Submodule UX overhaul update (was: What's cooking in git.git (Jan 2022, #07; Mon, 24))

From: Emily Shaffer <emilyshaffer@google.com>
Date: Jan 25, 2022, 19:45 UTC
Message-ID: <YfBTRuPrGGjepe+D@google.com>
In-Reply-To: <xmqq35lc53e9.fsf@gitster.g>

Finally finding the time to write up another "what's cooking as pertain to overhauling submodules UX" reply.

On Mon, Jan 24, 2022 at 11:39:58AM -0800, Junio C Hamano wrote:

Show 19 quoted lines

> --------------------------------------------------
> [Stalled]
> 
> * ar/submodule-update (2021-10-13) 9 commits
>  . submodule--helper: rename helper functions
>  . submodule--helper: remove unused helpers
>  . submodule: move core cmd_update() logic to C
>  . submodule--helper: run update using child process struct
>  . submodule--helper: allow setting superprefix for init_submodule()
>  . submodule--helper: refactor get_submodule_displaypath()
>  . submodule--helper: rename helpers for update-clone
>  . submodule--helper: get remote names from any repository
>  . submodule--helper: split up ensure_core_worktree()
> 
>  Rewrite of "git submodule update" in C.
> 
>  Expecting a reroll?
>  cf. <YWiXL+plA7GHfuVv@google.com>
>  source: <20211013051805.45662-10-raykar.ath@gmail.com>

Sounds like Glen is interested in this series with regard to some edge cases about cloning newly added submodules during a 'git fetch --recurse-submodules'. I'm curious to hear from Atharva about the status of this work :)

Show 10 quoted lines

> --------------------------------------------------
> [Cooking]
> 
> * js/apply-partial-clone-filters-recursively (2022-01-21) 1 commit
>  - clone, submodule: pass partial clone filters to submodules
> 
>  "git clone --filter=... --recurse-submodules" only makes the
>  top-level a partial clone, while submodules are fully cloned.  This
>  behaviour is changed to pass the same filter down to the submodules.
>  source: <50ebf7bd39adf34fa4ada27cd433d81b5381abe5.1642735881.git.steadmon@google.com>

Could use some comments on this. Looks like so far only Junio has replied. If anybody's interested, this one is a very big deal for us in terms of scaling Git for very large repos, or for superprojects with many submodules, or both. :)

Show 17 quoted lines

> 
> * ds/sparse-checkout-requires-per-worktree-config (2022-01-14) 6 commits
>  . worktree: copy sparse-checkout patterns and config on add
>  . sparse-checkout: use repo_config_set_worktree_gently()
>  . config: add repo_config_set_worktree_gently()
>  . worktree: add 'init-worktree-config' subcommand
>  . config: make some helpers repo-aware
>  . setup: use a repository when upgrading format
> 
>  "git sparse-checkout" wants to work with per-worktree configration,
>  but did not work well in a worktree attached to a bare repository.
> 
>  Expecting an update.
>  cf. <1db0f601-4769-15c0-cd58-ecddfa1fc9d5@gmail.com>
>  Introduces new leaks.
>  cf. https://github.com/git/git/runs/4823667255?check_suite_focus=true
>  source: <pull.1101.v3.git.1640727143.gitgitgadget@gmail.com>

Looks like this is getting some conversation from Eric and Elijah, but I'll call my team's attention to it too - sounds really useful.

Show 14 quoted lines

> 
> * gc/branch-recurse-submodules (2022-01-10) 6 commits
>  - branch: add --recurse-submodules option for branch creation
>  - builtin/branch: clean up action-picking logic in cmd_branch()
>  - branch: add a dry_run parameter to create_branch()
>  - branch: make create_branch() always create a branch
>  - branch: move --set-upstream-to behavior to dwim_and_setup_tracking()
>  - Merge branch 'js/branch-track-inherit' into gc/branch-recurse-submodules
> 
>  "git branch" learned the "--recurse-submodules" option.
> 
>  Expecting a reroll.
>  cf. <kl6l7db6kvp2.fsf@chooglen-macbookpro.roam.corp.google.com>
>  source: <20211220233459.45739-1-chooglen@google.com>

Looks like the reroll came just after this mail: <20220124204442.39353-1-chooglen@google.com>

Show 11 quoted lines

> * jt/conditional-config-on-remote-url (2022-01-18) 2 commits
>   (merged to 'next' on 2022-01-19 at 3c2df266eb)
>  + config: include file if remote URL matches a glob
>  + config: make git_config_include() static
> 
>  The conditional inclusion mechanism of configuration files using
>  "[includeIf <condition>]" learns to base its decision on the
>  URL of the remote repository the repository interacts with.
> 
>  Will cook in 'next'.
>  source: <cover.1642527965.git.jonathantanmy@google.com>

Very cool, thanks. We are hoping to be able to use this for distributing configs like sparsity filters, submodule behavior, config-based hooks, etc. in our controlled-environment /etc/gitconfig workplace. This kind of conditional include will let us say something like, "teams who are using https://some/remote have agreed to be guinea pigs" and curate a special config for the things we want them to try. I'm very excited to see it :)

Show 26 quoted lines

> 
> * ab/config-based-hooks-2 (2022-01-07) 17 commits
>   (merged to 'next' on 2022-01-19 at 594b6da22c)
>  + run-command: remove old run_hook_{le,ve}() hook API
>  + receive-pack: convert push-to-checkout hook to hook.h
>  + read-cache: convert post-index-change to use hook.h
>  + commit: convert {pre-commit,prepare-commit-msg} hook to hook.h
>  + git-p4: use 'git hook' to run hooks
>  + send-email: use 'git hook run' for 'sendemail-validate'
>  + git hook run: add an --ignore-missing flag
>  + hooks: convert worktree 'post-checkout' hook to hook library
>  + hooks: convert non-worktree 'post-checkout' hook to hook library
>  + merge: convert post-merge to use hook.h
>  + am: convert applypatch-msg to use hook.h
>  + rebase: convert pre-rebase to use hook.h
>  + hook API: add a run_hooks_l() wrapper
>  + am: convert {pre,post}-applypatch to use hook.h
>  + gc: use hook library for pre-auto-gc hook
>  + hook API: add a run_hooks() wrapper
>  + hook: add 'run' subcommand
> 
>  More "config-based hooks".
> 
>  Will cook in 'next'.
>  source: <cover-v6-00.17-00000000000-20211222T035755Z-avarab@gmail.com>
>

Very excited to see this one go in and looking forward to the next set! By the way, this reduced the number of patches Google is carrying internally on top of 'next' from 46 to 29. :) (All but 7 of those are the rest of config-based-hooks.)

It might seem slightly selfish for me to include config-based-hooks in the "submodules update" letter, but this is actually very important for submodules too - without config-based hooks, we don't have a good way to distribute a hook across an entire Git superproject-and-submodules codebase. With config-based hooks + "config.superproject" shared between the submodules and superproject, this becomes a very easy story :)

Thanks!

Show 14 quoted lines

> 
> * es/superproject-aware-submodules (2021-11-18) 5 commits
>  - submodule: use config to find superproject worktree
>  - submodule: record superproject gitdir during 'update'
>  - submodule: record superproject gitdir during absorbgitdirs
>  - introduce submodule.superprojectGitDir record
>  - t7400-submodule-basic: modernize inspect() helper
> 
>  A configuration variable in a submodule points at the location of
>  the superproject it is bound to (RFC).
> 
>  Expecting a reroll.
>  cf. <20211117234300.2598132-1-jonathantanmy@google.com>
>  source: <20211117005701.371808-1-emilyshaffer@google.com>

Hoping to reroll this week. I have been ignoring this to make more room for the discussion and followup work from <YbvBvch8JcHED+A9@google.com>, which I think is all distributed now.

By the way, speaking of that mail, for the past 2 weeks or so all the patches from the Google Git team - so me, Glen Choo, Josh Steadmon, Jonathan Tan, Jonathan Nieder, and Calvin Wan - have been undergoing an internal review of cover letter, --- description, and commit message before being sent. If anybody has feedback on how those parts of our series are looking lately, I'd be really interested in hearing it.

Thanks all.
 - Emily