Re: [PATCH v4 3/4] submodule: support running in multiple worktree setup

From: Duy Nguyen <pclouds@gmail.com>
Date: Jul 27, 2016, 15:40 UTC
Message-ID: <CACsJy8Dhd2YLmNoRN=j0PQeyG+8=8MALsiw611HMhi2zk_8ouQ@mail.gmail.com>
In-Reply-To: <CAGZ79kYGj7q=SQyHvFdmXasJppTVw56xSBMiSERdx22B+A68gQ@mail.gmail.com>

On Tue, Jul 26, 2016 at 8:15 PM, Stefan Beller <sbeller@google.com> wrote:

>> How to store stuff in .git is the implementation details that the user
>> does not care about.
>
> They do unfortunately. :(

Well.. i mean the structure of .git. If .git gets big, yeah many people will get pissed.

> My sudden interest in worktrees came up when I learned the
> `--reference` flag for submodule operations is broken for
> our use case, and instead of fixing the `--reference` flag,
> I think the worktree approach is generally saner (i.e. with the

I don't know exactly what that --reference problem is, but keep in mind you still have to support single-worktree use case. If it's broken in single-worktree, somebody still has to fix it.

> references you may have nasty gc issues IIUC, but in the
> worktree world gc knows about all the working trees, detached
> heads and branches.)

True, but not yet. git-gc now does not know about all detached heads and reflogs and have cause grief for some people. This should be fixed soon.

Show 60 quoted lines

>> As long as we keep the behavior the same (they
>> can still "git submodule init" and stuff in the new worktree), sharing
>> the same object store makes sense (pros: lower disk consumption, cons:
>> none).
>
> So I think the current workflow for submodules
> may need some redesign anyway as the submodule
> commands were designed with a strict "one working
> tree only" assumption.
>
> Submodule URLs  are stored in 3 places:
>  A) In the .gitmodules file of the superproject
>  B) In the option submodule.<name>.URL in the superproject
>  C) In the remote.origin.URL in the submodule
>
> A) is a recommendation from the superproject to make life
> of downstream easier to find and setup the whole thing.
> You can ignore that if you want, though generally a caring
> upstream provides good URLs here.
>
> C) is where we actually fetch from (and hope it has all
> the sha1s that are recorded as gitlinks in the superproejct)
>
> B) seems like a hack to enable the workflow as below:
>
> Current workflow for handling submodule URLs:
>  1) Clone the superproject
>  2) Run git submodule init on desired submodules
>  3) Inspect .git/config to see if any submodule URL needs adaption
>  4) Run git submodule update to obtain the submodules from
>     the configured place
>  5) In case of superproject adapting the URL
>     -> git submodule sync, which overwrites the submodule.<name>.URL in the
>     superprojects .git/config as well as configuring the
> remote."$remote".url in the submodule
>  6) In case of users desire to change the URL
>     -> No one command to solve it; possible workaround: edit
>     .gitmodules and git submodule sync, or configure  the submodule.<name>.URL
>     in the superprojects .git/config as well as configuring the
> remote."$remote".url in
>     the submodule separately. Although just changing the submodules remote works
>     just as well (until you remove and re-clone the submodule)
>
> One could imagine another workflow:
>  1) clone the superproject, which creates empty repositories for the
>     submodules
>  (2) from the prior workflow is gone
>  3) instead of inspecting .git/config you can directly manipulate the
>     remote.$remote.url configuration in the submodule.
>  4) Run git submodule update to obtain the submodules from
>     the configured place
>
> The current workflow is setup that way because historically you had
> the submodules .git dir inside the submodule, which would be gone
> if you deleted a submodule. So if you later checkout an earlier version'
> that had a submodule, you are missing the objects and more importantly
> configuration where to get them from.
>
> This is now fixed by keeping the actual submodules git dir inside
> the superprojects git dir.

Hmm.. sounds good, but I'm no judge when it comes to submodules :)

Show 5 quoted lines

>> Hmm.. I didn't realize this. But then I have never given much thought
>> about submodules, probably because I have an alternative solution for
>> it (or some of its use cases) anyway :)
>
> What is that?

Narrow clone (making progress but not there yet). I know it does not cover all cases (e.g. submodule can provide separate access control, and even using different dvcs system in theory).

Show 13 quoted lines

>> OK so it's already a problem. But if we keep sharing submodule stuff
>> in .git/config, there's a _new_ problem: when you "submodule init" a
>> worktree, .git/config is now tailored for the current worktree, when
>> you move back to the previous worktree, you need to "submodule init"
>> again.
>
> "Moving back" sounds like you use the worktree feature for short lived
> things only. (e.g. in the man page you refer to the hot fix your boss wants
> you to make urgently).
>
> I thought the worktree feature is more useful for long term "branches",
> e.g. I have one worktree of git now that tracks origin/master so I can
> use that to "make install" to repair my local broken version of git.

I use it for both. Sometimes you just want to fix something and not mess up your current worktree.

Show 5 quoted lines

> (I could have a worktree "continuous integration", where I only run tests
> in. I could have one worktree for Documentation changes only.)
>
> This long lived stuff probably doesn't make sense for the a single
> repository,

It does. You can switch branches in all worktrees. I have a worktree specifically for building mingw32 stuff (separate config.mak and stuff). When I'm done with a branch on my normal worktree, I could move over there, check out the same branch then try mingw32 build. If it fails I can fix it right right there and update the branch. When everything is ok, I just move back to my normal worktree and continue.

You can achieve the same thing with multiple clones, but it's inconvenient (you need to fetch and push...) and multiple clones take up more space.

Show 9 quoted lines

>> So moving to multiple worktrees setup changes how the user uses
>> submodule, not good in my opinion.
>
> Because the submodule user API is built on the strong assumption
> of "one working tree only", we have to at least slightly adapt.
>
> So instead of cloning a submodule in a worktree we could just
> setup a submodule worktree as well there?
> (i.e. that saves us network as well as disk)

You still need to clone once then somehow associate the clone with a submodule, I think.

> For worktrees there is no "worktree rm" as it would probably promise
> a bit more than the man pages suggestion of rm -rf $worktree && git
> worktree prune.

Oh there will be, sooooon :D I prefer not to do that command sequence manually, especially when "rm -rf" is involved. "git worktree remove" can refuse to delete when your worktree is dirty so you don't accidentally rm and cry later.

-- 
Duy