threads / patch / 35550

v2, 21 partsRe: [PATCH v2 00/21] Support multiple worktrees

Subject: Re: [PATCH v2 00/21] Support multiple worktrees

## tl;dr

7 messages between Dec 19, 2013 and Dec 28, 2013. Diffs are folded; open one to read it.

replies: 6people: 2as markdown or json

Duy Nguyen· Dec 19, 2013, 14:12 UTC · lore

I've got a better version [1] that fixes everything I can think of (there's still some room for improvements). I'm going to use it a bit longer before reposting again. But here's its basic design without going down to code

New .git file format includes two lines: -- 8< -- gitid: <id> gitdir: <path> -- 8< --

Which would set $GIT_COMMON_DIR to <path> and $GIT_DIR to <path>/repos/<id>. Repository split is the same as before, worktree stuff in $GIT_DIR, the rest in $GIT_COMMON_DIR. This .git file format takes precedence over core.worktree but can still be overriden with $GIT_WORK_TREE. The main interface to create new worktree is "git checkout --to".

"repos" belongs to $GIT_COMMON_DIR (i.e. shared across all checkouts). The new worktrees (which I call "linked checkouts") can also access HEAD of the original worktree via a virtual path "main/HEAD". This makes it possible for a linked checkout to detach HEAD of the main one.

There are three entries in repos/<id>: "gitdir" should point to the .git file that points it back here. Every time a linked checkout is used, git should update mtime of this "gitdir" file to help pruning. It should update the file content too if the repo is moved. "link" is a hardlink to .git file, if supported, again for pruning support. "locked", if exists, means no automatic pruning (e.g. the linked checkout is on a portable device).

The interesting thing is support for third party scripts (or hooks, maybe) so that they could work with both old and new git versions without some sort of git version/feature detection. Of course old git versions will only work with ordinary worktrees. To that end, "git rev-parse --git-dir" behavior could be changed by two environment variables. $GIT_ONE_PATH makes 'rev-parse --git-dir' return the .git _file_ in this case, which makes it much easier to pass the repo's checkout view around with "git --git-dir=... ".$GIT_COMMON_DIR_PATH makes 'rev-parse --git-dir' return $GIT_COMMON_DIR if it's from a linked checkout, or $GIT_DIR otherwise. This makes 'rev-parse --git-dir' falls back safely when running using old git versions. The last patch in [1] that updates git-completion.bash could demonstrate how it's used.

[1] https://github.com/pclouds/git.git checkout-new-worktree
-- 
Duy
Junio C Hamano· Dec 20, 2013, 20:32 UTC · re: Duy Nguyen · lore
Duy Nguyen <pclouds@gmail.com> writes:
Show 23 quoted lines
> I've got a better version [1] that fixes everything I can think of
> (there's still some room for improvements). I'm going to use it a bit
> longer before reposting again. But here's its basic design without
> going down to code
>
> New .git file format includes two lines:
> -- 8< --
> gitid: <id>
> gitdir: <path>
> -- 8< --
>
> Which would set $GIT_COMMON_DIR to <path> and $GIT_DIR to
> <path>/repos/<id>. Repository split is the same as before, worktree
> stuff in $GIT_DIR, the rest in $GIT_COMMON_DIR. This .git file format
> takes precedence over core.worktree but can still be overriden with
> $GIT_WORK_TREE. The main interface to create new worktree is "git
> checkout --to".
>
> "repos" belongs to $GIT_COMMON_DIR (i.e. shared across all checkouts).
> The new worktrees (which I call "linked checkouts") can also access
> HEAD of the original worktree via a virtual path "main/HEAD". This
> makes it possible for a linked checkout to detach HEAD of the main
> one.

I am not happy with the choice of "main/HEAD" that would squat on a good name for remote-tracking branch (i.e. s/origin/main/), though. $GIT_DIR/COMMON_HEAD perhaps?

Show 10 quoted lines
> The interesting thing is support for third party scripts (or hooks,
> maybe) so that they could work with both old and new git versions
> without some sort of git version/feature detection. Of course old git
> versions will only work with ordinary worktrees. To that end, "git
> rev-parse --git-dir" behavior could be changed by two environment
> variables. $GIT_ONE_PATH makes 'rev-parse --git-dir' return the .git
> _file_ in this case, which makes it much easier to pass the repo's
> checkout view around with "git --git-dir=... ".$GIT_COMMON_DIR_PATH
> makes 'rev-parse --git-dir' return $GIT_COMMON_DIR if it's from a
> linked checkout, or $GIT_DIR otherwise.
I do not understand why you need to go such a route.

Existing scripts that works only in a real repository will only know "git rev-parse --git-dir" as the way to get the real GIT_DIR and would not care about the "common" thing. Scripts updated to work well with the "common" thing needs to be aware of the "common" thing anyway, so adding "git rev-parse --common-git-dir" or somesuch that only these updated knows would be sufficient, no?

Duy Nguyen· Dec 21, 2013, 02:00 UTC · re: Junio C Hamano · lore
On Sat, Dec 21, 2013 at 3:32 AM, Junio C Hamano <gitster@pobox.com> wrote:
Show 29 quoted lines
> Duy Nguyen <pclouds@gmail.com> writes:
>
>> I've got a better version [1] that fixes everything I can think of
>> (there's still some room for improvements). I'm going to use it a bit
>> longer before reposting again. But here's its basic design without
>> going down to code
>>
>> New .git file format includes two lines:
>> -- 8< --
>> gitid: <id>
>> gitdir: <path>
>> -- 8< --
>>
>> Which would set $GIT_COMMON_DIR to <path> and $GIT_DIR to
>> <path>/repos/<id>. Repository split is the same as before, worktree
>> stuff in $GIT_DIR, the rest in $GIT_COMMON_DIR. This .git file format
>> takes precedence over core.worktree but can still be overriden with
>> $GIT_WORK_TREE. The main interface to create new worktree is "git
>> checkout --to".
>>
>> "repos" belongs to $GIT_COMMON_DIR (i.e. shared across all checkouts).
>> The new worktrees (which I call "linked checkouts") can also access
>> HEAD of the original worktree via a virtual path "main/HEAD". This
>> makes it possible for a linked checkout to detach HEAD of the main
>> one.
>
> I am not happy with the choice of "main/HEAD" that would squat on a
> good name for remote-tracking branch (i.e. s/origin/main/), though.
> $GIT_DIR/COMMON_HEAD perhaps?

It's not just about HEAD. Anything worktree-specific of the main checkout can be accessed this way, e.g. main/index, main/FETCH_HEAD.... and it's not exactly "common" because it's worktree info. Maybe 1ST_ as the prefix (e.g. 1ST_HEAD, 1ST_index...) ?

Show 19 quoted lines
>> The interesting thing is support for third party scripts (or hooks,
>> maybe) so that they could work with both old and new git versions
>> without some sort of git version/feature detection. Of course old git
>> versions will only work with ordinary worktrees. To that end, "git
>> rev-parse --git-dir" behavior could be changed by two environment
>> variables. $GIT_ONE_PATH makes 'rev-parse --git-dir' return the .git
>> _file_ in this case, which makes it much easier to pass the repo's
>> checkout view around with "git --git-dir=... ".$GIT_COMMON_DIR_PATH
>> makes 'rev-parse --git-dir' return $GIT_COMMON_DIR if it's from a
>> linked checkout, or $GIT_DIR otherwise.
>
> I do not understand why you need to go such a route.
>
> Existing scripts that works only in a real repository will only know
> "git rev-parse --git-dir" as the way to get the real GIT_DIR and
> would not care about the "common" thing.  Scripts updated to work
> well with the "common" thing needs to be aware of the "common" thing
> anyway, so adding "git rev-parse --common-git-dir" or somesuch that
> only these updated knows would be sufficient, no?

It simplifies the changes, if the new script is to work with both old and new git versions it may have to write

DIR=`git rev-parse --git-common-dir 2>/dev/null || git rev-parse --git-dir`
the env way makes it
DIR=`GIT_COMMON_DIR=1 git rev-parse --git-dir`
-- 
Duy
Junio C Hamano· Dec 22, 2013, 06:38 UTC · re: Duy Nguyen · lore
Duy Nguyen <pclouds@gmail.com> writes:
Show 9 quoted lines
>> I am not happy with the choice of "main/HEAD" that would squat on a
>> good name for remote-tracking branch (i.e. s/origin/main/), though.
>> $GIT_DIR/COMMON_HEAD perhaps?
>
> It's not just about HEAD. Anything worktree-specific of the main
> checkout can be accessed this way, e.g. main/index,
> main/FETCH_HEAD.... and it's not exactly "common" because it's
> worktree info. Maybe 1ST_ as the prefix (e.g. 1ST_HEAD, 1ST_index...)
> ?

Do we even need to expose them as ref-like things as a part of the external API/UI in the first place? For end-user scripts that want to operate in a real or borrowing worktree, there should be no reason to touch this "other" repository directly. Things like "if one of the wortrees tries to check out a branch that is already checked out elsewhere, error out" policy may need to consult the other worktrees via $GIT_COMMON_DIR mechanism but at that level we have all the control without contaminating end-user facing ref namespace in a way main/FETCH_HEAD... do. You said

    This makes it possible for a linked checkout to detach HEAD of
    the main one.

but I think that is misguided---it just makes it easier to confuse users, if done automatically and without any policy knob. It instead should error out, while saying which worktree has the branch in question checked out. After all, checking out a branch that is checked out in another worktree (not the common one) needs the same caution, so "main/HEAD" is not the only special one.

And if your updated "git checkout 'frotz'" gives a clear report of which worktree has the branch 'frotz' checked out, the user can go there to detach the HEAD in that worktree to detach with

	git -C $the_other_one checkout HEAD^0
if he chooses to.
Duy Nguyen· Dec 22, 2013, 08:44 UTC · re: Junio C Hamano · lore
On Sun, Dec 22, 2013 at 1:38 PM, Junio C Hamano <gitster@pobox.com> wrote:
Show 21 quoted lines
> Duy Nguyen <pclouds@gmail.com> writes:
>
>>> I am not happy with the choice of "main/HEAD" that would squat on a
>>> good name for remote-tracking branch (i.e. s/origin/main/), though.
>>> $GIT_DIR/COMMON_HEAD perhaps?
>>
>> It's not just about HEAD. Anything worktree-specific of the main
>> checkout can be accessed this way, e.g. main/index,
>> main/FETCH_HEAD.... and it's not exactly "common" because it's
>> worktree info. Maybe 1ST_ as the prefix (e.g. 1ST_HEAD, 1ST_index...)
>> ?
>
> Do we even need to expose them as ref-like things as a part of the
> external API/UI in the first place?  For end-user scripts that want
> to operate in a real or borrowing worktree, there should be no
> reason to touch this "other" repository directly.  Things like "if
> one of the wortrees tries to check out a branch that is already
> checked out elsewhere, error out" policy may need to consult the
> other worktrees via $GIT_COMMON_DIR mechanism but at that level we
> have all the control without contaminating end-user facing ref
> namespace in a way main/FETCH_HEAD... do.

No, external API/UI is just extra bonus. We need to (or should) do so in order to handle $GIT_COMMON_DIR/HEAD exactly like how we do normal refs. Given any ref, git_path(ref) gives the path to that ref, git_path("logs/%s", ref) gives the path of its reflog. By mapping special names to real paths behind git_path(), We can feed $GIT_COMMON_DIR/HEAD (under special names) to refs.c and it'll handle correctly without any changes for special cases.

Show 19 quoted lines
> You said
>
>     This makes it possible for a linked checkout to detach HEAD of
>     the main one.
>
> but I think that is misguided---it just makes it easier to confuse
> users, if done automatically and without any policy knob. It instead
> should error out, while saying which worktree has the branch in
> question checked out. After all, checking out a branch that is
> checked out in another worktree (not the common one) needs the same
> caution, so "main/HEAD" is not the only special one.
>
> And if your updated "git checkout 'frotz'" gives a clear report of
> which worktree has the branch 'frotz' checked out, the user can go
> there to detach the HEAD in that worktree to detach with
>
>         git -C $the_other_one checkout HEAD^0
>
> if he chooses to.

Jonathan mentions about the "checkout in portable device" case that would make the above a bit unnatural as you just can't "cd" there (git update-ref still works).

I don't see any problems with checking out a branch multiple times. I may want to try modifying something in the branch that will be thrown away in the end. It's when the user updates a branch that we should either error+reject or detach other checkouts. I guess it's up to the user to decide which way they want. The error+reject way may make the user hunt through dead checkouts waiting to be pruned. But we can start with error+reject then add an option to auto-detach.

-- 
Duy
Junio C Hamano· Dec 26, 2013, 17:12 UTC · re: Duy Nguyen · lore
Duy Nguyen <pclouds@gmail.com> writes:
Show 15 quoted lines
> On Sun, Dec 22, 2013 at 1:38 PM, Junio C Hamano <gitster@pobox.com> wrote:
>
>> Do we even need to expose them as ref-like things as a part of the
>> external API/UI in the first place?  For end-user scripts that want
>> to operate in a real or borrowing worktree, there should be no
>> reason to touch this "other" repository directly.  Things like "if
>> one of the wortrees tries to check out a branch that is already
>> checked out elsewhere, error out" policy may need to consult the
>> other worktrees via $GIT_COMMON_DIR mechanism but at that level we
>> have all the control without contaminating end-user facing ref
>> namespace in a way main/FETCH_HEAD... do.
>
> No, external API/UI is just extra bonus. We need to (or should) do so
> in order to handle $GIT_COMMON_DIR/HEAD exactly like how we do normal
> refs.
And that is what I consider a confusion-trigger, not a nice bonus.

I do not think it is particularly a good idea to contaminate end-user namespace for this kind of "peek another repository" special operation.

In order to handle your "multiple worktrees manipulating the same branch" case sanely, you need to be aware of not just the real repository your worktree is borrowing from, but also _all_ the other worktrees that borrow from that same real repository, so a single "main" virtual namespace will not cut it. You will be dealing with an unbounded set of HEADs, one for each such worktree.

Can't we do this by adding a "with this real repository" layer, e.g. "resolve HEAD wrt that repository", somewhat similar to how we peek into submodule namespaces?

Duy Nguyen· Dec 28, 2013, 02:46 UTC · re: Junio C Hamano · lore
On Fri, Dec 27, 2013 at 12:12 AM, Junio C Hamano <gitster@pobox.com> wrote:
Show 30 quoted lines
> Duy Nguyen <pclouds@gmail.com> writes:
>
>> On Sun, Dec 22, 2013 at 1:38 PM, Junio C Hamano <gitster@pobox.com> wrote:
>>
>>> Do we even need to expose them as ref-like things as a part of the
>>> external API/UI in the first place?  For end-user scripts that want
>>> to operate in a real or borrowing worktree, there should be no
>>> reason to touch this "other" repository directly.  Things like "if
>>> one of the wortrees tries to check out a branch that is already
>>> checked out elsewhere, error out" policy may need to consult the
>>> other worktrees via $GIT_COMMON_DIR mechanism but at that level we
>>> have all the control without contaminating end-user facing ref
>>> namespace in a way main/FETCH_HEAD... do.
>>
>> No, external API/UI is just extra bonus. We need to (or should) do so
>> in order to handle $GIT_COMMON_DIR/HEAD exactly like how we do normal
>> refs.
>
> And that is what I consider a confusion-trigger, not a nice bonus.
>
> I do not think it is particularly a good idea to contaminate
> end-user namespace for this kind of "peek another repository"
> special operation.
>
> In order to handle your "multiple worktrees manipulating the same
> branch" case sanely, you need to be aware of not just the real
> repository your worktree is borrowing from, but also _all_ the other
> worktrees that borrow from that same real repository, so a single
> "main" virtual namespace will not cut it. You will be dealing with
> an unbounded set of HEADs, one for each such worktree.

Yes. My problem is, while all secondary worktrees are in $GIT_DIR/repos and their HEADs can be accessed there with "repos/xxx/HEAD", the first worktree's HEAD can't be accessed this way because "HEAD" in a linked checkouts means repos/<my worktree>/HEAD.

> Can't we do this by adding a "with this real repository" layer,
> e.g. "resolve HEAD wrt that repository", somewhat similar to how we
> peek into submodule namespaces?
Hmm.. never thought of it like a "submodule". Thanks for the idea.
-- 
Duy

← back to recent threads