threads / patch / 31064

patchSolve git-submodule issues with detached work trees

Subject: [PATCH] Solve git-submodule issues with detached work trees

## tl;dr

11 messages between Jul 22, 2012 and Jul 23, 2012. Diffs are folded; open one to read it.

replies: 10people: 4as markdown or json

Daniel Graña· Jul 22, 2012, 14:49 UTC · lore

A common way to track dotfiles with git is using GIT_DIR and GIT_WORK_TREE to move repository out of ~/.git with something like:

    git init --bare ~/.dotfiles
    alias dotfiles="GIT_DIR=~/.dotfiles GIT_WORK_TREE=~ git"
    dotfiles add ~/.bashrc
    dotfiles commit -a -m "add my bashrc"
    ...
but git-submodule complains when trying to add submodules:
    dotfiles submodule add http://path.to/submodule
    fatal: working tree '/home/user' already exists.
    git --git-dir ~/.dotfiles submodule add http://path.to/submodule
    fatal: /usr/lib/git-core/git-submodule cannot be used without a
working tree.
Signed-off-by: Daniel Graña <dangra@gmail.com>
---
 git-submodule.sh                       |    7 +++-
 t/t7409-submodule-detached-worktree.sh |   61 ++++++++++++++++++++++++++++++++
 2 files changed, 66 insertions(+), 2 deletions(-)
 create mode 100755 t/t7409-submodule-detached-worktree.sh
Show changes to 2 files +66 −2

git-submodule.sh, t/t7409-submodule-detached-worktree.sh

diff --git a/git-submodule.sh b/git-submodule.sh
index 5629d87..88ee4ea 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -181,8 +181,11 @@ module_clone()
 		rm -f "$gitdir/index"
 	else
 		mkdir -p "$gitdir_base"
-		git clone $quiet -n ${reference:+"$reference"} \
-			--separate-git-dir "$gitdir" "$url" "$sm_path" ||
+		(
+			clear_local_git_env
+			git clone $quiet -n ${reference:+"$reference"} \
+				--separate-git-dir "$gitdir" "$url" "$sm_path"
+		) ||
 		die "$(eval_gettext "Clone of '\$url' into submodule path
'\$sm_path' failed")"
 	fi

diff --git a/t/t7409-submodule-detached-worktree.sh
b/t/t7409-submodule-detached-worktree.sh
new file mode 100755
index 0000000..db75642
--- /dev/null
+++ b/t/t7409-submodule-detached-worktree.sh
@@ -0,0 +1,61 @@
+#!/bin/sh
+#
+# Copyright (c) 2012 Daniel Graña
+#
+
+test_description='Test submodules on detached working tree
+
+This test verifies that "git submodule" initialization, update and
addition works
+on detahced working trees
+'
+
+TEST_NO_CREATE_REPO=1
+. ./test-lib.sh
+
+test_expect_success 'submodule on detached working tree' '
+	git init --bare remote &&
+	test_create_repo bundle1 &&
+	(cd bundle1 && test_commit "shoot") &&
+	mkdir home &&
+	(
+		cd home &&
+		export GIT_WORK_TREE="$(pwd)" GIT_DIR="$(pwd)/.dotfiles" &&
+		git clone --bare ../remote .dotfiles &&
+		git submodule add ../bundle1 .vim/bundle/sogood &&
+		test_commit "sogood" &&
+		git push origin master
+	) &&
+	mkdir home2 &&
+	(
+		cd home2 &&
+		export GIT_WORK_TREE="$(pwd)" GIT_DIR="$(pwd)/.dotfiles" &&
+		git clone --bare ../remote .dotfiles &&
+		git submodule update --init
+	)
+'
+
+test_expect_success 'submodule on detached working pointed by core.worktree' '
+	mkdir home3 &&
+	(
+		cd home3 &&
+		export GIT_DIR="$(pwd)/.dotfiles" &&
+		git clone --bare ../remote "$GIT_DIR" &&
+		git config core.bare false &&
+		git config core.worktree .. &&
+		git submodule add ../bundle1 .vim/bundle/dupe &&
+		test_commit "dupe" &&
+		git push origin master
+	) &&
+	(
+		cd home &&
+		export GIT_DIR="$(pwd)/.dotfiles" &&
+		git config core.bare false &&
+		git config core.worktree .. &&
+		git pull &&
+		git submodule update &&
+		git submodule status &&
+		test -d .vim/bundle/dupe
+	)
+'
+
+test_done

-- 
1.7.5.4
Junio C Hamano· Jul 23, 2012, 05:09 UTC · re: Daniel Graña · lore

Re: [PATCH] Solve git-submodule issues with detached work trees

Daniel Graña <dangra@gmail.com> writes:
Show 21 quoted lines
> A common way to track dotfiles with git is using GIT_DIR and
> GIT_WORK_TREE to move repository out of ~/.git with something like:
>
>     git init --bare ~/.dotfiles
>     alias dotfiles="GIT_DIR=~/.dotfiles GIT_WORK_TREE=~ git"
>
>     dotfiles add ~/.bashrc
>     dotfiles commit -a -m "add my bashrc"
>     ...
>
> but git-submodule complains when trying to add submodules:
>
>     dotfiles submodule add http://path.to/submodule
>     fatal: working tree '/home/user' already exists.
>
>     git --git-dir ~/.dotfiles submodule add http://path.to/submodule
>     fatal: /usr/lib/git-core/git-submodule cannot be used without a
> working tree.
>
> Signed-off-by: Daniel Graña <dangra@gmail.com>
> ---

I think this is in line with what we discussed earlier on list when the interaction between GIT_DIR/GIT_WORK_TREE and submodules came up the last time. Jens?

Show 22 quoted lines
>  git-submodule.sh                       |    7 +++-
>  t/t7409-submodule-detached-worktree.sh |   61 ++++++++++++++++++++++++++++++++
>  2 files changed, 66 insertions(+), 2 deletions(-)
>  create mode 100755 t/t7409-submodule-detached-worktree.sh
>
> diff --git a/git-submodule.sh b/git-submodule.sh
> index 5629d87..88ee4ea 100755
> --- a/git-submodule.sh
> +++ b/git-submodule.sh
> @@ -181,8 +181,11 @@ module_clone()
>  		rm -f "$gitdir/index"
>  	else
>  		mkdir -p "$gitdir_base"
> -		git clone $quiet -n ${reference:+"$reference"} \
> -			--separate-git-dir "$gitdir" "$url" "$sm_path" ||
> +		(
> +			clear_local_git_env
> +			git clone $quiet -n ${reference:+"$reference"} \
> +				--separate-git-dir "$gitdir" "$url" "$sm_path"
> +		) ||
>  		die "$(eval_gettext "Clone of '\$url' into submodule path
> '\$sm_path' failed")"
Line-wrapped broken patch?
Show 18 quoted lines
>  	fi
>
> diff --git a/t/t7409-submodule-detached-worktree.sh
> b/t/t7409-submodule-detached-worktree.sh
> new file mode 100755
> index 0000000..db75642
> --- /dev/null
> +++ b/t/t7409-submodule-detached-worktree.sh
> @@ -0,0 +1,61 @@
> +#!/bin/sh
> +#
> +# Copyright (c) 2012 Daniel Graña
> +#
> +
> +test_description='Test submodules on detached working tree
> +
> +This test verifies that "git submodule" initialization, update and
> addition works
Line-wrapped broken patch?
Show 19 quoted lines
> +on detahced working trees
> +'
> +
> +TEST_NO_CREATE_REPO=1
> +. ./test-lib.sh
> +
> +test_expect_success 'submodule on detached working tree' '
> +	git init --bare remote &&
> +	test_create_repo bundle1 &&
> +	(cd bundle1 && test_commit "shoot") &&
> +	mkdir home &&
> +	(
> +		cd home &&
> +		export GIT_WORK_TREE="$(pwd)" GIT_DIR="$(pwd)/.dotfiles" &&
> +		git clone --bare ../remote .dotfiles &&
> +		git submodule add ../bundle1 .vim/bundle/sogood &&
> +		test_commit "sogood" &&
> +		git push origin master
> +	) &&

Don't you want to verify not just commands succeed, but leave a reasonable result, e.g. by running "git rev-parse HEAD" in "remote" and comparing the output with "git rev-parse HEAD" in .dotfiles, or something?

Show 6 quoted lines
> +	mkdir home2 &&
> +	(
> +		cd home2 &&
> +		export GIT_WORK_TREE="$(pwd)" GIT_DIR="$(pwd)/.dotfiles" &&
> +		git clone --bare ../remote .dotfiles &&
> +		git submodule update --init

Likewise. What state, other than "submodule update --init" does not barf, do you expect to see?

Show 14 quoted lines
> +	)
> +'
> +
> +test_expect_success 'submodule on detached working pointed by core.worktree' '
> +	mkdir home3 &&
> +	(
> +		cd home3 &&
> +		export GIT_DIR="$(pwd)/.dotfiles" &&
> +		git clone --bare ../remote "$GIT_DIR" &&
> +		git config core.bare false &&
> +		git config core.worktree .. &&
> +		git submodule add ../bundle1 .vim/bundle/dupe &&
> +		test_commit "dupe" &&
> +		git push origin master
Likewise.
Show 10 quoted lines
> +	) &&
> +	(
> +		cd home &&
> +		export GIT_DIR="$(pwd)/.dotfiles" &&
> +		git config core.bare false &&
> +		git config core.worktree .. &&
> +		git pull &&
> +		git submodule update &&
> +		git submodule status &&
> +		test -d .vim/bundle/dupe

Likewise. Is there something you want to verify in the branches in the submodule and its working tree, other than the existence of that directory?

> +	)
> +'
> +
> +test_done
Jens Lehmann· Jul 23, 2012, 17:38 UTC · re: Junio C Hamano · lore

Re: [PATCH] Solve git-submodule issues with detached work trees

Am 23.07.2012 07:09, schrieb Junio C Hamano:
Show 27 quoted lines
> Daniel Graña <dangra@gmail.com> writes:
> 
>> A common way to track dotfiles with git is using GIT_DIR and
>> GIT_WORK_TREE to move repository out of ~/.git with something like:
>>
>>     git init --bare ~/.dotfiles
>>     alias dotfiles="GIT_DIR=~/.dotfiles GIT_WORK_TREE=~ git"
>>
>>     dotfiles add ~/.bashrc
>>     dotfiles commit -a -m "add my bashrc"
>>     ...
>>
>> but git-submodule complains when trying to add submodules:
>>
>>     dotfiles submodule add http://path.to/submodule
>>     fatal: working tree '/home/user' already exists.
>>
>>     git --git-dir ~/.dotfiles submodule add http://path.to/submodule
>>     fatal: /usr/lib/git-core/git-submodule cannot be used without a
>> working tree.
>>
>> Signed-off-by: Daniel Graña <dangra@gmail.com>
>> ---
> 
> I think this is in line with what we discussed earlier on list when
> the interaction between GIT_DIR/GIT_WORK_TREE and submodules came up
> the last time.  Jens?

Yes, I think this is the only way submodules in current git can be used with the GIT_DIR and GIT_WORK_TREE environment variables: set them when adding or initializing the submodule and always use the same settings when accessing them later. Daniel's dotfile alias achieves exactly that, so his fix looks good. But I agree the tests should be improved as you already pointed out.

Junio C Hamano· Jul 23, 2012, 18:21 UTC · re: Jens Lehmann · lore

Re: [PATCH] Solve git-submodule issues with detached work trees

Jens Lehmann <Jens.Lehmann@web.de> writes:
Show 35 quoted lines
> Am 23.07.2012 07:09, schrieb Junio C Hamano:
>> Daniel Graña <dangra@gmail.com> writes:
>> 
>>> A common way to track dotfiles with git is using GIT_DIR and
>>> GIT_WORK_TREE to move repository out of ~/.git with something like:
>>>
>>>     git init --bare ~/.dotfiles
>>>     alias dotfiles="GIT_DIR=~/.dotfiles GIT_WORK_TREE=~ git"
>>>
>>>     dotfiles add ~/.bashrc
>>>     dotfiles commit -a -m "add my bashrc"
>>>     ...
>>>
>>> but git-submodule complains when trying to add submodules:
>>>
>>>     dotfiles submodule add http://path.to/submodule
>>>     fatal: working tree '/home/user' already exists.
>>>
>>>     git --git-dir ~/.dotfiles submodule add http://path.to/submodule
>>>     fatal: /usr/lib/git-core/git-submodule cannot be used without a
>>> working tree.
>>>
>>> Signed-off-by: Daniel Graña <dangra@gmail.com>
>>> ---
>> 
>> I think this is in line with what we discussed earlier on list when
>> the interaction between GIT_DIR/GIT_WORK_TREE and submodules came up
>> the last time.  Jens?
>
> Yes, I think this is the only way submodules in current git can
> be used with the GIT_DIR and GIT_WORK_TREE environment variables:
> set them when adding or initializing the submodule and always use
> the same settings when accessing them later. Daniel's dotfile
> alias achieves exactly that, so his fix looks good. But I agree
> the tests should be improved as you already pointed out.

Thanks for a quick review. The "the only way ... in current git can be used" part makes it sound as if it is a somewhat suboptimal ugly workaround, but if that is what you meant, what is a more optimal and less ugly way that you have in mind?

If you want to move .git out of way with GIT_DIR and if you want to sit somewhere different from the top of your working tree, you must use GIT_WORK_TREE (or core.worktree) to tell where the top resides, whether your project use submodules or not, so I do not think it is an ugly workaround but is the one true way to use such a dismembered layout, I would think.

Jens Lehmann· Jul 23, 2012, 20:02 UTC · re: Junio C Hamano · lore

Re: [PATCH] Solve git-submodule issues with detached work trees

Am 23.07.2012 20:21, schrieb Junio C Hamano:
Show 42 quoted lines
> Jens Lehmann <Jens.Lehmann@web.de> writes:
> 
>> Am 23.07.2012 07:09, schrieb Junio C Hamano:
>>> Daniel Graña <dangra@gmail.com> writes:
>>>
>>>> A common way to track dotfiles with git is using GIT_DIR and
>>>> GIT_WORK_TREE to move repository out of ~/.git with something like:
>>>>
>>>>     git init --bare ~/.dotfiles
>>>>     alias dotfiles="GIT_DIR=~/.dotfiles GIT_WORK_TREE=~ git"
>>>>
>>>>     dotfiles add ~/.bashrc
>>>>     dotfiles commit -a -m "add my bashrc"
>>>>     ...
>>>>
>>>> but git-submodule complains when trying to add submodules:
>>>>
>>>>     dotfiles submodule add http://path.to/submodule
>>>>     fatal: working tree '/home/user' already exists.
>>>>
>>>>     git --git-dir ~/.dotfiles submodule add http://path.to/submodule
>>>>     fatal: /usr/lib/git-core/git-submodule cannot be used without a
>>>> working tree.
>>>>
>>>> Signed-off-by: Daniel Graña <dangra@gmail.com>
>>>> ---
>>>
>>> I think this is in line with what we discussed earlier on list when
>>> the interaction between GIT_DIR/GIT_WORK_TREE and submodules came up
>>> the last time.  Jens?
>>
>> Yes, I think this is the only way submodules in current git can
>> be used with the GIT_DIR and GIT_WORK_TREE environment variables:
>> set them when adding or initializing the submodule and always use
>> the same settings when accessing them later. Daniel's dotfile
>> alias achieves exactly that, so his fix looks good. But I agree
>> the tests should be improved as you already pointed out.
> 
> Thanks for a quick review.  The "the only way ... in current git can
> be used" part makes it sound as if it is a somewhat suboptimal ugly
> workaround, but if that is what you meant, what is a more optimal
> and less ugly way that you have in mind?

I don't see it as an ugly workaround, I just don't see a way to get the flexibility GIT_DIR and GIT_WORK_TREE offer for repos without submodules (e.g. put the work tree temporarily somewhere else for deployment or compare the current work tree with the history in a different git directory) without changing core git. I have no idea how many people use this flexibility but when submodules are involved, you have to stick to the choices you made in the first place or things will break. Right now the two relative links tie exactly two directories together as git dir and work tree. I suspect e.g. some setups using symlinks could benefit if we could get rid of one of them or even both.

We could get rid of the core.worktree setting by assuming that the directory a gitfile was found in is the root of the repo's work tree (unless configured otherwise). And we could introduce a new gitfile notation which gives the git dir relative to a parent repo's git dir (e.g. "parent: modules/name" or such). But I'm not sure if it is worth the hassle.

Show 6 quoted lines
> If you want to move .git out of way with GIT_DIR and if you want to
> sit somewhere different from the top of your working tree, you must
> use GIT_WORK_TREE (or core.worktree) to tell where the top resides,
> whether your project use submodules or not, so I do not think it is
> an ugly workaround but is the one true way to use such a dismembered
> layout, I would think.
That is a perfect use case and we should make it work.
Junio C Hamano· Jul 23, 2012, 20:34 UTC · re: Jens Lehmann · lore

Re: [PATCH] Solve git-submodule issues with detached work trees

Jens Lehmann <Jens.Lehmann@web.de> writes:
> We could get rid of the core.worktree setting by assuming that the
> directory a gitfile was found in is the root of the repo's work
> tree (unless configured otherwise).

Now you lost me. If you have .git that is not a directory but is a gitfile, then you do not need GIT_DIR nor GIT_WORK_TREE in the first place, no?

Jens Lehmann· Jul 23, 2012, 23:17 UTC · re: Junio C Hamano · lore

Re: [PATCH] Solve git-submodule issues with detached work trees

Am 23.07.2012 22:34, schrieb Junio C Hamano:
Show 9 quoted lines
> Jens Lehmann <Jens.Lehmann@web.de> writes:
> 
>> We could get rid of the core.worktree setting by assuming that the
>> directory a gitfile was found in is the root of the repo's work
>> tree (unless configured otherwise).
> 
> Now you lost me.  If you have .git that is not a directory but is a
> gitfile, then you do not need GIT_DIR nor GIT_WORK_TREE in the first
> place, no?

Not inside the submodule, me thinks they only make sense in the superproject (that's why we clean the environment before working inside the submodule). But setting the superproject's GIT_WORK_TREE to something else won't work for an already initialized submodule, as the core.worktree setting will still point to the old work tree which was set when the submodule was initialized. One could expect the submodule's work tree to be $GIT_WORK_TREE/$sm_path when GIT_WORK_TREE is set, but it isn't.

Junio C Hamano· Jul 23, 2012, 23:42 UTC · re: Jens Lehmann · lore

Re: [PATCH] Solve git-submodule issues with detached work trees

Jens Lehmann <Jens.Lehmann@web.de> writes:
> Not inside the submodule, me thinks they only make sense in the
> superproject (that's why we clean the environment before working
> inside the submodule).

Yes, that is fundamental and the only sane behaviour that comes from what submodules are. They are free-standing projects. Not clearing these environments when Git recurses into submodule was simply a bug (the original bug was that we added recursion without thinking these things through).

Hence...
> But setting the superproject's GIT_WORK_TREE
> to something else won't work for an already initialized submodule,

... if you point the _root_ of the toplevel superproject with GIT_WORK_TREE (and the repository of the superproject with GIT_DIR), and chdir into one of its submodule's working tree, we shouldn't say $GIT_WORK_TREE/$smpath is the submodule world. That will make it impossible to work only on submodules by setting GIT_WORK_TREE to point at its root level (and the submodule repository with GIT_DIR).

Daniel Graña· Jul 23, 2012, 18:22 UTC · re: Jens Lehmann · lore

Re: [PATCH] Solve git-submodule issues with detached work trees

On Mon, Jul 23, 2012 at 2:38 PM, Jens Lehmann <Jens.Lehmann@web.de> wrote:
Show 35 quoted lines
> Am 23.07.2012 07:09, schrieb Junio C Hamano:
>> Daniel Graña <dangra@gmail.com> writes:
>>
>>> A common way to track dotfiles with git is using GIT_DIR and
>>> GIT_WORK_TREE to move repository out of ~/.git with something like:
>>>
>>>     git init --bare ~/.dotfiles
>>>     alias dotfiles="GIT_DIR=~/.dotfiles GIT_WORK_TREE=~ git"
>>>
>>>     dotfiles add ~/.bashrc
>>>     dotfiles commit -a -m "add my bashrc"
>>>     ...
>>>
>>> but git-submodule complains when trying to add submodules:
>>>
>>>     dotfiles submodule add http://path.to/submodule
>>>     fatal: working tree '/home/user' already exists.
>>>
>>>     git --git-dir ~/.dotfiles submodule add http://path.to/submodule
>>>     fatal: /usr/lib/git-core/git-submodule cannot be used without a
>>> working tree.
>>>
>>> Signed-off-by: Daniel Graña <dangra@gmail.com>
>>> ---
>>
>> I think this is in line with what we discussed earlier on list when
>> the interaction between GIT_DIR/GIT_WORK_TREE and submodules came up
>> the last time.  Jens?
>
> Yes, I think this is the only way submodules in current git can
> be used with the GIT_DIR and GIT_WORK_TREE environment variables:
> set them when adding or initializing the submodule and always use
> the same settings when accessing them later. Daniel's dotfile
> alias achieves exactly that, so his fix looks good. But I agree
> the tests should be improved as you already pointed out.
Hi Jens, Junio.

Thanks for reviewing the patch, glad to hear this is acceptable way to fix my itch.

I will work soon on getting the test cases improved and the patch submitted without line wrapping.

thanks again, Daniel.

Richard Hartmann· Jul 23, 2012, 18:25 UTC · re: Junio C Hamano · lore

Re: [PATCH] Solve git-submodule issues with detached work trees

On Mon, Jul 23, 2012 at 7:09 AM, Junio C Hamano <gitster@pobox.com> wrote:
> I think this is in line with what we discussed earlier on list when
> the interaction between GIT_DIR/GIT_WORK_TREE and submodules came up
> the last time.  Jens?
Yes, it is.

I still have your email marked as TODO to get back to you as I am still unsure how to solve this whole mess in a way that works with vcsh[1].

Richard
PS: Thanks for being so dedicated you all. I really do appreciate this
more than you most likely suspect.
[1] vcsh

← back to recent threads