[git-users] Problem using detached worktrees with commands implemented in scripts

In Git, one can set up a repository with a "detached worktree", where the .git directory is not a subdirectory of the top directory of the work tree.

In general, Git commands on a repository with a detached worktree can be executed by cd'ing into the directory containing the .git directory, and executing the Git command there. E.g., "git add" and "git commit" execute as one would expect. (I think they can also be executed by cd'ing to the worktree and setting GIT_DIR.)

However, this approach does not work with "git filter-branch", which objects with "You need to run this command from the toplevel of the working tree."

I suspect that it does not work with other Git commands that are implemented with shell scripts. The problem appears to be in the git-sh-setup script, which is called by the Git shell scripts to set up the environment and do preliminary tests.

It seems to me that this inconsistency between the script commands and the binary commands can be fixed by updating git-sh-setup in this way:

--- git-sh-setup.Custom.orig 2013-06-20 12:59:45.000000000 -0400 +++ git-sh-setup 2013-10-07 22:34:06.719946134 -0400

worley@alum.mit.edu (Dale R. Worley) writes:

Eh? News to me; it might happened to have appeared to work by accident, but that is not by design.

IIRC, the intended use pattern (i.e. the change that introduced GIT_DIR and GIT_WORK_TREE environment variables was designed to support) for such a working tree is to:

 - export GIT_DIR that points at the correct .git directory;

 - export GIT_WORK_TREE that points at the correct top-level of such
   a working tree; and then

 - run the commands anywhere in the working tree, as if you did not
   export these two environment variables and instead had the .git
   directory at the usual place in the working tree.

It _is_ possible that we may have broken this canonical use pattern over time with more recent updates; I do not think we have extensive test coverage for "detached worktree" use case in the first place.

Not exporting GIT_DIR variable in sh-setup was done not by accident but as a very deliberate design choice, IIRC.

From: "Junio C Hamano" <gitster@pobox.com>

I think it is this part in Dale's original email

"However, this approach does not work with "git filter-branch", which objects with "You need to run this command from the toplevel of the working tree."

that is the problem Dale has seen. IIRC there are a few commands that do require to be run from the toplevel ('git bisect' I think is another), and the detection process for 'toplevel' may not work properly when in a separated work-tree environment.

Perhaps something to consider.

Philip

"Philip Oakley" <philipoakley@iee.org> writes:

Without GIT_WORK_TREE exported to point at the top-level, there is nothing that lets us "detect" it, as the working tree does not have ".git" directory to tell us to stop, no?

From: "Junio C Hamano" <gitster@pobox.com>

"No", but not in that way.

My point (to Dale) was, as you state, that the "cd to top level" was (IIUC) the probable causes of the fault, and that a documentation update would probably be appropriate for the discussion on exporting GIT_WORK_TREE, and that it would specifically mention those git commands that needed to "cd to top level", and hence would not work in such an environment. (I wasn't sure where the appropriate "cd to top level" function was)

An explanation here on the list wouldn't solve the problems for others who are yet to make the same mistake, hence the implied suggestion.

Philip 

"Philip Oakley" <philipoakley@iee.org> writes:

I understand what you mean by these last two lines. It was unclear to me which part of our documentation needs updating and how, and that was (and still is) what I was primarily interested in finding out.

From: "Junio C Hamano" <gitster@pobox.com>

I was expecting that the places would be in git(1) [git.txt] and config(1) [config.txt], in the enironment variables GIT_WORK_TREE section and core.worktree sections repectively. However what the right text would be hasn't been fully determined yet, as it should be clear about which commands don't follow the stated 'rules'. Dale's use case does appear to be stretching...

Philip 

From: "Philip Oakley" <philipoakley@iee.org>

A bit more looking gave that the cd_to_toplevel () in git-sh-setup.sh directly uses `git rev-parse --show-toplevel`, which simply returns work_tree (static char *work_tree; in environment.c, with comment /* This is set by setup_git_dir_gently() and/or git_default_config() */), apparently without a check for the GIT_WORK_TREE.

One option may be to either protect the cd_to_toplevel code with a check of `git rev-parse --local-env-vars` to see if GIT_WORK_TREE is present. Or create `git rev-parse --work-dir` to match `--git-dir`. This would be a code level fix. This makes the assumption that if a deteched GIT_WORK_TREE is set then it is the top level.

In terms of command scripts that use git-sh-setup.sh we have a longish list, so a full list in the documentation is probably unreasonable (which suggests that a code fix would be more apprpriate)

commands:

git-am git-bisect git-filter-branch git-instaweb git-lost-found git-merge-one-file git-mergetool git-pull git-quiltimport git-rebase git-repack git-request-pull git-stash git-submodule git-web--browse

git\contrib\*various*

Philip

Philip Oakley wrote:

Getting closer. :)

The usual way to use GIT_WORK_TREE is along with GIT_DIR. That takes you into the setup_explicit_git_dir() codepath, which does respect GIT_WORK_TREE as it should. (setup_discovered_git_dir does, too.)

The strange behavior you ran into is that unlike, say, git-pull.sh and git-am.sh, filter-branch does not set SUBDIRECTORY_OK, store the prefix from 'git rev-parse --show-prefix', and then cd_to_toplevel at the top of the script. In other words, nobody bothered to make it work from anywhere other than the toplevel of the worktree to begin with, and nobody wanted it enough to fix it later.

Hope that helps, Jonathan

From: "Jonathan Nieder" <jrnieder@gmail.com>

It was Dale that had the problem, I was just suggesting where he might want to look... ;-)

When re-checking the manual's git(1) env variable section the comment that implies this didn't read well "The value will not be used in combination...". The section probably needs that to be stated explicitly ("an exported GIT_WORK_TREE is ignored if GIT_DIR is not set").

I maybe wrong, but I thought that in Dale's case he was already at the same level as the GIT_WORK_TREE he had set, so may not have expected to need a cd_to_toplevel

Dale did propose a patch in http://thread.gmane.org/gmane.comp.version-control.git/236260 as the error was from later in the setup script.

Philip 

It seems to me that what is missing is a description of the circumstances under which Git can be run. With Subversion (the only other source control system I know in detail), the working tree that is operated on is at and below the cwd, and the working tree always points to the repository. (A subdirectory of a working tree is also a valid working tree.)

With Git, it seems that the basic usage is that Git searches upward from the cwd to find the top of the work tree, which is distinguished by having a .git subdirectory. The rules when the worktree is detached are more complicated, and don't seem to be written in any single place.

Dale

I must admit I've never seen the design (and I personally doubt that the design has ever been written down). But at least the following commands work correctly on a detached worktree if the current directory contains the .git directory, because I am using them in a production manner:

    git add
    git cat-file
    git commit
    git commit-tree
    git config
    git gc
    git log
    git ls-tree
    git reset
    git rev-list
    git update-ref

In my situation, the worktree is not, in my mind, dependent on the repository; the repository is intended to keep backups of the contents of the directories that are worktree. Indeed, one could establish several detached repositories to back up different subsets of the same worktree. So it is conceptually natural to execute Git in the repository directory. And, after all, the current directory identifies the repository and the repository contains a pointer to the worktree.

The intention of my change is that it appears that all of the failures of my use pattern are when the command is implemented by a shell script, and it appears that all shell scripts initially invoke git-sh-setup.

The change specifically detects my use pattern and, for the remainder of the script, changes the use pattern into a pattern closely related to the one that Junio documents:

 - export GIT_DIR that points at the correct .git directory;

 - GIT_WORK_TREE is left unset

 - set the current directory to the top of the working tree

Perhaps the change should also set GIT_WORK_TREE. I haven't noticed setting GIT_WORK_TREE to be necessary for "git filter-branch", but perhaps that is coincidental.

It seems to me that this change would uniformly support the use pattern I use without affecting Git's behavior in any other case.

Dale

worley@alum.mit.edu (Dale R. Worley) writes:

If you have this:

	/repositories/proj.git/{refs,objects,...}
	/working/trees/proj-wt1/

where proj-wt1 is a working tree for that proj.git repository, the idea was to set these:

	GIT_DIR=/repositories/proj.git
        GIT_WORK_TREE=/working/trees/proj-wt1
        export GIT_DIR GIT_WORK_TREE

and then working in /working/trees/proj-wt1 or any of its subdirectory should work as if you did not have these two environment variables and had /working/trees/proj-wt1/.git instead of /repositories/proj.git as the repository. To make that use case work was the motivation behind these environment variables.

	Side note: without GIT_WORK_TREE environment (or
	core.worktree), there is no way to tell where the top level
	is, so you were limited to always be at the top level of
	your working tree if you used GIT_DIR to refer to a
	repository that is not embedded in your working tree.  There
	were some changes in this area, but I do not recall the
	details offhand.

Now, when you say "the cwd contains the .git directory", do you mean

	cd /repositories
        git add ../working/trees/proj-wt1/file

updates "file" in the /repositories/proj.git/index? Or do you mean this?

	cd /repositories/proj.git
        git add ../../working/trees/proj-wt1/file

Or this?

	cd /repositories
	edit ../working/trees/proj-wt1/file
	git add file

Most of the commands you listed do not need to look at the actual working tree files, so I would expect e.g. "git log" or "git log paths..." to work but I am wondering what your definition of "works" with respect to the pathspecs, especially when you talk about starting Git command _outside_ the working tree (whether the working tree has its repository embedded in it is not very relevant).

That's not true. The core.worktree config variable tells the top of the worktree, so once you've located the repository, you know where the worktree is.

Indeed, it's not clear why GIT_WORK_TREE exists, as that allows the user to set GIT_WORK_TREE inconsistently with core.worktree. (What happens if you do that?)

Dale

worley@alum.mit.edu (Dale R. Worley) writes:

Read the second line again, perhaps?

The configuration item came _way_ later than the environment, and we need to keep users and scripts from old world working, that is why.

OK, that explains a great deal. IIRC, I first became aware that detached worktrees are possible through the documentation of core.worktree. As Git's architecture has a tight binding between the repository and the worktree, it made a great deal of sense to me that the repository points to the detached worktree. And the absence of core.worktree, a non-detached worktree, is essentially equivalent to having core.worktree specify the directory containing the .git directory.

So the obvious way (to me) to invoke Git with a detached worktree is to set GIT_DIR to point to the repository, and the repository points to the root of the worktree. If the command operates on the worktree, Git can compare the cwd with the worktree root to determine the relative path of files.

(And you can see that in this situation, Git doesn't have to search upward to try to determine where the worktree root is.)

What you're saying is that there's an older mode of operation where the repository does not point to the worktree. Instead, the caller has to set GIT_DIR to locate the repository and GIT_WORK_TREE to locate the worktree. This would be prone to error, because the user is responsible for always pairing the repository with the correct worktree; it doesn't enforce the architectural assumption that the repository is paired with a work tree.

Dale

The pattern I use is to have this:

	/repository/.git
	/working/...

then

	cd /repository
	git add /working/x/y/z

works as you'd expect it to. "git rm" seems to work correctly under these circumstances as well.

I seem to recall that using relative <path> values doesn't work under some conditions involving symbolic links, but I can't recall the details right now.

The above pattern is what I mean, where the cwd is not within the work tree.

Dale