behaviour of git diff, GIT_DIR & checked out tree

I'm doing something "sensible people don't do" -- I know this -- but I'm getting results I don't understand, so I'd appreciate any insight.

In a git tracked tree rooted at $HOME/V with git-dir of $HOME/V/.git, if I'm in $HOME/V then

git diff master@{midnight}

tells me the difference between the current modified files in the tree being tracked in V and the specified commit. In a different directory, OUTSIDE of $HOME/V, I tried

env GIT_DIR=$HOME/V/.git git diff master@{midnight}

to get the same effect but, whilst I do get a diff output, it looks like a diff of the commit against an empty tree. (Using

env GIT_DIR=$HOME/V/.git git diff HEAD master@{midnight}

works ok.) I suppose I could alias it to

cd $HOME/V && command && cd -

but that seems a bit clunky; is there another way to explicitly say "working tree for repo with this GIT_DIR" to git diff?

[Why I'm doing this: I actually work on files in the $HOME/V via various other directories containing symlinks to the basic files (with tracked files being pointed to by multiple symlinks). So I'm almost never actually "in" the tracked tree. This usage makes more sense for my tasks than being within the work tree.]