Re: Bug: All git operations fail when .git contains a non-existent gitdir

From: David Aguilar <davvid@gmail.com>
Date: Aug 30, 2021, 00:38 UTC
Message-ID: <CAJDDKr7vXRxhwuTN=xFxqQkt_zt_Hpi3bYfc7rotcUAFjau9Sg@mail.gmail.com>
In-Reply-To: <CAFSh4Uyr5v9Ao-j0j7yO_HkUZSovBmSg7ADia7XCNZfsspFUYg@mail.gmail.com>

On Thu, Jul 22, 2021 at 6:16 AM Tom Cook <tom.k.cook@gmail.com> wrote:

Show 12 quoted lines

>
> The easiest way to reproduce it is this:
>
> $ mkdir test
> $ cd test
> $ echo "gitdir: /foo/bar" > .git
> $ git ls-remote https://github.com/torvalds/linux
>
> We happen to use overlay mounts in our build system in a way that maps
> a git submodule from one place to another so that its "gitdir" is
> invalid and then attempt a `git ls-remote` from that location which
> unexpectedly fails.  But the above reproduces the problem well enough.

I'm making a guess about your setup. To me it sounds like tweaking the overlayfs or bind mounts to also map the .git directory of the parent repository (at just the right relative location inside the build environment) is the right approach towards making the .git-file links happy.

Submodules point to the parent repository's .git/modules/ directory these days so poking those paths through into the mapped file system is the solution that's going to lead to the least amount of hard-to-diagnose issues in the future.

The .git-file uses relative paths so it is possible to remap and relocate them as you're doing, but it's going to require another mount in the right relative location to do it.

It seems doable, even for most edge cases. For example if you end up mapping some/deeply/nested/submod into /shallow inside your container(?) and the .git-file contains "../../../../.git/modules/submod", that might seem impossible to resolve from a "/shallow" directory, but actually it's fine.

The solution in that situation is to map the parent repo's .git/ dir to the "/.git" filesystem root inside the build environment. Unix treats "/../../../../../../" as equivalent to "/" so if the .git-file has a bunch of "../../../" parent-dir traversing going on then it'll still work from a one-level deep "/shallow" directory. The parent git repo directory would be expected to be found at "/.git" (with a "modules" directory inside it) because the parent directory of "/" is also "/".

This should mesh right in with the setup you described because you're already using overlay mounts to synthesize a filesystem.

There are edge cases where this approach breaks down, though. One of these cases is submodules inside submodules. Luckily it doesn't sound like you have that problem.

There are patches now that special-case being able to run ls-remote <url> in a broken setup, but the real fix is to extend the build environment's filesystem view so that the submodules have access to the underlying .git/modules/ storage.

Hopefully it's just a --mount or --volume docker(?) command-line flag that has to be extended to poke the parent .git/ repository contents into the environment. That's the avenue I would explore if I were already down a path of using mount tricks.

I hope that helps and maybe sparks an idea that you can run with. I certainly don't mean it to be a dismissive, "go change your setup" kind of advice, but it would be interesting to know more details since it seems like achieving correctness is going to require doing that based on the limited information in this thread about the specifics of the setup.

Also, I wouldn't be surprised if, even after the "git ls-remote" robustness fixes are applied, you'll just stumble onto the next git command that can only be resolved by mapping in the parent repo's .git directory.

-- David