Bug with Git shallow clones and submodules

We recently came across a bug in one of our Git projects which make use of submodules. The basic problem is we have a Git repo that has 1 submodule within it which is set up to check out a specific branch, and when we attempt to perform a shallow clone on the parent repo the checkout fails with a "Server does not allow request for unadvertised object" error. After doing a lot of testing and reading online I found this StackOverflow thread (https://stackoverflow.com/a/61492323) that explains our problem in great detail, which indicates to me that this is likely not a problem specific to us, so I thought I'd report it to see if it could be fixed.

In our specific use case, we found the problem specifically when running a build of our project using Gitlab CI. This tool automatically performs a shallow checkout of projects being built by using a "--depth=50" parameter. This caused odd Git checkout errors to appear on our CI builds which weren't directly reproducible by our developers local builds. Upon closer examination we were able to reproduce outside of the CI environment by performing a checkout as follows:

    git clone --recurse-submodules --depth=50  git@<path_to_repo>.git .

Based on my initial research, which was helped greatly by the StackOverflow thread I linked to above, I was able to deduce the following contributing factors:

The submodule that was causing us grief was the dpdk library located here: http://git.dpdk.org/dpdk-stable/

This repository has a default branch named "default" which has only 2 commits on it: one for initializing the repo and one additional commit All releases of this library are created from branches named by the release version, and they all seem disjoint from one another, and they are all disjoint from the default branch (ie: changes from the release branches are never merged into the default branch)

our parent repo, which is an internal / non-public repository that can not be shared, has a single submodule defined in it with the following .gitmodule definition:

[submodule "dpdk-stable"] path = dpdk-stable url = https://git@dpdk.org/git/dpdk-stable branch = 19.11

the specific revision we are checking out has the OID 78bc27846101e481438a98c68cac47e4476085c0 which corresponds to the 19.11.8 tag the following git clone operation succeeds:

git clone --recurse-submodules git@<path_to_repo>.git .

the following git clone operation fails:

git clone --recurse-submodules --depth=50  git@<path_to_repo>.git .

we can work around the problem by doing a shallow clone with the no-single-branch parameter, as in:

git clone --recurse-submodules --depth=50 --no-single-branch git@<path_to_repo>.git .

Based on my current level of understanding, Git is automatically setting the --single-branch flag when cloning submodules but only when doing a shallow clone of the parent. This seems wrong to me. In addition, even if Git was going to enforce the single branch option on submodules it stands to reason that it should be respecting the branch defined in the .gitmodules file (ie: 19.11 in our case) instead of using the default branch (ie: "default" in this case). My guess is, if it were respecting the branch name then we wouldn't be getting the "unadvertised object" error because the referenced OID would exist in the submodule checkout because it would have the correct branch checked out.

For reference, this problem appears to be reproducible on several different platforms (ie: several different Linux distros and Mac OSX at least) and several different Git versions up to v2.32.0 which is the latest version at the moment. Hopefully this problem can be fixed in a future release of the command line tools. In the meantime we've had to hack around the problem in our Gitlab CI builds by disabling the shallow checkout features, which is less than ideal, so again I am hopeful that this can be fixed directly in Git itself.

Kevin Phillips Lead Developer, Configuration Management Team, IBM