nested submodules detection w/o .gitmodules file

Hello Git,

I have a pretty long investigation has been started from usage 3dparty projects related directly or indirectly to the git submodules:

`svn externals replacement` : https://github.com/chronoxor/gil/issues/6 `svn complete replacement for externals` : https://github.com/dirk-thomas/vcstool/issues/243

And stumbled on this discussion:

`nested submodules detection w/o .gitmodules file` : https://github.com/gitextensions/gitextensions/discussions/10644 (https://github.com/gitextensions/gitextensions/issues/10642)

The main question here is that, could the git have has submodules without `.submodules` file?

If no, then all side projects which utilizes it's own input file for the externals may subsequentially fail:

https://github.com/chronoxor/gil https://github.com/dirk-thomas/vcstool https://github.com/ingydotnet/git-subrepo

If yes, then other projects which does rely on the `.submodules` would have not actual or even invalid state:

https://github.com/gitextensions/gitextensions

Or even the github itself: 

`Zip archive to include submodule` : https://github.com/dear-github/dear-github/issues/214

(`[PATCH] archive: add –recurse-submodules to git-archive command` : https://git.github.io/rev_news/2022/11/30/edition-93/, https://lore.kernel.org/git/pull.1359.git.git.1665597148042.gitgitgadget@gmail.com/)

Mine point here is that:

Git database is a primary storage. The `.gitmodules` file is not a primary storage, so can be in not an actual or desync state with the database. And any application or a 3dparty project must read the database directly.

But another problem here is that the git still does not have a stable API for that. For example, a submodule can be declared directly from the `.git/config` file in a working copy:

https://git-scm.com/docs/git-submodule#Documentation/git-submodule.txt-init--ltpathgt82308203 https://git-scm.com/docs/gitsubmodules#_active_submodules

So, who is right and what is wrong here?

On Sun, Jan 22, 2023 at 12:45 PM Andry <andry@inbox.ru> wrote:

There is a little nuance here. Git can have nested repositories in a few different ways; submodules are just one of them. A submodule is the combination of a gitlink object in the object graph *and* a corresponding entry in the .gitmodules file. It's certainly possible to embed a nested repository in other ways, such as by ignoring the .gitmodules file, but then your nested repository is no longer a submodule, and operations which recurse over submodules will not consider that nested repository. The reason is that we need to understand where to clone the submodule from - that information isn't contained in the superproject's repository URL, and it can't be contained in the gitlink directly (which is in essence just a commit object, but one that exists in the nested repository, not in the superproject repository). If we didn't have a way of writing down the submodule's remote URL and version controlling it, we wouldn't have a way to populate the submodules when a user is cloning.

The .gitmodules exists to help at clone time; it's possible, as I think you're pointing out, to have some intermediate state locally. But this file is what needs to be the source of truth for putting together the repository on a new machine for the first time.

Hello Emily,

Monday, January 23, 2023, 11:12:32 PM, you wrote:

ES> There is a little nuance here. Git can have nested repositories in a few different ways; submodules are just one of them. ES> A submodule is the combination of a gitlink object in the object graph *and* a corresponding entry in the .gitmodules file. ES> It's certainly possible to embed a nested repository in other ways, such as by ignoring the .gitmodules file, but then your ES> nested repository is no longer a submodule, and operations which recurse over submodules will not consider that nested repository.

But it does consider. See the output of the `git submodule summary` command: https://github.com/gitextensions/gitextensions/discussions/10644#discussioncomment-4688533

ES> The reason is that we need to understand where to clone the submodule from - that information isn't contained in the superproject's ES> repository URL

Yes, it does clone "externally", but after a clone it is contained in the `.git/config` of a subproject:

[submodule]
        active = .
[remote "origin"]
        url = ...

ES> , and it can't be contained in the gitlink directly (which is in essence just a commit object, but one that exists ES> in the nested repository, not in the superproject repository). If we didn't have a way of writing down the submodule's remote URL ES> and version controlling it, we wouldn't have a way to populate the submodules when a user is cloning.

The `git submodule summary` nevertheless can populate submodules after that, so this is not the matter.

ES> The .gitmodules exists to help at clone time; it's possible, as I ES> think you're pointing out, to have some intermediate state locally. ES> But this file is what needs to be the source of truth for putting ES> together the repository on a new machine for the first time.

Yes, the original clone command is superseded in those projects.

Hello Git,

Sunday, January 22, 2023, 11:34:03 PM, you wrote:

A> Hello Git,

A> I have a pretty long investigation has been started from usage 3dparty projects related directly or indirectly to the git submodules: A> ... A> So, who is right and what is wrong here?

Is nobody interested in the issue?