# nested submodules detection w/o .gitmodules file 4 messages from 2023-01-22 to 2023-02-03. Participants: Andry, Emily Shaffer. Thread: https://gitlist.dev/t/59141 ## Andry, 2023-01-22 20:34 Subject: nested submodules detection w/o .gitmodules file Message-ID: <1716310675.20230122233403@inbox.ru> URL: https://gitlist.dev/e/1716310675.20230122233403%40inbox.ru ``` Hello Git, I have a pretty long investigation has been started from usage 3dparty projects related directly or indirectly to the git submodules: `svn externals replacement` : https://github.com/chronoxor/gil/issues/6 `svn complete replacement for externals` : https://github.com/dirk-thomas/vcstool/issues/243 And stumbled on this discussion: `nested submodules detection w/o .gitmodules file` : https://github.com/gitextensions/gitextensions/discussions/10644 (https://github.com/gitextensions/gitextensions/issues/10642) The main question here is that, could the git have has submodules without `.submodules` file? If no, then all side projects which utilizes it's own input file for the externals may subsequentially fail: https://github.com/chronoxor/gil https://github.com/dirk-thomas/vcstool https://github.com/ingydotnet/git-subrepo If yes, then other projects which does rely on the `.submodules` would have not actual or even invalid state: https://github.com/gitextensions/gitextensions Or even the github itself: `Zip archive to include submodule` : https://github.com/dear-github/dear-github/issues/214 (`[PATCH] archive: add –recurse-submodules to git-archive command` : https://git.github.io/rev_news/2022/11/30/edition-93/, https://lore.kernel.org/git/pull.1359.git.git.1665597148042.gitgitgadget@gmail.com/) Mine point here is that: Git database is a primary storage. The `.gitmodules` file is not a primary storage, so can be in not an actual or desync state with the database. And any application or a 3dparty project must read the database directly. But another problem here is that the git still does not have a stable API for that. For example, a submodule can be declared directly from the `.git/config` file in a working copy: https://git-scm.com/docs/git-submodule#Documentation/git-submodule.txt-init--ltpathgt82308203 https://git-scm.com/docs/gitsubmodules#_active_submodules So, who is right and what is wrong here? ``` ## Emily Shaffer, 2023-01-23 20:12 Subject: Re: nested submodules detection w/o .gitmodules file Message-ID: URL: https://gitlist.dev/e/CAJoAoZkjfOQwkeQqzQY5qDo7Md5QWSz1pOTBQKHL5KwNu2VoDg%40mail.gmail.com In-Reply-To: <1716310675.20230122233403@inbox.ru> ``` On Sun, Jan 22, 2023 at 12:45 PM Andry wrote: > > Hello Git, > > I have a pretty long investigation has been started from usage 3dparty projects related directly or indirectly to the git submodules: > > `svn externals replacement` : https://github.com/chronoxor/gil/issues/6 > `svn complete replacement for externals` : https://github.com/dirk-thomas/vcstool/issues/243 > > And stumbled on this discussion: > > `nested submodules detection w/o .gitmodules file` : https://github.com/gitextensions/gitextensions/discussions/10644 (https://github.com/gitextensions/gitextensions/issues/10642) > > The main question here is that, could the git have has submodules without `.submodules` file? There is a little nuance here. Git can have nested repositories in a few different ways; submodules are just one of them. A submodule is the combination of a gitlink object in the object graph *and* a corresponding entry in the .gitmodules file. It's certainly possible to embed a nested repository in other ways, such as by ignoring the .gitmodules file, but then your nested repository is no longer a submodule, and operations which recurse over submodules will not consider that nested repository. The reason is that we need to understand where to clone the submodule from - that information isn't contained in the superproject's repository URL, and it can't be contained in the gitlink directly (which is in essence just a commit object, but one that exists in the nested repository, not in the superproject repository). If we didn't have a way of writing down the submodule's remote URL and version controlling it, we wouldn't have a way to populate the submodules when a user is cloning. > > If no, then all side projects which utilizes it's own input file for the externals may subsequentially fail: > > https://github.com/chronoxor/gil > https://github.com/dirk-thomas/vcstool > https://github.com/ingydotnet/git-subrepo > > If yes, then other projects which does rely on the `.submodules` would have not actual or even invalid state: > > https://github.com/gitextensions/gitextensions > > Or even the github itself: > > `Zip archive to include submodule` : https://github.com/dear-github/dear-github/issues/214 > > (`[PATCH] archive: add –recurse-submodules to git-archive command` : https://git.github.io/rev_news/2022/11/30/edition-93/, https://lore.kernel.org/git/pull.1359.git.git.1665597148042.gitgitgadget@gmail.com/) > > Mine point here is that: > > Git database is a primary storage. The `.gitmodules` file is not a primary storage, so can be in not an actual or desync state with the database. > And any application or a 3dparty project must read the database directly. The .gitmodules exists to help at clone time; it's possible, as I think you're pointing out, to have some intermediate state locally. But this file is what needs to be the source of truth for putting together the repository on a new machine for the first time. > > But another problem here is that the git still does not have a stable API for that. > For example, a submodule can be declared directly from the `.git/config` file in a working copy: > > https://git-scm.com/docs/git-submodule#Documentation/git-submodule.txt-init--ltpathgt82308203 > https://git-scm.com/docs/gitsubmodules#_active_submodules > > So, who is right and what is wrong here? > ``` ## Andry, 2023-01-24 14:54 Subject: Re: nested submodules detection w/o .gitmodules file Message-ID: <1204599253.20230124175436@inbox.ru> URL: https://gitlist.dev/e/1204599253.20230124175436%40inbox.ru In-Reply-To: ``` Hello Emily, Monday, January 23, 2023, 11:12:32 PM, you wrote: ES> There is a little nuance here. Git can have nested repositories in a few different ways; submodules are just one of them. ES> A submodule is the combination of a gitlink object in the object graph *and* a corresponding entry in the .gitmodules file. ES> It's certainly possible to embed a nested repository in other ways, such as by ignoring the .gitmodules file, but then your ES> nested repository is no longer a submodule, and operations which recurse over submodules will not consider that nested repository. But it does consider. See the output of the `git submodule summary` command: https://github.com/gitextensions/gitextensions/discussions/10644#discussioncomment-4688533 ES> The reason is that we need to understand where to clone the submodule from - that information isn't contained in the superproject's ES> repository URL Yes, it does clone "externally", but after a clone it is contained in the `.git/config` of a subproject: [submodule] active = . [remote "origin"] url = ... ES> , and it can't be contained in the gitlink directly (which is in essence just a commit object, but one that exists ES> in the nested repository, not in the superproject repository). If we didn't have a way of writing down the submodule's remote URL ES> and version controlling it, we wouldn't have a way to populate the submodules when a user is cloning. The `git submodule summary` nevertheless can populate submodules after that, so this is not the matter. ES> The .gitmodules exists to help at clone time; it's possible, as I ES> think you're pointing out, to have some intermediate state locally. ES> But this file is what needs to be the source of truth for putting ES> together the repository on a new machine for the first time. Yes, the original clone command is superseded in those projects. ``` ## Andry, 2023-02-03 01:04 Subject: Re: nested submodules detection w/o .gitmodules file Message-ID: <1628944263.20230203040412@inbox.ru> URL: https://gitlist.dev/e/1628944263.20230203040412%40inbox.ru In-Reply-To: <1716310675.20230122233403@inbox.ru> ``` Hello Git, Sunday, January 22, 2023, 11:34:03 PM, you wrote: A> Hello Git, A> I have a pretty long investigation has been started from usage 3dparty projects related directly or indirectly to the git submodules: A> ... A> So, who is right and what is wrong here? Is nobody interested in the issue? ```