{"thread":{"id":"59141","subject":"nested submodules detection w/o .gitmodules file","startedAt":"2023-01-22T20:45:34Z","lastAt":"2023-02-03T01:03:34Z","messageCount":4,"participants":["Andry","Emily Shaffer"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"470890","messageId":"1716310675.20230122233403@inbox.ru","threadId":"59141","inReplyTo":null,"subject":"nested submodules detection w/o .gitmodules file","fromName":"Andry","fromEmail":"andry@inbox.ru","sentAt":"2023-01-22T20:34:03Z","receivedAt":"2023-01-22T20:45:34Z","isPatch":false,"sender":{"key":"andry@inbox.ru","avatar":null},"body":"Hello Git,\n\nI have a pretty long investigation has been started from usage 3dparty projects related directly or indirectly to the git submodules:\n\n`svn externals replacement` : https://github.com/chronoxor/gil/issues/6\n`svn complete replacement for externals` : https://github.com/dirk-thomas/vcstool/issues/243\n\nAnd stumbled on this discussion:\n\n`nested submodules detection w/o .gitmodules file` : https://github.com/gitextensions/gitextensions/discussions/10644 (https://github.com/gitextensions/gitextensions/issues/10642)\n\nThe main question here is that, could the git have has submodules without `.submodules` file?\n\nIf no, then all side projects which utilizes it's own input file for the externals may subsequentially fail:\n\nhttps://github.com/chronoxor/gil\nhttps://github.com/dirk-thomas/vcstool\nhttps://github.com/ingydotnet/git-subrepo\n\nIf yes, then other projects which does rely on the `.submodules` would have not actual or even invalid state:\n\nhttps://github.com/gitextensions/gitextensions\n\nOr even the github itself: \n\n`Zip archive to include submodule` : https://github.com/dear-github/dear-github/issues/214\n\n(`[PATCH] archive: add –recurse-submodules to git-archive command` : https://git.github.io/rev_news/2022/11/30/edition-93/, https://lore.kernel.org/git/pull.1359.git.git.1665597148042.gitgitgadget@gmail.com/)\n\nMine point here is that:\n\nGit database is a primary storage. The `.gitmodules` file is not a primary storage, so can be in not an actual or desync state with the database.\nAnd any application or a 3dparty project must read the database directly.\n\nBut another problem here is that the git still does not have a stable API for that.\nFor example, a submodule can be declared directly from the `.git/config` file in a working copy: \n\nhttps://git-scm.com/docs/git-submodule#Documentation/git-submodule.txt-init--ltpathgt82308203\nhttps://git-scm.com/docs/gitsubmodules#_active_submodules\n\nSo, who is right and what is wrong here?\n\n"},{"id":"470948","messageId":"CAJoAoZkjfOQwkeQqzQY5qDo7Md5QWSz1pOTBQKHL5KwNu2VoDg@mail.gmail.com","threadId":"59141","inReplyTo":"1716310675.20230122233403@inbox.ru","subject":"Re: nested submodules detection w/o .gitmodules file","fromName":"Emily Shaffer","fromEmail":"nasamuffin@google.com","sentAt":"2023-01-23T20:12:32Z","receivedAt":"2023-01-23T20:12:52Z","isPatch":false,"sender":{"key":"nasamuffin@google.com","avatar":"https://avatars.githubusercontent.com/u/1606826?v=4"},"body":"On Sun, Jan 22, 2023 at 12:45 PM Andry <andry@inbox.ru> wrote:\n>\n> Hello Git,\n>\n> I have a pretty long investigation has been started from usage 3dparty projects related directly or indirectly to the git submodules:\n>\n> `svn externals replacement` : https://github.com/chronoxor/gil/issues/6\n> `svn complete replacement for externals` : https://github.com/dirk-thomas/vcstool/issues/243\n>\n> And stumbled on this discussion:\n>\n> `nested submodules detection w/o .gitmodules file` : https://github.com/gitextensions/gitextensions/discussions/10644 (https://github.com/gitextensions/gitextensions/issues/10642)\n>\n> The main question here is that, could the git have has submodules without `.submodules` file?\n\nThere is a little nuance here. Git can have nested repositories in a\nfew different ways; submodules are just one of them. A submodule is\nthe combination of a gitlink object in the object graph *and* a\ncorresponding entry in the .gitmodules file. It's certainly possible\nto embed a nested repository in other ways, such as by ignoring the\n.gitmodules file, but then your nested repository is no longer a\nsubmodule, and operations which recurse over submodules will not\nconsider that nested repository. The reason is that we need to\nunderstand where to clone the submodule from - that information isn't\ncontained in the superproject's repository URL, and it can't be\ncontained in the gitlink directly (which is in essence just a commit\nobject, but one that exists in the nested repository, not in the\nsuperproject repository). If we didn't have a way of writing down the\nsubmodule's remote URL and version controlling it, we wouldn't have a\nway to populate the submodules when a user is cloning.\n\n>\n> If no, then all side projects which utilizes it's own input file for the externals may subsequentially fail:\n>\n> https://github.com/chronoxor/gil\n> https://github.com/dirk-thomas/vcstool\n> https://github.com/ingydotnet/git-subrepo\n>\n> If yes, then other projects which does rely on the `.submodules` would have not actual or even invalid state:\n>\n> https://github.com/gitextensions/gitextensions\n>\n> Or even the github itself:\n>\n> `Zip archive to include submodule` : https://github.com/dear-github/dear-github/issues/214\n>\n> (`[PATCH] archive: add –recurse-submodules to git-archive command` : https://git.github.io/rev_news/2022/11/30/edition-93/, https://lore.kernel.org/git/pull.1359.git.git.1665597148042.gitgitgadget@gmail.com/)\n>\n> Mine point here is that:\n>\n> Git database is a primary storage. The `.gitmodules` file is not a primary storage, so can be in not an actual or desync state with the database.\n> And any application or a 3dparty project must read the database directly.\n\nThe .gitmodules exists to help at clone time; it's possible, as I\nthink you're pointing out, to have some intermediate state locally.\nBut this file is what needs to be the source of truth for putting\ntogether the repository on a new machine for the first time.\n\n>\n> But another problem here is that the git still does not have a stable API for that.\n> For example, a submodule can be declared directly from the `.git/config` file in a working copy:\n>\n> https://git-scm.com/docs/git-submodule#Documentation/git-submodule.txt-init--ltpathgt82308203\n> https://git-scm.com/docs/gitsubmodules#_active_submodules\n>\n> So, who is right and what is wrong here?\n>\n"},{"id":"470978","messageId":"1204599253.20230124175436@inbox.ru","threadId":"59141","inReplyTo":"CAJoAoZkjfOQwkeQqzQY5qDo7Md5QWSz1pOTBQKHL5KwNu2VoDg@mail.gmail.com","subject":"Re: nested submodules detection w/o .gitmodules file","fromName":"Andry","fromEmail":"andry@inbox.ru","sentAt":"2023-01-24T14:54:36Z","receivedAt":"2023-01-24T14:53:55Z","isPatch":false,"sender":{"key":"andry@inbox.ru","avatar":null},"body":"Hello Emily,\n\n\n\nMonday, January 23, 2023, 11:12:32 PM, you wrote:\n\nES> There is a little nuance here. Git can have nested repositories in a few different ways; submodules are just one of them.\nES> A submodule is the combination of a gitlink object in the object graph *and* a corresponding entry in the .gitmodules file.\nES> It's certainly possible to embed a nested repository in other ways, such as by ignoring the .gitmodules file, but then your\nES> nested repository is no longer a submodule, and operations which recurse over submodules will not consider that nested repository.\n\nBut it does consider. See the output of the `git submodule summary` command:\nhttps://github.com/gitextensions/gitextensions/discussions/10644#discussioncomment-4688533\n\nES> The reason is that we need to understand where to clone the submodule from - that information isn't contained in the superproject's\nES> repository URL\n\nYes, it does clone \"externally\", but after a clone it is contained in the `.git/config` of a subproject:\n\n[submodule]\n        active = .\n[remote \"origin\"]\n        url = ...\n\nES> , and it can't be contained in the gitlink directly (which is in essence just a commit object, but one that exists\nES> in the nested repository, not in the superproject repository). If we didn't have a way of writing down the submodule's remote URL\nES> and version controlling it, we wouldn't have a way to populate the submodules when a user is cloning.\n\nThe `git submodule summary` nevertheless can populate submodules after that, so this is not the matter.\n\nES> The .gitmodules exists to help at clone time; it's possible, as I\nES> think you're pointing out, to have some intermediate state locally.\nES> But this file is what needs to be the source of truth for putting\nES> together the repository on a new machine for the first time.\n\nYes, the original clone command is superseded in those projects.\n\n"},{"id":"471411","messageId":"1628944263.20230203040412@inbox.ru","threadId":"59141","inReplyTo":"1716310675.20230122233403@inbox.ru","subject":"Re: nested submodules detection w/o .gitmodules file","fromName":"Andry","fromEmail":"andry@inbox.ru","sentAt":"2023-02-03T01:04:12Z","receivedAt":"2023-02-03T01:03:34Z","isPatch":false,"sender":{"key":"andry@inbox.ru","avatar":null},"body":"Hello Git,\n\n\n\nSunday, January 22, 2023, 11:34:03 PM, you wrote:\n\nA> Hello Git,\n\nA> I have a pretty long investigation has been started from usage 3dparty projects related directly or indirectly to the git submodules:\nA> ...\nA> So, who is right and what is wrong here?\n\nIs nobody interested in the issue?\n\n"}]}