{"thread":{"id":"58566","subject":"issue with submodules using origin remote unexpectadly","startedAt":"2022-10-04T17:44:04Z","lastAt":"2022-10-12T00:14:04Z","messageCount":5,"participants":["Jacob Keller","Glen Choo"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"464205","messageId":"CA+P7+xrnAXceHJbBYV8Z9Ay9xbkzpiX3P0wXLwN7jfnifW2OBQ@mail.gmail.com","threadId":"58566","inReplyTo":null,"subject":"issue with submodules using origin remote unexpectadly","fromName":"Jacob Keller","fromEmail":"jacob.keller@gmail.com","sentAt":"2022-10-04T17:43:10Z","receivedAt":"2022-10-04T17:44:04Z","isPatch":false,"sender":{"key":"jacob.keller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/874719?v=4"},"body":"Hi,\n\nI've been running into an issue with submodules trying to fetch using\n\"origin\" even though the submodule does not have a remote by that\nname. (I recently switched the submodule remote name)\n\nremote: Enumerating objects: 210, done.\nremote: Counting objects: 100% (207/207), done.\nremote: Compressing objects: 100% (54/54), done.\nremote: Total 210 (delta 123), reused 197 (delta 119), pack-reused 3\nReceiving objects: 100% (210/210), 107.20 KiB | 4.29 MiB/s, done.\nResolving deltas: 100% (123/123), completed with 48 local objects.\nFrom <redacted>\n...\nFetching submodule submodule\nFrom <redacted>\n   85e0da7533d9..80cc886f1187  <redacted>\nFetching submodule submodule2\nfatal: 'origin' does not appear to be a git repository\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists.\nErrors during submodule fetch:\n        submodule2\n\n\nI thought at first there might be some configuration value in the\nparent that is selecting origin, but I can 't find one. I also thought\nmaybe its some sort of defaulting to origin but I looked through\nsubmodule code and didn't find one.\n\nIs it just that submodule code is not resilient against changing the\ndefault remote name away from origin?\n\nThanks,\nJake\n"},{"id":"464206","messageId":"kl6lzgeblap2.fsf@chooglen-macbookpro.roam.corp.google.com","threadId":"58566","inReplyTo":"CA+P7+xrnAXceHJbBYV8Z9Ay9xbkzpiX3P0wXLwN7jfnifW2OBQ@mail.gmail.com","subject":"Re: issue with submodules using origin remote unexpectadly","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-10-04T18:12:41Z","receivedAt":"2022-10-04T18:12:54Z","isPatch":false,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"Hi Jacob! Thanks for the report!\n\nI think we'd need a bit more info, specifically:\n\n- How did you rename the remote?\n- What command is doing the submodule fetch?\n\nOr, if you could include a reproduction script, that would be really\nhelpful :)\n\nJacob Keller <jacob.keller@gmail.com> writes:\n\n> Hi,\n>\n> I've been running into an issue with submodules trying to fetch using\n> \"origin\" even though the submodule does not have a remote by that\n> name. (I recently switched the submodule remote name)\n\nHow did you rename the remote? e.g. did you run `git remote rename`, or\ndid you rename it manually? I'll come back to why this might be\nimportant...\n\n>\n> remote: Enumerating objects: 210, done.\n> remote: Counting objects: 100% (207/207), done.\n> remote: Compressing objects: 100% (54/54), done.\n> remote: Total 210 (delta 123), reused 197 (delta 119), pack-reused 3\n> Receiving objects: 100% (210/210), 107.20 KiB | 4.29 MiB/s, done.\n> Resolving deltas: 100% (123/123), completed with 48 local objects.\n> From <redacted>\n> ...\n> Fetching submodule submodule\n> From <redacted>\n>    85e0da7533d9..80cc886f1187  <redacted>\n> Fetching submodule submodule2\n> fatal: 'origin' does not appear to be a git repository\n> fatal: Could not read from remote repository.\n>\n> Please make sure you have the correct access rights\n> and the repository exists.\n> Errors during submodule fetch:\n>         submodule2\n\nI assume this is `git fetch` running in the superproject?\n\n> I thought at first there might be some configuration value in the\n> parent that is selecting origin, but I can 't find one. I also thought\n> maybe its some sort of defaulting to origin but I looked through\n> submodule code and didn't find one.\n>\n> Is it just that submodule code is not resilient against changing the\n> default remote name away from origin?\n\nYes, in a sense, though I'm not sure if this is the source of your\nproblems.\n\nWhen fetching with `git fetch`, submodules are fetched without\nspecifying the remote name, which means Git guesses which remote you\nwant to fetch from, which is documented at\nhttps://git-scm.com/docs/git-fetch. I believe (I haven't reread this\nvery closely) this is, in order:\n\n- The remote of your branch, i.e. the value of the config value\n  `branch.<name>.remote`\n- origin\n\nThis is why `git remote rename` matters, because `git remote rename`\nwill also rename `branch.<name>.remote`. If you have a branch checked\nout in your submodule, you might want to double-check your .git/config.\n\nBut... I'll mention another wrinkle for completeness' sake (though I\ndon't think it applies to you). If you fetch using `git submodule\nupdate`, the submodule is fetched using a _named_ remote, specifically:\n\n- If the superproject has a branch checked out, it uses the name of the\n  superproject branch's remote.\n- If the superproject does not have a branch checked out, it uses\n  \"origin\".\n\n>\n> Thanks,\n> Jake\n"},{"id":"464632","messageId":"CA+P7+xq3NGGTq2U2Di9ZTBO3buqeEMb2smC=v78V9Subt2cDMw@mail.gmail.com","threadId":"58566","inReplyTo":"kl6lzgeblap2.fsf@chooglen-macbookpro.roam.corp.google.com","subject":"Re: issue with submodules using origin remote unexpectadly","fromName":"Jacob Keller","fromEmail":"jacob.keller@gmail.com","sentAt":"2022-10-11T21:15:59Z","receivedAt":"2022-10-11T21:16:16Z","isPatch":false,"sender":{"key":"jacob.keller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/874719?v=4"},"body":"On Tue, Oct 4, 2022 at 11:12 AM Glen Choo <chooglen@google.com> wrote:\n>\n> Hi Jacob! Thanks for the report!\n>\n\nThanks for responding!\n\n> I think we'd need a bit more info, specifically:\n>\n> - How did you rename the remote?\n> - What command is doing the submodule fetch?\n>\n\nI am doing a git pull --rebase in the superproject, and I renamed the\nremote in the submodule using git remote rename.\n\n> Or, if you could include a reproduction script, that would be really\n> helpful :)\n>\n\nI'm not sure how to do this, because it is only an intermittent\nfailure. I suspect it has to do with when the submodule actually needs\nto update.\n\nPerhaps I can come up with something though. If I can, I'll send it as\na new test.\n\n> Jacob Keller <jacob.keller@gmail.com> writes:\n>\n> > Hi,\n> >\n> > I've been running into an issue with submodules trying to fetch using\n> > \"origin\" even though the submodule does not have a remote by that\n> > name. (I recently switched the submodule remote name)\n>\n> How did you rename the remote? e.g. did you run `git remote rename`, or\n> did you rename it manually? I'll come back to why this might be\n> important...\n>\n\nI renamed it using git remote rename.\n\n> >\n> > remote: Enumerating objects: 210, done.\n> > remote: Counting objects: 100% (207/207), done.\n> > remote: Compressing objects: 100% (54/54), done.\n> > remote: Total 210 (delta 123), reused 197 (delta 119), pack-reused 3\n> > Receiving objects: 100% (210/210), 107.20 KiB | 4.29 MiB/s, done.\n> > Resolving deltas: 100% (123/123), completed with 48 local objects.\n> > From <redacted>\n> > ...\n> > Fetching submodule submodule\n> > From <redacted>\n> >    85e0da7533d9..80cc886f1187  <redacted>\n> > Fetching submodule submodule2\n> > fatal: 'origin' does not appear to be a git repository\n> > fatal: Could not read from remote repository.\n> >\n> > Please make sure you have the correct access rights\n> > and the repository exists.\n> > Errors during submodule fetch:\n> >         submodule2\n>\n> I assume this is `git fetch` running in the superproject?\n>\n\nIts git pull --rebase, but I suppose as part of this it will run\nsomething equivalent to git fetch?\n\n> > I thought at first there might be some configuration value in the\n> > parent that is selecting origin, but I can 't find one. I also thought\n> > maybe its some sort of defaulting to origin but I looked through\n> > submodule code and didn't find one.\n> >\n> > Is it just that submodule code is not resilient against changing the\n> > default remote name away from origin?\n>\n> Yes, in a sense, though I'm not sure if this is the source of your\n> problems.\n>\n\nHmm.\n\n> When fetching with `git fetch`, submodules are fetched without\n> specifying the remote name, which means Git guesses which remote you\n> want to fetch from, which is documented at\n> https://git-scm.com/docs/git-fetch. I believe (I haven't reread this\n> very closely) this is, in order:\n>\n> - The remote of your branch, i.e. the value of the config value\n>   `branch.<name>.remote`\n\nSo basically if its checked out to a branch it will fetch from the\nremote of that branch, but...\n\n> - origin\n>\n\nIt defaults to origin, so if you have the usual \"checked out as a\ndetached head\" style of submodule, it can't find the remote branch.\n\n> This is why `git remote rename` matters, because `git remote rename`\n> will also rename `branch.<name>.remote`. If you have a branch checked\n> out in your submodule, you might want to double-check your .git/config.\n>\n\nI do not have a branch checked out, I have it checked out as detached\nhead, as is the usual style for \"git submodule update\" would do.\n\n> But... I'll mention another wrinkle for completeness' sake (though I\n> don't think it applies to you). If you fetch using `git submodule\n> update`, the submodule is fetched using a _named_ remote, specifically:\n>\n> - If the superproject has a branch checked out, it uses the name of the\n>   superproject branch's remote.\n\nRight, so that explains why I can re-run git submodule update after a\ngit pull --rebase and it works.\n\nIn theory wouldn't it make more sense to use the remote based on the\nURL of the .gitmodules file?\n\n> - If the superproject does not have a branch checked out, it uses\n>   \"origin\".\n>\n\nI suppose one option would be to make this configurable. I started\nusing \"upstream\" as the default remote name for most of my\nrepositories when I began working with forks a lot more.\n\n> >\n> > Thanks,\n> > Jake\n"},{"id":"464634","messageId":"kl6lilkq0zpr.fsf@chooglen-macbookpro.roam.corp.google.com","threadId":"58566","inReplyTo":"CA+P7+xq3NGGTq2U2Di9ZTBO3buqeEMb2smC=v78V9Subt2cDMw@mail.gmail.com","subject":"Re: issue with submodules using origin remote unexpectadly","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2022-10-11T22:20:32Z","receivedAt":"2022-10-11T22:21:32Z","isPatch":false,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"Jacob Keller <jacob.keller@gmail.com> writes:\n\n> On Tue, Oct 4, 2022 at 11:12 AM Glen Choo <chooglen@google.com> wrote:\n>>\n>> Hi Jacob! Thanks for the report!\n>>\n>\n> Thanks for responding!\n\n:) \n\n>> Or, if you could include a reproduction script, that would be really\n>> helpful :)\n>>\n>\n> I'm not sure how to do this, because it is only an intermittent\n> failure. I suspect it has to do with when the submodule actually needs\n> to update.\n>\n> Perhaps I can come up with something though. If I can, I'll send it as\n> a new test.\n\nThat would be greatly appreciated, thanks!\n\nIf you find code pointers useful,\n\n- builtin/submodule--helper.c:fetch_in_submodule() contains the logic\n  for fetching during \"git submodule update\"\n\n- submodule.c:fetch_submodules() contains the logic for fetching during\n  \"git fetch --recurse-submodules\" (which is invoked by \"git pull\n  --recurse-submodules\").\n\n\n>> >\n>> > remote: Enumerating objects: 210, done.\n>> > remote: Counting objects: 100% (207/207), done.\n>> > remote: Compressing objects: 100% (54/54), done.\n>> > remote: Total 210 (delta 123), reused 197 (delta 119), pack-reused 3\n>> > Receiving objects: 100% (210/210), 107.20 KiB | 4.29 MiB/s, done.\n>> > Resolving deltas: 100% (123/123), completed with 48 local objects.\n>> > From <redacted>\n>> > ...\n>> > Fetching submodule submodule\n>> > From <redacted>\n>> >    85e0da7533d9..80cc886f1187  <redacted>\n>> > Fetching submodule submodule2\n>> > fatal: 'origin' does not appear to be a git repository\n>> > fatal: Could not read from remote repository.\n>> >\n>> > Please make sure you have the correct access rights\n>> > and the repository exists.\n>> > Errors during submodule fetch:\n>> >         submodule2\n>>\n>> I assume this is `git fetch` running in the superproject?\n>>\n>\n> Its git pull --rebase, but I suppose as part of this it will run\n> something equivalent to git fetch?\n\nUnfortunately, this doesn't narrow it down much because \"git pull\n--recurse-submodules\" runs _both_ \"git fetch --recurse-submodules\" _and_\n\"git submodule update [--rebase]\" ;) Without more context, it's not\nclear which of those is failing.\n\n>> When fetching with `git fetch`, submodules are fetched without\n>> specifying the remote name, which means Git guesses which remote you\n>> want to fetch from, which is documented at\n>> https://git-scm.com/docs/git-fetch. I believe (I haven't reread this\n>> very closely) this is, in order:\n>>\n>> - The remote of your branch, i.e. the value of the config value\n>>   `branch.<name>.remote`\n>\n> So basically if its checked out to a branch it will fetch from the\n> remote of that branch, but...\n>\n>> - origin\n>>\n>\n> It defaults to origin, so if you have the usual \"checked out as a\n> detached head\" style of submodule, it can't find the remote branch.\n\nYes, this sounds about right. I was quite certain that we only default\nto \"origin\", but I observe that \"git fetch\" doesn't fail if there is\nonly one remote and it is not named \"origin\". Perhaps I'm mistaken, or I\nsimply couldn't track down that logic.\n\n>> But... I'll mention another wrinkle for completeness' sake (though I\n>> don't think it applies to you). If you fetch using `git submodule\n>> update`, the submodule is fetched using a _named_ remote, specifically:\n>>\n>> - If the superproject has a branch checked out, it uses the name of the\n>>   superproject branch's remote.\n>\n> Right, so that explains why I can re-run git submodule update after a\n> git pull --rebase and it works.\n>\n> In theory wouldn't it make more sense to use the remote based on the\n> URL of the .gitmodules file?\n\nAh, yes that's one possibility we (the folks working on an improved\nSubmodules UX) have considered. Another would be to teach submodules to\nactually use branches correctly and to use the remotes of the branches.\n\nIn general, the project tries not to respect config coming directly from\n.gitmodules (c.f. [1]), but I agree that there's a lot of room for\nimprovement.\n\n[1] https://lore.kernel.org/git/xmqq35bze3rr.fsf@gitster.g\n\n>> - If the superproject does not have a branch checked out, it uses\n>>   \"origin\".\n>>\n>\n> I suppose one option would be to make this configurable. I started\n> using \"upstream\" as the default remote name for most of my\n> repositories when I began working with forks a lot more.\n\nMy hope is that the work I mentioned earlier makes this code obsolete\nand nobody ever has to configure this ;)\n\n>\n>> >\n>> > Thanks,\n>> > Jake\n"},{"id":"464644","messageId":"CA+P7+xqf_Q35C0VT8A-zCRf46zSbXHhH5EhTo2vLvTJ9B6jyow@mail.gmail.com","threadId":"58566","inReplyTo":"kl6lilkq0zpr.fsf@chooglen-macbookpro.roam.corp.google.com","subject":"Re: issue with submodules using origin remote unexpectadly","fromName":"Jacob Keller","fromEmail":"jacob.keller@gmail.com","sentAt":"2022-10-12T00:13:50Z","receivedAt":"2022-10-12T00:14:04Z","isPatch":false,"sender":{"key":"jacob.keller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/874719?v=4"},"body":"On Tue, Oct 11, 2022 at 3:20 PM Glen Choo <chooglen@google.com> wrote:\n>\n> Jacob Keller <jacob.keller@gmail.com> writes:\n>\n> > On Tue, Oct 4, 2022 at 11:12 AM Glen Choo <chooglen@google.com> wrote:\n> >>\n> >> Hi Jacob! Thanks for the report!\n> >>\n> >\n> > Thanks for responding!\n>\n> :)\n>\n> >> Or, if you could include a reproduction script, that would be really\n> >> helpful :)\n> >>\n> >\n> > I'm not sure how to do this, because it is only an intermittent\n> > failure. I suspect it has to do with when the submodule actually needs\n> > to update.\n> >\n> > Perhaps I can come up with something though. If I can, I'll send it as\n> > a new test.\n>\n> That would be greatly appreciated, thanks!\n>\n> If you find code pointers useful,\n>\n> - builtin/submodule--helper.c:fetch_in_submodule() contains the logic\n>   for fetching during \"git submodule update\"\n>\n> - submodule.c:fetch_submodules() contains the logic for fetching during\n>   \"git fetch --recurse-submodules\" (which is invoked by \"git pull\n>   --recurse-submodules\").\n>\n\nI was able to get a test highlighting the failure. It shows the case\nof a single remote working but adding another remote causes it to fail\nas it falls back to the 'origin'.\n\n>\n> >> >\n> >> > remote: Enumerating objects: 210, done.\n> >> > remote: Counting objects: 100% (207/207), done.\n> >> > remote: Compressing objects: 100% (54/54), done.\n> >> > remote: Total 210 (delta 123), reused 197 (delta 119), pack-reused 3\n> >> > Receiving objects: 100% (210/210), 107.20 KiB | 4.29 MiB/s, done.\n> >> > Resolving deltas: 100% (123/123), completed with 48 local objects.\n> >> > From <redacted>\n> >> > ...\n> >> > Fetching submodule submodule\n> >> > From <redacted>\n> >> >    85e0da7533d9..80cc886f1187  <redacted>\n> >> > Fetching submodule submodule2\n> >> > fatal: 'origin' does not appear to be a git repository\n> >> > fatal: Could not read from remote repository.\n> >> >\n> >> > Please make sure you have the correct access rights\n> >> > and the repository exists.\n> >> > Errors during submodule fetch:\n> >> >         submodule2\n> >>\n> >> I assume this is `git fetch` running in the superproject?\n> >>\n> >\n> > Its git pull --rebase, but I suppose as part of this it will run\n> > something equivalent to git fetch?\n>\n> Unfortunately, this doesn't narrow it down much because \"git pull\n> --recurse-submodules\" runs _both_ \"git fetch --recurse-submodules\" _and_\n> \"git submodule update [--rebase]\" ;) Without more context, it's not\n> clear which of those is failing.\n>\n\nIt's definitely \"git fetch --recurse-submodules\", the new test should show this.\n\n> >> When fetching with `git fetch`, submodules are fetched without\n> >> specifying the remote name, which means Git guesses which remote you\n> >> want to fetch from, which is documented at\n> >> https://git-scm.com/docs/git-fetch. I believe (I haven't reread this\n> >> very closely) this is, in order:\n> >>\n> >> - The remote of your branch, i.e. the value of the config value\n> >>   `branch.<name>.remote`\n> >\n> > So basically if its checked out to a branch it will fetch from the\n> > remote of that branch, but...\n> >\n> >> - origin\n> >>\n> >\n> > It defaults to origin, so if you have the usual \"checked out as a\n> > detached head\" style of submodule, it can't find the remote branch.\n>\n> Yes, this sounds about right. I was quite certain that we only default\n> to \"origin\", but I observe that \"git fetch\" doesn't fail if there is\n> only one remote and it is not named \"origin\". Perhaps I'm mistaken, or I\n> simply couldn't track down that logic.\n>\n\nWe definitely default to the single/lone remote, I have two tests, one\nwhich shows the single remote working and another which shows the\nadditional remote causing the failure.\n\n> >> But... I'll mention another wrinkle for completeness' sake (though I\n> >> don't think it applies to you). If you fetch using `git submodule\n> >> update`, the submodule is fetched using a _named_ remote, specifically:\n> >>\n> >> - If the superproject has a branch checked out, it uses the name of the\n> >>   superproject branch's remote.\n> >\n> > Right, so that explains why I can re-run git submodule update after a\n> > git pull --rebase and it works.\n> >\n> > In theory wouldn't it make more sense to use the remote based on the\n> > URL of the .gitmodules file?\n>\n> Ah, yes that's one possibility we (the folks working on an improved\n> Submodules UX) have considered. Another would be to teach submodules to\n> actually use branches correctly and to use the remotes of the branches.\n>\n\nYes, if we can have it checkout on a branch and just rewind that\nbranch to match the expected commit instead of having it in a detached\nstate, things would be much easier. I recall work being done on this\nyears ago, but it is quite a thorny problem.\n\n> In general, the project tries not to respect config coming directly from\n> .gitmodules (c.f. [1]), but I agree that there's a lot of room for\n> improvement.\n>\n\nRight. I think I'd rather go with a config option inside the\n.git/config [submodule] section. I don't think gitmodules itself needs\nto know this, just that the parent project could be informed of what\nremote to default to when fetching inside the submodule. That or\nsomehow unify the git submodule update code with the recursive\nfetching?\n\n> [1] https://lore.kernel.org/git/xmqq35bze3rr.fsf@gitster.g\n>\n> >> - If the superproject does not have a branch checked out, it uses\n> >>   \"origin\".\n> >>\n> >\n> > I suppose one option would be to make this configurable. I started\n> > using \"upstream\" as the default remote name for most of my\n> > repositories when I began working with forks a lot more.\n>\n> My hope is that the work I mentioned earlier makes this code obsolete\n> and nobody ever has to configure this ;)\n>\n\nYea. I definitely like the idea of using branches instead of a\ndetached head state.\n\nI think for now I can avoid this by just disabling recursive fetch in\nmy config, which at least gets around the problem well enough.\n\nAnother alternative I thought was maybe \"try to fetch every remote\"\ninstead of trying to fetch only a single remote?\n\n> >\n> >> >\n> >> > Thanks,\n> >> > Jake\n"}]}