{"thread":{"id":"56362","subject":"Automatically Handling Using/Checking Out Branches With One or More Different Contained Submodules?","startedAt":"2021-08-24T12:01:26Z","lastAt":"2021-08-28T01:23:47Z","messageCount":3,"participants":["Bryce Glover","Philippe Blain"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"433542","messageId":"CALH-JHvKjK7KU+Z_R7kG291DQKyb3f=LwxcbP4fn-qL2eeosBQ@mail.gmail.com","threadId":"56362","inReplyTo":"CALH-JHvNHVvPWap8eiMaQ6HAJOBa4W5xuA9V_s7zPLubgwaRow@mail.gmail.com","subject":"Automatically Handling Using/Checking Out Branches With One or More Different Contained Submodules?","fromName":"Bryce Glover","fromEmail":"randomdsdevel@gmail.com","sentAt":"2021-08-24T12:00:00Z","receivedAt":"2021-08-24T12:01:26Z","isPatch":false,"sender":{"key":"randomdsdevel@gmail.com","avatar":null},"body":"(Note:  If this question would fit better on the git-users Google\nGroup, I apologize, but I saw that, unlike there — unless I overlooked\nsomething? —, you could send messages here even if you weren't a list\nsubscriber.)\n\nTo whom it may concern,\n\n     Currently, the only method I've seen that you can reliably use to\nswitch between different branches when they don't all have the same\ncontained submodules comes from the Stack Overflow answer at\n<https://stackoverflow.com/a/64690495/3319611>.  I'll reproduce the\nBash snippet it presents as a solution here for completeness's sake:\n\n```\nexport TARGET_BRANCH=\"my-branch-name\"\nexport CURRENT_BRANCH=$(git rev-parse --abbrev-ref HEAD)\nif [ -f \".gitmodules\" ]; then\n  git submodules deinit --all\n  mkdir -p .git/git-tools/modules\n  mv .git/modules .git/git-tools/modules/$CURRENT_BRANCH\nfi\n\ngit checkout $TARGET_BRANCH\n\nif [ -f \".gitmodules\" ]; then\n  if [ -f \".git/git-tools/modules/$TARGET_BRANCH\" ]; then\n    git mv .git/git-tools/modules/$TARGET_BRANCH .git/modules\n  fi\n\n  git submodule sync && git submodule update --init\nfi\n```\n\nThis involves invoking some actions before '`git checkout`,' so I\ncouldn't have a couple of Git hooks handle this since, per '`git help\nhooks`,' Git doesn't implement a 'pre-checkout' hook, only a\npost-checkout one.  That wouldn't be enough of a use case to motivate\nadding that, though, would it?  Alternatively, '`git checkout`'\nwould, ideally, handle this automatically, perhaps when requested by\nflag if it wouldn't make sense for this behavior to be the default\none.  I don't know if I'd personally be up to contributing either one\nor both of either of those approaches, at least not right away, but,\nhypothetically, how involved might that turn out to be?\n\nCurious,\n     Bryce Glover, an amateur Git user\n     RandomDSdevel@gmail.com\n"},{"id":"433975","messageId":"5a70d535-47b0-a4ea-b4e4-572a1bcfe997@gmail.com","threadId":"56362","inReplyTo":"CALH-JHvKjK7KU+Z_R7kG291DQKyb3f=LwxcbP4fn-qL2eeosBQ@mail.gmail.com","subject":"Re: Automatically Handling Using/Checking Out Branches With One or More Different Contained Submodules?","fromName":"Philippe Blain","fromEmail":"levraiphilippeblain@gmail.com","sentAt":"2021-08-27T22:48:21Z","receivedAt":"2021-08-27T22:48:13Z","isPatch":false,"sender":{"key":"levraiphilippeblain@gmail.com","avatar":"https://avatars.githubusercontent.com/u/44212482?v=4"},"body":"Hi Bryce,\n\nLe 2021-08-24 à 08:00, Bryce Glover a écrit :\n> (Note:  If this question would fit better on the git-users Google\n> Group, I apologize, but I saw that, unlike there — unless I overlooked\n> something? —, you could send messages here even if you weren't a list\n> subscriber.)\n> \n> To whom it may concern,\n> \n>       Currently, the only method I've seen that you can reliably use to\n> switch between different branches when they don't all have the same\n> contained submodules comes from the Stack Overflow answer at\n> <https://stackoverflow.com/a/64690495/3319611>.  I'll reproduce the\n> Bash snippet it presents as a solution here for completeness's sake:\n> \n> ```\n> export TARGET_BRANCH=\"my-branch-name\"\n> export CURRENT_BRANCH=$(git rev-parse --abbrev-ref HEAD)\n> if [ -f \".gitmodules\" ]; then\n>    git submodules deinit --all\n>    mkdir -p .git/git-tools/modules\n>    mv .git/modules .git/git-tools/modules/$CURRENT_BRANCH\n> fi\n> \n> git checkout $TARGET_BRANCH\n> \n> if [ -f \".gitmodules\" ]; then\n>    if [ -f \".git/git-tools/modules/$TARGET_BRANCH\" ]; then\n>      git mv .git/git-tools/modules/$TARGET_BRANCH .git/modules\n>    fi\n> \n>    git submodule sync && git submodule update --init\n> fi\n> ```\n> \n> This involves invoking some actions before '`git checkout`,' so I\n> couldn't have a couple of Git hooks handle this since, per '`git help\n> hooks`,' Git doesn't implement a 'pre-checkout' hook, only a\n> post-checkout one.  That wouldn't be enough of a use case to motivate\n> adding that, though, would it?  Alternatively, '`git checkout`'\n> would, ideally, handle this automatically, perhaps when requested by\n> flag if it wouldn't make sense for this behavior to be the default\n> one.  I don't know if I'd personally be up to contributing either one\n> or both of either of those approaches, at least not right away, but,\n> hypothetically, how involved might that turn out to be?\nThe script above is in my opinion complicated and uneeded. I will copy below some content\nfrom the original stackoverflow post, and reply inline. The bottom line is:\n\n1. 'git checkout --recurse-submodules $ref' should always work in an ideal\nworld, but there are still some missing pieces as of today.\n2. 'git checkout $ref && git submodule sync --recursive && git submodule update\n--recursive' should always work.\n\nHere goes:\n\n> The problem I am facing is that I cannot figure out a set of git commands that\n> will consistently work in all cases when switching between branches if the\n> branch has submodules. In my test, I have one repo with 3 branches, one branch\n> has no submodules (master), another module [sic, should be \"branch\"] has 1 submodule (submodule-test),\n> and another branch has another submodule (submodule-test2) that points to a\n> different repository (at the same path). \n[...]\n\n> In example here is one set of commands and their failure in git 2.20.1\n> \n> cd /tmp\n> git clone git@github.com:simpleviewinc/git-tools-test.git ./checkout --recurse-submodules\n> cd checkout\n> git checkout submodule-test\n> git submodule sync\n> git submodule update\n> # branch submodule-test fully checked out, all submodules downloaded, looking good!\n> git submodule deinit --all\n> git checkout submodule-test2\n> git submodule sync\n> git submodule update\n> fatal: remote error: upload-pack: not our ref c1bba6e3969937125248ee46e308a8efec8ac654\n> Fetched in submodule path 'submodule', but it did not contain c1bba6e3969937125248ee46e308a8efec8ac654. Direct fetching of that commit failed.\n> It fails because it uses the wrong submodule remote, even though I thought that was the explicit purpose of submodule sync. \n\nThe reason it fails is because the 'git submodule sync' did *not* update the\n'remote.origin.url' configuration in the Git config file of the submodule, i.e.\n/tmp/checkout/.git/modules/submodule/config, despite (confusingly) outputing\n\"Synchronizing submodule url for 'submodule'\".  It does change the\n'submodule.$name.url' value in the config file *of the superproject*, though,\n(/tmp/checkout/.git/config) but 'git submodule update' only uses\n'remote.origin.url' in the submodule's config file (if it exists).\n\nThe reason why the second 'git submodule sync' does not change\nremote.origin.url is because the submodule has been deinitialized when the\ncommand is run, because the previous command is 'git submodule deinit --all',\nwhich deinitializes all submodules. So this second step of changing\n'remote.origin.url' is skipped for deinitialized submodules (this fact is\nmissing from the doc).\n\nThe following sequence would be the correct one:\n\ngit clone git@github.com:simpleviewinc/git-tools-test.git ./checkout [--recurse-submodules] # --recurse-submodules is optional\ncd checkout\ngit checkout submodule-test\ngit submodule update --init\ngit checkout submodule-test2\ngit submodule sync\ngit submodule update\n# and we can switch back\ngit checkout submodule-test\ngit submodule sync\ngit submodule update\n\nNote that the correct command to initialize  all submodules is 'git submodule\ninit' or, to initialize, clone and check them out in a single step, 'git\nsubmodule update --init'. The fact that the first 'git submodule sync' works to\ninitialize the submodule is in fact due to using '--recurse-submodules' for\n'git clone', which sets 'submodule.active' to the match-all pathspec '.' in the\nsuperproject's config, and 'git submodule sync' recurses into *active*\nsubmodules (this is also missing from the doc).\n\nSo for 'git checkout', what you want to achieve can be done with a\npost-checkout hook:\n\n```shell\n#!/bin/sh\n\n# If the checkout was a branch checkout [1], update the submodules\n# to the commits recorded in the superproject\n# [1] https://git-scm.com/docs/githooks#_post_checkout\n\nprevious_head=$1\nnew_head=$2\ncheckout_type_flag=$3\n\nif [ \"$checkout_type_flag\" -eq 1 ] ; then\n   git submodule sync --recursive\n   git submodule update --init --recursive\nfi\n```\n\nHere I add '--recursive' to both commands for extra safety in case any of your\nsubmodules themselve contain submodules.\n\nNow, about 'git checkout --recurse-submodules':\n\n> I tried --recurse-submodules but that fails too, but this time when checking\n> from a branch without submodules to a branch with submodules.\n> \n> cd /tmp\n> git clone git@github.com:simpleviewinc/git-tools-test.git ./checkout --recurse-submodules\n> cd checkout\n> git checkout submodule-test --recurse-submodules\n> fatal: not a git repository: ../.git/modules/submodule\n> fatal: could not reset submodule index\n\nYeah, that one is bad. The reason it fails is because:\n\n1. 'git clone --recurse-submodules' in fact runs a simple 'git submodule update\n--init --recursive' at the end of the process, at the step where it's checking\nout the working tree. This means that only submodules present *in the branch\nbeing checked out* get initialized and cloned.\n\n2. 'git clone --recurse-submodules' *always* writes 'submodule.active=.' to the\nsuperproject's config.\n\n3. 'git checkout --recurse-submodules' recurses into active submodules, and for\nthat it needs access to the Git repository of the submodule, which does not\nexist yet since it was not cloned.\n\nThis is the same error you get when you try to recursively checkout an older\nrevision that contains a submodule that was since deleted [1], [2].\n\nI suggested in those threads a few ways all this could be improved; here is my\nup to date take on the subject:\n\n1. 'git clone --recurse-submodules' should be able to at least clone *all*\nsubmodules for *all* branches that are cloned, and put their Git directory in\n.git/modules/.  This would allow your use case to \"just work\" with 'git\ncheckout --recurse-submodules'\n\n2. 'git clone --recurse-submodules' could also be taught to clone all\nsubmodules for *all* revisions of *all* branches that are cloned. This would\nallow the \"deleted submodule\" cases I mentioned to work, but would not be\nwanted in all situations, so it could be a supplementary flag to 'git clone'.\n\n3. 'git fetch' should be taught to clone new submodules.\n\n4. 'git checkout --recurse-submodules' could be taught to clone missing\nsubmodules and fetch missing submodules commits.  This would cover both your\nuse case as well as the \"deleted submodule\" use case.\n\n5. In any case, 'git checkout --recurse-submodules' should not abort midway if\nit can't find the Git repository of the submodule and leave '.gitmodules'\nuntracked in the working tree, and a 'config' file in .git/modules/$name/ with\nonly 'core.worktree' set.\n\nHope that helps,\n\nPhilippe Blain.\n\n[1] https://lore.kernel.org/git/20200501005432.h62dnpkx7feb7rto@glandium.org/T/#u\n[2] https://lore.kernel.org/git/CAE5ih78zCR0ZdHAjoxguUb3Y6KFkZcoxJjhS7rkbtZpr+d1n=g@mail.gmail.com/t/#u\n"},{"id":"433994","messageId":"CALH-JHuTvAKtBwYaxAs8Z6Xdw5RBY4JZ-d9srq3Q_MxBCSnh6A@mail.gmail.com","threadId":"56362","inReplyTo":"5a70d535-47b0-a4ea-b4e4-572a1bcfe997@gmail.com","subject":"Re: Automatically Handling Using/Checking Out Branches With One or More Different Contained Submodules?","fromName":"Bryce Glover","fromEmail":"randomdsdevel@gmail.com","sentAt":"2021-08-28T01:23:04Z","receivedAt":"2021-08-28T01:23:47Z","isPatch":false,"sender":{"key":"randomdsdevel@gmail.com","avatar":null},"body":"Dear Mr. Blain,\n\n     Yes, that does help.  Thank you very much for the extensive reply\nand detailed explanation!\n\nMuch appreciated,\n     Bryce Glover\n     RandomDSdevel@gmail.com\n"}]}