{"thread":{"id":"49000","subject":"Fetch on submodule update","startedAt":"2018-08-01T17:18:51Z","lastAt":"2018-08-06T15:44:10Z","messageCount":6,"participants":["Robert Dailey","Jonathan Nieder"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"354170","messageId":"CAHd499B2VingKU-+7idv9FROYgCmOAzZ7YAQgWdwY1YUeTUVHA@mail.gmail.com","threadId":"49000","inReplyTo":null,"subject":"Fetch on submodule update","fromName":"Robert Dailey","fromEmail":"rcdailey.lists@gmail.com","sentAt":"2018-08-01T17:18:46Z","receivedAt":"2018-08-01T17:18:51Z","isPatch":false,"sender":{"key":"rcdailey.lists@gmail.com","avatar":null},"body":"Problem: I want to avoid recursively fetching submodules when I run a\n`fetch` command, and instead defer that operation to the next\n`submodule update`. Essentially I want `fetch.recurseSubmodules` to be\n`false`, and `get submodule update` to do exactly what it does with\nthe `--remote` option, but still use the SHA1 of the submodule instead\nof updating to the tip of the specified branch in the git modules\nconfig.\n\nI hope that makes sense. The reason for this ask is to\nimprove/streamline workflow in parent repositories. There are cases\nwhere I want to quickly fetch only the parent repository, even if a\nsubmodule changes, to perform some changes that do not require the\nsubmodule itself (yet). Then at a later time, do `submodule update`\nand have it automatically fetch when the SHA1 it's updating to does\nnot exist (because the former fetch operation for the submodule was\nskipped). For my case, it's very slow to wait on submodules to\nrecursively fetch when I only wanted to fetch the parent repo for the\nspecific task I plan to do.\n\nIs this possible right now through some variation of configuration?\n"},{"id":"354204","messageId":"20180801223439.GA189024@aiede.svl.corp.google.com","threadId":"49000","inReplyTo":"CAHd499B2VingKU-+7idv9FROYgCmOAzZ7YAQgWdwY1YUeTUVHA@mail.gmail.com","subject":"Re: Fetch on submodule update","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2018-08-01T22:34:39Z","receivedAt":"2018-08-01T22:34:49Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi,\n\nRobert Dailey wrote:\n\n> Problem: I want to avoid recursively fetching submodules when I run a\n> `fetch` command, and instead defer that operation to the next\n> `submodule update`. Essentially I want `fetch.recurseSubmodules` to be\n> `false`, and `get submodule update` to do exactly what it does with\n> the `--remote` option, but still use the SHA1 of the submodule instead\n> of updating to the tip of the specified branch in the git modules\n> config.\n>\n> I hope that makes sense. The reason for this ask is to\n> improve/streamline workflow in parent repositories. There are cases\n> where I want to quickly fetch only the parent repository, even if a\n> submodule changes, to perform some changes that do not require the\n> submodule itself (yet). Then at a later time, do `submodule update`\n> and have it automatically fetch when the SHA1 it's updating to does\n> not exist (because the former fetch operation for the submodule was\n> skipped). For my case, it's very slow to wait on submodules to\n> recursively fetch when I only wanted to fetch the parent repo for the\n> specific task I plan to do.\n>\n> Is this possible right now through some variation of configuration?\n\nCan you say more about the overall workflow?  This seems quite different\nfrom what we've been designing --recurse-submodules around:\n\n- avoiding the end user ever having to use the \"git submodule\" command,\n  except to add, remove, or reconfigure submodules\n\n- treating the whole codebase as something like one project, so that\n  \"git checkout --recurse-submodules <commit>\" always checks out the\n  same state\n\nMore details about the application would help with better\nunderstanding whether it can fit into this framework, or whether it's\na case where you'd want to set \"submodule.recurse\" to false to have\nmore manual control.\n\nThanks and hope that helps,\nJonathan\n"},{"id":"354212","messageId":"20180802060842.GB10757@aiede.svl.corp.google.com","threadId":"49000","inReplyTo":"CAHd499B2VingKU-+7idv9FROYgCmOAzZ7YAQgWdwY1YUeTUVHA@mail.gmail.com","subject":"Re: Fetch on submodule update","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2018-08-02T06:08:42Z","receivedAt":"2018-08-02T06:08:48Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi again,\n\nRobert Dailey wrote:\n\n> Problem: I want to avoid recursively fetching submodules when I run a\n> `fetch` command, and instead defer that operation to the next\n> `submodule update`. Essentially I want `fetch.recurseSubmodules` to be\n> `false`, and `get submodule update` to do exactly what it does with\n> the `--remote` option, but still use the SHA1 of the submodule instead\n> of updating to the tip of the specified branch in the git modules\n> config.\n\nI think I misread this the first time.  I got distracted by your\nmention of the --remote option, but you mentioned you want to use the\nSHA-1 of the submodule listed, so that was silly of me.\n\nI think you'll find that \"git fetch --no-recurse-submodules\" and \"git\nsubmodule update\" do exactly what you want.  \"git submodule update\"\ndoes perform a fetch (unless you pass --no-fetch).\n\nLet me know how it goes. :)\n\nI'd still be interested in hearing more about the nature of the\nsubmodules involved --- maybe `submodule.fetchJobs` would help, or\nmaybe this is a workflow where a tool that transparently fetches\nsubmodules on demand like\nhttps://gerrit.googlesource.com/gitfs/+/master/docs/design.md would be\nuseful (I'm not recommending using slothfs for this today, since it's\nread-only, but it illustrates the idea).\n\nThanks,\nJonathan\n"},{"id":"354612","messageId":"CAHd499CFHUwpNdYp2sMjChi_WWzSihi5VRzLMQbKsGrKf_Ktcg@mail.gmail.com","threadId":"49000","inReplyTo":"20180802060842.GB10757@aiede.svl.corp.google.com","subject":"Re: Fetch on submodule update","fromName":"Robert Dailey","fromEmail":"rcdailey.lists@gmail.com","sentAt":"2018-08-06T14:45:01Z","receivedAt":"2018-08-06T14:45:07Z","isPatch":false,"sender":{"key":"rcdailey.lists@gmail.com","avatar":null},"body":"On Thu, Aug 2, 2018 at 1:08 AM, Jonathan Nieder <jrnieder@gmail.com> wrote:\n> I think I misread this the first time.  I got distracted by your\n> mention of the --remote option, but you mentioned you want to use the\n> SHA-1 of the submodule listed, so that was silly of me.\n>\n> I think you'll find that \"git fetch --no-recurse-submodules\" and \"git\n> submodule update\" do exactly what you want.  \"git submodule update\"\n> does perform a fetch (unless you pass --no-fetch).\n>\n> Let me know how it goes. :)\n>\n> I'd still be interested in hearing more about the nature of the\n> submodules involved --- maybe `submodule.fetchJobs` would help, or\n> maybe this is a workflow where a tool that transparently fetches\n> submodules on demand like\n> https://gerrit.googlesource.com/gitfs/+/master/docs/design.md would be\n> useful (I'm not recommending using slothfs for this today, since it's\n> read-only, but it illustrates the idea).\n\nHi thanks for your response, sorry I am a bit late getting back with you.\n\nMaybe my workflow is dated, because I'm still used to treating\nsubmodules as distinctly separated and independent things. I realize\nsubmodule recursion is becoming more inherent in many high level git\ncommands, but outside of git there are separation issues that make\nthis workflow doomed to be non-seamless. For example, pull requests\nwill never offer the same uniformity: You will still have 1 pull\nrequest per submodule. There's also the issue of log audits: You\ncannot use blame, log, bisect, or other \"diagnostic\" commands to\nintrospect into submodules \"as if\" they were subtree or something of\nthe like (i.e. truly part of the DAG). A more realistic example of one\nof the common questions I still can't answer easily is: \"How do you\ndetermine which commit in a submodule made it into which release of\nthe software?\" In the case where the parent repository has the\nannotated tags (representing software release milestones), and the\nsubmodule is just a common library (which does not have those tags and\nhas no release cycle). Anyway, none of these issues are particularly\nrelated but they do contribute to the answer to your question\nregarding my workflow and use cases. The list goes on but I hope you\nget the idea.\n\nSome of the more functional issues are performance related: I am aware\nenough, at times, that I can save time (in both local operations and\nnetwork overhead) by skipping submodules. For example, if I know that\nI'm merging mainline branches, I do not need to mess with the\nsubmodules (I can fetch, merge, commit, push from the parent repo\nwithout messing with the submodules. This saves me time). If\n`fetchJobs` was also `updateJobs`, i.e. you could update submodules in\nparallel too, that might make this less of an issue. Think of\nrepositories [like boost][1] that have (I think) over a hundred\nsibling submodules: Fetching 8 in parallel *and* doing `submodule\nupdate` in parallel 8 times might also speed things up. There's also\n`git status`, that if it recurses into submodules, is also\nsignificantly slow in the boost case (I'm not sure if it is\nparallelized).\n\nAgain, none of this is particularly related, but just to give you more\ncontext on the \"why\" for my ask. Sorry if I'm dragging this out too\nfar.\n\nThe TLDR is that I do prefer the manual control. Automatic would be\ngreat if submodules were treated as integrated in a similar manner to\nsubtree, but it's not there. I wasn't aware that `submodule update`\ndid a fetch, because sometimes if I do that, I get errors saying SHA1\nis not present (because the submodule did not get fetched). Granted I\nhaven't seen this in a while, so maybe the fetch on submodule update\nis a newer feature. Do you know what triggers the fetch on update\nwithout --remote? Is it the missing SHA1 that triggers it, or is it\nfetching unconditionally?\n\nThanks for confirming it behaves as I already wanted. And as you can\ntell, I'm also happy to further discuss motivation / use cases /\ndetails related to overall usage of submodules if you'd like. I'm\nhappy to help however I can!\n\n[1]: https://github.com/boostorg/boost\n"},{"id":"354628","messageId":"20180806154122.GB105466@aiede.svl.corp.google.com","threadId":"49000","inReplyTo":"CAHd499CFHUwpNdYp2sMjChi_WWzSihi5VRzLMQbKsGrKf_Ktcg@mail.gmail.com","subject":"Re: Fetch on submodule update","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2018-08-06T15:41:22Z","receivedAt":"2018-08-06T15:41:27Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Robert Dailey wrote:\n\n>                                                  Automatic would be\n> great if submodules were treated as integrated in a similar manner to\n> subtree, but it's not there. I wasn't aware that `submodule update`\n> did a fetch, because sometimes if I do that, I get errors saying SHA1\n> is not present (because the submodule did not get fetched). Granted I\n> haven't seen this in a while, so maybe the fetch on submodule update\n> is a newer feature. Do you know what triggers the fetch on update\n> without --remote? Is it the missing SHA1 that triggers it, or is it\n> fetching unconditionally?\n\nThanks for this and the rest of the context you sent.  It's very\nhelpful.\n\nThe relevant code in git-submodule.sh is\n\n\t# Run fetch only if $sha1 isn't present or it\n\t# is not reachable from a ref.\n\tis_tip_reachable \"$sm_path\" \"$sha1\" ||\n\tfetch_in_submodule \"$sm_path\" $depth ||\n\tsay \"$(eval_gettext \"Unable to fetch in submodule path '\\$displaypath'\")\"\n\n\t# Now we tried the usual fetch, but $sha1 may\n\t# not be reachable from any of the refs\n\tis_tip_reachable \"$sm_path\" \"$sha1\" ||\n\tfetch_in_submodule \"$sm_path\" $depth \"$sha1\" ||\n\tdie \"$(eval_gettext \"Fetched in submodule path '\\$displaypath', but it did not contain \\$sha1. Direct fetching of that commit failed.\")\"\n\nThe fallback to fetching by SHA-1 was introduced in v2.8.0-rc0~9^2\n(submodule: try harder to fetch needed sha1 by direct fetching sha1,\n2018-02-23).\n\nJonathan\n"},{"id":"354629","messageId":"CAHd499AcgCNfZtthwsqiki7JKrDTmMzq48z12H5Mf+mLKrf_QA@mail.gmail.com","threadId":"49000","inReplyTo":"20180806154122.GB105466@aiede.svl.corp.google.com","subject":"Re: Fetch on submodule update","fromName":"Robert Dailey","fromEmail":"rcdailey.lists@gmail.com","sentAt":"2018-08-06T15:44:05Z","receivedAt":"2018-08-06T15:44:10Z","isPatch":false,"sender":{"key":"rcdailey.lists@gmail.com","avatar":null},"body":"On Mon, Aug 6, 2018 at 10:41 AM, Jonathan Nieder <jrnieder@gmail.com> wrote:\n> Robert Dailey wrote:\n>\n>>                                                  Automatic would be\n>> great if submodules were treated as integrated in a similar manner to\n>> subtree, but it's not there. I wasn't aware that `submodule update`\n>> did a fetch, because sometimes if I do that, I get errors saying SHA1\n>> is not present (because the submodule did not get fetched). Granted I\n>> haven't seen this in a while, so maybe the fetch on submodule update\n>> is a newer feature. Do you know what triggers the fetch on update\n>> without --remote? Is it the missing SHA1 that triggers it, or is it\n>> fetching unconditionally?\n>\n> Thanks for this and the rest of the context you sent.  It's very\n> helpful.\n>\n> The relevant code in git-submodule.sh is\n>\n>         # Run fetch only if $sha1 isn't present or it\n>         # is not reachable from a ref.\n>         is_tip_reachable \"$sm_path\" \"$sha1\" ||\n>         fetch_in_submodule \"$sm_path\" $depth ||\n>         say \"$(eval_gettext \"Unable to fetch in submodule path '\\$displaypath'\")\"\n>\n>         # Now we tried the usual fetch, but $sha1 may\n>         # not be reachable from any of the refs\n>         is_tip_reachable \"$sm_path\" \"$sha1\" ||\n>         fetch_in_submodule \"$sm_path\" $depth \"$sha1\" ||\n>         die \"$(eval_gettext \"Fetched in submodule path '\\$displaypath', but it did not contain \\$sha1. Direct fetching of that commit failed.\")\"\n>\n> The fallback to fetching by SHA-1 was introduced in v2.8.0-rc0~9^2\n> (submodule: try harder to fetch needed sha1 by direct fetching sha1,\n> 2018-02-23).\n\nYep, that's the root cause; I was basing my concerns on a legacy\nissue. I just had avoided using `update` when I expected a fetch, so I\nnever saw the issue again, and thus didn't realize it was corrected.\nVery helpful. Thanks again!\n"}]}