{"thread":{"id":"60845","subject":"submodules: multiple alternative relative URL schemes?","startedAt":"2024-02-04T22:14:32Z","lastAt":"2024-02-04T22:14:32Z","messageCount":1,"participants":["Matthew Ogilvie"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"487902","messageId":"ZcAMQpQlVsV2w2y2@comcast.net","threadId":"60845","inReplyTo":null,"subject":"submodules: multiple alternative relative URL schemes?","fromName":"Matthew Ogilvie","fromEmail":"mmogilvi+git@zoho.com","sentAt":"2024-02-04T22:14:26Z","receivedAt":"2024-02-04T22:14:32Z","isPatch":false,"sender":{"key":"mmogilvi+git@zoho.com","avatar":null},"body":"Background:\n\nAt work we are switching to submodules to share various\nutility code between mostly-unrelated projects and programs.\n\nWhen working on any one project, I commonly fetch and push\npreliminary changes beween numerous different sandboxes in\nvarious places (same machine, different machines, VM's, etc).\nBasically, I make extensive use of git's distributed nature.\n\nIssue:\n\nHowever, submodules don't seem to streamline fetching/pushing\nthe whole collection of modules together from varied sources\nvery well, without manually configuring or manipulating each\nsubmodule and its' \"remote\"s individually.  For example, if\nI recursively clone from a local sandbox with local changes\ninto another local sandbox, it will clone the main superproject\nfrom the local source, but the submodules will be downloaded\nfrom the URLs in the .gitmodules file, not the local copies\n(that might have local changes that will be missed).\n\n-----\n\nRelative URLs could maybe help, except there seems to be at least\ntwo common relative URL repository schemes/organizations that\nare often needed, and probably more:\n\n1. Subproject bare repositories are immediately adjacent to the\nsuperproject repository, such that relative URLs like\n\"../submodule1.git\" should work.\n\nThis is probably common in web-based hosting of personal \"forks\"\nlike in github, bitbucket, or similar.\n\nIt is also the most obvious way to organize multiple\nsemi-related repositories on a small self-hosted server.\n\n2. Or you want to fetch changes between sandboxes, perhaps\nto test code changes on multiple platforms, or to continue someone\nelse's incomplete work.  In this case, relative links like\n\"./.git/modules/submodule1\" would probably work.  But bare/origin/main\nrepositories would rarely be organized like this.\n\n3. \"Official\" configured upstream URLs may also be organized in other\nways.  For example, in something like github, different upstream URLs\nmay be \"owned\" by different owners, so they may URLs like\n\"BASE/OWNER1/MODULE1.git\", \"BASE/OWNER2/MODULE2.git\",\nwith different and semi-random \"OWNER\"s, not amenable to template\nstrings (see (E) below).  Or different submodule upstreams may be on\ncompletely different servers...\n\n-----\n\nBasically, I would like to restore or at least improve git's\nability to transparently/smoothly work in a \"distributed\" manner\n(no \"blesssed main upstream repositories\") when a project\nrequires certain submodules.\n\nSome Ideas:\n\nA. Is there something I'm missing?  Is there some good solution\nI just haven't stumbled over?  Any documentation URLs and/or\nkeywords I should look for?\n\nB. Of course no matter what, you could always set up all your various\nremote URLs individually manually, or write special project-specific\nhelper scripts.  But something more automated and built-in\nwould be nice...\n\nC. future?: Potentially, the .gitmodules file could define multiple\ndifferent URLs for each submodule, distinguished by named URL\n\"schemes\" that could be specified on the command line.\n\"git clone\", \"git fetch\", \"git push\", \"git submodule\", etc could\nall take an optional \"scheme\" argument of some kind to indicate\nwhich sets of URLs to use...\n\nD. future?: Potentially hard-code some schemes?  1 and 2 are probably the\nmost common alternative relative URL schemes, so it might be worth\nexplicitly supporting them without requiring any explicit configuration\nchanges.  Maybe this would be adequate by itself; don't even support\nother \"alternative\" named schemes (multiple 3's)?\n\nE. future?: In some cases, a \"scheme\" for a whole set of URLs\ncould maybe be simplified into a single URL template string.\nSubstitute in the \"base\" and the submodule name to get a\nspecific URL...\n\nF. Potentially add some optional fallback logic?  If it fails to\nfetch/clone from one URL, try another scheme instead?  (But this\nmight leave things in a confusingly unpredictable/mixed state,\nparticularly if the network is flakey...)\n\nG. Different approach: Instead of separate repositories, maybe\nthe submodules could essentially all be in one repository, but\ndistinguished by using different tags/namespaces in the \"refs\"\nhierarchy.  (\"refs/heads/SUBMODULE/WHATEVER\"?\n\"refs/SUBMODULE/heads/WHATEVER\"?)  The \"original/main\" upstream\nmight still split up repositories, but clones/local changes/etc\ncould be more self-contained this way...  (Working out all the\nnuances to streamline this idea might be a whole separate\ndiscussion...)\n\nH. \"git subtree\" instead of submodules: If it was up to me, I like\nthe subtree solution.  Most of the time you can ignore the fact that\nsome code is shared, and only need to worry about it when actually\nsyncing/merging shared code between different projects, which is\ngenerally much rarer than day-to-day development tasks.\nBut unfortunately subtrees seems to be somewhat unpopular, as\ncurrently implemented: Isolated in \"contrib\", doing graph traversal\nlogic in shell script of all things (very slow, especially on\nWindows where many of my coworkers reside), etc.  And ultimately\nin my case I think I've already lost this argument.  So I'm trying\nto figure out how to make submodules work better.\n\nI. Any other ideas?\n\n-----\n\nIn the short term, I expect I'll have to resort to (B) (custom scripts).\n\nIn the longer term, depending on what people think and if I can find\nthe time, maybe I could actually implement some of these ideas\nin git for a future release?  What do people think of these ideas?\n\n                - Matthew Ogilvie\n"}]}