{"thread":{"id":"14505","subject":"[PATCH] Teach git submodule update to use distributed repositories","startedAt":"2008-07-17T12:08:19Z","lastAt":"2008-07-21T10:59:42Z","messageCount":23,"participants":["Nigel Magnay","Johannes Schindelin","Petr Baudis","Jakub Narebski","Junio C Hamano","Mark Levedahl"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"83672","messageId":"320075ff0807170508j3d3c1ef8j49df576fc47debe2@mail.gmail.com","threadId":"14505","inReplyTo":null,"subject":"[PATCH] Teach git submodule update to use distributed repositories","fromName":"Nigel Magnay","fromEmail":"nigel.magnay@gmail.com","sentAt":"2008-07-17T12:08:19Z","receivedAt":"2008-07-17T12:08:19Z","isPatch":true,"sender":{"key":"nigel.magnay@gmail.com","avatar":"https://gravatar.com/avatar/d85cf38287bef3a8e4fa02358d2756d7589f8676c5eeb881ce2f6d731e4526c3?d=mp&s=160"},"body":"When doing a git submodule update, it fetches any missing submodule\ncommits from the repository specified in .gitmodules. If you instead\nwant to pull from another repository, you currently need to do a fetch\nin each submodule by hand.\n\nSigned-off-by: Nigel Magnay \n---\nThis is my first attempt at adding things to help everyday usage of\ngit submodule.\n\nI don't usually write much shell script; and it's my first patch, so\nit's possible there are better ways to do these things..\n\n git-submodule.sh | 33 +++++++++++++++++++++++++++++++--\n 1 files changed, 31 insertions(+), 2 deletions(-)\n\ndiff --git a/git-submodule.sh b/git-submodule.sh\nindex 9228f56..40e1aa1 100755\n--- a/git-submodule.sh\n+++ b/git-submodule.sh\n@@ -5,7 +5,7 @@\n # Copyright (c) 2007 Lars Hjemli\n\n USAGE=\"[--quiet] [--cached] \\\n-[add [-b branch] ]|[status|init|update\n[-i|--init]|summary [-n|--summary-limit ] []] \\\n+[add [-b branch] ]|[status|init|update [-i|--init]\n[-o|--origin ] [-r|-refspec ]|summary\n[-n|--summary-limit ] []] \\\n [--] [...]\"\n OPTIONS_SPEC=\n . git-sh-setup\n@@ -15,6 +15,8 @@ command=\n branch=\n quiet=\n cached=\n+repository=\n+refspec=\n\n #\n # print stuff on stdout unless -q was specified\n@@ -270,6 +272,14 @@ cmd_update()\n \t\t\tshift\n \t\t\tcmd_init \"$@\" || return\n \t\t\t;;\n+\t\t-o|--origin)\n+\t\t\tshift\n+\t\t\trepository=$1\n+\t\t\t;;\n+\t\t-r|--refspec)\n+\t\t\tshift\n+\t\t\trefspec=$1\n+\t\t\t;;\n \t\t--)\n \t\t\tshift\n \t\t\tbreak\n@@ -311,7 +321,9 @@ cmd_update()\n\n \t\tif test \"$subsha1\" != \"$sha1\"\n \t\tthen\n-\t\t\t(unset GIT_DIR; cd \"$path\" && git-fetch &&\n+\t\t\tset_submodule_repository \"$repository\" \"$path\"\n+\n+\t\t\t(unset GIT_DIR; cd \"$path\" && git-fetch \"$subrepo\" \"$refspec\" &&\n \t\t\t\tgit-checkout -q \"$sha1\") ||\n \t\t\tdie \"Unable to checkout '$sha1' in submodule path '$path'\"\n\n@@ -320,6 +332,23 @@ cmd_update()\n \tdone\n }\n\n+#\n+# If we asked for a repository such as 'origin', just pass this through\n+# otherwise, try to calculate what the repository URL might be by\n+# adding the submodule path to the url, subtracting any /.git first\n+#\n+set_submodule_repository() {\n+\n+\tif [ -z `echo \"$1\" | grep '/'` ]\n+\tthen\n+\t\t# This is not a URL - just use the name\n+\t\tsubrepo=\"$1\"\n+\telse\n+\t\t# This is a URL. Chop off /.git if it's there, and add submodule path\n+\t\tsubrepo=\"${1/%\\/.git/}/$2\"\n+\tfi\n+}\n+\n set_name_rev () {\n \trevname=$( (\n \t\tunset GIT_DIR\n-- \n1.5.6.2\n"},{"id":"83673","messageId":"alpine.DEB.1.00.0807171311010.8986@racer","threadId":"14505","inReplyTo":"320075ff0807170508j3d3c1ef8j49df576fc47debe2@mail.gmail.com","subject":"Re: [PATCH] Teach git submodule update to use distributed repositories","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2008-07-17T12:13:11Z","receivedAt":"2008-07-17T12:13:11Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Thu, 17 Jul 2008, Nigel Magnay wrote:\n\n> When doing a git submodule update, it fetches any missing submodule\n> commits from the repository specified in .gitmodules.\n\nHuh? It takes what is in .git/config! Not what is in .gitmodules.\n\nSo if you have another remote (or URL, e.g. if you have ssh:// access, but \nthe .gitmodules file lists git://), just edit .git/config.\n\nI meant, that is the whole _point_ of having a two-step init/update \nprocedure.\n\nCiao,\nDscho\n"},{"id":"83675","messageId":"320075ff0807170521s26693381m60648468cce1c41c@mail.gmail.com","threadId":"14505","inReplyTo":"320075ff0807170520r200e546ejbad2ed103bd65f82@mail.gmail.com","subject":"Re: [PATCH] Teach git submodule update to use distributed repositories","fromName":"Nigel Magnay","fromEmail":"nigel.magnay@gmail.com","sentAt":"2008-07-17T12:21:28Z","receivedAt":"2008-07-17T12:21:28Z","isPatch":true,"sender":{"key":"nigel.magnay@gmail.com","avatar":"https://gravatar.com/avatar/d85cf38287bef3a8e4fa02358d2756d7589f8676c5eeb881ce2f6d731e4526c3?d=mp&s=160"},"body":"On Thu, Jul 17, 2008 at 1:13 PM, Johannes Schindelin\n wrote:\n> Hi,\n>\n> On Thu, 17 Jul 2008, Nigel Magnay wrote:\n>\n>> When doing a git submodule update, it fetches any missing submodule\n>> commits from the repository specified in .gitmodules.\n>\n> Huh? It takes what is in .git/config! Not what is in .gitmodules.\n>\n\nHuh? And where does .git/config get it from? Oh, that's right, .gitmodules.\n\n> So if you have another remote (or URL, e.g. if you have ssh:// access, but\n> the .gitmodules file lists git://), just edit .git/config.\n>\n\nSo for my usecase, you'd have me go in and change *evey single one* of\nmy submodule refs from the centralised repository, *every time* I want\nto do a peer review?\n\nDoesn't the current system strike you as being somewhat centralised in nature?\n\n> I meant, that is the whole _point_ of having a two-step init/update\n> procedure.\n>\n\nAre you just determined that submodules should remain useless for \"the\nrest of us\"?\n\n> Ciao,\n> Dscho\n>\n"},{"id":"83681","messageId":"alpine.DEB.1.00.0807171351380.8986@racer","threadId":"14505","inReplyTo":"320075ff0807170521s26693381m60648468cce1c41c@mail.gmail.com","subject":"Re: [PATCH] Teach git submodule update to use distributed repositories","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2008-07-17T12:58:07Z","receivedAt":"2008-07-17T12:58:07Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Thu, 17 Jul 2008, Nigel Magnay wrote:\n\n> On Thu, Jul 17, 2008 at 1:13 PM, Johannes Schindelin\n> wrote:\n>\n> > On Thu, 17 Jul 2008, Nigel Magnay wrote:\n> >\n> >> When doing a git submodule update, it fetches any missing submodule \n> >> commits from the repository specified in .gitmodules.\n> >\n> > Huh? It takes what is in .git/config! Not what is in .gitmodules.\n> \n> Huh? And where does .git/config get it from? Oh, that's right, \n> .gitmodules.\n\nOh, that's right, after \"git submodule init\". Right before you are \nsupposed to change them if your setup commands that.\n\n> > So if you have another remote (or URL, e.g. if you have ssh:// access, \n> > but the .gitmodules file lists git://), just edit .git/config.\n> \n> So for my usecase, you'd have me go in and change *evey single one* of \n> my submodule refs from the centralised repository, *every time* I want \n> to do a peer review?\n\nNo.\n\n> Doesn't the current system strike you as being somewhat centralised in \n> nature?\n\nNo.\n\n> > I meant, that is the whole _point_ of having a two-step init/update \n> > procedure.\n> \n> Are you just determined that submodules should remain useless for \"the \n> rest of us\"?\n\nNo.\n\nIf you really need to change the \"origin\" back and forth between reviews, \nwhile the committed state of the superproject stays the same, then \nsomething is seriously awkward and needs to be streamlined in your setup.\n\nBecause when the superproject's revision stays the same, \"git submodule \nupdate\" may fetch additional objects if you specify another remote, but it \nwill check out just the same revisions of the submodules. Because they \nwere committed as such.\n\nBut if you want to get objects from another server (as opposed to update \nthe submodules' working directories to the latest committed revisions), \nwhich happens to have the identical layout of the principal server (which \nI would deem another setup peculiarity to be fixed), you might want to \nlook into the recurse patch that was flying about on this list a few \nmonths back.\n\nHth,\nDscho\n"},{"id":"83690","messageId":"320075ff0807170703l57fe26d2h1e9c4db1c38dd6f1@mail.gmail.com","threadId":"14505","inReplyTo":"alpine.DEB.1.00.0807171351380.8986@racer","subject":"Re: [PATCH] Teach git submodule update to use distributed repositories","fromName":"Nigel Magnay","fromEmail":"nigel.magnay@gmail.com","sentAt":"2008-07-17T14:03:30Z","receivedAt":"2008-07-17T14:03:30Z","isPatch":true,"sender":{"key":"nigel.magnay@gmail.com","avatar":"https://gravatar.com/avatar/d85cf38287bef3a8e4fa02358d2756d7589f8676c5eeb881ce2f6d731e4526c3?d=mp&s=160"},"body":"On Thu, Jul 17, 2008 at 1:58 PM, Johannes Schindelin\n wrote:\n> Hi,\n>\n> On Thu, 17 Jul 2008, Nigel Magnay wrote:\n>\n>> On Thu, Jul 17, 2008 at 1:13 PM, Johannes Schindelin\n>> wrote:\n>>\n>> > On Thu, 17 Jul 2008, Nigel Magnay wrote:\n>> >\n>> >> When doing a git submodule update, it fetches any missing submodule\n>> >> commits from the repository specified in .gitmodules.\n>> >\n>> > Huh? It takes what is in .git/config! Not what is in .gitmodules.\n>>\n>> Huh? And where does .git/config get it from? Oh, that's right,\n>> .gitmodules.\n>\n> Oh, that's right, after \"git submodule init\". Right before you are\n> supposed to change them if your setup commands that.\n>\n>> > So if you have another remote (or URL, e.g. if you have ssh:// access,\n>> > but the .gitmodules file lists git://), just edit .git/config.\n>>\n>> So for my usecase, you'd have me go in and change *evey single one* of\n>> my submodule refs from the centralised repository, *every time* I want\n>> to do a peer review?\n>\n> No.\n>\n>> Doesn't the current system strike you as being somewhat centralised in\n>> nature?\n>\n> No.\n>\n>> > I meant, that is the whole _point_ of having a two-step init/update\n>> > procedure.\n>>\n>> Are you just determined that submodules should remain useless for \"the\n>> rest of us\"?\n>\n> No.\n>\n> If you really need to change the \"origin\" back and forth between reviews,\n> while the committed state of the superproject stays the same, then\n> something is seriously awkward and needs to be streamlined in your setup.\n>\n> Because when the superproject's revision stays the same, \"git submodule\n> update\" may fetch additional objects if you specify another remote, but it\n> will check out just the same revisions of the submodules. Because they\n> were committed as such.\n>\n> But if you want to get objects from another server (as opposed to update\n> the submodules' working directories to the latest committed revisions),\n> which happens to have the identical layout of the principal server (which\n> I would deem another setup peculiarity to be fixed), you might want to\n> look into the recurse patch that was flying about on this list a few\n> months back.\n\nThe layout wouldn't be the same - the submodules would be in the\ncorresponding subdirectories (I guess it could have some other,\nstranger layout, but I'd consider that peculiar). So you're right, the\nlayout is different, which makes editing the config all the more\ntedious.\n\nI don't want to change the *origin* back and forth. I want to be able\nto use repos with submodules in them as easily and as transparently\nand in the same distributed way as git allows me to do if they don't\ncontain submodules. I.E I don't want it to be such a sisyphean\nchallenge every time with umpteen scripts to complete a usecase that\nreally ought to be supported as standard. The very first thing that\nI've hit is that submodule update only talks to origin, so 'git pull\nfred && git submodule update' falls flat on its face. Why am I being\nforced to update config just to have a look-see at fred's project?\n\nYour attitude seems to be that the status-quo is in some way\ndesirable; \"It's no wonder that this tool is awkward to use in your\nworkflow.\". This workflow is really common, and there's actual, real\npeople on this list complaining about it. Don't we think it could be\nimproved to be non-awkward ?\n\nIn the ideal UI, it ought to be possible to make the use of projects\nwith submodules (almost) completely transparent, like it is in the\nvcs-that-dare-not-speak-it's-name.\n"},{"id":"83692","messageId":"alpine.DEB.1.00.0807171513560.8986@racer","threadId":"14505","inReplyTo":"320075ff0807170703l57fe26d2h1e9c4db1c38dd6f1@mail.gmail.com","subject":"Re: [PATCH] Teach git submodule update to use distributed repositories","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2008-07-17T14:16:20Z","receivedAt":"2008-07-17T14:16:20Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Thu, 17 Jul 2008, Nigel Magnay wrote:\n\n> Your attitude seems to be that the status-quo is in some way desirable; \n> \"It's no wonder that this tool is awkward to use in your workflow.\". \n> This workflow is really common, and there's actual, real people on this \n> list complaining about it. Don't we think it could be improved to be \n> non-awkward ?\n\nI do not think that the status quo is the best possible.\n\nBut I think that the way you go makes things so confusing that those who \nuse it apart from you will have problems.\n\nFor example, in your setup everybody would have to install _different_ \nremotes in every submodule.\n\nAnd then some would ask themselves why the original origin was not good \nenough.\n\nAnd others would specify \"-o origin\" all the time, thinking it was \nrequired.\n\nThere must be a better way to promote submodules to a usable state,\nDscho\n"},{"id":"83695","messageId":"20080717143833.GV32184@machine.or.cz","threadId":"14505","inReplyTo":"320075ff0807170508j3d3c1ef8j49df576fc47debe2@mail.gmail.com","subject":"Re: [PATCH] Teach git submodule update to use distributed repositories","fromName":"Petr Baudis","fromEmail":"pasky@suse.cz","sentAt":"2008-07-17T14:38:33Z","receivedAt":"2008-07-17T14:38:33Z","isPatch":true,"sender":{"key":"pasky@ucw.cz","avatar":"https://avatars.githubusercontent.com/u/18439?v=4"},"body":"On Thu, Jul 17, 2008 at 01:08:19PM +0100, Nigel Magnay wrote:\n> When doing a git submodule update, it fetches any missing submodule\n> commits from the repository specified in .gitmodules. If you instead\n> want to pull from another repository, you currently need to do a fetch\n> in each submodule by hand.\n> \n> Signed-off-by: Nigel Magnay \n\nI don't think it is good idea to hijack git submodule update for this.\nThis command has a specific purpose:\n\n\t\"When I pulled new version of the main tree, bring my\n\tsubmodule checkouts in line with whatever is specified\n\twithin the new tree revision.\"\n\nYour usage scenario has nothing to do with that, it is about \"batch\nmanipulation\" of all the submodules at once in a certain way. I think\nusing the same command for two conceptually pretty much unrelated\npurposes will only clutter up the UI, and we should think of a better\ngeneral interface pattern for these operations.\n\nIn the new git-submodule description, it is said that\n\n\t\"This command will manage the tree entries and contents of the\n\tgitmodules file for you.\"\n\nand I think we should keep it at this; anything that is related to\nsubmodules, but does not do this directly, would IMHO live better\nas some kind of \"submodule-recursive\" extension of other existing\ncommands. Say, would this particular need of yours be served by a\nhypothetical command like\n\n\tgit checkout --submodules nifty\n\nto check out branch nifty of all submodules or am I misunderstanding\nwhat are you trying to achieve?\n\nIf not, then actually even _much_ more elegant solution for this\nparticular problem would be to store submodule.*.branch in .gitmodules\nappropriate to the -b parameter of git submodule add. Then, in branch\n'nifty' of the main project, you would set submodule.*.branch to 'nifty'\ntoo. Then, in order to bring all the submodules to the latest version,\nI could imagine something like\n\n\tgit pull --submodules\n\n(and possibly just abort at the first sight of a conflict, for\nstarters).\n\nLet's figure up some UI that is nifty and clean. ;-)\n\n-- \n\t\t\t\tPetr \"Pasky\" Baudis\nGNU, n. An animal of South Africa, which in its domesticated state\nresembles a horse, a buffalo and a stag. In its wild condition it is\nsomething like a thunderbolt, an earthquake and a cyclone. -- A. Pierce\n"},{"id":"83700","messageId":"320075ff0807170807l1537e34ev510deda537e4d11e@mail.gmail.com","threadId":"14505","inReplyTo":"alpine.DEB.1.00.0807171513560.8986@racer","subject":"Re: [PATCH] Teach git submodule update to use distributed repositories","fromName":"Nigel Magnay","fromEmail":"nigel.magnay@gmail.com","sentAt":"2008-07-17T15:07:11Z","receivedAt":"2008-07-17T15:07:11Z","isPatch":true,"sender":{"key":"nigel.magnay@gmail.com","avatar":"https://gravatar.com/avatar/d85cf38287bef3a8e4fa02358d2756d7589f8676c5eeb881ce2f6d731e4526c3?d=mp&s=160"},"body":"On Thu, Jul 17, 2008 at 3:16 PM, Johannes Schindelin\n wrote:\n> Hi,\n>\n> On Thu, 17 Jul 2008, Nigel Magnay wrote:\n>\n>> Your attitude seems to be that the status-quo is in some way desirable;\n>> \"It's no wonder that this tool is awkward to use in your workflow.\".\n>> This workflow is really common, and there's actual, real people on this\n>> list complaining about it. Don't we think it could be improved to be\n>> non-awkward ?\n>\n> I do not think that the status quo is the best possible.\n>\n> But I think that the way you go makes things so confusing that those who\n> use it apart from you will have problems.\n>\nOk\n\n> For example, in your setup everybody would have to install _different_\n> remotes in every submodule.\n>\n> And then some would ask themselves why the original origin was not good\n> enough.\n>\n> And others would specify \"-o origin\" all the time, thinking it was\n> required.\n>\n> There must be a better way to promote submodules to a usable state,\n\nMy attempt was to try and do some small simple things, but you could\nwell be right, that might make some commands bloat out with\nunneccessary options just to get something done, and that would be\nbad.\n\nStepping back - lets try to come up with a better way (please comment\nand and critique)\n\nWhat we'd like (to start with) is for\n$ git pull fred\n\nperhaps with --submodules (as Petr mentions), perhaps with config\nsettings and caveats, to produce a result that means you don't need to\nbe aware that there were submodules, they're automatically fetched and\nupdated based on commits that may only exist in fred's repository.\n\nSo currently, you can do\n$ git pull origin && git submodule init && git submodule update\n\nAnd it works, but\n\n$ git pull fred\n$ git submodule update\n\nCan leave you with problems, because if a submodule wasn't pushed to\norigin, you won't have it available. This is because the commands are\nequivalent to\n\n$ git pull fred\nfor each submodule()\n cd submodule\n git fetch origin\n git checkout \n\nSo somehow, you need to replace 'git fetch origin' with the \"correct\"\nrepository (on fred's computer). My patch was really just about being\nable to pass parameters to 'git fetch'. The problems are that if you\ndid\n\n$ git submodule update fred\n\nUnless each submodule had a [remote] specified for \"fred\", you'd be\nstuffed. But what you could do is either by passing the right URL, or\nlooking at the superproject [remote] for \"fred\" - i.e: If in the\nsuperproject you have\n\n[remote \"fred\"]\n url = ssh://git@fred.local/pub/scm/git/workspace/thing/.git\n[submodule \"module\"]\n url = ssh://git@repo/pub/scm/git/module.git\n\nThen the submodule \"module\" on fred, if it's a working-copy, can be calculated\n ssh://git@fred.local/pub/scm/git/workspace/thing/module/.git\n\nIf it isn't a WC then you'd have to have a [remote \"fred\"] in that\nsubmodule, but I'm thinking that'd be a rare case.\n\nI'd assumed (possibly wrongly?) that there was resistance to putting\nany of the submodule logic in things other than git-submodules.\n\nAs a starter for 10, how about\n- a '--submodules' option to git fetch / pull\n- using the remote name if known, calculate it if not based on the above\n\nWDYT?\n"},{"id":"83733","messageId":"20080717182253.GZ32184@machine.or.cz","threadId":"14505","inReplyTo":"320075ff0807170807l1537e34ev510deda537e4d11e@mail.gmail.com","subject":"Re: [PATCH] Teach git submodule update to use distributed repositories","fromName":"Petr Baudis","fromEmail":"pasky@suse.cz","sentAt":"2008-07-17T18:22:53Z","receivedAt":"2008-07-17T18:22:53Z","isPatch":true,"sender":{"key":"pasky@ucw.cz","avatar":"https://avatars.githubusercontent.com/u/18439?v=4"},"body":"On Thu, Jul 17, 2008 at 04:07:11PM +0100, Nigel Magnay wrote:\n> And it works, but\n> \n> $ git pull fred\n> $ git submodule update\n> \n> Can leave you with problems, because if a submodule wasn't pushed to\n> origin, you won't have it available. This is because the commands are\n> equivalent to\n> \n> $ git pull fred\n> for each submodule()\n> cd submodule\n> git fetch origin\n> git checkout \n\nOh! So, only after replying to most of your mail, I have realized what\nare you talking about all the time - _just_ this particular failure\nmode:\n\n\t\"Someone pushed out a repository repointing submodules to\n\tinvalid commits, and instead of waiting for the person to fix\n\tthis breakage, we want to do a one-off fetch of all submodules\n\tfrom a different repository.\"\n\nThere's nothing else you're trying to solve by this, right?\n\n\nNow, I think that this is a completely wrong problem to solve. Your\ngitweb is going to be broken, everyone has to jump through hoops because\nof this, and that all just because of a single mistake. It shouldn't\nhave _happenned_ in the first place.\n\nSo the proper solution for this should be to make an update hook that\nwill simply not _let_ you push out a tree that's broken like this.\nSomething like this (completely untested):\n\ndie() { echo \"$@\"; exit 1; }\ngit rev-list ^$2 $3 | while read commit; do\n\tgit show $commit:.gitmodules >/tmp/gm$$\n\tgit config -f /tmp/gm$$ --get-regexp 'submodule\\..*\\.path' |\n\t\tcut -d ' ' -f 1 |\n\t\tsed 's/^.*\\.//; s/\\..*$//;' |\n\t\twhile read submodule; do\n\t\t\tpath=$(git config -f /tmp/gm$$ \"submodule.$submodule.path\")\n\t\t\turl=$(git config -f /tmp/gm$$ \"submodule.$submodule.url\")\n\t\t\tentry=$(git ls-tree $commit \"$path\")\n\t\t\t[ -n \"$entry\" ] || die \"submodule $submodule points at a non-existing path\"\n\t\t\t[ \"$(echo \"$entry\" | cut -d ' ' -f 1)\" = \"160000\" ] || die \"submodule $submodule does not point to a gitlink entry\"\n\t\t\t\n\t\t\tsubcommit=\"$(echo \"$entry\" | cut -d ' ' -f 2)\"\n\t\t\turlhash=\"$(echo \"$url\" | sha1sum | cut -d ' ' -f 1)\"\n\t\t\t# We keep local copies of submodule repositories\n\t\t\t# for commit existence checking\n\t\t\techo \"Please wait, updating $url cache...\"\n\t\t\tif [ -d /tmp/ucache/$urlhash ]; then\n\t\t\t (cd /tmp/ucache/$urlhash && git fetch)\n\t\t\telse\n\t\t\t git clone --bare \"$url\" /tmp/ucache/$urlhash\n\t\t\tfi\n\t\t\t[ \"$(git --git-dir=/tmp/ucache/$urlhash cat-file -t \"$subcommit\" 2>/dev/null)\" = \"commit\" ] || die \"submodule $submodule does not point at an existing commit\"\n\t\tdone\n\tdone\n\nComments? If it seems good, it might be worth including in\ncontrib/hooks/. Maybe even in the default update hook, controlled by\na config option.\n\nAll the troubles here stem from the fact that normally, Git will not let\nyou push any invalid state to the server. This is not completely true in\nthis case, but we should prevent this behaviour instead of inventing\nhacks to work it around.\n\n> Unless each submodule had a [remote] specified for \"fred\", you'd be\n> stuffed. But what you could do is either by passing the right URL, or\n> looking at the superproject [remote] for \"fred\" - i.e: If in the\n> superproject you have\n> \n> [remote \"fred\"]\n> url = ssh://git@fred.local/pub/scm/git/workspace/thing/.git\n> [submodule \"module\"]\n> url = ssh://git@repo/pub/scm/git/module.git\n> \n> Then the submodule \"module\" on fred, if it's a working-copy, can be calculated\n> ssh://git@fred.local/pub/scm/git/workspace/thing/module/.git\n> \n> If it isn't a WC then you'd have to have a [remote \"fred\"] in that\n> submodule, but I'm thinking that'd be a rare case.\n\nThis is ultra-evil. I think assuming things like this is extremely dirty\nand not reasonable for a universal code, _unless_ we explicitly decide\nthat this is a new convention you want to introduce as a recommendation.\nBut you should've been very clear about this upfront.\n\n_If_ you still insist on the one-off fetches for some reason, I think\nit's reasonable to provide your own simple script for your users that\nwill autogenerate these URLs appropriately for your particular setup.\nI don't think there is any real need for a more generic solution.\n\n> I'd assumed (possibly wrongly?) that there was resistance to putting\n> any of the submodule logic in things other than git-submodules.\n\nAre you following the thread about submodule support for git mv, git rm?\n\n-- \n\t\t\t\tPetr \"Pasky\" Baudis\nGNU, n. An animal of South Africa, which in its domesticated state\nresembles a horse, a buffalo and a stag. In its wild condition it is\nsomething like a thunderbolt, an earthquake and a cyclone. -- A. Pierce\n"},{"id":"83842","messageId":"320075ff0807180111q4ca55cc4v15487af35f6fa63c@mail.gmail.com","threadId":"14505","inReplyTo":"20080717182253.GZ32184@machine.or.cz","subject":"Re: [PATCH] Teach git submodule update to use distributed repositories","fromName":"Nigel Magnay","fromEmail":"nigel.magnay@gmail.com","sentAt":"2008-07-18T08:11:53Z","receivedAt":"2008-07-18T08:11:53Z","isPatch":true,"sender":{"key":"nigel.magnay@gmail.com","avatar":"https://gravatar.com/avatar/d85cf38287bef3a8e4fa02358d2756d7589f8676c5eeb881ce2f6d731e4526c3?d=mp&s=160"},"body":"On Thu, Jul 17, 2008 at 7:22 PM, Petr Baudis wrote:\n> On Thu, Jul 17, 2008 at 04:07:11PM +0100, Nigel Magnay wrote:\n>> And it works, but\n>>\n>> $ git pull fred\n>> $ git submodule update\n>>\n>> Can leave you with problems, because if a submodule wasn't pushed to\n>> origin, you won't have it available. This is because the commands are\n>> equivalent to\n>>\n>> $ git pull fred\n>> for each submodule()\n>> cd submodule\n>> git fetch origin\n>> git checkout \n>\n> Oh! So, only after replying to most of your mail, I have realized what\n> are you talking about all the time - _just_ this particular failure\n> mode:\n>\n> \"Someone pushed out a repository repointing submodules to\n> invalid commits, and instead of waiting for the person to fix\n> this breakage, we want to do a one-off fetch of all submodules\n> from a different repository.\"\n>\n> There's nothing else you're trying to solve by this, right?\n>\n\nNo.\n\"Someone says 'please review the state of my tree, _before_ I push it\nout to a (central) repository\"\n\nFred is a person (and != origin). His tree(s) are entirely correct and\nconsistent, and he doesn't yet wish to push to origin (and perhaps he\ncannot, because he does not have permission to do so).\n\nAll the tutorials give credit to the fact that in git you don't need a\ncentral server - you can pull directly from people. Except in the case\nwhere you're using submodules, where you're basically forced to\nhand-modify .git/config (in this instance, to point to where 'fred' is\nstoring his submodule trees) before doing a submodule update. This\nmakes git complicated for users.\n\nI'm trying to improve the UI for projects using submodules to make it\nmostly transparent; the best way I can come up with is to pick on\nindividual usecases and show that they're a particular pain and that\nperhaps they don't need to be.\n\n>\n> Now, I think that this is a completely wrong problem to solve. Your\n> gitweb is going to be broken, everyone has to jump through hoops because\n> of this, and that all just because of a single mistake. It shouldn't\n> have _happenned_ in the first place.\n>\n> So the proper solution for this should be to make an update hook that\n> will simply not _let_ you push out a tree that's broken like this.\n> Something like this (completely untested):\n>\n> die() { echo \"$@\"; exit 1; }\n> git rev-list ^$2 $3 | while read commit; do\n> git show $commit:.gitmodules >/tmp/gm$$\n> git config -f /tmp/gm$$ --get-regexp 'submodule\\..*\\.path' |\n> cut -d ' ' -f 1 |\n> sed 's/^.*\\.//; s/\\..*$//;' |\n> while read submodule; do\n> path=$(git config -f /tmp/gm$$ \"submodule.$submodule.path\")\n> url=$(git config -f /tmp/gm$$ \"submodule.$submodule.url\")\n> entry=$(git ls-tree $commit \"$path\")\n> [ -n \"$entry\" ] || die \"submodule $submodule points at a non-existing path\"\n> [ \"$(echo \"$entry\" | cut -d ' ' -f 1)\" = \"160000\" ] || die \"submodule $submodule does not point to a gitlink entry\"\n>\n> subcommit=\"$(echo \"$entry\" | cut -d ' ' -f 2)\"\n> urlhash=\"$(echo \"$url\" | sha1sum | cut -d ' ' -f 1)\"\n> # We keep local copies of submodule repositories\n> # for commit existence checking\n> echo \"Please wait, updating $url cache...\"\n> if [ -d /tmp/ucache/$urlhash ]; then\n> (cd /tmp/ucache/$urlhash && git fetch)\n> else\n> git clone --bare \"$url\" /tmp/ucache/$urlhash\n> fi\n> [ \"$(git --git-dir=/tmp/ucache/$urlhash cat-file -t \"$subcommit\" 2>/dev/null)\" = \"commit\" ] || die \"submodule $submodule does not point at an existing commit\"\n> done\n> done\n>\n> Comments? If it seems good, it might be worth including in\n> contrib/hooks/. Maybe even in the default update hook, controlled by\n> a config option.\n>\n> All the troubles here stem from the fact that normally, Git will not let\n> you push any invalid state to the server. This is not completely true in\n> this case, but we should prevent this behaviour instead of inventing\n> hacks to work it around.\n>\n>> Unless each submodule had a [remote] specified for \"fred\", you'd be\n>> stuffed. But what you could do is either by passing the right URL, or\n>> looking at the superproject [remote] for \"fred\" - i.e: If in the\n>> superproject you have\n>>\n>> [remote \"fred\"]\n>> url = ssh://git@fred.local/pub/scm/git/workspace/thing/.git\n>> [submodule \"module\"]\n>> url = ssh://git@repo/pub/scm/git/module.git\n>>\n>> Then the submodule \"module\" on fred, if it's a working-copy, can be calculated\n>> ssh://git@fred.local/pub/scm/git/workspace/thing/module/.git\n>>\n>> If it isn't a WC then you'd have to have a [remote \"fred\"] in that\n>> submodule, but I'm thinking that'd be a rare case.\n>\n> This is ultra-evil. I think assuming things like this is extremely dirty\n> and not reasonable for a universal code, _unless_ we explicitly decide\n> that this is a new convention you want to introduce as a recommendation.\n> But you should've been very clear about this upfront.\n>\n> _If_ you still insist on the one-off fetches for some reason, I think\n> it's reasonable to provide your own simple script for your users that\n> will autogenerate these URLs appropriately for your particular setup.\n> I don't think there is any real need for a more generic solution.\n>\n>> I'd assumed (possibly wrongly?) that there was resistance to putting\n>> any of the submodule logic in things other than git-submodules.\n>\n> Are you following the thread about submodule support for git mv, git rm?\n>\n> --\n> Petr \"Pasky\" Baudis\n> GNU, n. An animal of South Africa, which in its domesticated state\n> resembles a horse, a buffalo and a stag. In its wild condition it is\n> something like a thunderbolt, an earthquake and a cyclone. -- A. Pierce\n>\n"},{"id":"83845","messageId":"m363r3y42v.fsf@localhost.localdomain","threadId":"14505","inReplyTo":"320075ff0807180111q4ca55cc4v15487af35f6fa63c@mail.gmail.com","subject":"Re: [PATCH] Teach git submodule update to use distributed repositories","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2008-07-18T08:45:23Z","receivedAt":"2008-07-18T08:45:23Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"\"Nigel Magnay\" writes:\n\n> On Thu, Jul 17, 2008 at 7:22 PM, Petr Baudis wrote:\n>> On Thu, Jul 17, 2008 at 04:07:11PM +0100, Nigel Magnay wrote:\n>>> And it works, but\n>>>\n>>> $ git pull fred\n>>> $ git submodule update\n>>>\n>>> Can leave you with problems, because if a submodule wasn't pushed to\n>>> origin, you won't have it available. This is because the commands are\n>>> equivalent to\n>>>\n>>> $ git pull fred\n>>> for each submodule()\n>>> cd submodule\n>>> git fetch origin\n>>> git checkout \n\n> \"Someone says 'please review the state of my tree, _before_ I push it\n> out to a (central) repository\"\n> \n> Fred is a person (and != origin). His tree(s) are entirely correct and\n> consistent, and he doesn't yet wish to push to origin (and perhaps he\n> cannot, because he does not have permission to do so).\n> \n> All the tutorials give credit to the fact that in git you don't need a\n> central server - you can pull directly from people. Except in the case\n> where you're using submodules, where you're basically forced to\n> hand-modify .git/config (in this instance, to point to where 'fred' is\n> storing his submodule trees) before doing a submodule update. This\n> makes git complicated for users.\n> \n> I'm trying to improve the UI for projects using submodules to make it\n> mostly transparent; the best way I can come up with is to pick on\n> individual usecases and show that they're a particular pain and that\n> perhaps they don't need to be.\n\nI _think_ that you can currently work around this problem by using\nURL rewriting (url..insteadOf).\n\n-- \nJakub Narebski\nPoland\nShadeHawk on #git\n"},{"id":"83847","messageId":"7vwsjj8t3s.fsf@gitster.siamese.dyndns.org","threadId":"14505","inReplyTo":"m363r3y42v.fsf@localhost.localdomain","subject":"Re: [PATCH] Teach git submodule update to use distributed repositories","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2008-07-18T09:00:23Z","receivedAt":"2008-07-18T09:00:23Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jakub Narebski writes:\n\n> \"Nigel Magnay\" writes:\n>\n>> Fred is a person (and != origin). His tree(s) are entirely correct and\n>> consistent, and he doesn't yet wish to push to origin (and perhaps he\n>> cannot, because he does not have permission to do so).\n>> ...\n> I _think_ that you can currently work around this problem by using\n> URL rewriting (url..insteadOf).\n\nDoesn't it also involve config modification?\n\nI think the right thing to do for this kind of \"trial merge\" should be the\nsame as cases that do not involve submodules. You *DO NOT* give a handy\nway to muck with your configuration to make \"origin\" point at fred.\nInstead, you would do something like:\n\n\t$ git fetch ../fred master\n $ git checkout FETCH_HEAD\n ... review test fix ...\n\t... when you are done, go back, discarding the state from Fred\n $ git checkout master\n\nWhat submodule changes from the above workflow would be what happens after\nyou switch to the trial state (the above example detaches HEAD temporarily\nwhile peeking into Fred's history). It is understandable that you would\nwant to script something that recurses into the submodules that you have\nchecked out (or submodules that Fred wants you to look at), do the\nequivalent of \"git fetch ../fred\" you did at the toplevel to automate that\nstep, but I very much agree with Pasky here in that it feels very wrong to\nhijack \"submodule update\" for it.\n"},{"id":"83848","messageId":"200807181107.18098.jnareb@gmail.com","threadId":"14505","inReplyTo":"7vwsjj8t3s.fsf@gitster.siamese.dyndns.org","subject":"Re: [PATCH] Teach git submodule update to use distributed repositories","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2008-07-18T09:07:17Z","receivedAt":"2008-07-18T09:07:17Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Junio C Hamano wrote:\n\n> [...] It is understandable that you would\n> want to script something that recurses into the submodules that you have\n> checked out (or submodules that Fred wants you to look at), do the\n> equivalent of \"git fetch ../fred\" you did at the toplevel to automate that\n> step, but I very much agree with Pasky here in that it feels very wrong to\n> hijack \"submodule update\" for it.\n\nThere were two proposals how to deal with fetching all submodules:\n(a) git-submodule recursing into submodules, IIRC even with some\nimplementation (b) new \"git submodule fetch\" command.\n\n-- \nJakub Narebski\nPoland\n"},{"id":"83852","messageId":"20080718091608.GL10151@machine.or.cz","threadId":"14505","inReplyTo":"320075ff0807180111q4ca55cc4v15487af35f6fa63c@mail.gmail.com","subject":"Re: [PATCH] Teach git submodule update to use distributed repositories","fromName":"Petr Baudis","fromEmail":"pasky@suse.cz","sentAt":"2008-07-18T09:16:08Z","receivedAt":"2008-07-18T09:16:08Z","isPatch":true,"sender":{"key":"pasky@ucw.cz","avatar":"https://avatars.githubusercontent.com/u/18439?v=4"},"body":" Hi,\n\n _please_, trim the parts of quoted e-mails that you are not reacting\nto. It makes your mails easier to read.\n\nOn Fri, Jul 18, 2008 at 09:11:53AM +0100, Nigel Magnay wrote:\n> No.\n> \"Someone says 'please review the state of my tree, _before_ I push it\n> out to a (central) repository\"\n> \n> Fred is a person (and != origin). His tree(s) are entirely correct and\n> consistent, and he doesn't yet wish to push to origin (and perhaps he\n> cannot, because he does not have permission to do so).\n> \n> All the tutorials give credit to the fact that in git you don't need a\n> central server - you can pull directly from people. Except in the case\n> where you're using submodules, where you're basically forced to\n> hand-modify .git/config (in this instance, to point to where 'fred' is\n> storing his submodule trees) before doing a submodule update. This\n> makes git complicated for users.\n\nOk! Handling this case makes sense, though I would have wished you to\nword this use case this clearly from the beginning; or maybe I'm just\nslow. :-)\n\nNow, we (at least we two) agree that this use case is worth supporting,\nI still don't like the solution you propose, though. The problem that we\nare trying to solve is:\n\n\t\"How do we mass-supply custom submodule URLs when publishing the\n\tcustomized main repository at a custom location too?\"\n\nNow, the most natural solution is for Fred to actually customize\n.gitmodules content when committing the submodule updates:\n\n (i) Either just give submodule update a hypothetical flag that will\nignore .git/config for that particular run or,\n\n (ii) even much better, actually change logical submodule names in\n.gitmodules; this is appropriate as you want the modules to actually\npoint at a significantly different repository. Thus,\n\n\t[submodule \"boo\"]\n\tpath=boo\n\turl=git://repo.or.cz/boo.git\n\nwill become\n\n\t[submodule \"boo/fred\"]\n\tpath=boo\n\turl=git://repo.or.cz/boo/fred.git\n\n Also, you will be able to redefine the URL of boo/fred too in\n.git/config (e.g. you're behind a firewall that lets only HTTP\nthrough; I'm actually behind such a firewall these days at my\n(non-SUSE ;) work).\n\n\nThis should be reasonably elegant, works with no Git changes, however\nstill has one significant problem - you very much do not want such a\n.gitmodules change in any of the commits you merge, since it breaks\nbisectability in case Fred or his repositories go away.\n\nIn that case, several possibilities come up on my mind:\n\n (1) Fred will prepare special branch for testing with modified\n.gitmodules and then for a merge he offers a different branch with clean\n.gitmodules. This works, but it is obnoxious.\n\n (2) Fred will pass a patch for .gitmodules as a part of his review\nrequest. This works too and is obnoxious in slightly different aspects\nthan (1).\n\n (3) Fred will offer a rewrite rule that you will pass to submodule\nupdate, like your solution proposed, but much more universal so that it\nis not tailored just to your particular repository hierarchy. A simple\nsed script could work fine.\n\n (4) Something else that I'm not realizing.\n\n-- \n\t\t\t\tPetr \"Pasky\" Baudis\nGNU, n. An animal of South Africa, which in its domesticated state\nresembles a horse, a buffalo and a stag. In its wild condition it is\nsomething like a thunderbolt, an earthquake and a cyclone. -- A. Pierce\n"},{"id":"83853","messageId":"320075ff0807180218k7cfd4b07l67a1c82af0d61653@mail.gmail.com","threadId":"14505","inReplyTo":"200807181107.18098.jnareb@gmail.com","subject":"Re: [PATCH] Teach git submodule update to use distributed repositories","fromName":"Nigel Magnay","fromEmail":"nigel.magnay@gmail.com","sentAt":"2008-07-18T09:18:27Z","receivedAt":"2008-07-18T09:18:27Z","isPatch":true,"sender":{"key":"nigel.magnay@gmail.com","avatar":"https://gravatar.com/avatar/d85cf38287bef3a8e4fa02358d2756d7589f8676c5eeb881ce2f6d731e4526c3?d=mp&s=160"},"body":"On Fri, Jul 18, 2008 at 10:07 AM, Jakub Narebski wrote:\n> Junio C Hamano wrote:\n>\n>> [...] It is understandable that you would\n>> want to script something that recurses into the submodules that you have\n>> checked out (or submodules that Fred wants you to look at), do the\n>> equivalent of \"git fetch ../fred\" you did at the toplevel to automate that\n>> step, but I very much agree with Pasky here in that it feels very wrong to\n>> hijack \"submodule update\" for it.\n>\n> There were two proposals how to deal with fetching all submodules:\n> (a) git-submodule recursing into submodules, IIRC even with some\n> implementation (b) new \"git submodule fetch\" command.\n>\n\nYes - I think there's a few more options and possible combinations\n\na. git submodule update having to recurse into\nsubmodules (a)(original patch)\nb. git submodule fetch\nc. git fetch --submodules\nd. git fetch (automatically recurse if there are submodules)\ne. git fetch (automatically recurse if there is some setting in .git/config)\n\nI started at (a) and agree that it's a bad choice.\nAny of b-e would work for me.\nMy (personal) preferences would be for d/e, then c, then b - but -\nthat's based on my belief that submodules are a pretty fundamental\nthing and having a separate UI is bad.\n"},{"id":"83856","messageId":"320075ff0807180236u4e7f5f9bm81b702d14c052de8@mail.gmail.com","threadId":"14505","inReplyTo":"20080718091608.GL10151@machine.or.cz","subject":"Re: [PATCH] Teach git submodule update to use distributed repositories","fromName":"Nigel Magnay","fromEmail":"nigel.magnay@gmail.com","sentAt":"2008-07-18T09:36:51Z","receivedAt":"2008-07-18T09:36:51Z","isPatch":true,"sender":{"key":"nigel.magnay@gmail.com","avatar":"https://gravatar.com/avatar/d85cf38287bef3a8e4fa02358d2756d7589f8676c5eeb881ce2f6d731e4526c3?d=mp&s=160"},"body":"On Fri, Jul 18, 2008 at 10:16 AM, Petr Baudis wrote:\n> snip\n>\n> \"How do we mass-supply custom submodule URLs when publishing the\n> customized main repository at a custom location too?\"\n>\nYes - that is an additional problem.\n\nIf I may expand the usecase just so it's clear (and to check we're\ntalkiing the same language)\n\nI do something like\n$ git remote add fred git://fredcomputer/superproject/.git\n$ git fetch --submodules fred\n\nAnd when the recursive fetching enters a submodule, it is trying\nitself to do something like\n$ git fetch fred\n\nAt which point\n1) the submodule also has a remote specified for fred. In which case\nit can continue\n2) the submodule doesn't have remote specified for fred. How to solve\nthis case? (I.E how does 'my' git 'discover' where fred's git\nrepositories are for the submodules?)\n a) By getting some information from fred, either in *Fred's*\nsuperproject .git/config (or some other readable file)\n b) By reading some information out of the superproject .gitmodules\nthat has been fetched from fred\n c) By calculating a relative URL based on the supposition that fred\nhas working copies laid out in the filesystem.\n\nI was tentatively suggesing (c), with a backup of (a) for the minority\ncases where you weren't pulling from a person but from a mirror or\nsomething. Having the client edit config files just feels like a hack\nto me, regardless of whether scripts are enabled to do it.\n"},{"id":"83861","messageId":"20080718100048.GN10151@machine.or.cz","threadId":"14505","inReplyTo":"320075ff0807180236u4e7f5f9bm81b702d14c052de8@mail.gmail.com","subject":"Re: [PATCH] Teach git submodule update to use distributed repositories","fromName":"Petr Baudis","fromEmail":"pasky@suse.cz","sentAt":"2008-07-18T10:00:48Z","receivedAt":"2008-07-18T10:00:48Z","isPatch":true,"sender":{"key":"pasky@ucw.cz","avatar":"https://avatars.githubusercontent.com/u/18439?v=4"},"body":"On Fri, Jul 18, 2008 at 10:36:51AM +0100, Nigel Magnay wrote:\n> On Fri, Jul 18, 2008 at 10:16 AM, Petr Baudis wrote:\n> > snip\n> >\n> > \"How do we mass-supply custom submodule URLs when publishing the\n> > customized main repository at a custom location too?\"\n> >\n> Yes - that is an additional problem.\n\nWait, I'm lost again - _additional_ problem? How does it differ from the\n_original_ problem, how does it differ from what you're explaining below\nand how does what you're explaining below differ from the original\nproblem?\n\nOr are we talking exclusively about what I summed up above now?\n\n> If I may expand the usecase just so it's clear (and to check we're\n> talkiing the same language)\n> \n> I do something like\n> $ git remote add fred git://fredcomputer/superproject/.git\n> $ git fetch --submodules fred\n\nI think you mean git pull --submodules fred. Well, actually, you want to\npull the main repository, then submodule update (_not_ pull in the\nsubmodules). See? This is part of the \"semantic swamp\" I mentioned\nbefore.\n\nI think it should be somehow part of the _main_ project's fred branch\nthat in this branch, the subprojects should be fetched from a different\nlocation; thus, you would still do\n\n\t$ git remote add fred git://fredcomputer/superproject/.git\n\t$ git pull fred\n\t$ git submodule update\n\nwhere either the submodule update takes the info from fred's adjusted\n.gitmodules, or it is an implicit part of the branch as in fred tells\nyou to run the update command with some extra arguments.\n\nHowever, I still believe the information should primarily stem from the\nmain repository; consider e.g. if you do not have some of the submodules\nchecked out when you switch to fred, then figure out that in fred's\nbranch, you really do want them checked out.\n\n-- \n\t\t\t\tPetr \"Pasky\" Baudis\nGNU, n. An animal of South Africa, which in its domesticated state\nresembles a horse, a buffalo and a stag. In its wild condition it is\nsomething like a thunderbolt, an earthquake and a cyclone. -- A. Pierce\n"},{"id":"83866","messageId":"320075ff0807180420k4b28c317mc026713b22c44839@mail.gmail.com","threadId":"14505","inReplyTo":"20080718100048.GN10151@machine.or.cz","subject":"Re: [PATCH] Teach git submodule update to use distributed repositories","fromName":"Nigel Magnay","fromEmail":"nigel.magnay@gmail.com","sentAt":"2008-07-18T11:20:13Z","receivedAt":"2008-07-18T11:20:13Z","isPatch":true,"sender":{"key":"nigel.magnay@gmail.com","avatar":"https://gravatar.com/avatar/d85cf38287bef3a8e4fa02358d2756d7589f8676c5eeb881ce2f6d731e4526c3?d=mp&s=160"},"body":"On Fri, Jul 18, 2008 at 11:00 AM, Petr Baudis wrote:\n> On Fri, Jul 18, 2008 at 10:36:51AM +0100, Nigel Magnay wrote:\n>> On Fri, Jul 18, 2008 at 10:16 AM, Petr Baudis wrote:\n>> > snip\n>> >\n>> > \"How do we mass-supply custom submodule URLs when publishing the\n>> > customized main repository at a custom location too?\"\n>> >\n>> Yes - that is an additional problem.\n>\n> Wait, I'm lost again - _additional_ problem? How does it differ from the\n> _original_ problem, how does it differ from what you're explaining below\n> and how does what you're explaining below differ from the original\n> problem?\n>\nIn addition to the problem of needing to execute multiple commands and\nedit files to acheive what is a rather simple usecase, there is the\nadditional problem of discovering (for a third party) a url for where\ntheir submodules are stored.\n\n> Or are we talking exclusively about what I summed up above now?\n>\nIn this part of the thread. The first part seems to have broad\nagreement that a command could be added / modified, but not yet what\nit should look like.\n\n>> If I may expand the usecase just so it's clear (and to check we're\n>> talkiing the same language)\n>>\n>> I do something like\n>> $ git remote add fred git://fredcomputer/superproject/.git\n>> $ git fetch --submodules fred\n>\n> I think you mean git pull --submodules fred. Well, actually, you want to\n> pull the main repository, then submodule update (_not_ pull in the\n> submodules). See? This is part of the \"semantic swamp\" I mentioned\n> before.\n\nAh - I understand. You're saying \"you can't pull submodules when you\npull the supermodule, because you don't know which submodules might be\nneeded until you also merge / checkout the desired revision\" ?\n\nAck.\n\n>\n> I think it should be somehow part of the _main_ project's fred branch\n> that in this branch, the subprojects should be fetched from a different\n> location; thus, you would still do\n>\n> $ git remote add fred git://fredcomputer/superproject/.git\n> $ git pull fred\n> $ git submodule update\n>\n\nYes, that makes sense.\n\n> where either the submodule update takes the info from fred's adjusted\n> .gitmodules, or it is an implicit part of the branch as in fred tells\n> you to run the update command with some extra arguments.\n>\n> However, I still believe the information should primarily stem from the\n> main repository; consider e.g. if you do not have some of the submodules\n> checked out when you switch to fred, then figure out that in fred's\n> branch, you really do want them checked out.\n>\n\nYes.\nReferring to your earlier mail, I'm now preferring \"(4) Something else\nthat I'm not realizing.\" ;-)\n\nHm. It feels like each person could have some 'local' info in their\n.gitmodules, and rules around merging; but I'm not sure of exactly\nwhat, or how.\n"},{"id":"83882","messageId":"20080718144325.GR10151@machine.or.cz","threadId":"14505","inReplyTo":"320075ff0807180420k4b28c317mc026713b22c44839@mail.gmail.com","subject":"Re: [PATCH] Teach git submodule update to use distributed repositories","fromName":"Petr Baudis","fromEmail":"pasky@suse.cz","sentAt":"2008-07-18T14:43:25Z","receivedAt":"2008-07-18T14:43:25Z","isPatch":true,"sender":{"key":"pasky@ucw.cz","avatar":"https://avatars.githubusercontent.com/u/18439?v=4"},"body":"On Fri, Jul 18, 2008 at 12:20:13PM +0100, Nigel Magnay wrote:\n> On Fri, Jul 18, 2008 at 11:00 AM, Petr Baudis wrote:\n> > On Fri, Jul 18, 2008 at 10:36:51AM +0100, Nigel Magnay wrote:\n> >> On Fri, Jul 18, 2008 at 10:16 AM, Petr Baudis wrote:\n> >> > snip\n> >> >\n> >> > \"How do we mass-supply custom submodule URLs when publishing the\n> >> > customized main repository at a custom location too?\"\n> >> >\n> >> Yes - that is an additional problem.\n> >\n> > Wait, I'm lost again - _additional_ problem? How does it differ from the\n> > _original_ problem, how does it differ from what you're explaining below\n> > and how does what you're explaining below differ from the original\n> > problem?\n> >\n> In addition to the problem of needing to execute multiple commands and\n> edit files to acheive what is a rather simple usecase, there is the\n> additional problem of discovering (for a third party) a url for where\n> their submodules are stored.\n\nI see. That's interconnected as a single \"How to check Fred's work\"\nproblem for me. :-)\n\n> >> If I may expand the usecase just so it's clear (and to check we're\n> >> talkiing the same language)\n> >>\n> >> I do something like\n> >> $ git remote add fred git://fredcomputer/superproject/.git\n> >> $ git fetch --submodules fred\n> >\n> > I think you mean git pull --submodules fred. Well, actually, you want to\n> > pull the main repository, then submodule update (_not_ pull in the\n> > submodules). See? This is part of the \"semantic swamp\" I mentioned\n> > before.\n> \n> Ah - I understand. You're saying \"you can't pull submodules when you\n> pull the supermodule, because you don't know which submodules might be\n> needed until you also merge / checkout the desired revision\" ?\n> \n> Ack.\n\nThat is something I might agree with, but my point is that within the\nsubmodule,\n\n\tgit pull\n\nsimply isn't a sensible operation at all! You don't want to do any\nmerges or whatever, just bring the submodule to a defined commit id.\nSo you want to do something significantly different:\n\n\tgit fetch\n\tgit reset --hard \n\nAnd that's what 'git submodule update' already does.\n\n> Hm. It feels like each person could have some 'local' info in their\n> .gitmodules, and rules around merging; but I'm not sure of exactly\n> what, or how.\n\nAgain, when customizing .gitmodules, you need to either give up on\n\n\t(i) bisectability; it's not good enough to restore the canonical\n\t.gitmodules contents on merge, since the bisect can run into one\n\tof the commits on fred' branchs\n\n\t(ii) publishing the exact same branch for testing and merging\n\nBut I start to feel that the tradeoff of (ii) is really not so bad at\nalland this would be perhaps the most elegant solution. You can either\n\n\t(a) make two parallel branches, one with your .gitmodules and\n\tone with the upstream's\n\n\t(b) probably better, stick a commit at the top of your branch\n\tthat will change .gitmodules to your locations; others can\n\tcheck out fred, test things out, then merge fred^; you can even\n\tgenerally go back in fred's commits if you just 'git submodule\n\tupdate' right after checking fred out, since all the required\n\tsubmodule commits will be probably already fetched.\n\nSo I say just go for the (ii)-(b) combination. :-)\n\n-- \n\t\t\t\tPetr \"Pasky\" Baudis\nAs in certain cults it is possible to kill a process if you know\nits true name. -- Ken Thompson and Dennis M. Ritchie\n"},{"id":"83883","messageId":"320075ff0807180809x599aefafw2c7fe88fea2691d2@mail.gmail.com","threadId":"14505","inReplyTo":"20080718144325.GR10151@machine.or.cz","subject":"Re: [PATCH] Teach git submodule update to use distributed repositories","fromName":"Nigel Magnay","fromEmail":"nigel.magnay@gmail.com","sentAt":"2008-07-18T15:09:40Z","receivedAt":"2008-07-18T15:09:40Z","isPatch":true,"sender":{"key":"nigel.magnay@gmail.com","avatar":"https://gravatar.com/avatar/d85cf38287bef3a8e4fa02358d2756d7589f8676c5eeb881ce2f6d731e4526c3?d=mp&s=160"},"body":">> Ah - I understand. You're saying \"you can't pull submodules when you\n>> pull the supermodule, because you don't know which submodules might be\n>> needed until you also merge / checkout the desired revision\" ?\n>>\n>> Ack.\n>\n> That is something I might agree with, but my point is that within the\n> submodule,\n>\n> git pull\n>\n> simply isn't a sensible operation at all! You don't want to do any\n> merges or whatever, just bring the submodule to a defined commit id.\n> So you want to do something significantly different:\n>\n> git fetch\n> git reset --hard \n>\n> And that's what 'git submodule update' already does.\n>\n\nI wasn't wanting to do pull there - but either way, I agree :-)\n\n>> Hm. It feels like each person could have some 'local' info in their\n>> .gitmodules, and rules around merging; but I'm not sure of exactly\n>> what, or how.\n>\n> Again, when customizing .gitmodules, you need to either give up on\n>\n> (i) bisectability; it's not good enough to restore the canonical\n> .gitmodules contents on merge, since the bisect can run into one\n> of the commits on fred' branchs\n>\n> (ii) publishing the exact same branch for testing and merging\n>\n> But I start to feel that the tradeoff of (ii) is really not so bad at\n> alland this would be perhaps the most elegant solution. You can either\n>\n> (a) make two parallel branches, one with your .gitmodules and\n> one with the upstream's\n>\n> (b) probably better, stick a commit at the top of your branch\n> that will change .gitmodules to your locations; others can\n> check out fred, test things out, then merge fred^; you can even\n> generally go back in fred's commits if you just 'git submodule\n> update' right after checking fred out, since all the required\n> submodule commits will be probably already fetched.\n>\n> So I say just go for the (ii)-(b) combination. :-)\n>\n\nHmm. Locally modifying my .gitmodules still feels bad because I don't\nlike either of those tradeoffs (but I don't have any sensible\nsuggestion yet).\n\nAs a bit of background (as to why I'd dislike (a) and (b)), we had a\nteam switch to git, and one of the really nice things is the ability\nto share stuff around and branch freely - but the flipside of that is\nthat we tend to push to a central repo more rarely, so the advantages\nof an continuous integration server become less. What we did is to\ntell a centralised CI server the URLs of all the team's git\nrepositories, and it would periodically pull from them, speculatively\ncompile anything new, and run the big suite of tests - finishing up by\nemailling them a heads-up that a particular state in their repo is\n'bad'.\n\nThis was really popular as it was demonstrably better than anything we\ncould do with svn, and best of all, it's pretty much transparent - as\na user you don't have to do anything at all.\n\nI could do it now by hacking about with files; it'd just be nice to\nkeep it transparent and make it a directly supported feature.\n"},{"id":"83885","messageId":"20080718154959.GS10151@machine.or.cz","threadId":"14505","inReplyTo":"320075ff0807180809x599aefafw2c7fe88fea2691d2@mail.gmail.com","subject":"Re: [PATCH] Teach git submodule update to use distributed repositories","fromName":"Petr Baudis","fromEmail":"pasky@suse.cz","sentAt":"2008-07-18T15:49:59Z","receivedAt":"2008-07-18T15:49:59Z","isPatch":true,"sender":{"key":"pasky@ucw.cz","avatar":"https://avatars.githubusercontent.com/u/18439?v=4"},"body":"On Fri, Jul 18, 2008 at 04:09:40PM +0100, Nigel Magnay wrote:\n> Hmm. Locally modifying my .gitmodules still feels bad because I don't\n> like either of those tradeoffs (but I don't have any sensible\n> suggestion yet).\n> \n> As a bit of background (as to why I'd dislike (a) and (b)), we had a\n> team switch to git, and one of the really nice things is the ability\n> to share stuff around and branch freely - but the flipside of that is\n> that we tend to push to a central repo more rarely, so the advantages\n> of an continuous integration server become less. What we did is to\n> tell a centralised CI server the URLs of all the team's git\n> repositories, and it would periodically pull from them, speculatively\n> compile anything new, and run the big suite of tests - finishing up by\n> emailling them a heads-up that a particular state in their repo is\n> 'bad'.\n> \n> This was really popular as it was demonstrably better than anything we\n> could do with svn, and best of all, it's pretty much transparent - as\n> a user you don't have to do anything at all.\n> \n> I could do it now by hacking about with files; it'd just be nice to\n> keep it transparent and make it a directly supported feature.\n\nIn that case you would need the \"URL mappings\", perhaps as a per-remote\nattribute. That is, you could configure:\n\n\t\"When I am doing git pull fred, do git submodule update but\n\tapply remote.fred.subrewrite sed script on each URL before\n\tfetching the submodule.\"\n\nStill, that feels quite hackish to me, and I'm not convinced that your\nworkflow cannot be adjusted so that users merge only the next-to-last\ncommit of a branch instead of the last one.\n\n-- \n\t\t\t\tPetr \"Pasky\" Baudis\nAs in certain cults it is possible to kill a process if you know\nits true name. -- Ken Thompson and Dennis M. Ritchie\n"},{"id":"83920","messageId":"48811B63.4090005@gmail.com","threadId":"14505","inReplyTo":"20080718154959.GS10151@machine.or.cz","subject":"Re: [PATCH] Teach git submodule update to use distributed repositories","fromName":"Mark Levedahl","fromEmail":"mlevedahl@gmail.com","sentAt":"2008-07-18T22:38:27Z","receivedAt":"2008-07-18T22:38:27Z","isPatch":true,"sender":{"key":"mdl123@verizon.net","avatar":"https://avatars.githubusercontent.com/u/5302462?v=4"},"body":"Petr Baudis wrote:\n> On Fri, Jul 18, 2008 at 04:09:40PM +0100, Nigel Magnay wrote:\n> \n> In that case you would need the \"URL mappings\", perhaps as a per-remote\n> attribute. That is, you could configure:\n> \n> \t\"When I am doing git pull fred, do git submodule update but\n> \tapply remote.fred.subrewrite sed script on each URL before\n> \tfetching the submodule.\"\n> \n> Still, that feels quite hackish to me, and I'm not convinced that your\n> workflow cannot be adjusted so that users merge only the next-to-last\n> commit of a branch instead of the last one.\n> \n\nThere really are two distinct forms of submodule URL's supported by \ngit-submodule: absolute and relative. The first says \"always go to repository x \non server y\", and is the correct form for a *very* loosely coupled submodule. \nHowever, it requires a lot of hacking to support fetching from an alternate \nlocation.\n\nThe relative form says \"go to this location *relative* to the superproject's \nrepository\". Using this form greatly eases the use case. Basically, fred has his \ntree of trees on his system, arranged exactly as they are on the main server. If \nyou do a git fetch \"fred\" into superproject, then submodule update, submodule \nshould be able to find the related submdodules on \"fred\" and get the data \nrelatively easily.\n\nI actually submitted a patch series a while back that does this, but no-one on \nthe list cared for the use case it supported so that series died.\n\nMark\n"},{"id":"84177","messageId":"320075ff0807210359r31d81d63i4d0584c4a1aab4c1@mail.gmail.com","threadId":"14505","inReplyTo":"20080718154959.GS10151@machine.or.cz","subject":"Re: [PATCH] Teach git submodule update to use distributed repositories","fromName":"Nigel Magnay","fromEmail":"nigel.magnay@gmail.com","sentAt":"2008-07-21T10:59:42Z","receivedAt":"2008-07-21T10:59:42Z","isPatch":true,"sender":{"key":"nigel.magnay@gmail.com","avatar":"https://gravatar.com/avatar/d85cf38287bef3a8e4fa02358d2756d7589f8676c5eeb881ce2f6d731e4526c3?d=mp&s=160"},"body":"> In that case you would need the \"URL mappings\", perhaps as a per-remote\n> attribute. That is, you could configure:\n>\n> \"When I am doing git pull fred, do git submodule update but\n> apply remote.fred.subrewrite sed script on each URL before\n> fetching the submodule.\"\n>\n> Still, that feels quite hackish to me, and I'm not convinced that your\n> workflow cannot be adjusted so that users merge only the next-to-last\n> commit of a branch instead of the last one.\n>\n\nHm - I'm still disliking having 'special' commits to change\n.gitmodules. I can hack scripts to make it work, but it would be nice\nto have a UI that is generally useful.\n\nThinking out loud, could we have in .git/config something like\n\n[submodule \"moduleA\"]\n url = git://origin.com/path/to/.git # Current place of origin\n fred.url = git://fredcomputer/path/to/freds/moduleA.git # where\nfred declares moduleA to come from\n local = git://myhost/working/copy/super/moduleA/.git # where other\npeople can get access to *my* moduleA repo\n\nSo if I look in the git repository of fred (as specified in my [remote\n\"fred\"], I can see their \"local\" entry, and enter that as fred.url in\nmy config\n\nAnd the ability to do (e.g)\n\n$ git submodule init fred\n$ git submodule update fred\n\n?\n"}]}