{"thread":{"id":"14480","subject":"git submodules and commit","startedAt":"2008-07-16T10:32:48Z","lastAt":"2008-07-18T16:11:54Z","messageCount":14,"participants":["Nigel Magnay","Johannes Sixt","Petr Baudis","Avery Pennarun","Ping Yin"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"83494","messageId":"320075ff0807160332k5e49c256tb4191de628ecf41c@mail.gmail.com","threadId":"14480","inReplyTo":"320075ff0807160331j30e8f832m4de3e3bbe9c26801@mail.gmail.com","subject":"git submodules and commit","fromName":"Nigel Magnay","fromEmail":"nigel.magnay@gmail.com","sentAt":"2008-07-16T10:32:48Z","receivedAt":"2008-07-16T10:32:48Z","isPatch":false,"sender":{"key":"nigel.magnay@gmail.com","avatar":"https://gravatar.com/avatar/d85cf38287bef3a8e4fa02358d2756d7589f8676c5eeb881ce2f6d731e4526c3?d=mp&s=160"},"body":"I wonder if this is a fairly common pattern. We tend to have modules\nas git repositories, and projects that tie together those git\nrepositories as submodules. In general, > 90% of the work is done in\none module, and the following stanza gets used a lot:\n\ncd /proj/modA\ngit commit -s -m \"Some change\"\ngit push\n\ncd ..\ngit add modA\ngit commit -s -m \"Some change (modA)\"\ngit push\n\nBut since this is much more cumbersome than (say) \"svn ci\", what often\nhappens is developers just commit into modA, then carry on. Or for\npeople just learning git, they somtimes screw up, and push the parent\nproj but not the child modA\n\nThis is a shame, as it means any external people pulling updates\ndirectly from proj will not get this change (e.g. CI tools\nspeculatively compiling against every developer tree).\n\nFor me, in some really high proportion of cases, I think I want 'git\ncommit' to mean 'commit to any child repositories, any sibling\nrepositories, and any parent repositories (updating the submodule sha1\nas appropriate). In other words, 'pretend like the whole thing is one\nbig repo'.\n\nI guess it probably gets sticky when there are merge conflicts. Is\nanyone working on this kind of thing; I might be able to give some\ntime to help work on it?\n"},{"id":"83501","messageId":"487DD1C7.3070701@viscovery.net","threadId":"14480","inReplyTo":"320075ff0807160332k5e49c256tb4191de628ecf41c@mail.gmail.com","subject":"Re: git submodules and commit","fromName":"Johannes Sixt","fromEmail":"j.sixt@viscovery.net","sentAt":"2008-07-16T10:47:35Z","receivedAt":"2008-07-16T10:47:35Z","isPatch":false,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Nigel Magnay schrieb:\n> For me, in some really high proportion of cases, I think I want 'git\n> commit' to mean 'commit to any child repositories, any sibling\n> repositories, and any parent repositories (updating the submodule sha1\n> as appropriate). In other words, 'pretend like the whole thing is one\n> big repo'.\n\nAnd I think that this is the problem: If this way of commiting your\nchanges is *required* in the *majority* of cases, then you are IMO outside\nthe intended use-case of submodules. You are better served by really\nmaking this one big repo.\n\nIMO, submodules are to be used if you can afford to advance parent project\nand submodules at different paces; i.e. if the parent project can work\nwith newer versions of the submodules (and possibly in a degraded mode\neven with outdated versions).\n\n-- Hannes\n"},{"id":"83504","messageId":"320075ff0807160402s7429291ela288b42d99c1ec53@mail.gmail.com","threadId":"14480","inReplyTo":"487DD1C7.3070701@viscovery.net","subject":"Re: git submodules and commit","fromName":"Nigel Magnay","fromEmail":"nigel.magnay@gmail.com","sentAt":"2008-07-16T11:02:19Z","receivedAt":"2008-07-16T11:02:19Z","isPatch":false,"sender":{"key":"nigel.magnay@gmail.com","avatar":"https://gravatar.com/avatar/d85cf38287bef3a8e4fa02358d2756d7589f8676c5eeb881ce2f6d731e4526c3?d=mp&s=160"},"body":"On Wed, Jul 16, 2008 at 11:47 AM, Johannes Sixt <j.sixt@viscovery.net> wrote:\n> Nigel Magnay schrieb:\n>> For me, in some really high proportion of cases, I think I want 'git\n>> commit' to mean 'commit to any child repositories, any sibling\n>> repositories, and any parent repositories (updating the submodule sha1\n>> as appropriate). In other words, 'pretend like the whole thing is one\n>> big repo'.\n>\n> And I think that this is the problem: If this way of commiting your\n> changes is *required* in the *majority* of cases, then you are IMO outside\n> the intended use-case of submodules. You are better served by really\n> making this one big repo.\n>\n\nHm - then my contention is that the scope of submodules needs to be\nexpanded (or something needs to be built on top).\n\nOne-big-repo doesn't fly - > 75% of the code volume (the 'other'\nmodules) are shared between multiple projects. In SVN these are just\nsvn:externals (which has it's own imperfections).\n\nI think it's a common usecase. You have 'shared' modules and\n'project-specific' modules[*]. The 'shared' modules you hope don't\nchange very much, but they are part of the overall project\nconfiguration - it's really nice that you can branch so easily in git,\nthen get the module owner to merge those changes into the next release\nat their leisure. The superproject then represents the correct\nconfiguration of submodule trees to make a valid build.\n\nThe machinery has everything that's required, it's just the user\nexperience sucks :(\n\n[*] actually there's more subtlety, there's 'shared', 'product' and\n'project', so some 'specific' modules are potentially re-shared\nelsewhere.\n> IMO, submodules are to be used if you can afford to advance parent project\n> and submodules at different paces; i.e. if the parent project can work\n> with newer versions of the submodules (and possibly in a degraded mode\n> even with outdated versions).\n>\n> -- Hannes\n>\n"},{"id":"83505","messageId":"487DDCFC.9020007@viscovery.net","threadId":"14480","inReplyTo":"320075ff0807160402s7429291ela288b42d99c1ec53@mail.gmail.com","subject":"Re: git submodules and commit","fromName":"Johannes Sixt","fromEmail":"j.sixt@viscovery.net","sentAt":"2008-07-16T11:35:24Z","receivedAt":"2008-07-16T11:35:24Z","isPatch":false,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Nigel Magnay schrieb:\n> On Wed, Jul 16, 2008 at 11:47 AM, Johannes Sixt <j.sixt@viscovery.net> wrote:\n>> Nigel Magnay schrieb:\n>>> For me, in some really high proportion of cases, I think I want 'git\n>>> commit' to mean 'commit to any child repositories, any sibling\n>>> repositories, and any parent repositories (updating the submodule sha1\n>>> as appropriate). In other words, 'pretend like the whole thing is one\n>>> big repo'.\n>> And I think that this is the problem: If this way of commiting your\n>> changes is *required* in the *majority* of cases, then you are IMO outside\n>> the intended use-case of submodules. You are better served by really\n>> making this one big repo.\n>>\n> \n> Hm - then my contention is that the scope of submodules needs to be\n> expanded (or something needs to be built on top).\n> \n> One-big-repo doesn't fly - > 75% of the code volume (the 'other'\n> modules) are shared between multiple projects. In SVN these are just\n> svn:externals (which has it's own imperfections).\n> \n> I think it's a common usecase. You have 'shared' modules and\n> 'project-specific' modules[*]. The 'shared' modules you hope don't\n> change very much, but they are part of the overall project\n> configuration - it's really nice that you can branch so easily in git,\n> then get the module owner to merge those changes into the next release\n> at their leisure. The superproject then represents the correct\n> configuration of submodule trees to make a valid build.\n\nAh, is this your actual scenario? Just to make sure we are talking about\nthe same thing:\n\n- You own superproject P.\n- $Maintainer owns submodule S.\n- You use S in P.\n- You make changes to S that you would like $Maintainer to include in the\nnext release.\nx You use in P your changes to S while $Maintainer has not yet released a\nnew version of S with your changes.\n- Finally your changes arrive via the new release of S.\n\nThat *is* the intended use-case for submodules. But you have to play the\ngame by the rules:\n\n- $Maintainer defines the official states of S.\n\n- You must never commit an unofficial state of S in P.\n\nThe critical step in above list I marked with x:\n\n- During the period where only *you* have the new changes to S, you must\n*not* commit your submodule state to P. Instead, you write P in such a way\nthat it can work with both the old version of S and the upcoming release\nthat will have your changes[*]. This way you make sure that your consumers\nof P always have a working version regardless of which version of S they use.\n\n- After you have received the new release of S from $Maintainer, you\ncommit the new state of S in P. And if you are nice to your consumers of\nP, then you *do not* remove the workaround from P just yet, so that you\ndon't force them to upgrade S. You will remove it later only if it becomes\na maintainance burden.\n\n[*] If it is not possible to make P work with old and new versions, then\nyou have to work closely with the $Maintainer so that you never need\ncommit an unofficial state of S into P.\n\n-- Hannes\n"},{"id":"83509","messageId":"20080716121124.GM32184@machine.or.cz","threadId":"14480","inReplyTo":"487DDCFC.9020007@viscovery.net","subject":"Re: git submodules and commit","fromName":"Petr Baudis","fromEmail":"pasky@suse.cz","sentAt":"2008-07-16T12:11:24Z","receivedAt":"2008-07-16T12:11:24Z","isPatch":false,"sender":{"key":"pasky@ucw.cz","avatar":"https://avatars.githubusercontent.com/u/18439?v=4"},"body":"On Wed, Jul 16, 2008 at 01:35:24PM +0200, Johannes Sixt wrote:\n> Ah, is this your actual scenario? Just to make sure we are talking about\n> the same thing:\n> \n> - You own superproject P.\n> - $Maintainer owns submodule S.\n> - You use S in P.\n> - You make changes to S that you would like $Maintainer to include in the\n> next release.\n> x You use in P your changes to S while $Maintainer has not yet released a\n> new version of S with your changes.\n> - Finally your changes arrive via the new release of S.\n> \n> That *is* the intended use-case for submodules. But you have to play the\n> game by the rules:\n> \n> - $Maintainer defines the official states of S.\n> \n> - You must never commit an unofficial state of S in P.\n\nI think the issue here is that $Maintainer = him (or Maintainers(P) =\nMaintainers(S), in general); the workflow you described still works, but\nis overly complicated and that is the original complaint.\n\n\t\t\t\tPetr \"Pasky\" Baudis\n"},{"id":"83512","messageId":"320075ff0807160548qae5d702jafe3df63363c512c@mail.gmail.com","threadId":"14480","inReplyTo":"487DDCFC.9020007@viscovery.net","subject":"Re: git submodules and commit","fromName":"Nigel Magnay","fromEmail":"nigel.magnay@gmail.com","sentAt":"2008-07-16T12:48:09Z","receivedAt":"2008-07-16T12:48:09Z","isPatch":false,"sender":{"key":"nigel.magnay@gmail.com","avatar":"https://gravatar.com/avatar/d85cf38287bef3a8e4fa02358d2756d7589f8676c5eeb881ce2f6d731e4526c3?d=mp&s=160"},"body":"> Ah, is this your actual scenario? Just to make sure we are talking about\n> the same thing:\n>\n> - You own superproject P.\n> - $Maintainer owns submodule S.\n> - You use S in P.\n> - You make changes to S that you would like $Maintainer to include in the\n> next release.\n> x You use in P your changes to S while $Maintainer has not yet released a\n> new version of S with your changes.\n> - Finally your changes arrive via the new release of S.\n>\n> That *is* the intended use-case for submodules. But you have to play the\n> game by the rules:\n>\n\nYes, that is the situation - with the proviso that it's not always\nclear in company environments who $Maintainer actually is. For\nexample, if the only changes occurring in S come from me, then chances\nare come release cycle, $Maintainer == me.\n\nP and S aren't distant projects, they're closely coupled.\n\n> - $Maintainer defines the official states of S.\n>\nYes - there is one branch ('master') which the changes eventually\nshould be merged to, and releases will be performed on\n\n> - You must never commit an unofficial state of S in P.\n>\n\nIf by that you mean that the only person to move the branch 'master'\nis $Maintainer, then I agree.\nIf by that you mean that you can't commit at all to the S tree (and\nthe S submodule pointer) then I don't agree, and I think that's a\nserious limitation in productivity.\n\n> The critical step in above list I marked with x:\n>\n> - During the period where only *you* have the new changes to S, you must\n> *not* commit your submodule state to P. Instead, you write P in such a way\n> that it can work with both the old version of S and the upcoming release\n> that will have your changes[*]. This way you make sure that your consumers\n> of P always have a working version regardless of which version of S they use.\n>\n\nJust to be clear - there's more than just 'me' working on P - there's\na whole team of people working on it. And there's Q R S and T teams\nalso working on projects that also have S.\n\nChanges that happen to S are, often, new features or bug fixes. We\ncan't just stop because there isn't an 'official' version of S yet\n(and the official version might end up simply being a FF anyway), so\nsaying 'don't commit your submodule state to P' is unrealistic.\n\nAnd that should be the big advantage of git. If we suddenly find we\nneed some additional functionality in S, we just add it to our\nP-branch-of-S. The $Maintainer (if he exists) can review these\nupcoming changes in the tree, and merge them to master as appropriate\n(or work with the projects to iron out cross-branch\nincompatibilities). The best example is that S is a \"product\", and (by\nmanagement decree), the only product changes that happen will occur\nbecause of *projects* (like P). And we can do this (and it's\ninfinitely better than svn, where 'ooh, branches too hard, everyone in\n[P-T] just commit to trunk'. But the UI is an ache.\n\n\n> - After you have received the new release of S from $Maintainer, you\n> commit the new state of S in P. And if you are nice to your consumers of\n> P, then you *do not* remove the workaround from P just yet, so that you\n> don't force them to upgrade S. You will remove it later only if it becomes\n> a maintainance burden.\n>\nMaintaining backwards compatibility isn't an issue at all for us.\n\n> [*] If it is not possible to make P work with old and new versions, then\n> you have to work closely with the $Maintainer so that you never need\n> commit an unofficial state of S into P.\n>\n> -- Hannes\n>\n>\n"},{"id":"83514","messageId":"487DF9BB.10107@viscovery.net","threadId":"14480","inReplyTo":"320075ff0807160548qae5d702jafe3df63363c512c@mail.gmail.com","subject":"Re: git submodules and commit","fromName":"Johannes Sixt","fromEmail":"j.sixt@viscovery.net","sentAt":"2008-07-16T13:38:03Z","receivedAt":"2008-07-16T13:38:03Z","isPatch":false,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Nigel Magnay schrieb:\n> P and S aren't distant projects, they're closely coupled.\n\nAnd I'm saying that submodules are designed for *loosely* coupled projects.\n\nIt's no wonder that this tool is awkward to use in your workflow.\n\n-- Hannes\n"},{"id":"83516","messageId":"320075ff0807160703v3f16ff5bue722b760ad66488e@mail.gmail.com","threadId":"14480","inReplyTo":"487DF9BB.10107@viscovery.net","subject":"Re: git submodules and commit","fromName":"Nigel Magnay","fromEmail":"nigel.magnay@gmail.com","sentAt":"2008-07-16T14:03:41Z","receivedAt":"2008-07-16T14:03:41Z","isPatch":false,"sender":{"key":"nigel.magnay@gmail.com","avatar":"https://gravatar.com/avatar/d85cf38287bef3a8e4fa02358d2756d7589f8676c5eeb881ce2f6d731e4526c3?d=mp&s=160"},"body":"On Wed, Jul 16, 2008 at 2:38 PM, Johannes Sixt <j.sixt@viscovery.net> wrote:\n> Nigel Magnay schrieb:\n>> P and S aren't distant projects, they're closely coupled.\n>\n> And I'm saying that submodules are designed for *loosely* coupled projects.\n>\n> It's no wonder that this tool is awkward to use in your workflow.\n>\n\nOk in a sense. I don't think it's particularly clear from the\ndocumentation that this is a limitation of submodules though.\n\nGiven that\n- The only way in git to separate out re-usable modules is by the use\nof submodules\nand\n- It's a pretty common usecase for these submodules to be interrelated\nand\n- Looking over the list archives, it seems this is quite common complaint\n\n\"I really like the git submodule implementation, I just don't like how\nhard it is to work with\"\n\n \"The current behaviour strongly encourages me to avoid submodules\nwhen I would otherwise like to use them, just to keep the rest of my\nteam members (who are not git experts) from going insane.\"\n\n \"For my use case, I passionately dislike the fact that a submodule is\nnot updated automatically.  There's never a time when I don't want to\nupdate the submodule.  The submodule is a very important piece of our\nproject and the super-project depends on it being at the right\nversion.\"\n\nand\n- All the technical capability is there, it's just the porcelain\nthat's causing the friction.\nthen\n would this not seem to be an area that could be improved? Even if it\nwere an optional mode of working?\n"},{"id":"83517","messageId":"20080716141738.GN32184@machine.or.cz","threadId":"14480","inReplyTo":"320075ff0807160703v3f16ff5bue722b760ad66488e@mail.gmail.com","subject":"Re: git submodules and commit","fromName":"Petr Baudis","fromEmail":"pasky@suse.cz","sentAt":"2008-07-16T14:17:38Z","receivedAt":"2008-07-16T14:17:38Z","isPatch":false,"sender":{"key":"pasky@ucw.cz","avatar":"https://avatars.githubusercontent.com/u/18439?v=4"},"body":"On Wed, Jul 16, 2008 at 03:03:41PM +0100, Nigel Magnay wrote:\n> - All the technical capability is there, it's just the porcelain\n> that's causing the friction.\n> then\n>  would this not seem to be an area that could be improved? Even if it\n> were an optional mode of working?\n\nSo, were there already any patches posted to add such a functionality\nthat were rejected? If not, apparently noone cared _enough_, yet. ;-)\nYou may be the first!\n\nI don't know if there are any _present_ \"free developers\" willing to\npick up this task now.  For many (most?) Git developers, submodules\nsimply aren't a priority.  For me, they actually currently are, but I\nprobably won't want to use them in your way either (even though I can\nagree that your sentiments are valid), so I will personally invest my\ntime in doing other things than figuring out the precise semantics\nthese operations should have etc.\n\n-- \n\t\t\t\tPetr \"Pasky\" Baudis\nGNU, n. An animal of South Africa, which in its domesticated state\nresembles a horse, a buffalo and a stag. In its wild condition it is\nsomething like a thunderbolt, an earthquake and a cyclone. -- A. Pierce\n"},{"id":"83518","messageId":"320075ff0807160731g2537780fja1d6f5664163e876@mail.gmail.com","threadId":"14480","inReplyTo":"20080716141738.GN32184@machine.or.cz","subject":"Re: git submodules and commit","fromName":"Nigel Magnay","fromEmail":"nigel.magnay@gmail.com","sentAt":"2008-07-16T14:31:46Z","receivedAt":"2008-07-16T14:31:46Z","isPatch":false,"sender":{"key":"nigel.magnay@gmail.com","avatar":"https://gravatar.com/avatar/d85cf38287bef3a8e4fa02358d2756d7589f8676c5eeb881ce2f6d731e4526c3?d=mp&s=160"},"body":"> On Wed, Jul 16, 2008 at 03:03:41PM +0100, Nigel Magnay wrote:\n>> - All the technical capability is there, it's just the porcelain\n>> that's causing the friction.\n>> then\n>>  would this not seem to be an area that could be improved? Even if it\n>> were an optional mode of working?\n>\n> So, were there already any patches posted to add such a functionality\n> that were rejected? If not, apparently noone cared _enough_, yet. ;-)\n> You may be the first!\n>\n> I don't know if there are any _present_ \"free developers\" willing to\n> pick up this task now.  For many (most?) Git developers, submodules\n> simply aren't a priority.  For me, they actually currently are, but I\n> probably won't want to use them in your way either (even though I can\n> agree that your sentiments are valid), so I will personally invest my\n> time in doing other things than figuring out the precise semantics\n> these operations should have etc.\n>\n\nThat's cool. I was guessing it might be the case (or alternatively\nthat someone might say 'yeah, but it's 25% of the way there'); my\noriginal query was also one of an offer of help ;-) My guess though is\nthat the core-devs have much more connected neural pathways at\nthinking about the problems around the edge cases to be able to give\nwarnings of  'there be dragons'!\n\n\nNigel\n"},{"id":"83527","messageId":"32541b130807160843k25f1d7d3u8bfecd6c1c6eab91@mail.gmail.com","threadId":"14480","inReplyTo":"320075ff0807160332k5e49c256tb4191de628ecf41c@mail.gmail.com","subject":"Re: git submodules and commit","fromName":"Avery Pennarun","fromEmail":"apenwarr@gmail.com","sentAt":"2008-07-16T15:43:03Z","receivedAt":"2008-07-16T15:43:03Z","isPatch":false,"sender":{"key":"apenwarr@gmail.com","avatar":"https://avatars.githubusercontent.com/u/20592?v=4"},"body":"On 7/16/08, Nigel Magnay <nigel.magnay@gmail.com> wrote:\n> I wonder if this is a fairly common pattern. We tend to have modules\n>  as git repositories, and projects that tie together those git\n>  repositories as submodules. [and submodules are necessary because they're\n>  shared between multiple supermodules].\n\nI have exactly the same problem as you, and have been working on\nimproving my own workflow so that someday I can offer patches that\nmight be generally applicable.\n\nIn the meantime, my solution is... some shell scripts checked in at\nthe top level of my project. :)\n\nIn one of my applications, I have a /wv submodule, which provides a\ncross-platform build environment.  That environment respectively\ncontains a /wv/wvstreams submodule, which is a library that we use.\n\nWhen I make a change to wvstreams that's needed for my application, I\nneed to check into wvstreams, then check that link into wv, then check\nthat link into the application.  Then, when I push, I have to make\nsure to always push wvstreams first, then wv, then application, or\nelse other users can end up with \"commit id xxxxxx not found\" type\nerrors.\n\nSo basically, committing is always harmless, since I can do anything I\nwant in my own repo (and I want to be able to update wvstreams\n*without* always updating wv, and so on).  The tricky part is pushing.\n Here's the script I wrote to make sure I don't screw up when pushing:\n\n\n~/src/vx-lin $ cat push-git-modules\n#!/bin/sh -x\nset -e\ntest -e wv/wvstreams/Makefile\n(cd wv/wvstreams && git push origin HEAD:master) &&\n(cd wv && git push origin HEAD:master) &&\ngit push origin HEAD:master ||\necho \"Failed!\"\n\n\nNow, this script is pretty flawed.  Notably, it always pushes to the\n'master' branch, which is stupid.  However, it works in our particular\nworkflow, because wvstreams isn't being modified by too many\ndevelopers and it's okay if we all commit to master.  This is also\naided by the fact that people are trained to push only after they've\nmade all the unit tests pass, etc.  And further, individual apps don't\nhave to update their wvstreams to the latest anyway unless they really\nneed the latest changes, which is a wonderful feature of git\nsubmodules.\n\nNow, sometimes the above push script will fail.  In my experience,\nthis is only when someone else has pushed in something before you,\nwhich means a fast-forward is not possible on at least one of the\nrepos.  When that happens, you have to pull first, using this script:\n\n~/src/vx-lin $ cat newest-git-modules\n#!/bin/sh -x\nset -e\ntest -e wv/wvstreams/Makefile\ngit pull origin master &&\n(cd wv && git pull origin master) &&\n(cd wv/wvstreams && git pull origin master) ||\necho \"Failed!\"\n\nThis pulls in the latest version of application, wv, and wvstreams, in\nthat order, and stops in case of any merge conflicts so that you can\nresolve them by hand.  It's safe to run the above script more than\nonce in case you're not sure if it's done or not.\n\nAfter pulling the new modules, you may need to make new commits to\nupdate to the latest submodule commits - if that's indeed what you\nwant.  And then you can run push-git-modules, and be reasonably\nassured that it will work (unless someone made another push while you\nwere fixing conflicts).\n\nFinally, I have another script that retrieves the *currently linked*\nversion of the git modules.  I wish git-checkout would do this\nautomatically, but it doesn't, for apparently-difficult-to-resolve\nsafety reasons.  Anyway, note that this script uses the existence of\nsubmodule/Makefile as \"proof\" that the submodule was checked out\ncorrectly.\n\n\n~/src/vx-lin $ cat get-git-modules\n#!/bin/sh -x\nset -e\ngit submodule init\ngit submodule update\ntest -e wv/Makefile\n(cd wv && git submodule init && git submodule update)\ntest -e wv/wvstreams/Makefile\n\n\n>  I guess it probably gets sticky when there are merge conflicts. Is\n>  anyone working on this kind of thing; I might be able to give some\n>  time to help work on it?\n\nSo as you can see, my scripts are crappy.  However, they have already\ndrastically reduced the number of mistakes made by developers in my\ngroup (especially commits lost due to 'git submodule update' at the\nwrong time, and pushes of the supermodule before the submodule).\n\nIf you want to work with me on my new submodule workflow (and I'd\ncertainly appreciate it!) then I'd suggest one or more of the\nfollowing starting points:\n\n- Take the recursive push, pull, and update operations described\nabove, make them general (ie. not referring to my submodules by name\n:)), and add them as commands in the real git-submodule script.  The\ntrickiest part here will be figuring out which remote branch to\npush/pull.\n\n- Perhaps add a \"recursive commit\" operation that recursively\nauto-commits submodule refs, for use after running the\nnewest-git-modules script.  The commit message could be auto-generated\nusing something like \"git-whatchanged\" on the submodule.\n\n- See what can be done about making git-checkout automatically\ngit-submodule-update *if and only if* the currently checked-out commit\nof the submodule exactly matches the one that was checked out last\ntime, *and* the desired commit is already available in the submodule\nrepo (which is not necessarily the case, if you haven't fetched it\nyet).  That is, as with any file in git, if it hasn't changed from the\none in the repo, you know you won't lose any information if you just\nauto-replace it with the new version.\n\n- Fix git-submodule-update to not just switch submodule branches if\nyou've made checkins in that submodule.  Right now, commits to a\nsubmodule by default don't go to any branch, so if you subsequently\nrun git-submodule-update, your commits are lost (except for the\nreflog).  This is very un-git-like in general, and\ngit-submodule-update should be much more polite.\n\nNote that git-submodule is only about 800 lines of shell.  It's\nremarkably straightforward to make it do whatever you want.  The hard\npart is figuring out what you want, and making sure you don't stomp on\n*other* people's workflows while you're there.\n\nAlso note that even if you don't contribute any of the above, I'm\nplanning to someday make time to do it myself :)  But don't hold your\nbreath.  I've been busy.\n\nHave fun,\n\nAvery\n"},{"id":"83658","messageId":"320075ff0807170247g7bb18252ma50b202e1d762296@mail.gmail.com","threadId":"14480","inReplyTo":"32541b130807160843k25f1d7d3u8bfecd6c1c6eab91@mail.gmail.com","subject":"Re: git submodules and commit","fromName":"Nigel Magnay","fromEmail":"nigel.magnay@gmail.com","sentAt":"2008-07-17T09:47:29Z","receivedAt":"2008-07-17T09:47:29Z","isPatch":false,"sender":{"key":"nigel.magnay@gmail.com","avatar":"https://gravatar.com/avatar/d85cf38287bef3a8e4fa02358d2756d7589f8676c5eeb881ce2f6d731e4526c3?d=mp&s=160"},"body":"On Wed, Jul 16, 2008 at 4:43 PM, Avery Pennarun <apenwarr@gmail.com> wrote:\n> On 7/16/08, Nigel Magnay <nigel.magnay@gmail.com> wrote:\n>> I wonder if this is a fairly common pattern. We tend to have modules\n>>  as git repositories, and projects that tie together those git\n>>  repositories as submodules. [and submodules are necessary because they're\n>>  shared between multiple supermodules].\n>\n> I have exactly the same problem as you, and have been working on\n> improving my own workflow so that someday I can offer patches that\n> might be generally applicable.\n>\n> In the meantime, my solution is... some shell scripts checked in at\n> the top level of my project. :)\n>\n> In one of my applications, I have a /wv submodule, which provides a\n> cross-platform build environment.  That environment respectively\n> contains a /wv/wvstreams submodule, which is a library that we use.\n>\n> When I make a change to wvstreams that's needed for my application, I\n> need to check into wvstreams, then check that link into wv, then check\n> that link into the application.  Then, when I push, I have to make\n> sure to always push wvstreams first, then wv, then application, or\n> else other users can end up with \"commit id xxxxxx not found\" type\n> errors.\n>\n> So basically, committing is always harmless, since I can do anything I\n> want in my own repo (and I want to be able to update wvstreams\n> *without* always updating wv, and so on).  The tricky part is pushing.\n>  Here's the script I wrote to make sure I don't screw up when pushing:\n>\n>\n> ~/src/vx-lin $ cat push-git-modules\n> #!/bin/sh -x\n> set -e\n> test -e wv/wvstreams/Makefile\n> (cd wv/wvstreams && git push origin HEAD:master) &&\n> (cd wv && git push origin HEAD:master) &&\n> git push origin HEAD:master ||\n> echo \"Failed!\"\n>\n>\n> Now, this script is pretty flawed.  Notably, it always pushes to the\n> 'master' branch, which is stupid.  However, it works in our particular\n> workflow, because wvstreams isn't being modified by too many\n> developers and it's okay if we all commit to master.  This is also\n> aided by the fact that people are trained to push only after they've\n> made all the unit tests pass, etc.  And further, individual apps don't\n> have to update their wvstreams to the latest anyway unless they really\n> need the latest changes, which is a wonderful feature of git\n> submodules.\n>\n\nYes - I use something rather similar on my desktop. The unfortunate\nthing is that I know how submodules work, and am happy with the\nscripts. My users are sometimes in the 'git gui' types - not as\ntechnically literate, and likely on Windows.\n\n> Now, sometimes the above push script will fail.  In my experience,\n> this is only when someone else has pushed in something before you,\n> which means a fast-forward is not possible on at least one of the\n> repos.  When that happens, you have to pull first, using this script:\n>\n> ~/src/vx-lin $ cat newest-git-modules\n> #!/bin/sh -x\n> set -e\n> test -e wv/wvstreams/Makefile\n> git pull origin master &&\n> (cd wv && git pull origin master) &&\n> (cd wv/wvstreams && git pull origin master) ||\n> echo \"Failed!\"\n>\n> This pulls in the latest version of application, wv, and wvstreams, in\n> that order, and stops in case of any merge conflicts so that you can\n> resolve them by hand.  It's safe to run the above script more than\n> once in case you're not sure if it's done or not.\n>\n> After pulling the new modules, you may need to make new commits to\n> update to the latest submodule commits - if that's indeed what you\n> want.  And then you can run push-git-modules, and be reasonably\n> assured that it will work (unless someone made another push while you\n> were fixing conflicts).\n>\n\nYeah - this happens a lot. If someone else commits to the\nsuper-project before you, it's always a conflict. What's annoying is\nthere's no way around it (though resolution is easy - force to current\n- but it this is a big bit of what confuses my users. They say 'but I\nalready resolved the merges in the submodule itself'. I'm not sure\nthere's an easy way around it though - and this is part of my worry\nthat there's hidden complexity with trying to make it 'look like 1 big\nrepo').\n\n> Finally, I have another script that retrieves the *currently linked*\n> version of the git modules.  I wish git-checkout would do this\n> automatically, but it doesn't, for apparently-difficult-to-resolve\n> safety reasons.  Anyway, note that this script uses the existence of\n> submodule/Makefile as \"proof\" that the submodule was checked out\n> correctly.\n>\n>\n> ~/src/vx-lin $ cat get-git-modules\n> #!/bin/sh -x\n> set -e\n> git submodule init\n> git submodule update\n> test -e wv/Makefile\n> (cd wv && git submodule init && git submodule update)\n> test -e wv/wvstreams/Makefile\n>\n>\n>>  I guess it probably gets sticky when there are merge conflicts. Is\n>>  anyone working on this kind of thing; I might be able to give some\n>>  time to help work on it?\n>\n> So as you can see, my scripts are crappy.  However, they have already\n> drastically reduced the number of mistakes made by developers in my\n> group (especially commits lost due to 'git submodule update' at the\n> wrong time, and pushes of the supermodule before the submodule).\n>\n\nYeah. I have an additional usecase, which is around pulling from\nanother user. If they've made changes in their tree(s) that they want\nto get reviewed, normally I could do something like\n\ngit fetch ssh://joebloggs.computer/blah +refs/heads/*:refs/remotes/joebloggs/*\n\nBut if they've made cross-module changes, I'm SOL, as fetching their\nsuper-project will have references to commits that aren't in the repo\nmentioned in .gitmodules (only in joebloggs's tree) - so doing git\nsubmodule update doesn't help. I have to go into each submodule and\nexplicitly fetch. It feels wierdly centralised for this otherwise\ndistributed tool.\n\n> If you want to work with me on my new submodule workflow (and I'd\n> certainly appreciate it!) then I'd suggest one or more of the\n> following starting points:\n>\n> - Take the recursive push, pull, and update operations described\n> above, make them general (ie. not referring to my submodules by name\n> :)), and add them as commands in the real git-submodule script.  The\n> trickiest part here will be figuring out which remote branch to\n> push/pull.\n>\n\nWhat's bugging me is I'm not sure that it's the right place. It seems\n(to me) that having the only place that knows about submodules being\nthe 'git submodules' script isn't right. What users want is 'git fetch\n<blah>' to do the lot - that, for the most, it ought to do the\nsubmodule init, update and clever stuff automatically. That if 'git\nfetch' is porcelain, then the porcelain needs to call the\ngit-submodule stuff.\n\nBut - perhaps it's best to approach it as scripts for now :)\n\n> - Perhaps add a \"recursive commit\" operation that recursively\n> auto-commits submodule refs, for use after running the\n> newest-git-modules script.  The commit message could be auto-generated\n> using something like \"git-whatchanged\" on the submodule.\n>\nHm - I'd be happy with the same commt message in all modules. What I\nwant is to be able to do (from the top) 'git commit -a' or the same\nwith the GUI, and see all the files to be committed regardless of\nwhether they're in a submodule or not.\n\nI'm guessing you probably need to build a tree of submodules, and\ncommit from the tips backwards towards the top level superproject.\n\nThis is what the users want - something that mirrors 'svn ci' at the\ntop level - \"Please Check All My stuff in\".\n\n> - See what can be done about making git-checkout automatically\n> git-submodule-update *if and only if* the currently checked-out commit\n> of the submodule exactly matches the one that was checked out last\n> time, *and* the desired commit is already available in the submodule\n> repo (which is not necessarily the case, if you haven't fetched it\n> yet).  That is, as with any file in git, if it hasn't changed from the\n> one in the repo, you know you won't lose any information if you just\n> auto-replace it with the new version.\n>\n> - Fix git-submodule-update to not just switch submodule branches if\n> you've made checkins in that submodule.  Right now, commits to a\n> submodule by default don't go to any branch, so if you subsequently\n> run git-submodule-update, your commits are lost (except for the\n> reflog).  This is very un-git-like in general, and\n> git-submodule-update should be much more polite.\nWe always move back onto a branch immediately after submodule update,\nwhich is another thing to forget!\n\n>\n> Note that git-submodule is only about 800 lines of shell.  It's\n> remarkably straightforward to make it do whatever you want.  The hard\n> part is figuring out what you want, and making sure you don't stomp on\n> *other* people's workflows while you're there.\n>\nTotally.\n\n> Also note that even if you don't contribute any of the above, I'm\n> planning to someday make time to do it myself :)  But don't hold your\n> breath.  I've been busy.\n>\nDitto.\n\n> Have fun,\n>\n> Avery\n>\n"},{"id":"83701","messageId":"32541b130807170812s7586da8ejbeeee64806490b13@mail.gmail.com","threadId":"14480","inReplyTo":"320075ff0807170247g7bb18252ma50b202e1d762296@mail.gmail.com","subject":"Re: git submodules and commit","fromName":"Avery Pennarun","fromEmail":"apenwarr@gmail.com","sentAt":"2008-07-17T15:12:55Z","receivedAt":"2008-07-17T15:12:55Z","isPatch":false,"sender":{"key":"apenwarr@gmail.com","avatar":"https://avatars.githubusercontent.com/u/20592?v=4"},"body":"On 7/17/08, Nigel Magnay <nigel.magnay@gmail.com> wrote:\n> Yeah - this happens a lot. If someone else commits to the\n>  super-project before you, it's always a conflict. What's annoying is\n>  there's no way around it (though resolution is easy - force to current\n>  - but it this is a big bit of what confuses my users. They say 'but I\n>  already resolved the merges in the submodule itself'. I'm not sure\n>  there's an easy way around it though - and this is part of my worry\n>  that there's hidden complexity with trying to make it 'look like 1 big\n>  repo').\n\nThis might not be as hard as it sounds.  We probably just need to\nteach the supermodule how to merge gitlinks safely.  So basically, if\nI moved the gitlink from A to B, and he moved it from A to C, then it\nneeds to check whether a fast forward merge already exists for the\nsubmodule to combine B and C.  This is easier than it sounds, because\nif I *already* ran my newest-git-modules script in the inner module,\nthen I've already manually resolved the merge in question, so that B\n*does* actually contain C.\n\nRight now, such a thing results in a conflict.  It isn't really a\nconflict though, it's a fast forward, and the supermodule's merge\nshould ideally just notice that and run with it.\n\nSadly I know very little about the merge code.  But I would be happy\nto help you test a patch that implemented this :)\n\nA slightly more advanced version of the same would automatically walk\ninto the submodule and ask it to merge B and C.  I suspect that is way\nmore complicated than it sounds at first glance, though (particularly\nif the new B or C gitlink doesn't have A as a parent at all, which\ncouldn't happen in a unified git repo, but is perfectly allowable with\nsubmodules).\n\nWith anything like this, there's always the question of what happens\nif you haven't done a \"fetch\" in the submodule yet; I think reverting\nto the current behaviour is fine in that case, because I can make\nnewest-git-modules to always fetch before trying anything anyway.\n\n> Yeah. I have an additional usecase, which is around pulling from\n>  another user. If they've made changes in their tree(s) that they want\n>  to get reviewed, normally I could do something like\n>\n>  git fetch ssh://joebloggs.computer/blah +refs/heads/*:refs/remotes/joebloggs/*\n>\n>  But if they've made cross-module changes, I'm SOL, as fetching their\n>  super-project will have references to commits that aren't in the repo\n>  mentioned in .gitmodules (only in joebloggs's tree) - so doing git\n>  submodule update doesn't help. I have to go into each submodule and\n>  explicitly fetch. It feels wierdly centralised for this otherwise\n>  distributed tool.\n\nOne slightly non-obvious option here is to actually use the *same*\nrepo for all your supermodules and submodules, then use \".\" as the\nrepo path in your .gitmodules.  The original clone is huge that way,\nbut it makes it obvious how to get any objects that you're missing.\n\nThen you could construct your submodules using --reference the\nsupermodule.  Thus, doing a \"fetch\" of your user's supermodule, you'll\nalso get all the objects it references.\n\nNote that I've only basically tried out this technique.  I think it's\nthe one for me, but I haven't experimented with it enough to know any\npitfalls.  When I've brought it up on the list, it's been shot down\nbecause it wouldn't work for gigantic mega-repositories like KDE where\nthe whole point of submodules is to *not* download all the modules\nevery time.  It works for me, though, because my software doesn't even\n*build* unless I have all the modules.\n\n(And before anyone asks, yes, it still makes sense to use submodules\nbecause some of the modules are shared with other projects.)\n\n> What's bugging me is I'm not sure that it's the right place. It seems\n>  (to me) that having the only place that knows about submodules being\n>  the 'git submodules' script isn't right. What users want is 'git fetch\n>  <blah>' to do the lot - that, for the most, it ought to do the\n>  submodule init, update and clever stuff automatically. That if 'git\n>  fetch' is porcelain, then the porcelain needs to call the\n>  git-submodule stuff.\n\nThere is some architectural elegance to the fact that the gitlink\nstuff is almost completely abstract (just a number, really) in the\ncore of git, and is only made \"real\" by running git-submodule, which\nactually extracts files and makes .git dirs and fetches submodules and\nwhatnot.\n\nHowever, it's architectural elegance, not UI elegance.  As a user, I\nmostly don't want to have to care whether a particular directory is a\n\"submodule\" or not, so the extra push and commit and fetch steps\nbecome tedious.  From the point of view of UI, I agree with you.\n\nPerhaps this is a plumbing vs. porcelain issue.  I don't think\ngit-submodule has made an attempt to separate the two, since it seems\nto be porcelain, but there's no \"submodule plumbing\" underneath\n(AFAICS) that things like git-fetch and git-commit and git-push can\nplug into.\n\n>  But - perhaps it's best to approach it as scripts for now :)\n\nI suspect so :)\n\n> Hm - I'd be happy with the same commt message in all modules. What I\n>  want is to be able to do (from the top) 'git commit -a' or the same\n>  with the GUI, and see all the files to be committed regardless of\n>  whether they're in a submodule or not.\n\nThat actually wouldn't work very well for me.  I do need the commits\nseparated, because that's why I'm using submodules in the first place\ninstead of the \"subtree\" merge strategy.\n\nBasically, I'm still planning on contributing patches to my class\nlibrary upstream, and the patches need to talk about how they affect\nthe *library*, not what I changed in my application.  So I *would*\nwant to write separate commit messages in all cases.  I can see how\nother people might not, though.\n\n>  This is what the users want - something that mirrors 'svn ci' at the\n>  top level - \"Please Check All My stuff in\".\n\nNote that submodules are more like svn:externals, which also require\nyou to commit each module separately.  One big difference there is\nthat you don't need to commit to the supermodule each time you commit\nto the submodule, but that's only because svn:externals by default\nlinks to a branch, not to a particular revision.  The\nrevision-specific linking is very worthwhile, I think, so requiring an\nextra commit is mostly okay here.\n\nPerhaps automating the extra commit would be nice in some cases, but\nfor me, for example, I tend to combine my \"update to newest version of\nsubmodule\" commit with some changes to the supermodule, since the\nreason I updated was to implement this new feature.\n\nHave fun,\n\nAvery\n"},{"id":"83886","messageId":"46dff0320807180911h63999019q5dd56e9341d0fbb6@mail.gmail.com","threadId":"14480","inReplyTo":"32541b130807160843k25f1d7d3u8bfecd6c1c6eab91@mail.gmail.com","subject":"Re: git submodules and commit","fromName":"Ping Yin","fromEmail":"pkufranky@gmail.com","sentAt":"2008-07-18T16:11:54Z","receivedAt":"2008-07-18T16:11:54Z","isPatch":false,"sender":{"key":"pkufranky@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5346?v=4"},"body":"On Wed, Jul 16, 2008 at 11:43 PM, Avery Pennarun <apenwarr@gmail.com> wrote:\n> On 7/16/08, Nigel Magnay <nigel.magnay@gmail.com> wrote:\n\n> If you want to work with me on my new submodule workflow (and I'd\n> certainly appreciate it!) then I'd suggest one or more of the\n> following starting points:\n>\n> - Take the recursive push, pull, and update operations described\n> above, make them general (ie. not referring to my submodules by name\n> :)), and add them as commands in the real git-submodule script.  The\n> trickiest part here will be figuring out which remote branch to\n> push/pull.\n\nSee http://article.gmane.org/gmane.comp.version-control.git/69834\n([PATCH] Added recurse command to git submodule)\nOr search \"submodule recursive\" in gmane.\n\nThe recursive pull,diff,status for submodule is implemented by Imran M\nYousuf. And IIRC, with this patch, you can walk through the submodule\nhierarchy to exectute any command.\n\n\n\n\n-- \nPing Yin\n"}]}