{"thread":{"id":"18545","subject":"[RFC GSoC 2009: git-submodule for multiple, active developers on active trees]","startedAt":"2009-03-25T20:14:58Z","lastAt":"2009-04-01T16:13:57Z","messageCount":12,"participants":["P Baker","Shawn O. Pearce","Johannes Schindelin","Andreas Ericsson"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"109408","messageId":"526944450903251314o622711b5u3665bac90398d6be@mail.gmail.com","threadId":"18545","inReplyTo":null,"subject":"[RFC GSoC 2009: git-submodule for multiple, active developers on active trees]","fromName":"P Baker","fromEmail":"me@retrodict.com","sentAt":"2009-03-25T20:14:58Z","receivedAt":"2009-03-25T20:14:58Z","isPatch":false,"sender":{"key":"me@retrodict.com","avatar":null},"body":"Greetings, I've been working on this for a while, but figured I'd send\nit out while I've still got some time left before I submit it!\n\nAny comments/questions would be welcome, as I'd really love to spend a\nsummer working on git.\n\nAbstract:\n\tThis project focuses on upgrading git-submodule to manage code\ncreated in external projects in ways that allow users to safely branch\nand merge that code without loss of data or routine merge conflicts.\nThis will incorporate some changes made on the ‘pu’ branch, but will\nalso include making substantial changes to git-submodule underlying\ncode.\n\nContent:\ngit-submodule is currently a good tool designed to allow developers to\nleverage the work of others by incorporating external code into a\nproject. However, its implementation is underdeveloped as most core\nusers/developers are not heavy users of the application. In contrast\nto much of the rest of the project, these holes create usage problems\nwhen git does not act according to developers’ expectations. This\nproject would devote a summer of work to filling in the gaps so that\ngit-submodules contains the features necessary to fully exploit its\npotential to track, update and edit external codebases incorporated\ninto a super-project.\n\nAs opposed to “remotes,” which also incorporate external code into a\nproject, submodules maintain the distinct nature of code and separate\nthe projects’ history. This is perfect for the intended nature of\n“embedding foreign repositories in dedicated subdirectories of the\nsource tree.” For example, managing plug-ins within a larger,\nstandalone project that depends on the plug-ins. However, the\nshortcomings of git-submodule create headaches for developers\nattempting to use git and might prevent its adoption among those\ndevelopers not willing to either create laborious workarounds or\nexplicitly create manual management techniques.\n\nAdding the features to fully enable git-submodule would allow heavy\nusers of projects built on other actively developed projects to use\ngit to manage this interaction in intuitive and predictable ways.\nAdding this feature set would give current users a desired tool, boost\ngit’s credibility by providing a common feature among revision\nsystems, make git’s adoption for new and existing projects easier and,\nas a result, likely boost git’s usage.\n\nThis project will consist of several stages: an initial community\nbased design review and investigation of specific requirements;\nspecifying and documenting the planned changes; writing and debugging\nthe code and related tests; and finally merging it into a public\nrelease. The tentative timeline is:\n\nEnd of May – Conclusively finish the public discussion regarding where\ngit-submodules needs to go\nBeginning of June – Produce final specifications (including method stubs)\nMiddle of July – Finish active code and test development\nEnd of July – Merge code into production release, fix public submitted bugs\nMiddle of August – Prepare code for final release and finish\nuser-facing documentation\n\nThis timeline should allow adequate flexibility while establishing\ndeadlines that ensure that the project will be completed in a timely\nand efficient manner.\n\nA few specific changes that this project will likely include are:\n\n*use .git instead of .gitmodules\n*move objects of submodules into .git/ directory\n*git submodule update --init should initialize nested levels of submodules\n*protect changes in local submodules when doing “git submodule update”\n\nThese changes, compiled from feature requests on the git mailing list\nand formulated in response to blog posts regarding git-submodule’s\nissues, are representative of the full list of changes. Most\ndevelopment will need to occur within git-submodules.sh, without\nchanging much plumbing, however, other files might be affected by more\nsubstantial changes.\n\nWhile git-submodule is stable and operational, it is not widely\nupdated and has not seen much change beyond bug-fixes. After reaching\nits current feature complete status, the last time a feature of any\nnovelty was included in a public release was August 2008. However,\nsome work has already been started on the ‘pu’ branch, which will need\nto be reviewed and probably incorporated into this project. At the\nconclusion of the project, one metric by which to evaluate its success\nwill be its acceptance in online communities. The final goal is to\nmake git a top-tier version control system in its management of\nexternal code repositories.\n\nMy main usage of git started when managing a summer project as an\nintern that had many of the requirements that make git-submodule\nproblematic: built in Ruby on Rails and dependent on other plug-ins,\nsome of which were managed in SVN. Even though the Ruby on Rails\ncommunity has popularized within itself the use of sub-modules to\nmanage plug-ins and quite a bit has been written on the topic, a\nsignificant portion end either in frustration or convoluted\nwork-arounds. The problems and extremely confusing nature of\ngit-submodule led me to give up on it altogether (an unfortunately\ncommon occurrence), and manage the code and updates to it by hand.\n\nI currently am finishing my sophomore year as an Electrical Engineer\nat the University of Pennsylvania (and dual-majoring in the Wharton\nSchool of Business). I started to develop professionally when I took a\nyear off between high school and college to work for a small firm in\nSilicon Valley, California developing diagnostic imaging software for\nquality assurance and research on their products. Last summer I\ndeveloped an Ruby on Rails-based engine to test user-designed\ninvestment strategies for QED Benchmark, a boutique hedge fund.\n\nThanks,\nPhill Baker\n"},{"id":"109914","messageId":"20090330153245.GD23521@spearce.org","threadId":"18545","inReplyTo":"526944450903251314o622711b5u3665bac90398d6be@mail.gmail.com","subject":"Re: [RFC GSoC 2009: git-submodule for multiple, active developers on active trees]","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-03-30T15:32:45Z","receivedAt":"2009-03-30T15:32:45Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Hi!  As someone who has coded around git submodule by creating \"repo\"\nfor Android, I'm certainly interested in git submodule improvements,\nso this is a great idea for a GSoC project.  I have some comments\nbelow that may help improve your proposal before you submit it.\n\nP Baker <me@retrodict.com> wrote:\n> Abstract:\n> \tThis project focuses on upgrading git-submodule to manage code\n> created in external projects in ways that allow users to safely branch\n> and merge that code without loss of data or routine merge conflicts.\n> This will incorporate some changes made on the ???pu??? branch, but will\n> also include making substantial changes to git-submodule underlying\n> code.\n...\n> As opposed to ???remotes,??? which also incorporate external code into a\n> project,\n\nI'm not sure what you mean by that.  Typically a \"remote\" in Git is\nthought to be a configuration that says where to download a fork of\nthis project from.  By default you get one remote, called \"origin\",\nwhich is where you initially cloned your fork from, but you can add\nmany more, such as other developers you frequently collaborate with.\n\nThis is quite different from the problem that submodule tries\nto address, as its dealing with forks of of the *same* project.\nBut a submodule is trying to point to forks of *other* projects,\nwhose histories are (possibly) unrelated to this project's history.\n\n> The tentative timeline is:\n> \n> End of May ??? Conclusively finish the public discussion regarding where\n> git-submodules needs to go\n> Beginning of June ??? Produce final specifications (including method stubs)\n> Middle of July ??? Finish active code and test development\n> End of July ??? Merge code into production release, fix public submitted bugs\n> Middle of August ??? Prepare code for final release and finish\n> user-facing documentation\n\nIMHO, this is too vague.  *What exactly* are the features you want\nto add to git submodule?  Break this down by features, not by phases\nof coding.\n\nFurther, you spend roughly a month writing method stubs.\nMy experience with such development practices is that you will\nget frustrated by not having the code working, get bored with it,\nand walk away.  Or at best, you'll be able to stub it all out,\nbut will need to redo most of the stubs because you find later on\nwhile writing the implementation code that you need to pass data\nthrough that you didn't initially anticipate.\n\nAlso, we very much prefer Git patches to update the documentation\nat the same time that the code changes.  Maybe its done in the same\npatch, if the code+doc update are relatively small, or maybe its done\nin two patches in the same series (code change, then doc update),\nbut the general guideline is that both code and documentation should\nbe updated at roughly the same time (e.g. same day for Junio when\nhe merges the series down to master).  This way the documentation\ndoesn't stray too far from the code its describing.\n\n> A few specific changes that this project will likely include are:\n> \n> *use .git instead of .gitmodules\n> *move objects of submodules into .git/ directory\n> *git submodule update --init should initialize nested levels of submodules\n> *protect changes in local submodules when doing ???git submodule update???\n\nAs I said above, I'd like to see this described in the timeline\nbetter, each of these could be done independently, so you could work\non one item try to get it completed, tested, documented, and merged\ninto Junio's tree, and then start the next item.  At worst at the\nend of the summer you'll have a fraction of these done, merged,\nand available for users, which is better than trying to do it all\nand failing to get none merged.\n\nI'd like to know more about each of these items, and less about\nthe general reasoning of where you got these feature ideas from.\nWhat exactly are you talking about changing, and why?  I don't need\nto see detailed code at this stage, but I'd like a better description\nof the user-visible changes that each bullet point might cause,\nand why you feel this change is better than what we have today.\n \n-- \nShawn.\n"},{"id":"110016","messageId":"526944450903310830q5f56fe82xb64ae8dc3c954ffb@mail.gmail.com","threadId":"18545","inReplyTo":"20090330153245.GD23521@spearce.org","subject":"Re: [RFC GSoC 2009: git-submodule for multiple, active developers on active trees]","fromName":"P Baker","fromEmail":"me@retrodict.com","sentAt":"2009-03-31T15:30:40Z","receivedAt":"2009-03-31T15:30:40Z","isPatch":false,"sender":{"key":"me@retrodict.com","avatar":null},"body":"Great, thanks Shawn. I've included my responses to your\nquestions/comments below, and would love to continue the dialog - any\nhelp with the application would be much appreciated. I'll work my\nresponses into my application and send it out again.\n\nOn 3/30/09, Shawn O. Pearce <spearce@spearce.org> wrote:\n> Hi!  As someone who has coded around git submodule by creating \"repo\"\n>  for Android, I'm certainly interested in git submodule improvements,\n>  so this is a great idea for a GSoC project.  I have some comments\n>  below that may help improve your proposal before you submit it.\n>\n\nI was going to send out another request for comments to the list, but\nyou beat me to it. The first part of my proposal is to get community\ninvolvement, is this mailing list the best place to do that, or\nsomething external, like a publicly editable site (e.g. a survey or\nwiki)?\n\n>\n> > As opposed to ???remotes,??? which also incorporate external code into a\n>  > project,\n>\n>\n> I'm not sure what you mean by that.  Typically a \"remote\" in Git is\n>  thought to be a configuration that says where to download a fork of\n>  this project from.  By default you get one remote, called \"origin\",\n>  which is where you initially cloned your fork from, but you can add\n>  many more, such as other developers you frequently collaborate with.\n>\n>  This is quite different from the problem that submodule tries\n>  to address, as its dealing with forks of of the *same* project.\n>  But a submodule is trying to point to forks of *other* projects,\n>  whose histories are (possibly) unrelated to this project's history.\n>\n\nRight. My point is that due to either perceived or real problems with\ngit-submodules, some folks have abandoned it and instead used\nworkarounds (see\nhttp://flavoriffic.blogspot.com/2008/05/managing-git-submodules-with-gitrake.html?showComment=1210125780000#c1605130977296198852\nfor one example of such a comment). That so much effort has been put\ninto solutions external to git, shows that there is some pent up\ndemand for a solution built into git.\n\n>\n>  > The tentative timeline is:\n>  >\n>  > End of May ??? Conclusively finish the public discussion regarding where\n>  > git-submodules needs to go\n>  > Beginning of June ??? Produce final specifications (including method stubs)\n>  > Middle of July ??? Finish active code and test development\n>  > End of July ??? Merge code into production release, fix public submitted bugs\n>  > Middle of August ??? Prepare code for final release and finish\n>  > user-facing documentation\n>\n>\n> IMHO, this is too vague.  *What exactly* are the features you want\n>  to add to git submodule?  Break this down by features, not by phases\n>  of coding.\n>\n\nThe features I have been considering are:\n*move objects of submodules into base .git/ directory\n**This would, as I understand it: protect submodules from being\noverwritten and changes lost when switching between branches of the\nsuperproject that might or might not contain the submodules and\ncentralize their management into one location.  The added benefits of\nfully using git's ability to branch and merge submodules makes it\nworth adding some complexity within the .git directory.\n\n*use .git instead of .gitmodules\n**I actually don't know why this was included with the project\ndescription, I searched for an explanation of the desired name change\non the mailing list and in commit messages, but came up with nothing.\n\n*git submodule update --init should initialize nested levels of submodules\n**As an ease of use command, either an additional flag to recurse can\nbe added, or it can act by default. As a requested feature on the\nmailing list, this is worth implementing.\n\n*ability to update submodule pulled from svn repo\n**One workaround is to clone it as local copy using git-svn and then\nimport that local clone as a submodule; clearly a clunky solution.\nThere are many requests for this feature (see\nhttp://panthersoftware.com/articles/view/4/git-svn-dcommit-workaround-for-git-submodules\nfor a typical example), and it makes sense integrating git-submodule\nwith git-svn would expand submodule's usefulness.\n\n*make submodules deal with updated references\n**Instead of issuing merge conflicts on updated submodule references,\nthis will allow submodules on default detached HEAD so that changes\nfrom the local repo can be committed without first pulling changes\nfrom the shared repo. See\nhttp://flavoriffic.blogspot.com/2008/05/managing-git-submodules-with-gitrake.html?showComment=1211380200000#c3897235118548537475\nfor an explanation of how this made 'submodules...unsuitable for\nactive development'. Clearly losing this kind of functionality impairs\nthe overall usability of git and should be fixed.\n\n*protect changes in local submodules when doing “git submodule update”\n**This is similar to the previous point, in that changes need to be\nprotected or merged or warnings issued when updating the submodule.\nThe potential to lose work with no warning is a big no-no.\n\n*make git submodules easy to remove\n** See http://pitupepito.homelinux.org/?p=24, for an example of why\nthis is a pain. Adding a submodule has ui, removing one should as\nwell.\n\n>  Further, you spend roughly a month writing method stubs.\n\nWeek max: end of May to beginning of June, but if I\n\n>\n>  Also, we very much prefer Git patches to update the documentation\n>  at the same time that the code changes.\n\nFair enough. I was planning on starting documentation during the\nstubbing phase, and then finishing it once having written the code.\n\n>\n>\n>  > A few specific changes that this project will likely include are:\n>  >\n>  > *use .git instead of .gitmodules\n>  > *move objects of submodules into .git/ directory\n>  > *git submodule update --init should initialize nested levels of submodules\n>\n> > *protect changes in local submodules when doing ???git submodule update???\n>\n>  As I said above, I'd like to see this described in the timeline\n>  better, each of these could be done independently, so you could work\n>  on one item try to get it completed, tested, documented, and merged\n>  into Junio's tree, and then start the next item.\n\nOk, makes sense. I can redo the timeline like that, I'll order them by\nmy priority and look for community input on re-ranking them and adding\nor subtracting features.\n\n\n>\n>  I'd like to know more about each of these items, and less about\n>  the general reasoning of where you got these feature ideas from.\n>  What exactly are you talking about changing, and why? I don't need\n>  to see detailed code at this stage, but I'd like a better description\n>  of the user-visible changes that each bullet point might cause,\n>  and why you feel this change is better than what we have today.\n>\n\nSee above.\n\nPhill Baker\n"},{"id":"110022","messageId":"alpine.DEB.1.00.0903311749160.7052@intel-tinevez-2-302","threadId":"18545","inReplyTo":"526944450903310830q5f56fe82xb64ae8dc3c954ffb@mail.gmail.com","subject":"Re: [RFC GSoC 2009: git-submodule for multiple, active developers on active trees]","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2009-03-31T15:57:27Z","receivedAt":"2009-03-31T15:57:27Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nI am a rather intense user of submodules, so I am quite interested.\n\nPlease take my comments as encouragement rather than discouragement.\n\nOn Tue, 31 Mar 2009, P Baker wrote:\n\n> On 3/30/09, Shawn O. Pearce <spearce@spearce.org> wrote:\n> > IMHO, this is too vague.  *What exactly* are the features you want to \n> > add to git submodule?  Break this down by features, not by phases of \n> > coding.\n> \n> The features I have been considering are:\n> *move objects of submodules into base .git/ directory\n> **This would, as I understand it: protect submodules from being\n> overwritten and changes lost when switching between branches of the\n> superproject that might or might not contain the submodules and\n> centralize their management into one location.  The added benefits of\n> fully using git's ability to branch and merge submodules makes it\n> worth adding some complexity within the .git directory.\n\nThe main problem with renaming/deleting is not the repository of the \nsubmodule, but the working directoy.\n\n> *use .git instead of .gitmodules\n> **I actually don't know why this was included with the project\n> description, I searched for an explanation of the desired name change\n> on the mailing list and in commit messages, but came up with nothing.\n\nAFAICT somebody thought that the information about the locations of the \nsubmodules should be in .git/ rather than in the working directory.  But \nof course, that is wrong: you want it to be tracked.\n\n> *git submodule update --init should initialize nested levels of submodules\n> **As an ease of use command, either an additional flag to recurse can\n> be added, or it can act by default. As a requested feature on the\n> mailing list, this is worth implementing.\n\nI thought there was a patch to support \"git submodule recurse\"?  That \nwould be rather less limited than yet another option to submodule update.\n\n> *ability to update submodule pulled from svn repo\n> **One workaround is to clone it as local copy using git-svn and then\n> import that local clone as a submodule; clearly a clunky solution.\n> There are many requests for this feature (see\n> http://panthersoftware.com/articles/view/4/git-svn-dcommit-workaround-for-git-submodules\n> for a typical example), and it makes sense integrating git-submodule\n> with git-svn would expand submodule's usefulness.\n\nI do not think that this would be good.  Both \"git svn\" and \"git \nsubmodule\" are rather complex by now, and mixing them would only \ncomplicate code.\n\n> *make submodules deal with updated references\n> **Instead of issuing merge conflicts on updated submodule references,\n> this will allow submodules on default detached HEAD so that changes\n> from the local repo can be committed without first pulling changes\n> from the shared repo.\n\nI'd rather call this \"make git-submodule help with merging\".\n\n> *protect changes in local submodules when doing “git submodule update”\n> **This is similar to the previous point, in that changes need to be\n> protected or merged or warnings issued when updating the submodule.\n> The potential to lose work with no warning is a big no-no.\n\nOne word: Reflogs.\n\n> *make git submodules easy to remove\n> ** See http://pitupepito.homelinux.org/?p=24, for an example of why\n> this is a pain. Adding a submodule has ui, removing one should as\n> well.\n\nAFAIR there was already a patch to implement this, but the OP apparently \ndid not address all issues.\n\n> >  Further, you spend roughly a month writing method stubs.\n> \n> Week max: end of May to beginning of June, but if I\n\n... yes?\n\nCiao,\nDscho\n"},{"id":"110076","messageId":"526944450903311532u24eb74fby1f558c1bef5c653a@mail.gmail.com","threadId":"18545","inReplyTo":"alpine.DEB.1.00.0903311749160.7052@intel-tinevez-2-302","subject":"Re: [RFC GSoC 2009: git-submodule for multiple, active developers on active trees]","fromName":"P Baker","fromEmail":"me@retrodict.com","sentAt":"2009-03-31T22:32:00Z","receivedAt":"2009-03-31T22:32:00Z","isPatch":false,"sender":{"key":"me@retrodict.com","avatar":null},"body":"Thanks for the comments, my replies are included. It's good to see\nsome core folks are big users!\n\nOn Tue, Mar 31, 2009 at 11:57 AM, Johannes Schindelin\n<Johannes.Schindelin@gmx.de> wrote:\n> Hi,\n>\n> I am a rather intense user of submodules, so I am quite interested.\n>\n\nI'm curious, where/under what circumstances do you tend to use it?\n\n> Please take my comments as encouragement rather than discouragement.\n>\n\nAlways.\n\n>> *move objects of submodules into base .git/ directory\n>> **This would, as I understand it: protect submodules from being\n>> overwritten and changes lost when switching between branches of the\n>> superproject that might or might not contain the submodules and\n>> centralize their management into one location.  The added benefits of\n>> fully using git's ability to branch and merge submodules makes it\n>> worth adding some complexity within the .git directory.\n>\n> The main problem with renaming/deleting is not the repository of the\n> submodule, but the working directoy.\n>\n\nMy understanding is that since the submodule objects (history) is\nstored in a .git directory in the subdirectory where the submodule is\nlocated, removing that subdirectory during checkout of a branch that\ndoes not include that submodule eliminates the .git directory as well.\nMoving the objects from the submodule's .git directory to the base\n.git directory would seem to alleviate this problem.\n\n>> *use .git instead of .gitmodules\n>> **I actually don't know why this was included with the project\n>> description, I searched for an explanation of the desired name change\n>> on the mailing list and in commit messages, but came up with nothing.\n>\n> AFAICT somebody thought that the information about the locations of the\n> submodules should be in .git/ rather than in the working directory.  But\n> of course, that is wrong: you want it to be tracked.\n>\n\nSo, in looking back through the archives of the mailing list there\nseems to be some disagreement between using .gitmodules and\n.git/config to track submodules.\n\n>> *git submodule update --init should initialize nested levels of submodules\n>> **As an ease of use command, either an additional flag to recurse can\n>> be added, or it can act by default. As a requested feature on the\n>> mailing list, this is worth implementing.\n>\n> I thought there was a patch to support \"git submodule recurse\"?  That\n> would be rather less limited than yet another option to submodule update.\n>\n\nThere is a git submodule foreach command, but it doesn't look like the\npatch for git submodule recurse\n(http://marc.info/?l=git&m=120997867213008&w=2) has been incorporated\ninto a public release.\n\nThat is one route, on the other hand, the default action is also open\nto question. When I update a submodule, I would probably expect that\nanything it depends on is also updated. The default action probably\nshould be recursive.\n\n>> *ability to update submodule pulled from svn repo\n>> **One workaround is to clone it as local copy using git-svn and then\n>> import that local clone as a submodule; clearly a clunky solution.\n>> There are many requests for this feature (see\n>> http://panthersoftware.com/articles/view/4/git-svn-dcommit-workaround-for-git-submodules\n>> for a typical example), and it makes sense integrating git-submodule\n>> with git-svn would expand submodule's usefulness.\n>\n> I do not think that this would be good.  Both \"git svn\" and \"git\n> submodule\" are rather complex by now, and mixing them would only\n> complicate code.\n>\n\nHm, point well taken, but it would seem to have enormous benefit for a\nlot of people. I can move it down the priority list, but I'd like to\ninclude it in the proposal - complexity alone isn't a good reason to\navoid something.\n\nI also believe that the workaround described, if incorporated into\ngit-submodules.sh in an appropriate way might open up possibilities\nfor further improvement. The UI would change much, seems like it'd\njust be detection of pointing to a SVN repo instead of a git repo and\nthen hooking into git svn calls instead of regular git calls. This\nbrings up the possibility that git submodules should abstract its\nrepository handling in much the same way that git does. I'm not\nfamiliar with the code, but this seems more like calling other\nplumbing hooks than anything else.\n\n>> *make submodules deal with updated references\n>> **Instead of issuing merge conflicts on updated submodule references,\n>> this will allow submodules on default detached HEAD so that changes\n>> from the local repo can be committed without first pulling changes\n>> from the shared repo.\n>\n> I'd rather call this \"make git-submodule help with merging\".\n>\n\nBetter name. Duly noted. Will change.\n\n>> *protect changes in local submodules when doing “git submodule update”\n>> **This is similar to the previous point, in that changes need to be\n>> protected or merged or warnings issued when updating the submodule.\n>> The potential to lose work with no warning is a big no-no.\n>\n> One word: Reflogs.\n>\n\nI  haven't used reflogs, but it doesn't seem to fix the problem (maybe\nyou can explain?): simply knowing where/what the reference is, doesn't\nmean that git-submodule looks at it, obeys the reference or issues\nwarnings when it should. The problem as stated\n(http://flavoriffic.blogspot.com/2008/05/managing-git-submodules-with-gitrake.html?showComment=1211380200000#c3897235118548537475)\nwas that git submodule update would silently overwrite any local\nchanges with the remote version (i.e. git did not check to see if the\nlocal reference was different than the remote reference when\nupdating).\n\n>> *make git submodules easy to remove\n>> ** See http://pitupepito.homelinux.org/?p=24, for an example of why\n>> this is a pain. Adding a submodule has ui, removing one should as\n>> well.\n>\n> AFAIR there was already a patch to implement this, but the OP apparently\n> did not address all issues.\n>\n\nYep, found it on the mailing list. Obviously, part of the project\nwould be to resolve those final issues.\n\nPhillip Baker\n"},{"id":"110077","messageId":"alpine.DEB.1.00.0904010058490.6616@intel-tinevez-2-302","threadId":"18545","inReplyTo":"526944450903311532u24eb74fby1f558c1bef5c653a@mail.gmail.com","subject":"Re: [RFC GSoC 2009: git-submodule for multiple, active developers on active trees]","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2009-03-31T23:05:07Z","receivedAt":"2009-03-31T23:05:07Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Tue, 31 Mar 2009, P Baker wrote:\n\n> On Tue, Mar 31, 2009 at 11:57 AM, Johannes Schindelin\n> <Johannes.Schindelin@gmx.de> wrote:\n> \n> >> *move objects of submodules into base .git/ directory\n> >> **This would, as I understand it: protect submodules from being\n> >> overwritten and changes lost when switching between branches of the\n> >> superproject that might or might not contain the submodules and\n> >> centralize their management into one location.  The added benefits of\n> >> fully using git's ability to branch and merge submodules makes it\n> >> worth adding some complexity within the .git directory.\n> >\n> > The main problem with renaming/deleting is not the repository of the\n> > submodule, but the working directoy.\n> >\n> \n> My understanding is that since the submodule objects (history) is\n> stored in a .git directory in the subdirectory where the submodule is\n> located, removing that subdirectory during checkout of a branch that\n> does not include that submodule eliminates the .git directory as well.\n> Moving the objects from the submodule's .git directory to the base\n> .git directory would seem to alleviate this problem.\n\nMy point was more about \"you cannot just remove the subdirectory, or you \n_will_ lose data\".\n\n> >> *use .git instead of .gitmodules\n> >> **I actually don't know why this was included with the project\n> >> description, I searched for an explanation of the desired name change\n> >> on the mailing list and in commit messages, but came up with nothing.\n> >\n> > AFAICT somebody thought that the information about the locations of the\n> > submodules should be in .git/ rather than in the working directory.  But\n> > of course, that is wrong: you want it to be tracked.\n> \n> So, in looking back through the archives of the mailing list there\n> seems to be some disagreement between using .gitmodules and\n> .git/config to track submodules.\n\nNo.  .gitmodules has the default information, and \"git submodule init\" \nbrings that into .git/config, to be overridden by the user if she so \nlikes.\n\n> >> *git submodule update --init should initialize nested levels of submodules\n> >> **As an ease of use command, either an additional flag to recurse can\n> >> be added, or it can act by default. As a requested feature on the\n> >> mailing list, this is worth implementing.\n> >\n> > I thought there was a patch to support \"git submodule recurse\"?  That\n> > would be rather less limited than yet another option to submodule update.\n> \n> There is a git submodule foreach command, but it doesn't look like the\n> patch for git submodule recurse\n> (http://marc.info/?l=git&m=120997867213008&w=2) has been incorporated\n> into a public release.\n> \n> That is one route, on the other hand, the default action is also open\n> to question. When I update a submodule, I would probably expect that\n> anything it depends on is also updated. The default action probably\n> should be recursive.\n\nNo.  Not at all.  At least in my usage, submodules are mostly optional.  \nIOW I have ways in my projects to cope with the absence of a checkout.\n\n> >> *ability to update submodule pulled from svn repo\n> >> **One workaround is to clone it as local copy using git-svn and then\n> >> import that local clone as a submodule; clearly a clunky solution.\n> >> There are many requests for this feature (see\n> >> http://panthersoftware.com/articles/view/4/git-svn-dcommit-workaround-for-git-submodules\n> >> for a typical example), and it makes sense integrating git-submodule\n> >> with git-svn would expand submodule's usefulness.\n> >\n> > I do not think that this would be good.  Both \"git svn\" and \"git\n> > submodule\" are rather complex by now, and mixing them would only\n> > complicate code.\n> \n> Hm, point well taken, but it would seem to have enormous benefit for a\n> lot of people. I can move it down the priority list, but I'd like to\n> include it in the proposal - complexity alone isn't a good reason to\n> avoid something.\n\nComplexity is often a good sign of bad design.\n\nIn this case, I want to point out that there has been a better design \nalready:\n\nhttp://thread.gmane.org/gmane.comp.version-control.git/114545\n\n(Unfortunately, Daniel decided to post the follow-up patches in different \nthreads; that will make it hard for you to find them.)\n\nCiao,\nDscho\n"},{"id":"110080","messageId":"526944450903311649q358d43edkf07e2e5058a9e527@mail.gmail.com","threadId":"18545","inReplyTo":"alpine.DEB.1.00.0904010058490.6616@intel-tinevez-2-302","subject":"Re: [RFC GSoC 2009: git-submodule for multiple, active developers on active trees]","fromName":"P Baker","fromEmail":"me@retrodict.com","sentAt":"2009-03-31T23:49:25Z","receivedAt":"2009-03-31T23:49:25Z","isPatch":false,"sender":{"key":"me@retrodict.com","avatar":null},"body":"I'll paraphrase to see if I understand your points:\n\n*Moving objects from submodule .git directories into the base .git/\ndirectory would protect the submodules and is a good idea.\n*Moving to a .git/ file from .gitmodules should be taken off of the\ngoal list (I went back and read this thread:\nhttp://thread.gmane.org/gmane.comp.version-control.git/78605; seemed\nto clear things up).\n*git submodule recurse would be a good option (not as a default), if\nthe remaining issues are resolved.\n*It would be a good idea for git submodule to work with foreign VCS,\nthrough Daniel's patches.\n\nI appreciate the guidance, it's helping me to see that some of this\nwork has already been done, it needs to be finished and pushed into a\npublic release. As an intense user of submodules, what does it do\npoorly/not do for your needs?\n\nThanks,\n\nPhillip Baker\n\nOn Tue, Mar 31, 2009 at 7:05 PM, Johannes Schindelin\n<Johannes.Schindelin@gmx.de> wrote:\n> Hi,\n>\n> On Tue, 31 Mar 2009, P Baker wrote:\n>\n>> On Tue, Mar 31, 2009 at 11:57 AM, Johannes Schindelin\n>> <Johannes.Schindelin@gmx.de> wrote:\n>>\n>> >> *move objects of submodules into base .git/ directory\n>> >> **This would, as I understand it: protect submodules from being\n>> >> overwritten and changes lost when switching between branches of the\n>> >> superproject that might or might not contain the submodules and\n>> >> centralize their management into one location.  The added benefits of\n>> >> fully using git's ability to branch and merge submodules makes it\n>> >> worth adding some complexity within the .git directory.\n>> >\n>> > The main problem with renaming/deleting is not the repository of the\n>> > submodule, but the working directoy.\n>> >\n>>\n>> My understanding is that since the submodule objects (history) is\n>> stored in a .git directory in the subdirectory where the submodule is\n>> located, removing that subdirectory during checkout of a branch that\n>> does not include that submodule eliminates the .git directory as well.\n>> Moving the objects from the submodule's .git directory to the base\n>> .git directory would seem to alleviate this problem.\n>\n> My point was more about \"you cannot just remove the subdirectory, or you\n> _will_ lose data\".\n>\n>> >> *use .git instead of .gitmodules\n>> >> **I actually don't know why this was included with the project\n>> >> description, I searched for an explanation of the desired name change\n>> >> on the mailing list and in commit messages, but came up with nothing.\n>> >\n>> > AFAICT somebody thought that the information about the locations of the\n>> > submodules should be in .git/ rather than in the working directory.  But\n>> > of course, that is wrong: you want it to be tracked.\n>>\n>> So, in looking back through the archives of the mailing list there\n>> seems to be some disagreement between using .gitmodules and\n>> .git/config to track submodules.\n>\n> No.  .gitmodules has the default information, and \"git submodule init\"\n> brings that into .git/config, to be overridden by the user if she so\n> likes.\n>\n>> >> *git submodule update --init should initialize nested levels of submodules\n>> >> **As an ease of use command, either an additional flag to recurse can\n>> >> be added, or it can act by default. As a requested feature on the\n>> >> mailing list, this is worth implementing.\n>> >\n>> > I thought there was a patch to support \"git submodule recurse\"?  That\n>> > would be rather less limited than yet another option to submodule update.\n>>\n>> There is a git submodule foreach command, but it doesn't look like the\n>> patch for git submodule recurse\n>> (http://marc.info/?l=git&m=120997867213008&w=2) has been incorporated\n>> into a public release.\n>>\n>> That is one route, on the other hand, the default action is also open\n>> to question. When I update a submodule, I would probably expect that\n>> anything it depends on is also updated. The default action probably\n>> should be recursive.\n>\n> No.  Not at all.  At least in my usage, submodules are mostly optional.\n> IOW I have ways in my projects to cope with the absence of a checkout.\n>\n>> >> *ability to update submodule pulled from svn repo\n>> >> **One workaround is to clone it as local copy using git-svn and then\n>> >> import that local clone as a submodule; clearly a clunky solution.\n>> >> There are many requests for this feature (see\n>> >> http://panthersoftware.com/articles/view/4/git-svn-dcommit-workaround-for-git-submodules\n>> >> for a typical example), and it makes sense integrating git-submodule\n>> >> with git-svn would expand submodule's usefulness.\n>> >\n>> > I do not think that this would be good.  Both \"git svn\" and \"git\n>> > submodule\" are rather complex by now, and mixing them would only\n>> > complicate code.\n>>\n>> Hm, point well taken, but it would seem to have enormous benefit for a\n>> lot of people. I can move it down the priority list, but I'd like to\n>> include it in the proposal - complexity alone isn't a good reason to\n>> avoid something.\n>\n> Complexity is often a good sign of bad design.\n>\n> In this case, I want to point out that there has been a better design\n> already:\n>\n> http://thread.gmane.org/gmane.comp.version-control.git/114545\n>\n> (Unfortunately, Daniel decided to post the follow-up patches in different\n> threads; that will make it hard for you to find them.)\n>\n> Ciao,\n> Dscho\n>\n>\n"},{"id":"110085","messageId":"alpine.DEB.1.00.0904010247170.10279@pacific.mpi-cbg.de","threadId":"18545","inReplyTo":"526944450903311649q358d43edkf07e2e5058a9e527@mail.gmail.com","subject":"Re: [RFC GSoC 2009: git-submodule for multiple, active developers on active trees]","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2009-04-01T00:58:31Z","receivedAt":"2009-04-01T00:58:31Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Tue, 31 Mar 2009, P Baker wrote:\n\n> I'll paraphrase to see if I understand your points:\n> \n> *Moving objects from submodule .git directories into the base .git/\n> directory would protect the submodules and is a good idea.\n\nNo, I did not say that.\n\nI said that moving submodules' working directory need to protected when \nrenaming/deleting submodules.\n\nEven worse, I think that moving the .git/ directory into the \nsuperproject's .git/ would be at least quite a bit awkward in the nested \ncase.\n\n> *Moving to a .git/ file from .gitmodules should be taken off of the\n> goal list (I went back and read this thread:\n> http://thread.gmane.org/gmane.comp.version-control.git/78605; seemed\n> to clear things up).\n\nCan't follow links here, as I am reading this offline, so cannot comment.\n\n> *git submodule recurse would be a good option (not as a default), if\n> the remaining issues are resolved.\n\nDefinitely.\n\n> *It would be a good idea for git submodule to work with foreign VCS,\n> through Daniel's patches.\n\nBut that would not only apply to submodules, but rather all repositories, \nto the point that \"git submodule\" does not need any change.\n\n> I appreciate the guidance, it's helping me to see that some of this work \n> has already been done, it needs to be finished and pushed into a public \n> release. As an intense user of submodules, what does it do poorly/not do \n> for your needs?\n\nOne gripe I have, but which should be rather easy to fix: \"git checkout -- \nsubmodule/\" does not update the index, last time I checked.  (It correctly \ndoes not touch the submodule's working directory.)\n\nAnother one: The most common mistake with submodules is to commit and push \nthe superproject, after having committed (but not pushed) in the \nsubmodule.  Not sure how that could be helped.\n\nFurther, often it would come in rather handy to be able to say something \nlike \"git diff $REVISION_AS_COMMITTED_IN_THE_SUPERPROJECT\" from within \nthe submodule...\n\ngit submodule summary should output to the pager by default.\n\nOh, and it would not hurt performance on Windows at all if git-submodule \nwould be finally made a builtin.\n\nCiao,\nDscho\n"},{"id":"110087","messageId":"526944450903311947w2f398c71n95a4a7aa47ecdb7f@mail.gmail.com","threadId":"18545","inReplyTo":"alpine.DEB.1.00.0904010247170.10279@pacific.mpi-cbg.de","subject":"Re: [RFC GSoC 2009: git-submodule for multiple, active developers on active trees]","fromName":"P Baker","fromEmail":"me@retrodict.com","sentAt":"2009-04-01T02:47:36Z","receivedAt":"2009-04-01T02:47:36Z","isPatch":false,"sender":{"key":"me@retrodict.com","avatar":null},"body":">  > *Moving objects from submodule .git directories into the base .git/\n>  > directory would protect the submodules and is a good idea.\n>\n>\n> No, I did not say that.\n>\n>  Even worse, I think that moving the .git/ directory into the\n>  superproject's .git/ would be at least quite a bit awkward in the nested\n>  case.\n>\n\nTthe initial prompt for the proposal was: \"Rewrite git-submodule,\nplacing the repository for each referenced submodules in the\nsuperproject's $GIT_DIR/modules...This resolves issues related to\nswitching between versions of the superproject...\" The prompt, and\npast experience with git, helped me to form my proposal which it seems\nwould fix numerous problems with git submodule, with the implied cost\nof some awkwardness/complexity. Am I misunderstanding the prompt? Or\ndo you think this could be accomplished more elegantly?\n\n>  I said that moving submodules' working directory need to protected when\n>  renaming/deleting submodules.\n\nI'm sorry, I still don't understand. Where would this occur? What is\nbeing protected? What is the submodules' working directory? I'm still\nlearning the intricacies of git, so I'd appreciate any pointers you\ncan give.\n\n>\n>\n>  > *It would be a good idea for git submodule to work with foreign VCS,\n>  > through Daniel's patches.\n>\n>\n> But that would not only apply to submodules, but rather all repositories,\n>  to the point that \"git submodule\" does not need any change.\n>\n>\n\nFair enough. There's plenty of other work to be done!\n\n>  > I appreciate the guidance, it's helping me to see that some of this work\n>  > has already been done, it needs to be finished and pushed into a public\n>  > release. As an intense user of submodules, what does it do poorly/not do\n>  > for your needs?\n>\n>\n> One gripe I have, but which should be rather easy to fix: \"git checkout --\n>  submodule/\" does not update the index, last time I checked.  (It correctly\n>  does not touch the submodule's working directory.)\n>\n\nI'll add it to the list. In terms of general gripes: git submodule add\n(or all of git submodule?) handles relative links poorly (see\nhttp://kerneltrap.org/mailarchive/git/2007/12/10/485597). And the\n'Gotchas' listed at\nhttp://git.or.cz/gitwiki/GitSubmoduleTutorial#head-a3cba9cbd1e125c0667dfb3b9249100be7f815ad.\n\n>  Another one: The most common mistake with submodules is to commit and push\n>  the superproject, after having committed (but not pushed) in the\n>  submodule.  Not sure how that could be helped.\n>\n\nSeems like this is on the git submodule wiki 'Gotcha' list, too.\nThere's a spectrum of options: failing, warning, generating an output\nmessage, etc. I think it is worth working on. What is git's policy on\ninterrupting users when their actions _could_ be counterproductive to\ntheir intentions? Would hooks on the submodule's commit written by the\nuser fix this? That's not a built-in solution.\n\n>  Further, often it would come in rather handy to be able to say something\n>  like \"git diff $REVISION_AS_COMMITTED_IN_THE_SUPERPROJECT\" from within\n>  the submodule...\n>\n\nThat sounds complex, and would break expectations. This would only\nwork if git in the submodule working directory knows its a submodule.\nIs there a way to reference it's super project?\n\n>  git submodule summary should output to the pager by default.\n>\nAdded to the list.\n\n>  Oh, and it would not hurt performance on Windows at all if git-submodule\n>  would be finally made a builtin.\n\nYou mean rewriting git-submodule.sh in C? What other impacts might that have?\n\nThanks,\n\nPhillip Baker\n\n>  Ciao,\n>  Dscho\n>\n>\n"},{"id":"110090","messageId":"49D30907.6090606@op5.se","threadId":"18545","inReplyTo":"alpine.DEB.1.00.0904010247170.10279@pacific.mpi-cbg.de","subject":"Re: [RFC GSoC 2009: git-submodule for multiple, active developers on active trees]","fromName":"Andreas Ericsson","fromEmail":"ae@op5.se","sentAt":"2009-04-01T06:26:15Z","receivedAt":"2009-04-01T06:26:15Z","isPatch":false,"sender":{"key":"ae@op5.se","avatar":"https://gravatar.com/avatar/426e89595c75a8f5252dd0c989e5fabe5bcac616e68557427ad9aef6b0ca342a?d=mp&s=160"},"body":"Johannes Schindelin wrote:\n> Hi,\n> \n> On Tue, 31 Mar 2009, P Baker wrote:\n> \n>> I'll paraphrase to see if I understand your points:\n>>\n>> *Moving objects from submodule .git directories into the base .git/\n>> directory would protect the submodules and is a good idea.\n> \n> No, I did not say that.\n> \n> I said that moving submodules' working directory need to protected when \n> renaming/deleting submodules.\n> \n> Even worse, I think that moving the .git/ directory into the \n> superproject's .git/ would be at least quite a bit awkward in the nested \n> case.\n> \n\nNot necessarily. The .git directory of a submodule need not be named .git\ninside the superprojects .git directory. I could well imagine something\nlike this:\n\n.git/modules/submod(.git)/modules/nested-submod(.git)\n\nFor deeply nested submodules (eurgh), one might run into path length limit\nissues though. The point is that we will need some library-like function\nto find the repository of the submodule. Once that's done, the same call\nwith a different $gitdir should be able to find the nested submodule.\n\nI'm also thinking of libgit2 here, where each repository will be\nrepresented as a struct that must be passed to the various $gitdir\nsearching functions. This is necessary to allow a single program to access\nmultiple repositories, and the .git/modules scheme makes supporting\nsubmodules in the library quite trivial.\n\n-- \nAndreas Ericsson                   andreas.ericsson@op5.se\nOP5 AB                             www.op5.se\nTel: +46 8-230225                  Fax: +46 8-230231\n\nConsidering the successes of the wars on alcohol, poverty, drugs and\nterror, I think we should give some serious thought to declaring war\non peace.\n"},{"id":"110146","messageId":"alpine.DEB.1.00.0904011806330.13502@intel-tinevez-2-302","threadId":"18545","inReplyTo":"526944450903311947w2f398c71n95a4a7aa47ecdb7f@mail.gmail.com","subject":"Re: [RFC GSoC 2009: git-submodule for multiple, active developers on active trees]","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2009-04-01T16:10:41Z","receivedAt":"2009-04-01T16:10:41Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Tue, 31 Mar 2009, P Baker wrote:\n\n> >  > *Moving objects from submodule .git directories into the base .git/ \n> >  > directory would protect the submodules and is a good idea.\n> >\n> > No, I did not say that.\n> >\n> >  Even worse, I think that moving the .git/ directory into the \n> >  superproject's .git/ would be at least quite a bit awkward in the \n> >  nested case.\n> >\n> \n> Tthe initial prompt for the proposal was: \"Rewrite git-submodule,\n> placing the repository for each referenced submodules in the\n> superproject's $GIT_DIR/modules...This resolves issues related to\n> switching between versions of the superproject...\" The prompt, and\n> past experience with git, helped me to form my proposal which it seems\n> would fix numerous problems with git submodule, with the implied cost\n> of some awkwardness/complexity. Am I misunderstanding the prompt? Or\n> do you think this could be accomplished more elegantly?\n\nWell, I think the focus here is wrong.  The focus should be on the working \ndirectory as hinted here:\n\n> >  I said that moving submodules' working directory need to protected \n> >  when renaming/deleting submodules.\n> \n> I'm sorry, I still don't understand. Where would this occur? What is \n> being protected? What is the submodules' working directory? I'm still \n> learning the intricacies of git, so I'd appreciate any pointers you can \n> give.\n\nIf your superproject deletes a submodule, what should happen with the \nworking directory?\n\nAnd what should happen if the submodule is _moved_?  (Which is not as \neasily detected as with renamed files or directories.)\n\n> >  Further, often it would come in rather handy to be able to say \n> >  something like \"git diff $REVISION_AS_COMMITTED_IN_THE_SUPERPROJECT\" \n> >  from within the submodule...\n> >\n> \n> That sounds complex, and would break expectations. This would only\n> work if git in the submodule working directory knows its a submodule.\n\n... or can detect easily that it is a submodule.\n\n> Is there a way to reference it's super project?\n\nNo.\n\n> >  Oh, and it would not hurt performance on Windows at all if \n> >  git-submodule would be finally made a builtin.\n> \n> You mean rewriting git-submodule.sh in C? What other impacts might that \n> have?\n\nJunio would hate it, I am sure.\n\nCiao,\nDscho\n"},{"id":"110148","messageId":"alpine.DEB.1.00.0904011811460.13502@intel-tinevez-2-302","threadId":"18545","inReplyTo":"49D30907.6090606@op5.se","subject":"Re: [RFC GSoC 2009: git-submodule for multiple, active developers on active trees]","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2009-04-01T16:13:57Z","receivedAt":"2009-04-01T16:13:57Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Wed, 1 Apr 2009, Andreas Ericsson wrote:\n\n> Johannes Schindelin wrote:\n> \n> > On Tue, 31 Mar 2009, P Baker wrote:\n> > \n> > > I'll paraphrase to see if I understand your points:\n> > >\n> > > *Moving objects from submodule .git directories into the base .git/\n> > > directory would protect the submodules and is a good idea.\n> > \n> > No, I did not say that.\n> > \n> > I said that moving submodules' working directory need to protected when\n> > renaming/deleting submodules.\n> > \n> > Even worse, I think that moving the .git/ directory into the superproject's\n> > .git/ would be at least quite a bit awkward in the nested case.\n> > \n> \n> Not necessarily. The .git directory of a submodule need not be named .git\n> inside the superprojects .git directory. I could well imagine something\n> like this:\n> \n> .git/modules/submod(.git)/modules/nested-submod(.git)\n> \n> For deeply nested submodules (eurgh), one might run into path length limit\n> issues though. The point is that we will need some library-like function\n> to find the repository of the submodule. Once that's done, the same call\n> with a different $gitdir should be able to find the nested submodule.\n\nIt appears to me as a solution in need of a problem.\n\n> I'm also thinking of libgit2 here, where each repository will be \n> represented as a struct that must be passed to the various $gitdir \n> searching functions. This is necessary to allow a single program to \n> access multiple repositories, and the .git/modules scheme makes \n> supporting submodules in the library quite trivial.\n\nFirst: libgit2 is not an issue for this thread AFAIAC.\n\nSecond: accessing submodules' repositories is quite trivial at the moment.  \nSo much so that git-submodule can still get away with being a shell \nscript.\n\nCiao,\nDscho\n"}]}