{"thread":{"id":"22081","subject":"RFC: display dirty submodule working directory in git gui and gitk","startedAt":"2010-01-02T15:33:22Z","lastAt":"2010-01-07T11:04:17Z","messageCount":45,"participants":["Jens Lehmann","Johannes Schindelin","Heiko Voigt","Nguyen Thai Ngoc Duy","Avery Pennarun","Junio C Hamano","Shawn O. Pearce","Johan Herland","Pau Garcia i Quiles","Nanako Shiraishi","Miles Bader"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"130686","messageId":"4B3F6742.6060402@web.de","threadId":"22081","inReplyTo":null,"subject":"RFC: display dirty submodule working directory in git gui and gitk","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2010-01-02T15:33:22Z","receivedAt":"2010-01-02T15:33:22Z","isPatch":false,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Now that we have much better output when displaying diffs of\nsubmodules in git gui and gitk (many thanks to all involved!),\nanother usability issue shows up: A dirty working directory of\na submodule isn't visible in git gui or gitk.\n\nSo you might think a \"submodule update\" would be ok - as you\nsee no changes - just too see it fail because the submodules\nworking directory is dirty.\n\nOr - even worse - you /think/ you committed your changes in\na submodule while you didn't. That can lead to 'interesting'\nproblems which can be pretty hard to diagnose (like breaking\nbuilds on other peoples machines).\n\n\nA possible solution could look like this:\n\nAFAICS, git gui and gitk use \"git diff-files\" both to get the\nfile names of unstaged local changes and to later display the\nactual differences.\n\nIf they could tell the diff core to also check the submodule\nworking directories and to output an extra line - maybe\nsomething like \"Submodule <name> contains uncommitted local\nchanges\" - when a submodules working directory is dirty,\ngit gui and gitk could show the submodules state adequately.\n\n\nWhat do you think about this approach?\n"},{"id":"130757","messageId":"alpine.DEB.1.00.1001041038520.4985@pacific.mpi-cbg.de","threadId":"22081","inReplyTo":"4B3F6742.6060402@web.de","subject":"Re: RFC: display dirty submodule working directory in git gui and gitk","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2010-01-04T09:44:55Z","receivedAt":"2010-01-04T09:44:55Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Sat, 2 Jan 2010, Jens Lehmann wrote:\n\n> Now that we have much better output when displaying diffs of submodules \n> in git gui and gitk (many thanks to all involved!), another usability \n> issue shows up: A dirty working directory of a submodule isn't visible \n> in git gui or gitk.\n> \n> So you might think a \"submodule update\" would be ok - as you see no \n> changes - just too see it fail because the submodules working directory \n> is dirty.\n> \n> Or - even worse - you /think/ you committed your changes in a submodule \n> while you didn't. That can lead to 'interesting' problems which can be \n> pretty hard to diagnose (like breaking builds on other peoples \n> machines).\n> \n> \n> A possible solution could look like this:\n> \n> AFAICS, git gui and gitk use \"git diff-files\" both to get the file names \n> of unstaged local changes and to later display the actual differences.\n> \n> If they could tell the diff core to also check the submodule working \n> directories and to output an extra line - maybe something like \n> \"Submodule <name> contains uncommitted local changes\" - when a \n> submodules working directory is dirty, git gui and gitk could show the \n> submodules state adequately.\n\nThe real problem is that submodules in the current form are not very well \ndesigned.  For example, a submodule being at a different commit than in \nthe superproject's index is not as fatal as the submodule having changes.\n\nSo in the long run, IMHO a proper redesign of the submodules would not \nmake only a little sense (it does not help, though, that those who \nimplemented and furthered the current approach over other discussed \napproaches do not use submodules themselves -- not even now).\n\nIn ths short run, we can paper over the shortcomings of the submodules by \nintroducing a command line option \"--include-submodules\" to \nupdate-refresh, diff-files and diff-index, though.\n\nThe implementation might be a bit tricky as parts of Git's source code \nstill use the_index, but at least adding the submodule's object database \nis no longer that difficult.\n\nCiao,\nDscho\n"},{"id":"130760","messageId":"61083.85.16.196.198.1262601871.squirrel@archive.darksea.de","threadId":"22081","inReplyTo":"alpine.DEB.1.00.1001041038520.4985@pacific.mpi-cbg.de","subject":"Re: RFC: display dirty submodule working directory in git gui and gitk","fromName":"Heiko Voigt","fromEmail":"hvoigt@hvoigt.net","sentAt":"2010-01-04T10:44:31Z","receivedAt":"2010-01-04T10:44:31Z","isPatch":false,"sender":{"key":"hvoigt@hvoigt.net","avatar":"https://avatars.githubusercontent.com/u/184958?v=4"},"body":"Hi,\n\nJohannes wrote:\n> The real problem is that submodules in the current form are not very well\n> designed.  For example, a submodule being at a different commit than in\n> the superproject's index is not as fatal as the submodule having changes.\n>\n> So in the long run, IMHO a proper redesign of the submodules would not\n> make only a little sense (it does not help, though, that those who\n> implemented and furthered the current approach over other discussed\n> approaches do not use submodules themselves -- not even now).\n\nDo you mean the complete workflow (submodules are links to other git repos)\nor the current implementation? Do you have links to other design\napproaches/threads? Would be nice if we could take that into account for any\ndecision.\n\ncheers Heiko\n"},{"id":"130761","messageId":"alpine.DEB.1.00.1001041157020.3695@intel-tinevez-2-302","threadId":"22081","inReplyTo":"61083.85.16.196.198.1262601871.squirrel@archive.darksea.de","subject":"submodules, was Re: RFC: display dirty submodule working directory in git gui and gitk","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2010-01-04T11:46:49Z","receivedAt":"2010-01-04T11:46:49Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Mon, 4 Jan 2010, Heiko Voigt wrote:\n\n> Johannes wrote:\n> > The real problem is that submodules in the current form are not very \n> > well designed.  For example, a submodule being at a different commit \n> > than in the superproject's index is not as fatal as the submodule \n> > having changes.\n> >\n> > So in the long run, IMHO a proper redesign of the submodules would not \n> > make only a little sense (it does not help, though, that those who \n> > implemented and furthered the current approach over other discussed \n> > approaches do not use submodules themselves -- not even now).\n> \n> Do you mean the complete workflow (submodules are links to other git \n> repos) or the current implementation? Do you have links to other design \n> approaches/threads? Would be nice if we could take that into account for \n> any decision.\n\nUnfortunately, I do not have any information about different approaches \nexcept the approach Subversion takes.  While Subversion's externals are \nnot perfect for all applications, for some, they are.  So I consider this \na serious shortcoming that Git does not support that workflow (and in \nfact, AFAIR Shawn's repo does not use submodules for that exact reason).\n\nBut I think that an important precondition to come up with a better design \nof the submodules is to have suffered the current implementation in \nreal-world work using submodules. (Which reminds me very much of the \nautocrlf mess.)\n\nCiao,\nDscho\n"},{"id":"130784","messageId":"4B421F90.4090402@web.de","threadId":"22081","inReplyTo":"alpine.DEB.1.00.1001041038520.4985@pacific.mpi-cbg.de","subject":"Re: RFC: display dirty submodule working directory in git gui and gitk","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2010-01-04T17:04:16Z","receivedAt":"2010-01-04T17:04:16Z","isPatch":false,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 04.01.2010 10:44, schrieb Johannes Schindelin:\n> The real problem is that submodules in the current form are not very well \n> designed.\n\nIMVHO using the tree sha1 for a submodule seems to be the 'natural' way\nto include another git repo. And it gives the reproducibility i expect\nfrom a scm. Or am i missing something?\n\nIt looks to me as most shortcomings come from the fact that most git\ncommands tend to ignore submodules (and if they don't, like git gui and\ngitk do now, they e.g. only show certain aspects of their state).\n\nSubmodules are in heavy use in our company since last year. Virtually\nevery patch i submitted for submodules came from that experience and\nscratched an itch i or one of my colleagues had (and the situation did\nalready improve noticeably by the few things we changed). We are still\nconvinced that using submodules was the right decision. But some work\nhas still to be done to be able to use them easily and to get rid of\nsome pitfalls.\n\n\n> In ths short run, we can paper over the shortcomings of the submodules by \n> introducing a command line option \"--include-submodules\" to \n> update-refresh, diff-files and diff-index, though.\n\nMaybe this is the way to go for now (and hopefully we can turn this\noption on by default later because we did the right thing ;-).\n"},{"id":"130785","messageId":"fcaeb9bf1001040951r3f797750o5ebd25e93c0272ea@mail.gmail.com","threadId":"22081","inReplyTo":"alpine.DEB.1.00.1001041038520.4985@pacific.mpi-cbg.de","subject":"Re: RFC: display dirty submodule working directory in git gui and gitk","fromName":"Nguyen Thai Ngoc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2010-01-04T17:51:36Z","receivedAt":"2010-01-04T17:51:36Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On 1/4/10, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:\n> The real problem is that submodules in the current form are not very well\n>  designed.  For example, a submodule being at a different commit than in\n>  the superproject's index is not as fatal as the submodule having changes.\n>\n>  So in the long run, IMHO a proper redesign of the submodules would not\n>  make only a little sense (it does not help, though, that those who\n>  implemented and furthered the current approach over other discussed\n>  approaches do not use submodules themselves -- not even now).\n>\n>  In ths short run, we can paper over the shortcomings of the submodules by\n>  introducing a command line option \"--include-submodules\" to\n>  update-refresh, diff-files and diff-index, though.\n\nIncidentally I was just drafting git-super.sh it see how far it goes.\nThe goal was to implement some cross-module operations over time. \"git\nsuper status\", \"git super commit\" and others could be handy.\n-- \nDuy\n"},{"id":"130786","messageId":"32541b131001041029t5adc535bt9681d33174042871@mail.gmail.com","threadId":"22081","inReplyTo":"alpine.DEB.1.00.1001041157020.3695@intel-tinevez-2-302","subject":"Re: submodules, was Re: RFC: display dirty submodule working directory in git gui and gitk","fromName":"Avery Pennarun","fromEmail":"apenwarr@gmail.com","sentAt":"2010-01-04T18:29:27Z","receivedAt":"2010-01-04T18:29:27Z","isPatch":false,"sender":{"key":"apenwarr@gmail.com","avatar":"https://avatars.githubusercontent.com/u/20592?v=4"},"body":"On Mon, Jan 4, 2010 at 6:46 AM, Johannes Schindelin\n<Johannes.Schindelin@gmx.de> wrote:\n> But I think that an important precondition to come up with a better design\n> of the submodules is to have suffered the current implementation in\n> real-world work using submodules. (Which reminds me very much of the\n> autocrlf mess.)\n\nI suffered the current implementation, which is why I wrote\ngit-subtree :)  I'm still suffering, though; git-subtree works much\nbetter for my own use cases, but after some experience with it, I'm\nstill not totally happy.\n\nFor me one big problem comes down to producing accurate output for\n'git log'.  git submodules assume that the history inside the module\nis entirely separate (you need to run multiple 'git log' instances to\nsee the full history); git-subtree assumes that it's entirely\nintegrated.  In that sense, git-subtree is somewhat more in line with\nthe core principle of git (we track the history of \"the content\", not\nany particular file or subdir).  Unfortunately, it also exposes a\nproblem with that core principle: taken to its extreme, \"the content\"\nincludes all data in the universe.  And while git could branch and\nmerge the universe very efficiently in about O(log n) time, 'git log'\noutput gets less useful about O(n) with the size of the tree.\n\nNeither git-subtree nor git submodules seem to help with this \"log\npollution\" problem very much - but I don't know what to do that would\nbe better.\n\nOutside of this, my major problem with submodules is they use separate\nwork trees and repositories, and thus require lots of extra\nhousekeeping to get anything done.  I'd be much happier if submodules\nwould share the same objects/packs/.gitdir/refs/indexfile as the\nsuperproject, and the *only* thing special about them would be that\nthe superproject's tree points at a commit object instead of a tree\nobject.  In other words, I think the actual repo format is correct\nas-is, but the tools surrounding it cause a lot of confusion.\n\nImagine if cloning a superproject also checked out the subproject\ntransparently, and committing dirty data inside the subproject's tree\ncreated a new commit object for the subproject, then tacked that\ncommit object into the superproject's index for a later commit\n(exactly as changing a subdir creates a new tree object that the\nparent directory can refer to).\n\nThis doesn't solve some use cases, however, such as ones where people\nreally don't want to check out (or even fetch) the contents of some\nsubmodules, even when they check out the superproject.  The current\nimplementation *does* handle that situation.  I'm not sure how many\npeople rely on that behaviour, though.  (And maybe the correct\nsolution to *that* is proper support for sparse clone/checkout\nregardless of submodules.)\n\nHave fun,\n\nAvery\n"},{"id":"130787","messageId":"4B423633.6090603@web.de","threadId":"22081","inReplyTo":"fcaeb9bf1001040951r3f797750o5ebd25e93c0272ea@mail.gmail.com","subject":"Re: RFC: display dirty submodule working directory in git gui and gitk","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2010-01-04T18:40:51Z","receivedAt":"2010-01-04T18:40:51Z","isPatch":false,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 04.01.2010 18:51, schrieb Nguyen Thai Ngoc Duy:\n> Incidentally I was just drafting git-super.sh it see how far it goes.\n> The goal was to implement some cross-module operations over time. \"git\n> super status\", \"git super commit\" and others could be handy.\n\nHm, i'm not sure if this will really help us. I would rather see \"git\nstatus\" and friends do the right thing for submodules too. Maybe this\nhas to be configurable but i think the separate commands that one has\nto use for submodules now are part of the usability problems we are\nseeing.\n\nIMHO putting the functionality of \"git submodule summary\" into \"git\ndiff\" was a step in the right direction. This thread is about adding a\nline to the diff output when diffing against the working directory and\na submodule has a dirty working directory too. Then you can ask \"git\ndiff\" and it tells you anything you need to know about the submodule\nbefore committing or checking out in the supermodule (And IMO later on\n\"git status\" should give us this information too).\n"},{"id":"130789","messageId":"7viqbhelmh.fsf@alter.siamese.dyndns.org","threadId":"22081","inReplyTo":"4B423633.6090603@web.de","subject":"Re: RFC: display dirty submodule working directory in git gui and gitk","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2010-01-04T19:05:42Z","receivedAt":"2010-01-04T19:05:42Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jens Lehmann <Jens.Lehmann@web.de> writes:\n\n> Am 04.01.2010 18:51, schrieb Nguyen Thai Ngoc Duy:\n>> Incidentally I was just drafting git-super.sh it see how far it goes.\n>> The goal was to implement some cross-module operations over time. \"git\n>> super status\", \"git super commit\" and others could be handy.\n>\n> Hm, i'm not sure if this will really help us. I would rather see \"git\n> status\" and friends do the right thing for submodules too. Maybe this\n> has to be configurable but i think the separate commands that one has\n> to use for submodules now are part of the usability problems we are\n> seeing.\n>\n> IMHO putting the functionality of \"git submodule summary\" into \"git\n> diff\" was a step in the right direction. This thread is about adding a\n> line to the diff output when diffing against the working directory and\n> a submodule has a dirty working directory too. Then you can ask \"git\n> diff\" and it tells you anything you need to know about the submodule\n> before committing or checking out in the supermodule (And IMO later on\n> \"git status\" should give us this information too).\n\nBoth will be valid approaches to work toward the same goal.  A separate\nprototype implementation can be a way to easily figure out what the\ndesired features are.\n\nIf \"git super status\" does turns out to be consistent with what \"git\nstatus\" is supposed to do, you can decide to fold that into the latter at\nthat point.  On the other hand, information people may want from \"git\nsuper status\" could be different from what people want \"git status\" from,\nin which case it might be better to either become a new option to \"git\nstatus\", or become a new subcommand to \"git submodule\".\n\nYou start the prototype by changing \"git status\" and later decide that the\nend result either needs to become an optional behaviour, or maybe even a\nseparate command.  Either way the end result will be the same---a good\nfeature to help people is placed at the most logical place.\n\nFor the past 12 months, you and Johan Herland were the people who had more\nthan one patches with substance to git-submodule.sh and I would really\nappreciate and at the same time want to encourage experimentation by\npeople like you who are heavy users with need for a better submodule\nsupport.\n\nThanks.\n"},{"id":"130791","messageId":"4B423E1A.7070504@web.de","threadId":"22081","inReplyTo":"32541b131001041029t5adc535bt9681d33174042871@mail.gmail.com","subject":"Re: submodules, was Re: RFC: display dirty submodule working directory in git gui and gitk","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2010-01-04T19:14:34Z","receivedAt":"2010-01-04T19:14:34Z","isPatch":false,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 04.01.2010 19:29, schrieb Avery Pennarun:\n> For me one big problem comes down to producing accurate output for\n> 'git log'.  git submodules assume that the history inside the module\n> is entirely separate (you need to run multiple 'git log' instances to\n> see the full history); git-subtree assumes that it's entirely\n> integrated.  In that sense, git-subtree is somewhat more in line with\n> the core principle of git (we track the history of \"the content\", not\n> any particular file or subdir).  Unfortunately, it also exposes a\n> problem with that core principle: taken to its extreme, \"the content\"\n> includes all data in the universe.  And while git could branch and\n> merge the universe very efficiently in about O(log n) time, 'git log'\n> output gets less useful about O(n) with the size of the tree.\n> \n> Neither git-subtree nor git submodules seem to help with this \"log\n> pollution\" problem very much - but I don't know what to do that would\n> be better.\n\nI think this depends extremely on the use case and may even differ\nfrom submodule to submodule. It might be desirable to be able to\nspecify which submodule logs you want to see, because only the user\nknows what is important for him. But you should be able to ask \"git\nlog\" directly without forking it in every submodule you care about,\nno?\n\nThere has been a thread between Junio and Heiko about group mappings\nfor submodules. Maybe the configuration could be extended to contain\ninformation about what submodule should add to the superprojects log?\nhttp://thread.gmane.org/gmane.comp.version-control.git/130928/\n\n\n> Outside of this, my major problem with submodules is they use separate\n> work trees and repositories, and thus require lots of extra\n> housekeeping to get anything done.  I'd be much happier if submodules\n> would share the same objects/packs/.gitdir/refs/indexfile as the\n> superproject, and the *only* thing special about them would be that\n> the superproject's tree points at a commit object instead of a tree\n> object.  In other words, I think the actual repo format is correct\n> as-is, but the tools surrounding it cause a lot of confusion.\n\nI don't care deeply where the objects live but agree about the repo\nformat and the confusion ;-)\n\n\n> Imagine if cloning a superproject also checked out the subproject\n> transparently,\n\nThat would be great (at least at checkout time, after clone you\nmight wanna decide which submodules to initialize first - unless\ngroup mappings are working). Right now we use post-checkout hooks\nto do that.\n\n\n> and committing dirty data inside the subproject's tree\n> created a new commit object for the subproject, then tacked that\n> commit object into the superproject's index for a later commit\n> (exactly as changing a subdir creates a new tree object that the\n> parent directory can refer to).\n\nThat would be a nice feature.\n\n\n> This doesn't solve some use cases, however, such as ones where people\n> really don't want to check out (or even fetch) the contents of some\n> submodules, even when they check out the superproject.  The current\n> implementation *does* handle that situation.  I'm not sure how many\n> people rely on that behaviour, though.  (And maybe the correct\n> solution to *that* is proper support for sparse clone/checkout\n> regardless of submodules.)\n\nWe do rely on this behavior. But sparse clone or group mappings\ncould replace that need.\n"},{"id":"130792","messageId":"4B423FAA.8070106@web.de","threadId":"22081","inReplyTo":"7viqbhelmh.fsf@alter.siamese.dyndns.org","subject":"Re: RFC: display dirty submodule working directory in git gui and gitk","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2010-01-04T19:21:14Z","receivedAt":"2010-01-04T19:21:14Z","isPatch":false,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 04.01.2010 20:05, schrieb Junio C Hamano:\n> Jens Lehmann <Jens.Lehmann@web.de> writes:\n> \n>> Am 04.01.2010 18:51, schrieb Nguyen Thai Ngoc Duy:\n>>> Incidentally I was just drafting git-super.sh it see how far it goes.\n>>> The goal was to implement some cross-module operations over time. \"git\n>>> super status\", \"git super commit\" and others could be handy.\n>>\n>> Hm, i'm not sure if this will really help us. I would rather see \"git\n>> status\" and friends do the right thing for submodules too. Maybe this\n>> has to be configurable but i think the separate commands that one has\n>> to use for submodules now are part of the usability problems we are\n>> seeing.\n\n> Both will be valid approaches to work toward the same goal.  A separate\n> prototype implementation can be a way to easily figure out what the\n> desired features are.\n\n> For the past 12 months, you and Johan Herland were the people who had more\n> than one patches with substance to git-submodule.sh and I would really\n> appreciate and at the same time want to encourage experimentation by\n> people like you who are heavy users with need for a better submodule\n> support.\n\nRight. It was not my intention to discourage such experimentations with\nmy reply. I'm sorry if my email made this impression.\n"},{"id":"130798","messageId":"20100104222701.GE22872@spearce.org","threadId":"22081","inReplyTo":"alpine.DEB.1.00.1001042217370.4985@pacific.mpi-cbg.de","subject":"Re: submodules' shortcomings, was Re: RFC: display dirty submodule working directory in git gui and gitk","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2010-01-04T22:27:01Z","receivedAt":"2010-01-04T22:27:01Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:\n> Besides, as long as there is enough reason to have out-of-Git alternative \n> solutions such as repo, submodules deserve to be 2nd-class citizens.\n\nIf I didn't think I'd be shot by current submodule users, I'd offer\nto write a full replacement based around the current in repository\nformat, but with sane features like we have in repo.\n\nActually, that's why repo happened.  I felt like submodules was\nalready too frozen to accept a different approach.  And another\nguy here thought XML might be a solution to a problem...  :-|\n\n-- \nShawn.\n"},{"id":"130797","messageId":"alpine.DEB.1.00.1001042217370.4985@pacific.mpi-cbg.de","threadId":"22081","inReplyTo":"4B421F90.4090402@web.de","subject":"submodules' shortcomings, was Re: RFC: display dirty submodule working directory in git gui and gitk","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2010-01-04T22:29:08Z","receivedAt":"2010-01-04T22:29:08Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Mon, 4 Jan 2010, Jens Lehmann wrote:\n\n> Am 04.01.2010 10:44, schrieb Johannes Schindelin:\n> > The real problem is that submodules in the current form are not very \n> > well designed.\n> \n> IMVHO using the tree sha1 for a submodule seems to be the 'natural' way \n> to include another git repo. And it gives the reproducibility i expect \n> from a scm. Or am i missing something?\n\nYou do remember the discussion at the Alles wird Git about the need for \nSubversion external-like behavior, right?\n\n> It looks to me as most shortcomings come from the fact that most git \n> commands tend to ignore submodules (and if they don't, like git gui and \n> gitk do now, they e.g. only show certain aspects of their state).\n\nIt is not only ignoring.  It is not being able to cope with the state only \nsubmodules can be in (see below).\n\n> Submodules are in heavy use in our company since last year. Virtually \n> every patch i submitted for submodules came from that experience and \n> scratched an itch i or one of my colleagues had (and the situation did \n> already improve noticeably by the few things we changed). We are still \n> convinced that using submodules was the right decision. But some work \n> has still to be done to be able to use them easily and to get rid of \n> some pitfalls.\n\nSubmodules may be the best way you have in Git for your workflow ATM.  \nBut that does not mean that the submodule design is in any way \nthought-through.\n\nJust a few shortcomings that do show up in my main project (and to a \nsmall extent in msysGit, as you are probably aware):\n\n- submodules were designed with a strong emphasis on not being forced to \n  check them out.  But Git makes it very unconvenient to actually check \n  submodules out, let alone check them out at clone-time.  And it is \n  outright impossible to _enforce_ a submodule to be checked out.\n\n- among other use cases, submodules are recommended for sharing content \n  between two different repositories. But it is part of the design that it \n  is _very_ easy to forget to commit, or push the changes in the submodule \n  that are required for the integrity of the superproject.\n\n- that use case -- sharing content between different repositories -- is \n  not really supported by submodules, but rather an afterthought.  This is \n  all too obvious when you look at the restriction that the shared content \n  must be in a single subdirectory.\n\n- submodules would be a perfect way to provide a fast-forward-only media \n  subdirectory that is written to by different people (artists) than to \n  the superproject (developers).  But there is no mechanism to enforce \n  shallow fetches, which means that this use case cannot be handled \n  efficiently using Git.\n\n- related are the use cases where it is desired not to have a fixed \n  submodule tip committed to the superproject, but always to update to the \n  current, say, master (like Subversion's externals).  This use case has \n  been wished away by the people who implemented submodules in Git.  But \n  reality has this nasty habit of ignoring your wishes, does it not?\n\n- there have been patches supporting rebasing submodules, i.e.  \n  submodules where a \"git submodule update\" rebases the current branch to \n  the revision committed to the superproject rather than detaching the \n  HEAD, which everybody who ever contributed to a project with submodules \n  should agree is a useful thing. But the patches only have been discussed \n  to death, to the point where the discussion's information content was \n  converging to zero, yet the patches did not make it into Git.  (FWIW \n  this is one reason why I refuse to write patches to git-submodule.sh: I \n  refuse to let my time to be wasted like that.)\n\n- working directories with GIT_DIRs are a very different beast from single \n  files.  That alone leads to a _lot_ of problems.  The original design of \n  Git had only a couple of states for named content (AKA files): clean, \n  added, removed, modified.  The states that are possible with submodules \n  are for the most part not handled _at all_ by most Git commands (and it \n  is sometimes very hard to decide what would be the best way to handle \n  those states, either).  Just think of a submodule at a different \n  revision than committed in the superproject, with uncommitted changes, \n  ignored and unignored files, a few custom hooks, a bit of additional \n  metadata in the .git/config, and just for fun, a few temporary files in \n  .git/ which are used by the hooks.\n\n- while it might be called clever that the submodules' metadata are stored \n  in .gitmodules in the superproject (and are therefore naturally tracked \n  with Git), the synchronization with .git/config is performed exactly \n  once -- when you initialize the submodule.  You are likely to miss out \n  on _every_ change you pulled into the superproject.\n\nAll in all, submodules are very clumsy to work with, and you are literally \nforced to provide scripts in the superproject to actually work with the \nsubmodules.\n\n> > In ths short run, we can paper over the shortcomings of the submodules \n> > by introducing a command line option \"--include-submodules\" to \n> > update-refresh, diff-files and diff-index, though.\n> \n> Maybe this is the way to go for now (and hopefully we can turn this \n> option on by default later because we did the right thing ;-).\n\nI do not think that --include-submodules is a good default.  It is just \ntoo expensive in terms of I/O even to check the status in a superproject \nwith a lot of submodules.\n\nBesides, as long as there is enough reason to have out-of-Git alternative \nsolutions such as repo, submodules deserve to be 2nd-class citizens.\n\nCiao,\nDscho\n"},{"id":"130799","messageId":"32541b131001041435t7373990dxcab10a2d8aa346d8@mail.gmail.com","threadId":"22081","inReplyTo":"20100104222701.GE22872@spearce.org","subject":"Re: submodules' shortcomings, was Re: RFC: display dirty submodule working directory in git gui and gitk","fromName":"Avery Pennarun","fromEmail":"apenwarr@gmail.com","sentAt":"2010-01-04T22:35:39Z","receivedAt":"2010-01-04T22:35:39Z","isPatch":false,"sender":{"key":"apenwarr@gmail.com","avatar":"https://avatars.githubusercontent.com/u/20592?v=4"},"body":"On Mon, Jan 4, 2010 at 5:27 PM, Shawn O. Pearce <spearce@spearce.org> wrote:\n> Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:\n>> Besides, as long as there is enough reason to have out-of-Git alternative\n>> solutions such as repo, submodules deserve to be 2nd-class citizens.\n>\n> If I didn't think I'd be shot by current submodule users, I'd offer\n> to write a full replacement based around the current in repository\n> format, but with sane features like we have in repo.\n\nPerhaps write it and call it 'git sub' or something.  Put them both\nin, and let users decide which they want to use.  Or, like git\nsubtree, maintain it separately.\n\nPersonally, I've avoided tools like repo because they seem to try to\nkidnap my *entire* git experience, most of which is already fine.\nIt's just submodules that are crazy.  I think it's probably similar\nfor other people.\n\nAvery\n"},{"id":"130801","messageId":"32541b131001041453l25409c41y3aadf749b1308f01@mail.gmail.com","threadId":"22081","inReplyTo":"alpine.DEB.1.00.1001042217370.4985@pacific.mpi-cbg.de","subject":"Re: submodules' shortcomings, was Re: RFC: display dirty submodule working directory in git gui and gitk","fromName":"Avery Pennarun","fromEmail":"apenwarr@gmail.com","sentAt":"2010-01-04T22:53:32Z","receivedAt":"2010-01-04T22:53:32Z","isPatch":false,"sender":{"key":"apenwarr@gmail.com","avatar":"https://avatars.githubusercontent.com/u/20592?v=4"},"body":"On Mon, Jan 4, 2010 at 5:29 PM, Johannes Schindelin\n<Johannes.Schindelin@gmx.de> wrote:\n> On Mon, 4 Jan 2010, Jens Lehmann wrote:\n>> IMVHO using the tree sha1 for a submodule seems to be the 'natural' way\n>> to include another git repo. And it gives the reproducibility i expect\n>> from a scm. Or am i missing something?\n>\n> You do remember the discussion at the Alles wird Git about the need for\n> Subversion external-like behavior, right?\n\nI'm not sure why this is such an issue.  Basically, non-version-locked\nsubmodules are about the easiest thing in the world; that's why CVS\nand SVN supported them first.  (SVN later added version-locking like\ngit has.)\n\nAll you need is a .gitignore entry and a trivial script that checks\nout the external.  If you want to be fancy, this operation could be\npart of git, but it's such a totally different case (and an easy one,\nno less) that I think it ought to be treated totally seperately.\n\n> - among other use cases, submodules are recommended for sharing content\n>  between two different repositories. But it is part of the design that it\n>  is _very_ easy to forget to commit, or push the changes in the submodule\n>  that are required for the integrity of the superproject.\n[...]\n> - working directories with GIT_DIRs are a very different beast from single\n>  files.  That alone leads to a _lot_ of problems.  The original design of\n>  Git had only a couple of states for named content (AKA files): clean,\n>  added, removed, modified.  The states that are possible with submodules\n>  are for the most part not handled _at all_ by most Git commands (and it\n>  is sometimes very hard to decide what would be the best way to handle\n>  those states, either).  Just think of a submodule at a different\n>  revision than committed in the superproject, with uncommitted changes,\n>  ignored and unignored files, a few custom hooks, a bit of additional\n>  metadata in the .git/config, and just for fun, a few temporary files in\n>  .git/ which are used by the hooks.\n\n\nI think this is primarily because checked-out submodules currently\nhave their own .git directories (with their own config, index, etc).\nIf they were considered *part* of the subproject's repo checkout, and\nupdated upon switching branches, etc, this whole class of problems\nwould go away.\n\n> - that use case -- sharing content between different repositories -- is\n>  not really supported by submodules, but rather an afterthought.  This is\n>  all too obvious when you look at the restriction that the shared content\n>  must be in a single subdirectory.\n\nI haven't found the subdir requirement to be much of an issue, at\nleast on Unix where I can simply work around it using symlinks from\nthe superproject into the subproject.  It's obviously more gross on\nWindows, but I've worked around it there too.  This one isn't a daily\naggravation for me, though maybe it is for others.  And any cure I can\nthink of sounds rather worse than the disease.\n\n> - submodules would be a perfect way to provide a fast-forward-only media\n>  subdirectory that is written to by different people (artists) than to\n>  the superproject (developers).  But there is no mechanism to enforce\n>  shallow fetches, which means that this use case cannot be handled\n>  efficiently using Git.\n\nI doubt you want to \"enforce\" shallow fetches.  And if you just want\nto \"allow\" shallow fetches, or default to shallow fetches, I'd think\nit would be pretty easy to add.  This hasn't been important to me\neither.  (It seems to be not too important to git users in general, or\ngit's support *in general* for shallow repositories would be more\nfeatureful.)\n\n> - while it might be called clever that the submodules' metadata are stored\n>  in .gitmodules in the superproject (and are therefore naturally tracked\n>  with Git), the synchronization with .git/config is performed exactly\n>  once -- when you initialize the submodule.  You are likely to miss out\n>  on _every_ change you pulled into the superproject.\n\nThis could be fixed too, though I gave up on git-submodule before I\nbothered to fix it myself.\n\nThe correct solution here is simply to not ever copy the settings from\n.gitmodules into .git/config.  Instead, git-submodule should read\n.gitmodules as defaults, and then override those defaults with\nanything in .git/config.  99% of users will probably not need to ever\nput any of their settings in .git/config, and so this problem\ndisappears.\n\n> All in all, submodules are very clumsy to work with, and you are literally\n> forced to provide scripts in the superproject to actually work with the\n> submodules.\n\nAgreed; I do this in every project which uses git-submodule.  (And\nfrom doing so, I learned that the value-added of git-submodule is\nnearly zero.  My script does most of the work, and it could just as\neasily check out the submodule as a git repo too.  I could even choose\nto version-lock or not version-lock the checked-out submodule: just\nhardcode the commitid into my script!)\n\n> I do not think that --include-submodules is a good default.  It is just\n> too expensive in terms of I/O even to check the status in a superproject\n> with a lot of submodules.\n\nI've thought about this a lot, and I think having a special case for\nsubmodules here is the wrong line of thinking.  A big project\n*without* submodules has this same problem.  The \"real\" solution is to\njust make status checks faster.\n\n(This is actually possible to do: in the extreme case, you just have a\ndaemon running with inotify or the Windows equivalent.  TortoiseSvn\nreputedly does something like this.  I've thought of writing such a\ndaemon myself to just twiddle --assume-{un,}changed flags at the right\ntimes, particularly since status checks in Windows are so ridiculously\nslow.  But I got frustrated when it was *still* slow even after\nsetting --assume-unchanged on all the files in the index.  git still\nscans directories to detect *unknown* files, and there seems to be no\nway to turn it off or, moreover, to provide the list of unknown files\nfrom some other source.)\n\nHave fun,\n\nAvery\n"},{"id":"130828","messageId":"4B42F425.4010901@web.de","threadId":"22081","inReplyTo":"alpine.DEB.1.00.1001042217370.4985@pacific.mpi-cbg.de","subject":"Re: submodules' shortcomings, was Re: RFC: display dirty submodule working directory in git gui and gitk","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2010-01-05T08:11:17Z","receivedAt":"2010-01-05T08:11:17Z","isPatch":false,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 04.01.2010 23:29, schrieb Johannes Schindelin:\n> You do remember the discussion at the Alles wird Git about the need for \n> Subversion external-like behavior, right?\n\nYup. But never having used svn, let alone externals, i think i just\ndid not get it then ;-)\n\n\n> - submodules were designed with a strong emphasis on not being forced to \n>   check them out.  But Git makes it very unconvenient to actually check \n>   submodules out, let alone check them out at clone-time.  And it is \n>   outright impossible to _enforce_ a submodule to be checked out.\n\nAbsolutely. But i think the group mappings discussed by Junio and Heiko\nare a good starting point to solve that problem:\nhttp://thread.gmane.org/gmane.comp.version-control.git/130928/\n\nThis should be solvable by putting the necessary information into\n.gitmodules and have git clone use it.\n\n\n> - among other use cases, submodules are recommended for sharing content \n>   between two different repositories. But it is part of the design that it \n>   is _very_ easy to forget to commit, or push the changes in the submodule \n>   that are required for the integrity of the superproject.\n\nDefinitely (and if i got that right, svn externals have the same problem).\n\nWhat about checking for every submodule before a push in the superproject\nthat its HEAD is on a remote branch? I don't think we can provide full\nsafety here, but we could handle the 99% case of a forgotten push in the\nsubmodule. This could even be done with a rather simple hook (if we had a\npre-push hook that is :-).\n\n\n> - that use case -- sharing content between different repositories -- is \n>   not really supported by submodules, but rather an afterthought.  This is \n>   all too obvious when you look at the restriction that the shared content \n>   must be in a single subdirectory.\n\nI don't see that as a problem (and it's the same with svn externals, no?).\n\nAnd having worked for a long time with a RCS variant which allowed\n\"projects\" to contain an arbitrary list of files, i don't think this is\na problem (but forgetting to add new files to this list really is, so\nputting everything in one directory is *much* safer IMHO).\nAnd: almost all files were properly grouped in directories after a decade\nof development even though that was not enforced by the scm at all.\n\n\n> - related are the use cases where it is desired not to have a fixed \n>   submodule tip committed to the superproject, but always to update to the \n>   current, say, master (like Subversion's externals).  This use case has \n>   been wished away by the people who implemented submodules in Git.  But \n>   reality has this nasty habit of ignoring your wishes, does it not?\n\nHaving read up about svn externals in the meantime, what about something\nlike this:\n- Add a command like \"git submodule forward\" (as update is already in\n  use) that takes an optional -b <branchname>. It does a fetch in the\n  submodule, then tries to fast forward (or rebase) to master or the\n  branch given and stages this commit in the superproject. This should\n  be the equivalent to doing an \"svn update\" in a repo with externals.\n  Or am i missing something?\n  (And we could avoid the detached HEAD in the fast forward case by\n  really checking out the branch in the submodule)\n- We could also add an option to \"git submodule add\" to specify the\n  default branch name for forward.\n\n\n> - while it might be called clever that the submodules' metadata are stored \n>   in .gitmodules in the superproject (and are therefore naturally tracked \n>   with Git), the synchronization with .git/config is performed exactly \n>   once -- when you initialize the submodule.  You are likely to miss out \n>   on _every_ change you pulled into the superproject.\n\nYes. This synchronization could be either obsoleted by only using\n.gitmodules or automated.\n\n\n> Besides, as long as there is enough reason to have out-of-Git alternative \n> solutions such as repo, submodules deserve to be 2nd-class citizens.\n\nI think in the long run to make submodules first class citizens the\nfollowing submodule commands must be obsoleted by their regular git\nparts: init (by git clone), status (by git status), update (by git\ncheckout), summary (already in git diff thanks to your patch) and sync\n(maybe Avery's idea of only relying on .gitmodules and not copying data\nint .git/config would solve this).\nThat would leave git submodule add, foreach and maybe a command to do\nwhat svn update does for externals and another to manipulate things like\ngroup membership etc..\n\n\nWhich reminds me of Sverre's quote from Alles Wird Git:\n\"Yes, it is possible. But it will be hard.\"\n"},{"id":"130831","messageId":"7v1vi428w0.fsf@alter.siamese.dyndns.org","threadId":"22081","inReplyTo":"4B42F425.4010901@web.de","subject":"Re: submodules' shortcomings, was Re: RFC: display dirty submodule working directory in git gui and gitk","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2010-01-05T09:33:51Z","receivedAt":"2010-01-05T09:33:51Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jens Lehmann <Jens.Lehmann@web.de> writes:\n\n> Am 04.01.2010 23:29, schrieb Johannes Schindelin:\n> ...\n>> - submodules were designed with a strong emphasis on not being forced to \n>>   check them out.  But Git makes it very unconvenient to actually check \n>>   submodules out, let alone check them out at clone-time.  And it is \n>>   outright impossible to _enforce_ a submodule to be checked out.\n>\n> Absolutely. But i think the group mappings discussed by Junio and Heiko\n> are a good starting point to solve that problem:\n> http://thread.gmane.org/gmane.comp.version-control.git/130928/\n>\n> This should be solvable by putting the necessary information into\n> .gitmodules and have git clone use it.\n\nI sense there is a chicken and egg problem, but I'll let it pass for now.\n\n>> - among other use cases, submodules are recommended for sharing content \n>>   between two different repositories. But it is part of the design that it \n>>   is _very_ easy to forget to commit, or push the changes in the submodule \n>>   that are required for the integrity of the superproject.\n>\n> Definitely (and if i got that right, svn externals have the same problem).\n>\n> What about checking for every submodule before a push in the superproject\n> that its HEAD is on a remote branch? I don't think we can provide full\n> safety here, but we could handle the 99% case of a forgotten push in the\n> submodule. This could even be done with a rather simple hook (if we had a\n> pre-push hook that is :-).\n\nYou don't need \"pre-push\" hook, if the eventual goal is to integrate this\ninto \"git push\" proper; it can notice submodule directories, descending\ninto them, check if the remote lacks the necessary commit and invoke \"git\npush\" via run_command() interface as needed.\n\n>> - related are the use cases where it is desired not to have a fixed \n>>   submodule tip committed to the superproject, but always to update to the \n>>   current, say, master (like Subversion's externals).  This use case has \n>>   been wished away by the people who implemented submodules in Git.  But \n>>   reality has this nasty habit of ignoring your wishes, does it not?\n>\n> Having read up about svn externals in the meantime, what about something\n> like this:\n> - Add a command like \"git submodule forward\" (as update is already in\n>   use) that takes an optional -b <branchname>. It does a fetch in the\n>   submodule, then tries to fast forward (or rebase) to master or the\n>   branch given and stages this commit in the superproject. This should\n>   be the equivalent to doing an \"svn update\" in a repo with externals.\n>   Or am i missing something?\n>   (And we could avoid the detached HEAD in the fast forward case by\n>   really checking out the branch in the submodule)\n> - We could also add an option to \"git submodule add\" to specify the\n>   default branch name for forward.\n\nInstead of recording a specific submodule commit in the superproject, we\ncould record a branch name (this would need a separate \"gitlink\" type of\nobject we toyed around during the early days of submodule design) to say\n\"the tip of the branch\".\n\nBut there is a difference between a distributed system and a centralized\none like Subversion.  When you say \"tip of the branch\", you have to say\n\"which repository\".  If your position is that _any_ repository will do as\nlong as the commit is at the tip of the named branch, that is like saying\nyou don't care what commit it really is, as you are free to muck with\nbranch heads in your copy of submodule repository, by adding commits, or\nresetting new ones away.  For that matter, your 'master' branch in the\nsubmodule repository may not build-on/fork-from the 'master' branch in the\nupstream of it, so even \"tip of the branch by _this name_\" is still fuzzy.\n\nI am not saying \"any commit will do\" is necessarily a bad position to\ntake.  But people who claim they want to say \"this branch\" need to realize\nwhat they are really saying: whatever you record in the superproject\ncommit is immaterial.  In other words, \"this superproject will work no\nmatter which version of submodule is checked out at its location\".\n\nThatv actually is a very valid thing to say in some situations (Dscho\nmentioned different versions of artwork checked out as a submodule in a\ndeveloper's superproject to build an app).  Interestingly enough, some\npeople seem to think that we place too much importance on not having to\ncheck out submodules, but it indeed is a very natural extention of \"any\ncommit will do\".  If the configuration you chose for your build does not\ndepend on any files from there, it will truly be \"any commit will do\",\nincluding \"nothing checked out there is just fine\".\n\nSo it is not necessarily a bad thing if the commit checked out in the\nsubmodule repository is different from what the superproject records in\nits index when a commit is made in the superproject.  We allow committing\nwith local changes in regular files, while we do notify the users about\nthem to avoid mistakes.  We should give the same kind of notification\nabout submodules, but the \"local changes\" need to be thought out more\ncarefully than plain files in the superproject itself.  Does uncommitted\nchanges in the index of submodule repository count?  Local changes in the\nwork tree files?  What about untracked files that the user might have\nforgot to add?  Should they be warned?  What about the commit in the\nsubmodule repository being a non-descendant of the commit recorded in the\nHEAD of the superproject's tree, resulting in a non-ff change at the\nsubmodule level?\n\nWhat this also means is that it is important to\n\n (1) be able to simply be a user of the submodule (in such a scenario, the\n     developer who uses artwork from designer's repository does _not_ want\n     to commit the submodule, but he does want to have a recent checkout\n     of it, and he might even make some tweaks); and\n\n (2) being able to commit the state of the superproject, even if there is\n     a mismatch between the submodule commit recorded in the superproject\n     and the actual version that is checked into the authoritative\n     submodule project by the designer (perhaps he hasn't pulled in the\n     submodule while traveling).\n\nIn other words, even if the default is made to \"always clone and checkout\nall the submodules, and before allowing anything be done in the higher\nlevels of superprojects, submodules must be made in sync with their\nlatest\", there has to be a way to override such a rigid constraints for\nthe resulting system to be usable.\n\n> I think in the long run to make submodules first class citizens the\n> following submodule commands must be obsoleted by their regular git\n> parts: init (by git clone), status (by git status), update (by git\n> checkout), summary (already in git diff thanks to your patch) and sync\n> (maybe Avery's idea of only relying on .gitmodules and not copying data\n> int .git/config would solve this).\n\nI think \"clone\" has a chicken-and-egg problem.  If all of your project\nparticipant are expected to check out all the submodules, are expected to\nmake commits in all of them, and essentially have to track everything in\nsync, then \"clone\" can obviously do that without asking what kind of\nparticipant you are [*1*].  Otherwise, you need to have some mechanism\n(e.g. \"group mapping\" you mentioned earlier) for the user to specify \"I am\ninterested in these submodules\" before the actual sub-clones to happen,\nbut until you clone the superproject that has some description for that\nmechanism to use, and the user to see what's available, you cannot say\nwhat kind of participant you are.  It has to become two-step process;\neither \"clone\" going interactive in the middle, or you let the clone to\nhappen and then \"submodule init\" to express that information.\n\n\n[Footnote]\n\n*1* of course, in such a scenario you have to question what you are using\nsubmodules for.\n"},{"id":"130832","messageId":"alpine.DEB.1.00.1001051032440.4985@pacific.mpi-cbg.de","threadId":"22081","inReplyTo":"4B42F425.4010901@web.de","subject":"Re: submodules' shortcomings, was Re: RFC: display dirty submodule working directory in git gui and gitk","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2010-01-05T09:46:11Z","receivedAt":"2010-01-05T09:46:11Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Tue, 5 Jan 2010, Jens Lehmann wrote:\n\n> Am 04.01.2010 23:29, schrieb Johannes Schindelin:\n> \n> > - submodules were designed with a strong emphasis on not being forced \n> >   to check them out.  But Git makes it very unconvenient to actually \n> >   check submodules out, let alone check them out at clone-time.  And \n> >   it is outright impossible to _enforce_ a submodule to be checked \n> >   out.\n> \n> Absolutely. But i think the group mappings discussed by Junio and Heiko\n> are a good starting point to solve that problem:\n> http://thread.gmane.org/gmane.comp.version-control.git/130928/\n> \n> This should be solvable by putting the necessary information into\n> .gitmodules and have git clone use it.\n\nAnd of course, existing Git versions will not handle it correctly.  \nJudging from the rebasing-submodule patch, the next Git version will not \nhandle it either.\n\nBut you're correct, one has to start _somewhere_.\n\n> > - among other use cases, submodules are recommended for sharing \n> >   content between two different repositories. But it is part of the \n> >   design that it is _very_ easy to forget to commit, or push the \n> >   changes in the submodule that are required for the integrity of the \n> >   superproject.\n> \n> Definitely (and if i got that right, svn externals have the same problem).\n\nYes, svn externals have that problem.  But we do not need to take the svn \nexternals example more seriously than it deserves: it illustrates a valid \nuse case that is not handled by submodules.  But svn externals are not \nwhat I would call \"elegant design\" either.\n\n> What about checking for every submodule before a push in the \n> superproject that its HEAD is on a remote branch? I don't think we can \n> provide full safety here, but we could handle the 99% case of a \n> forgotten push in the submodule. This could even be done with a rather \n> simple hook (if we had a pre-push hook that is :-).\n\nThe problem with hooks is that for security reasons, every user has to \ninstall them in every repository herself (unless she is working on a \nmachine serviced by an overzealous administrator).\n\n> > - that use case -- sharing content between different repositories -- \n> >   is not really supported by submodules, but rather an afterthought.  \n> >   This is all too obvious when you look at the restriction that the \n> >   shared content must be in a single subdirectory.\n> \n> I don't see that as a problem (and it's the same with svn externals, no?).\n> \n> And having worked for a long time with a RCS variant which allowed\n> \"projects\" to contain an arbitrary list of files, i don't think this is\n> a problem (but forgetting to add new files to this list really is, so\n> putting everything in one directory is *much* safer IMHO).\n> And: almost all files were properly grouped in directories after a decade\n> of development even though that was not enforced by the scm at all.\n\nThat happens to be the case here, I agree.\n\nBut I have a use case here where the shared content is _not_ a library \nthat can live in a subdirectory naturally.\n\n> > - related are the use cases where it is desired not to have a fixed \n> >   submodule tip committed to the superproject, but always to update to \n> >   the current, say, master (like Subversion's externals).  This use \n> >   case has been wished away by the people who implemented submodules \n> >   in Git.  But reality has this nasty habit of ignoring your wishes, \n> >   does it not?\n> \n> Having read up about svn externals in the meantime, what about something\n> like this:\n> - Add a command like \"git submodule forward\" (as update is already in\n>   use) that takes an optional -b <branchname>. It does a fetch in the\n>   submodule, then tries to fast forward (or rebase) to master or the\n>   branch given and stages this commit in the superproject. This should\n>   be the equivalent to doing an \"svn update\" in a repo with externals.\n>   Or am i missing something?\n\nYes.  It is not the decision of the fetcher, but of the guy who adds the \nsubmodule to decide what it is.\n\n> - We could also add an option to \"git submodule add\" to specify the\n>   default branch name for forward.\n\nThat's an obvious precondition for proper always-tip-submodules.  But \nGit's core data structure, the index, does not allow for it.  _That_ is \nthe difficulty, not what the user interface would look like.\n\n> > - while it might be called clever that the submodules' metadata are \n> >   stored in .gitmodules in the superproject (and are therefore \n> >   naturally tracked with Git), the synchronization with .git/config is \n> >   performed exactly once -- when you initialize the submodule.  You \n> >   are likely to miss out on _every_ change you pulled into the \n> >   superproject.\n> \n> Yes. This synchronization could be either obsoleted by only using\n> .gitmodules or automated.\n\nI start to wonder whether the insistence that .gitmodules' settings must \nbe overrideable makes any sense in practice.\n\n> > Besides, as long as there is enough reason to have out-of-Git \n> > alternative solutions such as repo, submodules deserve to be 2nd-class \n> > citizens.\n> \n> I think in the long run to make submodules first class citizens the\n> following submodule commands must be obsoleted by their regular git\n> parts: init (by git clone), status (by git status), update (by git\n> checkout), summary (already in git diff thanks to your patch) and sync\n> (maybe Avery's idea of only relying on .gitmodules and not copying data\n> int .git/config would solve this).\n\nAvery's idea was to make .gitmodules overrideable in .git/config, which \nwould share almost all the shortcomings I listed for the current solution.\n\n> That would leave git submodule add, foreach and maybe a command to do \n> what svn update does for externals and another to manipulate things like \n> group membership etc..\n> \n> Which reminds me of Sverre's quote from Alles Wird Git: \"Yes, it is \n> possible. But it will be hard.\"\n\nYeah, it will be hard.  Especially since the fact that submodule is a \nbloated shell script has outlived its usefulness by far.  (It would be \ndifferent if it was a nice, small, elegant script, but you have looked at \nit, so you know why I am disgusted.)\n\nCiao,\nDscho\n"},{"id":"130835","messageId":"alpine.DEB.1.00.1001051102530.4985@pacific.mpi-cbg.de","threadId":"22081","inReplyTo":"7v1vi428w0.fsf@alter.siamese.dyndns.org","subject":"Re: submodules' shortcomings, was Re: RFC: display dirty submodule working directory in git gui and gitk","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2010-01-05T10:07:44Z","receivedAt":"2010-01-05T10:07:44Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Tue, 5 Jan 2010, Junio C Hamano wrote:\n\n> Jens Lehmann <Jens.Lehmann@web.de> writes:\n> \n> > Am 04.01.2010 23:29, schrieb Johannes Schindelin:\n> > ...\n> >> - among other use cases, submodules are recommended for sharing \n> >>   content between two different repositories. But it is part of the \n> >>   design that it is _very_ easy to forget to commit, or push the \n> >>   changes in the submodule that are required for the integrity of the \n> >>   superproject.\n> >\n> > Definitely (and if i got that right, svn externals have the same \n> > problem).\n> >\n> > What about checking for every submodule before a push in the \n> > superproject that its HEAD is on a remote branch? I don't think we can \n> > provide full safety here, but we could handle the 99% case of a \n> > forgotten push in the submodule. This could even be done with a rather \n> > simple hook (if we had a pre-push hook that is :-).\n> \n> You don't need \"pre-push\" hook, if the eventual goal is to integrate this\n> into \"git push\" proper; it can notice submodule directories, descending\n> into them, check if the remote lacks the necessary commit and invoke \"git\n> push\" via run_command() interface as needed.\n\nThat is obvious, _iff_ we make the necessary changes in core Git.  Jens' \npoint was that you can do it with hooks, too.\n\n> >> - related are the use cases where it is desired not to have a fixed \n> >>   submodule tip committed to the superproject, but always to update \n> >>   to the current, say, master (like Subversion's externals).  This \n> >>   use case has been wished away by the people who implemented \n> >>   submodules in Git.  But reality has this nasty habit of ignoring \n> >>   your wishes, does it not?\n> >\n> > Having read up about svn externals in the meantime, what about \n> > something like this:\n> > - Add a command like \"git submodule forward\" (as update is already in \n> >   use) that takes an optional -b <branchname>. It does a fetch in the \n> >   submodule, then tries to fast forward (or rebase) to master or the \n> >   branch given and stages this commit in the superproject. This should \n> >   be the equivalent to doing an \"svn update\" in a repo with externals.  \n> >   Or am i missing something?  (And we could avoid the detached HEAD in \n> >   the fast forward case by really checking out the branch in the \n> >   submodule)\n> > - We could also add an option to \"git submodule add\" to specify the \n> >   default branch name for forward.\n> \n> Instead of recording a specific submodule commit in the superproject, we\n> could record a branch name (this would need a separate \"gitlink\" type of\n> object we toyed around during the early days of submodule design) to say\n> \"the tip of the branch\".\n\nYes, and it would be as limited (but in a different way) as the current \ngitlink.\n\nYou might argue that \"gitlink\" in its current form has not raised too many \ncomplaints.  But that is only because next to nobody uses submodules \nunless forced to.\n\n> But there is a difference between a distributed system and a centralized \n> one like Subversion.  When you say \"tip of the branch\", you have to say \n> \"which repository\".  If your position is that _any_ repository will do \n> as long as the commit is at the tip of the named branch, that is like \n> saying you don't care what commit it really is, as you are free to muck \n> with branch heads in your copy of submodule repository, by adding \n> commits, or resetting new ones away.  For that matter, your 'master' \n> branch in the submodule repository may not build-on/fork-from the \n> 'master' branch in the upstream of it, so even \"tip of the branch by \n> _this name_\" is still fuzzy.\n> \n> I am not saying \"any commit will do\" is necessarily a bad position to \n> take.  But people who claim they want to say \"this branch\" need to \n> realize what they are really saying: whatever you record in the \n> superproject commit is immaterial.  In other words, \"this superproject \n> will work no matter which version of submodule is checked out at its \n> location\".\n> \n> Thatv actually is a very valid thing to say in some situations (Dscho\n> mentioned different versions of artwork checked out as a submodule in a\n> developer's superproject to build an app).  Interestingly enough, some\n> people seem to think that we place too much importance on not having to\n> check out submodules, but it indeed is a very natural extention of \"any\n> commit will do\".  If the configuration you chose for your build does not\n> depend on any files from there, it will truly be \"any commit will do\",\n> including \"nothing checked out there is just fine\".\n\nCome on Junio, do not insult my intelligence.\n\nYou know all too well about scenarios where a superproject tracks a \n3rd-party project which the superproject's developers do not contribute \nto.\n\n\"nothing checked out there is just fine\".  Pfff.  That's ridiculous.  \nYou'll have to try much harder than that.\n\nCiao,\nJohannes\n"},{"id":"130838","messageId":"4B43292C.5060106@web.de","threadId":"22081","inReplyTo":"7v1vi428w0.fsf@alter.siamese.dyndns.org","subject":"Re: submodules' shortcomings, was Re: RFC: display dirty submodule working directory in git gui and gitk","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2010-01-05T11:57:32Z","receivedAt":"2010-01-05T11:57:32Z","isPatch":false,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 05.01.2010 10:33, schrieb Junio C Hamano:\n> So it is not necessarily a bad thing if the commit checked out in the\n> submodule repository is different from what the superproject records in\n> its index when a commit is made in the superproject.  We allow committing\n> with local changes in regular files, while we do notify the users about\n> them to avoid mistakes.  We should give the same kind of notification\n> about submodules, but the \"local changes\" need to be thought out more\n> carefully than plain files in the superproject itself.  Does uncommitted\n> changes in the index of submodule repository count?  Local changes in the\n> work tree files?  What about untracked files that the user might have\n> forgot to add?  Should they be warned?  What about the commit in the\n> submodule repository being a non-descendant of the commit recorded in the\n> HEAD of the superproject's tree, resulting in a non-ff change at the\n> submodule level?\n\nCommitting in the superproject with any dirty state in a submodule\nshould always work (same as it does with local changes in regular files),\nbut be visible for the user (again as local changes in regular files are).\nRight now we do not show enough information about a submodule to protect\nthe user from accidentally throwing away changes made inside it.\nThe only thing we show right now are the differences between submodule\ncommits and what the superproject has in its index and in its commits.\nMissing are:\n\n  a) modified files\n     I think these have to be shown, no matter if they are checked into\n     the submodules index or not (because until they are committed, they\n     can't be staged in the superproject anyway).\n\n  b) new unignored files\n     IMO these files should show up too (the superproject doesn't show\n     ignored files, the submodule state shouldn't do that either). But\n     OTOH i don't see a possibility for loss of data when this state is\n     not shown.\n\n  c) a detached HEAD not on any local *or* remote branch\n     This can be fatal when doing a reset, revert or checkout, so it\n     should be shown. Alternatively when applied on a submodule, forcing\n     could be disabled to let the command fail instead of throwing stuff\n     away.\n\n  d) a detached HEAD not on any remote branch\n     AFAICS this is only important for a push, and could just error out\n     there.\n\n(But i don't think it is necessary to show detailed information, just\nwhat type of states are found in the submodule)\n\nConcerning Dscho's remarks about the performace impact: We could control\nthis behavior via .gitmodules too (and later have different settings\nfor the submodules depending on the group the user chose). So you could\nturn these checks off for repos where you don't care, saving the time to\ngo through the whole working directory of the submodule. But i would vote\nfor the default to show at least case a) and maybe even c) to follow the\nprinciple of least surprise.\n\n\n> I think \"clone\" has a chicken-and-egg problem.  If all of your project\n> participant are expected to check out all the submodules, are expected to\n> make commits in all of them, and essentially have to track everything in\n> sync, then \"clone\" can obviously do that without asking what kind of\n> participant you are [*1*].  Otherwise, you need to have some mechanism\n> (e.g. \"group mapping\" you mentioned earlier) for the user to specify \"I am\n> interested in these submodules\" before the actual sub-clones to happen,\n> but until you clone the superproject that has some description for that\n> mechanism to use, and the user to see what's available, you cannot say\n> what kind of participant you are.  It has to become two-step process;\n> either \"clone\" going interactive in the middle, or you let the clone to\n> happen and then \"submodule init\" to express that information.\n\nYes, we can leave it that way for now (first \"clone\" and then \"submodule\ninit <the submodules you need>\"). We can migrate to the \"group mapping\"\nfunctionality later (which would then allow to force certain submodules\nto always be populated because they appear in every group).\n"},{"id":"130839","messageId":"4B432E6F.8000806@web.de","threadId":"22081","inReplyTo":"alpine.DEB.1.00.1001051032440.4985@pacific.mpi-cbg.de","subject":"Re: submodules' shortcomings, was Re: RFC: display dirty submodule working directory in git gui and gitk","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2010-01-05T12:19:59Z","receivedAt":"2010-01-05T12:19:59Z","isPatch":false,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 05.01.2010 10:46, schrieb Johannes Schindelin:\n> But I have a use case here where the shared content is _not_ a library \n> that can live in a subdirectory naturally.\n\nYes, we had to reorganize a major part of one project too. Heiko could\ntell more about that.\n\n\n>> Having read up about svn externals in the meantime, what about something\n>> like this:\n>> - Add a command like \"git submodule forward\" (as update is already in\n>>   use) that takes an optional -b <branchname>. It does a fetch in the\n>>   submodule, then tries to fast forward (or rebase) to master or the\n>>   branch given and stages this commit in the superproject. This should\n>>   be the equivalent to doing an \"svn update\" in a repo with externals.\n>>   Or am i missing something?\n> \n> Yes.  It is not the decision of the fetcher, but of the guy who adds the \n> submodule to decide what it is.\n>\n>> - We could also add an option to \"git submodule add\" to specify the\n>>   default branch name for forward.\n> \n> That's an obvious precondition for proper always-tip-submodules.  But \n> Git's core data structure, the index, does not allow for it.  _That_ is \n> the difficulty, not what the user interface would look like.\n\nI have never experienced (and never had the need for) such an always-tip\nscenario and therefore still seem to have difficulties to grok it. I\nassume you always want to have the newest tip at /checkout/ time, not at\n/commit/ time? Then my proposal would really not help you.\n\n\n> I start to wonder whether the insistence that .gitmodules' settings must \n> be overrideable makes any sense in practice.\n\nI know of none, maybe someone else can speak up here?\n(And even if it is overrideable, do the settings necessarily have to be\ncopied into .git/config when they aren't even overridden?)\n"},{"id":"130844","messageId":"20100105142727.GA83546@book.hvoigt.net","threadId":"22081","inReplyTo":"alpine.DEB.1.00.1001051032440.4985@pacific.mpi-cbg.de","subject":"Re: Re: submodules' shortcomings, was Re: RFC: display dirty submodule working directory in git gui and gitk","fromName":"Heiko Voigt","fromEmail":"hvoigt@hvoigt.net","sentAt":"2010-01-05T14:27:27Z","receivedAt":"2010-01-05T14:27:27Z","isPatch":false,"sender":{"key":"hvoigt@hvoigt.net","avatar":"https://avatars.githubusercontent.com/u/184958?v=4"},"body":"On Tue, Jan 05, 2010 at 10:46:11AM +0100, Johannes Schindelin wrote:\n> On Tue, 5 Jan 2010, Jens Lehmann wrote:\n> > Yes. This synchronization could be either obsoleted by only using\n> > .gitmodules or automated.\n> \n> I start to wonder whether the insistence that .gitmodules' settings must \n> be overrideable makes any sense in practice.\n\nI just read this and felt the need to comment.\n\nYes, it definitely makes sense in practise to have it overrideable\notherwise we loose the distributed nature of git for submodules.\n\nImagine you fork a project and you want to work with others on a change\nthat involves chaning a subproject. If you can not override .gitmodules\nyou can only work on the central repository.\n\nI am actually working like this in practise. I have a private clone of\nall the subprojects msysgit has and commit/push locally first. Once I\nsense the change is going to be useful for a wider audience I send it\nupstream. This would be more uncomfortable if it is not overideable.\n\nBut I know what you mean by the general confusion about manual updates.\nSo how about an approach like this:\n\n* clone will initialise all submodules in .git/config from .gitmodules\n\n* if a change in .gitmodules happens git scans .git/config for that\n  entry and in case nothing is there it syncronises the new one and\n  notifies the user.\n\n* if a change in .gitmodules happens and the entry before was the same\n  in .git/config we also automatically update that entry there.\n\n* In every other case we just leave .git/config alone.\n\nDid I miss anything? I think you should get the idea and that it could\nget rid of the confusion caused by manual .gitmodule updates.\n\ncheers Heiko\n\nP.S.: Additionally (for my use case) we could add a \"hint mechanism\"\nwhich allows git to \"guess\" a new submodules address. For example in\ncase I have all my local clones on \"git@my.server.net:<modulename>.git\".\nNow when a new submodule gets seen in .gitmodules it will infer the\naddress from the hint configuration and not take the original one from\nupstream.\n"},{"id":"130845","messageId":"201001051607.46310.johan@herland.net","threadId":"22081","inReplyTo":"20100105142727.GA83546@book.hvoigt.net","subject":"Re: submodules' shortcomings, was Re: RFC: display dirty submodule working directory in git gui and gitk","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2010-01-05T15:07:45Z","receivedAt":"2010-01-05T15:07:45Z","isPatch":false,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"On Tuesday 05 January 2010, Heiko Voigt wrote:\n> P.S.: Additionally (for my use case) we could add a \"hint mechanism\"\n> which allows git to \"guess\" a new submodules address. For example in\n> case I have all my local clones on\n> \"git@my.server.net:<modulename>.git\". Now when a new submodule gets\n> seen in .gitmodules it will infer the address from the hint\n> configuration and not take the original one from upstream.\n\nThis can be achieved today, if the upstream .gitmodules uses relative \nsubmodule URLs. I normally place super-repo and submodules in a single \ndirectory on the server, and use submodule URLs of the \nform \"../<modulename>.git\". Now, downstream developers can \"git \nclone --mirror\" the repos from my server, and - as long as they \npreserve the directory layout - provide their own complete server \nmirror, without editing .gitmodules. Granted, the existing submodule \ntools don't make working with relative submodule URLs particularily \neasy...\n\n\n...Johan\n\n-- \nJohan Herland, <johan@herland.net>\nwww.herland.net\n"},{"id":"130846","messageId":"alpine.DEB.1.00.1001051627250.3361@intel-tinevez-2-302","threadId":"22081","inReplyTo":"20100105142727.GA83546@book.hvoigt.net","subject":"Re: Re: submodules' shortcomings, was Re: RFC: display dirty submodule working directory in git gui and gitk","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2010-01-05T15:30:49Z","receivedAt":"2010-01-05T15:30:49Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Tue, 5 Jan 2010, Heiko Voigt wrote:\n\n> On Tue, Jan 05, 2010 at 10:46:11AM +0100, Johannes Schindelin wrote:\n> > On Tue, 5 Jan 2010, Jens Lehmann wrote:\n> > > Yes. This synchronization could be either obsoleted by only using\n> > > .gitmodules or automated.\n> > \n> > I start to wonder whether the insistence that .gitmodules' settings must \n> > be overrideable makes any sense in practice.\n> \n> I just read this and felt the need to comment.\n> \n> Yes, it definitely makes sense in practise to have it overrideable\n> otherwise we loose the distributed nature of git for submodules.\n\nAFAICT you can use url.<base>.insteadOf for that.\n\nOr maybe even better use a different remote for that, as you are likely \nwanting to stay up-to-date with the upstream projects even if you work on \nthe stuff locally.\n\n> But I know what you mean by the general confusion about manual updates.\n> So how about an approach like this:\n> \n> * clone will initialise all submodules in .git/config from .gitmodules\n> \n> * if a change in .gitmodules happens git scans .git/config for that\n>   entry and in case nothing is there it syncronises the new one and\n>   notifies the user.\n> \n> * if a change in .gitmodules happens and the entry before was the same\n>   in .git/config we also automatically update that entry there.\n> \n> * In every other case we just leave .git/config alone.\n\nI'm sorry, but this is the kind of stuff I am seeing in Git: a lot of \nreally complicated design with a lot of corner cases, put on top of a \nreally simple and elegant design.\n\nSo I'd like to see a solution that is obviously superior by being \nplain simple.\n\nCiao,\nDscho\n"},{"id":"130853","messageId":"7vd41oz9mp.fsf@alter.siamese.dyndns.org","threadId":"22081","inReplyTo":"4B43292C.5060106@web.de","subject":"Re: submodules' shortcomings, was Re: RFC: display dirty submodule working directory in git gui and gitk","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2010-01-05T18:31:26Z","receivedAt":"2010-01-05T18:31:26Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jens Lehmann <Jens.Lehmann@web.de> writes:\n\n> The only thing we show right now are the differences between submodule\n> commits and what the superproject has in its index and in its commits.\n> Missing are:\n>\n>   a) modified files\n> ...\n>   b) new unignored files\n>      IMO these files should show up too (the superproject doesn't show\n>      ignored files, the submodule state shouldn't do that either). But\n>      OTOH i don't see a possibility for loss of data when this state is\n>      not shown.\n\nI don't know if we are talking about the same scenario.  What I had in\nmind was:\n\n    cd sub\n    edit new-file\n    tests ok and be happy\n    git commit\n    cd ..\n    git status\n    git commit\n\nforgetting that only you have sub/new-file in the world.  It is not loss\nof data, but still bad.  Forgetting to add a new-file and committing in a\nproject without submodule doesn't lose data, but the resulting commit will\nbe seen as broken by other people.\n\n>   c) a detached HEAD not on any local *or* remote branch\n>      This can be fatal when doing a reset, revert or checkout, so it\n>      should be shown. Alternatively when applied on a submodule, forcing\n>      could be disabled to let the command fail instead of throwing stuff\n>      away.\n\nSorry, I am lost.  Are you worried about \"reset/revert/checkout\" in the\nsuperproject?  What destructive things do these operations do that you\nconsider \"fatal\"?  I am especially puzzled by \"revert\", as \"commit\",\n\"cherry-pick\", and \"merge\" would have the same \"fatal\" effect as \"revert\",\nbut I don't get what \"fatality\" you are talking about here.\n\n>   d) a detached HEAD not on any remote branch\n>      AFAICS this is only important for a push, and could just error out\n>      there.\n\nLikewise.\n\n>> I think \"clone\" has a chicken-and-egg problem.  If all of your project\n>> ...\n>> what kind of participant you are.  It has to become two-step process;\n>> either \"clone\" going interactive in the middle, or you let the clone to\n>> happen and then \"submodule init\" to express that information.\n>\n> Yes, we can leave it that way for now (first \"clone\" and then \"submodule\n> init <the submodules you need>\"). We can migrate to the \"group mapping\"\n> functionality later (which would then allow to force certain submodules\n> to always be populated because they appear in every group).\n\nEven with group mapping, you need to clone the superproject first, before\nseeing the mapping (which I would assume comes in the superproject).  And\nyou need to see the mapping to decide what group you belong to.  After\nthat you can finally drive sub-clone to continue (e.g. I work in the\ndocumentation area, and the group mapping has 'docs' that lets me pull in\nsubmodules for doc/ and common/ directories, without src/ submodule --- I\ncan only learn that the submodules I am interested in are called 'docs' by\ngroup name or doc/ and common/ subdirectories _after_ I get the clone of\nthe superproject).\n\nI don't know if \"this appears in all groups so let's always sub-clone it\"\nis very useful in practice, but some sort of mandatory clone/checkout\nmechanism would be handy.\n"},{"id":"130858","messageId":"4B439A86.3020500@web.de","threadId":"22081","inReplyTo":"7vd41oz9mp.fsf@alter.siamese.dyndns.org","subject":"Re: submodules' shortcomings, was Re: RFC: display dirty submodule working directory in git gui and gitk","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2010-01-05T20:01:10Z","receivedAt":"2010-01-05T20:01:10Z","isPatch":false,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 05.01.2010 19:31, schrieb Junio C Hamano:\n> Jens Lehmann <Jens.Lehmann@web.de> writes:\n>>   b) new unignored files\n>>      IMO these files should show up too (the superproject doesn't show\n>>      ignored files, the submodule state shouldn't do that either). But\n>>      OTOH i don't see a possibility for loss of data when this state is\n>>      not shown.\n> \n> I don't know if we are talking about the same scenario.  What I had in\n> mind was:\n> \n>     cd sub\n>     edit new-file\n>     tests ok and be happy\n>     git commit\n>     cd ..\n>     git status\n>     git commit\n> \n> forgetting that only you have sub/new-file in the world.  It is not loss\n> of data, but still bad.  Forgetting to add a new-file and committing in a\n> project without submodule doesn't lose data, but the resulting commit will\n> be seen as broken by other people.\n\nI'm not quite sure, i was rather thinking about something like this:\n\n    cd sub\n    edit new-file\n    cd ..\n    <use sub/new-file here, test ok and be happy>\n    git status\n    git commit\n    git push\n\ngit status won't show you that sub has any new files and so you won't be\nreminded that you still have to add, commit and push it in the submodule\nbefore you should even commit, let alone push in the superproject.\n\nIt is a possible breakage for other people if sub/new-file stays unnoticed.\nThat's IMO a good point for showing these files too.\n\n\n>>   c) a detached HEAD not on any local *or* remote branch\n>>      This can be fatal when doing a reset, revert or checkout, so it\n>>      should be shown. Alternatively when applied on a submodule, forcing\n>>      could be disabled to let the command fail instead of throwing stuff\n>>      away.\n> \n> Sorry, I am lost.  Are you worried about \"reset/revert/checkout\" in the\n> superproject?  What destructive things do these operations do that you\n> consider \"fatal\"?  I am especially puzzled by \"revert\", as \"commit\",\n> \"cherry-pick\", and \"merge\" would have the same \"fatal\" effect as \"revert\",\n> but I don't get what \"fatality\" you are talking about here.\n\nSorry, that was an incomplete description on my part.\n\nMy mind had already been warped into in the - hopefully not too distant -\nfuture where these commands will be able to recurse into submodules too\n(I ran into this issue recently while trying to teach git gui to revert\nsubmodules). Right now we are blind for this state of the submodule unless\nyou go inside and use \"git status\" and friends there. And if you use e.g.\n\"git reset --hard\" there, you can loose the commits on HEAD which aren't\non any branch.\n\n\n>>   d) a detached HEAD not on any remote branch\n>>      AFAICS this is only important for a push, and could just error out\n>>      there.\n> \n> Likewise.\n\nThis can be bad in the same way that new unignored files can be (and\nthere is no time travel involved this time ;-). With HEAD i meant the\nsubmodule commit committed and about to be pushed in the supermodule\n(which happens to be the HEAD of the submodule most of the time, but\nnot always). So you committed sub/new-file but didn't push it anywhere.\nThis can lead to breakage for other people even with current git. I\nthink push could check for this and error out, as pushing out a\nreferenced submodule commit which is not pushed anywhere makes no sense.\n\nBut right now i don't believe we would have to show that in the output\nof git diff-files and git status, because it is only relevant at the\ntime when you actually want to push the superproject.\n\n\n>> Yes, we can leave it that way for now (first \"clone\" and then \"submodule\n>> init <the submodules you need>\"). We can migrate to the \"group mapping\"\n>> functionality later (which would then allow to force certain submodules\n>> to always be populated because they appear in every group).\n> \n> Even with group mapping, you need to clone the superproject first, before\n> seeing the mapping (which I would assume comes in the superproject).  And\n> you need to see the mapping to decide what group you belong to.  After\n> that you can finally drive sub-clone to continue (e.g. I work in the\n> documentation area, and the group mapping has 'docs' that lets me pull in\n> submodules for doc/ and common/ directories, without src/ submodule --- I\n> can only learn that the submodules I am interested in are called 'docs' by\n> group name or doc/ and common/ subdirectories _after_ I get the clone of\n> the superproject).\n\nI think we agree here.\n"},{"id":"130861","messageId":"3af572ac1001051238t63e07a25hf9dd77056b79be89@mail.gmail.com","threadId":"22081","inReplyTo":"alpine.DEB.1.00.1001042217370.4985@pacific.mpi-cbg.de","subject":"Re: submodules' shortcomings, was Re: RFC: display dirty submodule working directory in git gui and gitk","fromName":"Pau Garcia i Quiles","fromEmail":"pgquiles@elpauer.org","sentAt":"2010-01-05T20:38:01Z","receivedAt":"2010-01-05T20:38:01Z","isPatch":false,"sender":{"key":"pgquiles@elpauer.org","avatar":null},"body":"Hello,\n\nLet me pop here to support Johannes: I agree with every single point\nhe enumerated. Every. Single. Point.\n\nFor instance, I'd like to have a 'cmake' repository where I store all\nthe FindBlah.cmake modules, so that I can share them from every\nrepository, and not worry about users changing and committing in the\nmain project instead of the submodule. I can't. Subversion externals\nstill rule in that regard.\n\nOn Mon, Jan 4, 2010 at 11:29 PM, Johannes Schindelin\n<Johannes.Schindelin@gmx.de> wrote:\n> Hi,\n>\n> On Mon, 4 Jan 2010, Jens Lehmann wrote:\n>\n>> Am 04.01.2010 10:44, schrieb Johannes Schindelin:\n>> > The real problem is that submodules in the current form are not very\n>> > well designed.\n>>\n>> IMVHO using the tree sha1 for a submodule seems to be the 'natural' way\n>> to include another git repo. And it gives the reproducibility i expect\n>> from a scm. Or am i missing something?\n>\n> You do remember the discussion at the Alles wird Git about the need for\n> Subversion external-like behavior, right?\n>\n>> It looks to me as most shortcomings come from the fact that most git\n>> commands tend to ignore submodules (and if they don't, like git gui and\n>> gitk do now, they e.g. only show certain aspects of their state).\n>\n> It is not only ignoring.  It is not being able to cope with the state only\n> submodules can be in (see below).\n>\n>> Submodules are in heavy use in our company since last year. Virtually\n>> every patch i submitted for submodules came from that experience and\n>> scratched an itch i or one of my colleagues had (and the situation did\n>> already improve noticeably by the few things we changed). We are still\n>> convinced that using submodules was the right decision. But some work\n>> has still to be done to be able to use them easily and to get rid of\n>> some pitfalls.\n>\n> Submodules may be the best way you have in Git for your workflow ATM.\n> But that does not mean that the submodule design is in any way\n> thought-through.\n>\n> Just a few shortcomings that do show up in my main project (and to a\n> small extent in msysGit, as you are probably aware):\n>\n> - submodules were designed with a strong emphasis on not being forced to\n>  check them out.  But Git makes it very unconvenient to actually check\n>  submodules out, let alone check them out at clone-time.  And it is\n>  outright impossible to _enforce_ a submodule to be checked out.\n>\n> - among other use cases, submodules are recommended for sharing content\n>  between two different repositories. But it is part of the design that it\n>  is _very_ easy to forget to commit, or push the changes in the submodule\n>  that are required for the integrity of the superproject.\n>\n> - that use case -- sharing content between different repositories -- is\n>  not really supported by submodules, but rather an afterthought.  This is\n>  all too obvious when you look at the restriction that the shared content\n>  must be in a single subdirectory.\n>\n> - submodules would be a perfect way to provide a fast-forward-only media\n>  subdirectory that is written to by different people (artists) than to\n>  the superproject (developers).  But there is no mechanism to enforce\n>  shallow fetches, which means that this use case cannot be handled\n>  efficiently using Git.\n>\n> - related are the use cases where it is desired not to have a fixed\n>  submodule tip committed to the superproject, but always to update to the\n>  current, say, master (like Subversion's externals).  This use case has\n>  been wished away by the people who implemented submodules in Git.  But\n>  reality has this nasty habit of ignoring your wishes, does it not?\n>\n> - there have been patches supporting rebasing submodules, i.e.\n>  submodules where a \"git submodule update\" rebases the current branch to\n>  the revision committed to the superproject rather than detaching the\n>  HEAD, which everybody who ever contributed to a project with submodules\n>  should agree is a useful thing. But the patches only have been discussed\n>  to death, to the point where the discussion's information content was\n>  converging to zero, yet the patches did not make it into Git.  (FWIW\n>  this is one reason why I refuse to write patches to git-submodule.sh: I\n>  refuse to let my time to be wasted like that.)\n>\n> - working directories with GIT_DIRs are a very different beast from single\n>  files.  That alone leads to a _lot_ of problems.  The original design of\n>  Git had only a couple of states for named content (AKA files): clean,\n>  added, removed, modified.  The states that are possible with submodules\n>  are for the most part not handled _at all_ by most Git commands (and it\n>  is sometimes very hard to decide what would be the best way to handle\n>  those states, either).  Just think of a submodule at a different\n>  revision than committed in the superproject, with uncommitted changes,\n>  ignored and unignored files, a few custom hooks, a bit of additional\n>  metadata in the .git/config, and just for fun, a few temporary files in\n>  .git/ which are used by the hooks.\n>\n> - while it might be called clever that the submodules' metadata are stored\n>  in .gitmodules in the superproject (and are therefore naturally tracked\n>  with Git), the synchronization with .git/config is performed exactly\n>  once -- when you initialize the submodule.  You are likely to miss out\n>  on _every_ change you pulled into the superproject.\n>\n> All in all, submodules are very clumsy to work with, and you are literally\n> forced to provide scripts in the superproject to actually work with the\n> submodules.\n>\n>> > In ths short run, we can paper over the shortcomings of the submodules\n>> > by introducing a command line option \"--include-submodules\" to\n>> > update-refresh, diff-files and diff-index, though.\n>>\n>> Maybe this is the way to go for now (and hopefully we can turn this\n>> option on by default later because we did the right thing ;-).\n>\n> I do not think that --include-submodules is a good default.  It is just\n> too expensive in terms of I/O even to check the status in a superproject\n> with a lot of submodules.\n>\n> Besides, as long as there is enough reason to have out-of-Git alternative\n> solutions such as repo, submodules deserve to be 2nd-class citizens.\n>\n> Ciao,\n> Dscho\n>\n> --\n> To unsubscribe from this list: send the line \"unsubscribe git\" in\n> the body of a message to majordomo@vger.kernel.org\n> More majordomo info at  http://vger.kernel.org/majordomo-info.html\n>\n\n\n\n-- \nPau Garcia i Quiles\nhttp://www.elpauer.org\n(Due to my workload, I may need 10 days to answer)\n"},{"id":"130867","messageId":"20100106073718.6117@nanako3.lavabit.com","threadId":"22081","inReplyTo":"20100105142727.GA83546@book.hvoigt.net","subject":"Re: submodules' shortcomings, was Re: RFC: display dirty submodule working directory in git gui and gitk","fromName":"Nanako Shiraishi","fromEmail":"nanako3@lavabit.com","sentAt":"2010-01-05T22:37:18Z","receivedAt":"2010-01-05T22:37:18Z","isPatch":false,"sender":{"key":"nanako3@lavabit.com","avatar":"https://gravatar.com/avatar/3777b9e201c5883a62b1a6fdf7c53f2d712d1d80989146063ea861e33aad72a8?d=mp&s=160"},"body":"Quoting Heiko Voigt <hvoigt@hvoigt.net>\n\n> On Tue, Jan 05, 2010 at 10:46:11AM +0100, Johannes Schindelin wrote:\n>> On Tue, 5 Jan 2010, Jens Lehmann wrote:\n>> > Yes. This synchronization could be either obsoleted by only using\n>> > .gitmodules or automated.\n>> \n>> I start to wonder whether the insistence that .gitmodules' settings must \n>> be overrideable makes any sense in practice.\n>\n> I just read this and felt the need to comment.\n>\n> Yes, it definitely makes sense in practise to have it overrideable\n> otherwise we loose the distributed nature of git for submodules.\n>\n> Imagine you fork a project and you want to work with others on a change\n> that involves chaning a subproject. If you can not override .gitmodules\n> you can only work on the central repository.\n>\n> I am actually working like this in practise. I have a private clone of\n> all the subprojects msysgit has and commit/push locally first. Once I\n> sense the change is going to be useful for a wider audience I send it\n> upstream. This would be more uncomfortable if it is not overideable.\n>\n> But I know what you mean by the general confusion about manual updates.\n> So how about an approach like this:\n>\n> * clone will initialise all submodules in .git/config from .gitmodules\n>\n> * if a change in .gitmodules happens git scans .git/config for that\n>   entry and in case nothing is there it syncronises the new one and\n>   notifies the user.\n>\n> * if a change in .gitmodules happens and the entry before was the same\n>   in .git/config we also automatically update that entry there.\n>\n> * In every other case we just leave .git/config alone.\n>\n> Did I miss anything? I think you should get the idea and that it could\n> get rid of the confusion caused by manual .gitmodule updates.\n>\n> cheers Heiko\n>\n> P.S.: Additionally (for my use case) we could add a \"hint mechanism\"\n> which allows git to \"guess\" a new submodules address. For example in\n> case I have all my local clones on \"git@my.server.net:<modulename>.git\".\n> Now when a new submodule gets seen in .gitmodules it will infer the\n> address from the hint configuration and not take the original one from\n> upstream.\n\nThanks for sharing your thoughts. I find this discussion very interesting.\n\nI found this other discussion in the design area enlightening.\n\nhttp://thread.gmane.org/gmane.comp.version-control.git/47466/focus=47621\n\nIt was before I started using git heavily and I don't see many people who were in the discussion yet in the current thread, but I think it is worth reading.\n\nP.S. A happy new year to everybody!\n\n-- \nNanako Shiraishi\nhttp://ivory.ap.teacup.com/nanako3/\n"},{"id":"130871","messageId":"alpine.DEB.1.00.1001052357500.4985@pacific.mpi-cbg.de","threadId":"22081","inReplyTo":"7vd41oz9mp.fsf@alter.siamese.dyndns.org","subject":"Re: submodules' shortcomings, was Re: RFC: display dirty submodule working directory in git gui and gitk","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2010-01-05T23:02:24Z","receivedAt":"2010-01-05T23:02:24Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Tue, 5 Jan 2010, Junio C Hamano wrote:\n\n> Jens Lehmann <Jens.Lehmann@web.de> writes:\n> \n> >> I think \"clone\" has a chicken-and-egg problem.  If all of your \n> >> project ... what kind of participant you are.  It has to become \n> >> two-step process; either \"clone\" going interactive in the middle, or \n> >> you let the clone to happen and then \"submodule init\" to express that \n> >> information.\n> >\n> > Yes, we can leave it that way for now (first \"clone\" and then \n> > \"submodule init <the submodules you need>\"). We can migrate to the \n> > \"group mapping\" functionality later (which would then allow to force \n> > certain submodules to always be populated because they appear in every \n> > group).\n> \n> Even with group mapping, you need to clone the superproject first, before\n> seeing the mapping (which I would assume comes in the superproject).\n\nThat's just like saying \"you only see the URL first, and you have to clone \nbefore you see what the project is about\".\n\nSo in effect you are saying that things are bad.  But you do not take the \nleap of imagination to say what we need to improve.\n\nThere are quite a number of settings which could benefit from git-clone -- \nfinally -- learning to take more information than just the URL; autocrlf \nand submodules' \"grouping\" (which is a lousy name, by the way) being the \nmost prominent examples (which the core Git developers very obviously do \nnot use, otherwise the state of things would not be as sorry as it is).\n\nCiao,\nDscho\n"},{"id":"130872","messageId":"alpine.DEB.1.00.1001060005010.4985@pacific.mpi-cbg.de","threadId":"22081","inReplyTo":"3af572ac1001051238t63e07a25hf9dd77056b79be89@mail.gmail.com","subject":"cmake, was Re: submodules' shortcomings","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2010-01-05T23:06:24Z","receivedAt":"2010-01-05T23:06:24Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Tue, 5 Jan 2010, Pau Garcia i Quiles wrote:\n\n> For instance, I'd like to have a 'cmake' repository where I store all\n> the FindBlah.cmake modules, so that I can share them from every\n> repository, and not worry about users changing and committing in the\n> main project instead of the submodule.\n\n... which reminds me... it was you who wanted to provide a working recipe \nto compile and install CMake on msysGit, right?\n\nWhat happened in the meantime?\n\nCiao,\nDscho\n"},{"id":"130873","messageId":"alpine.DEB.1.00.1001060009530.4985@pacific.mpi-cbg.de","threadId":"22081","inReplyTo":"20100106073718.6117@nanako3.lavabit.com","subject":"Re: submodules' shortcomings, was Re: RFC: display dirty submodule working directory in git gui and gitk","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2010-01-05T23:13:32Z","receivedAt":"2010-01-05T23:13:32Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Wed, 6 Jan 2010, Nanako Shiraishi wrote:\n\n> I found this other discussion in the design area enlightening.\n> \n> http://thread.gmane.org/gmane.comp.version-control.git/47466/focus=47621\n\nCould you be so kind and summarize the result of the thread in something \nlike 2000 characters?\n\nI am sorry, but what with the recent trend of a precious few Git mailing \nlist members using up my weekly Git time budget in less than half a day, \njust by me reading their mails, it would be nice if at least _some_ \ndiscussions on the list could be concise and to the point.\n\nThanks,\nDscho\n"},{"id":"130881","messageId":"7vbph8oxg0.fsf@alter.siamese.dyndns.org","threadId":"22081","inReplyTo":"4B439A86.3020500@web.de","subject":"Re: submodules' shortcomings, was Re: RFC: display dirty submodule working directory in git gui and gitk","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2010-01-06T01:04:47Z","receivedAt":"2010-01-06T01:04:47Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jens Lehmann <Jens.Lehmann@web.de> writes:\n\n> Am 05.01.2010 19:31, schrieb Junio C Hamano:\n>> Jens Lehmann <Jens.Lehmann@web.de> writes:\n>>>   b) new unignored files\n>>>      IMO these files should show up too (the superproject doesn't show\n>>>      ignored files, the submodule state shouldn't do that either). But\n>>>      OTOH i don't see a possibility for loss of data when this state is\n>>>      not shown.\n>> \n>> I don't know if we are talking about the same scenario.  What I had in\n>> mind was:\n>> \n>>     cd sub\n>>     edit new-file\n>>     tests ok and be happy\n>>     git commit\n>>     cd ..\n>>     git status\n>>     git commit\n>> \n>> forgetting that only you have sub/new-file in the world.  It is not loss\n>> of data, but still bad.  Forgetting to add a new-file and committing in a\n>> project without submodule doesn't lose data, but the resulting commit will\n>> be seen as broken by other people.\n>\n> I'm not quite sure, i was rather thinking about something like this:\n>\n>     cd sub\n>     edit new-file\n>     cd ..\n>     <use sub/new-file here, test ok and be happy>\n>     git status\n>     git commit\n>     git push\n>\n> git status won't show you that sub has any new files and so you won't be\n> reminded that you still have to add, commit and push it in the submodule\n> before you should even commit, let alone push in the superproject.\n>\n> It is a possible breakage for other people if sub/new-file stays unnoticed.\n> That's IMO a good point for showing these files too.\n\nYeah, your \"i don't see a possibility for lost of data when this state is\nnot shown\" confused me into thinking as if you were saying it is not _too_\nbad if we didn't show the information.\n\nAfter all we _were_ in agreement.  We both think the user should be told\nabout untracked files in submodule directory when inspecting the status to\nmake a commit in the superproject.\n"},{"id":"130886","messageId":"3af572ac1001051717u7757f0dep9392fbb7b02cbbca@mail.gmail.com","threadId":"22081","inReplyTo":"alpine.DEB.1.00.1001060005010.4985@pacific.mpi-cbg.de","subject":"Re: cmake, was Re: submodules' shortcomings","fromName":"Pau Garcia i Quiles","fromEmail":"pgquiles@elpauer.org","sentAt":"2010-01-06T01:17:45Z","receivedAt":"2010-01-06T01:17:45Z","isPatch":false,"sender":{"key":"pgquiles@elpauer.org","avatar":null},"body":"On Wed, Jan 6, 2010 at 12:06 AM, Johannes Schindelin\n<Johannes.Schindelin@gmx.de> wrote:\n> Hi,\n>\n> On Tue, 5 Jan 2010, Pau Garcia i Quiles wrote:\n>\n>> For instance, I'd like to have a 'cmake' repository where I store all\n>> the FindBlah.cmake modules, so that I can share them from every\n>> repository, and not worry about users changing and committing in the\n>> main project instead of the submodule.\n>\n> ... which reminds me... it was you who wanted to provide a working recipe\n> to compile and install CMake on msysGit, right?\n\nRight\n\n> What happened in the meantime?\n\nWhat happened is I was very busy until November. Now I've got some free time.\n\nAt this moment, what stops me from beginning this project is a simple\nquestion: is it worth my time? From the discussion a few months ago,\nit looked like it would the a second-class citizen and never replace\nthe existing buildsystems, so I really wonder if I should spend me\ntime porting git to CMake, or I should focus on other projects which\nwould gladly receive my contributions. If you honestly think it's\nworth it, just tell me and I'll start the port to CMake immediately.\n\n-- \nPau Garcia i Quiles\nhttp://www.elpauer.org\n(Due to my workload, I may need 10 days to answer)\n"},{"id":"130893","messageId":"buovdffoo5t.fsf@dhlpc061.dev.necel.com","threadId":"22081","inReplyTo":"3af572ac1001051717u7757f0dep9392fbb7b02cbbca@mail.gmail.com","subject":"Re: cmake, was Re: submodules' shortcomings","fromName":"Miles Bader","fromEmail":"miles@gnu.org","sentAt":"2010-01-06T04:25:18Z","receivedAt":"2010-01-06T04:25:18Z","isPatch":false,"sender":{"key":"miles@gnu.org","avatar":"https://gravatar.com/avatar/01069b69593af7bff28e2f97afeb3644ae6fe2f5f56cb3a8cf34c5fb8c36efe5?d=mp&s=160"},"body":"Pau Garcia i Quiles <pgquiles@elpauer.org> writes:\n> At this moment, what stops me from beginning this project is a simple\n> question: is it worth my time? From the discussion a few months ago,\n> it looked like it would the a second-class citizen and never replace\n> the existing buildsystems, so I really wonder if I should spend me\n> time porting git to CMake, or I should focus on other projects which\n> would gladly receive my contributions. If you honestly think it's\n> worth it, just tell me and I'll start the port to CMake immediately.\n\nIt sounds like it's you who want it, so aren't you the best person to\nmake that judgement...?  It seems very unlikely for cmake to replace\nanything.\n\n-Miles\n\n-- \nPoliteness, n. The most acceptable hypocrisy.\n"},{"id":"130908","messageId":"alpine.DEB.1.00.1001061021370.11013@intel-tinevez-2-302","threadId":"22081","inReplyTo":"3af572ac1001051717u7757f0dep9392fbb7b02cbbca@mail.gmail.com","subject":"Re: cmake, was Re: submodules' shortcomings","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2010-01-06T09:24:06Z","receivedAt":"2010-01-06T09:24:06Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Wed, 6 Jan 2010, Pau Garcia i Quiles wrote:\n\n> On Wed, Jan 6, 2010 at 12:06 AM, Johannes Schindelin\n> <Johannes.Schindelin@gmx.de> wrote:\n> >\n> > On Tue, 5 Jan 2010, Pau Garcia i Quiles wrote:\n> >\n> >> For instance, I'd like to have a 'cmake' repository where I store all \n> >> the FindBlah.cmake modules, so that I can share them from every \n> >> repository, and not worry about users changing and committing in the \n> >> main project instead of the submodule.\n> >\n> > ... which reminds me... it was you who wanted to provide a working \n> > recipe to compile and install CMake on msysGit, right?\n> \n> Right\n> \n> > What happened in the meantime?\n> \n> What happened is I was very busy until November. Now I've got some free \n> time.\n> \n> At this moment, what stops me from beginning this project is a simple \n> question: is it worth my time?\n\nWell, I thought you wanted to show that CMake is superior to what we have \nright now, and for me as msysGit maintainer, that implies that CMake \nactually works within msysGit.\n\nNow, I do not think that it is hard to get CMake to compile in msysGit, \nbut then, I just lost access to the last Windows computer, so I cannot do \nthat myself.\n\nAs Miles said, it is up to you to decide whether it is so complicated, or \nwhether CMake is likely not to convince, that the time balance turns out \npositive or negative.\n\nCiao,\nDscho\n"},{"id":"130921","messageId":"4B4498BC.5040400@web.de","threadId":"22081","inReplyTo":"7vbph8oxg0.fsf@alter.siamese.dyndns.org","subject":"Re: submodules' shortcomings, was Re: RFC: display dirty submodule working directory in git gui and gitk","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2010-01-06T14:05:48Z","receivedAt":"2010-01-06T14:05:48Z","isPatch":false,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 06.01.2010 02:04, schrieb Junio C Hamano:\n> After all we _were_ in agreement.  We both think the user should be told\n> about untracked files in submodule directory when inspecting the status to\n> make a commit in the superproject.\n\nThanks. So i'll take a closer look at the diff core (but i suspect i'll\nneed some time until i can come up with some patches because i don't know\nthis part of git very well).\n"},{"id":"130923","messageId":"7vbph7181x.fsf@alter.siamese.dyndns.org","threadId":"22081","inReplyTo":"4B4498BC.5040400@web.de","subject":"Re: submodules' shortcomings, was Re: RFC: display dirty submodule working directory in git gui and gitk","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2010-01-06T17:01:46Z","receivedAt":"2010-01-06T17:01:46Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jens Lehmann <Jens.Lehmann@web.de> writes:\n\n> Am 06.01.2010 02:04, schrieb Junio C Hamano:\n>> After all we _were_ in agreement.  We both think the user should be told\n>> about untracked files in submodule directory when inspecting the status to\n>> make a commit in the superproject.\n>\n> Thanks. So i'll take a closer look at the diff core (but i suspect i'll\n> need some time until i can come up with some patches because i don't know\n> this part of git very well).\n\nI don't see a direct connection between \"the user should be told about\nuntracked in the submodule before committing\" and diffcore.  It is just\nthe matter of \"git status\" and \"git commit\" running another instance of\n\"git status\" via run_command() interface in the submodule directory, no?\n"},{"id":"130930","messageId":"fcaeb9bf1001060923m6559f00bp794bb5fdd4af704c@mail.gmail.com","threadId":"22081","inReplyTo":"7vbph7181x.fsf@alter.siamese.dyndns.org","subject":"Re: submodules' shortcomings, was Re: RFC: display dirty submodule working directory in git gui and gitk","fromName":"Nguyen Thai Ngoc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2010-01-06T17:23:35Z","receivedAt":"2010-01-06T17:23:35Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On 1/7/10, Junio C Hamano <gitster@pobox.com> wrote:\n> Jens Lehmann <Jens.Lehmann@web.de> writes:\n>\n>\n> > Am 06.01.2010 02:04, schrieb Junio C Hamano:\n>  >> After all we _were_ in agreement.  We both think the user should be told\n>  >> about untracked files in submodule directory when inspecting the status to\n>  >> make a commit in the superproject.\n>  >\n>  > Thanks. So i'll take a closer look at the diff core (but i suspect i'll\n>  > need some time until i can come up with some patches because i don't know\n>  > this part of git very well).\n>\n>\n> I don't see a direct connection between \"the user should be told about\n>  untracked in the submodule before committing\" and diffcore.  It is just\n>  the matter of \"git status\" and \"git commit\" running another instance of\n>  \"git status\" via run_command() interface in the submodule directory, no?\n\nYou would need to rewrite file paths so that files in submodules are\nalso relative to the same directory as files in supermodule (I tried\nto do that with GIT_WORK_TREE and needed to change a bit). Or you\ncould show each \"git status\" output separately, which does not look as\nnice as the former in my opinion.\n-- \nDuy\n"},{"id":"130932","messageId":"7vljgbw21x.fsf@alter.siamese.dyndns.org","threadId":"22081","inReplyTo":"fcaeb9bf1001060923m6559f00bp794bb5fdd4af704c@mail.gmail.com","subject":"Re: submodules' shortcomings, was Re: RFC: display dirty submodule working directory in git gui and gitk","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2010-01-06T17:55:38Z","receivedAt":"2010-01-06T17:55:38Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nguyen Thai Ngoc Duy <pclouds@gmail.com> writes:\n\n> On 1/7/10, Junio C Hamano <gitster@pobox.com> wrote:\n>> Jens Lehmann <Jens.Lehmann@web.de> writes:\n>>\n>>\n>> > Am 06.01.2010 02:04, schrieb Junio C Hamano:\n>>  >> After all we _were_ in agreement.  We both think the user should be told\n>>  >> about untracked files in submodule directory when inspecting the status to\n>>  >> make a commit in the superproject.\n>>  >\n>>  > Thanks. So i'll take a closer look at the diff core (but i suspect i'll\n>>  > need some time until i can come up with some patches because i don't know\n>>  > this part of git very well).\n>>\n>>\n>> I don't see a direct connection between \"the user should be told about\n>>  untracked in the submodule before committing\" and diffcore.  It is just\n>>  the matter of \"git status\" and \"git commit\" running another instance of\n>>  \"git status\" via run_command() interface in the submodule directory, no?\n>\n> You would need to rewrite file paths so that files in submodules are\n> also relative to the same directory as files in supermodule (I tried\n> to do that with GIT_WORK_TREE and needed to change a bit). Or you\n> could show each \"git status\" output separately, which does not look as\n> nice as the former in my opinion.\n\nYou could show output separately if you want, but I think that is a\nseparate issue.\n\nI was envisioning that the \"git status\" in submodule will be run with its\nrecent --porcelain option, and \"git status\" or \"git commit\" would read it\nto postprocess and incorporate into its own output.\n"},{"id":"130936","messageId":"4B44D45D.3070509@web.de","threadId":"22081","inReplyTo":"7vbph7181x.fsf@alter.siamese.dyndns.org","subject":"Re: submodules' shortcomings, was Re: RFC: display dirty submodule working directory in git gui and gitk","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2010-01-06T18:20:13Z","receivedAt":"2010-01-06T18:20:13Z","isPatch":false,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 06.01.2010 18:01, schrieb Junio C Hamano:\n> Jens Lehmann <Jens.Lehmann@web.de> writes:\n> \n>> Am 06.01.2010 02:04, schrieb Junio C Hamano:\n>>> After all we _were_ in agreement.  We both think the user should be told\n>>> about untracked files in submodule directory when inspecting the status to\n>>> make a commit in the superproject.\n>>\n>> Thanks. So i'll take a closer look at the diff core (but i suspect i'll\n>> need some time until i can come up with some patches because i don't know\n>> this part of git very well).\n> \n> I don't see a direct connection between \"the user should be told about\n> untracked in the submodule before committing\" and diffcore.  It is just\n> the matter of \"git status\" and \"git commit\" running another instance of\n> \"git status\" via run_command() interface in the submodule directory, no?\n\nBasically yes. But i also would like to teach \"git diff\" (when diffing\nagainst the working directory of the superproject) to show these\nsubmodule states too so that git gui and gitk will display them.\n"},{"id":"130937","messageId":"fcaeb9bf1001061022j57981a90x249f82c6dfd2f92d@mail.gmail.com","threadId":"22081","inReplyTo":"7vljgbw21x.fsf@alter.siamese.dyndns.org","subject":"Re: submodules' shortcomings, was Re: RFC: display dirty submodule working directory in git gui and gitk","fromName":"Nguyen Thai Ngoc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2010-01-06T18:22:37Z","receivedAt":"2010-01-06T18:22:37Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On 1/7/10, Junio C Hamano <gitster@pobox.com> wrote:\n> Nguyen Thai Ngoc Duy <pclouds@gmail.com> writes:\n>  > You would need to rewrite file paths so that files in submodules are\n>  > also relative to the same directory as files in supermodule (I tried\n>  > to do that with GIT_WORK_TREE and needed to change a bit). Or you\n>  > could show each \"git status\" output separately, which does not look as\n>  > nice as the former in my opinion.\n>\n>\n> You could show output separately if you want, but I think that is a\n>  separate issue.\n>\n>  I was envisioning that the \"git status\" in submodule will be run with its\n>  recent --porcelain option, and \"git status\" or \"git commit\" would read it\n>  to postprocess and incorporate into its own output.\n\nNice option! I had to call a few \"git diff\" for that just because I\ndid not catch up with recent Git development :-(\n-- \nDuy\n"},{"id":"130939","messageId":"4B44D73F.6000607@web.de","threadId":"22081","inReplyTo":"7vljgbw21x.fsf@alter.siamese.dyndns.org","subject":"Re: submodules' shortcomings, was Re: RFC: display dirty submodule working directory in git gui and gitk","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2010-01-06T18:32:31Z","receivedAt":"2010-01-06T18:32:31Z","isPatch":false,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 06.01.2010 18:55, schrieb Junio C Hamano:\n> Nguyen Thai Ngoc Duy <pclouds@gmail.com> writes:\n> \n>> On 1/7/10, Junio C Hamano <gitster@pobox.com> wrote:\n>>> Jens Lehmann <Jens.Lehmann@web.de> writes:\n>>>\n>>>\n>>>> Am 06.01.2010 02:04, schrieb Junio C Hamano:\n>>>  >> After all we _were_ in agreement.  We both think the user should be told\n>>>  >> about untracked files in submodule directory when inspecting the status to\n>>>  >> make a commit in the superproject.\n>>>  >\n>>>  > Thanks. So i'll take a closer look at the diff core (but i suspect i'll\n>>>  > need some time until i can come up with some patches because i don't know\n>>>  > this part of git very well).\n>>>\n>>>\n>>> I don't see a direct connection between \"the user should be told about\n>>>  untracked in the submodule before committing\" and diffcore.  It is just\n>>>  the matter of \"git status\" and \"git commit\" running another instance of\n>>>  \"git status\" via run_command() interface in the submodule directory, no?\n>>\n>> You would need to rewrite file paths so that files in submodules are\n>> also relative to the same directory as files in supermodule (I tried\n>> to do that with GIT_WORK_TREE and needed to change a bit). Or you\n>> could show each \"git status\" output separately, which does not look as\n>> nice as the former in my opinion.\n> \n> You could show output separately if you want, but I think that is a\n> separate issue.\n> \n> I was envisioning that the \"git status\" in submodule will be run with its\n> recent --porcelain option, and \"git status\" or \"git commit\" would read it\n> to postprocess and incorporate into its own output.\n\nAnd i thought about printing just one line for each dirty submodule that\ncontains uncommitted and/or new files. I did not intend to list every\nfile, for the same reason a \"git diff --submodule\" only shows the first\nline of the commit messages, not the actual differences of all changed\nfiles in the submodule. I am not against being able to show all files\ntoo, but i really would want to have an option to get a short output for\ngit gui and gitk.\n"},{"id":"130944","messageId":"7vbph7uhn9.fsf@alter.siamese.dyndns.org","threadId":"22081","inReplyTo":"4B44D73F.6000607@web.de","subject":"Re: submodules' shortcomings, was Re: RFC: display dirty submodule working directory in git gui and gitk","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2010-01-06T20:01:46Z","receivedAt":"2010-01-06T20:01:46Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jens Lehmann <Jens.Lehmann@web.de> writes:\n\n> Am 06.01.2010 18:55, schrieb Junio C Hamano:\n>> I was envisioning that the \"git status\" in submodule will be run with its\n>> recent --porcelain option, and \"git status\" or \"git commit\" would read it\n>> to postprocess and incorporate into its own output.\n>\n> And i thought about printing just one line for each dirty submodule that\n> contains uncommitted and/or new files. I did not intend to list every\n> file, for the same reason a \"git diff --submodule\" only shows the first\n> line of the commit messages, not the actual differences of all changed\n> files in the submodule. I am not against being able to show all files\n> too, but i really would want to have an option to get a short output for\n> git gui and gitk.\n\nI don't think what you are saying is inconsistent with \"git status/commit\nthat reads from 'git status --porcelain' it runs in a submodule directory,\npostprocesses it and incorporates it into its own output.\"  When the\nsub-status reports changes, your \"postprocess\" would condense it down to\n\"this has a potential change that user could want to commit\".  How the\ndirtiness is shown is entirely up to the caller that detected that change.\n\nLet's explain it in another way.\n\nThe original \"diff\" for a submodule entry was implemented by preparing a\n\n\t\"Subproject commit %s\\n\"\n\nline for the submodule commit recorded in the preimage and postimage, and\ncompare these as if they are one-line files.  When the postimage was work\ntree, it looked at submodule's .git/HEAD to learn what to stuff in %s\nthere.\n\nBut nobody forced you to limit the check only to .git/HEAD in the\nsubmodule.  To make the comparison richer, you could check if the\nsubmodule directory is dirty (and we have already discussed the potential\ndefinition of dirtiness earlier), and add \"-dirty\" in the string as well.\nWith such a change, if you make some changes to a file in the work tree of\nthe submodule after a clean \"clone\", \"git diff\" between the index and the\nwork tree would report:\n\n\t-Subproject commit 37bae10e38a66e4f1ddd5350daded00b21735126\n\t+Subproject commit 37bae10e38a66e4f1ddd5350daded00b21735126-dirty\n\nThe suggestion to read from \"status --porcelain\" that is run in the\nsubmodule directory was about how to implement the part that determines\nthis \"dirtiness\" information, and not about how that dirtiness is\nexpressed in the output.  The above is an illustration that even the\ntraditional output format can be made aware of this submodule dirtiness\ncheck.  \"diff --submodule\" can express that dirtiness information in any\nway it wants.\n"},{"id":"130952","messageId":"4B44FE58.3030209@web.de","threadId":"22081","inReplyTo":"7vbph7uhn9.fsf@alter.siamese.dyndns.org","subject":"Re: submodules' shortcomings, was Re: RFC: display dirty submodule working directory in git gui and gitk","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2010-01-06T21:19:20Z","receivedAt":"2010-01-06T21:19:20Z","isPatch":false,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 06.01.2010 21:01, schrieb Junio C Hamano:\n> Jens Lehmann <Jens.Lehmann@web.de> writes:\n> \n>> Am 06.01.2010 18:55, schrieb Junio C Hamano:\n>>> I was envisioning that the \"git status\" in submodule will be run with its\n>>> recent --porcelain option, and \"git status\" or \"git commit\" would read it\n>>> to postprocess and incorporate into its own output.\n>>\n>> And i thought about printing just one line for each dirty submodule that\n>> contains uncommitted and/or new files. I did not intend to list every\n>> file, for the same reason a \"git diff --submodule\" only shows the first\n>> line of the commit messages, not the actual differences of all changed\n>> files in the submodule. I am not against being able to show all files\n>> too, but i really would want to have an option to get a short output for\n>> git gui and gitk.\n> \n> I don't think what you are saying is inconsistent with \"git status/commit\n> that reads from 'git status --porcelain' it runs in a submodule directory,\n> postprocesses it and incorporates it into its own output.\"  When the\n> sub-status reports changes, your \"postprocess\" would condense it down to\n> \"this has a potential change that user could want to commit\".  How the\n> dirtiness is shown is entirely up to the caller that detected that change.\n> \n> Let's explain it in another way.\n> \n> The original \"diff\" for a submodule entry was implemented by preparing a\n> \n> \t\"Subproject commit %s\\n\"\n> \n> line for the submodule commit recorded in the preimage and postimage, and\n> compare these as if they are one-line files.  When the postimage was work\n> tree, it looked at submodule's .git/HEAD to learn what to stuff in %s\n> there.\n> \n> But nobody forced you to limit the check only to .git/HEAD in the\n> submodule.  To make the comparison richer, you could check if the\n> submodule directory is dirty (and we have already discussed the potential\n> definition of dirtiness earlier), and add \"-dirty\" in the string as well.\n> With such a change, if you make some changes to a file in the work tree of\n> the submodule after a clean \"clone\", \"git diff\" between the index and the\n> work tree would report:\n> \n> \t-Subproject commit 37bae10e38a66e4f1ddd5350daded00b21735126\n> \t+Subproject commit 37bae10e38a66e4f1ddd5350daded00b21735126-dirty\n> \n> The suggestion to read from \"status --porcelain\" that is run in the\n> submodule directory was about how to implement the part that determines\n> this \"dirtiness\" information, and not about how that dirtiness is\n> expressed in the output.  The above is an illustration that even the\n> traditional output format can be made aware of this submodule dirtiness\n> check.  \"diff --submodule\" can express that dirtiness information in any\n> way it wants.\n\nI see, we seem to agree again :-)\n\nWhile looking into \"git status\" in the last hours i became aware that\nthere is some infrastructure for calling \"git submodule summary\" (when\nthat is enabled via \"git config status.submodulesummary\"). I think this\ncan be extended to transfer the dirty information from \"git diff\n--submodule\" (which can and should replace \"git submodule summary\" IMO)\ninto \"git status\".\n\nWill send a patch for discussion tomorrow, i have to get some sleep now.\n"},{"id":"130990","messageId":"20100107200417.6117@nanako3.lavabit.com","threadId":"22081","inReplyTo":"alpine.DEB.1.00.1001060009530.4985@pacific.mpi-cbg.de","subject":"Re: submodules' shortcomings, was Re: RFC: display dirty submodule working directory in git gui and gitk","fromName":"Nanako Shiraishi","fromEmail":"nanako3@lavabit.com","sentAt":"2010-01-07T11:04:17Z","receivedAt":"2010-01-07T11:04:17Z","isPatch":false,"sender":{"key":"nanako3@lavabit.com","avatar":"https://gravatar.com/avatar/3777b9e201c5883a62b1a6fdf7c53f2d712d1d80989146063ea861e33aad72a8?d=mp&s=160"},"body":"Quoting Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:\n\n> On Wed, 6 Jan 2010, Nanako Shiraishi wrote:\n>\n>> I found this other discussion in the design area enlightening.\n>> \n>> http://thread.gmane.org/gmane.comp.version-control.git/47466/focus=47621\n>\n> Could you be so kind and summarize the result of the thread in something \n> like 2000 characters?\n\nSorry, but I only said \"enlightening\". There wasn't a conclusion that lets you stop thinking and just go ahead implementing the design specified in the thread, if that is what you are looking for.\n\nInstead, let me tell you an example of what I found enlightening. It isn't a summary of the result. I don't think there was a *result*; otherwise somebody already would have implemented it.\n\nI often wonder why 'git-submodule init' copies data to .git/config file. If .gitmodules file gives the default and I can use .git/config file to override it, it seems stupid to copy entries between these files. I can just keep using data from .gitmodules file until I need to override something.\n\nReading the thread made me realize how wrong I was. It became very clear why .gitmodules file shouldn't even be the default that is read when no entries is in .git/config file and why .git/config file should be the only thing that is used at runtime.\n\nUnfortunately I can't summarize the reason in '2000 characters', so you need read the thread yourself if you are interested. The key concept that I was missing was that remote repositories can move or change over time, and you may want to check out and interact with a very old version of your supermodule. The .gitmodules file checked out in such a case still records old information. Treating .gitmodules file as a hint and always looking into .git/config file is a part of the fundamental solution to that problem, but I didn't even realize that such an issue existed when I read the current discussion until I found the old thread.\n\nI think the 'git-submodule' script is mainly based on the 'three-level thing Steven Grimm suggested', but it doesn't seem to implement all the ideas in the thread yet. It gives no interactive prompt to suggest URL from 'git-submodule init' command. Neither it records which URLs have been seen with subproject.*.seen variable. But the issues that high level design must take into account looks very well thought out already.\n\n-- \nNanako Shiraishi\nhttp://ivory.ap.teacup.com/nanako3/\n"}]}