{"thread":{"id":"3035","subject":"RFC: Subprojects","startedAt":"2006-01-11T15:58:23Z","lastAt":"2006-02-21T07:57:14Z","messageCount":56,"participants":["Simon Richter","Johannes Schindelin","Linus Torvalds","Alexander Litvinov","Martin Langhoff","Anand Kumria","Alex Riesen","Daniel Barkalow","Junio C Hamano","A Large Angry SCM","Tom Prince","Petr Baudis","Andreas Ericsson","Josef Weidendorfer","Craig Schlenter","Uwe Zeisberger"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"14473","messageId":"43C52B1F.8020706@hogyros.de","threadId":"3035","inReplyTo":null,"subject":"RFC: Subprojects","fromName":"Simon Richter","fromEmail":"simon.richter@hogyros.de","sentAt":"2006-01-11T15:58:23Z","receivedAt":"2006-01-11T15:58:23Z","isPatch":false,"sender":{"key":"simon.richter@hogyros.de","avatar":"https://gravatar.com/avatar/1192aa9fa5dd19ce258b12b044cc27111dd7cdb58d920dc2123a02a24f55b5c5?d=mp&s=160"},"body":"Hello,\n\none thing that I have been missing so far in all SCM systems apart from \nCVS (and there it's just coincidence) is the ability to include a \nproject as part of a bigger project. Developing software for embedded \nsystems, I need that feature fairly often, for example the source tree \nfor a particular device almost always contains one or more Linux trees, \nsome binutils, gcc and gdb stuff and so on.\n\nThe changes necessary here would be fairly simple: \"tree\" objects would \npoint to a \"commit\" or a \"tag\" object when a subproject is used.\n\nIn the working directory, this would be represented by a .git directory \nthat contains a symref to the embedding project instead of the objects \ndirectory. Head pointers are only required if you intend to push changes \nupstream to the maintainer of the embedded project. Each subproject has \nits own index.\n\nWould such a feature make sense, and what behaviour would make the most \nsense for the various operations (e.g. shall commits in the inner \nproject propagate to the outer?)?\n\n Simon\n"},{"id":"14474","messageId":"Pine.LNX.4.63.0601111740220.17966@wbgn013.biozentrum.uni-wuerzburg.de","threadId":"3035","inReplyTo":"43C52B1F.8020706@hogyros.de","subject":"Re: RFC: Subprojects","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2006-01-11T16:44:53Z","receivedAt":"2006-01-11T16:44:53Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Wed, 11 Jan 2006, Simon Richter wrote:\n\n> one thing that I have been missing so far in all SCM systems apart from CVS\n> (and there it's just coincidence) is the ability to include a project as part\n> of a bigger project. Developing software for embedded systems, I need that\n> feature fairly often, for example the source tree for a particular device\n> almost always contains one or more Linux trees, some binutils, gcc and gdb\n> stuff and so on.\n\nWhat I do: I call it a branch. While this might seem technically \nincorrect, it is not.\n\nAnd since the subprojects are really independent, you can connect them by \nan octopus.\n\n> The changes necessary here would be fairly simple: \"tree\" objects would point\n> to a \"commit\" or a \"tag\" object when a subproject is used.\n\nSorry, we discussed similar things already. It is not necessary to change \nthe structure. Even more: it makes no sense. Why would you want to have \ntwo or more commit messages for the same revision?\n\nRemember: trees, commits and tags (objects in general) are immutable. You \nmay think that you just commit a new revision of the subproject, and it is \npicked up by the overall project, but that is not the case!\n\n> In the working directory, this would be represented by a .git directory that\n> contains a symref to the embedding project instead of the objects directory.\n> Head pointers are only required if you intend to push changes upstream to the\n> maintainer of the embedded project. Each subproject has its own index.\n\nYou can do this like I said: use branches (and possibly a common \nGIT_OBJECT_DIRECTORY to save on disk space).\n\nHth,\nDscho\n"},{"id":"14476","messageId":"43C537C9.4090206@hogyros.de","threadId":"3035","inReplyTo":"Pine.LNX.4.63.0601111740220.17966@wbgn013.biozentrum.uni-wuerzburg.de","subject":"Re: RFC: Subprojects","fromName":"Simon Richter","fromEmail":"simon.richter@hogyros.de","sentAt":"2006-01-11T16:52:25Z","receivedAt":"2006-01-11T16:52:25Z","isPatch":false,"sender":{"key":"simon.richter@hogyros.de","avatar":"https://gravatar.com/avatar/1192aa9fa5dd19ce258b12b044cc27111dd7cdb58d920dc2123a02a24f55b5c5?d=mp&s=160"},"body":"Hello,\n\nJohannes Schindelin wrote:\n\n> And since the subprojects are really independent, you can connect them by \n> an octopus.\n\nThe important thing for me is that I need to be able to transfer them \neasily, or turn a subdirectory into a subproject or vice versa.\n\n> Sorry, we discussed similar things already. It is not necessary to change \n> the structure. Even more: it makes no sense. Why would you want to have \n> two or more commit messages for the same revision?\n\nBecause the commit affects both the subproject and the master project.\n\n> Remember: trees, commits and tags (objects in general) are immutable. You \n> may think that you just commit a new revision of the subproject, and it is \n> picked up by the overall project, but that is not the case!\n\nThis is why I asked for intended behaviour on commit in a subproject. It \nis pretty obvious that the master project would need a new tree object \nto reference the new version of the subproject, and hence, a new commit \nto keep it all together (and correctly so, since I would like my master \nproject to refer to that particular version of the subproject that is \nknown to work).\n\n> You can do this like I said: use branches (and possibly a common \n> GIT_OBJECT_DIRECTORY to save on disk space).\n\nYes, however that wouldn't cover consistency between the subprojects, \nwould it?\n\n Simon\n"},{"id":"14479","messageId":"Pine.LNX.4.64.0601110928350.5073@g5.osdl.org","threadId":"3035","inReplyTo":"43C537C9.4090206@hogyros.de","subject":"Re: RFC: Subprojects","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2006-01-11T17:42:44Z","receivedAt":"2006-01-11T17:42:44Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Wed, 11 Jan 2006, Simon Richter wrote:\n> \n> The important thing for me is that I need to be able to transfer them easily,\n> or turn a subdirectory into a subproject or vice versa.\n\nTurning a _snapshot_ of a subproject into a subdirectory is easy: you can \nliterally just create a subdirectory, copy it there, and it will re-use \nall the objects that the subproject uses (ie the top-level project will \nhave a \"tree\" entry that just points to the same tree entry as the \ntop-level commit in the sub-project).\n\nHowever, while that works as a way to import snapshots, it doesn't work in \nany other way. It allows you to share objects with the \"real project\", and \nit's space-efficient etc, but there's no shared history, and you cannot \nmerge back-and-forth, which is probably what you really want to do.\n\nQuite frankly, you really probably want more of a \"git-aware symlink\" kind \nof thing. I'd really hesitate (in fact, I'd object) to re-use the existing \n\"tree\" type for it, but you're not the only one to have asked for \nsubproject support, so this is clearly not a odd request.\n\n> > Sorry, we discussed similar things already. It is not necessary to change\n> > the structure. Even more: it makes no sense. Why would you want to have two\n> > or more commit messages for the same revision?\n> \n> Because the commit affects both the subproject and the master project.\n\nWhat we _could_ do is for you to first do a commit in the \"independent\" \nsubproject (it really would be a totally independent git repository in all \nways: you could continue to merge it with other subprojects of the same \ntype), and then you could commit a new pointer to that subproject in the \nmaster project. \n\nThe two would really be fundamentally independent: they'd be two different \ngit projects, one would just have a strange kind of \"symlink\" to the \nother, which would include a name and the top commit SHA1 of the other \nproject.\n\nGetting everything to work reasonably seamlessly would be potentially \npainful (getting \"git diff\" to recurse into the subdirectory correctly is \nnon-trivial: you'd have a separate \".git/index\" file for it), but it \nsounds doable.\n\nI'd suggest adding a new kind of object (\"gitlink\") which has some \nwell-specified format (20-byte SHA1 + ASCII C string \"name\" - the name \ntranslation to external repository would be done in the .git/config file \nof the \"outer\" project). Then a special file mode to indicate that in the \n\"struct tree\", and support for \"git-update-cache\" to understand how such \nan object is really tied into the \"/.git/HEAD\" file rather than \nthe rest of the directory contents.\n\nThen a \"git fetch\" would have to be taught to recursively fetch the other \nsubproject when the \"gitlink\" changes.\n\nIt should be doable: somebody could try to implement a rough first draft \n(maybe not very seamless at first).\n\n\t\tLinus\n"},{"id":"14492","messageId":"43C55FEF.2090108@hogyros.de","threadId":"3035","inReplyTo":"Pine.LNX.4.64.0601110928350.5073@g5.osdl.org","subject":"Re: RFC: Subprojects","fromName":"Simon Richter","fromEmail":"simon.richter@hogyros.de","sentAt":"2006-01-11T19:43:43Z","receivedAt":"2006-01-11T19:43:43Z","isPatch":false,"sender":{"key":"simon.richter@hogyros.de","avatar":"https://gravatar.com/avatar/1192aa9fa5dd19ce258b12b044cc27111dd7cdb58d920dc2123a02a24f55b5c5?d=mp&s=160"},"body":"Hi,\n\nLinus Torvalds wrote:\n\n> Turning a _snapshot_ of a subproject into a subdirectory is easy: you can \n> literally just create a subdirectory, copy it there, and it will re-use \n> all the objects that the subproject uses (ie the top-level project will \n> have a \"tree\" entry that just points to the same tree entry as the \n> top-level commit in the sub-project).\n\nExactly. My proposal is to allow the tree object to point to the \ntoplevel commit object directly, thus importing the entire project[1].\n\n> However, while that works as a way to import snapshots, it doesn't work in \n> any other way. It allows you to share objects with the \"real project\", and \n> it's space-efficient etc, but there's no shared history, and you cannot \n> merge back-and-forth, which is probably what you really want to do.\n\nWell, the history cannot be really shared, as turning subtrees to \nprojects and vice versa is a valid use case (and in fact something I do \npretty often in my projects, as various \"helper\" classes evolve into \n\"utility\" libraries). Since the subproject needs to be self-contained, \nthe history before it became a separate project will be difficult to \nrepresent, to say the least.\n\n> Quite frankly, you really probably want more of a \"git-aware symlink\" kind \n> of thing. I'd really hesitate (in fact, I'd object) to re-use the existing \n> \"tree\" type for it, but you're not the only one to have asked for \n> subproject support, so this is clearly not a odd request.\n\n[...]\n\n> What we _could_ do is for you to first do a commit in the \"independent\" \n> subproject (it really would be a totally independent git repository in all \n> ways: you could continue to merge it with other subprojects of the same \n> type), and then you could commit a new pointer to that subproject in the \n> master project. \n\nExactly. The questions I posed in the last paragraph of the initial \nmail, rewritten for clarity, would be\n - \"should cg-commit automatically create a commit in the master \nproject when a change in the subproject is committed?\", and\n - \"should cg-commit automatically commit all changes to subprojects \nwhen a path that has been listed on the command line contains a \nsubproject?\".\n\nThere are three cases, basically:\n\n - change to subproject, part of a larger set of changes, not ready for \nprime time: A commit in the subproject, master left alone (obviously, \nthe directory would show as \"modified\".\n - change to subproject, fixing a bug that affects the master project. \nYou'd expect these to happen often, as I could fix stuff that the master \ndoesn't care about in another tree. In this case, you'd want a new \ncommit to happen in the master as well, for everyone to enjoy.\n - change to subproject and master that need to go in sync, like \nrenaming a configure option. Obviously, this can also happen after an \nupdate of the subproject, so creating a new commit on the master after \nan update is bad, but normal behaviour for update would be to merge and \ncreate a commit instantly if there were no conflicts.\n\n> The two would really be fundamentally independent: they'd be two different \n> git projects, one would just have a strange kind of \"symlink\" to the \n> other, which would include a name and the top commit SHA1 of the other \n> project.\n\nWell, that would be exactly what the tree contains. One could do an \nadditional level of indirection with another object that just points to \nan sha1 (because its name is given by the tree referencing it).\n\nHaving such an object would mainly have the advantage of being extensible.\n\n> I'd suggest adding a new kind of object (\"gitlink\") which has some \n> well-specified format (20-byte SHA1 + ASCII C string \"name\" - the name \n> translation to external repository would be done in the .git/config file \n> of the \"outer\" project).\n\nWell, most people would likely not care about the external repository of \nthe subproject, as they can get all the objects from the master project. \nI'm not even sure the subproject needs an \"origin\" branch by default, as \nyou can push changes to the master project's maintainers who would then \neventually push them on to the subproject's maintainers (in fact I \nbelieve it's vital to be able to keep changes to the subproject in the \nmaster project so development can go on while the subproject maintainers \nreview the changes.\n\n> Then a special file mode to indicate that in the \n> \"struct tree\", and support for \"git-update-cache\" to understand how such \n> an object is really tied into the \"/.git/HEAD\" file rather than \n> the rest of the directory contents.\n\nThat sounds pretty simple actually.\n\n> Then a \"git fetch\" would have to be taught to recursively fetch the other \n> subproject when the \"gitlink\" changes.\n\nI think that would be mostly implicit if we were to use a direct \ntree->commit reference, but can be implemented trivially even for the \nlink objects.\n\n\n> It should be doable: somebody could try to implement a rough first draft \n> (maybe not very seamless at first).\n\nIndeed. I just wanted to have rough use cases thrown at me before I even \nthink of implementing something like it.\n\n Simon\n\n[1] There is a slight problem with that approach: When you cherry-pick \nchanges into a subproject (like I did today with the asm-arm/uaccess.h \nconstness fix), the subproject will have a separate branch whose head is \nknown to the outer project only. When the subproject gets merged with \nits origin later, that branch is no longer needed, and it makes a lot of \nsense to have the master project reference origin again. This means that \nif you look at the history of the inner project from the POV of the \nouter, the new commit is no longer a descendant of the old, but in fact \nit may be a good idea to attempt a fast-forward merge nevertheless as \ngoing through the common ancestor is very likely to cause conflicts \nhere, and those conflicts have already been resolved, or you wouldn't be \nseeing an updated subproject link (i.e. you just want to preserve local \nchanges and stuff committed on top of the old reference).\n"},{"id":"14497","messageId":"Pine.LNX.4.64.0601111157020.5073@g5.osdl.org","threadId":"3035","inReplyTo":"43C55FEF.2090108@hogyros.de","subject":"Re: RFC: Subprojects","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2006-01-11T20:06:45Z","receivedAt":"2006-01-11T20:06:45Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Wed, 11 Jan 2006, Simon Richter wrote:\n> \n> Exactly. The questions I posed in the last paragraph of the initial mail,\n> rewritten for clarity, would be\n> - \"should cg-commit automatically create a commit in the master project when\n> a change in the subproject is committed?\", and\n\nNo.\n\nDon't commit in the \"parent\" thing automatically. The sub-project should \nbe as independent as possible, and you should see a commit to that to be \nnothing more than \"editing\" a regular file in the top-level project.\n\nIn many ways, if you decide to look into doing a \"gitlink\" kind of object, \nthat object really _does_ conceptually point to the .git/HEAD file in the \nsubproject. So when you do a commit in the subproject, conceptually that \nis no different from editing the .git/HEAD.\n\nThen, when you want to commit the _dependency_ of the top-level project on \nthe sub-project, you commit in the top level. That commit probably does \nother things too: it probably also commits the code in the top level that \nnow depends on the sub-project changes.\n\nSo don't tie the two together any more than necessary. My suggested usage \ncase has the big advantage that the sub-project is much less tightly \ncoupled, so you can do things like \"git pull\" _inside_ the subproject, to \nupdate it, and then do a big compile in the top-level project to reflect \nthe changes (and perhaps update stuff at the top level to conform to \nchanges in the sub-project), and then commit in the top independently \n(which will now automatically pick up the changes to .git/HEAD that \"git \npull\" did on the subproject).\n\n> - \"should cg-commit automatically commit all changes to subprojects when a\n> path that has been listed on the command line contains a subproject?\".\n\nAgain, I'd really suggest not. If you keep the \"gitlink\" really meaning \nthe \".git/HEAD\" contents of the subproject (which is a good semantic \nrule), then if there is any dirty state in the sub-project, it is totally \nirrelevant to a \"git commit\". Because it's dirty, it's not part of of \n.git/HEAD yet.\n\nNow, obviously you should make \"git status\" _talk_ about the fact that the \nsub-project is dirty, so that the committer sees that the sub-project \nneeds committing first (the same way \"git status\" now informs git commit \nabout dirty files that haven't been updated).\n\n(I'm saying \"git\" here all the time rather than cg-, because not only \ndon't I know cogito very well, I think you want to do most of the core at \nthe git level, and just teach cg about the new capability).\n\n\t\t\tLinus\n"},{"id":"14530","messageId":"200601120919.08354.lan@ac-sw.com","threadId":"3035","inReplyTo":"43C52B1F.8020706@hogyros.de","subject":"Re: RFC: Subprojects","fromName":"Alexander Litvinov","fromEmail":"lan@ac-sw.com","sentAt":"2006-01-12T03:19:08Z","receivedAt":"2006-01-12T03:19:08Z","isPatch":false,"sender":{"key":"lan@ac-sw.com","avatar":null},"body":"On Wednesday 11 January 2006 21:58, Simon Richter wrote:\n> Hello,\n>\n> one thing that I have been missing so far in all SCM systems apart from\n> CVS (and there it's just coincidence) is the ability to include a\n> project as part of a bigger project. \n\nI really miss this feature. This is the last stopper for moving from CVS to \ngit for out project.\n"},{"id":"14531","messageId":"46a038f90601112046u13d7075dsc2108111e2462152@mail.gmail.com","threadId":"3035","inReplyTo":"200601120919.08354.lan@ac-sw.com","subject":"Re: RFC: Subprojects","fromName":"Martin Langhoff","fromEmail":"martin.langhoff@gmail.com","sentAt":"2006-01-12T04:46:09Z","receivedAt":"2006-01-12T04:46:09Z","isPatch":false,"sender":{"key":"martin.langhoff@gmail.com","avatar":"https://gravatar.com/avatar/1e3f311b6c4c15836501901ca58f8c0b0667246488084ba524d8bc9867e22fd9?d=mp&s=160"},"body":"On 1/12/06, Alexander Litvinov wrote:\n> On Wednesday 11 January 2006 21:58, Simon Richter wrote:\n> > Hello,\n> >\n> > one thing that I have been missing so far in all SCM systems apart from\n> > CVS (and there it's just coincidence) is the ability to include a\n> > project as part of a bigger project.\n>\n> I really miss this feature. This is the last stopper for moving from CVS to\n> git for out project.\n\nWhat about using nested checkouts? They work great with git as-is,\njust add an .gitignore file.\n\nAs Linus points out, there are many good reasons why a top-level\ncommit should _not_ commit the nested subproject. And once you are\nobserving that rule, what's left then? git status and git diff \ncan show an aggregate of top-level and nested subprojects, but that's\nease-of-use -- not something only.\n\nWhat is your show stopper?\n\ncheers,\n\n\nmartin\n"},{"id":"14532","messageId":"200601121125.33696.lan@ac-sw.com","threadId":"3035","inReplyTo":"46a038f90601112046u13d7075dsc2108111e2462152@mail.gmail.com","subject":"Re: RFC: Subprojects","fromName":"Alexander Litvinov","fromEmail":"lan@ac-sw.com","sentAt":"2006-01-12T05:25:33Z","receivedAt":"2006-01-12T05:25:33Z","isPatch":false,"sender":{"key":"lan@ac-sw.com","avatar":null},"body":"On Thursday 12 January 2006 10:46, Martin Langhoff wrote:\n> > I really miss this feature. This is the last stopper for moving from CVS\n> > to git for out project.\n>\n> What about using nested checkouts? They work great with git as-is,\n> just add an .gitignore file.\n>\n> As Linus points out, there are many good reasons why a top-level\n> commit should _not_ commit the nested subproject. And once you are\n> observing that rule, what's left then? git status and git diff \n> can show an aggregate of top-level and nested subprojects, but that's\n> ease-of-use -- not something only.\n>\n> What is your show stopper?\n\nI would agree to make separate commits for each sub project.\n\n1. I need to have ability to make tags, branches thru all subprojects.\n2. Update (pull) sould update each subproject, it is hard to update them by \nhands.\n3. The need of some sort of checkout script (can be solved by storing this \nscript in base project, but it would be much nicer allow git fetch all \nsubprojects)\n\nNothing else I can imagine.\n"},{"id":"14534","messageId":"46a038f90601112139l2f2bde5bx15102a1afcf4ec25@mail.gmail.com","threadId":"3035","inReplyTo":"200601121125.33696.lan@ac-sw.com","subject":"Re: RFC: Subprojects","fromName":"Martin Langhoff","fromEmail":"martin.langhoff@gmail.com","sentAt":"2006-01-12T05:39:10Z","receivedAt":"2006-01-12T05:39:10Z","isPatch":false,"sender":{"key":"martin.langhoff@gmail.com","avatar":"https://gravatar.com/avatar/1e3f311b6c4c15836501901ca58f8c0b0667246488084ba524d8bc9867e22fd9?d=mp&s=160"},"body":"On 1/12/06, Alexander Litvinov wrote:\n> > What is your show stopper?\n>\n> I would agree to make separate commits for each sub project.\n>\n> 1. I need to have ability to make tags, branches thru all subprojects.\n\nI suspect that this is a bad idea -- for the same reason as committing\nto a subproject is a bad idea. The subprojects most likely have their\nown external repositories -- and lifecycles of their own. The same\nheadname/branchname won't do.\n\n> 2. Update (pull) sould update each subproject, it is hard to update them by\n> hands.\n\nA simple shellscript can help you here.\n\n> 3. The need of some sort of checkout script (can be solved by storing this\n> script in base project, but it would be much nicer allow git fetch all\n> subprojects)\n\nAs you say, a bootstrapping shellscript can sort this out.\n\nSounds quite doable ;-)\n\n(have to warn you though -- git is quite addictive. there's no going back...)\n\ncheers,\n\n\nmartin\n"},{"id":"14536","messageId":"pan.2006.01.12.07.20.12.411828@progsoc.org","threadId":"3035","inReplyTo":"200601121125.33696.lan@ac-sw.com","subject":"Re: RFC: Subprojects","fromName":"Anand Kumria","fromEmail":"wildfire@progsoc.org","sentAt":"2006-01-12T07:20:13Z","receivedAt":"2006-01-12T07:20:13Z","isPatch":false,"sender":{"key":"wildfire@progsoc.org","avatar":null},"body":"On Thu, 12 Jan 2006 11:25:33 +0600, Alexander Litvinov wrote:\n\n> On Thursday 12 January 2006 10:46, Martin Langhoff wrote:\n>> > I really miss this feature. This is the last stopper for moving from CVS\n>> > to git for out project.\n>>\n>> What about using nested checkouts? They work great with git as-is,\n>> just add an .gitignore file.\n>>\n>> As Linus points out, there are many good reasons why a top-level\n>> commit should _not_ commit the nested subproject. And once you are\n>> observing that rule, what's left then? git status and git diff \n>> can show an aggregate of top-level and nested subprojects, but that's\n>> ease-of-use -- not something only.\n>>\n>> What is your show stopper?\n> \n> I would agree to make separate commits for each sub project.\n> \n> 1. I need to have ability to make tags, branches thru all subprojects.\n> 2. Update (pull) sould update each subproject, it is hard to update them by \n> hands.\n> 3. The need of some sort of checkout script (can be solved by storing this \n> script in base project, but it would be much nicer allow git fetch all \n> subprojects)\n> \n> Nothing else I can imagine.\n\nIt sounds like you want 'config-manager',\nhttp://packages.debian.org/unstable/devel/config-manager, it doesn't\nsupport git (yet) but I can't imagine it is hard to add that support in.\n\nCheers,\nAnand\n"},{"id":"14539","messageId":"200601121436.52827.lan@ac-sw.com","threadId":"3035","inReplyTo":"46a038f90601112139l2f2bde5bx15102a1afcf4ec25@mail.gmail.com","subject":"Re: RFC: Subprojects","fromName":"Alexander Litvinov","fromEmail":"lan@ac-sw.com","sentAt":"2006-01-12T08:36:52Z","receivedAt":"2006-01-12T08:36:52Z","isPatch":false,"sender":{"key":"lan@ac-sw.com","avatar":null},"body":"> > 1. I need to have ability to make tags, branches thru all subprojects.\n>\n> I suspect that this is a bad idea -- for the same reason as committing\n> to a subproject is a bad idea. The subprojects most likely have their\n> own external repositories -- and lifecycles of their own. The same\n> headname/branchname won't do.\n\nThis is one main idea of supporing subprojects. Everything else I already can \ndo. I want to be able to make tag over composite project and be able to fetch \ntagged files later. The same with branches.\n\nI cleary understand if I made tag/branch on subproject outside my composite \nproject I will not be able to work with it - this is ok.\n\nBut tag/branches on whole composite project is \"the must\".\n\nI hope it is possible to teach git (or may be something else) to scan all \nsubprojects and fetch common tags/branches and work with them.\n"},{"id":"14541","messageId":"81b0412b0601120058u6e60f009u5caf83ba574aaa7e@mail.gmail.com","threadId":"3035","inReplyTo":"200601121436.52827.lan@ac-sw.com","subject":"Re: RFC: Subprojects","fromName":"Alex Riesen","fromEmail":"raa.lkml@gmail.com","sentAt":"2006-01-12T08:58:23Z","receivedAt":"2006-01-12T08:58:23Z","isPatch":false,"sender":{"key":"raa.lkml@gmail.com","avatar":"https://avatars.githubusercontent.com/u/324101?v=4"},"body":"On 1/12/06, Alexander Litvinov wrote:\n> > > 1. I need to have ability to make tags, branches thru all subprojects.\n> >\n> > I suspect that this is a bad idea -- for the same reason as committing\n> > to a subproject is a bad idea. The subprojects most likely have their\n> > own external repositories -- and lifecycles of their own. The same\n> > headname/branchname won't do.\n>\n> This is one main idea of supporing subprojects. Everything else I already can\n> do. I want to be able to make tag over composite project and be able to fetch\n> tagged files later. The same with branches.\n>\n> I cleary understand if I made tag/branch on subproject outside my composite\n> project I will not be able to work with it - this is ok.\n>\n> But tag/branches on whole composite project is \"the must\".\n\nThe Linus' proposal of gitlink will probably help you here: gitlink will be\ntagged as well, so you just have to teach git-checkout about checking\nout subprojects.\n"},{"id":"14548","messageId":"Pine.LNX.4.64.0601120826010.25300@iabervon.org","threadId":"3035","inReplyTo":"46a038f90601112046u13d7075dsc2108111e2462152@mail.gmail.com","subject":"Re: RFC: Subprojects","fromName":"Daniel Barkalow","fromEmail":"barkalow@iabervon.org","sentAt":"2006-01-12T13:38:32Z","receivedAt":"2006-01-12T13:38:32Z","isPatch":false,"sender":{"key":"barkalow@iabervon.org","avatar":"https://avatars.githubusercontent.com/u/55364219?v=4"},"body":"On Thu, 12 Jan 2006, Martin Langhoff wrote:\n\n> What about using nested checkouts? They work great with git as-is,\n> just add an .gitignore file.\n> \n> As Linus points out, there are many good reasons why a top-level\n> commit should _not_ commit the nested subproject. And once you are\n> observing that rule, what's left then? git status and git diff \n> can show an aggregate of top-level and nested subprojects, but that's\n> ease-of-use -- not something only.\n> \n> What is your show stopper?\n\nThe core structural thing (which I'm not sure CVS handles) is having each \ncommit of the outer project specify the commit of the inner project that \nit contains in some way. This would be good with CVS, but is vital with \ngit, because there's no way of estimating it when you don't have a linear \nhistory. (With CVS, you could say that the inner project version for a \ngiven outer project version should be the version that was current when \nthe outer project was committed. But that isn't well-defined for git.) If \nyou try to debug anything involving the history, you'd have problems with \nchoosing versions of the two projects that don't actually match.\n\n\t-Daniel\n*This .sig left intentionally blank*\n"},{"id":"14649","messageId":"7vacdzkww3.fsf@assigned-by-dhcp.cox.net","threadId":"3035","inReplyTo":"Pine.LNX.4.64.0601110928350.5073@g5.osdl.org","subject":"Re: RFC: Subprojects","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2006-01-14T08:59:40Z","receivedAt":"2006-01-14T08:59:40Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Linus Torvalds writes:\n\n> I'd suggest adding a new kind of object (\"gitlink\") which has some \n> well-specified format (20-byte SHA1 + ASCII C string \"name\" - the name \n> translation to external repository would be done in the .git/config file \n> of the \"outer\" project). Then a special file mode to indicate that in the \n> \"struct tree\", and support for \"git-update-cache\" to understand how such \n> an object is really tied into the \"/.git/HEAD\" file rather than \n> the rest of the directory contents.\n>\n> Then a \"git fetch\" would have to be taught to recursively fetch the other \n> subproject when the \"gitlink\" changes.\n\nThere are two positive properties about this setup, and one\nnegative:\n\n + The contained project is kept totally independent and does\n not have to know it is contained.\n\n + The tree for the contained project can be rooted anywhere in\n the containing project's tree.\n\n - The contained project cannot be rooted at the same level or\n higher than the containing project; the containing project\n can only delegate a whole subdirectory to the contained\n project.\n\nThe \"embedded software\" example Simon originally suggested can\nbe represented with the above. I'll think aloud for a while\nhere, because I am of a slow kind who needs a more-or-less\nconcrete illustration to understand what is being discussed\n(that is primarily why I have not said anything on this topic so\nfar).\n\nThe \"containing\" project would have a handful \"gitlink\" objects\namong other things. The toplevel tree object from a commit in\nsuch a project might look like this (mode bits 0160000 is\nS_IFDIR|S_IFLNK, which is what this thing is):\n\n\t$ git ls-tree HEAD\n 0100644 blob 012345... Makefile\n 0100644 blob 123456... README\n 0160000 link 234567... gcc-4.0\n 0160000 link 345678... linux-2.6\n\t0040000 tree 456789... src\n\t$ git cat-file -t 345678\n link\n $ git cat-file link 345678\n commit 87530db5ec7d519c7ba334e414307c5130ae2da8\n\turl git://...torvalds/linux-2.6.git/\n\n The upstream Linux 2.6 repository.\n\t$ cd linux-2.6 && git-rev-parse --verify HEAD\n 87530db5ec7d519c7ba334e414307c5130ae2da8\n\nURL will be used as a suggestion for people who cloned this tree\nto set up their repository. The place and method you clone from\nLinus tree might be different, so this has to stay suggestion\nand should be overridable by the repository owner. And to help\npeople at an unusual location you could have textual comment at\nthe end, just like tags.\n\nHow would this get set up initially? Here is one way.\n\n $ git init-db\n $ edit Makefile README src/*\n\t$ git clone git://...torvalds/linux-2.6.git/ linux-2.6\n $ git clone git://.../gcc-4.0.git/ gcc-4.0\n $ link=$(echo 'The upstream Linux 2.6 repository.' |\n git-mklink linux-2.6)\n\t$ git update-index --add --cacheinfo 0160000 $link linux-2.6\n $ : ;# same for gcc-4.0\n $ git add . ;# add the rest as usual\n\t$ git commit\n\nI presume that the index file have the \"gitlink\" object just\nlike in a tree object. The usual merge rules would apply to\nthose index entries; we should be able to treat gitlinks just\nlike we handle symlinks.\n\nInteresting would be \"git checkout-index linux-2.6\" (or what\n\"git read-tree -u\" does in this \"containing\" project for\nlinux-2.6 subdirectory). After descending into linux-2.6, it\nshould not just do \"git reset --hard $commit\" for the commit\nrecorded in the gitlink (the user may have local modifications\nin the subtree). Doing \"git update-ref HEAD $commit\" there is\nnot quite right either because the index there would then need\nto be adjusted as well. Perhaps the real core level commands\nsuch as \"checkout-index\" and \"read-tree -u\" should fail when the\nsubproject tree is dirty, just like \"read-tree -m old new\" does\nnot always have to succeed.\n\nWhat does \"git-diff-index/git-diff-tree/git-diff-files\" would do\nwith them?\n\n\t$ git-diff-files linux-2.6\n\nwould compare the commit recorded in the link and what is\nchecked out in the linux-2.6/.git/HEAD and report that\ndifference. So do other git-diff-* siblings. At the core level\nwe do not have to recurse and look at linux-2.6/.git/index (we\nmay end up doing so at the end, I dunno; initially we said at\nthe core level we do not have to generate patches but we ended\nup having -p option go all of git-diff-* siblings).\n\nFetching/cloning at the core level is easy. \"git-fetch-pack\"\nwould just need to do one level, but Porcelains need to address\nhow to actually arrange the subprojects cloning to happen, which\nis harder.\n\n\"git clone\" would say: \"Ah, now I see these gitlinks; we need to\nclone them. linux-2.6 directory needs to be populated with\ncommit 87530d from git://...torvalds/linux-2.6.git/ repository.\nWould this work for you, or would you use different mirror?\"\nand then it clones the repository and sets linux-2.6/.git/HEAD\nto the named commit and does a checkout. The URL used for this\nactual subcloning would need to be stored somewhere in $GIT_DIR/,\nperhaps in config as you suggested. I do not think we need a\nseparate name for it -- we can probably say \"linux-2.6\" for this\n(i.e. use the pathname itself as the key).\n\nWhat happens if the containing project wants to move these\ngitlinks (or remove them)? When checking out such a commit with\n\"git-read-tree -u\", would the subproject directory be wiped out\n(again, such a \"read-tree\" would be prevented if it would result\nin information loss)?\n\nAll of this sounds quite a lot of change with brittleness.\n\n\nNow I'll think aloud about a completely different design.\n\nWe could simply overlay the projects. I think this is what\nJohannes suggested earlier.\n\nYou keep one branch for each \"subproject\", and make commits into\neach branch (i.e. if you modified files for the upstream kernel,\nthe change is committed to the branch for linux-2.6 subproject),\nbut when checking things out, you do an equivalent of octopus\nmerge across subprojects.\n\nOne downside of this approach is we cannot re-root the\nsubprojects until we update read-tree and write-tree, but I\nsuspect that would be a lot smaller change. Once that is done,\nwe could:\n\n $ git init-db\n $ mkdir linux-2.6\n $ H=$(git-fetch-pack -k git://...torvalds/linux-2.6.git/ master)\n $ echo $H >.git/refs/heads/kernel\n $ : ;# same for gcc-4.0\n $ cat .git/config < + The contained project is kept totally independent and does\n> not have to know it is contained.\n> \n> + The tree for the contained project can be rooted anywhere in\n> the containing project's tree.\n\nRight.\n\n> - The contained project cannot be rooted at the same level or\n> higher than the containing project; the containing project\n> can only delegate a whole subdirectory to the contained\n> project.\n\nYes.\n\nHowever, I think this is actually a _huge_ advantage.\n\nThe thing is, if you do the contained projects as \"union projects\" as you \nsuggest, I will bet that it will really really suck, because it ends up \nlosing the two positives above.\n\nIn particular, any real independent project will have it's own \"Makefile\" \nor \"configure-in\", and often its own \"src\" subdirectory or other \npseudo-standard names.\n\nAnd the \"contained project as a link\" approach has zero problems with that \nat all, exactly because it keeps the projects clearly separate - just \nlinked (one way).\n\n> What does \"git-diff-index/git-diff-tree/git-diff-files\" would do\n> with them?\n\nI would actually argue that git itself wouldn't do a whole lot with them. \nThere are real advantages to seeing only the diffs wrt _one_ of the \nprojects, and I'd argue that\n\n\tgit-diff-*\n\nwould actually act like they now act for directories that they don't \nrecurse into, ie you'd see something like\n\n\t:160000 160000 5eb57670... 3f1a42aa... M\tsub-project\n\nand it would be up to higher-level porcelain to recurse.\n\nWhy? Partly because that's actually likely enough for a lot of users: you \n_can_ use just the raw git programs by just doing\n\n\tcd sub-project\n\tgit diff\n\t..\n\tgit commit\n\nand so technically you aren't really missing a lot. The capabilities are \nthere, you just have to do some more by hand (but in many ways that is \n_good_: it makes it obvious that you're really committing a _different_ \nsubproject).\n\nThe other reason? A lot of the git infrastructure really does only work on \nthe \"one project\" level. The programs work with _one_ index, not two. \nReading two trees is perfectly possible, but unless you keep them in \nseparate stages, you can't separate them afterwards. IOW, trying to be \nrecursive really does end up being a big change, for very little gain (and \nfor a lot of potential bugs and instability).\n\nIn contrast, doing it at a higher level means that you have a simple and \nreliable lower level that you can trust. Layering is good.\n\n> Fetching/cloning at the core level is easy. \"git-fetch-pack\"\n> would just need to do one level, but Porcelains need to address\n> how to actually arrange the subprojects cloning to happen, which\n> is harder.\n> \n> \"git clone\" would say: \"Ah, now I see these gitlinks; we need to\n> clone them.\n\nActually, I would say no - that's actually not a \"clone\" operation so much \nas a \"checkout\" operation. There are strong arguments that you should \n_not_ clone sub-projects when you clone the top-level project: there's no \nreason to. Anybody else who clones it will have all the information you \nhave, so cloning th esub-project is just extra work.\n\nSo only if you actually check it out (which is often in practice the \nsecond stage of the cloning, of course) do you want to fetch the \nsubproject too. But even then you might want to ask the user (he may have \na local repository for that sub-project somewhere else, so going to the \n\"canonical name\" might be the wrong thing to do - and he might not even \ncare, because he might want to work _just_ on the top-level project).\n\n> Now I'll think aloud about a completely different design.\n> \n> We could simply overlay the projects. I think this is what\n> Johannes suggested earlier.\n> \n> You keep one branch for each \"subproject\", and make commits into\n> each branch (i.e. if you modified files for the upstream kernel,\n> the change is committed to the branch for linux-2.6 subproject),\n> but when checking things out, you do an equivalent of octopus\n> merge across subprojects.\n\nI think this one has serious disadvantages:\n\n - it's much less obvious when there are common names and especially \n common subdirectories.\n - in _practice_, almost all sub-projects are kept in sub-directories. Are \n you doing to change the sub-project git tree? How are you going to \n merge back to the original sub-project?\n - iow, I think this only works for sub-projects that are totally \n controlled by the top-level project - in which case they might as well \n just be totally merged into the top level (the way we did with the \n \"tools\" project, and largely with \"gitk\").\n\nin the \"gitk\" case, we could actually continue to keep gitk a separate \nproject, but that was really fortunate: it's purely because gitk ends up \nbeing a single file, with no Makefile at all to build it independently \netc. The moment we integrated the \"tools\" sub-project into git, we lost \nthe ability to do that, exactly because they now needed to share Makefiles \netc, making all further development very inter-twined.\n\nPut another way: the moment you have linkages going both ways between the \nsubproject and the top-level project, it's no longer two separate \nprojects. At that point, it in practice becomes one, since the sub-project \ncan no longer do independent development without merging becoming a big \nissue.\n\nThe advantage of having a \"git link\" is exactly the fact that the \ndependency goes only one way. The subproject remains truly independent.\n\n\t\t\tLinus\n"},{"id":"14661","messageId":"43C951B6.5030607@gmail.com","threadId":"3035","inReplyTo":"Pine.LNX.4.64.0601141055210.13339@g5.osdl.org","subject":"Re: RFC: Subprojects","fromName":"A Large Angry SCM","fromEmail":"gitzilla@gmail.com","sentAt":"2006-01-14T19:32:06Z","receivedAt":"2006-01-14T19:32:06Z","isPatch":false,"sender":{"key":"gitzilla@gmail.com","avatar":"https://gravatar.com/avatar/354625c442439908ff3dd99757dee330e29e9df7847472384faf7a00add247fb?d=mp&s=160"},"body":"So far I've not seen any convincing arguments why the sub-projects can \nnot be managed by the Makefile, or equivalent, of the super-project. \nParticularly when the sub-projects have a life of their own.\n"},{"id":"14662","messageId":"Pine.LNX.4.64.0601141154590.13339@g5.osdl.org","threadId":"3035","inReplyTo":"43C951B6.5030607@gmail.com","subject":"Re: RFC: Subprojects","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2006-01-14T20:02:37Z","receivedAt":"2006-01-14T20:02:37Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Sat, 14 Jan 2006, A Large Angry SCM wrote:\n>\n> So far I've not seen any convincing arguments why the sub-projects can not be\n> managed by the Makefile, or equivalent, of the super-project. Particularly\n> when the sub-projects have a life of their own.\n\nNow, from a developer standpoint I actually agree with you. I find \nsub-projects totally useless - I'm much happier just having separate \ntrees.\n\nThe advantage (as far as I can tell) of sub-projects is not that they are \neasier to develop in, but that it's a total nightmare for the technical \n_user_ to download ten different projects from ten different sites, and \nconfigure them properly and install them in the right order, and keep them \nup-to-date.\n\nThere are projects that I simply gave up even trying to track: I wasn't \ninterested in being a developer per se, but I _was_ interested in trying \nto test and give feedback to the current development tree - but it was \njust too damn confusing to get it working.\n\nIf I could have just done a \"git clone \" to get it all, I'd \nhave been a much more productive user.\n\nThis is why I think sub-projects are more about \"git checkout\" and an \nautomated \"git fetch\" than anything else. Doing actual development etc you \ncan easily do one project at a time. \"git diff\" and \"git commit\" wouldn't \nneed any real ability to recurse into subprojects and try to make it \nseamless. And if you do a \"git pull\" that needs to do anything but \nfast-forward, you might as well resolve the sub-projects one by one.\n\n\t\tLinus\n"},{"id":"14663","messageId":"7vek3ah8f9.fsf@assigned-by-dhcp.cox.net","threadId":"3035","inReplyTo":"Pine.LNX.4.64.0601141055210.13339@g5.osdl.org","subject":"Re: RFC: Subprojects","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2006-01-14T20:16:26Z","receivedAt":"2006-01-14T20:16:26Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Linus Torvalds writes:\n\n> On Sat, 14 Jan 2006, Junio C Hamano wrote:\n>\n> The thing is, if you do the contained projects as \"union projects\" as you \n> suggest, I will bet that it will really really suck, because it ends up \n> losing the two positives above.\n\nAfter a good night's sleep, I agree. I have not thought things\nthrough and still have a feeling that (feasibilities aside) it\nwould be interesting if we can do a \"union projects\" a la \"union\nmounts\" (or translucent filesystem). But that \"interesting\"\nthing would probably not be very useful in practice.\n\n> would actually act like they now act for directories that they don't \n> recurse into, ie you'd see something like\n>\n> \t:160000 160000 5eb57670... 3f1a42aa... M\tsub-project\n>\n> and it would be up to higher-level porcelain to recurse.\n\nThis I agree with.\n\n> The other reason? A lot of the git infrastructure really does only work on \n> the \"one project\" level. The programs work with _one_ index, not two. \n> Reading two trees is perfectly possible, but unless you keep them in \n> separate stages, you can't separate them afterwards. IOW, trying to be \n> recursive really does end up being a big change, for very little gain (and \n> for a lot of potential bugs and instability).\n\nYup. BTW, I think with a couple of minor tweaking and giving it\nthe same restriction (\"two pluses and one negative\") as the\ngitlink proposal, the \"union\" approach would work equally well,\nperhaps with a simpler implementation. I'll think aloud about\nthis at the end.\n\n>> Fetching/cloning at the core level is easy. \"git-fetch-pack\"\n>> would just need to do one level, but Porcelains need to address\n>> how to actually arrange the subprojects cloning to happen, which\n>> is harder.\n>...\n> So only if you actually check it out (which is often in practice the \n> second stage of the cloning, of course) do you want to fetch the \n> subproject too.\n\nWe are in complete agreement here.\n\n> I think this one has serious disadvantages:\n>\n> - it's much less obvious when there are common names and especially \n> common subdirectories.\n> - in _practice_, almost all sub-projects are kept in sub-directories. Are \n> you doing to change the sub-project git tree? How are you going to \n> merge back to the original sub-project?\n> - iow, I think this only works for sub-projects that are totally \n> controlled by the top-level project - in which case they might as well \n> just be totally merged into the top level (the way we did with the \n> \"tools\" project, and largely with \"gitk\").\n\nYes, I agree to the above 100%; the serious disadvantages come\nfrom the fact that we do not have clear separation between\nsubprojects -- which new files belong to what subproject. I\nthink re-rooting read-tree and write-tree would help solving\nthat. After I wrote the message you are replying to, I came up\nwith a couple of tweaks.\n\n - Do the octopus-like thing, but always give subprojects a\n separate directories to work in.\n\n - Extend \"commit\" objects for the toplevel project to record\n what subprojects with what head commits are contained at\n which subdirectory. I wrote in the previous message to make\n subprojects heads parents of aggregate commits, but I think\n that one without \"where to\" information has a serious\n disadvantage when computing a merge.\n\nIn the \"embedded linux\" example that has \"linux-2.6\" and\n\"gcc-4.0\" projects as an externally controlled subprojects, and\nhas all the rest (including the toplevel Makefile) in \"master\"\nbranch:\n\n $ tar xf embed.tar embed && cd embed && git init-db\n $ git add . ;# toplevel Makefile and stuff\n $ git commit -a -m 'embedded repo - initial'\n\nAfter doing \"git-fetch-pack -k git://.../linux-2.6.git/ master\"\nand \"echo $H >.git/refs/heads/kernel\" (similar for gcc-4.0) to\nset up the branch heads (but we do not have any working tree\nfiles for these subprojects yet):\n\n\t$ git bind -m 'Bind kernel and gcc into us' \\\n \tkernel=linux-2.6 gcc=gcc-4.0\n\nwould prepare the subprojects binding (I am just looking for a\nbetter word --- I called it \"setup-overlay\" in the previous\nmessage). This would:\n\n - append the tree object in \"kernel\" commit object to the\n current index, rerooted at linux-2.6/; similar for \"gcc\" at\n gcc-4.0/. We may need a new mode and option for read-tree for\n this, or we may not. Internally this step would be scripted\n in \"git bind\" wrapper like this:\n\n\tgit read-tree --bind --prefix=linux-2.6 kernel\n\tgit read-tree --bind --prefix=gcc-4.0 gcc\n\n and would result in an index file that has these trees\n \"mounted\" at specified places. If you look at only the index\n file, you cannot tell this is an overlay, unlike gitlink\n scheme.\n\n - make a commit that records the tree object (the whole thing\n including the subproject trees), with the initial commit we\n made earlier as the sole parent commit, and additionally\n records the two subproject heads with bind points. This\n happens in the same \"git bind\" wrapper, and produces\n something like:\n\n\t$ git cat-file commit HEAD\n tree e9de76f2e141824439caa00a65e3b91d05d125c9\n parent bfca932434cc65e7aa90794e7c4d66f75d00b16a\n bind a8fe7257b8427d31cfcca0aa336335bb43689fc9 linux-2.6\n bind b3b2df23226634f42c9646bd7961fbea8b00f914 gcc-4.0\n author Junio C Hamano 1137205528 -0800\n committer Junio C Hamano 1137205528 -0800\n\n\tBind kernel and gcc into us.\n\n \"bind\" line needs to be taught to fsck-objects. The format\n is the object name of the commit followed by (c-style quoted)\n subdirectory name.\n\n - record the branch name vs subproject directory binding in\n $GIT_DIR/ somewhere, say $GIT_DIR/mtab ;-).\n\n\t$ cat .git/mtab\n\tkernel\tlinux-2.6\n gcc\tgcc-4.0\n\nAfter this, \"git checkout-index -f -q -u -a\" would populate the\nwhole thing. Instead of linux-2.6/.git/HEAD as in gitlink\nexample, I am using .git/refs/heads/kernel; this would not make\na semantic difference. One big difference however is I have\nonly one index file that controls the whole tree, without using\na separate linux-2.6/.git/index.\n\nAfter mucking with a file in linux-2.6/ subdirectory and nowhere\nelse, committing the result from the whole tree would work like\nthis:\n\n - Look at the current commit and notice the bind for two\n subdirectories; then look them up in $GIT_DIR/mtab to see\n which branches keep track of them.\n\n - Notice that there are modified paths in the index vs tree\n from the last commit under linux-2.6/ directory.\n\n - Write out only that part, re-rooted, into a tree.\n\n\tgit write-tree --prefix=linux-2.6\n\n - Make a commit to record that tree, with a parent set to the\n \"kernel\" branch head; update the \"kernel\" branch head at that\n commit.\n\n - Make another commit to record the tree made from the whole\n index (obviously linux-2.6 subdirectory would result in the\n same tree object we just committed in the subproject) with\n parent set to .git/HEAD and bind adjusted accordingly; update\n the \"HEAD\".\n\nNow I have to think about clones and merges but this is getting\ntoo long so I'll leave it to a separate message.\n"},{"id":"14665","messageId":"43C95F69.7090200@gmail.com","threadId":"3035","inReplyTo":"Pine.LNX.4.64.0601141154590.13339@g5.osdl.org","subject":"Re: RFC: Subprojects","fromName":"A Large Angry SCM","fromEmail":"gitzilla@gmail.com","sentAt":"2006-01-14T20:30:33Z","receivedAt":"2006-01-14T20:30:33Z","isPatch":false,"sender":{"key":"gitzilla@gmail.com","avatar":"https://gravatar.com/avatar/354625c442439908ff3dd99757dee330e29e9df7847472384faf7a00add247fb?d=mp&s=160"},"body":"Linus Torvalds wrote:\n> \n> On Sat, 14 Jan 2006, A Large Angry SCM wrote:\n>>So far I've not seen any convincing arguments why the sub-projects can not be\n>>managed by the Makefile, or equivalent, of the super-project. Particularly\n>>when the sub-projects have a life of their own.\n> \n> Now, from a developer standpoint I actually agree with you. I find \n> sub-projects totally useless - I'm much happier just having separate \n> trees.\n> \n> The advantage (as far as I can tell) of sub-projects is not that they are \n> easier to develop in, but that it's a total nightmare for the technical \n> _user_ to download ten different projects from ten different sites, and \n> configure them properly and install them in the right order, and keep them \n> up-to-date.\n> \n> There are projects that I simply gave up even trying to track: I wasn't \n> interested in being a developer per se, but I _was_ interested in trying \n> to test and give feedback to the current development tree - but it was \n> just too damn confusing to get it working.\n> \n> If I could have just done a \"git clone \" to get it all, I'd \n> have been a much more productive user.\n\n$ make get_sub_components\n\nThis can work with most any SCM (depending on your environment), is \namazingly flexible, and does not require special support in the SCM.\n\nThe \"get\" rule for each sub-project could be something like:\n\n\tgit_sub-project:\n\t\tmkdir sub-project\n\t\tcd sub-project\n\t\tgit-init-db\n\t\tgit-fetch \n\t\tgit-checkout \n\t\t$(MAKE) get_sub_components\n\n> \n> This is why I think sub-projects are more about \"git checkout\" and an \n> automated \"git fetch\" than anything else. Doing actual development etc you \n> can easily do one project at a time. \"git diff\" and \"git commit\" wouldn't \n> need any real ability to recurse into subprojects and try to make it \n> seamless. And if you do a \"git pull\" that needs to do anything but \n> fast-forward, you might as well resolve the sub-projects one by one.\n\nAnd all of this can be done today, without changing git, with more \nflexibility, with Make rules.\n"},{"id":"14669","messageId":"7vk6d2fsu6.fsf@assigned-by-dhcp.cox.net","threadId":"3035","inReplyTo":"43C95F69.7090200@gmail.com","subject":"Re: RFC: Subprojects","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2006-01-14T20:38:25Z","receivedAt":"2006-01-14T20:38:25Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"A Large Angry SCM writes:\n\n>> If I could have just done a \"git clone \" to get it all,\n>> I'd have been a much more productive user.\n>\n> $ make get_sub_components\n>\n> This can work with most any SCM (depending on your environment), is\n> amazingly flexible, and does not require special support in the SCM.\n\nI am with you two on this one, in principle, as a developer.\n\n> The \"get\" rule for each sub-project could be something like:\n>\n> \tgit_sub-project:\n> \t\tmkdir sub-project\n> \t\tcd sub-project\n> \t\tgit-init-db\n> \t\tgit-fetch \n> \t\tgit-checkout \n> \t\t$(MAKE) get_sub_components\n\nThere lies a drake here --- is not the same for\neverybody. It is not a big showstopper dragon, though.\n"},{"id":"14675","messageId":"46a038f90601141628n2ec32e8fy7fc23d8d7884c0f2@mail.gmail.com","threadId":"3035","inReplyTo":"7vk6d2fsu6.fsf@assigned-by-dhcp.cox.net","subject":"Re: RFC: Subprojects","fromName":"Martin Langhoff","fromEmail":"martin.langhoff@gmail.com","sentAt":"2006-01-15T00:28:58Z","receivedAt":"2006-01-15T00:28:58Z","isPatch":false,"sender":{"key":"martin.langhoff@gmail.com","avatar":"https://gravatar.com/avatar/1e3f311b6c4c15836501901ca58f8c0b0667246488084ba524d8bc9867e22fd9?d=mp&s=160"},"body":"On 1/15/06, Junio C Hamano wrote:\n> > The \"get\" rule for each sub-project could be something like:\n> >\n> > git_sub-project:\n> > mkdir sub-project\n> > cd sub-project\n> > git-init-db\n> > git-fetch \n> > git-checkout \n> > $(MAKE) get_sub_components\n>\n> There lies a drake here --- is not the same for\n> everybody. It is not a big showstopper dragon, though.\n\nWell, that /little complication/ applies to doing it in git too ;-)\nThere's no way to tell how the dev doing the top level checkout has\naccess to the subproject repos.\n\nI am with gitzilla on this one. Let the projects have their own\nbootstraping mechanisms, using make, ant or whatever catches their\nfancy. One of the great things about git is that it doesn't assume\nthat it's being used by all the projects in the world -- thanks to\nLinus' disregard for arbitrary metadata and to your git-cherry\nimplementation, it's all about the content -- and so it interoperates\ngreat with Arch, SVN, CVS, etc.\n\nHaving intra-git subproject support assumes that the subprojects are\nall in git. Heh! That covers about 0.001% of reality out there.\nPer-project bootstraping scripts will use whatever tools they need for\nthe checkout.\n\nAutomating the 'checkout' stage for git subprojects is trivial, and\nI'd argue not interesting enough to try and solve within git,\nspecially when most subprojects are going to be using a different SCM\nanyway. And all the *interesting* operations (branch, commit, tag) are\nperhaps indeed interesting problems to solve, but definite misfeatures\nin a tool that tries to be sane and minimalistic.\n\nIOWs, adding some repo metadata describing subprojects is the wrong\nthing to do, just like tracking patches via metadata would be the\nwrong thing to do. It's all about files -- which git handles\nmasterfully.\n\ncheers,\n\n\nmartin\n"},{"id":"14676","messageId":"7v4q4671tg.fsf@assigned-by-dhcp.cox.net","threadId":"3035","inReplyTo":"46a038f90601141628n2ec32e8fy7fc23d8d7884c0f2@mail.gmail.com","subject":"Re: RFC: Subprojects","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2006-01-15T00:49:15Z","receivedAt":"2006-01-15T00:49:15Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Martin Langhoff writes:\n\n> I am with gitzilla on this one. Let the projects have their own\n> bootstraping mechanisms, using make, ant or whatever catches their\n> fancy. One of the great things about git is that it doesn't assume\n> that it's being used by all the projects in the world -- thanks to\n> Linus' disregard for arbitrary metadata and to your git-cherry\n> implementation, it's all about the content -- and so it interoperates\n> great with Arch, SVN, CVS, etc.\n\nI had the exactly the same reaction when I saw the project\nbundling facility of Arch (tla 1.0 -- I do not know what the\nnewer versions use). It probably was a great way to tie two or\nmore Arch projects together, but it would quickly become less\nuseful once the component project is outside Arch space and the\ntoplevel project would end up with doing some Makefile targets\nlike ALASCM described.\n\nI hope this settles this issue and nobody would bring up \"Wee\nwant subprojects\" ever again ;-).\n"},{"id":"14677","messageId":"7vfynq484b.fsf@assigned-by-dhcp.cox.net","threadId":"3035","inReplyTo":"7vek3ah8f9.fsf@assigned-by-dhcp.cox.net","subject":"Re: RFC: Subprojects","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2006-01-15T01:01:24Z","receivedAt":"2006-01-15T01:01:24Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Continuing with the \"union\" approach...\n\nJunio C Hamano writes:\n\n> - append the tree object in \"kernel\" commit object to the\n> current index, rerooted at linux-2.6/; similar for \"gcc\" at\n> gcc-4.0/. We may need a new mode and option for read-tree for\n> this, or we may not. Internally this step would be scripted\n> in \"git bind\" wrapper like this:\n>\n> \tgit read-tree --bind --prefix=linux-2.6 kernel\n> \tgit read-tree --bind --prefix=gcc-4.0 gcc\n>\n> and would result in an index file that has these trees\n> \"mounted\" at specified places...\n\nClarification. By \"mounted\", I mean 'without affecting existing\nindex entries, create index entries from the tree, with all the\npaths have \"linux-2.6/\" prefixed to them'.\n\n> - record the branch name vs subproject directory binding in\n> $GIT_DIR/ somewhere, say $GIT_DIR/mtab ;-).\n>\n> \t$ cat .git/mtab\n> \tkernel\tlinux-2.6\n> gcc\tgcc-4.0\n\nI now realize this needs to be something like:\n\n\tmaster\tkernel=linux-2.6/ gcc=gcc-4.0/\n\nthat is, \"when on branch master, bind these two heads at these\ndirectories\", to allow switching to another branch and switching\nback to this branch. And the file should probably be called\n$GIT_DIR/modules, to parallel CVSROOT/modules file.\n\n>\t$ git cat-file commit HEAD\n> tree e9de76f2e141824439caa00a65e3b91d05d125c9\n> parent bfca932434cc65e7aa90794e7c4d66f75d00b16a\n> bind a8fe7257b8427d31cfcca0aa336335bb43689fc9 linux-2.6\n> bind b3b2df23226634f42c9646bd7961fbea8b00f914 gcc-4.0\n> author Junio C Hamano 1137205528 -0800\n> committer Junio C Hamano 1137205528 -0800\n>\n>\tBind kernel and gcc into us.\n>...\n> Now I have to think about clones and merges but this is getting\n> too long so I'll leave it to a separate message.\n\nThe core-level cloning would just \"clone\" the objects, treating\n\"bind\" line in the commit just like \"parent\" to pull necessary\nobjects.\n\nCheckout would involve the usual read-tree -u which extracts the\ntree (which is the whole tree, with files of the subprojects in\nit), and notices \"bind\" lines are there but there are no\nmatching $GIT_DIR/modules entries for those directories.\nProbably it would create $GIT_DIR/refs/heads/bind/a8fe725 for\nthe linux-2.6 subproject (what the original committer called\n\"kernel\" branch), and similarly for the gcc-4.0 subproject, add\nan appropriate entry to $GIT_DIR/modules file. The user would\nthen rename the branch names and optionally arrange remotes/\nfiles to update the bound branches appropriately:\n\n\t$ mv .git/refs/heads/bind/a8fe725 .git/refs/heads/kernel\n\nNow, let's say this \"master\" branch is checked out, and somehow\nthe \"kernel\" branch gets updated. That is, the commit recorded\non the \"bind\" line of the HEAD commit does not match the branch\nhead that can be found out via $GIT_DIR/modules file. This will\nnot happen if you are committing into the \"master\" branch using\nthe \"commit to subprojects and then to the toplevel project\"\nmechanism yourself, but it would happen if the \"kernel\" branch\nwas moved by \"git fetch\" fast-forwarding, or if you switched to\nthe \"kernel\" branch (which would essentially remove everything\nfrom your tree, and checkout the kernel source at the root\nlevel, not in linux-2.6/ subdirectory), did an upstream merge\nyourself, and switched back to the \"master\" branch.\n\nTo keep the problem simpler, let's say we only deal with the\ncase where \"kernel\" branch head is a fast-forward of what is on\n\"bind\" in the HEAD commit of \"master\" branch. Then \"checkout\"\nneeds to notice it, and check out the subdirectory from the\n\"kernel\" branch head (*not* using the object name on \"bind\"\nline).\n\nSo the outline of the \"checkout\" would be like this:\n\n * Read commit object from new HEAD.\n\n * For each \"bind\" line:\n\n If the subdirectory does not have a corresponding branch,\n create one in $GIT_DIR/refs/heads/bind/; record it in\n $GIT_DIR/modules for the new branch (otherwise leave branch\n as is).\n\n Make sure the commit recorded on \"bind\" line is an ancestor\n of the branch head. Otherwise it is an error and checkout is\n prevented until the \"kernel\" branch is resolved to be a\n descendant of it.\n\n Run \"read-tree -u --prefix=\" to merge in the subtree into the\n index, and update the working tree.\n\nAt this point, there may be mismatch between the tree in the\nHEAD and the working tree files and index, when subproject\ncommit recorded on the \"bind\" line is different from the\ncorresponding subproject branch head, and \"git diff\" would show\nit. When making a commit here, the \"subproject and then\ntoplevel\" commit scheme I described earlier would record the\ncurrent \"kernel\" branch head on the \"bind\" line in the new\ncommit, along with the tree object that contains the tree from\n\"kernel\" branch head commit as a subtree.\n\n\nAbout \"merge\", we should be able to do this:\n\n\t$ git checkout master ;# the whole mess\n $ git pull -b kernel git://..torvalds/linux-2.6.git/\n\nthat is, 'pull from this URL but into \"kernel\" branch not to the\ncurrent branch'. Independent of this \"subprojects\" topic,\nmerging in a separate temporary directory into non-current\nbranch is something we have talked about some time ago, and in\nthis particular case, instead of using a throw-away temporary\ndirectory, we have a pre-made directory to do the merge already,\nso let's say that is solved elsewhere first. Once we have that,\nthe above \"checkout\" would be able to integrate the result into\nthe \"master\" project.\n"},{"id":"14678","messageId":"20060115015550.GD9672@socrates","threadId":"3035","inReplyTo":"7v4q4671tg.fsf@assigned-by-dhcp.cox.net","subject":"Re: RFC: Subprojects","fromName":"Tom Prince","fromEmail":"tom.prince@ualberta.net","sentAt":"2006-01-15T01:55:50Z","receivedAt":"2006-01-15T01:55:50Z","isPatch":false,"sender":{"key":"tom.prince@ualberta.net","avatar":"https://gravatar.com/avatar/a0ad19caee7618876339485106ec994f5202505eecd210ba5c0bd869feaa555a?d=mp&s=160"},"body":"On Sat, Jan 14, 2006 at 04:49:15PM -0800, Junio C Hamano wrote:\n> Martin Langhoff writes:\n> \n> > I am with gitzilla on this one. Let the projects have their own\n> > bootstraping mechanisms, using make, ant or whatever catches their\n> > fancy. One of the great things about git is that it doesn't assume\n> > that it's being used by all the projects in the world -- thanks to\n> > Linus' disregard for arbitrary metadata and to your git-cherry\n> > implementation, it's all about the content -- and so it interoperates\n> > great with Arch, SVN, CVS, etc.\n> \n> \n> I hope this settles this issue and nobody would bring up \"Wee\n> want subprojects\" ever again ;-).\n> \n\nBut since we can import everything into a GIT repository, and have\n(some) tools for pushing changes back, we can pretend that it is being\nused for every project in the world.\n\n Tom\n"},{"id":"14684","messageId":"20060115150721.GE28365@pasky.or.cz","threadId":"3035","inReplyTo":"43C52B1F.8020706@hogyros.de","subject":"[RFC][PATCH] Cogito support for simple subprojects","fromName":"Petr Baudis","fromEmail":"pasky@suse.cz","sentAt":"2006-01-15T15:07:21Z","receivedAt":"2006-01-15T15:07:21Z","isPatch":true,"sender":{"key":"pasky@ucw.cz","avatar":"https://avatars.githubusercontent.com/u/18439?v=4"},"body":" Hello,\n\n I've tried to take a different approach - KISS and don't make the\nsubprojects part of the git-tracked tree but a thing purely local to\nyour particular checkout. Subprojects are simply listed in\n.git/subprojects and various commands are called recursively on them.\n\n - No auto-cloning of subprojects is possible.\n - Switching between branches and merging is troublesome in case the\n required subprojects arrangement changes inbetween. Now, this is\n a matter of taste - I don't see this as a huge problem since you\n can make special provisions for that and this should be going to be\n so rare that it's not worth optimizing for in my eyes.\n + It's simple.\n + It's flexible. You can have _optional_ subprojects - IIRC e.g.\n mplayer can use ffmpeg if it's checked out as a subproject but\n will use own copy if it's not there. You can clone subprojects\n based e.g. on selected features before compilation. If you do so,\n your GIT won't bother you with uncommitted local changes, and you\n will not have to filter this out from any other changes you are\n going to commit.\n\n The main goal of this is simply to be able to check out bunch of\nstuff to subdirectories and make cg-update update all of it, without\nany special scripts. ;-)\n\n This patch is just trivial proof-of-concept thing which makes only\ncg-update and cg-fetch aware of the subprojects; many commands still\nneed to be taught about subprojects but some don't - I currently don't\nthink a recursive cg-merge is a particularily good idea, for one.\nI think the good default is to make all read-only commands by default\nrecursive and all modifying commands by default non-recursive. (And\nit might be useful to be able to mark some subprojects read-only.)\n\n How to create a subproject? Simply cg-clone inside a working copy,\nit will register it automagically. So far there are no tools to further\nmaintain the subprojects, though, therefore a mv or rm needs to be\nfollowed by an appropriate modification of parent .git/subprojects.\n\n\n This is not committed yet - I'm curious about your opinions.\n\ndiff --git a/TODO b/TODO\nindex 0658b39..29daf30 100644\n--- a/TODO\n+++ b/TODO\n@@ -90,23 +90,13 @@ cg-*patch should be pre-1.0.)\n \n * cg-Xfetchprogress showing smooth progress for packfiles\n \n+* Enhance subprojects notion\n+\tSo far the subprojects support is trivial and prone to user error.\n+\tE.g. cg-add should check if it doesn't poke into a subproject,\n+\tcg-status should list subprojects, etc.\n \n-Post 1.0:\n-\n-* Subprojects\n-\tSupport a GIT project inside a GIT project:\n \n-\t\tx/.git\n-\t\tx/foo/bar/.git\n-\t\tx/foo/bar/baz/.git\n-\t\tx/quux/zot/.git\n-\n-\tThat means cg-update working recursively and cg-add'n'stuff\n-\tchecking if there isn't another .git along the path of its\n-\targument.\n-\n-\tNeeds more thought, especially wrt. fetching and merging\n-\trecursive semantics.\n+Post 1.0:\n \n * Comfortable cg-log\n \tProbably make it a real terminal application, not just less\ndiff --git a/cg-Xlib b/cg-Xlib\nindex 46a8a73..6e6265e 100755\n--- a/cg-Xlib\n+++ b/cg-Xlib\n@@ -197,6 +197,10 @@ list_untracked_files()\n \t\tif [ -f \"$EXCLUDEFILE\" ]; then\n \t\t\tEXCLUDE[${#EXCLUDE[@]}]=\"--exclude-from=$EXCLUDEFILE\"\n \t\tfi\n+\t\tEXCLUDEFILE=\"$_git/subprojects\"\n+\t\tif [ -f \"$EXCLUDEFILE\" ]; then\n+\t\t\tEXCLUDE[${#EXCLUDE[@]}]=\"--exclude-from=$EXCLUDEFILE\"\n+\t\tfi\n \t\t# This is just for compatibility (2005-09-16).\n \t\t# To be removed later.\n \t\tEXCLUDEFILE=\"$_git/exclude\"\n@@ -209,6 +213,29 @@ list_untracked_files()\n \tgit-ls-files -z --others \"${EXCLUDE[@]}\"\n }\n \n+# Usage: subprojects_recurse ACTIONNAME COMMAND...\n+# Run command recursively on subprojects, displaying warning using the\n+# ACTIONNAME string in case any of them failed.\n+subprojects_recurse()\n+{\n+\t[ -s \"$_git/subprojects\" ] || return 0\n+\tlocal failures=0 subprj= actionname=\"$1\" s=\n+\tlocal Actionname=\"$(echo \"$actionname\" | perl -pe '$_=ucfirst')\"\n+\tshift\n+\twhile IFS= read -r subprj; do\n+\t\techo \"Running $actionname in $subprj...\" >&2\n+\t\tif ( cd \"$subprj\" && \"$@\" ); then\n+\t\t\techo \"$Actionname in $subprj succeeded.\" >&2\n+\t\telse\n+\t\t\techo \"$Actionname in $subprj failed!\" >&2\n+\t\t\tfailures=$(($failures+1))\n+\t\tfi\n+\tdone <\"$_git/subprojects\"\n+\tlocal s=; if [ $failures -gt 1 ]; then s=s; fi\n+\t[ $failures -gt 0 ] && echo \"Warning: $failures subproject $actionname$s failed\" >&2\n+\treturn $failures\n+}\n+\n # Usage: showdate SECONDS TIMEZONE [FORMAT]\n # Display date nicely based on how GIT stores it.\n # Save the date to $_showdate\ndiff --git a/cg-clone b/cg-clone\nindex f86a548..dfb6dc8 100755\n--- a/cg-clone\n+++ b/cg-clone\n@@ -11,6 +11,15 @@\n # parameter is omitted, the basename of the source repository is used as the\n # destination.\n #\n+# If you are cloning inside another working tree, you are automatically\n+# establishing a subproject - that means that when you will update the\n+# parent project, this project will be auto-updated as well, and in the\n+# future, certain other operations may also recurse to subprojects. Use the\n+# -P option to prevent this from becoming a subproject. Also please note\n+# that the subprojects support is preliminary and subject to change. If you\n+# remove the subproject later, you must also remove the corresponding entry\n+# in '.git/subprojects' for now.\n+#\n # OPTIONS\n # -------\n # -l::\tSymlink the object database when cloning locally\n@@ -24,20 +33,28 @@\n #\tNote that you MUST NOT prune repository containing a symlink\n #\tor being symlinked to.\n #\n+# -P::\tPrevent this from becoming a subproject\n+#\tIf this option is passed, cg-clone will never create a subproject\n+#\teven if called inside working tree of another project.\n+#\n # -s::\tClone into the current directory\n #\tClone in the current directory instead of creating a new one.\n #\tSpecifying both -s and a destination directory makes no sense.\n+#\tThis also implies -P.\n \n-USAGE=\"cg-clone [-l] [-s] LOCATION [DESTDIR]\"\n+USAGE=\"cg-clone [-l] [-P] [-s] LOCATION [DESTDIR]\"\n _git_repo_unneeded=1\n \n . ${COGITO_LIB}cg-Xlib || exit 1\n \n same_dir=\n symlink=\n+may_subproject=1\n while optparse; do\n \tif optparse -l; then\n \t\tsymlink=1\n+\telif optparse -P; then\n+\t\tmay_subproject=\n \telif optparse -s; then\n \t\tsame_dir=1\n \telse\n@@ -65,7 +82,13 @@ else\n \tlocation=\"$location\"\n fi\n \n+parentprj=\n+parentpath=\n if [ ! \"$same_dir\" ]; then\n+\tif [ \"$may_subproject\" ]; then\n+\t\tparentprj=\"$(git-rev-parse --git-dir 2>/dev/null)\"\n+\t\tparentpath=\"$(git-rev-parse --show-prefix 2>/dev/null)$dir\"\n+\tfi\n \t[ -e \"$dir\" ] && die \"$dir/ already exists\"\n \tmkdir \"$dir\" || exit $?\n \tcd \"$dir\" || exit $?\n@@ -94,4 +117,14 @@ cp \"$_git/refs/heads/origin\" \"$_git/refs\n \tgit-update-index --refresh ||\n \texit 1\n \n+if [ \"$parentprj\" ]; then\n+\tcd ..\n+\tparentroot=\"${parentprj%.git}\"\n+\tif [ -z \"$parentroot\" ]; then\n+\t\tparentroot=\"$(pwd)\"\n+\tfi\n+\techo \"Registering as a subproject of $parentroot...\"\n+\techo \"$parentpath\" >>\"$parentprj\"/subprojects\n+fi\n+\n echo \"Cloned to $dir/ (origin $location available as branch \\\"origin\\\")\"\ndiff --git a/cg-fetch b/cg-fetch\nindex c3dfb81..9d44d9f 100755\n--- a/cg-fetch\n+++ b/cg-fetch\n@@ -28,6 +28,10 @@\n # -f:: Force the complete fetch even if the heads are the same.\n #\tForce the complete fetch even if the heads are the same.\n #\n+# -R:: Do not recurse to subprojects\n+#\tDo not recursively fetch the subprojects (see the `cg-clone`\n+#\tdocumentation for more information).\n+#\n # -v:: Enable verbosity\n #\tDisplay more verbose output - most notably list all the files\n #\ttouched by the fetched changes.\n@@ -53,7 +57,7 @@\n #\twon't unpack the transferred pack.\n \n \n-USAGE=\"cg-fetch [-f] [-v] [BRANCH_NAME]\"\n+USAGE=\"cg-fetch [-f] [-R] [-v] [BRANCH_NAME]\"\n \n . ${COGITO_LIB}cg-Xlib || exit 1\n deprecated_alias cg-fetch cg-pull\n@@ -234,12 +238,15 @@ fetch_tags()\n \n \n recovery=\n+recurse=1\n verbose=\n while optparse; do\n \tif optparse -f; then\n \t\t# When forcing, let the fetch tools make more extensive\n \t\t# walk over the dependency tree with --recover.\n \t\trecovery=--recover\n+\telif optparse -R; then\n+\t\trecurse=\n \telif optparse -v; then\n \t\tverbose=1\n \telse\n@@ -248,9 +255,10 @@ while optparse; do\n done\n \n name=\"${ARGS[0]}\"\n-\n [ \"$name\" ] || { [ -s \"$_git/branches/origin\" ] && name=origin; }\n+[ \"$recurse\" ] && subprojects_recurse \"fetch\" cg-fetch $recovery $verbose \"$name\"\n [ \"$name\" ] || die \"where to fetch from?\"\n+\n uri=$(cat \"$_git/branches/$name\" 2>/dev/null) || die \"unknown branch: $name\"\n \n rembranch=\ndiff --git a/cg-update b/cg-update\nindex 1d6e0a0..1b21338 100755\n--- a/cg-update\n+++ b/cg-update\n@@ -25,23 +25,30 @@\n # -f:: Force the complete fetch even if the heads are the same.\n #\tForce the complete fetch even if the heads are the same.\n #\n+# -R:: Do not recurse to subprojects\n+#\tDo not recursively update the subprojects (see the `cg-clone`\n+#\tdocumentation for more information).\n+#\n # --squash:: Use \"squash\" merge to record pending commits as a single merge commit\n #\t\"Squash\" merge - condense all the to-be-merged commits to a single\n #\tmerge commit. This is not to be used lightly; see the cg-merge\n #\tdocumenation for further details.\n \n-USAGE=\"cg-update [-f] [--squash] [BRANCH_NAME]\"\n+USAGE=\"cg-update [-f] [-R] [--squash] [BRANCH_NAME]\"\n _git_requires_root=1\n \n . ${COGITO_LIB}cg-Xlib || exit 1\n \n force=\n squash=\n+recurse=1\n while optparse; do\n \tif optparse -f; then\n \t\tforce=-f\n \telif optparse --squash; then\n \t\tsquash=--squash\n+\telif optparse -R; then\n+\t\trecurse=\n \telse\n \t\toptfail\n \tfi\n@@ -49,13 +56,14 @@ done\n \n name=\"${ARGS[0]}\"\n [ \"$name\" ] || { [ -s \"$_git/branches/origin\" ] && name=origin; }\n+[ \"$recurse\" ] && subprojects_recurse \"update\" cg-update $force $squash \"$name\"\n [ \"$name\" ] || die \"where to update from?\"\n \n # cg-merge can do better decision about fast-forwarding if it sees this.\n [ -s \"$_git/refs/heads/$name\" ] && export _cg_orig_head=\"$(cat \"$_git/refs/heads/$name\")\"\n \n if [ -s \"$_git/branches/$name\" ]; then\n-\tcg-fetch $force \"$name\" || exit 1\n+\tcg-fetch -R $force \"$name\" || exit 1\n else\n \techo \"Updating from a local branch.\"\n fi\ndiff --git a/t/t9215-update-recursive.sh b/t/t9215-update-recursive.sh\nnew file mode 100755\nindex 0000000..d0e27a4\n--- /dev/null\n+++ b/t/t9215-update-recursive.sh\n@@ -0,0 +1,54 @@\n+#!/usr/bin/env bash\n+#\n+# Copyright (c) 2005 Petr Baudis\n+#\n+test_description=\"Tests recursive cg-update functionality\n+\n+Create a subproject and then try to cg-update.\"\n+\n+. ./test-lib.sh\n+\n+mkdir prj1\n+echo file >prj1/file\n+test_expect_success 'initialize project 1' \\\n+\t\"(cd prj1 && cg-init -I && cg-add file && cg-commit -C -m\\\"Initial commit\\\")\"\n+\n+mkdir prj2\n+echo file >prj2/FILE\n+test_expect_success 'initialize project 2' \\\n+\t\"(cd prj2 && cg-init -I && cg-add FILE && cg-commit -C -m\\\"Initial commit\\\")\"\n+\n+test_expect_success 'clone project 1' \\\n+\t\"cg-clone prj1 clone\"\n+test_expect_success 'clone project 2 as subproject of project 1' \\\n+\t\"(cd clone && cg-clone ../prj2)\"\n+\n+test_expect_success 'commit to project 1' \\\n+\t\"(cd prj1 && echo foo >>file && cg-commit -m\\\"Commit in project 1\\\")\"\n+test_expect_success 'commit to project 2' \\\n+\t\"(cd prj2 && echo bar >>FILE && cg-commit -m\\\"Commit in project 2\\\")\"\n+\n+test_expect_success 'non-recursive update' \\\n+\t\"(cd clone && cg-update -R)\"\n+test_expect_success 'check if the prj1 head was updated in clone' \\\n+\t\"(cmp prj1/.git/refs/heads/master clone/.git/refs/heads/master)\"\n+test_expect_success 'check if the prj1 working copy was updated in clone' \\\n+\t\"(cmp prj1/file clone/file)\"\n+test_expect_failure 'check if the prj2 head was not updated in clone' \\\n+\t\"(cmp prj2/.git/refs/heads/master clone/prj2/.git/refs/heads/origin)\"\n+\n+test_expect_success 'commit to project 1' \\\n+\t\"(cd prj1 && echo baz >>file && cg-commit -m\\\"Commit in project 1\\\")\"\n+\n+test_expect_success 'recursive update' \\\n+\t\"(cd clone && cg-update)\"\n+test_expect_success 'check if the prj1 head was updated in clone' \\\n+\t\"(cmp prj1/.git/refs/heads/master clone/.git/refs/heads/master)\"\n+test_expect_success 'check if the prj1 working copy was updated in clone' \\\n+\t\"(cmp prj1/file clone/file)\"\n+test_expect_success 'check if the prj2 head was updated in clone' \\\n+\t\"(cmp prj2/.git/refs/heads/master clone/prj2/.git/refs/heads/master)\"\n+test_expect_success 'check if the prj2 working copy was updated in clone' \\\n+\t\"(cmp prj2/FILE clone/prj2/FILE)\"\n+\n+test_done\n\n\n-- \n\t\t\t\tPetr \"Pasky\" Baudis\nStuff: http://pasky.or.cz/\nOf the 3 great composers Mozart tells us what it's like to be human,\nBeethoven tells us what it's like to be Beethoven and Bach tells us\nwhat it's like to be the universe. -- Douglas Adams\n"},{"id":"14685","messageId":"Pine.LNX.4.64.0601150937500.13339@g5.osdl.org","threadId":"3035","inReplyTo":"20060115150721.GE28365@pasky.or.cz","subject":"Re: [RFC][PATCH] Cogito support for simple subprojects","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2006-01-15T17:38:14Z","receivedAt":"2006-01-15T17:38:14Z","isPatch":true,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Sun, 15 Jan 2006, Petr Baudis wrote:\n> \n> I've tried to take a different approach - KISS and don't make the\n> subprojects part of the git-tracked tree but a thing purely local to\n> your particular checkout. Subprojects are simply listed in\n> .git/subprojects and various commands are called recursively on them.\n\nThis seems very sane. Goodie.\n\n\t\tLinus\n"},{"id":"14687","messageId":"7vk6d1480z.fsf@assigned-by-dhcp.cox.net","threadId":"3035","inReplyTo":"20060115150721.GE28365@pasky.or.cz","subject":"Re: [RFC][PATCH] Cogito support for simple subprojects","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2006-01-15T19:15:40Z","receivedAt":"2006-01-15T19:15:40Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Petr Baudis writes:\n\n> ... I currently don't\n> think a recursive cg-merge is a particularily good idea, for one.\n> I think the good default is to make all read-only commands by default\n> recursive and all modifying commands by default non-recursive. (And\n> it might be useful to be able to mark some subprojects read-only.)\n\nAll sounds sane and simple. Good job!\n\nEspecially because this does not even try to let the project\nexpress its version dependencies on its subprojects, I like it\nfor its simplicity (which makes it very easy to explain, to my\nmind that is the biggest plus). However, I fear that others\nmight complain to say that the contained things do not deserve\nto be called \"subprojects\" if there is no version linkage [*1*].\n\nI think something like this is greatly helpful for people (like\nme) as \"an end user who builds from source out of SCM not from\ntarballs.\"\n\nIf you go this route, one minor concern is what the format of\nthe $GIT_DIR/subproject should be (and if they do not deserve to\nbe called \"subproject\" then what the name of the file be),\nbecause this may likely be a \"nice if they were compatible\" item\nacross Porcelains. The list of names separated with LF has two\nvery big pluses:\n\n - it is certainly the easiest to parse.\n\n - it can readily be used as --exclude-from file, as long as you\n do not care about another directory with a same name\n somewhere other than the subproject itself.\n\nEven if the limitation to the latter becomes a real issue, a\nseparate exclude-from file could easily be generated on the fly\n(prefix them with '/' to force \"not anywhere in the subtree but\nonly here\" matching, perhaps with escaping shell glob pattern\nwhile you are at it), which does not sound so bad.\n\n\n[Footnote]\n\n*1* I do not personally care about version linkage; this is me\nbeing lazy to avoid core side support ;-). Yesterday I was\nmucking with rev-list code to see how gitlink and/or bind commit\nwould affect what it needs to do (especially wrt its --objects\nflag), and I did not like the potential code impact I saw there.\n"},{"id":"14707","messageId":"Pine.LNX.4.64.0601152248030.25300@iabervon.org","threadId":"3035","inReplyTo":"46a038f90601141628n2ec32e8fy7fc23d8d7884c0f2@mail.gmail.com","subject":"Re: RFC: Subprojects","fromName":"Daniel Barkalow","fromEmail":"barkalow@iabervon.org","sentAt":"2006-01-16T05:06:18Z","receivedAt":"2006-01-16T05:06:18Z","isPatch":false,"sender":{"key":"barkalow@iabervon.org","avatar":"https://avatars.githubusercontent.com/u/55364219?v=4"},"body":"On Sun, 15 Jan 2006, Martin Langhoff wrote:\n\n> On 1/15/06, Junio C Hamano wrote:\n> > > The \"get\" rule for each sub-project could be something like:\n> > >\n> > > git_sub-project:\n> > > mkdir sub-project\n> > > cd sub-project\n> > > git-init-db\n> > > git-fetch \n> > > git-checkout \n> > > $(MAKE) get_sub_components\n> >\n> > There lies a drake here --- is not the same for\n> > everybody. It is not a big showstopper dragon, though.\n> \n> Well, that /little complication/ applies to doing it in git too ;-)\n> There's no way to tell how the dev doing the top level checkout has\n> access to the subproject repos.\n> \n> I am with gitzilla on this one. Let the projects have their own\n> bootstraping mechanisms, using make, ant or whatever catches their\n> fancy. One of the great things about git is that it doesn't assume\n> that it's being used by all the projects in the world -- thanks to\n> Linus' disregard for arbitrary metadata and to your git-cherry\n> implementation, it's all about the content -- and so it interoperates\n> great with Arch, SVN, CVS, etc.\n\nBut most of the content of the project that started this thread is the \nrevisions of the subprojects. Sure, it could all be done in the build \nsystem, but then it becomes impractical to manage. Git could refuse to \nsupport tracking the executable bit on files, or what directories things \nare in, and we could tell people to use their build systems to set these \nthings, but it would make the tool impractical to use. We want to track \nsome metadata, because it's actually important; what we don't want to \ntrack is the metadata that is local to the particular working tree. That's \nwhy we track only one executable bit, not a full set of permissions; it's \na matter of local policy who can interact with the files in a working \ntree, but it's part of the content whether a file is executable.\n\nSo the problem with handling subprojects with the build system is that it \nis too tempting to use the revision control system directly on the \nsubproject, at which point the thing you're developing and testing isn't \nat all what other people will get if they check out your commit. You want \n\"git status\" to report it as an uncommitted change if you have a different \nrevision of the subproject than your previous commit had, and it can't \ntell if this information is buried in the build system.\n\nI like Linus's proposal: which revision of which project goes where is \npart of the content, while how you manipulate data for that project is a \nmatter of local policy, and is not tracked, although it might be a good \nidea to let project provide overridable defaults (so that, if you're a \nrandom member of the general public and don't have a special method for \naccessing the repository, you don't have to track it down yourself).\n\nThe tricky question is whether we should permit the \"subproject\" objects \nto specify a revision that isn't a hash, for use in identifying revisions \nof subprojects in other systems.\n\n\t-Daniel\n*This .sig left intentionally blank*\n"},{"id":"14711","messageId":"200601161328.04985.lan@ac-sw.com","threadId":"3035","inReplyTo":"7vacdzkww3.fsf@assigned-by-dhcp.cox.net","subject":"Re: RFC: Subprojects","fromName":"Alexander Litvinov","fromEmail":"lan@ac-sw.com","sentAt":"2006-01-16T07:28:04Z","receivedAt":"2006-01-16T07:28:04Z","isPatch":false,"sender":{"key":"lan@ac-sw.com","avatar":null},"body":"On Saturday 14 January 2006 14:59, Junio C Hamano wrote:\n> Now I'll think aloud about a completely different design.\n>\n> We could simply overlay the projects.  I think this is what\n> Johannes suggested earlier.\n>\n> You keep one branch for each \"subproject\", and make commits into\n> each branch (i.e. if you modified files for the upstream kernel,\n> the change is committed to the branch for linux-2.6 subproject),\n> but when checking things out, you do an equivalent of octopus\n> merge across subprojects.\nIf I cleary understand this idea it is NOT that I dreaming about. Almost all \nour sub-projects are used in more than one project (imaging network layer \nlibrary). So variant with gitlink is that I willing.\n"},{"id":"14713","messageId":"81b0412b0601152348x1a8e7dy19dce8fcdd9b812b@mail.gmail.com","threadId":"3035","inReplyTo":"Pine.LNX.4.64.0601141154590.13339@g5.osdl.org","subject":"Re: RFC: Subprojects","fromName":"Alex Riesen","fromEmail":"raa.lkml@gmail.com","sentAt":"2006-01-16T07:48:12Z","receivedAt":"2006-01-16T07:48:12Z","isPatch":false,"sender":{"key":"raa.lkml@gmail.com","avatar":"https://avatars.githubusercontent.com/u/324101?v=4"},"body":"On 1/14/06, Linus Torvalds wrote:\n> > So far I've not seen any convincing arguments why the sub-projects can not be\n> > managed by the Makefile, or equivalent, of the super-project. Particularly\n> > when the sub-projects have a life of their own.\n>\n> Now, from a developer standpoint I actually agree with you. I find\n> sub-projects totally useless - I'm much happier just having separate\n> trees.\n>\n> The advantage (as far as I can tell) of sub-projects is not that they are\n> easier to develop in, but that it's a total nightmare for the technical\n> _user_ to download ten different projects from ten different sites, and\n> configure them properly and install them in the right order, and keep them\n> up-to-date.\n>\n> There are projects that I simply gave up even trying to track: I wasn't\n> interested in being a developer per se, but I _was_ interested in trying\n> to test and give feedback to the current development tree - but it was\n> just too damn confusing to get it working.\n>\n> If I could have just done a \"git clone \" to get it all, I'd\n> have been a much more productive user.\n>\n> This is why I think sub-projects are more about \"git checkout\" and an\n> automated \"git fetch\" than anything else. Doing actual development etc you\n> can easily do one project at a time. \"git diff\" and \"git commit\" wouldn't\n> need any real ability to recurse into subprojects and try to make it\n> seamless. And if you do a \"git pull\" that needs to do anything but\n> fast-forward, you might as well resolve the sub-projects one by one.\n\nThat is exactly how subprojects are used in Perforce- and ClearCase-like SCM:\nthe working tree is \"configured\" to contain the super-project (build\nconfiguration)\nand the actual work happens in the subproject and _only_ there. The mentioned\nsystems even have heavily used permission system just to prevent either\ncheckout or commit anywhere outside the area of responsibility of a developer.\n(The \"permissions\" are somehow pointless in git context, just mentioned them\nto underline the main point).\n"},{"id":"14722","messageId":"43CB7283.3090003@op5.se","threadId":"3035","inReplyTo":"200601161328.04985.lan@ac-sw.com","subject":"Re: RFC: Subprojects","fromName":"Andreas Ericsson","fromEmail":"ae@op5.se","sentAt":"2006-01-16T10:16:35Z","receivedAt":"2006-01-16T10:16:35Z","isPatch":false,"sender":{"key":"ae@op5.se","avatar":"https://gravatar.com/avatar/426e89595c75a8f5252dd0c989e5fabe5bcac616e68557427ad9aef6b0ca342a?d=mp&s=160"},"body":"Alexander Litvinov wrote:\n> On Saturday 14 January 2006 14:59, Junio C Hamano wrote:\n> \n>>Now I'll think aloud about a completely different design.\n>>\n>>We could simply overlay the projects. I think this is what\n>>Johannes suggested earlier.\n>>\n>>You keep one branch for each \"subproject\", and make commits into\n>>each branch (i.e. if you modified files for the upstream kernel,\n>>the change is committed to the branch for linux-2.6 subproject),\n>>but when checking things out, you do an equivalent of octopus\n>>merge across subprojects.\n> \n> If I cleary understand this idea it is NOT that I dreaming about. Almost all \n> our sub-projects are used in more than one project (imaging network layer \n> library). So variant with gitlink is that I willing.\n\n\nThen it isn't so much a subproject as a separate project of its own. \nOtherwise glibc would be a subproject of pretty much everything and \nthat's hardly a sane setup.\n\n-- \nAndreas Ericsson andreas.ericsson@op5.se\nOP5 AB www.op5.se\nTel: +46 8-230225 Fax: +46 8-230231\n"},{"id":"14725","messageId":"200601161144.48245.Josef.Weidendorfer@gmx.de","threadId":"3035","inReplyTo":"7vek3ah8f9.fsf@assigned-by-dhcp.cox.net","subject":"Re: RFC: Subprojects","fromName":"Josef Weidendorfer","fromEmail":"josef.weidendorfer@gmx.de","sentAt":"2006-01-16T10:44:48Z","receivedAt":"2006-01-16T10:44:48Z","isPatch":false,"sender":{"key":"josef.weidendorfer@gmx.de","avatar":null},"body":"On Saturday 14 January 2006 21:16, you wrote:\n> Yes, I agree to the above 100%; the serious disadvantages come\n> from the fact that we do not have clear separation between\n> subprojects -- which new files belong to what subproject. I\n> ...\n> - Extend \"commit\" objects for the toplevel project to record\n> what subprojects with what head commits are contained at\n> which subdirectory.\n\nThe suggested \"bind\" info in commit objects has the same problem\nas the original overlay: if the superproject already has a\nsubdirectory kernel/, and there is an additional \"bind\" specification\nin commits also for kernel/, what should be done?\n\nSo the gitlink object seems to be the only solution if we want to\nbind git versions of subprojects into a superproject.\n\nBut as this seems to make everything quite complex and not-obvious for\na user, I am with Paskys simple subproject idea.\n\nJosef\n"},{"id":"14740","messageId":"43CBEF47.7050607@gmail.com","threadId":"3035","inReplyTo":"Pine.LNX.4.64.0601152248030.25300@iabervon.org","subject":"Re: RFC: Subprojects","fromName":"A Large Angry SCM","fromEmail":"gitzilla@gmail.com","sentAt":"2006-01-16T19:08:55Z","receivedAt":"2006-01-16T19:08:55Z","isPatch":false,"sender":{"key":"gitzilla@gmail.com","avatar":"https://gravatar.com/avatar/354625c442439908ff3dd99757dee330e29e9df7847472384faf7a00add247fb?d=mp&s=160"},"body":"Daniel Barkalow wrote:\n[...]\n> \n> So the problem with handling subprojects with the build system is that it \n> is too tempting to use the revision control system directly on the \n> subproject, at which point the thing you're developing and testing isn't \n> at all what other people will get if they check out your commit. You want \n> \"git status\" to report it as an uncommitted change if you have a different \n> revision of the subproject than your previous commit had, and it can't \n> tell if this information is buried in the build system.\n\nUsing \"git-status\" is the wrong tool to use there. What you should be \nusing is \"make project_status\". Claiming \"that it is too tempting to use \nthe revision control system on the subproject\" is wrong; you should use \nthe SCM (of the subproject) to manage the subproject. You use the build \nsystem to manage the _entire_ project.\n\n> I like Linus's proposal: which revision of which project goes where is \n> part of the content, while how you manipulate data for that project is a \n> matter of local policy, and is not tracked, although it might be a good \n> idea to let project provide overridable defaults (so that, if you're a \n> random member of the general public and don't have a special method for \n> accessing the repository, you don't have to track it down yourself).\n\nI think Linus' proposal is an attempt to solve the problem in the wrong \nplace; it encumbers the SCM with features of limited applicability, that \nimpose a specific methodology on how to handle subprojects, and requires \nthat the SCM of the subproject be Git.\n\n\n> The tricky question is whether we should permit the \"subproject\" objects \n> to specify a revision that isn't a hash, for use in identifying revisions \n> of subprojects in other systems.\n\nWhy would you want to limit how required versions of subprojects are \nspecified? Your project policies and procedures may require that \nsubprojects be specified by a subproject SCM specific immutable revision \nbut the policies and procedures of other projects may not be so \nrestrictive and could accept a tag identifying the latest \"stable\" (or \nsomething) revision.\n\n--\n"},{"id":"14744","messageId":"Pine.LNX.4.64.0601161414080.25300@iabervon.org","threadId":"3035","inReplyTo":"43CBEF47.7050607@gmail.com","subject":"Re: RFC: Subprojects","fromName":"Daniel Barkalow","fromEmail":"barkalow@iabervon.org","sentAt":"2006-01-16T20:20:39Z","receivedAt":"2006-01-16T20:20:39Z","isPatch":false,"sender":{"key":"barkalow@iabervon.org","avatar":"https://avatars.githubusercontent.com/u/55364219?v=4"},"body":"On Mon, 16 Jan 2006, A Large Angry SCM wrote:\n\n> Daniel Barkalow wrote:\n> [...]\n> > \n> > So the problem with handling subprojects with the build system is that it is\n> > too tempting to use the revision control system directly on the subproject,\n> > at which point the thing you're developing and testing isn't at all what\n> > other people will get if they check out your commit. You want \"git status\"\n> > to report it as an uncommitted change if you have a different revision of\n> > the subproject than your previous commit had, and it can't tell if this\n> > information is buried in the build system.\n> \n> Using \"git-status\" is the wrong tool to use there. What you should be using is\n> \"make project_status\". Claiming \"that it is too tempting to use the revision\n> control system on the subproject\" is wrong; you should use the SCM (of the\n> subproject) to manage the subproject. You use the build system to manage the\n> _entire_ project.\n\nI'm talking about using \"git status\" on the main project, in case you're \nmisunderstanding me. If you can manage the entire project with the build \nsystem, then you don't need git or any version control at all, aside from \nyour build system. But you'd also lose the ability to use webgit, bisect, \ngitk, git log, and so forth on the project as a whole.\n\n> > The tricky question is whether we should permit the \"subproject\" objects to\n> > specify a revision that isn't a hash, for use in identifying revisions of\n> > subprojects in other systems.\n> \n> Why would you want to limit how required versions of subprojects are\n> specified? Your project policies and procedures may require that subprojects\n> be specified by a subproject SCM specific immutable revision but the policies\n> and procedures of other projects may not be so restrictive and could accept a\n> tag identifying the latest \"stable\" (or something) revision.\n\nIf you accept a tag identifying the latest stable revision, then you might \nas well not bother. The point of revision controlling a project is to be \nable to reconstruct previous states. If you allow any event, especially \noutside, unrelated, events to change the reconstructed state for a \nrevision, then this is not the case. Your normal debugging situation will \nbe \"It's broken, and I didn't change anything.\" because someone somewhere \nelse changed something, and you have no record of what last worked. And \nyou can obviously forget any hope of \"git bisect\" working.\n\n\t-Daniel\n*This .sig left intentionally blank*\n"},{"id":"14746","messageId":"7vek37rj83.fsf@assigned-by-dhcp.cox.net","threadId":"3035","inReplyTo":"200601161144.48245.Josef.Weidendorfer@gmx.de","subject":"Re: RFC: Subprojects","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2006-01-16T20:49:48Z","receivedAt":"2006-01-16T20:49:48Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Josef Weidendorfer writes:\n\n> The suggested \"bind\" info in commit objects has the same problem\n> as the original overlay: if the superproject already has a\n> subdirectory kernel/, and there is an additional \"bind\" specification\n> in commits also for kernel/, what should be done?\n>\n> So the gitlink object seems to be the only solution if we want to\n> bind git versions of subprojects into a superproject.\n\nIn \"pu\", I have some of the necessary basic pieces for \"bind\"\napproach, barely enough so that anybody interested could start\nprototyping using them as building blocks. It still has very\nrough edges; the missing includes rev-list and fsck-objects, so\nyou cannot do a send-pack or fetch-pack yet.\n\nYesterday I was working on \"gitlink\" approach to have similar\ncore-side support for prototyping. I haven't finished it into a\nbuildable state yet (it is not in \"pu\"), and I am pessimistic if\nI ever will X-<.\n\nI think the updated \"bind\" thing makes the two approaches\nsemantically equivalent (i.e. it does not allow an arbitrary\noverlayed setup anymore). We simply do not allow the\nconflicting \"bind\". So neither is the _only_ solution. We\nprobably could make both to work, but the details differ.\n\n * With \"gitlink\", the index of containing project never has\n subprojects parts of the tree, which I see it as an advantage\n compared to what \"bind\" does. It only has one \"gitlink\"\n entry per each subproject. update-index, read-tree,\n ls-files, diff-*, etc. needs to be aware of \"gitlink\" object.\n Especially tricky is read-tree. It needs to treat a\n \"gitlink\" object as a directory for D/F conflict detection\n purposes, but treat it similar to blobs in most other aspects\n (e.g. results in one entry in the index). The stat\n information update-index and diff-files uses for quick\n up-to-date check needs to be taught not to worry about the\n stat information of the subdirectory a \"gitlink\" object\n points at (e.g. if you do a whole-tree build, the timestamp\n of the directory would change, but that does not mean the\n subtree is dirty). tree/directory traversal code needs to be\n aware of \"gitlink\" and stop there. This approach involves\n quite a lot of code changes, mostly because what is in the\n current index never correspond to a directory on the\n filesystem but \"gitlink\" quacks like a directory.\n\n * With \"bind\", the index of containing project keeps the entire\n tree structure, including subproject part. In fact, there is\n no other separate index for the subproject part.\n\n An updated write-tree in \"pu\" can write a tree for only the\n subproject part with \"write-tree --prefix=/\" from such\n an index file, and read-tree can read with \"read-tree\n --prefix=/\" to graft a subproject tree on top of the\n current index contents. Without the --prefix, write-tree\n writes out the whole thing for a commit for the containing\n project, so if somebody cloned that superproject, getting the\n whole tree out in order to \"make\" is just the matter of doing\n a regular \"read-tree && checkout-index\".\n\n We could introduce \"bind the rest\" to make write-tree write\n out a tree that contains only the containing project part and\n not any of the subproject part (e.g. Makefile, README and\n src/ but not linux-2.6/ nor gcc-4.0/ in the earlier example).\n Essentially the contents of such a tree object would be the\n same as what \"gitlink\" approach would have had for the\n containing project in the index file, minus \"gitlink\" entries\n themselves). This is not so surprising, because the missing\n information \"gitlink\" approach recorded in the tree object\n itself is expressed on \"bind\" lines in the commit object with\n this approach.\n\nAn advantage with the \"bind\" approach, from the implementation\npoint of view, is that none of the \"index vs working tree\" part\nof the core needs to be modified (you would notice that many\nissues I had with trying \"gitlink\" I listed above are \"index vs\nworking tree\" issues). \"tree object vs index\" part needed to be\nenhanced somewhat (e.g. the re-rooting read-tree/write-tree with\nthe --prefix option) but it was not too painful.\n"},{"id":"14748","messageId":"43CC1D3E.1070700@gmail.com","threadId":"3035","inReplyTo":"Pine.LNX.4.64.0601161414080.25300@iabervon.org","subject":"Re: RFC: Subprojects","fromName":"A Large Angry SCM","fromEmail":"gitzilla@gmail.com","sentAt":"2006-01-16T22:25:02Z","receivedAt":"2006-01-16T22:25:02Z","isPatch":false,"sender":{"key":"gitzilla@gmail.com","avatar":"https://gravatar.com/avatar/354625c442439908ff3dd99757dee330e29e9df7847472384faf7a00add247fb?d=mp&s=160"},"body":"Daniel Barkalow wrote:\n> On Mon, 16 Jan 2006, A Large Angry SCM wrote:\n> \n>>Daniel Barkalow wrote:\n>>[...]\n>>>So the problem with handling subprojects with the build system is that it is\n>>>too tempting to use the revision control system directly on the subproject,\n>>>at which point the thing you're developing and testing isn't at all what\n>>>other people will get if they check out your commit. You want \"git status\"\n>>>to report it as an uncommitted change if you have a different revision of\n>>>the subproject than your previous commit had, and it can't tell if this\n>>>information is buried in the build system.\n>>Using \"git-status\" is the wrong tool to use there. What you should be using is\n>>\"make project_status\". Claiming \"that it is too tempting to use the revision\n>>control system on the subproject\" is wrong; you should use the SCM (of the\n>>subproject) to manage the subproject. You use the build system to manage the\n>>_entire_ project.\n> \n> I'm talking about using \"git status\" on the main project, in case you're \n> misunderstanding me. If you can manage the entire project with the build \n> system, then you don't need git or any version control at all, aside from \n> your build system. But you'd also lose the ability to use webgit, bisect, \n> gitk, git log, and so forth on the project as a whole.\n\nWhen you say \"main project\", do you mean the top level project and all \nof its subprojects? Or just the top level project without any of its \nsubprojects?\n\nA build system and a SCM working together can be a powerful very tool, \neven if one or both of the components is not. Since managing a project \nwith a build system and no SCM means that you don't have any history and \nsince you seem to want access to the project's history, I'll ignore you \nstatement. (Backups and directory snapshots are primitive SCMs.)\n\nConsider the following:\n\n1) For a project to be a sub-project, it must also be a project.\n\n2) So the standard SCM tools will work on the project.\n\n3) The super-project is also a project and the standard SCM tools will \nwork on it.\n\n4) Projects managed by an SCM are only considered consistent and usable \nat specific points in that project's history; it may be every recorded \npoint in it's history or it may be just a few specific recorded points.\n\n5) A super-project only cares about specific states of its sub-projects, \ncorresponding to points in the sub-project's history.\n\n6) For each sub-project, the super-project needs to record the \nsub-project's state identifier for each recorded point in the \nsuper-project's history.\n\n7) The super-project can record each sub-project's state identifier \nsomewhere in the build system.\n\n8) If the super-project's SCM is Git then webgit, bisect, gitk, git log, \nand so forth all work and will identify when and where the recorded \nstate identifier of each sub-project is changed.\n\n9) The information is available to the build system to permit using \ngit-bisect in the super-project and notice that the breakage occurred \nwhen the recorded state identifier of a sub-project changed. If the \nsub-project used Git, the build system can automatically start \ngit-bisect'ing in the sub-project.\n\n10) The information is available to the build system to permit doing \nsimilar things for git-log and so forth.\n\n11) Webgit and gitk are a little more work but by creating a git \nrepository that contains all of the history and refs of each project in \nthe entire project, you can navigate and view the history of any project \nin the entire project.\n\n12) If the SCM of some of the sub-projects is not Git, the build system \ncan still do git-bisect, git-log, etc. equivalents for the entire \nproject (subject to the limitations of the SCMs involved).\n\n\n>>>The tricky question is whether we should permit the \"subproject\" objects to\n>>>specify a revision that isn't a hash, for use in identifying revisions of\n>>>subprojects in other systems.\n>>Why would you want to limit how required versions of subprojects are\n>>specified? Your project policies and procedures may require that subprojects\n>>be specified by a subproject SCM specific immutable revision but the policies\n>>and procedures of other projects may not be so restrictive and could accept a\n>>tag identifying the latest \"stable\" (or something) revision.\n> \n> If you accept a tag identifying the latest stable revision, then you might \n> as well not bother. The point of revision controlling a project is to be \n> able to reconstruct previous states. If you allow any event, especially \n> outside, unrelated, events to change the reconstructed state for a \n> revision, then this is not the case. Your normal debugging situation will \n> be \"It's broken, and I didn't change anything.\" because someone somewhere \n> else changed something, and you have no record of what last worked. And \n> you can obviously forget any hope of \"git bisect\" working.\n\nCVS does not have the concept of \"hash\" for use in identifying the state \nof a particular set of files but it does have tags and CVS tags work \nlike Git tags. The form of identifier that is used to identify the \nparticular state of interest of a sub-project is dependent on the \npolicies and procedures of the super-project, and the SCM of the \nsub-project.\n\nThe need to reproduce a particular state at sometime in the future is \nwell understood, even in organizations that use tools that only support \ntags. Implying that it's not possible without using Git's immutable \nhashes as the state identifier is just wrong.\n\n--\n"},{"id":"14762","messageId":"Pine.LNX.4.64.0601170001130.25300@iabervon.org","threadId":"3035","inReplyTo":"7vek37rj83.fsf@assigned-by-dhcp.cox.net","subject":"Re: RFC: Subprojects","fromName":"Daniel Barkalow","fromEmail":"barkalow@iabervon.org","sentAt":"2006-01-17T05:46:40Z","receivedAt":"2006-01-17T05:46:40Z","isPatch":false,"sender":{"key":"barkalow@iabervon.org","avatar":"https://avatars.githubusercontent.com/u/55364219?v=4"},"body":"On Mon, 16 Jan 2006, Junio C Hamano wrote:\n\n> We could introduce \"bind the rest\" to make write-tree write\n> out a tree that contains only the containing project part and\n> not any of the subproject part (e.g. Makefile, README and\n> src/ but not linux-2.6/ nor gcc-4.0/ in the earlier example).\n> Essentially the contents of such a tree object would be the\n> same as what \"gitlink\" approach would have had for the\n> containing project in the index file, minus \"gitlink\" entries\n> themselves). This is not so surprising, because the missing\n> information \"gitlink\" approach recorded in the tree object\n> itself is expressed on \"bind\" lines in the commit object with\n> this approach.\n\nSo why not use the \"bind\" approach for the \"index vs working tree\" part, \nbut write out \"gitlink\"-style tree objects? I think putting the info in \nthe tree objects in the location the subproject would appear is nicer than \nhaving tree objects that tell only part of the story, and you don't have \nto worry about commits that stick a subproject on top of something in the \ntree.\n\nIn any case, I think it would be good to track where the subprojects are \nin some core state, and probably the right solution is to have special \nindex entries for them, in addition to having their contents in the index. \nI'm not seeing a clear way to get from commit objects with \"bind\" lines to \nan index with the appropriate things read and back otherwise.\n\nOne idea I toyed with a while ago for the index/working tree \nimplementation is having an index file per bound project, such that each \nproject has a completely ordinary index file, and you just need to tell \ncheckout-index where to write. This is especially cute because the index \nfile for the superproject doesn't need to know about the subprojects at \nall; they're not in that index, and the working tree is just directories \nof untracked files. Not sure if this is a useful idea at this point or \nnot.\n\n\t-Daniel\n*This .sig left intentionally blank*\n"},{"id":"14767","messageId":"7vfynnfkc8.fsf@assigned-by-dhcp.cox.net","threadId":"3035","inReplyTo":"Pine.LNX.4.64.0601170001130.25300@iabervon.org","subject":"Re: RFC: Subprojects","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2006-01-17T06:18:47Z","receivedAt":"2006-01-17T06:18:47Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Daniel Barkalow writes:\n\n> So why not use the \"bind\" approach for the \"index vs working tree\" part, \n> but write out \"gitlink\"-style tree objects?\n\nI said \"index vs working tree\" as a mere example, and never said\n\"gitlink\" is easier (or at least as easy as \"bind\") for \"tree\nobject vs index\" or \"tree object vs working tree through index\".\nIn fact I suspect those parts also need to be changed fairly\nheavily, and to be honest, I am not very much looking forward to\ninvestigating the details.\n\n> In any case, I think it would be good to track where the subprojects are \n> in some core state, and probably the right solution is to have special \n> index entries for them, in addition to having their contents in the index. \n\nActually, the \"special entry\" was what I found out to be quite a\npain, if you mean to have \"linux-2.6/\" in the index and have it\nused in some meaningful way. Further hacking and prototyping\n_might_ convince me otherwise, but I am not so optimistic at\nthis moment.\n\n> I'm not seeing a clear way to get from commit objects with \"bind\" lines to \n> an index with the appropriate things read and back otherwise.\n\nHere again I am thinking aloud, remembering the earlier example\nof an embedded linux project that ships with linux-2.6 and\ngcc-4.0, along with its own README and Makefile at the toplevel\nand src/ for its own sources. The tools at the tip of \"pu\"\nshould be able to let you do the following:\n\n\t$ git cat-file commit $such_toplevel_commit\n\ttree $tree\n parent $parent\n bind $primarysub /\n bind $linuxsub linux-2.6/\n bind $gccsub gcc-4.0/\n\tauthor A U Thor 1137392543 -0800\n\tcommmitter A U Thor 1137392543 -0800\n\n An example.\n\nwhere $tree is the object name of the whole tree (no \"gitlink\"\nobject), $primarysub and $linuxsub are the object names of\ncommit objects for the primary subproject (which sits at the\nrootlevel) and another subproject (which sits at linux-2.6/\nsubdirectory).\n\nTo make sure there is no misunderstanding:\n\n\t* \"git-ls-tree $tree\" would show the object name of\n $linuxsub^{tree} at path \"linux-2.6/\" because\n \"tree\" line of a commit describes the whole tree,\n including subprojects.\n\n\t* \"git-ls-tree $primarysub\" would show README,\n Makefile and src/ directories but not linux-2.6/ nor\n gcc-4.0/.\n\n\t* \"git-ls-tree $linuxsub\" would show COPYING, Makefile\n etc., not linux-2.6/COPYING.\n\nReading such a commit is easy:\n\n\t$ git-read-tree $tree ;# ;-)\n\nBut that is cheating. Constructing such an index can be done by:\n\n\t$ git-read-tree $primarysub\n $ git-read-tree --prefix=linux-2.6/ $linuxsub\n $ git-read-tree --prefix=gcc-4.0/ $gccsub\n\nWhen you have such an index, writing out various trees are:\n\n\t$ git-write-tree ;# $tree\n\t$ git-write-tree --prefix=linux-2.6/ ;# $linuxsub^{tree}\n\t$ git-write-tree --prefix=gcc-4.0/ ;# $gccsub^{tree}\n\t$ git-write-tree \\\n --bound=linux-2.6/ --bound=gcc-4.0/ ;# $primarysub^{tree}\n\nThe decision to use what --prefix and --bound and what tree(s)\nto write out must come from somewhere, and as you say it would\nbe nice if we _could_ stick them in the index as \"special\nentries\", but for the purpose of prototyping I am assuming I\nkeep that somewhere in $GIT_DIR/ (the \"mtab\" in the previous\nmessage. Maybe \"$GIT_DIR/bind\" is a good name?).\n"},{"id":"14780","messageId":"20060117140937.GI28365@pasky.or.cz","threadId":"3035","inReplyTo":"7vfynnfkc8.fsf@assigned-by-dhcp.cox.net","subject":"Re: RFC: Subprojects","fromName":"Petr Baudis","fromEmail":"pasky@suse.cz","sentAt":"2006-01-17T14:09:37Z","receivedAt":"2006-01-17T14:09:37Z","isPatch":false,"sender":{"key":"pasky@ucw.cz","avatar":"https://avatars.githubusercontent.com/u/18439?v=4"},"body":"Dear diary, on Tue, Jan 17, 2006 at 07:18:47AM CET, I got a letter\nwhere Junio C Hamano said that...\n> Here again I am thinking aloud, remembering the earlier example\n> of an embedded linux project that ships with linux-2.6 and\n> gcc-4.0, along with its own README and Makefile at the toplevel\n> and src/ for its own sources. The tools at the tip of \"pu\"\n> should be able to let you do the following:\n> \n> \t$ git cat-file commit $such_toplevel_commit\n> \ttree $tree\n> parent $parent\n> bind $primarysub /\n> bind $linuxsub linux-2.6/\n> bind $gccsub gcc-4.0/\n> \tauthor A U Thor 1137392543 -0800\n> \tcommmitter A U Thor 1137392543 -0800\n> \n> An example.\n> \n> where $tree is the object name of the whole tree (no \"gitlink\"\n> object), $primarysub and $linuxsub are the object names of\n> commit objects for the primary subproject (which sits at the\n> rootlevel) and another subproject (which sits at linux-2.6/\n> subdirectory).\n\nI perhaps missed this in the thread, but is it really so useful to bind\nthe subprojects to specific commits? If you care about reproducing\nspecific configuration, all you have to do is tag and seek recursively -\nand even having a separate tiny git branch tracking just a single file\nlisting the commit ids of subprojects seems more elegant to me than just\nforcing the specific commit ids. In the general case, I think it most\nusually goes \"this project#branch, the latest commit you can get\", so\nI'm not really convinced that you are optimizing for the right case at\nall.\n\n-- \n\t\t\t\tPetr \"Pasky\" Baudis\nStuff: http://pasky.or.cz/\nOf the 3 great composers Mozart tells us what it's like to be human,\nBeethoven tells us what it's like to be Beethoven and Bach tells us\nwhat it's like to be the universe. -- Douglas Adams\n"},{"id":"14782","messageId":"Pine.LNX.4.64.0601171122270.25300@iabervon.org","threadId":"3035","inReplyTo":"20060117140937.GI28365@pasky.or.cz","subject":"Re: RFC: Subprojects","fromName":"Daniel Barkalow","fromEmail":"barkalow@iabervon.org","sentAt":"2006-01-17T16:45:36Z","receivedAt":"2006-01-17T16:45:36Z","isPatch":false,"sender":{"key":"barkalow@iabervon.org","avatar":"https://avatars.githubusercontent.com/u/55364219?v=4"},"body":"On Tue, 17 Jan 2006, Petr Baudis wrote:\n\n> I perhaps missed this in the thread, but is it really so useful to bind\n> the subprojects to specific commits? If you care about reproducing\n> specific configuration, all you have to do is tag and seek recursively -\n> and even having a separate tiny git branch tracking just a single file\n> listing the commit ids of subprojects seems more elegant to me than just\n> forcing the specific commit ids. In the general case, I think it most\n> usually goes \"this project#branch, the latest commit you can get\", so\n> I'm not really convinced that you are optimizing for the right case at\n> all.\n\nThink from a debugging standpoint. You know that the main project worked \nwith a particular commit of the superproject. The bug you've found is \nrelated to the behavior of one of the subprojects in the the context of \nyour superproject, but you don't know this. In order to reproduce the \nworking version and search for the change that broken things, you need to \nbe able to identify which commits of subprojects were used in each commit \nof the superproject; these are almost certainly not the latest commits on \nany branch of the subproject. And if you're going to want to debug things \nlater, no commit of the superproject can just say to use the latest in the \nsubproject. You don't know what you make a commit whether it will turn out \nto be a configuration that you'll want to recreate later.\n\nNow it may be useful to have a tool to update all of the subprojects to \nthe latest versions, similar in end-user usage to pulling repositories, \nbut you still need to generate new commits when you do this, rather than \nreinterpreting the old commits with new content, so that you keep the \nhistory immutable. You also want to know when you've done this, so that \nyou don't clone a tree that's working fine and build it only to find that \nthe clone has fetched a new kernel version with different behavior without \nletting you know that anything has changed.\n\n\t-Daniel\n*This .sig left intentionally blank*\n"},{"id":"14785","messageId":"AD13D35F-7085-4F92-8BE6-B00C6056EA7A@gmail.com","threadId":"3035","inReplyTo":"Pine.LNX.4.64.0601171122270.25300@iabervon.org","subject":"Re: RFC: Subprojects","fromName":"Craig Schlenter","fromEmail":"craig.schlenter@gmail.com","sentAt":"2006-01-17T17:33:30Z","receivedAt":"2006-01-17T17:33:30Z","isPatch":false,"sender":{"key":"craig.schlenter@gmail.com","avatar":null},"body":"Hi\n\nFor reference, here is what subversion does:\n\nhttp://svnbook.red-bean.com/nightly/en/svn.advanced.externals.html\n\n--Craig\n"},{"id":"14786","messageId":"Pine.LNX.4.64.0601170928240.3240@g5.osdl.org","threadId":"3035","inReplyTo":"Pine.LNX.4.64.0601171122270.25300@iabervon.org","subject":"Re: RFC: Subprojects","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2006-01-17T17:38:13Z","receivedAt":"2006-01-17T17:38:13Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Tue, 17 Jan 2006, Daniel Barkalow wrote:\n> \n> Think from a debugging standpoint. You know that the main project worked \n> with a particular commit of the superproject.\n\nYes, there are real advantages to being able to tag a very specific \nversion of a tree.\n\nYou can do it manually (ie tag the versions of everything used), but \nthere's a real convenience to being able to say \"I want the tree to look \nexactly as it looked for our internal test-release that we shipped as a \npre-view to customer so-and-so\".\n\nYou can do it with ad-hoc build rules inside a company, but the likelihood \nthat they don't work all the time is pretty high. Somebody forgot to \nfollow the right procedure, and had updated a sub-tree without marking it, \nand now you can't reproduce the problem that a customer has with a debug \nbuild, because you have no way to reproduce the exact binary...\n\nIt's why people tag every file for huge trees under CVS for a release, and \naccept why building a release may take hours. It's crazy, yes, but there \nare other projects than just the BSD's that have that \"World\" mentality, \nwhere they want every single program under _one_ umbrella, so that they \ncan tag them all together.\n\nMe, I think it's crazy engineering (\"if you can't reproduce it with \nindividual projects, you're not doing programming, you're doing Voodoo\"), \nbut it's something that some organizations simply require.\n\nNow, it might be enough with a cogito approach of \".git/subprojects\", and \njust _version-control_ it in the top-level project, but then you'd need to \nmake sure that all the tools automatically update the version when they do \na \"pull\" or a \"commit\" on a subproject. But then it almost boils down to \n\"gitlink\"s after all.\n\n\t\tLinus\n"},{"id":"14787","messageId":"Pine.LNX.4.64.0601171150050.25300@iabervon.org","threadId":"3035","inReplyTo":"7vfynnfkc8.fsf@assigned-by-dhcp.cox.net","subject":"Re: RFC: Subprojects","fromName":"Daniel Barkalow","fromEmail":"barkalow@iabervon.org","sentAt":"2006-01-17T17:41:39Z","receivedAt":"2006-01-17T17:41:39Z","isPatch":false,"sender":{"key":"barkalow@iabervon.org","avatar":"https://avatars.githubusercontent.com/u/55364219?v=4"},"body":"On Mon, 16 Jan 2006, Junio C Hamano wrote:\n\n> Daniel Barkalow writes:\n> \n> > So why not use the \"bind\" approach for the \"index vs working tree\" part, \n> > but write out \"gitlink\"-style tree objects?\n> \n> I said \"index vs working tree\" as a mere example, and never said\n> \"gitlink\" is easier (or at least as easy as \"bind\") for \"tree\n> object vs index\" or \"tree object vs working tree through index\".\n> In fact I suspect those parts also need to be changed fairly\n> heavily, and to be honest, I am not very much looking forward to\n> investigating the details.\n\nI suspect that it's just as easy, except that you get confronted \nimmediately with the issues that you haven't dealt with in the bind \napproach (mentioned below). If you had parse_tree_buffer() just ignore \nthem, and had write-tree take a list of bind lines, that would match the \nstatus of your \"bind\" implementation, I think (except for the part you say \nis cheating).\n\nIncidentally, I don't think we'd want \"gitlink\" objects with the \"gitlink\" \napproach; we'd want trees to contain commit objects for subprojects. The \n\"gitlink\" thing that corresponds to \".git/HEAD\" isn't an object, it's a \ntree entry, which, like \".git/HEAD\" (or, more appropriately, \n\".git/refs/heads/something\") maps a name to the hash of a commit object.\n\n> > In any case, I think it would be good to track where the subprojects are \n> > in some core state, and probably the right solution is to have special \n> > index entries for them, in addition to having their contents in the index. \n> \n> Actually, the \"special entry\" was what I found out to be quite a\n> pain, if you mean to have \"linux-2.6/\" in the index and have it\n> used in some meaningful way. Further hacking and prototyping\n> _might_ convince me otherwise, but I am not so optimistic at\n> this moment.\n\nHmm... maybe libification should go ahead of subprojects. If access to the \nindex weren't so often open-coded, it would just be a matter of having \nthese entries in the data structure, but not actually returned by any \ncurrent call, and it would be just like they were in some other structure. \n\nActually, it should be easy to have them in the index file but not in the \nmain index data structure, by skipping over them in the for loop near the \nend of read_cache(). Put them in a separate structure, and write them back \nto the file in write_cache(), and have a different method entirely for \nchanging them, and they shouldn't affect the normal use of the index.\n\n> > I'm not seeing a clear way to get from commit objects with \"bind\" lines to \n> > an index with the appropriate things read and back otherwise.\n> \n> Here again I am thinking aloud, remembering the earlier example\n> of an embedded linux project that ships with linux-2.6 and\n> gcc-4.0, along with its own README and Makefile at the toplevel\n> and src/ for its own sources. The tools at the tip of \"pu\"\n> should be able to let you do the following:\n> \n> \t$ git cat-file commit $such_toplevel_commit\n> \ttree $tree\n> parent $parent\n> bind $primarysub /\n> bind $linuxsub linux-2.6/\n> bind $gccsub gcc-4.0/\n> \tauthor A U Thor 1137392543 -0800\n> \tcommmitter A U Thor 1137392543 -0800\n> \n> An example.\n> \n> where $tree is the object name of the whole tree (no \"gitlink\"\n> object), $primarysub and $linuxsub are the object names of\n> commit objects for the primary subproject (which sits at the\n> rootlevel) and another subproject (which sits at linux-2.6/\n> subdirectory).\n> \n> To make sure there is no misunderstanding:\n> \n> \t* \"git-ls-tree $tree\" would show the object name of\n> $linuxsub^{tree} at path \"linux-2.6/\" because\n> \"tree\" line of a commit describes the whole tree,\n> including subprojects.\n> \n> \t* \"git-ls-tree $primarysub\" would show README,\n> Makefile and src/ directories but not linux-2.6/ nor\n> gcc-4.0/.\n> \n> \t* \"git-ls-tree $linuxsub\" would show COPYING, Makefile\n> etc., not linux-2.6/COPYING.\n\nSide issue here: this implies that the kernel objects are in the \nsuperproject's repository, or at least accessible from it. So prune has to \nnot remove them. So, if you've committed changes to a subproject but not \nyet committed the fact that you want to use the changed subproject into \nthe superproject, fsck-objects has to find them somewhere.\n\n> Reading such a commit is easy:\n> \n> \t$ git-read-tree $tree ;# ;-)\n> \n> But that is cheating. \n\nThis is for backwards compatibility, I assume?\n\n> Constructing such an index can be done by:\n>\n> \t$ git-read-tree $primarysub\n> $ git-read-tree --prefix=linux-2.6/ $linuxsub\n> $ git-read-tree --prefix=gcc-4.0/ $gccsub\n> \n> When you have such an index, writing out various trees are:\n> \n> \t$ git-write-tree ;# $tree\n> \t$ git-write-tree --prefix=linux-2.6/ ;# $linuxsub^{tree}\n> \t$ git-write-tree --prefix=gcc-4.0/ ;# $gccsub^{tree}\n> \t$ git-write-tree \\\n> --bound=linux-2.6/ --bound=gcc-4.0/ ;# $primarysub^{tree}\n> \n> The decision to use what --prefix and --bound and what tree(s)\n> to write out must come from somewhere, and as you say it would\n> be nice if we _could_ stick them in the index as \"special\n> entries\", but for the purpose of prototyping I am assuming I\n> keep that somewhere in $GIT_DIR/ (the \"mtab\" in the previous\n> message. Maybe \"$GIT_DIR/bind\" is a good name?).\n\nThe hard thing here is getting the commits for the trees. The bind lines \nneed commits, which means either identifying that we already have the \ncorrect commit object, because we didn't change anything in the \nsubproject, or generating a new commit object with some message and the \nright parent. And we want to use commit objects, not tree objects, in the \nbind lines, so that, once we track a problem to the change of which commit \nis bound, we can treat the subproject as a project and debug it with \nbisect, rather than just having one tree that works and one that doesn't.\n\n\t-Daniel\n*This .sig left intentionally blank*\n"},{"id":"14795","messageId":"7vpsmq2tyb.fsf@assigned-by-dhcp.cox.net","threadId":"3035","inReplyTo":"Pine.LNX.4.64.0601171150050.25300@iabervon.org","subject":"Re: RFC: Subprojects","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2006-01-18T01:41:48Z","receivedAt":"2006-01-18T01:41:48Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Daniel Barkalow writes:\n\n> Incidentally, I don't think we'd want \"gitlink\" objects with the \"gitlink\" \n> approach; we'd want trees to contain commit objects for subprojects. The \n> \"gitlink\" thing that corresponds to \".git/HEAD\" isn't an object, it's a \n> tree entry, which, like \".git/HEAD\" (or, more appropriately, \n> \".git/refs/heads/something\") maps a name to the hash of a commit object.\n\n> Hmm... maybe libification should go ahead of subprojects. If access to the \n> index weren't so often open-coded, it would just be a matter of having \n> these entries in the data structure, but not actually returned by any \n> current call, and it would be just like they were in some other structure. \n\nAnd libification has been waiting for the core to settle ;-) We\nhave to start somewhere.\n\n> Actually, it should be easy to have them in the index file but not in the \n> main index data structure, by skipping over them in the for loop near the \n> end of read_cache()....\n\nYeah, I guess I was vaguely thinking along those lines while I\nwas driving to work this morning. I appreciate your spelling it\nout to make things clearer.\n\n> Side issue here: this implies that the kernel objects are in the \n> superproject's repository, or at least accessible from it. So prune has to \n> not remove them. So, if you've committed changes to a subproject but not \n> yet committed the fact that you want to use the changed subproject into \n> the superproject, fsck-objects has to find them somewhere.\n\nYes. I was planning to have \"$GIT_DIR/bind\" that says:\n\n\tmaster kernel=linux-2.6/ gcc=gcc-4.0/\n\nmeaning:\n\n\tThe project kept track by \"master\" branch binds the\n\tproject kept track by \"kernel\" branch as its subproject\n\tat its linux-2.6/ subdirectory.\n\nor something like that, so when you make a commit, you update\nthose other branches as needed. You already raised that issue\nat the end of your message, and I will explain how I think that\ncan/should be done as a response to that part later.\n\n>> Reading such a commit is easy:\n>> \n>> \t$ git-read-tree $tree ;# ;-)\n>> \n>> But that is cheating. \n>\n> This is for backwards compatibility, I assume?\n\nThis is done more for not having to touch *anything* that does\n\"index vs working file\", \"tree vs index\" and \"tree vs working\nfile via index\". It also is the easiest way to keep the \"a\ncommit object name can be used in place of the tree object name\nof the tree it contains\" invariant. Also I suspect this\norganization might help recursive subprojects, but if it is the\ncase, that is just a byproduct, not a design goal.\n\n>> When you have such an index, writing out various trees are:\n>> \n>> \t$ git-write-tree ;# $tree\n>> \t$ git-write-tree --prefix=linux-2.6/ ;# $linuxsub^{tree}\n>> \t$ git-write-tree --prefix=gcc-4.0/ ;# $gccsub^{tree}\n>> \t$ git-write-tree \\\n>> --bound=linux-2.6/ --bound=gcc-4.0/ ;# $primarysub^{tree}\n>\n> The hard thing here is getting the commits for the trees. The bind lines \n> need commits, which means either identifying that we already have the \n> correct commit object, because we didn't change anything in the \n> subproject, or generating a new commit object with some message and the \n> right parent. And we want to use commit objects, not tree objects, in the \n> bind lines, so that, once we track a problem to the change of which commit \n> is bound, we can treat the subproject as a project and debug it with \n> bisect, rather than just having one tree that works and one that doesn't.\n\nYour wording \"get the commit\" is a bit misleading. Even when\nthe tree for a subproject happens to match a commit in the\nsubproject in a distant past, we would not want to use it unless\nthe user explicitly asked for it. IOW, we do not actively go\nand look for a commit.\n\nOur subproject tree either matches the subproject branch head,\nin which case we just reuse it, or we make a new commit on top\nof that ourselves.\n\nLet's say my project breaks with the latest kernel, and I\nsuspect that it would work with v2.6.13 sources. To test that\ntheory, I could:\n\n $ git branch -f kernel v2.6.13 ;# rewind\n\n\t$ git ls-files linux-2.6/ |\n xargs git update-index --force-remove\n $ git read-tree --prefix=linux-2.6/ -u kernel\n\nto construct such a tree. Maybe the latter two-command sequence\n\"ls-files & read-tree --prefix\" sequence deserves to become a\ncommand, \"git update-subproject kernel\" [*1*].\n\nThe result may work as-is, or I may need to do some further\nfutzing in linux-2.6/ directory before the result works. Once\nthe result starts working, I'd want to make a commit:\n\n - I compare the result of write-tree for linux-2.6/ portion and\n the tree object name contained in the head commit of the\n \"kernel\" branch. If they match, then the current \"kernel\"\n branch head commit is what I'll place on the \"bind\" line in\n my commit; I do not have to make a new commit in the \"kernel\"\n subproject in this case.\n\n - If the tree object does not match the \"kernel\" head, that\n means I have tweaked the kernel part further, on top of\n v2.6.13. So I make a commit for the kernel subproject (whose\n parent is obviously v2.6.13), update the kernel branch head\n with that commit, and then record that tip-of-the-tree commit\n for the subproject on the \"bind\" line in my commit for the\n toplevel.\n\nOr let's say my project builds with the latest kernel (IOW, I\ndid not do the branch -f kernel in the above), and I made some\ncustom tweaks in the kernel area. The above precedure would\nresult in a new commit on top of the latest kernel, update the\n\"kernel\" branch head, and make a commit for the toplevel that\nrecords the updated \"kernel\" branch head on its \"bind\" line.\n\nNote that the above procedure did not use the commit object name\nrecorded on the \"bind\" line at all in either case. From the\nmechanism point of view, it is the right thing to do. From the\nusability point of view, however, we may want to take notice\nthat \"bind\" line commit and the bound branch head do not match,\nand remind/warn the user about it. If the reason why they are\ndifferent is because the user rewound the bound branch to use a\nknown working version, or made fixes in the subproject and\npulled the result into the bound branch (in which case there is\nno funny rewinding involved), then this warning is\nextraneous. But in the normal case of keep reusing the same\nvintage of subprojects (and maybe making necessary adjustments\nto subprojects while working on the main project), the commit\nobject on the \"bind\" line of the HEAD commit and bound branch\nhead should match.\n\n\n[Footnote]\n\n*1* One could also do a forward development on the kernel branch\nin a separate working tree and fetch from there. For example,\nif our example \"superproject\" is in embed/ directory, and there\nis a linux/ directory next to it to house a kernel repository,\nwe could:\n\n $ cd ../linux/\n $ edit && compile && test \n $ git commit -m 'Fix for upstream, not just for embed'\n\nto make an upstream fix, and then:\n\n $ cd ../embed/\n $ git fetch ../linux/ master:kernel\n\nto update the \"kernel\" subproject branch head. In such a case:\n\n\t$ git update-subproject kernel\n\nwould bring the subproject working tree and index up to date\nwith respect to the updated kernel branch.\n"},{"id":"14796","messageId":"7vy81eyz47.fsf@assigned-by-dhcp.cox.net","threadId":"3035","inReplyTo":"7vpsmq2tyb.fsf@assigned-by-dhcp.cox.net","subject":"Re: RFC: Subprojects","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2006-01-18T03:49:12Z","receivedAt":"2006-01-18T03:49:12Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> Daniel Barkalow writes:\n>\n>>> Reading such a commit is easy:\n>>> \n>>> \t$ git-read-tree $tree ;# ;-)\n>>> \n>>> But that is cheating. \n>>\n>> This is for backwards compatibility, I assume?\n>\n> This is done more for not having to touch *anything* that does\n> \"index vs working file\", \"tree vs index\" and \"tree vs working\n> file via index\". It also is the easiest way to keep the \"a\n> commit object name can be used in place of the tree object name\n> of the tree it contains\" invariant. Also I suspect this\n> organization might help recursive subprojects, but if it is the\n> case, that is just a byproduct, not a design goal.\n\nI started this \"bind\" design as a thought experiment, but I\nstarted to like it more and more.\n\nOne interesting outcome of keeping the whole tree in the index\nand the tree object recorded in the commit object of the\ntoplevel project is that a merge in the toplevel project \"just\nworks\".\n\nTo preserve our sanity, let's say we refuse to merge two commits\nthat have different sets of subprojects. That is, they must\nhave the \"bind\" lines for the same set of subdirectories. The\ncommits bound at these subdirectories do not need to match.\n\nBefore starting a merge, we require that the index is in sync\nwith the tree object recorded in the top commit, just like we do\nfor a normal merge[*1*]. Then we use the current merge\nmachinery that does not know anything about \"bind\" to perform\nthe merge, using the merge base of the toplevel project and\nusual three-way merge. From the mechanism point of view, there\nis no need to look at commits on \"bind\" line of either side to\ncome up with the resulting tree.\n\nWe could notice that the commit bound at linux-2.6/ subdirectory\nof one side is v2.6.15 and the other side is v2.6.16-rc1, and\nbecause one is a fast-forward of the other, choose to pick the\ntree associated with v2.6.16-rc1 commit without actually doing\nthe 3-way resolve of linux-2.6/ subtree part, but that is purely\na performance optimization [*2*].\n\nWhen writing out the merge result as a commit, we would create\n(this is the fun part) a commit for linux-2.6/ part that has two\nparents: the commits bound to linux-2.6/ tree from the two\ntoplevel commits being merged are the parents of such a\nsubproject commit. And the resulting toplevel merge commit\nwould have that commit object name on its \"bind\" line.\nObviously, when the bound subproject head of one side is a\nfast-forwad of the other, we do not create such a merge commit\nfor the subproject; instead, we just record the one that is\nahead on the \"bind\" line of the resulting toplevel merge commit.\n\n\n[Footnote]\n\n*1* As a side effect, this also ensures the index is in sync\nwith the bound commits of the subprojects. As an additional\nrequirement, we may want to enforce that the bound commits must\nmatch the branch heads that keep track of subprojects.\n\n*2* Of course, from the usability, safety and confusion\navoidance point of view, it _might_ make sense to require that\nbound commits are in such fast-forward relationship. But that\nis a policy issue; at the mechanism level, there is no need to\nimpose such a requirement.\n"},{"id":"14804","messageId":"200601181747.15609.lan@ac-sw.com","threadId":"3035","inReplyTo":"7vy81eyz47.fsf@assigned-by-dhcp.cox.net","subject":"Re: RFC: Subprojects","fromName":"Alexander Litvinov","fromEmail":"lan@ac-sw.com","sentAt":"2006-01-18T11:47:15Z","receivedAt":"2006-01-18T11:47:15Z","isPatch":false,"sender":{"key":"lan@ac-sw.com","avatar":null},"body":"> I started this \"bind\" design as a thought experiment, but I\n> started to like it more and more.\n>\n\nIs there a version of git with this to try it ?\n"},{"id":"14808","messageId":"43CE42A0.9040302@op5.se","threadId":"3035","inReplyTo":"200601181747.15609.lan@ac-sw.com","subject":"Re: RFC: Subprojects","fromName":"Andreas Ericsson","fromEmail":"ae@op5.se","sentAt":"2006-01-18T13:29:04Z","receivedAt":"2006-01-18T13:29:04Z","isPatch":false,"sender":{"key":"ae@op5.se","avatar":"https://gravatar.com/avatar/426e89595c75a8f5252dd0c989e5fabe5bcac616e68557427ad9aef6b0ca342a?d=mp&s=160"},"body":"Alexander Litvinov wrote:\n>>I started this \"bind\" design as a thought experiment, but I\n>>started to like it more and more.\n>>\n> \n> \n> Is there a version of git with this to try it ?\n\nThe pu branch of the official git repo.\n\n-- \nAndreas Ericsson andreas.ericsson@op5.se\nOP5 AB www.op5.se\nTel: +46 8-230225 Fax: +46 8-230231\n"},{"id":"14826","messageId":"7vy81dxy80.fsf@assigned-by-dhcp.cox.net","threadId":"3035","inReplyTo":"200601181747.15609.lan@ac-sw.com","subject":"Re: RFC: Subprojects","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2006-01-18T17:06:07Z","receivedAt":"2006-01-18T17:06:07Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Alexander Litvinov writes:\n\n>> I started this \"bind\" design as a thought experiment, but I\n>> started to like it more and more.\n>>\n>\n> Is there a version of git with this to try it ?\n\n\thttp://article.gmane.org/gmane.comp.version-control.git/14760\n\nAs I warned there, it still has quite rough edges, so do not use\nit on your production repositories.\n"},{"id":"14836","messageId":"Pine.LNX.4.64.0601181214150.25300@iabervon.org","threadId":"3035","inReplyTo":"7vpsmq2tyb.fsf@assigned-by-dhcp.cox.net","subject":"Re: RFC: Subprojects","fromName":"Daniel Barkalow","fromEmail":"barkalow@iabervon.org","sentAt":"2006-01-18T18:21:59Z","receivedAt":"2006-01-18T18:21:59Z","isPatch":false,"sender":{"key":"barkalow@iabervon.org","avatar":"https://avatars.githubusercontent.com/u/55364219?v=4"},"body":"On Tue, 17 Jan 2006, Junio C Hamano wrote:\n\n> Daniel Barkalow writes:\n> \n> > Incidentally, I don't think we'd want \"gitlink\" objects with the \"gitlink\" \n> > approach; we'd want trees to contain commit objects for subprojects. The \n> > \"gitlink\" thing that corresponds to \".git/HEAD\" isn't an object, it's a \n> > tree entry, which, like \".git/HEAD\" (or, more appropriately, \n> > \".git/refs/heads/something\") maps a name to the hash of a commit object.\n> \n> > Hmm... maybe libification should go ahead of subprojects. If access to the \n> > index weren't so often open-coded, it would just be a matter of having \n> > these entries in the data structure, but not actually returned by any \n> > current call, and it would be just like they were in some other structure. \n> \n> And libification has been waiting for the core to settle ;-) We\n> have to start somewhere.\n\nWell, we could do a pass at cleaning, organizing, and documenting the \ninternals, which is sort of the start to each of them.\n\n> > Side issue here: this implies that the kernel objects are in the \n> > superproject's repository, or at least accessible from it. So prune has to \n> > not remove them. So, if you've committed changes to a subproject but not \n> > yet committed the fact that you want to use the changed subproject into \n> > the superproject, fsck-objects has to find them somewhere.\n> \n> Yes. I was planning to have \"$GIT_DIR/bind\" that says:\n> \n> \tmaster kernel=linux-2.6/ gcc=gcc-4.0/\n> \n> meaning:\n> \n> \tThe project kept track by \"master\" branch binds the\n> \tproject kept track by \"kernel\" branch as its subproject\n> \tat its linux-2.6/ subdirectory.\n> \n> or something like that, so when you make a commit, you update\n> those other branches as needed. You already raised that issue\n> at the end of your message, and I will explain how I think that\n> can/should be done as a response to that part later.\n\nOkay, so you're using additional branch heads in the superproject to track \nthe current state of the subprojects. That makes sense, although I think \nit would confuse people less if they were held separately. IIRC, \nrefs/subprojects/kernel/heads/master is a perfectly good ref name these \ndays, so that might be a good idea. That would also mean that \nrefs/tags/v2.6.14 and refs/tags/v2.7.2.3 wouldn't get confused (being \nlinux and gcc tags, respectively), because they'd be under the appropriate \nsubprojects.\n\nI assume these get updated by checkout when you check out the commit with \nthem as bind lines?\n\n> >> Reading such a commit is easy:\n> >> \n> >> \t$ git-read-tree $tree ;# ;-)\n> >> \n> >> But that is cheating. \n> >\n> > This is for backwards compatibility, I assume?\n> \n> This is done more for not having to touch *anything* that does\n> \"index vs working file\", \"tree vs index\" and \"tree vs working\n> file via index\". It also is the easiest way to keep the \"a\n> commit object name can be used in place of the tree object name\n> of the tree it contains\" invariant. Also I suspect this\n> organization might help recursive subprojects, but if it is the\n> case, that is just a byproduct, not a design goal.\n\nAh, okay, so it's cheating for checkout, because checkout is supposed to \nunderstand everything, but not cheating for other things. I thought we \ndecided that the stuff that doesn't know about subprojects sees them as \nopaque, rather than as their contents, so your toplevel git diff doesn't \nshow you a millions lines when you switch from linux-2.6.14 to 15.\n\n> >> When you have such an index, writing out various trees are:\n> >> \n> >> \t$ git-write-tree ;# $tree\n> >> \t$ git-write-tree --prefix=linux-2.6/ ;# $linuxsub^{tree}\n> >> \t$ git-write-tree --prefix=gcc-4.0/ ;# $gccsub^{tree}\n> >> \t$ git-write-tree \\\n> >> --bound=linux-2.6/ --bound=gcc-4.0/ ;# $primarysub^{tree}\n> >\n> > The hard thing here is getting the commits for the trees. The bind lines \n> > need commits, which means either identifying that we already have the \n> > correct commit object, because we didn't change anything in the \n> > subproject, or generating a new commit object with some message and the \n> > right parent. And we want to use commit objects, not tree objects, in the \n> > bind lines, so that, once we track a problem to the change of which commit \n> > is bound, we can treat the subproject as a project and debug it with \n> > bisect, rather than just having one tree that works and one that doesn't.\n> \n> Your wording \"get the commit\" is a bit misleading. Even when\n> the tree for a subproject happens to match a commit in the\n> subproject in a distant past, we would not want to use it unless\n> the user explicitly asked for it. IOW, we do not actively go\n> and look for a commit.\n\nWe don't search the history for just any commit, but we have to look \nsomewhere...\n\n> Our subproject tree either matches the subproject branch head,\n> in which case we just reuse it, or we make a new commit on top\n> of that ourselves.\n\nI hadn't realized that the subprojects had branch heads. It makes much \nmore sense that you'd expect to be able to just write out bind lines if \nyou've got that information.\n\nI thought we decided that committing the superproject wouldn't commit the \nsubprojects. If this is what we decided, then the subproject tree is \nrequired to match the branch head, because we must have committed the \nsubproject already (which is good, because otherwise the user will get \nconfused about which commit message to write when).\n\n> Let's say my project breaks with the latest kernel, and I\n> suspect that it would work with v2.6.13 sources. To test that\n> theory, I could:\n> \n> $ git branch -f kernel v2.6.13 ;# rewind\n> \n> \t$ git ls-files linux-2.6/ |\n> xargs git update-index --force-remove\n> $ git read-tree --prefix=linux-2.6/ -u kernel\n> \n> to construct such a tree. Maybe the latter two-command sequence\n> \"ls-files & read-tree --prefix\" sequence deserves to become a\n> command, \"git update-subproject kernel\" [*1*].\n\nShouldn't \"git read-tree --prefix=linux-2.6/ -u kernel\" remove everything \nelse in the index in linux-2.6 itself, making the \"git update-index \n--force-remove\" unnecessary?\n\n> The result may work as-is, or I may need to do some further\n> futzing in linux-2.6/ directory before the result works. Once\n> the result starts working, I'd want to make a commit:\n> \n> - I compare the result of write-tree for linux-2.6/ portion and\n> the tree object name contained in the head commit of the\n> \"kernel\" branch. If they match, then the current \"kernel\"\n> branch head commit is what I'll place on the \"bind\" line in\n> my commit; I do not have to make a new commit in the \"kernel\"\n> subproject in this case.\n> \n> - If the tree object does not match the \"kernel\" head, that\n> means I have tweaked the kernel part further, on top of\n> v2.6.13. So I make a commit for the kernel subproject (whose\n> parent is obviously v2.6.13), update the kernel branch head\n> with that commit, and then record that tip-of-the-tree commit\n> for the subproject on the \"bind\" line in my commit for the\n> toplevel.\n\nEquivalently, you make a commit for the kernel subproject, update the \nbranch head, and start over; the first case should apply now.\n\n> Or let's say my project builds with the latest kernel (IOW, I\n> did not do the branch -f kernel in the above), and I made some\n> custom tweaks in the kernel area. The above precedure would\n> result in a new commit on top of the latest kernel, update the\n> \"kernel\" branch head, and make a commit for the toplevel that\n> records the updated \"kernel\" branch head on its \"bind\" line.\n> \n> Note that the above procedure did not use the commit object name\n> recorded on the \"bind\" line at all in either case. From the\n> mechanism point of view, it is the right thing to do. From the\n> usability point of view, however, we may want to take notice\n> that \"bind\" line commit and the bound branch head do not match,\n> and remind/warn the user about it. If the reason why they are\n> different is because the user rewound the bound branch to use a\n> known working version, or made fixes in the subproject and\n> pulled the result into the bound branch (in which case there is\n> no funny rewinding involved), then this warning is\n> extraneous. But in the normal case of keep reusing the same\n> vintage of subprojects (and maybe making necessary adjustments\n> to subprojects while working on the main project), the commit\n> object on the \"bind\" line of the HEAD commit and bound branch\n> head should match.\n\nI hope people will want to prepare their commits to the kernel subproject \nas would be suitable for pushing to Linus, which would suggest that they'd \ntend to do a commit in the kernel subproject embedded in their \nsuperproject separately from doing the commit in the superproject, and \nso the branch head would match the index but not the bind line when they \ngot to committing the superproject.\n\n\t-Daniel\n*This .sig left intentionally blank*\n"},{"id":"14839","messageId":"7v4q41wevw.fsf@assigned-by-dhcp.cox.net","threadId":"3035","inReplyTo":"Pine.LNX.4.64.0601181214150.25300@iabervon.org","subject":"Re: RFC: Subprojects","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2006-01-18T18:49:07Z","receivedAt":"2006-01-18T18:49:07Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Daniel Barkalow writes:\n\n> I assume these get updated by checkout when you check out the commit with \n> them as bind lines?\n\nI did not think of that when I wrote that message, but you are right.\n\n> Ah, okay, so it's cheating for checkout, because checkout is supposed to \n> understand everything, but not cheating for other things.\n\nYeah; to put it another way, read-tree of the toplevel tree and\nchecking it out is equivalent to do the skelton read-tree\nfollowed by --prefix read-tree of all the bound projects, so\ncheckout can optimize.\n\n> ... I thought we \n> decided that the stuff that doesn't know about subprojects sees them as \n> opaque, rather than as their contents, so your toplevel git diff doesn't \n> show you a millions lines when you switch from linux-2.6.14 to 15.\n\nIt was discussed in the context of \"gitlink\" approach as a way\nto keep things simple. In the \"bind\" approach, I am doing\nthings a bit differently, and this \"toplevel has everything\" is\none big difference.\n\n> I thought we decided that committing the superproject wouldn't\n> commit the subprojects.\n\nI see it as a policy. We can forbid the modification of the\nsubproject part of the index (i.e. detect and refuse to commit\nand/or do \"git reset --mixed\" only for the subproject part) so\nthat the commit outlined in the \"bind\" approach does not _have_\nto make a new commit, if you want to work that way. But if\nsomebody else wants to make a related set of changes to the\nsuperproject and bound subprojects, we _could_ allow a commit\nper subproject.\n\n> Shouldn't \"git read-tree --prefix=linux-2.6/ -u kernel\" remove everything \n> else in the index in linux-2.6 itself, making the \"git update-index \n> --force-remove\" unnecessary?\n\nI agree that \"-u\" should imply that. The current \"read-tree\n--prefix=linux-2.6/\" in proposed updates refuses if linux-2.6/\nappears in the original index.\n\n> I hope people will want to prepare their commits to the kernel subproject \n> as would be suitable for pushing to Linus, which would suggest that they'd \n> tend to do a commit in the kernel subproject embedded in their \n> superproject separately from doing the commit in the superproject, and \n> so the branch head would match the index but not the bind line when they \n> got to committing the superproject.\n\nYes, that is the workflow I outlined in the footnote part you\ndid not quote. I think it is cleaner to do things that way: to\nhave a separate, kernel-only repository+worktree and do pure\nkernel work there, and fetch into the superproject branch that\nkeeps track of the kernel subproject in that superproject.\n\nHaving more than one working tree with .git/, everything except\nHEAD and index undef which are symlinked to one copy, like you\ndo, would be a natural way to work.\n\n\tembed/.git/HEAD -> refs/heads/master\n\n\tembed/linux-2.6/.git/HEAD -> refs/heads/kernel\n\tembed/linux-2.6/.git/refs -> ../.git/refs\n\tembed/linux-2.6/.git/objects -> ../.git/objects\n\nThen, after hacking on the collective whole to make the whole\nthing work in \"embed\" directory, you would:\n\n\t$ cd linux-2.6\n $ git commit\n\nto make commit that can be sent Linus, at the same time updating\nthe \"kernel\" branch. Then come back to the toplevel, tell git\nthat you updated the \"kernel\" branch so it does not complain\nthat the \"bind\" in the HEAD commit does not match \"kernel\" head,\nand make a toplevel commit.\n"},{"id":"14847","messageId":"Pine.LNX.4.64.0601181359400.25300@iabervon.org","threadId":"3035","inReplyTo":"7v4q41wevw.fsf@assigned-by-dhcp.cox.net","subject":"Re: RFC: Subprojects","fromName":"Daniel Barkalow","fromEmail":"barkalow@iabervon.org","sentAt":"2006-01-18T19:29:02Z","receivedAt":"2006-01-18T19:29:02Z","isPatch":false,"sender":{"key":"barkalow@iabervon.org","avatar":"https://avatars.githubusercontent.com/u/55364219?v=4"},"body":"On Wed, 18 Jan 2006, Junio C Hamano wrote:\n\n> Daniel Barkalow writes:\n> \n> > ... I thought we \n> > decided that the stuff that doesn't know about subprojects sees them as \n> > opaque, rather than as their contents, so your toplevel git diff doesn't \n> > show you a millions lines when you switch from linux-2.6.14 to 15.\n> \n> It was discussed in the context of \"gitlink\" approach as a way\n> to keep things simple. In the \"bind\" approach, I am doing\n> things a bit differently, and this \"toplevel has everything\" is\n> one big difference.\n\nI thought that had been a question of what is best as an interface, but \neither is plausible.\n\n> > I thought we decided that committing the superproject wouldn't\n> > commit the subprojects.\n> \n> I see it as a policy. We can forbid the modification of the\n> subproject part of the index (i.e. detect and refuse to commit\n> and/or do \"git reset --mixed\" only for the subproject part) so\n> that the commit outlined in the \"bind\" approach does not _have_\n> to make a new commit, if you want to work that way. But if\n> somebody else wants to make a related set of changes to the\n> superproject and bound subprojects, we _could_ allow a commit\n> per subproject.\n\nI think it makes most sense, for the purpose of consolidating code paths, \nif the superproject may only be committed with clean subprojects; the \nporcelain has the option of responding to unclean subprojects by \ncommitting them to make them clean, and then there is only a single case \nfor how the superproject commit happens. I think it makes most sense as a \ncommand line option, like -a is; if you want to commit dirty suprojects, \nyou use --subprojects, and it does that. If you're not expecting to need \nit, you won't start doing the wrong commit.\n\n> > I hope people will want to prepare their commits to the kernel subproject \n> > as would be suitable for pushing to Linus, which would suggest that they'd \n> > tend to do a commit in the kernel subproject embedded in their \n> > superproject separately from doing the commit in the superproject, and \n> > so the branch head would match the index but not the bind line when they \n> > got to committing the superproject.\n> \n> Yes, that is the workflow I outlined in the footnote part you\n> did not quote. I think it is cleaner to do things that way: to\n> have a separate, kernel-only repository+worktree and do pure\n> kernel work there, and fetch into the superproject branch that\n> keeps track of the kernel subproject in that superproject.\n\nI actually meant that I expected people to go into superproject/linux-2.6, \nmake changes, and commit there, using the place it appears in their \nsuperproject working tree as a working tree for the subproject, so the \nopposite of your footnote, but still doing the subproject commit as a step \nbefore the superproject commit.\n\nFor example, they might want to send the subproject changes upstream as a \npatch, get feedback, reset the subproject, do revised changes, commit \nthat, get it merged upstream, and then commit the changes to the \nsuperproject, including in the message the fact that the changes have been \npushed upstream. But they may still want to do this all within the working \ntree of the superproject, so that they can test their changes in context.\n\n> Having more than one working tree with .git/, everything except\n> HEAD and index undef which are symlinked to one copy, like you\n> do, would be a natural way to work.\n> \n> \tembed/.git/HEAD -> refs/heads/master\n> \n> \tembed/linux-2.6/.git/HEAD -> refs/heads/kernel\n> \tembed/linux-2.6/.git/refs -> ../.git/refs\n> \tembed/linux-2.6/.git/objects -> ../.git/objects\n> \n> Then, after hacking on the collective whole to make the whole\n> thing work in \"embed\" directory, you would:\n> \n> \t$ cd linux-2.6\n> $ git commit\n> \n> to make commit that can be sent Linus, at the same time updating\n> the \"kernel\" branch. Then come back to the toplevel, tell git\n> that you updated the \"kernel\" branch so it does not complain\n> that the \"bind\" in the HEAD commit does not match \"kernel\" head,\n> and make a toplevel commit.\n\nI'm not sure having a .git directory for a subproject inside a \nsubdirectory of the superproejct's working tree is all that good an idea, \nand I don't think it should be necessary in any case, because the toplevel \nindex has all the information from the subproject index. The only think \nwould be having \"git commit\" notice what you're doing when you run it from \na directory that's a subproject.\n\n\t-Daniel\n*This .sig left intentionally blank*\n"},{"id":"15054","messageId":"20060123005056.GV28365@pasky.or.cz","threadId":"3035","inReplyTo":"7vek37rj83.fsf@assigned-by-dhcp.cox.net","subject":"Re: RFC: Subprojects","fromName":"Petr Baudis","fromEmail":"pasky@suse.cz","sentAt":"2006-01-23T00:50:56Z","receivedAt":"2006-01-23T00:50:56Z","isPatch":false,"sender":{"key":"pasky@ucw.cz","avatar":"https://avatars.githubusercontent.com/u/18439?v=4"},"body":"Dear diary, on Mon, Jan 16, 2006 at 09:49:48PM CET, I got a letter\nwhere Junio C Hamano said that...\n> We could introduce \"bind the rest\" to make write-tree write\n> out a tree that contains only the containing project part and\n> not any of the subproject part (e.g. Makefile, README and\n> src/ but not linux-2.6/ nor gcc-4.0/ in the earlier example).\n> Essentially the contents of such a tree object would be the\n> same as what \"gitlink\" approach would have had for the\n> containing project in the index file, minus \"gitlink\" entries\n> themselves). This is not so surprising, because the missing\n> information \"gitlink\" approach recorded in the tree object\n> itself is expressed on \"bind\" lines in the commit object with\n> this approach.\n\nNow, I must have missed the obvious again, but what is the point in\nhaving the write-tree --exclude stuff? My impression (also from your\nlater mail in this thread) is that now the moment you introduce any\nbinds, your top-level development changes to \"two-tiered\" - the\ntop-level project and the meta-project holding it all together. I'd say\nthat's pretty confusing and I don't see big gain in this; the simplicity\nof the original proposal was a lot more appealing.\n\n-- \n\t\t\t\tPetr \"Pasky\" Baudis\nStuff: http://pasky.or.cz/\nOf the 3 great composers Mozart tells us what it's like to be human,\nBeethoven tells us what it's like to be Beethoven and Bach tells us\nwhat it's like to be the universe. -- Douglas Adams\n"},{"id":"15055","messageId":"20060123012227.GW28365@pasky.or.cz","threadId":"3035","inReplyTo":"Pine.LNX.4.64.0601181214150.25300@iabervon.org","subject":"Re: RFC: Subprojects","fromName":"Petr Baudis","fromEmail":"pasky@suse.cz","sentAt":"2006-01-23T01:22:27Z","receivedAt":"2006-01-23T01:22:27Z","isPatch":false,"sender":{"key":"pasky@ucw.cz","avatar":"https://avatars.githubusercontent.com/u/18439?v=4"},"body":"Dear diary, on Wed, Jan 18, 2006 at 07:21:59PM CET, I got a letter\nwhere Daniel Barkalow said that...\n> Okay, so you're using additional branch heads in the superproject to track \n> the current state of the subprojects. That makes sense, although I think \n> it would confuse people less if they were held separately. IIRC, \n> refs/subprojects/kernel/heads/master is a perfectly good ref name these \n> days, so that might be a good idea. That would also mean that \n> refs/tags/v2.6.14 and refs/tags/v2.7.2.3 wouldn't get confused (being \n> linux and gcc tags, respectively), because they'd be under the appropriate \n> subprojects.\n\nI passionately agree - this is the only thing I do not like on the\ncurrent Junio's proposal (besides that top-level subproject confusion).\nThe way it is proposed, you are mixing different projects in a single\nrefs namespace and I think that's *really* confusing.\n\nBesides, you are going to get a lot of complications since to do merging\nproperly you need two heads per subproject (its 'master' and 'origin'\nheads; and it's useful to have e.g. all the upstream heads called\n'origin' since then you can say cg-fetch -r origin in the superproject\nand have all the subproject origins fetched as well) and you might want\nto have other subproject heads as well. Now, for different superproject\nheads, you want separate set of subproject heads. You can see the\ndownward spiral from here, I guess... And multiply all that by two since\nyou also have tags.\n\nIt actually took me a short while to realize that keeping separate\nsubproject/.git/refs makes no sense precisely because for different\nsuperproject heads, you want a different set of subproject refs.\nSo in line with Daniel's proposal, I'd propose:\n\n\trefs/subprojects///heads/master\n\n is the name of the current HEAD (${#refs/heads/}). \nis a little more tricky - this should be the part after the equal sign\nin .git/mtab (or .git/binds or .git/subprojects or whichever is the name\nof the day). Obviously, you can just figure out something, but I'd like\nto assign this automagically.\n\nOTOH, in Cogito I might as well just default to sha1 of something random\n(e.g. the path+commitid+time()) since I do not expect this to be\nnormally referenced by a human; I just intend to switch from refs/ to\nrefs/subprojects/// when dealing with the subproject\nexclusively. ($GIT_REF_DIR (by default $GIT_DIR/refs) would come useful;\nI'll probably whip up a patch when I get to finally need it.)\n\n> I hope people will want to prepare their commits to the kernel subproject \n> as would be suitable for pushing to Linus, which would suggest that they'd \n> tend to do a commit in the kernel subproject embedded in their \n> superproject separately from doing the commit in the superproject, and \n> so the branch head would match the index but not the bind line when they \n> got to committing the superproject.\n\nFWIW, my idea is that it should be \"a seamless experience for the user\"\n(tm) to do development in a subproject of another project, and I can see\nno reason why should that be hard to do in any way.\n\n-- \n\t\t\t\tPetr \"Pasky\" Baudis\nStuff: http://pasky.or.cz/\nOf the 3 great composers Mozart tells us what it's like to be human,\nBeethoven tells us what it's like to be Beethoven and Bach tells us\nwhat it's like to be the universe. -- Douglas Adams\n"},{"id":"16454","messageId":"20060220131659.GA8613@informatik.uni-freiburg.de","threadId":"3035","inReplyTo":"7vacdzkww3.fsf@assigned-by-dhcp.cox.net","subject":"Re: RFC: Subprojects","fromName":"Uwe Zeisberger","fromEmail":"zeisberg@informatik.uni-freiburg.de","sentAt":"2006-02-20T13:16:59Z","receivedAt":"2006-02-20T13:16:59Z","isPatch":false,"sender":{"key":"u.kleine-koenig@pengutronix.de","avatar":"https://gravatar.com/avatar/354b5e3ceb2806a2f1e1e382ac29ddbdad18288654da62b61eb13583a857eee7?d=mp&s=160"},"body":"Hello,\n\nJunio C Hamano wrote:\n> The \"containing\" project would have a handful \"gitlink\" objects\n> among other things. The toplevel tree object from a commit in\n> such a project might look like this (mode bits 0160000 is\n> S_IFDIR|S_IFLNK, which is what this thing is):\n> \n> $ git ls-tree HEAD\n> 0100644 blob 012345... Makefile\n> 0100644 blob 123456... README\n> 0160000 link 234567... gcc-4.0\n> 0160000 link 345678... linux-2.6\n> 0040000 tree 456789... src\n> $ git cat-file -t 345678\n> link\n> $ git cat-file link 345678\n> commit 87530db5ec7d519c7ba334e414307c5130ae2da8\n> url git://...torvalds/linux-2.6.git/\n> \n> The upstream Linux 2.6 repository.\n> $ cd linux-2.6 && git-rev-parse --verify HEAD\n> 87530db5ec7d519c7ba334e414307c5130ae2da8\n> \n> URL will be used as a suggestion for people who cloned this tree\n> to set up their repository.\nI'd prefer to have the objects needed to get the linux-2.6 tree in the\nobject db of the containing project. Then \"url\" is not needed, and you\ncould directly use the commit as value for the link. i.e.\n\n $ git ls-tree HEAD\n 0100644 blob 012345... Makefile\n 0100644 blob 123456... README\n 0160000 link 435363... gcc-4.0\n 0160000 link 87530d... linux-2.6\n 0040000 tree 456789... src\n\n(You could now rename \"link\" to \"commit\", but it would break the\nlayout.)\n\nMoreover I prefer the the link approach over the bind method. The\nreason is, that binds use information from the commit object to build\nthe wc other than the tree. Moreover the condition that the\n\"containing\" tree must not have an entry named linux-2.6 is handled\nimplicitly with links.\n\nPlease correct me if I'm wrong somewhere. It's some time ago I read the\npatches and this thread. This mail is the result of some thoughts in my\nvacation.\n\nBest regards\nUwe\n\n\n-- \nUwe Zeisberger\n\nhttp://www.google.com/search?q=1+year+divided+by+3+in+seconds\n"},{"id":"16496","messageId":"7v1wxx4011.fsf@assigned-by-dhcp.cox.net","threadId":"3035","inReplyTo":"20060220131659.GA8613@informatik.uni-freiburg.de","subject":"Re: RFC: Subprojects","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2006-02-21T07:57:14Z","receivedAt":"2006-02-21T07:57:14Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Uwe Zeisberger writes:\n\n> I'd prefer to have the objects needed to get the linux-2.6 tree in the\n> object db of the containing project. Then \"url\" is not needed, and you\n> could directly use the commit as value for the link.\n\n... which is actually closer to what bind commit approach gives\nyou. The tree object in a commit of the containing project has\nthe full tree object at path linux-2.6/. The \"bind\" lines in\nthe commit object are just notes that tell you where those\ntrees happen to came from.\n\n> ... Moreover the condition that the\n> \"containing\" tree must not have an entry named linux-2.6 is handled\n> implicitly with links.\n\nI had an impression that two approaches were more or less\nequivalent, especially the last round of bound commit approach.\nIt does not let anything to exist at the bound path in the\ncontaining project either (\"read-tree --prefix\" rejects it).\n\n> Please correct me if I'm wrong somewhere. It's some time ago I read the\n> patches and this thread. This mail is the result of some thoughts in my\n> vacation.\n\nI have to admit that I haven't thought about the issues involved\nfor a long time, having no great need nor desire for subprojects\nmyself, and especially with more generally useful stuff like\nperformance enhancement for pack generation to occupy me. I am\nnot sure I am much more qualified to comment than you are at\nthis point.\n\nThe bound commit lowlevel changes have been sitting in \"pu\" for\nabout a month by now, but nobody seems to be interested enough\nto start prototyping Porcelain around it. Neither the gitlink\napproach. After seeing not much interest on the list, I was\nhoping that I could retire both WIPs.\n"}]}