{"thread":{"id":"43127","subject":"Re: [RFC] Submodules in GIT","startedAt":"2006-11-20T21:51:16Z","lastAt":"2006-12-03T12:31:45Z","messageCount":10,"participants":["Jakub Narebski","Martin Waitz","Shawn Pearce","Petr Baudis","Robin Rosenberg"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"294854","messageId":"20061120215116.GA20736@admingilde.org","threadId":"43127","inReplyTo":null,"subject":"[RFC] Submodules in GIT","fromName":"Martin Waitz","fromEmail":"tali@admingilde.org","sentAt":"2006-11-20T21:51:16Z","receivedAt":"2006-11-20T21:51:16Z","isPatch":false,"sender":{"key":"tali@admingilde.org","avatar":"https://gravatar.com/avatar/3f89b03eee362187effabe257898735b475673a12265c398ea9161259ae91553?d=mp&s=160"},"body":"I am currently working on adding submodule support to GIT.\nHere I am presenting some prototyping work to show how submodules could\nbe implemented in GIT.\n\n\nWhat is a submodule\n-------------------\n\nA submodule in a GIT repository which is managed by another higher-layer\nparent repository.  Both the submodule and the parent repository can be\nused as a normal GIT repository but there are links between them.  That\nmeans that you can merge, push and pull both on the parent and on\nindividual submodule level.\n\nThe parent repository is special in that it does not only hold individual\nfiles but also complete submodule trees, complete with their own history.\nThe history of the parent and the submodule are independent, but each\nversion of the parent is linked to exactly one version of the submodule.\n\nThe submodule is special in that it has one special branch which is tracked\nby the parent.  Each time something is commited to this branch, this\nautomatically changes the parent, just like modifying normal files.  Each\ntime the parent gets updated, this submodule branch gets updated\nautomatically.\n\nAs a submodule has all the properties of a normal GIT repository,\nit can also contain submodules itself.\n\n\nPrevious submodule proposal\n---------------------------\n\nMy first experiment in implementing submodules was a very simple one: Store\nthe submodule refs in the working directory of the parent repository and\ninclude them in the version control.\n\nThis way the most important properties of submodules are easily to obtain.\nEach version of the parent stores the exact version of all submodules and\nwith the help of some shell scripts it is easily possible to update the\nsubmodule when there are updates to the parent repository.\n\n\nHowever, this easy approach has several drawbacks.  The most important one\nis that the GIT core does not know about submodules, they are only built on\ntop of them.  For this reason it is difficult to support fsck-objects and\nprune in such a setup.\n\nIn order to permanently keep all previous stages of the parent, there is\nthe need to also keep all versions of the submodules which have been linked\nto the parent.  All operations which have to walk the entire object database\nhave to know all possible references of the submodule.  This is easy if the\nsubmodule is only fast-forwarded, without switching branches and without\nremoving the submodule.  Under these conditions it is enough to walk the\ncurrent reference of the submodule which is stored in its parent.  But if\nthere are past versions of the submodule which are not reachable by the\ncurrent submodule commit then it is difficult to keep them in the database\nwhen the submodule is handled by a standard GIT core.  This could only be\nsolved by creating fake references for all possible versions of the\nsubmodule.\n\n\nNew approach\n------------\n\nIn order to make the GIT core know about all possible versions of the\nsubmodule it is not enough to store one reference in the parent working\ndirectory.  The GIT core has to be changed so that it knows about all\nsubmodule references while traversing the parent repository.  This means\nthat both the submodule and the parent repository have to use the same\nobject database.  (I already did this for my first experiment, but it was\nnot really neccessary at that time.)\n\nA submodule really is part of the parent tree, so it is very natural to\nadd the link to the submodule commit into the GIT tree data structure.\nIn addition to links to blobs and other trees, they can now also hold\na link to a commit, which in turn has the pointers to the submodule tree\nand its history.  In order to differenciate a submodule entry with\nnormal file or directory entries, they get a special file mode.\n\nDirectly including the submodules in the object database allows the\ntraversal of the entire repository, together with the parent and all\nsubmodules.  In this way it is possible to support fsck-objects and prune\nwhen they are executed in the parent repository.  However, submodules can\ncontain branches which are independent from the one stored in the parent.\nSo all references and the working directory index of the submodules have\nto be made available to the parent, too.\n\nWhen a parent containing submodules is checked out, the submodul entry is\nstored in the index, just like it is done for all normal files.  But instead\nof writing one file to the working directory, a complete GIT repository is\ncreated (with the object database linked to the parent one).  This submodule\ngets a reference \"refs/heads/module\" from the parent's index entry.\nThe index entry in the parent can be updated with update-index just\nlike other entries.\n\nWhen a merge in the parent has to resolve changes in the submodule,\nthen it does exactly the same as for files: at first it is tried to resolve\nit in the index and if this is not possible it will have store stages 1-3\nin the index and tries a content merge.  The only difference with submodules\nis that the content merge is not possible with a simple diff3 call, but\nthat the GIT merge machinery has to recurse into the submodule.\n\n\nImplementation\n--------------\n\nObviously, all the tree traversal routines have to be modified to recognize\na submodule and to correctly traverse it.  The submodule entry gets\na special S_IFSOCK file mode to distinguish it from other entries.\nThis special file mode is used for both the tree entries in the object\ndatabase and for the index entries of the submodule in the parent index.\n\nSome basic low-level commands now (more or less) cope with submodules.\nMerging of submodules has not yet been implemented.\n\nThe current status can be viewed in\nhttp://git.admingilde.org/tali/git.git/module2\n(on top of next)\n\nThe code now passes the small test-script so at least a little bit of\nit must be working ;-).  Please feel free to give it a try and complain\nthat it does not work the way you expect it.\n\n\nWhat's next?\n------------\n\nThe most important next step is to commit to some object database format\nfor submodules.  So please, do give feedback about the proposed changes\nto the tree object.\n\nThe second most important step is to make it possible to merge submodules.\n\nAfter that, for sure there are enough bugs to fix to keep me busy for some\ntime... ;-)\n\n-- \nMartin Waitz\n"},{"id":"296299","messageId":"ejtbbn$p8s$2@sea.gmane.org","threadId":"43127","inReplyTo":"20061120215116.GA20736@admingilde.org","subject":"Re: [RFC] Submodules in GIT","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2006-11-20T22:49:55Z","receivedAt":"2006-11-20T22:49:55Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Could you please add to http://git.or.cz/gitwiki/SubprojectSupport\n(even if all it would be is a link to archive of this thread)? TIA.\n\n-- \nJakub Narebski\nWarsaw, Poland\nShadeHawk on #git\n\n"},{"id":"294922","messageId":"20061121072137.GC3332@spearce.org","threadId":"43127","inReplyTo":"20061120215116.GA20736@admingilde.org","subject":"Re: [RFC] Submodules in GIT","fromName":"Shawn Pearce","fromEmail":"spearce@spearce.org","sentAt":"2006-11-21T07:21:37Z","receivedAt":"2006-11-21T07:21:37Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Martin Waitz <tali@admingilde.org> wrote:\n> I am currently working on adding submodule support to GIT.\n> Here I am presenting some prototyping work to show how submodules could\n> be implemented in GIT.\n\nAside from a GUI for Git that I can give to a non-technical\nuser and have them be productive with (and without getting \"this\nsucks!\" complaints from them), submodule support is the next highest\npriority feature for me in Git.  So as soon as I can get git-gui\nto that point I'll probably redirect as much of my \"Git time\"\nas I can to testing your submodule implementation.\n \n> When a merge in the parent has to resolve changes in the submodule,\n> then it does exactly the same as for files: at first it is tried to resolve\n> it in the index and if this is not possible it will have store stages 1-3\n> in the index and tries a content merge.  The only difference with submodules\n> is that the content merge is not possible with a simple diff3 call, but\n> that the GIT merge machinery has to recurse into the submodule.\n\nRight.  And what's really cool about that is many times a subproject\nmerge will be trivial, so top level project merges will be still be\ntrivial in index merges.  But what's complicated about that is you\nneed to make sure the subproject working directory is fast-forwarded\nto the new commit.  :)\n \n> The most important next step is to commit to some object database format\n> for submodules.  So please, do give feedback about the proposed changes\n> to the tree object.\n\nI think the S_IFSOCK approach is the right way to go here, and thus\nI'm reasonably happy with this repository format.  But Linus and\nJunio do tend to be better at this than I... :-)\n\n-- \n"},{"id":"296307","messageId":"20061122052936.GK7201@pasky.or.cz","threadId":"43127","inReplyTo":"20061120215116.GA20736@admingilde.org","subject":"Re: [RFC] Submodules in GIT","fromName":"Petr Baudis","fromEmail":"pasky@suse.cz","sentAt":"2006-11-22T05:29:36Z","receivedAt":"2006-11-22T05:29:36Z","isPatch":false,"sender":{"key":"pasky@ucw.cz","avatar":"https://avatars.githubusercontent.com/u/18439?v=4"},"body":"On Mon, Nov 20, 2006 at 10:51:16PM CET, Martin Waitz wrote:\n> The current status can be viewed in\n> http://git.admingilde.org/tali/git.git/module2\n> (on top of next)\n\n(For those wondering, a diff of the changes is at:\n\n\thttp://git.admingilde.org/tali/git.git/?a=commitdiff;h=module2;hp=next\n\nAny bright ideas on how to best make such diffs reachable in the UI?)\n\n-- \n\t\t\t\tPetr \"Pasky\" Baudis\nStuff: http://pasky.or.cz/\nThe meaning of Stonehenge in Traflamadorian, when viewed from above, is:\n\"Replacement part being rushed with all possible speed.\"\n"},{"id":"298255","messageId":"eksmrf$c33$1@sea.gmane.org","threadId":"43127","inReplyTo":"20061120215116.GA20736@admingilde.org","subject":"Re: [RFC] Submodules in GIT","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2006-12-02T20:16:42Z","receivedAt":"2006-12-02T20:16:42Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Few thoughts on this topic. Some of those are repeating what\nwas said eaelier\n\n1. Submodule (subproject) as commit-in-a-tree\n\nLet's try to paint a little diagram (attribution missing):\n\nbelonging to:\n/--------- supermodule -------\\    /---- submodule -------\\\n\ncommit -> tree +-> blob\n  |            +-> tree -> ...\n  |            +-----------------> commit -> tree -> ...\n  v                                  |\ncommit -> tree +-> ...               v\n  |            +-----------------> commit -> ...\n  v                        /         |\ncommit -> tree +-> ...    /          |\n  |            +---------/           v\n  |                                commit -> ...\n  v                                  |\ncommit -> tree +-> ...               v\n               +-----------------> commit\n\nBoth have their independent history, but they are linked as some\nsubmodule versions are part of the supermodule tree.\n\n\n2. Working area for project with submodules\n\nSubmodule as separate repository model\nsupermodule\n+ .git/  <------------------------.\n  + HEAD                          |\n  + index                         |\n  + objects/                      |\n  + objects/info/alternates ---.  |\n+ subdir1/                     |  |\n  + sub1file                   |  ^                   \n+ submodule/                   |  |\n  + .git                       v  |\n    + HEAD                     |  |\n    + index                    |  |\n    + objects/  <--------------'  |\n    +[objects/info/borrowers] ----'\n  + subsubdir/\n    + submfile\n+ file\n\nEmbedded submodule model\nsupermodule\n+ .git/\n  + HEAD\n  + index\n  + objects/\n  +[refs/submodules/submodule/HEAD]\n  +[refs/submodules/submodule/index]\n+ subdir1/\n  + sub1file\n+ submodule/\n  + subsubdir/\n    + submfile\n+ file\n\nThe [fictional] borrowers file is for git-prune and friends (also\ngit-repack with -d option) to not remove objects needed by supermodule\n(when for example submodule history got rewritten). But you can do\nwithout it, as long as you don't rewind or don't prune in\nsupermodule.\n\nThe problem with submodule as separate git repository is that if you\nmove submodule (subproject) somewhere else in the repository (or just\nrename it), you have to update alternates file... and this happens not\nonly on move itself, but also on checkout and reset. But that can be\nmanaged by having in alternates all possible places the submodule ends\ninto. I don't know if it is truly a problem.\n\nAlternate solution would be to have submodule objects [also] in the\nmain (superproject) object database (for example fetched from\nsubmodule object repository on supermodule commit with changing\nsubmodule).\n\nPerhaps instead of objects/info/alternates we should use\nobjects/info/modules, or even modules file (as top .git dir).\n\n\nThe problem with embedded submodule model is ensuring that changes in\nsubmodule go to submodule (using submodule refs; at least HEAD and\nsubmodule index). And there are troubles with treating submodule\nseparately, for example cloning submodule only, or fetching from\nsubmodule only.\n\n\n3. Output of git-ls-tree and git-ls-files (git-ls-index ;-) for\nproject with submodules.\n\n$ git ls-tree HEAD\n040000 tree 959dd5d97e665998eb26c764d3a889ae7903d9c2    subdir1\n140000 subm ccddf1d4b0cf7fd3a699d8b33cf5bc4c5c4435b7    submodule\n100644 blob a57a33b81ac6c9cb5ec0c833edc21bd66428d976    file\n\n$ git ls-tree -r -t HEAD\n040000 tree 959dd5d97e665998eb26c764d3a889ae7903d9c2    subdir1\n100644 blob 70d8b9838a7333bc5a1edb93cf0e9abdbcf146cc    subdir1/sub1file\n140000 subm ccddf1d4b0cf7fd3a699d8b33cf5bc4c5c4435b7    submodule\n040000 tree 959dd5d97e665998eb26c764d3a889ae7903d9c2    submodule/subsubdir\n100755 blob 6579f06b05c91f00f4f45015894f2bfab1076bf6    submodule/subsubdir/submfile\n100644 blob a57a33b81ac6c9cb5ec0c833edc21bd66428d976    file\n\n$ git ls-files --stages\n100644 70d8b9838a7333bc5a1edb93cf0e9abdbcf146cc 0   subdir1/sub1file\n140000 ccddf1d4b0cf7fd3a699d8b33cf5bc4c5c4435b7 0   submodule\n100644 a57a33b81ac6c9cb5ec0c833edc21bd66428d976 0   file\n\n\n4. Workflow(s) for project with submodules\n\n$ cd submodule\nsubmodule$ edit subsubdir/submfile\nsubmodule$ git update-index subsubdir/submfile  # this updates submodule index\nsubmodule$ git commit -m \"Submodule change\"     # this changes submodule HEAD\nsubmodule$ cd ..\n$ git update-index submodule                    # this updates index \n                                                  to submodule HEAD version\n$ git commit -m \"Change in submodule\"           # this updates HEAD\n\nOf course as usual you should be able to do \"git commit -a\" to skip\n\"git update-index\". One has to remember that \"git update-index\nsubmodule\" and \"git commit submodule\" uses HEAD version of submodule,\nnot the working area version.\n\nThere was an idea to update superproject index not to HEAD version\nbut to some specified branch version.\n\n\n5. Extended sha1 syntax for submodules\n\nFor [almost] all commands the commit-in-tree should\nbe viewed as tree-ish, for example in HEAD:submodule/subsubdir (is a\ntree), or HEAD:submodule/subsubdir/submfile (is a blob).\n\nCurrently a suffix ':' followed by a path names the blob or tree (or\ncommit) at the given path in the tree-ish object named by the part\nbefore the colon. You cannot currently use it recirsively, i.e. use\n<tree-ish>:<path> to refer to tree (or commit), and use ':' after\nthat, e.g. <tree-ish>:<path>:<subpath>... well, currently this has not\nmuch sense, as you can (and have to) use '/' as a separator.\n\nThere was proposal to use '//' as a way to force commit object in the\ntree to be treated as commit-ish, not as a tree, so you can apply all\nthe extended sha1 machinery suitable for commits like ^, ^n, ~n and\nalso probably ^@, but perhaps not @{n}. Then making ':' resursive\nwould be useful, for example:\n\n  HEAD^:submodule//~2:subsubdir/submfile\n\n\n-- \nJakub Narebski\nWarsaw, Poland\nShadeHawk on #git\n\n"},{"id":"298308","messageId":"200612030224.50592.robin.rosenberg.lists@dewire.com","threadId":"43127","inReplyTo":"eksmrf$c33$1@sea.gmane.org","subject":"Re: [RFC] Submodules in GIT","fromName":"Robin Rosenberg","fromEmail":"robin.rosenberg.lists@dewire.com","sentAt":"2006-12-03T01:24:50Z","receivedAt":"2006-12-03T01:24:50Z","isPatch":false,"sender":{"key":"robin.rosenberg@dewire.com","avatar":"https://avatars.githubusercontent.com/u/46357?v=4"},"body":"lördag 02 december 2006 21:16 skrev Jakub Narebski:\n> The problem with submodule as separate git repository is that if you\n> move submodule (subproject) somewhere else in the repository (or just\n> rename it), you have to update alternates file... and this happens not\n> only on move itself, but also on checkout and reset. But that can be\n> managed by having in alternates all possible places the submodule ends\n> into. I don't know if it is truly a problem.\n\nA nasty problem with separate repositories for submodules is that when you \nscrew up and git complains about everything you try do do, you previously \ncould do rm -rf *; git reset --hard and retry whatever you were trying to do. \nWith separate repositories your submodules will be resting in /dev/null, \nunless you're very, very careful. \n\n"},{"id":"296960","messageId":"200612030231.05900.jnareb@gmail.com","threadId":"43127","inReplyTo":"200612030224.50592.robin.rosenberg.lists@dewire.com","subject":"Re: [RFC] Submodules in GIT","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2006-12-03T01:31:05Z","receivedAt":"2006-12-03T01:31:05Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Dnia niedziela 3. grudnia 2006 02:24, Robin Rosenberg napisał:\n> lördag 02 december 2006 21:16 skrev Jakub Narebski:\n> > The problem with submodule as separate git repository is that if you\n> > move submodule (subproject) somewhere else in the repository (or just\n> > rename it), you have to update alternates file... and this happens not\n> > only on move itself, but also on checkout and reset. But that can be\n> > managed by having in alternates all possible places the submodule ends\n> > into. I don't know if it is truly a problem.\n> \n> A nasty problem with separate repositories for submodules is that when you \n> screw up and git complains about everything you try do do, you previously \n> could do rm -rf *; git reset --hard and retry whatever you were trying to do. \n> With separate repositories your submodules will be resting in /dev/null, \n> unless you're very, very careful. \n\nActually, rm -rf * is not needed for \"git reset --hard\" or\n\"git checkout -f\" to succeed.\n\n-- \nJakub Narebski\n"},{"id":"293885","messageId":"ekuaka$6km$1@sea.gmane.org","threadId":"43127","inReplyTo":"200612030224.50592.robin.rosenberg.lists@dewire.com","subject":"Re: [RFC] Submodules in GIT","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2006-12-03T11:00:20Z","receivedAt":"2006-12-03T11:00:20Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Robin Rosenberg wrote:\n\n> lördag 02 december 2006 21:16 skrev Jakub Narebski:\n>>\n>> The problem with submodule as separate git repository is that if you\n>> move submodule (subproject) somewhere else in the repository (or just\n>> rename it), you have to update alternates file... and this happens not\n>> only on move itself, but also on checkout and reset. But that can be\n>> managed by having in alternates all possible places the submodule ends\n>> into. I don't know if it is truly a problem.\n> \n> A nasty problem with separate repositories for submodules is that when you \n> screw up and git complains about everything you try do do, you previously \n> could do rm -rf *; git reset --hard and retry whatever you were trying to do. \n> With separate repositories your submodules will be resting in /dev/null, \n> unless you're very, very careful. \n\nThe solution to this concern could be having GIT_DIR for submodule\noutside it's working area, for example somewhere in GIT_DIR of\nthe supermodule, and use either symlink or (to be coded) .gitlink\nsymbolic reference to GIT_DIR file. Disadvantage of that is that it\nmoves troubles with moving subproject (although there are no troubles\nwith simple subproject directory renaming) from alternates file to\nGIT_DIR link representation in submodule.\n\nAs Linus said, there are advantages to having submodule repository use\nseparate object database (clone and other operation scaling, index size),\nand I think they outweight the troubles with moving/renaming the directory\nsubmodule resides in.\n\nP.S. I think that the problem with bad performance of too large index\nis similar to the problems filesystems have with directories with large\nnumber of files; some filesystems solved this problem.\n-- \nJakub Narebski\nWarsaw, Poland\nShadeHawk on #git\n\n"},{"id":"298486","messageId":"200612031322.54748.robin.rosenberg.lists@dewire.com","threadId":"43127","inReplyTo":"200612030231.05900.jnareb@gmail.com","subject":"Re: [RFC] Submodules in GIT","fromName":"Robin Rosenberg","fromEmail":"robin.rosenberg.lists@dewire.com","sentAt":"2006-12-03T12:22:54Z","receivedAt":"2006-12-03T12:22:54Z","isPatch":false,"sender":{"key":"robin.rosenberg@dewire.com","avatar":"https://avatars.githubusercontent.com/u/46357?v=4"},"body":"söndag 03 december 2006 02:31 skrev Jakub Narebski:\n> Actually, rm -rf * is not needed for \"git reset --hard\" or\n> \"git checkout -f\" to succeed.\n\nTrue, but git reset --hard isn't always enough and rm -rf * is the good ol' \nway of resetting things. Typically this comes from make being upset (or too \ncontent) with something.\n\n"},{"id":"297719","messageId":"200612031331.45410.jnareb@gmail.com","threadId":"43127","inReplyTo":"200612031322.54748.robin.rosenberg.lists@dewire.com","subject":"Re: [RFC] Submodules in GIT","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2006-12-03T12:31:45Z","receivedAt":"2006-12-03T12:31:45Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Robin Rosenberg wrote:\n> söndag 03 december 2006 02:31 skrev Jakub Narebski:\n>\n>> Actually, rm -rf * is not needed for \"git reset --hard\" or\n>> \"git checkout -f\" to succeed.\n> \n> True, but git reset --hard isn't always enough and rm -rf * is the good ol' \n> way of resetting things. Typically this comes from make being upset (or too \n> content) with something.\n\n$ git clean -d -x -q\n$ git reset --hard\n\n(or vice versa) then.\n\nBesides, \"rm -rf *\" should not remove hidden files and hidden directories,\nincluding .git directory. And you should take great care with \"rm -rf .*\"\nif it doesn't follow '..'\n-- \nJakub Narebski\n"}]}