{"thread":{"id":"31801","subject":"A design for subrepositories","startedAt":"2012-10-13T13:33:22Z","lastAt":"2012-10-19T20:09:35Z","messageCount":17,"participants":["Lauri Alanko","Junio C Hamano","perryh@pluto.rain.com","Jens Lehmann"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"201082","messageId":"20121013163322.685276teuhqhjc82.lealanko@webmail.helsinki.fi","threadId":"31801","inReplyTo":null,"subject":"A design for subrepositories","fromName":"Lauri Alanko","fromEmail":"la@iki.fi","sentAt":"2012-10-13T13:33:22Z","receivedAt":"2012-10-13T13:33:22Z","isPatch":false,"sender":{"key":"la@iki.fi","avatar":null},"body":"Hello.\n\nI intend to work on a \"subrepository\" tool for git, but before I \nembark on the actual programming, I thought to first invite comments \non the general design.\n\nSome background first. I know that there are several existing \napproaches already for managing nested repositories, but none of them \nquite seems to fit my purposes. My primary goal is to use git for home \ndirectory backup and mirroring, while the home directory itself may of \ncourse contain repositories.\n\nGit-subtree doesn't quite fit the bill. It allows merging a subtree \ninto a larger tree and then again splitting it out for exporting, but \nthis is tedious. More importantly, a merged tree gets branched along \nwith the containing tree, whereas I want to have subrepositories \nprecisely because the subtrees need to be branched independently of \nthe container.\n\nSubmodules are a bit closer to what I want, but they have clearly been \ndesigned for a different purpose: a repository with submodules is only \nsupposed to collate existing repositories, not act as a source for \nthem. So they aren't really faithful to the distributed nature of git: \nthere's no easy way to completely clone a repository and its submodules.\n\nMoreover, submodules have some other annoyances like not supporting \nbare repositories and checking out the submodules in detached heads.\n\nNow, in other circumstances I might just patch git-submodule to add \nthe features I want, but it turns out that it is written in shell. I \nknow that is a git tradition, but I'm going to get a bit religious \nhere: anything longer than a screenful shouldn't be written in shell, \nand I'm certainly not going to add more lines to an already overlong \nscript. Hence I'm going to write a separate tool using something a bit \nmore... structured. Probably Python with Dulwich.\n\nSo here are some preliminary thoughts on how the tool should work.\n\n\n* Repository layout\n\nEvery subrepository has a unique identifier. The heads of \nsubrepository are simply stored as heads in a subdirectory \nof the main repository: e.g. \nrefs/heads/subrepos//. Likewise for tags: \nrefs/tags/subrepos//.\n\nRationale: if we had fully independent repositories under the main \nrepository directory, like what git-submodule uses, there would be no \neasy way to enumerate all the existing subrepositories to copy them. \nSince the only thing we can directly list from a remote repository are \nreferences, it makes sense to store the subrepositories just as a \nbunch of them.\n\nThe reason for storing the subrepo references under refs/heads/ and \nrefs/tags/ (instead of, say, refs/subrepos/) is simply that this way \neverything is directly compatible with standard git tools: one can do \na normal git clone/push/pull for mirroring and backup purposes without \nany need for special tools. You only need tools once you operate on a \nworking tree.\n\n\n* Tree layout\n\nA tree can mount references of subrepositories. There are two \ncomponents to a mount: a gitlink under to a particular commit \nof a subrepo, and an entry in .gitrepos. This is very similar to how \ngit-submodule works.\n\nThe entry in .gitrepos specifies two things: the name of the \nsubrepository mounted under , and the active branch in that \nmount at the time of commit. So .gitrepos would look like this:\n\n[mount \"\"]\n subrepo = \n branch = \n\nRationale: by storing the active branch name we can cater for the very \ncommon case where we check out a gitlink pointing to the current head \nof the branch. Then, when we check out the subrepository at the mount \npoint, we can adjust HEAD to point to the correct branch.\n\nBy associating from a path to a subrepository (instead of the other \nway, as git-submodule does), we can have multiple mount points for the \nsame subrepository, presumably with different active branches. \nSometimes we want to have separate working trees for various branches, \nand it's good to be able to store this configuration in the containing \ntree.\n\n\n* Working tree layout\n\nWhen a tree containing mount points is checked out, a repository is \ncreated at each of those mount points. For every specified in \n.gitrepos with subrepo and active branch , and a \ngitlink in pointing to , we do the following:\n\n- Create a repository under /.git\n\n- Add the object store of the containing repository to \n/.git/objects/info/alternates\n\n- Pull (just copy, really) the containing repository's references to \nthe subrepository as follows:\n\n - refs/heads/subrepos//* -> refs/heads/*\n - refs/tags/subrepos//* -> refs/tags/*\n - refs/remotes//subrepos//* -> refs/remotes//*\n\n- If now in the subrepository refs/heads/ points to \n, set HEAD as a symref to it. Otherwise set a detached HEAD \ndirectly to .\n\n- Check out HEAD in the subrepository.\n\nRationale: it was a tempting idea to make refs/heads and refs/tags to \nbe symlinks directly to the correct subdirectories in the containing \nrepository, and likewise make objects/ directly a symlink to the \ncontaining repository's object store. However, this is not really \nfeasible due to packed-refs, and it would make symlinks a requirement, \nsomething that git tries to avoid. (Of course \"directory symrefs\" \nwould be a simple addition to the core.)\n\nMore importantly, a symlink to the object store would break git-gc. \nAlso, it would be ugly to have ref manipulations under the mount point \ndirectly affect the refs in the containing repository. It's better \nthat none of the changes under the mount point affect the containing \nrepository in any way before an explicit add and check-in. At this \npoint the refs are pulled back in the reverse direction.\n\n\n* Basic commands\n\n\n** git subrepo add []\n\nAdd a subrepository to the containing repository, or add the changes \nin a subrepository to the index.\n\nIf is not yet found in .gitrepos, must be specified. \nOtherwise is looked up from .gitrepos.\n\nThe command performs the following:\n\n- Add or update the gitlink to the index: git add \n- Add or change an entry in .gitmodules, setting mount..subname \nto and mount..branch to the active branch under \n(if any).\n- git add .gitmodules\n\n\n** git subrepo checkin [-f] [...]\n\nUpdate the subrepo references in the containing repository to the \nreferences in the mount points. This is meant to be run as a \npre-commit hook with no arguments.\n\nIf no paths are given, ... defaults to every mount path in \n.gitrepos that has been changed in the index. For each mounting \n, perform the following:\n\n- git fetch [-f] refs/heads/*:refs/heads/subrepos//\n- git fetch [-f] refs/tags/*:refs/tags/subrepos//\n\nIf [-f] is given, it is passed to git fetch.\n\nThe operation can fail in the unlikely case that there are multiple \nmount points for the same subrepository, and a branch has diverged \nbetween those mount points.\n\nNote: after this operation, any new objects that were added under the \nmount point are now duplicated in the containing repository. A git gc \nin the containing repository followed by a git gc in the mount point \nshould remove the now-redundant objects from the mount point.\n\nNote: the default paths overlook the spurious case where have modified \nthe head of a non-active branch under the mount point, but the active \nbranch (and hence the commit in the gitlink) have remained unchanged. \nI don't know if there's a reasonable way to make \"git subrepo add\" \nsomehow stage even these kinds of changes.\n\n\n** git subrepo checkout [...]\n\nCheck out the subrepositories at mount points ..., or at all the \nmount points if none are specified. This is meant to be run as a \npost-checkout hook with no arguments.\n\nThis is described above in \"Working tree layout\". If this is not an \ninitial checkout, then the first two steps are skipped and just the \nrefs and working tree are updated.\n\n\n** git subrepo mv \n\nMove a mount point: git mv the actual directory and adjust the path in \n.gitrepos and possibly the relative path in \n/.git/objects/info/alternates. (An absolute path would fix the \nlatter, but then we couldn't move the entire containing repository. \nThis is the lesser evil, IMHO.)\n\nGripe: why doesn't git support arbitrary metadata for tree entries? \nThen we wouldn't need to worry about syncing various path attributes \nthat are stored in separate files, but a simple git mv could \nautomatically move everything associated with the path.\n\n\n** git subrepo rm \n\nRemove the mount point and its entry in .gitrepos.\n\n\n* A variant design\n\nThe above design is straightforward to implement, but it has a bit of \nan ad-hoc feel in that we have these magic commands that transfer refs \nbetween the containing repository and the mount points. But there are \nalready standard tools for transferring refs: push and pull/fetch. It \nwould be more \"git-like\" to use these directly, and make the \ncontaining repository be simply a remote for the mount point. We need \na special remote for this purpose: git-remote-subrepo gives a \"view\" \nof the refs of a particular subrepo within the ref tree of the \ncontaining repository. It just makes the following translations for \npush and fetch:\n\nsubrepo:/// refs/heads/\n-> heads/subrepos//\n\nsubrepo:/// refs/tags/\n-> tags/subrepos//\n\nsubrepo://// refs/heads/ ->\n-> remotes//heads/subrepos//\n\nsubrepo://// refs/heads/ ->\n-> remotes//heads/subrepos//\n\nThen subrepo:/// is set as the origin in the \nmount point, so one can just do a normal git push to push the changes \nto the containing repository. Likewise, for all the remotes in the \ncontaining repository, a remote with the same name is created under \nthe mount point with the url \nsubrepo:////. Or it can be set to \ndirectly access the actual remote: \nsubrepo:///. It's a matter of taste.\n\nThe problem with explicit pushing to the containing repository is that \nthen changes to the refs happen completely independently of changes to \nthe gitlinks, and ideally these should be synchronized in a single \ncommit. So I'm not quite sure if the additional complexity of a remote \nhelper is warranted.\n\n\nI hope I managed to make some sense of what this is about. Questions \nand comments are appreciated.\n\nCheers,\n\n\nLauri\n"},{"id":"201083","messageId":"7vd30m2sbr.fsf@alter.siamese.dyndns.org","threadId":"31801","inReplyTo":"20121013163322.685276teuhqhjc82.lealanko@webmail.helsinki.fi","subject":"Re: A design for subrepositories","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-10-13T17:30:00Z","receivedAt":"2012-10-13T17:30:00Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Lauri Alanko\" writes:\n\n> I intend to work on a \"subrepository\" tool for git, but before I\n> embark on the actual programming, I thought to first invite comments\n> on the general design.\n>\n> Some background first. I know that there are several existing\n> approaches already for managing nested repositories, but none of them\n> quite seems to fit my purposes. My primary goal is to use git for home\n> directory backup and mirroring, while the home directory itself may of\n> course contain repositories.\n> ...\n> Submodules are a bit closer to what I want, but they have clearly been\n> designed for a different purpose: a repository with submodules is only\n> supposed to collate existing repositories, not act as a source for\n> them.\n\nI have a repository that covers my home directory and some of its\nsubdirectories have their own repositories.\n\nI had my home directory and its subdirectories before Git ever\nexisted, and I made my home directory and these subdirectories into\nseparate, nested Git repositories fairly early after I started\nmanaging them with Git---way before submodules were invented. Now\nthe subdirectory repositories are bound as submodules of the top\nlevel directory just fine.\n\nI push these out for safekeeping purposes, all of my machines get\ntheir copies from here, and some submodules are not cloned to work\nmachines (they house data of private nature). They are used just\nlike you are expected to use submodules. In fact, this is pretty\nmuch vanilla use case of submodules, I think.\n\nThey _all_ originate from under my home directory, not \"collating\nexisting repositories\" at all.\n\nHave you considered how you can _extend_ submodules support to\nsupport your use case better? I think that would be a much more\nuseful approach, as you are likely to get help from other people who\ndo use submodules.\n"},{"id":"201088","messageId":"507a3d9a.j+V+kkG9pRPJs4kM%perryh@pluto.rain.com","threadId":"31801","inReplyTo":"20121013163322.685276teuhqhjc82.lealanko@webmail.helsinki.fi","subject":"Re: A design for subrepositories","fromName":"","fromEmail":"perryh@pluto.rain.com","sentAt":"2012-10-13T21:20:42Z","receivedAt":"2012-10-13T21:20:42Z","isPatch":false,"sender":{"key":"perryh@pluto.rain.com","avatar":null},"body":"\"Lauri Alanko\" wrote:\n\n> I'm going to get a bit religious here:\n> anything longer than a screenful shouldn't be written in shell ...\n\nWhence cometh this religion? I've heard of a modularity principle\nwherein no one function, in any language, ought to be longer than a\npage, but what's special about shell that warrants such a further\nrestriction?\n\nBTW, to adherents of the mentioned religion, this:\nhttp://www.freebsd.org/cgi/cvsweb.cgi/ports/ports-mgmt/portmaster/files/Attic/portmaster.sh.in?rev=2.32;content-type=text/plain\n-- at just under 3600 lines -- is likely one of the greater heresies\naround :)\n"},{"id":"201087","messageId":"20121014002304.14167k2j2ctspiuw.lealanko@webmail.helsinki.fi","threadId":"31801","inReplyTo":"7vd30m2sbr.fsf@alter.siamese.dyndns.org","subject":"Re: A design for subrepositories","fromName":"Lauri Alanko","fromEmail":"la@iki.fi","sentAt":"2012-10-13T21:23:04Z","receivedAt":"2012-10-13T21:23:04Z","isPatch":false,"sender":{"key":"la@iki.fi","avatar":null},"body":"Quoting \"Junio C Hamano\" :\n> Now\n> the subdirectory repositories are bound as submodules of the top\n> level directory just fine.\n\nThis is indeed possible, but with some serious caveats.\n\nFirstly, if you simply do \"git submodule add ./foo\" (the obligatory \n\"./\" being quite an unobvious pitfall), you get something quite \nfragile, since now we have submodule.foo.url = ./foo. If the \nsubmodules ever get reorganized and foo is moved to ./bar, then it is \nimpossible to check out older versions or alternate branches, since \nthe submodule is no longer where it is expected to be at the origin.\n\nA more robust solution is to use submodule.foo.url = \n./.git/modules/foo, since logical name of a module doesn't change. \nThis seems quite kludgy, though, and this cannot be how git-submodule \nis supposed to be used.\n\nBut still, \"git submodule update\" only looks at the modules in the \ncurrently checked-out tree. If we have other branches or old tags that \nrefer to other submodules, there's no simple way to fetch those, too. \nAnd there is not even such a concept as a bare repository with modules.\n\nSo git-submodule is fundamentally a tool to attach repositories into a \ntree, not to attach repositories into a repository. That's why it's \nnot really fit for my purposes.\n\nThe core problem is that to clone an entire repository and all its \nsubmodules, there needs to be a way to list them all remotely. But the \ngit protocol doesn't just allow us to list the subdirectories under \n.git/modules. Still, there are several ways to do this:\n\n* Just read .gitmodules in every ref and find by brute force every \nsubmodule referred to even by a single ref. This doesn't really scale.\n\n* Maintain a list of all the submodules in a repository. This would \nhave to be in a separate metadata branch, and would get rather hairy \nwhen we need to merge from a remote that has added other submodules.\n\n* Represent the submodules as refs instead of independent \nrepositories. This is my proposal for subrepositories.\n\nHowever, I feel that all of these are too drastic changes to make in \ngit-submodule, given that it is already well-established.\n\nThe minor problems, like lack of active branch tracking and multiple \nmount points of a module, could in principle be fixed in \ngit-submodule. But again, I have no fondness for complex shell \nprogramming. Perhaps it was justified when the only interface to git's \nfunctionality were the command-line tools, but nowadays there are \nvarious ways to manipulate git repositories from real programming \nlanguages through real libraries (libgit2, dulwich, etc), and I prefer \nto use those, so I don't really have any motivation to touch \ngit-submodule.\n\n\nLauri\n"},{"id":"201115","messageId":"7vzk3p1xh3.fsf@alter.siamese.dyndns.org","threadId":"31801","inReplyTo":"20121014002304.14167k2j2ctspiuw.lealanko@webmail.helsinki.fi","subject":"Re: A design for subrepositories","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-10-14T04:36:24Z","receivedAt":"2012-10-14T04:36:24Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Lauri Alanko\" writes:\n\n> Firstly, if you simply do \"git submodule add ./foo\" (the obligatory\n> \"./\" being quite an unobvious pitfall), you get something quite\n> fragile, since now we have submodule.foo.url = ./foo. If the\n> submodules ever get reorganized and foo is moved to ./bar, then it is\n> impossible to check out older versions or alternate branches, since\n> the submodule is no longer where it is expected to be at the origin.\n\nIsn't that exactly what the \"module name\" vs \"module path\" mapping\nin .gitmodules file is meant to address?\n\n> But still, \"git submodule update\" only looks at the modules in the\n> currently checked-out tree. If we have other branches or old tags that\n> refer to other submodules, there's no simple way to fetch those, too.\n\nDidn't I already suggest you to think about how you can improve\nexisting \"git submodule\" to suit your use case better?\n"},{"id":"201159","messageId":"20121014131928.25943ezwa6fveyls.lealanko@webmail.helsinki.fi","threadId":"31801","inReplyTo":"7vzk3p1xh3.fsf@alter.siamese.dyndns.org","subject":"Re: A design for subrepositories","fromName":"Lauri Alanko","fromEmail":"la@iki.fi","sentAt":"2012-10-14T10:19:28Z","receivedAt":"2012-10-14T10:19:28Z","isPatch":false,"sender":{"key":"la@iki.fi","avatar":null},"body":"Quoting \"Junio C Hamano\" :\n\n>> If the\n>> submodules ever get reorganized and foo is moved to ./bar, then it is\n>> impossible to check out older versions or alternate branches, since\n>> the submodule is no longer where it is expected to be at the origin.\n>\n> Isn't that exactly what the \"module name\" vs \"module path\" mapping\n> in .gitmodules file is meant to address?\n\nYes, and as I showed after the part you quoted, it is possible to \nrefer to a module by name, although it looks like such a hack that I \ncan't imagine it's currently something that git-submodule is intended \nto support.\n\n>> But still, \"git submodule update\" only looks at the modules in the\n>> currently checked-out tree. If we have other branches or old tags that\n>> refer to other submodules, there's no simple way to fetch those, too.\n\n> Didn't I already suggest you to think about how you can improve\n> existing \"git submodule\" to suit your use case better?\n\nYes, and I listed three possible ways. Two of them seem technically \nunattractive, whereas one of them (submodules as ref directories) \nseems like a huge change that could introduce incompatibilities. That \nis why a separate tool seems like a cleaner choice.\n\nIf you want enhancements to git-submodule, at least deign to comment \non the issues above.\n\nThere is actually a fourth alternative: extend the git protocol so \nthat a remote repository could be queried for its list of submodules. \nBut this seems particularly icky: git is at its core such a low-level \nframework. Nested repositories are such a high-level concept that \nsomething is wrong if the core needs specialized support for it. The \nref directories approach, on the other hand, is completely transparent \nto standard tools.\n\n\nLauri\n"},{"id":"201172","messageId":"507ABDF3.4040106@web.de","threadId":"31801","inReplyTo":"20121014131928.25943ezwa6fveyls.lealanko@webmail.helsinki.fi","subject":"Re: A design for subrepositories","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2012-10-14T13:28:19Z","receivedAt":"2012-10-14T13:28:19Z","isPatch":false,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 14.10.2012 12:19, schrieb Lauri Alanko:\n> Quoting \"Junio C Hamano\" :\n> \n>>> If the\n>>> submodules ever get reorganized and foo is moved to ./bar, then it is\n>>> impossible to check out older versions or alternate branches, since\n>>> the submodule is no longer where it is expected to be at the origin.\n>>\n>> Isn't that exactly what the \"module name\" vs \"module path\" mapping\n>> in .gitmodules file is meant to address?\n> \n> Yes, and as I showed after the part you quoted, it is possible to refer to a module by name, although it looks like such a hack that I can't imagine it's currently something that git-submodule is intended to support.\n\nYour initial statement is not correct. It is possible to check out older\nversions or alternate branches (at least since we moved the .git directory\ninto the .git directory of the superproject). So no improvement gained\nhere by your proposal (although I concede that the current user experience\nis suboptimal until my recursive submodule update work hits mainline).\n\n>>> But still, \"git submodule update\" only looks at the modules in the\n>>> currently checked-out tree. If we have other branches or old tags that\n>>> refer to other submodules, there's no simple way to fetch those, too.\n\nDid you notice that \"git fetch\" fetches all those submodules too which\nhave been updated in the commits fetched for the superproject, no matter\non what branch they are on?\n\n>> Didn't I already suggest you to think about how you can improve\n>> existing \"git submodule\" to suit your use case better?\n> \n> Yes, and I listed three possible ways. Two of them seem technically unattractive, whereas one of them (submodules as ref directories) seems like a huge change that could introduce incompatibilities. That is why a separate tool seems like a cleaner choice.\n\nWhat's wrong with making git clone all submodules together with the\nsuperproject (when the user said he wants to update all submodules on\nclone too by setting a - still to be added - config option)? That's my\nplan to make automagic recursive submodule cloning work and it would\nclone all submodules seen in the history of the superproject to\n.git/modules so they could easily be checked out later (and those\npresent in the HEAD of the superproject will be checked out immediately\nlike \"git clone --recurse-submodules\" does right now). Were not there\nyet, but that's how I believe that should work.\n\n> There is actually a fourth alternative: extend the git protocol so that a remote repository could be queried for its list of submodules.\n\nThat information is contained in the different versions of the .gitmodules\nfile, so no need to extend anything here.\n\nI saw nothing in your proposal which couldn't been handled by submodules,\nand for every issue there already have been proposals on how to do that.\nSo adding another tool doesn't make any sense here. But you are welcome\nhelping us to improve the submodule script (and some core commands too)\nto make submodules cover your use case too.\n"},{"id":"201176","messageId":"20121014182746.42895rwvalv4uoz6.lealanko@webmail.helsinki.fi","threadId":"31801","inReplyTo":"507ABDF3.4040106@web.de","subject":"Re: A design for subrepositories","fromName":"Lauri Alanko","fromEmail":"la@iki.fi","sentAt":"2012-10-14T15:27:46Z","receivedAt":"2012-10-14T15:27:46Z","isPatch":false,"sender":{"key":"la@iki.fi","avatar":null},"body":"Quoting \"Jens Lehmann\" :\n\n>>>> If the\n>>>> submodules ever get reorganized and foo is moved to ./bar, then it is\n>>>> impossible to check out older versions or alternate branches, since\n>>>> the submodule is no longer where it is expected to be at the origin.\n\n> Your initial statement is not correct.\n\nPlease elaborate. My initial statement was about \"git submodule add \n./foo\", and this is what I get:\n\nla@bq:~/tmp$ git --version\ngit version 1.8.0.rc2.2.gfc364c7\nla@bq:~/tmp$ git init super\nInitialized empty Git repository in /home/la/tmp/super/.git/\nla@bq:~/tmp$ cd super\nla@bq:~/tmp/super$ echo foo > foo\nla@bq:~/tmp/super$ git add foo\nla@bq:~/tmp/super$ git ci -m foo\n[master (root-commit) a0dd543] foo\n 1 file changed, 1 insertion(+)\n create mode 100644 foo\nla@bq:~/tmp/super$ git init sub\nInitialized empty Git repository in /home/la/tmp/super/sub/.git/\nla@bq:~/tmp/super$ cd sub\nla@bq:~/tmp/super/sub$ echo bar > bar\nla@bq:~/tmp/super/sub$ git add bar\nla@bq:~/tmp/super/sub$ git ci -m bar\n[master (root-commit) a6ee6d6] bar\n 1 file changed, 1 insertion(+)\n create mode 100644 bar\nla@bq:~/tmp/super/sub$ cd ..\nla@bq:~/tmp/super$ git submodule add ./sub\nAdding existing repo at 'sub' to the index\nla@bq:~/tmp/super$ git ci -m sub\n[master cb289e8] sub\n 2 files changed, 4 insertions(+)\n create mode 100644 .gitmodules\n create mode 160000 sub\nla@bq:~/tmp/super$ git branch old\nla@bq:~/tmp/super$ git mv sub movedsub\nfatal: source directory is empty, source=sub, destination=movedsub\nla@bq:~/tmp/super$ mv sub movedsub\nla@bq:~/tmp/super$ git rm sub\nrm 'sub'\nla@bq:~/tmp/super$ git add movedsub\nla@bq:~/tmp/super$ git config -f .gitmodules submodule.sub.path movedsub\nla@bq:~/tmp/super$ git config -f .gitmodules submodule.sub.url ./movedsub\nla@bq:~/tmp/super$ git ci -am movedsub\n[master 5598bc0] movedsub\n 2 files changed, 2 insertions(+), 2 deletions(-)\n rename sub => movedsub (100%)\nla@bq:~/tmp/super$ cd ..\nla@bq:~/tmp$ git clone super superc\nCloning into 'superc'...\ndone.\nla@bq:~/tmp$ cd superc\nla@bq:~/tmp/superc$ git co old\nBranch old set up to track remote branch old from origin.\nSwitched to a new branch 'old'\nla@bq:~/tmp/superc$ git submodule update --init\nSubmodule 'sub' (/home/la/tmp/super/sub) registered for path 'sub'\nfatal: repository '/home/la/tmp/super/sub' does not exist\nClone of '/home/la/tmp/super/sub' into submodule path 'sub' failed\n\nSo a normal relative path in .gitmodules to inside the tree is \nfragile, since the location of the submodule can change.\n\n> Did you notice that \"git fetch\" fetches all those submodules too which\n> have been updated in the commits fetched for the superproject, no matter\n> on what branch they are on?\n\nNo. This would be great, but this is what I get:\n\nla@bq:~/tmp$ git init super\nInitialized empty Git repository in /home/la/tmp/super/.git/\nla@bq:~/tmp$ cd super\nla@bq:~/tmp/super$ echo foo > foo\nla@bq:~/tmp/super$ git add foo\nla@bq:~/tmp/super$ git ci -m foo\n[master (root-commit) 0f207c9] foo\n 1 file changed, 1 insertion(+)\n create mode 100644 foo\nla@bq:~/tmp/super$ git branch nosubs\nla@bq:~/tmp/super$ git init sub\nInitialized empty Git repository in /home/la/tmp/super/sub/.git/\nla@bq:~/tmp/super$ cd sub\nla@bq:~/tmp/super/sub$ echo bar > bar\nla@bq:~/tmp/super/sub$ git add bar\nla@bq:~/tmp/super/sub$ git ci -m bar\n[master (root-commit) 180c6c9] bar\n 1 file changed, 1 insertion(+)\n create mode 100644 bar\nla@bq:~/tmp/super/sub$ cd ..\nla@bq:~/tmp/super$ git submodule add ./sub\nAdding existing repo at 'sub' to the index\nla@bq:~/tmp/super$ git ci -m sub\n[master 16cff18] sub\n 2 files changed, 4 insertions(+)\n create mode 100644 .gitmodules\n create mode 160000 sub\nla@bq:~/tmp/super$ cd ..\nla@bq:~/tmp$ git clone super superc\nCloning into 'superc'...\ndone.\nla@bq:~/tmp$ cd superc\nla@bq:~/tmp/superc$ git submodule update --init\nSubmodule 'sub' (/home/la/tmp/super/sub) registered for path 'sub'\nCloning into 'sub'...\ndone.\nSubmodule path 'sub': checked out '180c6c979289f4e25525003673e51d0e39dab8f6'\nla@bq:~/tmp/superc$ cd ../super/sub\nla@bq:~/tmp/super/sub$ echo baz >> bar\nla@bq:~/tmp/super/sub$ git ci -am baz\n[master 652c8b3] baz\n 1 file changed, 1 insertion(+)\nla@bq:~/tmp/super/sub$ cd ..\nla@bq:~/tmp/super$ git ci -am subbaz\n[master c7c3bfc] subbaz\n 1 file changed, 1 insertion(+), 1 deletion(-)\nla@bq:~/tmp/super$ cd ../superc\nla@bq:~/tmp/superc$ git co nosubs\nwarning: unable to rmdir sub: Directory not empty\nBranch nosubs set up to track remote branch nosubs from origin.\nSwitched to a new branch 'nosubs'\nla@bq:~/tmp/superc$ git fetch --recurse-submodules=yes\nremote: Counting objects: 3, done.\nremote: Compressing objects: 100% (2/2), done.\nremote: Total 2 (delta 1), reused 0 (delta 0)\nUnpacking objects: 100% (2/2), done.\n From /home/la/tmp/super\n 16cff18..c7c3bfc master -> origin/master\nla@bq:~/tmp/superc$ git co master\nSwitched to branch 'master'\nYour branch is behind 'origin/master' by 1 commit, and can be fast-forwarded.\nla@bq:~/tmp/superc$ git fetch --recurse-submodules=yes\nFetching submodule sub\nremote: Counting objects: 5, done.\nremote: Total 3 (delta 0), reused 0 (delta 0)\nUnpacking objects: 100% (3/3), done.\n From /home/la/tmp/super/sub\n 180c6c9..652c8b3 master -> origin/master\n\nSo I had to checkout master in order to fetch the updates to the \nsubmodule used by master.\n\n> What's wrong with making git clone all submodules together with the\n> superproject (when the user said he wants to update all submodules on\n> clone too by setting a - still to be added - config option)?\n\nDepends on how it's done. In a previous mail I just considered various \nways to do it. If I see correctly, your choice is to read .gitmodules \nfrom every branch and every tag to find the total set of submodules \nused by the repository. As I said already, that is certainly possible, \nbut it's just not very scalable, if fetch operations slow down \nlinearly in the number of tags.\n\nBut no matter the technical issues, it seems that you at least have \nthe _intention_ to eventually support self-contained, HEAD-independent \nrepository collections. That already is valuable information, thanks.\n\n\nLauri\n"},{"id":"201181","messageId":"507AE3FC.3040200@web.de","threadId":"31801","inReplyTo":"20121014182746.42895rwvalv4uoz6.lealanko@webmail.helsinki.fi","subject":"Re: A design for subrepositories","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2012-10-14T16:10:36Z","receivedAt":"2012-10-14T16:10:36Z","isPatch":false,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 14.10.2012 17:27, schrieb Lauri Alanko:\n> Quoting \"Jens Lehmann\" :\n> \n>>>>> If the\n>>>>> submodules ever get reorganized and foo is moved to ./bar, then it is\n>>>>> impossible to check out older versions or alternate branches, since\n>>>>> the submodule is no longer where it is expected to be at the origin.\n> \n>> Your initial statement is not correct.\n> \n> Please elaborate. My initial statement was about \"git submodule add ./foo\", and this is what I get:\n> \n> la@bq:~/tmp$ git --version\n> git version 1.8.0.rc2.2.gfc364c7\n> la@bq:~/tmp$ git init super\n> Initialized empty Git repository in /home/la/tmp/super/.git/\n> la@bq:~/tmp$ cd super\n> la@bq:~/tmp/super$ echo foo > foo\n> la@bq:~/tmp/super$ git add foo\n> la@bq:~/tmp/super$ git ci -m foo\n> [master (root-commit) a0dd543] foo\n> 1 file changed, 1 insertion(+)\n> create mode 100644 foo\n> la@bq:~/tmp/super$ git init sub\n> Initialized empty Git repository in /home/la/tmp/super/sub/.git/\n> la@bq:~/tmp/super$ cd sub\n> la@bq:~/tmp/super/sub$ echo bar > bar\n> la@bq:~/tmp/super/sub$ git add bar\n> la@bq:~/tmp/super/sub$ git ci -m bar\n> [master (root-commit) a6ee6d6] bar\n> 1 file changed, 1 insertion(+)\n> create mode 100644 bar\n> la@bq:~/tmp/super/sub$ cd ..\n> la@bq:~/tmp/super$ git submodule add ./sub\n> Adding existing repo at 'sub' to the index\n> la@bq:~/tmp/super$ git ci -m sub\n> [master cb289e8] sub\n> 2 files changed, 4 insertions(+)\n> create mode 100644 .gitmodules\n> create mode 160000 sub\n> la@bq:~/tmp/super$ git branch old\n> la@bq:~/tmp/super$ git mv sub movedsub\n> fatal: source directory is empty, source=sub, destination=movedsub\n\nThis error here indicates that we didn't teach git to properly move\na submodule yet. It is one of my next goals to make \"git [submodule]\nmv sub movedsub\" do the right thing here. To do these steps manually\nyou'll additionally have to do the following before moving the\nsubmodule (because after moving it the relative paths will be broken):\n\n$ HASH=$(cd sub; git rev-parse HEAD)\n\n> la@bq:~/tmp/super$ mv sub movedsub\n\nCurrently it is better to remove the submodule here, as recreating it\nwith a \"git submodule update\" later will get the relative paths right.\n\n> la@bq:~/tmp/super$ git rm sub\n> rm 'sub'\n> la@bq:~/tmp/super$ git add movedsub\n\nAnd to git this adds a completely different submodule (as its name\nis not \"sub\"), which breaks your expectation. To do what you intended\nuse this line instead:\n\n$ git update-index --add --cacheinfo 160000 $HASH movedsub\n\n(With the \"--next\" option currently in the \"next\" branch of Junio's\nrepo a \"git submodule add --name sub movedsub\" should do the job.\nUntil then a bit more magic is necessary).\n\n> la@bq:~/tmp/super$ git config -f .gitmodules submodule.sub.path movedsub\n> la@bq:~/tmp/super$ git config -f .gitmodules submodule.sub.url ./movedsub\n> la@bq:~/tmp/super$ git ci -am movedsub\n> [master 5598bc0] movedsub\n> 2 files changed, 2 insertions(+), 2 deletions(-)\n> rename sub => movedsub (100%)\n> la@bq:~/tmp/super$ cd ..\n> la@bq:~/tmp$ git clone super superc\n> Cloning into 'superc'...\n> done.\n> la@bq:~/tmp$ cd superc\n> la@bq:~/tmp/superc$ git co old\n> Branch old set up to track remote branch old from origin.\n> Switched to a new branch 'old'\n> la@bq:~/tmp/superc$ git submodule update --init\n> Submodule 'sub' (/home/la/tmp/super/sub) registered for path 'sub'\n> fatal: repository '/home/la/tmp/super/sub' does not exist\n> Clone of '/home/la/tmp/super/sub' into submodule path 'sub' failed\n\nAnd that fails because to be able to clone a submodule it has to be\npushed into its own repo first, so it can be cloned from there somewhere\nelse. After doing that this will work.\n\n> So a normal relative path in .gitmodules to inside the tree is fragile, since the location of the submodule can change.\n\nAs I said, the current user experience is suboptimal. The test case\n'submodule update properly revives a moved submodule' in t7406 shows\nwhat has to be done with current git to properly move a submodule,\nwhich is way too much to remember for a regular git user.\n"},{"id":"201183","messageId":"507AE52A.3080807@web.de","threadId":"31801","inReplyTo":"20121014182746.42895rwvalv4uoz6.lealanko@webmail.helsinki.fi","subject":"Re: A design for subrepositories","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2012-10-14T16:15:38Z","receivedAt":"2012-10-14T16:15:38Z","isPatch":false,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 14.10.2012 17:27, schrieb Lauri Alanko:\n> Quoting \"Jens Lehmann\" :\n>> What's wrong with making git clone all submodules together with the\n>> superproject (when the user said he wants to update all submodules on\n>> clone too by setting a - still to be added - config option)?\n> \n> Depends on how it's done. In a previous mail I just considered various ways to do it. If I see correctly, your choice is to read .gitmodules from every branch and every tag to find the total set of submodules used by the repository. As I said already, that is certainly possible, but it's just not very scalable, if fetch operations slow down linearly in the number of tags.\n\nCurrently \"git fetch\" checks all newly fetched commits for changes in\ngitlinks too, so that would just add another file to that. And as a\nfetch is pretty much linear in the number of newly fetched commits\nanyway, its performance impact should be minimal.\n"},{"id":"201184","messageId":"507AE773.1010301@web.de","threadId":"31801","inReplyTo":"20121014182746.42895rwvalv4uoz6.lealanko@webmail.helsinki.fi","subject":"Re: A design for subrepositories","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2012-10-14T16:25:23Z","receivedAt":"2012-10-14T16:25:23Z","isPatch":false,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 14.10.2012 17:27, schrieb Lauri Alanko:\n> Quoting \"Jens Lehmann\" :\n>> Did you notice that \"git fetch\" fetches all those submodules too which\n>> have been updated in the commits fetched for the superproject, no matter\n>> on what branch they are on?\n> \n> No. This would be great, but this is what I get:\n> \n> la@bq:~/tmp$ git init super\n> Initialized empty Git repository in /home/la/tmp/super/.git/\n> la@bq:~/tmp$ cd super\n> la@bq:~/tmp/super$ echo foo > foo\n> la@bq:~/tmp/super$ git add foo\n> la@bq:~/tmp/super$ git ci -m foo\n> [master (root-commit) 0f207c9] foo\n> 1 file changed, 1 insertion(+)\n> create mode 100644 foo\n> la@bq:~/tmp/super$ git branch nosubs\n> la@bq:~/tmp/super$ git init sub\n> Initialized empty Git repository in /home/la/tmp/super/sub/.git/\n> la@bq:~/tmp/super$ cd sub\n> la@bq:~/tmp/super/sub$ echo bar > bar\n> la@bq:~/tmp/super/sub$ git add bar\n> la@bq:~/tmp/super/sub$ git ci -m bar\n> [master (root-commit) 180c6c9] bar\n> 1 file changed, 1 insertion(+)\n> create mode 100644 bar\n> la@bq:~/tmp/super/sub$ cd ..\n> la@bq:~/tmp/super$ git submodule add ./sub\n> Adding existing repo at 'sub' to the index\n> la@bq:~/tmp/super$ git ci -m sub\n> [master 16cff18] sub\n> 2 files changed, 4 insertions(+)\n> create mode 100644 .gitmodules\n> create mode 160000 sub\n> la@bq:~/tmp/super$ cd ..\n> la@bq:~/tmp$ git clone super superc\n> Cloning into 'superc'...\n> done.\n> la@bq:~/tmp$ cd superc\n> la@bq:~/tmp/superc$ git submodule update --init\n> Submodule 'sub' (/home/la/tmp/super/sub) registered for path 'sub'\n> Cloning into 'sub'...\n> done.\n> Submodule path 'sub': checked out '180c6c979289f4e25525003673e51d0e39dab8f6'\n> la@bq:~/tmp/superc$ cd ../super/sub\n> la@bq:~/tmp/super/sub$ echo baz >> bar\n> la@bq:~/tmp/super/sub$ git ci -am baz\n> [master 652c8b3] baz\n> 1 file changed, 1 insertion(+)\n> la@bq:~/tmp/super/sub$ cd ..\n> la@bq:~/tmp/super$ git ci -am subbaz\n> [master c7c3bfc] subbaz\n> 1 file changed, 1 insertion(+), 1 deletion(-)\n> la@bq:~/tmp/super$ cd ../superc\n> la@bq:~/tmp/superc$ git co nosubs\n> warning: unable to rmdir sub: Directory not empty\n> Branch nosubs set up to track remote branch nosubs from origin.\n> Switched to a new branch 'nosubs'\n> la@bq:~/tmp/superc$ git fetch --recurse-submodules=yes\n> remote: Counting objects: 3, done.\n> remote: Compressing objects: 100% (2/2), done.\n> remote: Total 2 (delta 1), reused 0 (delta 0)\n> Unpacking objects: 100% (2/2), done.\n> From /home/la/tmp/super\n> 16cff18..c7c3bfc master -> origin/master\n> la@bq:~/tmp/superc$ git co master\n> Switched to branch 'master'\n> Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded.\n> la@bq:~/tmp/superc$ git fetch --recurse-submodules=yes\n> Fetching submodule sub\n> remote: Counting objects: 5, done.\n> remote: Total 3 (delta 0), reused 0 (delta 0)\n> Unpacking objects: 100% (3/3), done.\n> From /home/la/tmp/super/sub\n> 180c6c9..652c8b3 master -> origin/master\n> \n> So I had to checkout master in order to fetch the updates to the submodule used by master.\n\nYes, when you switch to a branch which hasn't got that submodule at all\nthat is the case (as currently the .gitmodules found in the work tree is\nused to do the path -> name mapping). The culprit is the \"git fetch\" does\nnot yet examine the .gitmodules file of the commit it finds a submodule\nchange in, but uses the one currently found inside the work tree. But I'll\nhave to tackle too soon, as that also poses a problem when the submodule\nwas moved. So \"no matter what branch they are on\" is not always correct\nat the moment ;-)\n\nAgain, the user experience is currently suboptimal.\n"},{"id":"201188","messageId":"7v4nlxylpm.fsf@alter.siamese.dyndns.org","threadId":"31801","inReplyTo":"507AE773.1010301@web.de","subject":"Re: A design for subrepositories","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-10-14T18:04:05Z","receivedAt":"2012-10-14T18:04:05Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jens Lehmann writes:\n\n> Again, the user experience is currently suboptimal.\n\nYou mentioned multiple things in your responses that you are\nplanning to address, but I am wondering if the first step before\ndoing anything else is to have a list of known-to-be-suboptimal\nthings and publish it somewhere other people can find it. Then\nLauri or others may able to help code the design of the approach to\naddress them for items you already have designs for, and they may\neven be able to help designing the approach for the ones you don't.\n\nMore importantly, they do not have to waste time coming up with\nincompatible tools. Adding \"works in this scenario that is\ndifferent from those other slightly different tools\" to the mix of\nthird-party tool set would fragment and confuse the user base\n(\"which one of 47 different tools, all of which are incomplete,\nshould I use?\") and dilute developer attention. They all at some\npoint want to interact with the core side, and without an overall\nconsistent design and coordination, some of their demand on the core\nside would end up being imcompatible.\n\nThe \"just let .gitmodules record which branch is of interest,\nwithout checking out a specific commit bound to the superproject\ntree and using as a base for diff\" (aka floating submodule) could be\none of the items on the list, for example; to support it, we should\nnot have to throw the entire \"git submodule\" with the bathwater.\n\nThanks.\n"},{"id":"201192","messageId":"507B1335.10105@web.de","threadId":"31801","inReplyTo":"7v4nlxylpm.fsf@alter.siamese.dyndns.org","subject":"Re: A design for subrepositories","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2012-10-14T19:32:05Z","receivedAt":"2012-10-14T19:32:05Z","isPatch":false,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 14.10.2012 20:04, schrieb Junio C Hamano:\n> Jens Lehmann writes:\n> \n>> Again, the user experience is currently suboptimal.\n> \n> You mentioned multiple things in your responses that you are\n> planning to address, but I am wondering if the first step before\n> doing anything else is to have a list of known-to-be-suboptimal\n> things and publish it somewhere other people can find it. Then\n> Lauri or others may able to help code the design of the approach to\n> address them for items you already have designs for, and they may\n> even be able to help designing the approach for the ones you don't.\n\nI'm keeping such a list in the \"Issues still to be tackled in this\nrepo\" section of the Wiki page of my github repo:\n https://github.com/jlehmann/git-submod-enhancements/wiki\n\nCurrently that's just a collection of things to do and bugs to fix,\nbut if people are interested I'm willing to add descriptions of the\nsolutions I have in mind for those topics.\n\n> More importantly, they do not have to waste time coming up with\n> incompatible tools. Adding \"works in this scenario that is\n> different from those other slightly different tools\" to the mix of\n> third-party tool set would fragment and confuse the user base\n> (\"which one of 47 different tools, all of which are incomplete,\n> should I use?\") and dilute developer attention. They all at some\n> point want to interact with the core side, and without an overall\n> consistent design and coordination, some of their demand on the core\n> side would end up being imcompatible.\n> \n> The \"just let .gitmodules record which branch is of interest,\n> without checking out a specific commit bound to the superproject\n> tree and using as a base for diff\" (aka floating submodule) could be\n> one of the items on the list, for example; to support it, we should\n> not have to throw the entire \"git submodule\" with the bathwater.\n\nYup, that's also on that list under \"always tip\" mode.\n"},{"id":"201194","messageId":"20121015015934.1597359e76eaqvh2.lealanko@webmail.helsinki.fi","threadId":"31801","inReplyTo":"507AE773.1010301@web.de","subject":"Re: A design for subrepositories","fromName":"Lauri Alanko","fromEmail":"la@iki.fi","sentAt":"2012-10-14T22:59:34Z","receivedAt":"2012-10-14T22:59:34Z","isPatch":false,"sender":{"key":"la@iki.fi","avatar":null},"body":">> la@bq:~/tmp/super$ git mv sub movedsub\n>> fatal: source directory is empty, source=sub, destination=movedsub\n>\n> This error here indicates that we didn't teach git to properly move\n> a submodule yet. It is one of my next goals to make \"git [submodule]\n> mv sub movedsub\" do the right thing here.\n\nI'll digress here a bit: I'm not really fond of the idea of adding\nspecial-purpose support into the core git commands. It just makes them\nmessier, and there will always be other tools that won't be supported by\ngit directly. I'd much rather see an mv-hook that arbitrary extensions\ncould use to update metadata associated with a tree entry.\n\nIndeed, one of the reasons a separate tool seemed attractive to me was\nthat that way I could be sure that the tool was a high-level utility\nthat was completely implemented on top of basic low-level git\noperations. The fact that git's submodule support manifests as bits and\npieces in various parts of the core seems a bit worrisome to me.\n\n(Moreover, it's confusing to the user. I read the git-submodule man page\nand thought that that described all the available submodule operations.\nOnly now did I find out that clone and fetch also have built-in\nsubmodule functionality.)\n\n>> la@bq:~/tmp/super$ mv sub movedsub\n>\n> Currently it is better to remove the submodule here, as recreating it\n> with a \"git submodule update\" later will get the relative paths right.\n\nThis was a bit of a special case, as this was the original directory\nwhere we did \"git init sub\" and \"git submodule add ./sub\". So \"sub\"\nactually contains the real repository, not a gitlink to\n.git/modules/sub. Arguably \"git submodule add\" should move the local\nsubmodule's repository there.\n\n>> la@bq:~/tmp/super$ git rm sub\n>> rm 'sub'\n>> la@bq:~/tmp/super$ git add movedsub\n>\n> And to git this adds a completely different submodule (as its name\n> is not \"sub\"), which breaks your expectation.\n\nSubmodule? This is just a normal git add, not git submodule add. I\nthought this just adds to the index a gitlink with the head revision in\nmovedsub, which is the same as the head revision was in sub, so it's\ndetected as a move of a gitlink.\n\n> To do what you intended\n> use this line instead:\n>\n> $ git update-index --add --cacheinfo 160000 $HASH movedsub\n\nDoesn't this do exactly the same thing as \"git add\" for a directory\ncontaining a repository?\n\n>> la@bq:~/tmp/superc$ git submodule update --init\n>> Submodule 'sub' (/home/la/tmp/super/sub) registered for path 'sub'\n>> fatal: repository '/home/la/tmp/super/sub' does not exist\n>> Clone of '/home/la/tmp/super/sub' into submodule path 'sub' failed\n>\n> And that fails because to be able to clone a submodule it has to be\n> pushed into its own repo first, so it can be cloned from there somewhere\n> else. After doing that this will work.\n\nSorry, but I can't get this to work. To me it seems that when fetching\nsubmodules from the origin, submodule.sub.url has to point to the actual\nlocation of the repository, and if this is outdated or missing, the\nfetch won't work.\n\nIt would make sense that if the url is missing, the submodule repo\ninside origin's .git/modules would be used, but this doesn't seem to be\nthe case currently.\n\n> Currently \"git fetch\" checks all newly fetched commits for changes in\n> gitlinks too, so that would just add another file to that.\n\nI only now realized that it is indeed enough to check .gitmodules only\nin the _updated_ refs. The older refs are interested in their submodules\nonly up to a certain commit, and even if those submodules have been\nupdated upstream, we won't be interested in them until we have trees\nwith gitlinks pointing to the newer revisions.\n\nSo it turns out that my main technical argument against git-submodule's\npotential scalability was false, and it is indeed feasible to make it\nsupport all the features I require.\n\nHowever, \"always tip\" mode would break this, since then even non-updated\nbranches might be interested in upstream changes to a submodule.\n\n\nAnyway, I am a bit surprised to hear of such active development for\ngit-submodule. It's pretty old now (the shell script says 2007), and I\nthought that if it were to ever support the kind of basic functionality\nI require, it would do so already.\n\nHow soon do you envision support for bare repositories with submodules?\n\n\nLauri\n"},{"id":"201272","messageId":"507C439A.6060202@web.de","threadId":"31801","inReplyTo":"20121015015934.1597359e76eaqvh2.lealanko@webmail.helsinki.fi","subject":"Re: A design for subrepositories","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2012-10-15T17:10:50Z","receivedAt":"2012-10-15T17:10:50Z","isPatch":false,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 15.10.2012 00:59, schrieb Lauri Alanko:\n>>> la@bq:~/tmp/super$ git mv sub movedsub\n>>> fatal: source directory is empty, source=sub, destination=movedsub\n>>\n>> This error here indicates that we didn't teach git to properly move\n>> a submodule yet. It is one of my next goals to make \"git [submodule]\n>> mv sub movedsub\" do the right thing here.\n> \n> I'll digress here a bit: I'm not really fond of the idea of adding\n> special-purpose support into the core git commands. It just makes them\n> messier, and there will always be other tools that won't be supported by\n> git directly. I'd much rather see an mv-hook that arbitrary extensions\n> could use to update metadata associated with a tree entry.\n\nOne third of the participants of the GitSurvey2010 stated that they are\nusing submodules (e.g. more than gitattributes), so adding some support\nfor them into the core doesn't look that unwarranted to me. And believe\nme, without putting support in there the user experience will stay\nsuboptimal.\n\n> Indeed, one of the reasons a separate tool seemed attractive to me was\n> that that way I could be sure that the tool was a high-level utility\n> that was completely implemented on top of basic low-level git\n> operations. The fact that git's submodule support manifests as bits and\n> pieces in various parts of the core seems a bit worrisome to me.\n\nI see it the other way around: Due to the fact that submodules were\nonly accessible via the submodule script and not integrated into the\ncore made a lot of people (e.g. the Jenkins Git plugin we are using at\nwork) code around that. That wouldn't have been necessary if I would\nhave finished my submodule update work at that time.\n\nAnd e.g. you can't forget to add changes inside the submodule anymore\nsince diff and status learned to show those changes. And we still have\nmis-merges at my dayjob due to not updated submodules, which will go\naway the moment merge learns to update all submodules without merge\nconflicts. And so on.\n\n> (Moreover, it's confusing to the user. I read the git-submodule man page\n> and thought that that described all the available submodule operations.\n> Only now did I find out that clone and fetch also have built-in\n> submodule functionality.)\n\nThen the man page might need some overhaul. Care to take a look?\n\n>>> la@bq:~/tmp/super$ mv sub movedsub\n>>\n>> Currently it is better to remove the submodule here, as recreating it\n>> with a \"git submodule update\" later will get the relative paths right.\n> \n> This was a bit of a special case, as this was the original directory\n> where we did \"git init sub\" and \"git submodule add ./sub\". So \"sub\"\n> actually contains the real repository, not a gitlink to\n> .git/modules/sub. Arguably \"git submodule add\" should move the local\n> submodule's repository there.\n\nThat sounds like a good idea.\n\n>>> la@bq:~/tmp/super$ git rm sub\n>>> rm 'sub'\n>>> la@bq:~/tmp/super$ git add movedsub\n>>\n>> And to git this adds a completely different submodule (as its name\n>> is not \"sub\"), which breaks your expectation.\n> \n> Submodule? This is just a normal git add, not git submodule add. I\n> thought this just adds to the index a gitlink with the head revision in\n> movedsub, which is the same as the head revision was in sub, so it's\n> detected as a move of a gitlink.\n\nYou're free to use simple gitlinks, but then you can't expect existing\nand coming goodies - like git being able to move them around in the\nwork tree - work all by itself, because they are only possible with\nsubmodule support.\n\n>> To do what you intended\n>> use this line instead:\n>>\n>> $ git update-index --add --cacheinfo 160000 $HASH movedsub\n> \n> Doesn't this do exactly the same thing as \"git add\" for a directory\n> containing a repository?\n\nIn my test case \"git add movesub\" silently does nothing, as my\ndirectory is empty. So I need the update-index here.\n\n>>> la@bq:~/tmp/superc$ git submodule update --init\n>>> Submodule 'sub' (/home/la/tmp/super/sub) registered for path 'sub'\n>>> fatal: repository '/home/la/tmp/super/sub' does not exist\n>>> Clone of '/home/la/tmp/super/sub' into submodule path 'sub' failed\n>>\n>> And that fails because to be able to clone a submodule it has to be\n>> pushed into its own repo first, so it can be cloned from there somewhere\n>> else. After doing that this will work.\n> \n> Sorry, but I can't get this to work. To me it seems that when fetching\n> submodules from the origin, submodule.sub.url has to point to the actual\n> location of the repository, and if this is outdated or missing, the\n> fetch won't work.\n> \n> It would make sense that if the url is missing, the submodule repo\n> inside origin's .git/modules would be used, but this doesn't seem to be\n> the case currently.\n\nNo it isn't. Patches welcome ;-)\n\n> Anyway, I am a bit surprised to hear of such active development for\n> git-submodule. It's pretty old now (the shell script says 2007), and I\n> thought that if it were to ever support the kind of basic functionality\n> I require, it would do so already.\n\nSo much to do, so little time.\n\n> How soon do you envision support for bare repositories with submodules?\n\nI'm not sure what you mean by that and what your use case is, but I'll\nbe happy to discuss design issues with you. But as that is not my itch\nI don't expect to be working on that soon, as my next main goal is to\nget recursive checkout working (currently I'm removing the obstacles\nI find in my way towards that, but there is still quite some work to\ndo until I get there).\n"},{"id":"201540","messageId":"20121019033158.93403x1gcanyjo8u.lealanko@webmail.helsinki.fi","threadId":"31801","inReplyTo":"7v4nlxylpm.fsf@alter.siamese.dyndns.org","subject":"A design for distributed submodules","fromName":"Lauri Alanko","fromEmail":"la@iki.fi","sentAt":"2012-10-19T00:31:58Z","receivedAt":"2012-10-19T00:31:58Z","isPatch":false,"sender":{"key":"la@iki.fi","avatar":null},"body":"\nI think I finally agree that it's best to develop submodules further\nrather than introduce a new tool for the functionality I require. Here\nare some explicit proposals for submodules so we can at least establish\nagreement on what should be done. These are in order of decreasing\nimportance (to me).\n\n\n* Upstreamless submodules\n\nIf there is no 'url' key defined for a submodule in .gitconfig, there is\nno \"authoritative upstream\" for it. When a recursive\nfetch/pull/clone/push is performed on a remote superproject, its\nupstreamless submodules are also fetched/pulled/cloned/pushed directly\nfrom/to the submodule repositories under the superproject .git/modules.\nIf this is the first time that remote's submodules are accessed, that\nremote is initialized for the local submodules: the submodule of the\nremote superproject becomes a remote of the local submodule, and is\ngiven the same name as the remote of the superproject.\n\nSo, suppose we have a superproject with .gitmodules:\n\n[submodule \"sub\"]\n\tpath = sub\n\nwhich is hosted at repositories at URL1 and URL2. Then we do:\n\ngit clone --recursive URL1 super\ncd super\ngit remote add other URL2\ngit fetch --recursive URL2\n\nNow .git/modules/sub/config has:\n\n[remote \"origin\"]\n\turl = URL1/.git/modules/sub\n[remote \"other\"]\n\turl = URL2/.git/modules/sub\n\nSo the effect is similar to just setting the submodule's url as\n\".git/modules/sub\", except that:\n\n - it hides the implementation detail of the exact location of the\n submodule repository from the publicly visible configuration file\n\n - it also works with bare remotes (where the actual remote submodule\n location would be URL/modules/sub)\n\n - it allows multiple simultaneous superproject remotes (where\n git-submodule currently always resolves relative urls against\n branch.$branch.remote with no option to fetch from a different\n remote).\n\n\n* Submodule discovery across all refs\n\nThis is what Jens already mentioned. If we fetch multiple refs of a\nremote superproject, we also need to fetch _all_ of the submodules\nreferenced by _any_ of the refs, not just the ones in the currently\nactive branch. Finding the complete list of submodules probably has to\nbe implemented by reading .gitmodules in all of the (updated) refs,\nwhich is a bit ugly, but not too bad.\n\n\n* Recording the active branch of a submodule\n\nWhen a submodule is added, its active branch should be stored in\n.gitmodules as submodule.$sub.branch. Then, when the submodule is\nchecked out, and the head of that branch is the same as the commit in\nthe gitlink (i.e. the superproject tree is \"current\"), then that branch\nis set as the active branch in the checked-out submodule working tree.\nOtherwise, a detached head is used.\n\n\n* Multiple working trees for a submodule\n\nA superproject may have multiple paths for the same submodule,\npresumably for different commits. This is for cases where the\nsuperproject is a snapshot of a developer's directory hierarchy, and the\ndeveloper is simultaneously working on multiple branches of a submodule\nand it is convenient to have separate working trees for each of them.\n\nThis is a bit hard to express with the current .gitconfig format, since\npaths are attributes of repository ids instead of vice versa. I'd\nintroduce an alternative section format where you can say:\n\n[mount \"path1\"]\n module = sub\n branch = master\n\n[mount \"path2\"]\n module = sub\n branch = topic\n\nImplementing this is a bit intricate, since we need to use the\ngit-new-workdir method to create multiple working directories that share\nthe same refs, config, and object store, but have separate HEAD and\nindex. I think this is a problem with the repository layout: the\nnon-workdir-specific bits should all be in a single directory so that a\nsingle symlink would be enough.\n\n\nObviously, I'm willing to implement the above functionalities since I\nneed them. However, I think I'm going to work in Dulwich (which doesn't\ncurrently have any submodule support): a Python API is currently more\nimportant to me than a command-line tool, and the git.git codebase\ndoesn't look like a very attractive place to contribute anyway. No\noffense, it's just not to my tastes.\n\nSo the main reason I'd like to reach some tentative agreement about the\ndetails of the proposal is to ensure that _once_ someone finally\nimplements this kind of functionality in git.git, it will use the same\nconfiguration format and same conventions, so that it will be compatible\nwith my code. The compatibility between different tools is after all the\nmain reason for doing this stuff as an extension to submodules instead\nof something completely different.\n\n\nLauri\n"},{"id":"201564","messageId":"5081B37F.3010204@web.de","threadId":"31801","inReplyTo":"20121019033158.93403x1gcanyjo8u.lealanko@webmail.helsinki.fi","subject":"Re: A design for distributed submodules","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2012-10-19T20:09:35Z","receivedAt":"2012-10-19T20:09:35Z","isPatch":false,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 19.10.2012 02:31, schrieb Lauri Alanko:\n> I think I finally agree that it's best to develop submodules further\n> rather than introduce a new tool for the functionality I require. Here\n> are some explicit proposals for submodules so we can at least establish\n> agreement on what should be done. These are in order of decreasing\n> importance (to me).\n\nGood to hear that!\n\n> * Upstreamless submodules\n> \n> If there is no 'url' key defined for a submodule in .gitconfig, there is\n> no \"authoritative upstream\" for it. When a recursive\n> fetch/pull/clone/push is performed on a remote superproject, its\n> upstreamless submodules are also fetched/pulled/cloned/pushed directly\n> from/to the submodule repositories under the superproject .git/modules.\n> If this is the first time that remote's submodules are accessed, that\n> remote is initialized for the local submodules: the submodule of the\n> remote superproject becomes a remote of the local submodule, and is\n> given the same name as the remote of the superproject.\n> \n> So, suppose we have a superproject with .gitmodules:\n> \n> [submodule \"sub\"]\n> path = sub\n> \n> which is hosted at repositories at URL1 and URL2. Then we do:\n> \n> git clone --recursive URL1 super\n> cd super\n> git remote add other URL2\n> git fetch --recursive URL2\n> \n> Now .git/modules/sub/config has:\n> \n> [remote \"origin\"]\n> url = URL1/.git/modules/sub\n> [remote \"other\"]\n> url = URL2/.git/modules/sub\n\nSo you want to automatically propagate the new superproject remote\n\"other\" into the submodules?\n\n> So the effect is similar to just setting the submodule's url as\n> \".git/modules/sub\", except that:\n> \n> - it hides the implementation detail of the exact location of the\n> submodule repository from the publicly visible configuration file\n> \n> - it also works with bare remotes (where the actual remote submodule\n> location would be URL/modules/sub)\n> \n> - it allows multiple simultaneous superproject remotes (where\n> git-submodule currently always resolves relative urls against\n> branch.$branch.remote with no option to fetch from a different\n> remote).\n\nMaybe it's too late on a Friday evening in my timezone, but currently\nI can't wrap my mind around what you have in mind here ... will try\nagain later.\n\n> * Submodule discovery across all refs\n> \n> This is what Jens already mentioned. If we fetch multiple refs of a\n> remote superproject, we also need to fetch _all_ of the submodules\n> referenced by _any_ of the refs, not just the ones in the currently\n> active branch.\n\nThat is how things already work now (and it is done in an optimized\nway because we only do a fetch in a submodule when the referenced\ncommit isn't already present locally). But the current limitation\nis that only populated submodules are updated (we do a \"git fetch\"\ninside the submodules work tree), so e.g. currently we can't follow\nrenames. We should also do a fetch for submodules which aren't\nchecked out but whose repo is found in .git/modules/.\n\n> Finding the complete list of submodules probably has to\n> be implemented by reading .gitmodules in all of the (updated) refs,\n> which is a bit ugly, but not too bad.\n\nYes, this will be necessary to get the correct path -> name mapping\nfor submodules which aren't found in the work tree (e.g. because\nthey are renamed). (I will also need to peek into another commit's\n.gitmodules file to make the recursive checkout work for appearing\nsubmodules for the same reason)\n\n> * Recording the active branch of a submodule\n> \n> When a submodule is added, its active branch should be stored in\n> .gitmodules as submodule.$sub.branch. Then, when the submodule is\n> checked out, and the head of that branch is the same as the commit in\n> the gitlink (i.e. the superproject tree is \"current\"), then that branch\n> is set as the active branch in the checked-out submodule working tree.\n> Otherwise, a detached head is used.\n\nWe had some discussions about a \"floating\" submodule model where the\nsubmodules follow the tip of a branch configured in .gitmodules. That\nlooked similar to what you have in mind, except that the tip of that\nbranch is always used.\n\n> * Multiple working trees for a submodule\n> \n> A superproject may have multiple paths for the same submodule,\n> presumably for different commits. This is for cases where the\n> superproject is a snapshot of a developer's directory hierarchy, and the\n> developer is simultaneously working on multiple branches of a submodule\n> and it is convenient to have separate working trees for each of them.\n> \n> This is a bit hard to express with the current .gitconfig format, since\n> paths are attributes of repository ids instead of vice versa. I'd\n> introduce an alternative section format where you can say:\n> \n> [mount \"path1\"]\n> module = sub\n> branch = master\n> \n> [mount \"path2\"]\n> module = sub\n> branch = topic\n> \n> Implementing this is a bit intricate, since we need to use the\n> git-new-workdir method to create multiple working directories that share\n> the same refs, config, and object store, but have separate HEAD and\n> index. I think this is a problem with the repository layout: the\n> non-workdir-specific bits should all be in a single directory so that a\n> single symlink would be enough.\n\nI'm not sure how good that'll work. E.g. what happens if the user\nconfigures the URL of \"path1\" to something else? It looks to me like\nhaving the same repo copied under different .git/modules/ would\nbe a more robust approach, even though it wastes some disk space.\n\n> Obviously, I'm willing to implement the above functionalities since I\n> need them. However, I think I'm going to work in Dulwich (which doesn't\n> currently have any submodule support): a Python API is currently more\n> important to me than a command-line tool, and the git.git codebase\n> doesn't look like a very attractive place to contribute anyway. No\n> offense, it's just not to my tastes.\n> \n> So the main reason I'd like to reach some tentative agreement about the\n> details of the proposal is to ensure that _once_ someone finally\n> implements this kind of functionality in git.git, it will use the same\n> configuration format and same conventions, so that it will be compatible\n> with my code. The compatibility between different tools is after all the\n> main reason for doing this stuff as an extension to submodules instead\n> of something completely different.\n\nFair enough. But I fear unless we code the same functionality in both\nworlds at about the same time the assumption that it will be done in\nthe future in the git core in the same way you expect may fail.\n\nHaving said that: I expect to implement peeking into another commit's\n.gitmodules to read the config next after I finished the rm and mv for\nsubmodules (and intend to use it for doing a fetch first), so maybe we\ncan start with that?\n"}]}