{"thread":{"id":"33285","subject":"Composing git repositories","startedAt":"2013-03-26T07:56:33Z","lastAt":"2013-04-05T07:15:44Z","messageCount":43,"participants":["Ramkumar Ramachandra","Junio C Hamano","Jonathan Nieder","Jens Lehmann","Phil Hord","Seth Robertson","Jeff King","Duy Nguyen"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"212268","messageId":"CALkWK0=CsuAWQwk5Guf0pbC4_ZEoZiwQpamcRvBGz5LJ0QGKHg@mail.gmail.com","threadId":"33285","inReplyTo":null,"subject":"Composing git repositories","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-03-26T07:56:33Z","receivedAt":"2013-03-26T07:56:33Z","isPatch":false,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"(Changed subject)\n(+CC: Peff; I suspect he'll be interested in the repository\ncomposition discussion)\n\nJens Lehmann wrote:\n> Am 25.03.2013 20:57, schrieb Ramkumar Ramachandra:\n>> Okay, I'll do it step-by-step now, with a live example:\n>> [...]\n>> What is going on, seriously?\n>\n> Pilot error, mostly omitting the --recursive option and some\n> - fixable - usability issues. Patches welcome.\n\nAs a user inexperienced with recursive submodules (I've only used them\nin this repository), I found it highly confusing. Thanks for clearing\nthem up.\n\nIt was highly exaggerated and dramatized to make the following points clear:\n\n1. With the current limitation of cd-to-toplevel, it's extremely\nunpleasant to work with many levels of nesting.\n\n2. I thought no operation could be performed without cd-to-toplevel,\nas even 'git submodule foo' complains about this (instead of saying\n\"command not found\"). Having to go to each submodule subdirectory to\nperform operations is just horrible.\n\n3. To change a submodule, I'll have to propagate the change upwards,\ncreating new commits in each of the outer submodules, and the\nsuperproject.\n\nApart from the implementation glitches, I don't like the design;\nsubmodules don't compose well:\n\n1. There's an inherent asymmetry between the superproject and each of\nthe subprojects, because the superproject owns all the object stores.\nWhy is it absolutely necessary to relocate the object stores?\n\n2. The metadata is tracked as a file (.gitmodules) in the git\nrepository. While it makes it possible for different branches to have\ndifferent submodules, it's impossible to say 'git submodule sync\na/b/c/d' (see: 'repo sync'), where b, c haven't been initialized.\n\n3. The current implementation only allows me to compose with commit\nobjects, but what if I want to compose with refs? ie. What if I want\nto track the tip of the 'master' of a submodule in a superproject? I\ndon't want to commit-cascade everytime I make a minor change in a\ndeeply nested submodule.\n\nOther solutions such as repo (see: depot_tools) solve different\nproblems but are huge failures when it comes to composability; repo\nuses a central manifest repository, for example. Is it impossible to\nbuild a tool that can truly compose git repositories? I think\nsubmodules gets a lot of things Right, and we'd have to start from\nthere.\n\n>> This is just two levels of nesting: with more levels of nesting,\n>> things only get worse.\n>\n> Yes, you cannot have the cake and eat it. Either you incorporate\n> everything into a single repo (e.g. using subtree) and loose the\n> strong distinction which content belongs to which upstream repo\n> (which AFAIK is a valid choice unless you want to contribute back\n> to the submodule's upstream) or you'll have to cope with the\n> submodule borders showing up from time to time, reminding you\n> which part of the work tree has another upstream.\n\nI don't think it's impossible. We just haven't thought hard enough\nabout composition.\n"},{"id":"212294","messageId":"7vmwtqt8rs.fsf@alter.siamese.dyndns.org","threadId":"33285","inReplyTo":"CALkWK0=CsuAWQwk5Guf0pbC4_ZEoZiwQpamcRvBGz5LJ0QGKHg@mail.gmail.com","subject":"Re: Composing git repositories","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-03-26T16:39:51Z","receivedAt":"2013-03-26T16:39:51Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ramkumar Ramachandra writes:\n\n> Apart from the implementation glitches, I don't like the design;\n> submodules don't compose well:\n>\n> 1. There's an inherent asymmetry between the superproject and each of\n> the subprojects, because the superproject owns all the object stores.\n> Why is it absolutely necessary to relocate the object stores?\n\nImagine doing \"git checkout oldbranch\" in the superproject when your\ncurrent branch has a submodule A bound to it, but the oldbranch that\nis an old version of the superproject did not (yet) have it.\n\nYou obviously want the directory A disappear, but you would be\nunhappy if you have to lose A/.git and everything in it while doing\nso. If you are lucky, you can re-clone, but you may have your own\nchanges.\n\nSo you have to stash it somewhere. We could have made it to move\nthem to $HOME/.safeplace or somewhere totally unrelated to the\nsuperproject. So in that sense, the repositories are *not* owned by\nthe superproject in any way. However, you are working within the\ncontext of the superproject on the submodule after all, and\nsomewhere under $GIT_DIR/ of the superorject is not too wrong a\nplace to use as such a safe place.\n\n> 3. The current implementation only allows me to compose with commit\n> objects, but what if I want to compose with refs? ie. What if I want\n> to track the tip of the 'master' of a submodule in a superproject?\n\nLook for floating submodules in the list archive.\n"},{"id":"212372","messageId":"CALkWK0kNH2A4eLML22RTofarR3MB++OECiNXMi-bWLLMWK1GAg@mail.gmail.com","threadId":"33285","inReplyTo":"7vmwtqt8rs.fsf@alter.siamese.dyndns.org","subject":"Re: Composing git repositories","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-03-27T11:49:38Z","receivedAt":"2013-03-27T11:49:38Z","isPatch":false,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Junio C Hamano wrote:\n> So you have to stash it somewhere. We could have made it to move\n> them to $HOME/.safeplace or somewhere totally unrelated to the\n> superproject. So in that sense, the repositories are *not* owned by\n> the superproject in any way. However, you are working within the\n> context of the superproject on the submodule after all, and\n> somewhere under $GIT_DIR/ of the superorject is not too wrong a\n> place to use as such a safe place.\n\nThanks for the explanation. The paths in .git/modules are unnecessary\nugly and unwieldy, especially in the case of multiple levels of\nnesting: I'll look into converting it to a flat structure using\n.git, while handling name conflicts. I'll also look\ninto adding a feature to relocate this/ using an object store from an\nexisting clone.\n\n> Look for floating submodules in the list archive.\n\nThe most relevant message thread I could find was [1], back from 2011.\n You argue that floating submodules are Wrong, and that there is no\nreal usecase for it.\n\nLike I explained earlier, I'm looking at one tool that solves a\nsuperset of the problems mr, repo, submodules, subtrees, and other\ntools solve. I really like the way repo allows me to work on a meta\nproject like Android or ChromiumOS, but hate that it allows for zero\ncomposition.\n\nTo move forward, I have the following design thoughts (elaborating on\nmy previous email):\n\n1. If .gitmodules is tracked like a normal file, it is absolutely\nimpossible to tell the possible dependencies of the superproject\nwithout cloning it entirely, and looking at the .gitmodules file in\neach of the branches. Can't we have it as a special ref instead, so I\ncan `git fetch` just that ref to figure out the dependencies?\n\n2. True composition requires that I be able to specify the entire\nmanifest (for nested submodules) in the toplevel .gitmodules, or break\nit up as I see fit. This is currently impossible, and brings us back\nto #1: the manifest for b/, b/c are in the toplevel repo's special\nref, and I need to fetch c's special ref to figure out what d is (or\nerror out if no such thing exists).\n\n3. True floating submodules are impossible, because a change in the\nsubmodule means a change in the commit object referenced by the\nsuperproject's tree object; diff-tree will see that some content has\nchanged in the repository. We can represent that diff however we want\n(using diff.submodule), but we can't change the fact that the change\nhas to be committed. Fixing this will require nothing short of\nintroducing a new kind of object (say \"submodule\" object which can be\na concrete SHA-1 or point to a ref).\n\nDo you think thinking about these things is worthwhile? I see more\ncomplaints like [2] as git adoption in the industry increases, but we\nhave no solution: we can't make git scale to super-large repositories,\nand we have no real way to compose smaller repositories. Hacks like\nrepo sadden me.\n\n[1]: http://thread.gmane.org/gmane.comp.version-control.git/185164\n[2]: http://thread.gmane.org/gmane.comp.version-control.git/189776\n"},{"id":"212391","messageId":"7vvc8comj5.fsf@alter.siamese.dyndns.org","threadId":"33285","inReplyTo":"CALkWK0kNH2A4eLML22RTofarR3MB++OECiNXMi-bWLLMWK1GAg@mail.gmail.com","subject":"Re: Composing git repositories","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-03-27T16:06:06Z","receivedAt":"2013-03-27T16:06:06Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ramkumar Ramachandra writes:\n\n> Junio C Hamano wrote:\n>> So you have to stash it somewhere. We could have made it to move\n>> them to $HOME/.safeplace or somewhere totally unrelated to the\n>> superproject. So in that sense, the repositories are *not* owned by\n>> the superproject in any way. However, you are working within the\n>> context of the superproject on the submodule after all, and\n>> somewhere under $GIT_DIR/ of the superorject is not too wrong a\n>> place to use as such a safe place.\n>\n> Thanks for the explanation.\n\nWhat do you _exactly_ mean by that? You understood why things are\narranged in that way, and no longer think that it is unnecessary,\nugly and unwieldy to stash the real copy of $GIT_DIR of submodules\naway from their working trees and store them inside $GIT_DIR/modules\nof the superproject?\n\n> The paths in .git/modules are unnecessary ugly and unwieldy,\n> especially in the case of multiple levels of nesting...\n"},{"id":"212403","messageId":"CALkWK0nARWAtC-D3UiNLccuaSwjR6meJb+Cu590N=8Ti8O7OMg@mail.gmail.com","threadId":"33285","inReplyTo":"7vvc8comj5.fsf@alter.siamese.dyndns.org","subject":"Re: Composing git repositories","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-03-27T17:02:27Z","receivedAt":"2013-03-27T17:02:27Z","isPatch":false,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Junio C Hamano wrote:\n> Ramkumar Ramachandra writes:\n>> Junio C Hamano wrote:\n>>> So you have to stash it somewhere. We could have made it to move\n>>> them to $HOME/.safeplace or somewhere totally unrelated to the\n>>> superproject. So in that sense, the repositories are *not* owned by\n>>> the superproject in any way. However, you are working within the\n>>> context of the superproject on the submodule after all, and\n>>> somewhere under $GIT_DIR/ of the superorject is not too wrong a\n>>> place to use as such a safe place.\n>>\n>> Thanks for the explanation.\n>\n> What do you _exactly_ mean by that? You understood why things are\n> arranged in that way, and no longer think that it is unnecessary,\n> ugly and unwieldy to stash the real copy of $GIT_DIR of submodules\n> away from their working trees and store them inside $GIT_DIR/modules\n> of the superproject?\n\nIn essence, git commands are built to act on pure worktrees. It's\ntrivially Correct to pretend that an object store present in the\ntoplevel directory (as .git/) of the worktree doesn't exist, but it's\nquite non-trivial to handle a .git directory anywhere else in the\nworktree. Since we built git ground-up to act on a single\nrepository's worktree, embedding one repository inside another is a\nhack: as a \"workaround\", we simply relocate the object store of the\nsubmodule repository. Even then, working with one worktree embedded\ninside another is something git never designed for: it explains why I\nhave to literally fight with git when using submodules (no offense\nJens; it's a very hard problem).\n\nRepresenting submodules as commit objects in the tree is also a hack.\nI'm sorry, but a submodule is not a commit object. We need a fifth\nobject type if we want them to be first-class citizens.\n\nSorry, I'm deviating. I learnt why you think the hack is necessary\nand not \"too wrong\". As I explained above, the entire design is\nasymmetric and inelegant; I think we can do much better than this.\n"},{"id":"212409","messageId":"7vobe4n4oq.fsf@alter.siamese.dyndns.org","threadId":"33285","inReplyTo":"CALkWK0nARWAtC-D3UiNLccuaSwjR6meJb+Cu590N=8Ti8O7OMg@mail.gmail.com","subject":"Re: Composing git repositories","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-03-27T17:16:53Z","receivedAt":"2013-03-27T17:16:53Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ramkumar Ramachandra writes:\n\n> Sorry, I'm deviating. I learnt why you think the hack is necessary\n> and not \"too wrong\".\n\nOK.\n\n> As I explained above, the entire design is\n> asymmetric and inelegant; I think we can do much better than this.\n\nI personally find the \"explained above\" part making not much sense.\nMaybe others can comment on it.\n"},{"id":"212435","messageId":"20130327192630.GF28148@google.com","threadId":"33285","inReplyTo":"CALkWK0nARWAtC-D3UiNLccuaSwjR6meJb+Cu590N=8Ti8O7OMg@mail.gmail.com","subject":"Re: Composing git repositories","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2013-03-27T19:26:30Z","receivedAt":"2013-03-27T19:26:30Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Ramkumar Ramachandra wrote:\n\n> Even then, working with one worktree embedded\n> inside another is something git never designed for: it explains why I\n> have to literally fight with git when using submodules\n\nDo you mean that you wish you could ignore subrepository boundaries\nand use commands like\n\n\tgit clone --recurse-submodules http://git.zx2c4.com/cgit\n\tcd cgit\n\tvi git/cache.h\n\t... edit edit edit ...\n\tgit add --recurse-submodules git/cache.h\n\tgit commit --recurse-submodules\n\tgit push --recurse-submodules\n\n, possibly with configuration to allow the --recurse-submodules to be\nimplied, and have everything work out well?\n\nI think something like that is a goal for submodules in the long term,\nwith a caveat that there are complications in that different projects\n(the parent project and subproject) can have different contribution\nguidelines, review and release schedules, and so on.\n\nIf submodules are not working for you today, you may find some of\nJens's submodule improvement patches interesting, or you may want to\nlook into alternatives that make different assumptions, such as\nentirely independent repositories and tools like \"mr\" that iterate\nover them.\n\nHope that helps,\nJonathan\n"},{"id":"212449","messageId":"7vppyklhot.fsf@alter.siamese.dyndns.org","threadId":"33285","inReplyTo":"20130327192630.GF28148@google.com","subject":"Re: Composing git repositories","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-03-27T20:18:58Z","receivedAt":"2013-03-27T20:18:58Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Nieder writes:\n\n> Ramkumar Ramachandra wrote:\n>\n>> Even then, working with one worktree embedded\n>> inside another is something git never designed for: it explains why I\n>> have to literally fight with git when using submodules\n>\n> Do you mean that you wish you could ignore subrepository boundaries\n> and use commands like\n>\n> \tgit clone --recurse-submodules http://git.zx2c4.com/cgit\n> \tcd cgit\n> \tvi git/cache.h\n> \t... edit edit edit ...\n> \tgit add --recurse-submodules git/cache.h\n> \tgit commit --recurse-submodules\n> \tgit push --recurse-submodules\n>\n> , possibly with configuration to allow the --recurse-submodules to be\n> implied, and have everything work out well?\n> ...\n> I think something like that is a goal for submodules in the long\n> term,...\n\nAs you hinted with \"complications\" below, I have to wonder what\nshould happen when the above \"git add\" touches anything outside\n\"git\" subdirectory.\n\nBut such an administrative details (the project boundary is\nprimarily not an implementation detail but is a social issue) aside,\nI agree that overall it would be a good user experience.\n\nI however do not see the implementation detail of having (or not\nhaving) separate $GIT_DIR for component projects having anything to\ndo with the goal of that ideal.\n\nWhere and how do you envision the metainformation about the\ncomponent projects are stored in such a clone? It does not have to\nbe cgit/.git, but you would need to somehow store things we store in\n$GIT_DIR for cgit itself and git in the current system. If you pick\none location to store both, I would imagine that it would still be\nsomewhere under the cgit directory.\n\nAs I said in another thread, your top-level may be only a part in\nsomebody else's project, and what you consider just a part of your\nproject may be the whole project to somebody else. If you pick one\nlocation to store both for the above clone, e.g. cgit/.git (it could\nbe cgit/.ram-git or any other name), embedding it in a yet larger\nproject (perhaps having both cgit and gitolite to give a one-stop\nsolution for hosting services) later would face the same issue as\nRam seemed to be complaining. It needs to address what happens when\nthat cgit/.git (or whatever name) gets in the way in the scope of\nthe larger project. That is why I said Ram's rant, using subjective\nwords like \"elegant\", without sound technical justification, did not\nmake much sense to me.\n"},{"id":"212454","messageId":"20130327204213.GG28148@google.com","threadId":"33285","inReplyTo":"7vppyklhot.fsf@alter.siamese.dyndns.org","subject":"Re: Composing git repositories","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2013-03-27T20:42:13Z","receivedAt":"2013-03-27T20:42:13Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Junio C Hamano wrote:\n\n> I however do not see the implementation detail of having (or not\n> having) separate $GIT_DIR for component projects having anything to\n> do with the goal of that ideal.\n\nYeah, I think the current gitlink-instead-of-full-git-dir-for-submodules\nimplementation that can allow \"git rm\" to remove a submodule and\n\"git checkout --recurse-submodules\" to later revive it is good.\n"},{"id":"212477","messageId":"51537A7B.7050206@web.de","threadId":"33285","inReplyTo":"CALkWK0nARWAtC-D3UiNLccuaSwjR6meJb+Cu590N=8Ti8O7OMg@mail.gmail.com","subject":"Re: Composing git repositories","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2013-03-27T23:02:19Z","receivedAt":"2013-03-27T23:02:19Z","isPatch":false,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 27.03.2013 18:02, schrieb Ramkumar Ramachandra:\n> Junio C Hamano wrote:\n>> Ramkumar Ramachandra writes:\n>>> Junio C Hamano wrote:\n>>>> So you have to stash it somewhere. We could have made it to move\n>>>> them to $HOME/.safeplace or somewhere totally unrelated to the\n>>>> superproject. So in that sense, the repositories are *not* owned by\n>>>> the superproject in any way. However, you are working within the\n>>>> context of the superproject on the submodule after all, and\n>>>> somewhere under $GIT_DIR/ of the superorject is not too wrong a\n>>>> place to use as such a safe place.\n>>>\n>>> Thanks for the explanation.\n>>\n>> What do you _exactly_ mean by that? You understood why things are\n>> arranged in that way, and no longer think that it is unnecessary,\n>> ugly and unwieldy to stash the real copy of $GIT_DIR of submodules\n>> away from their working trees and store them inside $GIT_DIR/modules\n>> of the superproject?\n> \n> In essence, git commands are built to act on pure worktrees. It's\n> trivially Correct to pretend that an object store present in the\n> toplevel directory (as .git/) of the worktree doesn't exist, but it's\n> quite non-trivial to handle a .git directory anywhere else in the\n> worktree. Since we built git ground-up to act on a single\n> repository's worktree, embedding one repository inside another is a\n> hack: as a \"workaround\", we simply relocate the object store of the\n> submodule repository.\n\nSubmodules work pretty well, no matter if you call them a \"hack\".\nAnd what you call a \"workaround\" allows us to move, remove and\nrecreate submodules, which is one of *the* major inconveniences\nsubmodules currently have.\n\n> Even then, working with one worktree embedded\n> inside another is something git never designed for: it explains why I\n> have to literally fight with git when using submodules (no offense\n> Jens; it's a very hard problem).\n\nUnless you acknowledge that submodules are a different repo, you'll\nalways run into problems. I believe future enhancements will make\nthis less tedious, but in the end they will stay separate repos\n(which is the whole point, you'd want to use a different approach\n- e.g. subtree - if you want to put stuff from different upstreams\ninto a single repo without keeping the distinction where all that\ncame from).\n\n> Representing submodules as commit objects in the tree is also a hack.\n> I'm sorry, but a submodule is not a commit object. We need a fifth\n> object type if we want them to be first-class citizens.\n\nWhat else than a commit object should that be??? Submodules are\nthere to have a different upstream for a part of your work tree,\nand that means a commit in that repo is the only sane thing to\nrecord in the superproject. A lot of thought has been put into\nthis, and it is definitely a good choice [1].\n\n> Sorry, I'm deviating. I learnt why you think the hack is necessary\n> and not \"too wrong\". As I explained above, the entire design is\n> asymmetric and inelegant; I think we can do much better than this.\n\nHow? The \"submodules suck, we should try a completely different\napproach\" thingy comes up from time to time, but so far nobody\ncould provide a viable alternative to what we currently do.\n\nAnd apart from that, let's not forget we identified some valuable\nimprovements to submodules in this thread:\n\n*) Get rid of the \"toplevel\" requirement\n\n*) Add functionality to relocate the object store out of the work\n tree (either \"git submodule to-gitfile\" or something similar,\n maybe even as a separate script in contrib)\n\n*) Add an option to \"git submodule add\" (and/or maybe a config\n option) to relocate the object store immediately on adding an\n already present submodule\n\nAll of those are topics I like to see materialize, and you are\nwelcome to tackle them.\n\n\n[1]: http://article.gmane.org/gmane.comp.version-control.git/151857/\n"},{"id":"212497","messageId":"CALkWK0nfNCu775MBB-Y28=V93RkV24kbTLTDKWO2dZ-0yxX=Sw@mail.gmail.com","threadId":"33285","inReplyTo":"51537A7B.7050206@web.de","subject":"Re: Composing git repositories","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-03-28T09:16:43Z","receivedAt":"2013-03-28T09:16:43Z","isPatch":false,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Jens Lehmann wrote:\n> Unless you acknowledge that submodules are a different repo, you'll\n> always run into problems. I believe future enhancements will make\n> this less tedious, but in the end they will stay separate repos\n> (which is the whole point, you'd want to use a different approach\n> - e.g. subtree - if you want to put stuff from different upstreams\n> into a single repo without keeping the distinction where all that\n> came from).\n\nI acknowledge that it's a different repository. It's just that I\nthink that our current design has too many seams: why do you think\nit's impossible to make it seamless?\n\ngit-subtree is not an answer to anything. Dumping all the history\ninto one repository has its limited usecases, but it is no solution.\n\n> What else than a commit object should that be??? Submodules are\n> there to have a different upstream for a part of your work tree,\n> and that means a commit in that repo is the only sane thing to\n> record in the superproject. A lot of thought has been put into\n> this, and it is definitely a good choice [1].\n\nLinus argues that it shouldn't be a tree object, and I agree with\nthat. I don't see an argument that says that the commit object is a\nperfect fit (probably because it's not).\n\n> How? The \"submodules suck, we should try a completely different\n> approach\" thingy comes up from time to time, but so far nobody\n> could provide a viable alternative to what we currently do.\n\nMy argument is not \"submodules suck; we should throw them out of the\nwindow, and start from scratch\" at all. I'm merely questioning the\nfundamental assumptions that submodules make, instead of proposing\nthat we work around everything in shell. We don't have to be married\nto the existing implementation of submodules and try to fix all the\nproblems in shell.\n\n> And apart from that, let's not forget we identified some valuable\n> improvements to submodules in this thread:\n> [...]\n> All of those are topics I like to see materialize, and you are\n> welcome to tackle them.\n\nAllow me a few days to think about changing the fundamental building\nblocks to make our shell hackery easier.\n\nThanks.\n"},{"id":"212499","messageId":"CALkWK0nreJZX4msFET0a7cuUMWNbQhhqy+ezrkqYGqL4_a2duA@mail.gmail.com","threadId":"33285","inReplyTo":"20130327192630.GF28148@google.com","subject":"Re: Composing git repositories","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-03-28T10:01:46Z","receivedAt":"2013-03-28T10:01:46Z","isPatch":false,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Jonathan Nieder wrote:\n> Do you mean that you wish you could ignore subrepository boundaries\n> and use commands like\n>\n> git clone --recurse-submodules http://git.zx2c4.com/cgit\n> cd cgit\n> vi git/cache.h\n> ... edit edit edit ...\n> git add --recurse-submodules git/cache.h\n> git commit --recurse-submodules\n> git push --recurse-submodules\n>\n> , possibly with configuration to allow the --recurse-submodules to be\n> implied, and have everything work out well?\n\nDo you realize how difficult this is to implement? We'll need to\npatch all the git commands to essentially do what we'd get for free if\nthe submodule were a tree object instead of a commit object (although\nI'm not saying that's the Right thing to do). Some caveats:\n\n- If we maintain one global index, and try to emulate git-subtree\nusing submodules, we've lost. It's going to take freakin' ages to\nstat billions of files across hundreds of nested sumodules. One major\nadvantage of having repository boundaries is separate object stores,\nindexes, worktrees: little ones that git is designed to work with.\n\n- Auto-splitting commits that touch multiple submodules/ superproject\nat once. Although git-subtree does this, I think it's horribly ugly.\n\n- Auto-propagating commits upwards to the superproject is another big\nchallenge. I think the current design of anchoring to a specific\ncommit SHA-1 has its usecases, but is unwieldy when things become big.\n We have to fix that first.\n"},{"id":"212501","messageId":"CALkWK0=GcxBh9o+sF1Q8t6SC0JU=NmPyRg6tqaOKmkJ6qDvRCA@mail.gmail.com","threadId":"33285","inReplyTo":"7vppyklhot.fsf@alter.siamese.dyndns.org","subject":"Re: Composing git repositories","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-03-28T11:48:49Z","receivedAt":"2013-03-28T11:48:49Z","isPatch":false,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Junio C Hamano wrote:\n> As I said in another thread, your top-level may be only a part in\n> somebody else's project, and what you consider just a part of your\n> project may be the whole project to somebody else. If you pick one\n> location to store both for the above clone, e.g. cgit/.git (it could\n> be cgit/.ram-git or any other name), embedding it in a yet larger\n> project (perhaps having both cgit and gitolite to give a one-stop\n> solution for hosting services) later would face the same issue as\n> Ram seemed to be complaining. It needs to address what happens when\n> that cgit/.git (or whatever name) gets in the way in the scope of\n> the larger project. That is why I said Ram's rant, using subjective\n> words like \"elegant\", without sound technical justification, did not\n> make much sense to me.\n\nI was having a lot of difficulty writing down my thoughts. Thank you\nfor providing an illustrative example. It is terribly hard to do with\nour current implementation: we'd have to rewrite the \"gitdir: \" lines\nin all the .git files in the submodule trees and rebuild all the\n.git/modules paths. I'm thinking that we need to separate the object\nstores from the worktrees for good. For a project with no submodules,\nthe object store can be present in .git/ of the toplevel directory,\nlike it is now. The moment submodules are added, all the object\nstores should be relocated to a place outside the worktree. So my\n~/src might look like: dotfiles.git/, auto-complete.git/, magit.git/,\ngit-commit-mode.git/, yasnippet.git/ and dotfiles/. dotfiles/\ncontains lots of worktrees stitched together nicely, pointing to these\nobject stores in ~/src. This would certainly get rid of the asymmetry\nfor good.\n\nNow, we can focus our attention on composing git worktrees. What is a\nworktree? A tree object pointed to by the commit object referred to\nby HEAD. What we need to do is embed one tree inside another using a\nmediating object to establish repository boundaries, while not\nintroducing an ugly seam. If you think about it, the mediator we've\npicked conveys little/ no information to the parent; it says: \"there's\na commit with this SHA-1 present in this submodule, but I can't tell\nyou the commit message, tree object, branch, remote, or anything else\"\n(obviously because the commit isn't present in the parent's object\nstore). So, the mediator might as well have been a SHA-1 string. And\nwe have an ugly .gitmodules conveying the remote and the branch. Why\ncan't we stuff more information into the mediating object and get rid\nof .gitmodules altogether?\n\nOkay, here's a first draft of the new design. The new mediator object\nshould look like:\n\n name = git\n ref = v1.7.8\n\nThe name is looked up in refs/modules/, which in turn looks like:\n\n [submodule \"git\"]\n origin = gh:artagnon/git\n path = git\n [submodule \"magit\"]\n origin = gh:magit/magit\n path = git/extensions/magit\n\nThe ref could be 'master', 'HEAD~1', or even a commit SHA-1 (to do the\ncurrent anchored-submodules).\nFinally, there's a .git file in the worktree, which contains a\n\"gitdir: \" line pointing to the object store, as before.\n\nThis solves the two problems that I brought up earlier:\n- Floating submodules (which are _necessary_ if you don't want to\npropagate commits upwards to the root).\n- Initializing a nested submodule without having to initialize all the\nsubmodules in the path leading up to it.\n\nHowever, I suspect that we can put more information the mediator\nobject to make life easier for the parent repository and make seams\ndisappear. I'm currently thinking about what information git core\nneeds to behave smoothly with submodules.\n"},{"id":"212558","messageId":"20130328182140.GO28148@google.com","threadId":"33285","inReplyTo":"CALkWK0nreJZX4msFET0a7cuUMWNbQhhqy+ezrkqYGqL4_a2duA@mail.gmail.com","subject":"Re: Composing git repositories","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2013-03-28T18:21:40Z","receivedAt":"2013-03-28T18:21:40Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Ramkumar Ramachandra wrote:\n\n> Do you realize how difficult this is to implement? We'll need to\n> patch all the git commands to essentially do what we'd get for free if\n> the submodule were a tree object instead of a commit object (although\n> I'm not saying that's the Right thing to do).\n\nWhat are you talking about? Yes, of course I realize that recursing\nover subprojects that are managed as separate git repositories\nrequires writing new code. That's why people have started to write\nsuch code. They seem to think it's worth it.\n\nMeanwhile others with different designs in mind have written other\ntools. Use cases even overlap a little, so they can compete. That is\nexactly as it should be.\n"},{"id":"212575","messageId":"5154A577.2020103@web.de","threadId":"33285","inReplyTo":"CALkWK0nreJZX4msFET0a7cuUMWNbQhhqy+ezrkqYGqL4_a2duA@mail.gmail.com","subject":"Re: Composing git repositories","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2013-03-28T20:17:59Z","receivedAt":"2013-03-28T20:17:59Z","isPatch":false,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 28.03.2013 11:01, schrieb Ramkumar Ramachandra:\n> Jonathan Nieder wrote:\n>> Do you mean that you wish you could ignore subrepository boundaries\n>> and use commands like\n>>\n>> git clone --recurse-submodules http://git.zx2c4.com/cgit\n>> cd cgit\n>> vi git/cache.h\n>> ... edit edit edit ...\n>> git add --recurse-submodules git/cache.h\n>> git commit --recurse-submodules\n>> git push --recurse-submodules\n>>\n>> , possibly with configuration to allow the --recurse-submodules to be\n>> implied, and have everything work out well?\n> \n> Do you realize how difficult this is to implement? We'll need to\n> patch all the git commands to essentially do what we'd get for free if\n> the submodule were a tree object instead of a commit object (although\n> I'm not saying that's the Right thing to do). Some caveats:\n> \n> - If we maintain one global index, and try to emulate git-subtree\n> using submodules, we've lost. It's going to take freakin' ages to\n> stat billions of files across hundreds of nested sumodules. One major\n> advantage of having repository boundaries is separate object stores,\n> indexes, worktrees: little ones that git is designed to work with.\n\nAre you aware that current Git code already stats all files across\nall submodules recursive by default? So (again) no problem here, we\ndo that already (unless configured otherwise).\n\n> - Auto-splitting commits that touch multiple submodules/ superproject\n> at once. Although git-subtree does this, I think it's horribly ugly.\n\nYou don't like it, but what technical argument is hidden here I'm\nmissing?\n\n> - Auto-propagating commits upwards to the superproject is another big\n> challenge. I think the current design of anchoring to a specific\n> commit SHA-1 has its usecases, but is unwieldy when things become big.\n> We have to fix that first.\n\nWhat??? Again there is nothing to \"fix\" here, \"anchoring to a specific\ncommit SHA-1\" is *the* most prominent use case (think reproducibility\nof the whole work tree), floating submodules are the oddball here.\n"},{"id":"212577","messageId":"5154A725.7030609@web.de","threadId":"33285","inReplyTo":"CALkWK0=GcxBh9o+sF1Q8t6SC0JU=NmPyRg6tqaOKmkJ6qDvRCA@mail.gmail.com","subject":"Re: Composing git repositories","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2013-03-28T20:25:09Z","receivedAt":"2013-03-28T20:25:09Z","isPatch":false,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 28.03.2013 12:48, schrieb Ramkumar Ramachandra:\n> Okay, here's a first draft of the new design. The new mediator object\n> should look like:\n> \n> name = git\n> ref = v1.7.8\n> \n> The name is looked up in refs/modules/, which in turn looks like:\n> \n> [submodule \"git\"]\n> origin = gh:artagnon/git\n> path = git\n> [submodule \"magit\"]\n> origin = gh:magit/magit\n> path = git/extensions/magit\n\nWhat happens when you rename \"magit\" to \"foo\" in that branch and want\nto check out an older commit of the same branch? That is one of the\nreasons why that belongs in to a checked in .gitmodules and not\nsomeplace untracked.\n\n> This solves the two problems that I brought up earlier:\n> - Floating submodules (which are _necessary_ if you don't want to\n> propagate commits upwards to the root).\n\nIf you don't want that either don't use submodules or set the ignore\nconfig so you won't be bothered with any changes to the submodules.\nFloating up to the submodule's tip can be easily achieved with a\nscript (possibly checked in in the superproject). You loose the\nreproducibility by doing that, but that's what you asked for. No\nproblem here.\n\n> - Initializing a nested submodule without having to initialize all the\n> submodules in the path leading up to it.\n\nYou cannot access a nested sub-submodule without its parent telling\nyou what submodules it has. Otherwise the first level submodule would\nnot be self-contained, so you'll need to check it out too to access\nthe sub-submodules. Nothing to fix here either.\n\n> However, I suspect that we can put more information the mediator\n> object to make life easier for the parent repository and make seams\n> disappear. I'm currently thinking about what information git core\n> needs to behave smoothly with submodules.\n\nTo me your proposal is trying to fix non-issues and breaking stuff\nthat works, so I see no improvement.\n"},{"id":"212580","messageId":"5154AACC.7050006@web.de","threadId":"33285","inReplyTo":"CALkWK0nfNCu775MBB-Y28=V93RkV24kbTLTDKWO2dZ-0yxX=Sw@mail.gmail.com","subject":"Re: Composing git repositories","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2013-03-28T20:40:44Z","receivedAt":"2013-03-28T20:40:44Z","isPatch":false,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 28.03.2013 10:16, schrieb Ramkumar Ramachandra:\n> Jens Lehmann wrote:\n>> Unless you acknowledge that submodules are a different repo, you'll\n>> always run into problems. I believe future enhancements will make\n>> this less tedious, but in the end they will stay separate repos\n>> (which is the whole point, you'd want to use a different approach\n>> - e.g. subtree - if you want to put stuff from different upstreams\n>> into a single repo without keeping the distinction where all that\n>> came from).\n> \n> I acknowledge that it's a different repository. It's just that I\n> think that our current design has too many seams: why do you think\n> it's impossible to make it seamless?\n> \n> git-subtree is not an answer to anything. Dumping all the history\n> into one repository has its limited usecases, but it is no solution.\n\nGuess what: submodules are the solution for a certain set of use\ncases, and tools like subtree are a solution for another set of\nuse cases. There is no silver bullet.\n\n>> What else than a commit object should that be??? Submodules are\n>> there to have a different upstream for a part of your work tree,\n>> and that means a commit in that repo is the only sane thing to\n>> record in the superproject. A lot of thought has been put into\n>> this, and it is definitely a good choice [1].\n> \n> Linus argues that it shouldn't be a tree object, and I agree with\n> that. I don't see an argument that says that the commit object is a\n> perfect fit (probably because it's not).\n\nThere was discussion about what to record in the index/commit of\nthe superproject in early submodule days (some time before I became\ninvolved in Git, seems I currently cannot find a link to that). A\ncommit is the thing to record here because it *is* the perfect fit,\nas some years of submodule experience show.\n\n>> How? The \"submodules suck, we should try a completely different\n>> approach\" thingy comes up from time to time, but so far nobody\n>> could provide a viable alternative to what we currently do.\n> \n> My argument is not \"submodules suck; we should throw them out of the\n> window, and start from scratch\" at all. I'm merely questioning the\n> fundamental assumptions that submodules make, instead of proposing\n> that we work around everything in shell. We don't have to be married\n> to the existing implementation of submodules and try to fix all the\n> problems in shell.\n\nYou cannot simply change the fundamental assumptions of submodules\nand expect them to be the same thing afterwards. And it doesn't\nmatter at all if we \"fix all the problems in shell\" or in C-code,\nwe'll fix the remaining problems that are fixable in whatever part\nof Git it makes sense. And I don't have the impression you have an\nidea about what submodules are good at, where they can be improved\nand what problems they'll probably never solve.\n\n>> And apart from that, let's not forget we identified some valuable\n>> improvements to submodules in this thread:\n>> [...]\n>> All of those are topics I like to see materialize, and you are\n>> welcome to tackle them.\n> \n> Allow me a few days to think about changing the fundamental building\n> blocks to make our shell hackery easier.\n\nPlease go ahead, but if your goal is \"to make our shell hackery\neasier\" I'm not interested. I want to improve the user experience\nof submodules and don't care much in what language we achieve that.\nAnd I can't see anything fundamental being wrong with submodules but\nstrongly believe they are a perfect match for some very important\nuse cases (some of which I see happening at my $dayjob for some\nyears now), so I still don't see what you are trying to \"fix\" here.\n"},{"id":"212752","messageId":"CALkWK0k=g3iFjmpUQA1VkuH2kZsVX1_Hpo=LZ7CuotwHz_1++g@mail.gmail.com","threadId":"33285","inReplyTo":"5154AACC.7050006@web.de","subject":"Re: Composing git repositories","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-03-31T20:34:17Z","receivedAt":"2013-03-31T20:34:17Z","isPatch":false,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Thanks for taking the time and effort to review my thoughts.\n\nJens Lehmann wrote:\n> A\n> commit is the thing to record here because it *is* the perfect fit\n\nMight be, but saying that doesn't help one bit. I want to know why.\n\n> I want to improve the user experience\n> of submodules and don't care much in what language we achieve that.\n\nYou missed the point entirely. If git didn't have a commit object,\nwould you use a special kind of blob and code around everything to\navoid fixing a more fundamental issue?\n\n> What happens when you rename \"magit\" to \"foo\" in that branch and want\n> to check out an older commit of the same branch? That is one of the\n> reasons why that belongs in to a checked in .gitmodules and not\n> someplace untracked.\n\nGood point. I learnt something new.\n\n> Are you aware that current Git code already stats all files across\n> all submodules recursive by default? So (again) no problem here, we\n> do that already (unless configured otherwise).\n\nI didn't know that. Why does it do this?\n\n> Guess what: submodules are the solution for a certain set of use\n> cases, and tools like subtree are a solution for another set of\n> use cases. There is no silver bullet.\n\nThat's the core of your argument: submodules already solve what it\nwas meant to, and we can't get it to solve a larger class of problems.\n In other words, you're implying that it's impossible to build a tool\nthat will be able to compose git repositories in a way that solves a\nvery large class of problems. I don't see conclusive proof for this,\nso I have to disagree.\n\nTo summarize, everyone seems to be elated with the current state of\nsubmodules and is vehemently defending it. I'm a little unhappy, but\nam unable to express my discontent in better prose. Let's just go\nback to writing patches, and come back to this if and when I have a\nfull design.\n"},{"id":"212760","messageId":"20130331225747.GB11704@elie.Belkin","threadId":"33285","inReplyTo":"CALkWK0k=g3iFjmpUQA1VkuH2kZsVX1_Hpo=LZ7CuotwHz_1++g@mail.gmail.com","subject":"Re: Composing git repositories","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2013-03-31T22:57:48Z","receivedAt":"2013-03-31T22:57:48Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Ramkumar Ramachandra wrote:\n> Jens Lehmann wrote:\n\n>> A\n>> commit is the thing to record here because it *is* the perfect fit\n>\n> Might be, but saying that doesn't help one bit. I want to know why.\n[...]\n> To summarize, everyone seems to be elated with the current state of\n> submodules and is vehemently defending it. I'm a little unhappy, but\n> am unable to express my discontent in better prose. Let's just go\n> back to writing patches, and come back to this if and when I have a\n> full design.\n\nElated is probably not the right word. More \"annoyed at being told\ntheir work is ugly without an accompanying concrete and actionable bug\nreport\". :)\n\nIf you are curious, at a quieter time it might be useful to ask for\npointers to the discussions that led to the current design, and folks\non the list might be glad to help.\n\nHope that helps,\nJonathan\n"},{"id":"212761","messageId":"CABURp0q9mV+-tEtHGpE4mh9cdbhkA8fr4i7XpBtK0fpfSYg-+A@mail.gmail.com","threadId":"33285","inReplyTo":"CALkWK0k=g3iFjmpUQA1VkuH2kZsVX1_Hpo=LZ7CuotwHz_1++g@mail.gmail.com","subject":"Re: Composing git repositories","fromName":"Phil Hord","fromEmail":"phil.hord@gmail.com","sentAt":"2013-03-31T23:50:05Z","receivedAt":"2013-03-31T23:50:05Z","isPatch":false,"sender":{"key":"phil.hord@gmail.com","avatar":"https://avatars.githubusercontent.com/u/123908?v=4"},"body":"On Sun, Mar 31, 2013 at 4:34 PM, Ramkumar Ramachandra\n wrote:\n> Thanks for taking the time and effort to review my thoughts.\n>\n> Jens Lehmann wrote:\n>> A\n>> commit is the thing to record here because it *is* the perfect fit\n>\n> Might be, but saying that doesn't help one bit. I want to know why.\n>\n>> I want to improve the user experience\n>> of submodules and don't care much in what language we achieve that.\n>\n> You missed the point entirely. If git didn't have a commit object,\n> would you use a special kind of blob and code around everything to\n> avoid fixing a more fundamental issue?\n>\n>> What happens when you rename \"magit\" to \"foo\" in that branch and want\n>> to check out an older commit of the same branch? That is one of the\n>> reasons why that belongs in to a checked in .gitmodules and not\n>> someplace untracked.\n>\n> Good point. I learnt something new.\n>\n>> Are you aware that current Git code already stats all files across\n>> all submodules recursive by default? So (again) no problem here, we\n>> do that already (unless configured otherwise).\n>\n> I didn't know that. Why does it do this?\n>\n>> Guess what: submodules are the solution for a certain set of use\n>> cases, and tools like subtree are a solution for another set of\n>> use cases. There is no silver bullet.\n>\n> That's the core of your argument: submodules already solve what it\n> was meant to, and we can't get it to solve a larger class of problems.\n> In other words, you're implying that it's impossible to build a tool\n> that will be able to compose git repositories in a way that solves a\n> very large class of problems. I don't see conclusive proof for this,\n> so I have to disagree.\n\nI think it is possible to solve larger classes of problems with\nsubmodules, but it is a harder problem than you seem to think. In any\ncase I do not think you need to re-engineer submodules to improve\nthem.\n\nSumodules are good for preserving history. When properly managed,\nthey answer the question git always answers, \"Where was my code in the\npast?\" I would like proper management to be easier, but I understand\nwhy it is difficult; and I see it getting easier.\n\nSome users also want submodules to handle other tasks, like \"Import\nbranch-tracked upstream changes (i.e. git pull origin master).\" This\ntoo is useful, but it is a different problem than submodules'\nprimarily try to solve. But they do already solve _part_ of that\nproblem (\"Show me how these modules are related\"), so it seems a\ntrivial thing to ask them also to handle the \"floating branch\" task.\nThe trick is to handle this task in a way that does not break the task\nthey are designed and used for already.\n\nSome other users want submodules to solve the problem of composition,\nlike \"Show me the combined log of all these submodules.\" (Replace\n\"show log\" with \"diff\", \"merge\", \"bisect\" or even \"rebase\" if you\nlike.) I think this is where Jens is leaning when working to improve\nthe user experience. But this direction does not require\nre-architecting the fundamentals of submodules.\n\n\n> To summarize, everyone seems to be elated with the current state of\n> submodules and is vehemently defending it.\n\nThat's a gross overstatement. Everyone understands that is\ncomplicated and dangerous tweak a machine in operation. Such tweaks\nshould be safe, prudent and justifiable, in roughly that order.\n\nPhil\n"},{"id":"212762","messageId":"201304010016.r310G79C032108@no.baka.org","threadId":"33285","inReplyTo":"CALkWK0=CsuAWQwk5Guf0pbC4_ZEoZiwQpamcRvBGz5LJ0QGKHg@mail.gmail.com","subject":"Re: Composing git repositories","fromName":"Seth Robertson","fromEmail":"in-gitvger@baka.org","sentAt":"2013-04-01T00:16:07Z","receivedAt":"2013-04-01T00:16:07Z","isPatch":false,"sender":{"key":"in-gitvger@baka.org","avatar":null},"body":"\nIn message , Ramkumar Ramachandra writes:\n\n As a user inexperienced with recursive submodules (I've only used them\n in this repository), I found it highly confusing. Thanks for clearing\n them up.\n\nYou may want to investigate third party alternatives like gitslave\nhttp://gitslave.sf.net Depending on what problem you are trying to\nsolve, it can be better (or worse) to use compared to submodules.\n\nIt provides a usually conceptually easier method to group subprojects\ntogether into a superproject. You can replace practically any git\ncommand you want with \"gits\" and it will usually work as you might\nmore or less expect. Conceptually it has a list of git repositories\nand will execute the listed git command on each in turn.\n\nIt seems to resolve most of the issues that you raise, but of course\nit has some warts of its own. Some could be resolved with sufficient\neffort, others are fundamental. (An example of the latter, you cannot\ntrivially tell what commit in other repositories a particular user was\nat when he made a commit in a specific repository (absent a gits tag\nbeing created). An example of the former, if you have git output\npaging turned on and many subprojects to check out, `gits clone` pages\nthe output and more to the point, blocks the clones until you page\nthrough the output which you must typically do many times).\n\n\t\t\t\t\t-Seth Robertson\n"},{"id":"212790","messageId":"5159585A.3060907@web.de","threadId":"33285","inReplyTo":"CALkWK0k=g3iFjmpUQA1VkuH2kZsVX1_Hpo=LZ7CuotwHz_1++g@mail.gmail.com","subject":"Re: Composing git repositories","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2013-04-01T09:50:18Z","receivedAt":"2013-04-01T09:50:18Z","isPatch":false,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 31.03.2013 22:34, schrieb Ramkumar Ramachandra:\n>> Are you aware that current Git code already stats all files across\n>> all submodules recursive by default? So (again) no problem here, we\n>> do that already (unless configured otherwise).\n> \n> I didn't know that. Why does it do this?\n\nTo show the user work tree changes inside the submodules too. It\nwas really easy to forget to commit accompanying submodule changes\nwhen preparing a superproject commit before that (and that broke\nquite some builds at my $dayjob until we fixed that).\n\n>> Guess what: submodules are the solution for a certain set of use\n>> cases, and tools like subtree are a solution for another set of\n>> use cases. There is no silver bullet.\n> \n> That's the core of your argument: submodules already solve what it\n> was meant to, and we can't get it to solve a larger class of problems.\n> In other words, you're implying that it's impossible to build a tool\n> that will be able to compose git repositories in a way that solves a\n> very large class of problems. I don't see conclusive proof for this,\n> so I have to disagree.\n> \n> To summarize, everyone seems to be elated with the current state of\n> submodules and is vehemently defending it. I'm a little unhappy, but\n> am unable to express my discontent in better prose.\n\nI just think it is too early for the \"let's do things differently\nand redesign stuff\" phase. Before doing that we have to clear up\nthe things that currently don't work for you or are confusing you\nabout submodules and see if they are already solved (which could\nlead to a documentation update) or what it would take to fix them\nusing current submodule infrastructure. I have the strong feeling\nthat after we did that, no design changes are necessary anymore.\n"},{"id":"212792","messageId":"51597A37.1010301@web.de","threadId":"33285","inReplyTo":"CABURp0q9mV+-tEtHGpE4mh9cdbhkA8fr4i7XpBtK0fpfSYg-+A@mail.gmail.com","subject":"Re: Composing git repositories","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2013-04-01T12:14:47Z","receivedAt":"2013-04-01T12:14:47Z","isPatch":false,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 01.04.2013 01:50, schrieb Phil Hord:\n> On Sun, Mar 31, 2013 at 4:34 PM, Ramkumar Ramachandra\n> wrote:\n>> Jens Lehmann wrote:\n>>> Guess what: submodules are the solution for a certain set of use\n>>> cases, and tools like subtree are a solution for another set of\n>>> use cases. There is no silver bullet.\n>>\n>> That's the core of your argument: submodules already solve what it\n>> was meant to, and we can't get it to solve a larger class of problems.\n>> In other words, you're implying that it's impossible to build a tool\n>> that will be able to compose git repositories in a way that solves a\n>> very large class of problems. I don't see conclusive proof for this,\n>> so I have to disagree.\n> \n> I think it is possible to solve larger classes of problems with\n> submodules, but it is a harder problem than you seem to think. In any\n> case I do not think you need to re-engineer submodules to improve\n> them.\n> \n> Sumodules are good for preserving history. When properly managed,\n> they answer the question git always answers, \"Where was my code in the\n> past?\" I would like proper management to be easier, but I understand\n> why it is difficult; and I see it getting easier.\n\nExactly.\n\n> Some users also want submodules to handle other tasks, like \"Import\n> branch-tracked upstream changes (i.e. git pull origin master).\" This\n> too is useful, but it is a different problem than submodules'\n> primarily try to solve. But they do already solve _part_ of that\n> problem (\"Show me how these modules are related\"), so it seems a\n> trivial thing to ask them also to handle the \"floating branch\" task.\n> The trick is to handle this task in a way that does not break the task\n> they are designed and used for already.\n\nBut I think we recently learned to support that use case with\nsubmodules. I think there are two floating models:\n\n- Tracked:\n Follow a branch in the submodule and let git help you to advance\n the submodule to the tip of that branch at certain times, but\n still record a certain SHA-1 in the superproject to maintain\n reproducibility. We support this since 1.8.2 (see 06b1abb5 by\n Trevor).\n\n- Untracked:\n Some people just want \"the newest\" tip of a branch checked out in\n the submodule and update that from time to time (I suspect this\n is because they are used to SVN externals, which I believe work\n that way). You throw away reproducibility, which I think is not\n good and not the way I expect Git to work. But that use case is\n achieved with a simple script and some config settings telling\n Git to don't even look for changes in the submodule anymore, and\n submodule infrastructure will set up everything for you after\n cloning the superproject. You run your custom script from time\n to time and have a truly floating submodule.\n\nSo to me it looks we support both floating models with current Git's\nsubmodule infrastructure.\n\n> Some other users want submodules to solve the problem of composition,\n> like \"Show me the combined log of all these submodules.\" (Replace\n> \"show log\" with \"diff\", \"merge\", \"bisect\" or even \"rebase\" if you\n> like.) I think this is where Jens is leaning when working to improve\n> the user experience. But this direction does not require\n> re-architecting the fundamentals of submodules.\n\nCorrect. The only major change needed for that was to move the .git\ndirectories into the .git directory of the superproject to prepare\nfor recursive update. But that is done under the hood and didn't\ntouch the fundamentals of using gitlinks and .gitmodules, it is just\na change in the layout of the local clone.\n"},{"id":"212797","messageId":"CABURp0qnTLiB6zrdYfZfLP9+WJXT3umNggsfjT9TfDVn9V48HA@mail.gmail.com","threadId":"33285","inReplyTo":"51597A37.1010301@web.de","subject":"Re: Composing git repositories","fromName":"Phil Hord","fromEmail":"phil.hord@gmail.com","sentAt":"2013-04-01T14:49:01Z","receivedAt":"2013-04-01T14:49:01Z","isPatch":false,"sender":{"key":"phil.hord@gmail.com","avatar":"https://avatars.githubusercontent.com/u/123908?v=4"},"body":"On Mon, Apr 1, 2013 at 8:14 AM, Jens Lehmann wrote:\n> Am 01.04.2013 01:50, schrieb Phil Hord:\n>> On Sun, Mar 31, 2013 at 4:34 PM, Ramkumar Ramachandra\n>> wrote:\n>>> Jens Lehmann wrote:\n>>>> Guess what: submodules are the solution for a certain set of use\n>>>> cases, and tools like subtree are a solution for another set of\n>>>> use cases. There is no silver bullet.\n>>>\n>>> That's the core of your argument: submodules already solve what it\n>>> was meant to, and we can't get it to solve a larger class of problems.\n>>> In other words, you're implying that it's impossible to build a tool\n>>> that will be able to compose git repositories in a way that solves a\n>>> very large class of problems. I don't see conclusive proof for this,\n>>> so I have to disagree.\n>>\n>> I think it is possible to solve larger classes of problems with\n>> submodules, but it is a harder problem than you seem to think. In any\n>> case I do not think you need to re-engineer submodules to improve\n>> them.\n>>\n>> Sumodules are good for preserving history. When properly managed,\n>> they answer the question git always answers, \"Where was my code in the\n>> past?\" I would like proper management to be easier, but I understand\n>> why it is difficult; and I see it getting easier.\n>\n> Exactly.\n>\n>> Some users also want submodules to handle other tasks, like \"Import\n>> branch-tracked upstream changes (i.e. git pull origin master).\" This\n>> too is useful, but it is a different problem than submodules'\n>> primarily try to solve. But they do already solve _part_ of that\n>> problem (\"Show me how these modules are related\"), so it seems a\n>> trivial thing to ask them also to handle the \"floating branch\" task.\n>> The trick is to handle this task in a way that does not break the task\n>> they are designed and used for already.\n>\n> But I think we recently learned to support that use case with\n> submodules. I think there are two floating models:\n>\n> - Tracked:\n> Follow a branch in the submodule and let git help you to advance\n> the submodule to the tip of that branch at certain times, but\n> still record a certain SHA-1 in the superproject to maintain\n> reproducibility. We support this since 1.8.2 (see 06b1abb5 by\n> Trevor).\n\nThanks. I followed that thread closely, but I thought the patch had\nstalled again. I'm glad to see it in master\n\n> - Untracked:\n> Some people just want \"the newest\" tip of a branch checked out in\n> the submodule and update that from time to time (I suspect this\n> is because they are used to SVN externals, which I believe work\n> that way). You throw away reproducibility, which I think is not\n> good and not the way I expect Git to work. But that use case is\n> achieved with a simple script and some config settings telling\n> Git to don't even look for changes in the submodule anymore, and\n> submodule infrastructure will set up everything for you after\n> cloning the superproject. You run your custom script from time\n> to time and have a truly floating submodule.\n>\n> So to me it looks we support both floating models with current Git's\n> submodule infrastructure.\n\nWell, for that matter, the \"tracked\" floating tip was also a simple\nscript once a upon a time. To say we support both is nothing new.\nYou could say we supported both in 1.8.1, and now we support \"Tracked\"\na little bit better in 1.8.2.\n\nI think the difference is that everyone's expectation for \"Untracked\"\nis a little bit different; or maybe it is just dangerous enough that\nit should not be a core feature.\n\nBut I ain't complainin'.\n\nPhil\n"},{"id":"212924","messageId":"CALkWK0mgtfYFd+sT=J-hAMLq=HVF-_a-kT_xxE9-ZzfiBiFBQA@mail.gmail.com","threadId":"33285","inReplyTo":"20130331225747.GB11704@elie.Belkin","subject":"Re: Composing git repositories","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-04-02T17:44:49Z","receivedAt":"2013-04-02T17:44:49Z","isPatch":false,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Jonathan Nieder wrote:\n> Elated is probably not the right word. More \"annoyed at being told\n> their work is ugly without an accompanying concrete and actionable bug\n> report\". :)\n\nIf I had an actionable report, I'd have started hammering patches\ninstead of wasting everyone's time here. I'm was presenting fragments\nof my thoughts, hoping that it turn into concrete actionable work\nafter exchanging a few emails. I'm also annoyed that it didn't\nhappen.\n\n> If you are curious, at a quieter time it might be useful to ask for\n> pointers to the discussions that led to the current design, and folks\n> on the list might be glad to help.\n\nWill do. The search on GMane is no good, and taking a local dump to\nsearch using real tools is just too painful; does someone already have\na local dump?\n"},{"id":"212929","messageId":"20130402175839.GF24698@sigill.intra.peff.net","threadId":"33285","inReplyTo":"CALkWK0mgtfYFd+sT=J-hAMLq=HVF-_a-kT_xxE9-ZzfiBiFBQA@mail.gmail.com","subject":"Re: Composing git repositories","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2013-04-02T17:58:39Z","receivedAt":"2013-04-02T17:58:39Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Apr 02, 2013 at 11:14:49PM +0530, Ramkumar Ramachandra wrote:\n\n> > If you are curious, at a quieter time it might be useful to ask for\n> > pointers to the discussions that led to the current design, and folks\n> > on the list might be glad to help.\n> \n> Will do. The search on GMane is no good, and taking a local dump to\n> search using real tools is just too painful; does someone already have\n> a local dump?\n\nYes, I have a maildir with the complete list archive, which I index\nusing mairix (or sometimes grep if I'm doing something particularly\ntricky). I find the search on gmane to be abysmal.\n\nI'm happy to make my dump available to anyone who wants it, but it's\nkind of big (about 1.4G uncompressed).\n\n-Peff\n"},{"id":"212930","messageId":"7v7gkkern9.fsf@alter.siamese.dyndns.org","threadId":"33285","inReplyTo":"20130331225747.GB11704@elie.Belkin","subject":"Re: Composing git repositories","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-04-02T18:03:54Z","receivedAt":"2013-04-02T18:03:54Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Nieder writes:\n\n> If you are curious, at a quieter time it might be useful to ask for\n> pointers to the discussions that led to the current design, and folks\n> on the list might be glad to help.\n\nNot on the current design but the discussion before that round that\ninfluenced the outcome greatly was this:\n\n http://thread.gmane.org/gmane.comp.version-control.git/14486/focus=14492\n\nwhere we discussed a separate \"gitlink\" type of object.\n\nAnd obviously this discussion is also a must read:\n\n http://thread.gmane.org/gmane.comp.version-control.git/44106\n\nI vaguely recall asking (or seeing somebody ask) why Linus ended up\nwith using \"commit in index\" without introducing a separate gitlink\ntype, but I didn't find it. IIRC, the answer was \"it turned out\nthat we didn't need it\" or something like that, which I tend to\nagree.\n"},{"id":"212933","messageId":"CALkWK0nVax9HtM-M2zo-KH6U2jWznaUH9yBn4y1wqDW8f-mfOg@mail.gmail.com","threadId":"33285","inReplyTo":"51597A37.1010301@web.de","subject":"Re: Composing git repositories","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-04-02T18:35:54Z","receivedAt":"2013-04-02T18:35:54Z","isPatch":false,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Jens Lehmann wrote:\n> But I think we recently learned to support that use case with\n> submodules. I think there are two floating models:\n>\n> - Tracked:\n> [...]\n>\n> - Untracked:\n> Some people just want \"the newest\" tip of a branch checked out in\n> the submodule and update that from time to time (I suspect this\n> is because they are used to SVN externals, which I believe work\n> that way). You throw away reproducibility, which I think is not\n> good and not the way I expect Git to work.\n> [...]\n\nNope, it has nothing to do with SVN externals; I've never used them.\nAnd no, all repositories aren't created equal. I should be able to\nadd in magit.git into my dotfiles repository without worrying about\nwhich commits the other repositories were at a particular commit. If\nmy project depends on the bleeding edge of poppler and girarra, I\nshould always be able to tell what commits in each subproject the\nbuild was passing in. In other words, I should be able to freely\nmixed floating and fixed submodules. There's no reason for one to be\nRight, and the other to be a shunned second-class citizen.\n"},{"id":"212934","messageId":"20130402185426.GG28148@google.com","threadId":"33285","inReplyTo":"CALkWK0nVax9HtM-M2zo-KH6U2jWznaUH9yBn4y1wqDW8f-mfOg@mail.gmail.com","subject":"Re: Composing git repositories","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2013-04-02T18:54:26Z","receivedAt":"2013-04-02T18:54:26Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Ramkumar Ramachandra wrote:\n\n> I should be able to\n> add in magit.git into my dotfiles repository\n\nFor this, ideally what we'd want is a file that lists repositories\nthat should be cloned at the same time as cloning the dotfiles\nrepository, to reconstitute your dotfiles on a new machine. From then\non, when acting on the dotfiles repository (with \"git status\", \"git\ncommit\", etc) git should not pay attention to the magit subdirectory\nat all, because there is no coupling between the two.\n\nDid I get that right?\n\nThat sounds similar to what Junio does with the Meta subdirectory in\nhis git development worktree. I don't think submodules are a good\nfit, but it might make sense to start respecting a .motd file to allow\nthe following in a hypothetical world where everyone who clones git\nuses the same scripts Junio does:\n\n\t$ git clone git://repo.or.cz/git.git\n\tCloning into 'git'...\n\tremote: Counting objects: 151283, done.\n\tremote: Compressing objects: 100% (38546/38546), done.\n\tremote: Total 151283 (delta 111004), reused 151073 (delta 110797)\n\tReceiving objects: 100% (151283/151283), 36.39 MiB | 7.66 MiB/s, done.\n\tResolving deltas: 100% (111004/111004), done.\n\n\tDon't forget to \"git clone -b todo git://repo.or.cz/git.git git/Meta\"\n\tfor maintenance scripts.\n\t$\n\nThat would allow you to include an arbitrary setup script (including\ncloning dependencies as well as running \"autoreconf\" or whatever) and\ngive people cloning a quick reminder to inspect it if paranoid and\nthen run it.\n\nJonathan\n"},{"id":"212958","messageId":"7vr4isda1u.fsf@alter.siamese.dyndns.org","threadId":"33285","inReplyTo":"20130402185426.GG28148@google.com","subject":"Re: Composing git repositories","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-04-02T19:09:17Z","receivedAt":"2013-04-02T19:09:17Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Nieder writes:\n\n> ..., but it might make sense to start respecting a .motd file to allow\n> the following in a hypothetical world where everyone who clones git\n> uses the same scripts Junio does:\n>\n> \t$ git clone git://repo.or.cz/git.git\n> \tCloning into 'git'...\n> \tremote: Counting objects: 151283, done.\n> \tremote: Compressing objects: 100% (38546/38546), done.\n> \tremote: Total 151283 (delta 111004), reused 151073 (delta 110797)\n> \tReceiving objects: 100% (151283/151283), 36.39 MiB | 7.66 MiB/s, done.\n> \tResolving deltas: 100% (111004/111004), done.\n>\n> \tDon't forget to \"git clone -b todo git://repo.or.cz/git.git git/Meta\"\n> \tfor maintenance scripts.\n> \t$\n>\n> That would allow you to include an arbitrary setup script (including\n> cloning dependencies as well as running \"autoreconf\" or whatever) and\n> give people cloning a quick reminder to inspect it if paranoid and\n> then run it.\n\nI do not think motd is a good fit for the above; the message will\ndisappear any time once you are done cloning. Depending on how\ncommon the use of Meta/ should be, it is more appropriate to have\nsuch an instruction in README or some other places as tracked\ncontents.\n\nI do not know if the above is related to the problem Ram is trying\nto solve or a totally orthogonal issue, though.\n"},{"id":"212959","messageId":"CALkWK0kCcSgHfmTuQc-0XGHOdm6PPaVHqFeD4bko-zq3pH8mUw@mail.gmail.com","threadId":"33285","inReplyTo":"20130402185426.GG28148@google.com","subject":"Re: Composing git repositories","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-04-02T19:11:38Z","receivedAt":"2013-04-02T19:11:38Z","isPatch":false,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Jonathan Nieder wrote:\n> That sounds similar to what Junio does with the Meta subdirectory in\n> his git development worktree. I don't think submodules are a good\n> fit, but it might make sense to start respecting a .motd file to allow\n> the following in a hypothetical world where everyone who clones git\n> uses the same scripts Junio does:\n>\n> $ git clone git://repo.or.cz/git.git\n> Cloning into 'git'...\n> remote: Counting objects: 151283, done.\n> remote: Compressing objects: 100% (38546/38546), done.\n> remote: Total 151283 (delta 111004), reused 151073 (delta 110797)\n> Receiving objects: 100% (151283/151283), 36.39 MiB | 7.66 MiB/s, done.\n> Resolving deltas: 100% (111004/111004), done.\n>\n> Don't forget to \"git clone -b todo git://repo.or.cz/git.git git/Meta\"\n> for maintenance scripts.\n> $\n\nNope, it's not mandatory for everyone to use dotfiles.git in exactly\nthe same way either. In other words: I'm not sitting in an office and\nworking with my colleagues on exactly the same things, in exactly the\nsame way; wasn't that the Subversion age? Some might decide to\ninitialize a few submodules, change the URLs of some, and remove some.\n I'd want my private fork to have commits changing \"initialize\nsubmodule quux\" to \"don't initialize submodule quux\", and be able to\nrebase that on top of upstream. Why are you leaning towards solutions\nfor very narrow usecases?\n"},{"id":"212963","messageId":"CALkWK0nq-Cfkf+s4BWiFPAg4LYefW=+ifbazLUnvDv39OWhjcQ@mail.gmail.com","threadId":"33285","inReplyTo":"201304010016.r310G79C032108@no.baka.org","subject":"Re: Composing git repositories","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-04-02T19:19:07Z","receivedAt":"2013-04-02T19:19:07Z","isPatch":false,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Seth Robertson wrote:\n>\n> In message , Ramkumar Ramachandra writes:\n>\n> As a user inexperienced with recursive submodules (I've only used them\n> in this repository), I found it highly confusing. Thanks for clearing\n> them up.\n>\n> You may want to investigate third party alternatives like gitslave\n> http://gitslave.sf.net Depending on what problem you are trying to\n> solve, it can be better (or worse) to use compared to submodules.\n\nI'm not looking for yet another specialized tool for my special\ncombined-repository workflow. I'm looking to build one generalized\ntool that will be able to do various kinds of compositions equally\nwell.\n\n> [...]\n\nThanks. I learnt one more workflow that people want to use.\n\nIt's one large Perl script. Quite an involved hack.\n"},{"id":"212964","messageId":"20130402192017.GI28148@google.com","threadId":"33285","inReplyTo":"CALkWK0kCcSgHfmTuQc-0XGHOdm6PPaVHqFeD4bko-zq3pH8mUw@mail.gmail.com","subject":"Re: Composing git repositories","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2013-04-02T19:20:17Z","receivedAt":"2013-04-02T19:20:17Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Ramkumar Ramachandra wrote:\n> Jonathan Nieder wrote:\n\n>> $ git clone git://repo.or.cz/git.git\n[...]\n>> Don't forget to \"git clone -b todo git://repo.or.cz/git.git git/Meta\"\n>> for maintenance scripts.\n>> $\n>\n> Nope, it's not mandatory for everyone to use dotfiles.git in exactly\n> the same way either. In other words: I'm not sitting in an office and\n> working with my colleagues on exactly the same things, in exactly the\n> same way; wasn't that the Subversion age? Some might decide to\n> initialize a few submodules, change the URLs of some, and remove some.\n\nCan't a script pointed to in README handle all these things?\n\n> I'd want my private fork to have commits changing \"initialize\n> submodule quux\" to \"don't initialize submodule quux\", and be able to\n> rebase that on top of upstream.\n\nThese would be patches to comment or uncomment repositories in the\nlist used by your \"setup\" script.\n\n> Why are you leaning towards solutions\n> for very narrow usecases?\n\nI don't think this hostile way of explaining things is warranted. :/\n\nYours,\nJonathan\n"},{"id":"212966","messageId":"CALkWK0kFm8n9CgtvtW=b-JPKO-ZJBn0Dh6z9B0C0_7_EJAb_7A@mail.gmail.com","threadId":"33285","inReplyTo":"20130402192017.GI28148@google.com","subject":"Re: Composing git repositories","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-04-02T19:29:50Z","receivedAt":"2013-04-02T19:29:50Z","isPatch":false,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Jonathan Nieder wrote:\n> Ramkumar Ramachandra wrote:\n>> Jonathan Nieder wrote:\n>\n>>> $ git clone git://repo.or.cz/git.git\n> [...]\n>>> Don't forget to \"git clone -b todo git://repo.or.cz/git.git git/Meta\"\n>>> for maintenance scripts.\n>>> $\n>>\n>> Nope, it's not mandatory for everyone to use dotfiles.git in exactly\n>> the same way either. In other words: I'm not sitting in an office and\n>> working with my colleagues on exactly the same things, in exactly the\n>> same way; wasn't that the Subversion age? Some might decide to\n>> initialize a few submodules, change the URLs of some, and remove some.\n>\n> Can't a script pointed to in README handle all these things?\n\nIf it came to that, it's quite possible to write one giant Perl script\nto do whatever I want (see: git-slave).\n\nWhat will I be merging and rebasing? One configuration file stuffed\nwith miscellaneous repositories. Don't you think this is highly\nunpleasant? Why would I want to write slightly different scripts for\neach of my repositories? I'd want to write down several different\nadjustable knobs to use something that's well-integrated into core\ngit, like we do in .git/config (except we can't share a .gitconfig\nyet).\n\n> I don't think this hostile way of explaining things is warranted. :/\n\nSorry about that. No hostility intended.\n"},{"id":"212967","messageId":"CALkWK0nS7OT8XeGqFmotDjbcHj=rCVVs=C4RCu4e=zd-R4Ztbw@mail.gmail.com","threadId":"33285","inReplyTo":"20130402175839.GF24698@sigill.intra.peff.net","subject":"Re: Composing git repositories","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-04-02T19:33:20Z","receivedAt":"2013-04-02T19:33:20Z","isPatch":false,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Jeff King wrote:\n> I'm happy to make my dump available to anyone who wants it, but it's\n> kind of big (about 1.4G uncompressed).\n\nThanks. Can you put it up publicly somewhere (Dropbox comes to mind),\nand send me a link?\n"},{"id":"212973","messageId":"CALkWK0=HJP_ScNn7BQzvZeacFEUpN4i3MhYMUt1x0mHNcv3U4A@mail.gmail.com","threadId":"33285","inReplyTo":"CALkWK0kFm8n9CgtvtW=b-JPKO-ZJBn0Dh6z9B0C0_7_EJAb_7A@mail.gmail.com","subject":"Re: Composing git repositories","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2013-04-02T19:49:02Z","receivedAt":"2013-04-02T19:49:02Z","isPatch":false,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Ramkumar Ramachandra wrote:\n> What will I be merging and rebasing? One configuration file stuffed\n> with miscellaneous repositories. Don't you think this is highly\n> unpleasant?\n\nI spoke too fast. Isn't that exactly what we do with .gitmodules\ntoday (I'm not saying it's ideal, but I can't think of an\nalternative)?\n\nYes, you're right: a simple script with a configuration file can\nmanage only floating submodules quite well.\n"},{"id":"212978","messageId":"515B37E3.6030706@web.de","threadId":"33285","inReplyTo":"CALkWK0mgtfYFd+sT=J-hAMLq=HVF-_a-kT_xxE9-ZzfiBiFBQA@mail.gmail.com","subject":"Re: Composing git repositories","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2013-04-02T19:56:19Z","receivedAt":"2013-04-02T19:56:19Z","isPatch":false,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 02.04.2013 19:44, schrieb Ramkumar Ramachandra:\n> Jonathan Nieder wrote:\n>> Elated is probably not the right word. More \"annoyed at being told\n>> their work is ugly without an accompanying concrete and actionable bug\n>> report\". :)\n> \n> If I had an actionable report, I'd have started hammering patches\n> instead of wasting everyone's time here. I'm was presenting fragments\n> of my thoughts, hoping that it turn into concrete actionable work\n> after exchanging a few emails. I'm also annoyed that it didn't\n> happen.\n\nBut didn't we already note three worthwhile patches (get rid of the\ntop-level requirement and relocate a .git directory at \"add\" time\nand anytime later)?\n"},{"id":"212980","messageId":"515B38B0.2040705@web.de","threadId":"33285","inReplyTo":"CALkWK0nVax9HtM-M2zo-KH6U2jWznaUH9yBn4y1wqDW8f-mfOg@mail.gmail.com","subject":"Re: Composing git repositories","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2013-04-02T19:59:44Z","receivedAt":"2013-04-02T19:59:44Z","isPatch":false,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 02.04.2013 20:35, schrieb Ramkumar Ramachandra:\n> Jens Lehmann wrote:\n>> But I think we recently learned to support that use case with\n>> submodules. I think there are two floating models:\n>>\n>> - Tracked:\n>> [...]\n>>\n>> - Untracked:\n>> Some people just want \"the newest\" tip of a branch checked out in\n>> the submodule and update that from time to time (I suspect this\n>> is because they are used to SVN externals, which I believe work\n>> that way). You throw away reproducibility, which I think is not\n>> good and not the way I expect Git to work.\n>> [...]\n> \n> Nope, it has nothing to do with SVN externals; I've never used them.\n> And no, all repositories aren't created equal. I should be able to\n> add in magit.git into my dotfiles repository without worrying about\n> which commits the other repositories were at a particular commit. If\n> my project depends on the bleeding edge of poppler and girarra, I\n> should always be able to tell what commits in each subproject the\n> build was passing in. In other words, I should be able to freely\n> mixed floating and fixed submodules. There's no reason for one to be\n> Right, and the other to be a shunned second-class citizen.\n\nBut you can currently mix floating and fixed submodules, as each\nsubmodule can be configured differently. Or am I missing something\nhere?\n"},{"id":"213084","messageId":"7v7gkilry2.fsf@alter.siamese.dyndns.org","threadId":"33285","inReplyTo":"7v7gkkern9.fsf@alter.siamese.dyndns.org","subject":"Re: Composing git repositories","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-04-04T06:40:05Z","receivedAt":"2013-04-04T06:40:05Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> Jonathan Nieder writes:\n>\n>> If you are curious, at a quieter time it might be useful to ask for\n>> pointers to the discussions that led to the current design, and folks\n>> on the list might be glad to help.\n>\n> Not on the current design but the discussion before that round that\n> influenced the outcome greatly was this:\n>\n> http://thread.gmane.org/gmane.comp.version-control.git/14486/focus=14492\n>\n> where we discussed a separate \"gitlink\" type of object.\n>\n> And obviously this discussion is also a must read:\n>\n> http://thread.gmane.org/gmane.comp.version-control.git/44106\n>\n> I vaguely recall asking (or seeing somebody ask) why Linus ended up\n> with using \"commit in index\" without introducing a separate gitlink\n> type, but I didn't find it. IIRC, the answer was \"it turned out\n> that we didn't need it\" or something like that, which I tend to\n> agree.\n\nFound a bit more relevant and probably more important (at the design\nlevel) discussion history for people interested in understanding why\nthe things are as they are (without which we cannot make progress\nwhile avoiding mistakes):\n\n http://thread.gmane.org/gmane.comp.version-control.git/15072\n http://thread.gmane.org/gmane.comp.version-control.git/31941/focus=32302\n http://thread.gmane.org/gmane.comp.version-control.git/47466/focus=47621\n"},{"id":"213230","messageId":"CACsJy8C_dRqdPvAUW19zVLrJQGqFCRu_TaPMnRbkfgq+H9V2dw@mail.gmail.com","threadId":"33285","inReplyTo":"7v7gkilry2.fsf@alter.siamese.dyndns.org","subject":"Re: Composing git repositories","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2013-04-05T02:36:06Z","receivedAt":"2013-04-05T02:36:06Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Apr 4, 2013 at 5:40 PM, Junio C Hamano wrote:\n>> Not on the current design but the discussion before that round that\n>> influenced the outcome greatly was this:\n>>\n>> http://thread.gmane.org/gmane.comp.version-control.git/14486/focus=14492\n>>\n>> where we discussed a separate \"gitlink\" type of object.\n>>\n>> And obviously this discussion is also a must read:\n>>\n>> http://thread.gmane.org/gmane.comp.version-control.git/44106\n>>\n>\n> Found a bit more relevant and probably more important (at the design\n> level) discussion history for people interested in understanding why\n> the things are as they are (without which we cannot make progress\n> while avoiding mistakes):\n>\n> http://thread.gmane.org/gmane.comp.version-control.git/15072\n> http://thread.gmane.org/gmane.comp.version-control.git/31941/focus=32302\n> http://thread.gmane.org/gmane.comp.version-control.git/47466/focus=47621\n\nShould someone add these links to the source code (maybe as a comment\nin submodule.c, or above the definition of S_IFGITLINK in cache.h)? A\nbrief summary or outcome from these links in the comment would be\nnice.\n--\nDuy\n"},{"id":"213236","messageId":"7vwqshefyk.fsf@alter.siamese.dyndns.org","threadId":"33285","inReplyTo":"CACsJy8C_dRqdPvAUW19zVLrJQGqFCRu_TaPMnRbkfgq+H9V2dw@mail.gmail.com","subject":"Re: Composing git repositories","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-04-05T04:53:07Z","receivedAt":"2013-04-05T04:53:07Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Duy Nguyen writes:\n\n> Should someone add these links to the source code (maybe as a comment\n> in submodule.c, or above the definition of S_IFGITLINK in cache.h)?\n\nThey were given in response to a request for reading material to\nlearn background. Most of the straw-man outlines raised in these old\ndiscussoin threads are very different from what we ended up doing,\nand their value is mostly to learn what kind of use cases should one\nconsider if one were designing subproject support from scratch, what\npossible approaches were considered and what were found already\nlacking in them before we even had a working prototype (in other\nwords, to learn where one should _not_ go).\n\nLink themselves are not very useful and definitely not appropriate\nas a comment next to S_IFGITLINK, as S_IFGITLINK is not the only\nthing that implements the subproject support, and not all the ideals\ndreamed in these threads are realized with the implementation (yet).\n\n> A brief summary or outcome from these links in the comment would\n> be nice.\n\nA summary of what to consider in Documentation/technical/ somewhere\nmay be a very welcome addition. Thanks for volunteering ;-).\n"},{"id":"213239","messageId":"CACsJy8DX8xAc_CUFLYtGX9Ynm-kzgrbfCRFXaZaGd_+G4_imsg@mail.gmail.com","threadId":"33285","inReplyTo":"7vwqshefyk.fsf@alter.siamese.dyndns.org","subject":"Re: Composing git repositories","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2013-04-05T05:27:43Z","receivedAt":"2013-04-05T05:27:43Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Fri, Apr 5, 2013 at 3:53 PM, Junio C Hamano wrote:\n>> A brief summary or outcome from these links in the comment would\n>> be nice.\n>\n> A summary of what to consider in Documentation/technical/ somewhere\n> may be a very welcome addition. Thanks for volunteering ;-).\n\nNo thanks :-) I did not really follow this thread to make such\ncontribution. Still have some work to do with other topics.\n--\nDuy\n"},{"id":"213250","messageId":"515E7A20.1050003@web.de","threadId":"33285","inReplyTo":"CACsJy8DX8xAc_CUFLYtGX9Ynm-kzgrbfCRFXaZaGd_+G4_imsg@mail.gmail.com","subject":"Re: Composing git repositories","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2013-04-05T07:15:44Z","receivedAt":"2013-04-05T07:15:44Z","isPatch":false,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 05.04.2013 07:27, schrieb Duy Nguyen:\n> On Fri, Apr 5, 2013 at 3:53 PM, Junio C Hamano wrote:\n>>> A brief summary or outcome from these links in the comment would\n>>> be nice.\n>>\n>> A summary of what to consider in Documentation/technical/ somewhere\n>> may be a very welcome addition. Thanks for volunteering ;-).\n> \n> No thanks :-) I did not really follow this thread to make such\n> contribution. Still have some work to do with other topics.\n\nI'll do that (unless someone else steps up). I had these links in\nmy todo list with the intent of adding them to my github page, but\nI agree they make more sense in Documentation/technical. Will see\nwhen I find a time slot to read through the whole threads to give\nthem meaningful captions.\n"}]}