# Re: [RFC] Submodules in GIT 160 messages from 2006-11-28 to 2006-12-17. Participants: Martin Waitz, Josef Weidendorfer, Torgil Svensson, sf, Shawn Pearce, Linus Torvalds, Michael K. Edwards, Andreas Ericsson, Jakub Narebski, Sam Vilain, R. Steve McKown, Sven Verdoolaege, Uwe Kleine-Koenig, Andy Parkins, Stephan Feder, Jon Loeliger, Daniel Barkalow, Junio C Hamano, Alan Chandler, Steven Grimm. Thread: https://gitlist.dev/t/43065 ## Andy Parkins, 2006-11-28 10:29 Subject: Re: [RFC] Submodules in GIT Message-ID: <200611281029.11918.andyparkins@gmail.com> URL: https://gitlist.dev/e/200611281029.11918.andyparkins%40gmail.com In-Reply-To: <456C0313.3020308@op5.se> ``` On Tuesday 2006 November 28 09:36, Andreas Ericsson wrote: > I'd actually prefer the second solution here and let git print a list of > submodules with dirty state and ask for some sort of user-response > before creating the actual commit. As non-interactive commits should > always be clean, requiring user intervention on non-clean state should > be a safe thing to do. I'd agree. However, is there a need to require user intervention? Can we not make the following analogies to normal git operation: file in working directory -> submodule working directory file in index -> submodule repository It's perfectly possible to make a commit with different contents in the index and the working directory - it shows up in the git-status output very nicely. Why not deal with submodules in the same way? Now imagine the following repository: file1 file2 submodule1/file3 Make changes to file2 and file3, but don't update-index or commit. git-status would show: # Changed but not updated: # (use git-update-index to mark for commit) # # modified: file2 # dirty: submodule1 # nothing to commit Now "git-update-index file2" and git-status # Updated but not checked in: # (will commit) # # modified: file2 # # Changed but not updated: # (use git-update-index to mark for commit) # # dirty: submodule1/ # Now do a commit in submodule1/ and git-status in the supermodule. # Updated but not checked in: # (will commit) # # modified: file2 # submodule: submodule1/ # Obviously the detail would be different, but you get the idea. There is almost no difference between git-with-submodules and git-as-normal. I suppose there would actually need to be an extra step were the submodule is added to the supermodule index. So really there would be three states from git-status: # Updated but not checked in: # (will commit) # # modified: file2 # # Changed but not updated: # (use git-update-index to mark for commit) # # modified: file1 # submodule: submodule1/ # # Dirty submodules: # (commit changes in the submodule to clean) # # dirty: submodule1/ # Which means: since the last supermodule commit there has been * a change to file2, which is in the index and would be committed. * a change to file1, which is not in the index and won't be committed. * a commit to submodule1, which won't be committed * changes to the submodule working directory This really reinforces Linus's interpretation that submodules are directories - they would presumably just get a new object type and be referenced in the tree object. git-update-index would be blind to dirty submodules with no new commit, just as git-update-index on an unchanged file has no effect. Has this question been answered yet? How does the supermodule know which branch to track in the submodule? Does it simply track HEAD or when the submodule is added to the supermodule is it told which branch to track? I suppose it's got to be HEAD really hasn't it? Andy -- Dr Andy Parkins, M Eng (hons), MIEE ``` ## Jakub Narebski, 2006-11-28 10:50 Subject: Re: [RFC] Submodules in GIT Message-ID: URL: https://gitlist.dev/e/ekh45n%24rfc%241%40sea.gmane.org In-Reply-To: <200611281029.11918.andyparkins@gmail.com> ``` Andy Parkins wrote: > How does the supermodule know which > branch to track in the submodule?  Does it simply track HEAD or when the > submodule is added to the supermodule is it told which branch to track?  I > suppose it's got to be HEAD really hasn't it? I think that the proper place for that would be supermodule _index_. The supermodule tree would have commit entry, and the index would have symbolic branch (and perhaps some infor about where to find refs for submodule). This I guess breaks index abstraction slightly, but on the other hand allows for tracking non-HEAD branch of submodule, and for submodule to not know about supermodule at all... -- Jakub Narebski Warsaw, Poland ShadeHawk on #git ``` ## Andy Parkins, 2006-11-28 13:35 Subject: Re: [RFC] Submodules in GIT Message-ID: <200611281335.38728.andyparkins@gmail.com> URL: https://gitlist.dev/e/200611281335.38728.andyparkins%40gmail.com In-Reply-To: ``` On Tuesday 2006 November 28 10:50, Jakub Narebski wrote: > I think that the proper place for that would be supermodule _index_. > The supermodule tree would have commit entry, and the index would have > symbolic branch (and perhaps some infor about where to find refs for > submodule). > > This I guess breaks index abstraction slightly, but on the other hand > allows for tracking non-HEAD branch of submodule, and for submodule to > not know about supermodule at all... The reason I thought it would have to be HEAD at all times, is to prevent situations where the supermodule commit doesn't reflect the state of the current tree. Let's imagine that we're doing non-HEAD tracking in the supermodule. supermodule +-------- libsubmodule1 +-------- libsubmodule2 So, you do a "make" in supermodule; this of course will call make in each of the submodules. You test the output and find that it's all working nicely. Time for a supermodule commit. We want to freeze this working state. You commit and tag "supermodule-rc1" Unfortunately, during development, you've switched libsubmodule1 to branch "development", but supermodule isn't tracking libsubmodule1/HEAD it's tracking libsubmodule1/master. Your supermodule commit doesn't capture a snapshot of the tree you're using. Now you say to the mailing list "hey guys, can you test "supermodule-rc1"? They check it out, and find that everything is broken. Why? Because what you wanted to check in was libsubmodule@development, but what actually went in was libsubmodule@master. I think I've talked myself into the position where it definitely has to be HEAD being tracked in the submodules; anything else is a disaster waiting to happen because commit doesn't check in your current tree. Andy -- Dr Andy Parkins, M Eng (hons), MIEE ``` ## Shawn Pearce, 2006-11-28 15:44 Subject: Re: [RFC] Submodules in GIT Message-ID: <20061128154434.GD28337@spearce.org> URL: https://gitlist.dev/e/20061128154434.GD28337%40spearce.org In-Reply-To: <200611281335.38728.andyparkins@gmail.com> ``` Andy Parkins wrote: > I think I've talked myself into the position where it definitely has to be > HEAD being tracked in the submodules; anything else is a disaster waiting to > happen because commit doesn't check in your current tree. Yes, but not only that, HEAD is the only thing that fits with the rest of the git repository/index/working directory model. Lets review... What's HEAD? Its the commit which matches the index state as closely as possible, with the only differences being the changes in progress that are being prepared for the next commit (whose parent will be HEAD). If the index and working directory are both clean (no changes) then its also the current content of this directory, right? What's the index? Its what you are about to commit. What's the working directory? Its the current content, which may also be partially checked out or dirty. So HEAD in a submodule is the current content of that submodule. Therefore any update-index call on a submodule should load HEAD (totally ignoring whatever branch it refers to) into the supermodule index. -- ``` ## Andy Parkins, 2006-11-28 16:29 Subject: Re: [RFC] Submodules in GIT Message-ID: <200611281629.08636.andyparkins@gmail.com> URL: https://gitlist.dev/e/200611281629.08636.andyparkins%40gmail.com In-Reply-To: <20061128154434.GD28337@spearce.org> ``` On Tuesday 2006 November 28 15:44, Shawn Pearce wrote: > So HEAD in a submodule is the current content of that submodule. > Therefore any update-index call on a submodule should load HEAD > (totally ignoring whatever branch it refers to) into the supermodule > index. I was with you right up until here. Why should a submodule do anything to the supermodule? This is like saying, when I edit a working tree file, it should automatically call update-index. The supermodule index should only be updated in response to a manual update-index (or commit -a I suppose). Worse, if you allow that to happen, the supermodule can commit a state that cannot be retrieved from the submodule's repository. The ONLY thing a supermodule can record about a submodule is a commit. Changing the index doesn't create a commit, so it can't change anything in the supermodule. If you change the submodule index then that submodule is "dirty", this state has no parallel with normal git operation. The nearest thing is that you've changed a file but not saved it. Apart from showing the "dirty" state in the supermodule's git-status, I don't see that there is anything that the supermodule can do - it can't go around committing in a repository that it not itself. IMO, it should always be possible to take a submodule and work on it in isolation - in an extreme case, by moving it out of the supermodule tree entirely. In summary, from the supermodule's point of view: * A submodule with changed working directory is "dirty-wd" * A submodule with changed index is "dirty-idx" from the supermodule's * A submodule with changed HEAD (since the last supermodule commit) is "changed but not updated" and can hence be "update-index"ed into the supermodule * A submodule with changed HEAD that has been added to the supermodule index is "updated but not checked in" * A submodule with changed HEAD (since the last supermodule update-index) is both "changed but not updated" _and_ "updated but not checked in", just like any normal file. What's needed then: * A way of telling git to treat a particular directory as a submodule instead of a directory * git-status gets knowledge of how to check for "dirty" submodules * git-commit-tree learns about how to store "submodule" object types in trees. The submodule object type will be nothing more than the hash of the current HEAD commit. (This might be my ignorance, perhaps it's just update-index that needs to know this) I don't know enough about the plumbing to know if my description above is using the right nomenclature - I'm sure someone will correct me. In my head, it would look something like this: $ mkdir supermodule; cd supermodule $ git init-db $ git clone proto://host/submodule.git $ git add --submodule submodule $ git update-index submodule $ git commit -m "Added submodule to supermodule" [ edit submodule ] $ git status submodule is dirty, the working directory has changed [ update-index in submodule ] $ git status submodule is dirty, the index has changed [ commit in submodule ] $ git status submodule is changed but not updated $ git update-index submodule $ git status submodule is updated but not checked in $ git commit -m "Record submodule change in supermodule" Am I crazy? Andy -- Dr Andy Parkins, M Eng (hons), MIEE ``` ## Shawn Pearce, 2006-11-28 16:36 Subject: Re: [RFC] Submodules in GIT Message-ID: <20061128163651.GG28337@spearce.org> URL: https://gitlist.dev/e/20061128163651.GG28337%40spearce.org In-Reply-To: <200611281629.08636.andyparkins@gmail.com> ``` Andy Parkins wrote: > On Tuesday 2006 November 28 15:44, Shawn Pearce wrote: > > > So HEAD in a submodule is the current content of that submodule. > > Therefore any update-index call on a submodule should load HEAD > > (totally ignoring whatever branch it refers to) into the supermodule > > index. > > I was with you right up until here. > > Why should a submodule do anything to the supermodule? This is like saying, > when I edit a working tree file, it should automatically call update-index. > The supermodule index should only be updated in response to a manual > update-index (or commit -a I suppose). You misread my poorly written statement. :-) What I meant to say was that update-index run in the supermodule would load the submodule content into the supermodule index; much as an update-index on a file would load the content of that file into the index. > IMO, it should always be possible to take a submodule and work on it in > isolation - in an extreme case, by moving it out of the supermodule tree > entirely. Aside from sharing object directories, yes. > In summary, from the supermodule's point of view: > * A submodule with changed working directory is "dirty-wd" > * A submodule with changed index is "dirty-idx" from the supermodule's > * A submodule with changed HEAD (since the last supermodule commit) > is "changed but not updated" and can hence be "update-index"ed into the > supermodule > * A submodule with changed HEAD that has been added to the supermodule index > is "updated but not checked in" > * A submodule with changed HEAD (since the last supermodule update-index) is > both "changed but not updated" _and_ "updated but not checked in", just > like any normal file. > > What's needed then: > * A way of telling git to treat a particular directory as a submodule instead > of a directory > * git-status gets knowledge of how to check for "dirty" submodules > * git-commit-tree learns about how to store "submodule" object types in > trees. The submodule object type will be nothing more than the hash of the > current HEAD commit. (This might be my ignorance, perhaps it's just > update-index that needs to know this) Err, uhm, more like git-write-tree. git-commit-tree doesn't care about the tree content. And all of the tree reading code. And all object traversal code (e.g. rev-list --objects). Martin Waitz's submodule prototype has been working on those details. Its non-trivial due to the number of locations affected. > In my head, it would look something like this: > > $ mkdir supermodule; cd supermodule > $ git init-db > $ git clone proto://host/submodule.git > $ git add --submodule submodule > $ git update-index submodule > $ git commit -m "Added submodule to supermodule" > [ edit submodule ] > $ git status > submodule is dirty, the working directory has changed > [ update-index in submodule ] > $ git status > submodule is dirty, the index has changed > [ commit in submodule ] > $ git status > submodule is changed but not updated > $ git update-index submodule > $ git status > submodule is updated but not checked in > $ git commit -m "Record submodule change in supermodule" Yes, exactly my thoughts on the matter. > Am I crazy? Maybe, but I'm not a shrink. Your email looked sane. :-) -- ``` ## Jon Loeliger, 2006-11-28 17:38 Subject: Re: [RFC] Submodules in GIT Message-ID: <1164735520.4724.28.camel@cashmere.sps.mot.com> URL: https://gitlist.dev/e/1164735520.4724.28.camel%40cashmere.sps.mot.com In-Reply-To: <200611281629.08636.andyparkins@gmail.com> ``` On Tue, 2006-11-28 at 10:29, Andy Parkins wrote: > IMO, it should always be possible to take a submodule and work on it in > isolation - in an extreme case, by moving it out of the supermodule tree > entirely. This seems to me to be tantamount to saying something like: We need a "recursively defined git repository" that is representable as a git repository. That is, can the tree object be changed from containing just "blob" and "tree" references to also having a new "git" reference as well? jdl ``` ## Steven Grimm, 2006-11-28 19:58 Subject: Re: [RFC] Submodules in GIT Message-ID: <456C94E2.6010708@midwinter.com> URL: https://gitlist.dev/e/456C94E2.6010708%40midwinter.com In-Reply-To: <200611281335.38728.andyparkins@gmail.com> ``` Andy Parkins wrote: > Unfortunately, during development, you've switched libsubmodule1 to > branch "development", but supermodule isn't tracking libsubmodule1/HEAD it's > tracking libsubmodule1/master. Your supermodule commit doesn't capture a > snapshot of the tree you're using. > How about if the supermodule commit errors out by default if you commit a different submodule branch than the one you committed the previous time? Require the user to explicitly acknowledge that yes, they want to check in the contents of "development" now, even though the supermodule was tracking "master" before. Otherwise I think you could easily end up with just the opposite situation, where you forget you've checked out "development" for a moment to look at something, and end up inadvertently committing a bunch of stuff that's not ready for prime time yet. In a standalone git setting, that's no big deal since the commit only updates the current branch and doesn't touch the master branch, but (as I understand the proposal) in a supermodule setting you'd actually end up essentially doing a merge between your development branch and the previously committed master. Or maybe not a merge, but worse, you'd *replace* the previously committed master with what's in your dev branch. I think wanting to commit a submodule on a different branch than last time is probably not a typical day-to-day use case, so we should make sure the user really wants to do it (but allow it if so.) On a related note, it would be great from a usability point of view if there were a way to say "I always want to be on the same branch in all submodules and the supermodule." I think a common scenario will be that you are doing development that touches a couple of different applications and your development effort is really a single set of changes even though it happens to cross submodule boundaries. If this branches-in-sync option is turned on, I'd want "git checkout development" to check out the development branch in the entire set of repositories. More generally, while I 100% agree that it's very useful to be able to operate independently on each submodule, I think it's also going to be common to use submodules to selectively clone different pieces of a larger project. Say your current development effort needs server A, library B, and documentation C, and you want to have *just* those pieces in your environment. You don't particularly care about the details of how the system has assembled the pieces you want; you want to be able to make your changes and push them when you're done. They are really just pieces of a larger code base, not independent entities that happen to be pulled together into a composite workspace temporarily. For that use case, I don't want the system to act differently depending on whether server A and library B are in the same submodule or separate ones; I want to treat the supermodule as the repository, and the system should take care of the details of managing the submodules. When I do "git commit -a" I want it to give me one editor to write one commit comment that covers all of the changes I've made, and when I do "git checkout -b" I want a new branch to apply across all the files I'm working with. It is entirely possible that the above is a matter best left to the porcelain layer, and that's fine with me. But I think the Perforce-style "compose a single workspace out of different bits of a larger project" model is hugely useful and whatever submodule system Git ends up with, it should be able to emulate as much of that feature as possible. ``` ## Shawn Pearce, 2006-11-28 21:02 Subject: Re: [RFC] Submodules in GIT Message-ID: <20061128210211.GI28337@spearce.org> URL: https://gitlist.dev/e/20061128210211.GI28337%40spearce.org In-Reply-To: <456C94E2.6010708@midwinter.com> ``` Steven Grimm wrote: > Andy Parkins wrote: > >Unfortunately, during development, you've switched libsubmodule1 to > >branch "development", but supermodule isn't tracking libsubmodule1/HEAD > >it's tracking libsubmodule1/master. Your supermodule commit doesn't > >capture a snapshot of the tree you're using. > > > > Or maybe not a merge, but worse, you'd *replace* the > previously committed master with what's in your dev branch. Right, you would be replacing the prior branch of that submodule with the new submodule branch. I think the safety valve you are looking for here is two things: * don't automatically update the submodule's HEAD into the supermodule's index. * make sure the submodule's HEAD is a fast-forward of the supermodule's index, with a --force option to force it anyway. Otherwise the developer just has to know what he/she is doing. Today you can put stuff that isn't ready for prime-time into a repository on the wrong branch just by applying the wrong patch, or cherry-picking the wrong commit, etc... the user can (and will) make mistakes. But they can also easily recover from them by rewinding history and redoing it. > On a related note, it would be great from a usability point of view if > there were a way to say "I always want to be on the same branch in all > submodules and the supermodule." That's not really an issue. A branch doesn't exist just because you checked-out the branch, or because you created it. A branch exists because there were two or more commits (B and C) which use the same parent (A) and two or more of those commits survive, e.g. they have refs which point to them (directly or indirectly) or they were merged into another commit which itself survives. Therefore if the supermodule is on the "development branch" the submodules are also immediately on the same branch, because their HEADs are derived from whatever is stored in the supermodule's tree. And that tree is derived from whatever "development branch" means. Really what you want/need is a special head in the submodule which acts as the "branch that corresponds to the supermodule". This probably should just be a naked SHA1 stored in HEAD, which is committable only because a supermodule exists in a higher level directory. The fact that the submodule project has branches *at all* is totally irrelevant once you start to speak about that submodule within the supermodule, as its the supermodule which determines the branch of the submodule. > But I think the Perforce-style > "compose a single workspace out of different bits of a larger project" > model is hugely useful That's a mess. You start to get into weird cases where the directory structure expected by the build process is no longer intact, because the user has sliced it apart in weird ways. And there's no single version which corresponds to that workspace as (if I recall correctly) you can pick different tags or branches at will. I believe that ClearCase has the same bug. You also can't version that now spliced workspace, aside from taking the configuration file and putting that under version control too. However I think the proposal on the table will support that to some degree, in that you can take any version of any repository and embed it at any directory of any other repository. This means you can for example embed the Linux kernel, glibc and gcc projects into a larger "embedded device" repository, but you cannot alter the structure of any of those three projects without making your own locally developed branch of them. Which is actually the correct thing to do as any subslicing of a repository is exactly that: a locally developed branch of that repository. -- ``` ## Martin Waitz, 2006-11-29 16:03 Subject: Re: [RFC] Submodules in GIT Message-ID: <20061129160355.GF18810@admingilde.org> URL: https://gitlist.dev/e/20061129160355.GF18810%40admingilde.org In-Reply-To: <200611281335.38728.andyparkins@gmail.com> ``` hoi :) On Tue, Nov 28, 2006 at 01:35:37PM +0000, Andy Parkins wrote: > The reason I thought it would have to be HEAD at all times, is to prevent > situations where the supermodule commit doesn't reflect the state of the > current tree. The way I wanted to address this is to show in the supermodule git-status that the submodule is using another branch. That way you are warned and can decide not to commit the supermodule. I implemented tracking of refs/heads/master (not HEAD) without much thinking, and only recently began to think about possible problems with this approach. But I think it is an important design decision to take, so I'd like to have consensus here. Pro HEAD: - update-index on submodule really updates the supermodule index with a commit that resembles the working directory. Contra HEAD: - HEAD is not garanteed to be equal to the working directory anyway, you may have uncommitted changes. - when updating the supermodule, you have to take care that your submodules are on the right branch. You might for example have some testing-throwawy branch in one submodule and don't want to merge it with other changes yet. Pro refs/heads/master: - the supermodule really tracks one defined branch of development. - you can easily overwrite one submodule by changing to another branch, without fearing that changes in the supermodule change anything there. Contra refs/heads/master: - after updating the supermodule, you may not have the correct working directory checked out everywhere, because some submodules may be on a different branch. - there is one branch in the submodule which is special to all the other. I think that most of the disadvantages of refs/heads/master can be solved by printing the above-mentioned warning in git-status when the submodule is using another branch (similiar to the planned-but-not-implemented warn if the submodule has uncommited changes). I don't yet know how to cope with tracking HEAD directly, so I'm still in favor of tracking refs/heads/master, as already implemented. -- Martin Waitz ``` ## Martin Waitz, 2006-11-29 16:15 Subject: Re: [RFC] Submodules in GIT Message-ID: <20061129161543.GG18810@admingilde.org> URL: https://gitlist.dev/e/20061129161543.GG18810%40admingilde.org In-Reply-To: <200611281629.08636.andyparkins@gmail.com> ``` hoi :) On Tue, Nov 28, 2006 at 04:29:05PM +0000, Andy Parkins wrote: > In summary, from the supermodule's point of view: > * A submodule with changed working directory is "dirty-wd" > * A submodule with changed index is "dirty-idx" from the supermodule's > * A submodule with changed HEAD (since the last supermodule commit) > is "changed but not updated" and can hence be "update-index"ed into the > supermodule > * A submodule with changed HEAD that has been added to the supermodule index > is "updated but not checked in" > * A submodule with changed HEAD (since the last supermodule update-index) is > both "changed but not updated" _and_ "updated but not checked in", just > like any normal file. when tracking refs/heads/master instead of HEAD, you also get: * A submodule where HEAD is not pointing to refs/heads/master is "dirty-branch" or something. > What's needed then: > * A way of telling git to treat a particular directory as a submodule instead > of a directory This is handled by creating a GIT repository in this directory. My current implementation needs some more magic by the user to add it to the index, but I plan to change this to the way that GIT repositories will be recognized as possible submodules. > * git-status gets knowledge of how to check for "dirty" submodules This is on top of my TODO. > * git-commit-tree learns about how to store "submodule" object types in > trees. The submodule object type will be nothing more than the hash of the > current HEAD commit. (This might be my ignorance, perhaps it's just > update-index that needs to know this) it's only update-index that has to know this. Otherwise it would be implicitly updated and you would never get your "changed but not updated" status as above. -- Martin Waitz ``` ## Andy Parkins, 2006-11-29 20:00 Subject: Re: [RFC] Submodules in GIT Message-ID: <200611292000.23778.andyparkins@gmail.com> URL: https://gitlist.dev/e/200611292000.23778.andyparkins%40gmail.com In-Reply-To: <20061129160355.GF18810@admingilde.org> ``` On Wednesday 2006, November 29 16:03, Martin Waitz wrote: > The way I wanted to address this is to show in the supermodule > git-status that the submodule is using another branch. > That way you are warned and can decide not to commit the supermodule. The problem I see with tracking a particular branch is that it makes it less convenient to use git's quick-branching features in the submodules. Let's say I want to try something out quickly in a submodule, I make a branch, commit, commit, "hmm, looks good, let's snapshot it in the supermodule", make a supermodule branch, "oh no, I've got to tell the supermodule to track the new (but temporary) branch in the submodule do a commit, switch the submodule branch back to master, delete the temporary branch, remember that the supermodule is tracking that branch and tell the supermodule to track something else instead... It all seems too complicated to me. > Pro HEAD: > - update-index on submodule really updates the supermodule index with > a commit that resembles the working directory. Ouch. Why does the submodule need to update the supermodule index? That should be done by update-index in the supermodule. Further, how is the supermodule index going to represent working directory changes in the submodule? The only link between the two is a commit hash. It has to be like that otherwise you haven't made a supermodule-submodule, you've just made one super-repository. Also, if you don't store submodule commit hashes, then there is no way to guarantee that you're going to be able get back the state of the submodule again. > Contra HEAD: > - HEAD is not garanteed to be equal to the working directory anyway, > you may have uncommitted changes. That's the case for every file in a repository, so isn't really a worry. It's the equivalent of changing a file and not updating the index - who cares? As long as update-index tells you that the submodule is dirty and what to do to clean it, everything is great. > - when updating the supermodule, you have to take care that your > submodules are on the right branch. > You might for example have some testing-throwawy branch in one > submodule and don't want to merge it with other changes yet. What is the "right" branch though? As I said above, if you're tracking one branch in the submodule then you've effectively locked that submodule to that branch for all supermodule uses. Or you've made yourself a big rod to beat yourself with everytime you want to do some development on an "off" branch on the submodule. > Pro refs/heads/master: > - the supermodule really tracks one defined branch of development. Why is this a pro? > - you can easily overwrite one submodule by changing to another branch, > without fearing that changes in the supermodule change anything > there. You can always do that anyway by simply not running update-index for the submodule in the supermodule. > Contra refs/heads/master: > - after updating the supermodule, you may not have the correct working > directory checked out everywhere, because some submodules may be on a > different branch. This seems like the biggest problem to me - doesn't this negate all the advantages of a submodule system? After a check in, you have no idea if what you checked in was what was in your working tree. Andy -- Dr Andrew Parkins, M Eng (Hons), AMIEE ``` ## sf, 2006-11-30 11:57 Subject: Re: [RFC] Submodules in GIT Message-ID: <456EC738.6000103@b-i-t.de> URL: https://gitlist.dev/e/456EC738.6000103%40b-i-t.de In-Reply-To: <200611281629.08636.andyparkins@gmail.com> ``` Andy Parkins wrote: ... > Worse, if you allow that to happen, the supermodule can commit a state that > cannot be retrieved from the submodule's repository. The ONLY thing a > supermodule can record about a submodule is a commit. So what? You have a submodule commit that only exists in the supermodule. I fail to see the problem. The changes you made to the submodule _in the supermodule_ can later be pulled from wherever you want. Regards Stephan ``` ## Andreas Ericsson, 2006-11-30 12:16 Subject: Re: [RFC] Submodules in GIT Message-ID: <456ECBA5.7010409@op5.se> URL: https://gitlist.dev/e/456ECBA5.7010409%40op5.se In-Reply-To: <200611292000.23778.andyparkins@gmail.com> ``` Andy Parkins wrote: > On Wednesday 2006, November 29 16:03, Martin Waitz wrote: > >> - when updating the supermodule, you have to take care that your >> submodules are on the right branch. >> You might for example have some testing-throwawy branch in one >> submodule and don't want to merge it with other changes yet. > > What is the "right" branch though? As I said above, if you're tracking one > branch in the submodule then you've effectively locked that submodule to that > branch for all supermodule uses. Or you've made yourself a big rod to beat > yourself with everytime you want to do some development on an "off" branch on > the submodule. > Perhaps I'm just daft, but I fail to see how you can safely track a topic-branch that might get rewinded or rebased in the submodule without crippling the supermodule. Wasn't the intention that the supermodule has a new tree object (called "submodule") that points to a commit in the submodule from where it gets its tree and stuff? Is the intention that the supermodule pulls all of the submodules history into its own ODB? If so, what's the difference between just having one large repository. If not, how can you make it not break in case the commit it references in the submodule is pruned away? One possible way would ofcourse be to add something like this to the supermodule commit: submodule directory/commit-sha1 tree submodule-tree-sha1 but then you're in trouble because the supermodule will have the same files as all the submodules stored in its own tree. I'm confused. Could someone shed some light on how this sub-/super-module connection is supposed to work in the supermodule's commit objects? -- Andreas Ericsson andreas.ericsson@op5.se OP5 AB www.op5.se ``` ## Andy Parkins, 2006-11-30 12:40 Subject: Re: [RFC] Submodules in GIT Message-ID: <200611301240.22938.andyparkins@gmail.com> URL: https://gitlist.dev/e/200611301240.22938.andyparkins%40gmail.com In-Reply-To: <456ECBA5.7010409@op5.se> ``` On Thursday 2006 November 30 12:16, Andreas Ericsson wrote: > > What is the "right" branch though? As I said above, if you're tracking > > one branch in the submodule then you've effectively locked that submodule > > to that branch for all supermodule uses. Or you've made yourself a big > > rod to beat yourself with everytime you want to do some development on an > > "off" branch on the submodule. > > Perhaps I'm just daft, but I fail to see how you can safely track a > topic-branch that might get rewinded or rebased in the submodule without > crippling the supermodule. Wasn't the intention that the supermodule has Who said anything but rebase/rewind? As it happens though, I don't see why you can't (it wouldn't be pleasant though). A rebase or rewind still leaves the original commit in the object database, so provided no one runs git-prune, there is no catastrophic failure. > a new tree object (called "submodule") that points to a commit in the > submodule from where it gets its tree and stuff? Is the intention that > the supermodule pulls all of the submodules history into its own ODB? If > so, what's the difference between just having one large repository. If > not, how can you make it not break in case the commit it references in > the submodule is pruned away? I certainly never suggested anything /but/ storing a submodule type that stores the commit. The current debate is about whether the supermodule should track HEAD or some defined branch in the submodule. > but then you're in trouble because the supermodule will have the same > files as all the submodules stored in its own tree. I'm confused. Could > someone shed some light on how this sub-/super-module connection is > supposed to work in the supermodule's commit objects? I don't really know, I only joined in to stand up against commit in the supermodule triggering commits in the submodule. That lead to me trying to get an understanding of how it would work. As far as I can see, the only way a submodule is any use is if it is always a submodule-commit-hash that is noted in the supermodule tree object. That means that the supermodule will only commit clean submodules. The rest is just UI to show something useful in the difficult cases when the submodule tree is dirty. Andy -- Dr Andy Parkins, M Eng (hons), MIEE ``` ## Martin Waitz, 2006-11-30 17:06 Subject: Re: [RFC] Submodules in GIT Message-ID: <20061130170625.GH18810@admingilde.org> URL: https://gitlist.dev/e/20061130170625.GH18810%40admingilde.org In-Reply-To: <200611292000.23778.andyparkins@gmail.com> ``` hoi :) On Wed, Nov 29, 2006 at 08:00:22PM +0000, Andy Parkins wrote: > On Wednesday 2006, November 29 16:03, Martin Waitz wrote: > > > The way I wanted to address this is to show in the supermodule > > git-status that the submodule is using another branch. > > That way you are warned and can decide not to commit the supermodule. > > The problem I see with tracking a particular branch is that it makes it less > convenient to use git's quick-branching features in the submodules. Let's > say I want to try something out quickly in a submodule, I make a branch, > commit, commit, "hmm, looks good, let's snapshot it in the supermodule", make > a supermodule branch, "oh no, I've got to tell the supermodule to track the > new (but temporary) branch in the submodule do a commit, switch the submodule > branch back to master, delete the temporary branch, remember that the > supermodule is tracking that branch and tell the supermodule to track > something else instead... It all seems too complicated to me. What about: You decide to try something out quickly and create a new branch in the submodule. After you have verified that it works, you merge it to the submodules master branch and commit that to the supermodule. Not that complicated, isn't it? In fact, my current implementation does not even allow to change the branch name of the submodule which is tracked by the supermodule ;-). > > Pro HEAD: > > - update-index on submodule really updates the supermodule index with > > a commit that resembles the working directory. > > Ouch. Why does the submodule need to update the supermodule index? Please excuse that I am not an native english speaker and I may have caused some confusion here. > That should be done by update-index in the supermodule. That is exactly what I wanted to say. In the supermoduel you call update-index (with the submodule path as argument) to update the index of the supermodule. Just like normal files. Nothing new. > Further, how is the supermodule index going to represent working > directory changes in the submodule? The only link between the two is > a commit hash. It has to be like that otherwise you haven't made a > supermodule-submodule, you've just made one super-repository. Also, > if you don't store submodule commit hashes, then there is no way to > guarantee that you're going to be able get back the state of the > submodule again. This is handled in the next paragraph. The argument really is: HEAD always points to the checked out branch, so it really has a relationship to the working directory. > > Contra HEAD: > > - HEAD is not garanteed to be equal to the working directory anyway, > > you may have uncommitted changes. > > That's the case for every file in a repository, so isn't really a > worry. It's the equivalent of changing a file and not updating the > index - who cares? As long as update-index tells you that the > submodule is dirty and what to do to clean it, everything is great. Yes, it's not a real counter-argument, but it relativates the previous pro-argument. > > - when updating the supermodule, you have to take care that your > > submodules are on the right branch. > > You might for example have some testing-throwawy branch in one > > submodule and don't want to merge it with other changes yet. > > What is the "right" branch though? As I said above, if you're tracking one > branch in the submodule then you've effectively locked that submodule to that > branch for all supermodule uses. yes, but luckily GIT branches are very flexible. > Or you've made yourself a big rod to beat yourself with everytime you > want to do some development on an "off" branch on the submodule. I don't think it is that bad. > > Pro refs/heads/master: > > - the supermodule really tracks one defined branch of development. > > Why is this a pro? You always know which branch in the submodule is the "upstream" branch which is managed by the supermodule. You can easily have several topic-branches and merge updates from the master branch. otherwise you always have to remember which branch holds your current contents from the supermodule. When viewed from the supermodule, you are storing one branch per submodule in your tree. > > - you can easily overwrite one submodule by changing to another branch, > > without fearing that changes in the supermodule change anything > > there. > > You can always do that anyway by simply not running update-index for the > submodule in the supermodule. Suppose you are working on a complicated feature in one submodule. You create your own branch for that feature and work on it. Now you want to update your project, so you pull a new supermodule version. Now this pull also included one (for you unimportant) change in the submodule. I think it is more clear to update the master branch with the new version coming from the supermodule, while leaving your work intact (you haven't commited it to the supermodule yet, so the supermodule should not care about your changes, it's just some dirty tree). Then you can freely merge between your branch and master as you like and are not forced to merge at once. And perhaps you even do not want to merge at all, because you are on an experimental branch which really is mutually exclusive with the current supermodule contents. > > Contra refs/heads/master: > > - after updating the supermodule, you may not have the correct working > > directory checked out everywhere, because some submodules may be on a > > different branch. > > This seems like the biggest problem to me - doesn't this negate all the > advantages of a submodule system? After a check in, you have no idea if what > you checked in was what was in your working tree. Of course you know: git-status will tell it. This is no different to today, where you can commit while still leaving a part of the tree dirty. -- Martin Waitz ``` ## Andreas Ericsson, 2006-11-30 18:57 Subject: Re: [RFC] Submodules in GIT Message-ID: <456F29A2.1050205@op5.se> URL: https://gitlist.dev/e/456F29A2.1050205%40op5.se In-Reply-To: <20061130170625.GH18810@admingilde.org> ``` Martin Waitz wrote: > hoi :) > > On Wed, Nov 29, 2006 at 08:00:22PM +0000, Andy Parkins wrote: >> On Wednesday 2006, November 29 16:03, Martin Waitz wrote: >> >> Further, how is the supermodule index going to represent working >> directory changes in the submodule? The only link between the two is >> a commit hash. It has to be like that otherwise you haven't made a >> supermodule-submodule, you've just made one super-repository. Also, >> if you don't store submodule commit hashes, then there is no way to >> guarantee that you're going to be able get back the state of the >> submodule again. > > This is handled in the next paragraph. > The argument really is: HEAD always points to the checked out branch, > so it really has a relationship to the working directory. > >>> Contra HEAD: >>> - HEAD is not garanteed to be equal to the working directory anyway, >>> you may have uncommitted changes. >> That's the case for every file in a repository, so isn't really a >> worry. It's the equivalent of changing a file and not updating the >> index - who cares? As long as update-index tells you that the >> submodule is dirty and what to do to clean it, everything is great. > > Yes, it's not a real counter-argument, but it relativates the previous > pro-argument. > >>> - when updating the supermodule, you have to take care that your >>> submodules are on the right branch. >>> You might for example have some testing-throwawy branch in one >>> submodule and don't want to merge it with other changes yet. >> What is the "right" branch though? As I said above, if you're tracking one >> branch in the submodule then you've effectively locked that submodule to that >> branch for all supermodule uses. > > yes, but luckily GIT branches are very flexible. > There's no real technical reason for locking it to a single branch though, and in case of a fork in the upstream submodule project, you might suddenly decide that "the other team" is heading in a much more interesting direction and you want to use their work in your module instead. Will you now have to maintain a separate branch just to keep the same name as the branch the original team used? >> Or you've made yourself a big rod to beat yourself with everytime you >> want to do some development on an "off" branch on the submodule. > > I don't think it is that bad. > It could be, and as has already been stated, there's no real reason to limit this to a particular branch, so I don't see why we would want to impose such non-real restrictions. >>> Pro refs/heads/master: >>> - the supermodule really tracks one defined branch of development. >> Why is this a pro? > > You always know which branch in the submodule is the "upstream" branch > which is managed by the supermodule. No you don't. The branch-name might be moved to some other tip of the DAG, and that's exactly the same as changing the branch you're tracking. > You can easily have several topic-branches and merge updates from the > master branch. > otherwise you always have to remember which branch holds your current > contents from the supermodule. > No you don't. The only thing you need is the commit-sha. > When viewed from the supermodule, you are storing one branch per > submodule in your tree. > Wrong again. You're storing one particular point in the revision history. >>> - you can easily overwrite one submodule by changing to another branch, >>> without fearing that changes in the supermodule change anything >>> there. >> You can always do that anyway by simply not running update-index for the >> submodule in the supermodule. > > Suppose you are working on a complicated feature in one submodule. > You create your own branch for that feature and work on it. > Now you want to update your project, so you pull a new supermodule > version. Now this pull also included one (for you unimportant) change > in the submodule. git reset to the rescue. > I think it is more clear to update the master branch with the new > version coming from the supermodule, while leaving your work intact > (you haven't commited it to the supermodule yet, so the supermodule > should not care about your changes, it's just some dirty tree). > Then you can freely merge between your branch and master as you like and > are not forced to merge at once. And perhaps you even do not want to > merge at all, because you are on an experimental branch which really is > mutually exclusive with the current supermodule contents. > This is all just policy though. Tools that enforce a certain policy are not good tools. The only problem I'm seeing atm is that the supermodule somehow has to mark whatever commits it's using from the submodule inside the submodule repo so that they effectively become un-prunable, otherwise the supermodule may some day find itself with a history that it can't restore. The really major problem with this is that now you'll have one repository of the submodule that is actually special, so it's not certain you can go and use any repository at all of the submodule code, since the upstream repo most likely won't be all that interested in having all of that meta-data inside it. In reality, I'm sure this will be a small problem though, as submodules that are in reality projects which the supermodule's maintainer isn't the owner of will most likely never rewind their history beyond the supermodules stored commit. It's something fsck will have to be taught to watch for though. -- Andreas Ericsson andreas.ericsson@op5.se OP5 AB www.op5.se Tel: +46 8-230225 Fax: +46 8-230231 ``` ## Andy Parkins, 2006-12-01 09:02 Subject: Re: [RFC] Submodules in GIT Message-ID: <200612010902.51264.andyparkins@gmail.com> URL: https://gitlist.dev/e/200612010902.51264.andyparkins%40gmail.com In-Reply-To: <20061130170625.GH18810@admingilde.org> ``` On Thursday 2006 November 30 17:06, Martin Waitz wrote: > You can easily have several topic-branches and merge updates from the > master branch. > otherwise you always have to remember which branch holds your current > contents from the supermodule. WHAT? I've got to make merges (that I don't necessarily want) in order to commit in the supermodule? This completely negates any useful functioning of branches in the submodule. I want to be able to make a quick development branch in the submodule and NOT merge that code into master and then be able to still commit that in the supermodule. I think you're imagining the binding between the super and sub is very much tighter than it should be. What if I'm working on a development version of the supermodule, which includes a stable version of the submodule? Vice versa? > When viewed from the supermodule, you are storing one branch per > submodule in your tree. That prevents me "trying something out" on a topic branch in the submodule. Here's a scenario using my suggested "supermodule tracks submodule HEAD" method. * You're developerA * Make a development branch in the supermodule * In the submodule, make a whole load of topic branches * Make a development branch in the submodule * Merge the topic branches into the development branch of the submodule * Commit in the supermodule. This capture * Tag that commit "my-tested-arrangement-of-submodule-features" * Push that tag to the central repository - tell the world. * DeveloperB checks out that tag and tries it. Great stuff. Now: here's the secret fact that I didn't tell you that will break your "supermodule tracks submodule branch" method. DeveloperB has decided to have this in his remote: Pull: refs/heads/master:refs/heads/upstream/master Oops. The supermodule, which has been told to track the "master" branch in the submodule is tracking different things in developerA's repository from developerB's repository. Worse, what if developerB did this: Pull: refs/heads/master:refs/heads/development Pull: refs/heads/development:refs/heads/master Branches are completely arbitrary per-repository. You cannot rely on them being consistent between different repositories. If you store the name of a submodule branch in a supermodule - that supermodule is only valid for that one special case of your particular version of the submodule. Andy -- Dr Andy Parkins, M Eng (hons), MIEE ``` ## Martin Waitz, 2006-12-01 11:00 Subject: Re: [RFC] Submodules in GIT Message-ID: <20061201110032.GL18810@admingilde.org> URL: https://gitlist.dev/e/20061201110032.GL18810%40admingilde.org In-Reply-To: <200612010902.51264.andyparkins@gmail.com> ``` hoi :) On Fri, Dec 01, 2006 at 09:02:48AM +0000, Andy Parkins wrote: > On Thursday 2006 November 30 17:06, Martin Waitz wrote: > > > You can easily have several topic-branches and merge updates from the > > master branch. > > otherwise you always have to remember which branch holds your current > > contents from the supermodule. > > WHAT? I've got to make merges (that I don't necessarily want) in > order to commit in the supermodule? This completely negates any > useful functioning of branches in the submodule. I want to be able to > make a quick development branch in the submodule and NOT merge that > code into master and then be able to still commit that in the > supermodule. exactly! Please think about it. If you track HEAD, then this means that you track HEAD. In _both_ directions! So you not only store your submodule HEAD commit in the supermodule when you do commit to the supermodule, it also means that your submodule HEAD will be updated when you update your supermodule. And what happens if you already commited something to HEAD in the mean time? Exactly: a merge is needed. And you are right: you might not want to do this now, because you branched off, because you _wanted_ to have some development which is _independent_ to the current supermodule work. So tracking HEAD really makes branching in the submodule hard to work with. What does the supermodule provide to the submodule? It stores one reference to a commit sha1. Just like a reference inside refs/heads inside the submodule. There really is not much difference between the sha1 stored inside the supermodules tree and one stored inside refs/. So from the submodules point of view, the supermodule is not much more then one special branch. But it is not possible to use the supermodule index directly as one "magic" branch for several reasons. So we need synchronization methods between the index entry for the submodule which is stored in the supermodule and the references in the submodule. These are git-update-index/git-commit and git-checkout, both called explicitly or implicitly in the supermodule. And I really think it makes sense to have a one-to-one relationship between the submodule "branch" stored in the supermodule and the branchname used in the submodule. > I think you're imagining the binding between the super and sub is very much > tighter than it should be. What if I'm working on a development version of > the supermodule, which includes a stable version of the submodule? Vice > versa? I don't see your problem here. > > When viewed from the supermodule, you are storing one branch per > > submodule in your tree. > > That prevents me "trying something out" on a topic branch in the submodule. > Here's a scenario using my suggested "supermodule tracks submodule HEAD" > method. > > * You're developerA > * Make a development branch in the supermodule > * In the submodule, make a whole load of topic branches > * Make a development branch in the submodule > * Merge the topic branches into the development branch of the submodule > * Commit in the supermodule. This capture > * Tag that commit "my-tested-arrangement-of-submodule-features" > * Push that tag to the central repository - tell the world. > * DeveloperB checks out that tag and tries it. Great stuff. This is still supposed to be a distributed system. DeveloperB does not only check out the whole project including several modules. He is also supposed to _work_ with it. What if DeveloperB also has several topic branches? When he checks out the new supermodule, only his current HEAD in the submodule will be updated. So he first has to change to some supermodule-tracking branch inside the submodule, then pull the supermodule updates, then eventually merge the new contents of his supermodule-tracking branch into his topic branches. So why not make this "let's update one supermodule-tracking-branch" automatic? > Now: here's the secret fact that I didn't tell you that will break > your "supermodule tracks submodule branch" method. DeveloperB has > decided to have this in his remote: > Pull: refs/heads/master:refs/heads/upstream/master > Oops. The supermodule, which has been told to track the "master" > branch in the submodule is tracking different things in developerA's > repository from developerB's repository. So what? He can do to the repository whatever he wants? He wants to change one submodule to a different branch? He can do so! But please do not expect the system to magically be able to resolve problems. If you _by intent_ changed the submodule to another branch which is incompatible to the one used in the submodule you can't expect that this is magically merged. This is the same as with normal files. Sure you can replace one file with new contents that are different to the one used by someone else. Don't expect this can be merged automatically. So now you have two forks/branches of the project. So what? Same for a system including submodules: If you change one submodule to a totally different branch, then you effectivley forked/branched the entire project. (Nomenclature is a bit difficult here: what I mean by totally different branch is: the submodule commit tracked by the supermodule is not directly connected to the one tracked by an old version of the supermodule). So whenever you introduce conflicting changes somewhere in the project (be it in a submodule or in a file) you _always_ fork/branch the entire project (i.e. the topmost supermodule). You can't circumvent that. So what are submodule branches good for then? To store other lines of development which are not yet / not any more tracked by the supermodule. Perhaps you store references to branches stored in another supermodule, or another standalone repository. Or a temporary branch which is only used for testing. There are really many possiblilities. But they all have one thing in common: they are not meant to be tracked by the supermodule. > Branches are completely arbitrary per-repository. Yes, but a submodule is special here: it really has one special branch. The module is not independent any more. That is the _nature_ of a submodule. > You cannot rely on them being consistent between different > repositories. Sure, we are in a distributed system. But the supermodule always has to know which branch in the submodule has to be tracked. The easiest thing is to always use the default refs/heads/master. Surely this could be changed if there is a need. -- Martin Waitz ``` ## sf, 2006-12-01 12:03 Subject: Re: [RFC] Submodules in GIT Message-ID: <45701A24.5060500@b-i-t.de> URL: https://gitlist.dev/e/45701A24.5060500%40b-i-t.de In-Reply-To: <456F29A2.1050205@op5.se> ``` Andreas Ericsson wrote: ... > The only problem I'm seeing atm is that the supermodule somehow has to > mark whatever commits it's using from the submodule inside the submodule > repo so that they effectively become un-prunable, otherwise the > supermodule may some day find itself with a history that it can't restore. That has nothing to do with submodules. What you state here is the problem of alternate repositories. There are two solutions: 1. Do not use alternates. 2. Do not prune a repository that is used as an alternate repository by other repositories. For the submodule discussion that would mean: 1. Only fetch and work on branches of submodules you are interested in. It does not matter that the origin repository contains (probably orders of magnitude) more data. You will never touch that. 2. You can never prune the main (the supermodule's) repository, at least not with what git provides today. That is why the sanest approach to subprojects is to put commits into tree objects, define a way to name these commits and make git understand these new commit names. Done. Works. Regards Stephan ``` ## sf, 2006-12-01 12:09 Subject: Re: [RFC] Submodules in GIT Message-ID: <45701B8D.1030508@b-i-t.de> URL: https://gitlist.dev/e/45701B8D.1030508%40b-i-t.de In-Reply-To: <20061201110032.GL18810@admingilde.org> ``` Martin Waitz wrote: ... > So you not only store your submodule HEAD commit in the supermodule when you > do commit to the supermodule, it also means that your submodule HEAD > will be updated when you update your supermodule. Why the magic? The typical workflow in git is 1. You work on a branch, i.e. edit and commit and so on. 2. At some point, you decide to share the work you did on that branch (e-mail a patch, merge into another branch, push upstream or let it by pulled by upstream) I fail to understand why these two steps have to be mixed up. Someone care to explain? Regards Stephan ``` ## Martin Waitz, 2006-12-01 12:11 Subject: Re: [RFC] Submodules in GIT Message-ID: <20061201121110.GP18810@admingilde.org> URL: https://gitlist.dev/e/20061201121110.GP18810%40admingilde.org In-Reply-To: <45701A24.5060500@b-i-t.de> ``` hoi :) On Fri, Dec 01, 2006 at 01:03:48PM +0100, sf wrote: > Andreas Ericsson wrote: > 2. You can never prune the main (the supermodule's) repository, at least > not with what git provides today. It even already works (well, not with what git provides today, but with my implementation). git-prune simply walks all the submodules, too, when doing it's reachability analysis. What does not work is a prune inside the submodule, because it does not know about all the commits used by the supermodule. -- Martin Waitz ``` ## Martin Waitz, 2006-12-01 12:12 Subject: Re: [RFC] Submodules in GIT Message-ID: <20061201121234.GQ18810@admingilde.org> URL: https://gitlist.dev/e/20061201121234.GQ18810%40admingilde.org In-Reply-To: <45701B8D.1030508@b-i-t.de> ``` hoi :) On Fri, Dec 01, 2006 at 01:09:49PM +0100, sf wrote: > Martin Waitz wrote: > ... > >So you not only store your submodule HEAD commit in the supermodule when > >you > >do commit to the supermodule, it also means that your submodule HEAD > >will be updated when you update your supermodule. > > Why the magic? The typical workflow in git is > > 1. You work on a branch, i.e. edit and commit and so on. > 2. At some point, you decide to share the work you did on that branch > (e-mail a patch, merge into another branch, push upstream or let it by > pulled by upstream) 3. Other people want to use your new work. -- Martin Waitz ``` ## sf, 2006-12-01 13:05 Subject: Re: [RFC] Submodules in GIT Message-ID: <4570289D.9050802@b-i-t.de> URL: https://gitlist.dev/e/4570289D.9050802%40b-i-t.de In-Reply-To: <20061201121234.GQ18810@admingilde.org> ``` Martin Waitz wrote: > hoi :) > > On Fri, Dec 01, 2006 at 01:09:49PM +0100, sf wrote: >> Martin Waitz wrote: >> ... >> >So you not only store your submodule HEAD commit in the supermodule when >> >you >> >do commit to the supermodule, it also means that your submodule HEAD >> >will be updated when you update your supermodule. >> >> Why the magic? The typical workflow in git is >> >> 1. You work on a branch, i.e. edit and commit and so on. >> 2. At some point, you decide to share the work you did on that branch >> (e-mail a patch, merge into another branch, push upstream or let it by >> pulled by upstream) > > 3. Other people want to use your new work. Sorry, if that was not obvious: You actually procceed with one of the options I listed in Step 2. What I wanted to state is that with git you do not mix up committing (which is local to your repository and your branch) and publishing. Regards Stephan ``` ## sf, 2006-12-01 13:21 Subject: Re: [RFC] Submodules in GIT Message-ID: <45702C50.9050307@b-i-t.de> URL: https://gitlist.dev/e/45702C50.9050307%40b-i-t.de In-Reply-To: <20061201121110.GP18810@admingilde.org> ``` Martin Waitz wrote: > hoi :) > > On Fri, Dec 01, 2006 at 01:03:48PM +0100, sf wrote: >> Andreas Ericsson wrote: >> 2. You can never prune the main (the supermodule's) repository, at least >> not with what git provides today. > > It even already works (well, not with what git provides today, but with > my implementation). git-prune simply walks all the submodules, too, when > doing it's reachability analysis. > > What does not work is a prune inside the submodule, because it does not > know about all the commits used by the supermodule. I just had a short (really short) look at your work. My impression is that your repository setup is much too complicated. As I proposed elsewhere: For submodules to work you only need to allow commits in tree objects (that is what your implementation requires as well). Everything else is in the tools. Much simpler. Regards Stephan ``` ## Martin Waitz, 2006-12-01 13:35 Subject: Re: [RFC] Submodules in GIT Message-ID: <20061201133558.GU18810@admingilde.org> URL: https://gitlist.dev/e/20061201133558.GU18810%40admingilde.org In-Reply-To: <4570289D.9050802@b-i-t.de> ``` hoi :) On Fri, Dec 01, 2006 at 02:05:33PM +0100, sf wrote: > >On Fri, Dec 01, 2006 at 01:09:49PM +0100, sf wrote: > >>Martin Waitz wrote: > >>>So you not only store your submodule HEAD commit in the supermodule > >>>when you do commit to the supermodule, it also means that your > >>>submodule HEAD will be updated when you update your supermodule. > >> > >>Why the magic? The typical workflow in git is > >> > >>1. You work on a branch, i.e. edit and commit and so on. > >>2. At some point, you decide to share the work you did on that branch > >>(e-mail a patch, merge into another branch, push upstream or let it by > >>pulled by upstream) > > > >3. Other people want to use your new work. > > Sorry, if that was not obvious: You actually procceed with one of the > options I listed in Step 2. What I wanted to state is that with git you > do not mix up committing (which is local to your repository and your > branch) and publishing. I guess you are refering to not mix up committing to the submodule and updating the supermodule index. These are really two separate steps, I just combined them above because I wanted to put emphasis on the other part: it is not a one-way flow, it is bidirectional, so your HEAD would have to changed if the supermodule gets updated. And I consider changing HEAD, without looking at the branch it points to, to be a bad thing. -- Martin Waitz ``` ## Martin Waitz, 2006-12-01 13:43 Subject: Re: [RFC] Submodules in GIT Message-ID: <20061201134311.GV18810@admingilde.org> URL: https://gitlist.dev/e/20061201134311.GV18810%40admingilde.org In-Reply-To: <45702C50.9050307@b-i-t.de> ``` hoi :) On Fri, Dec 01, 2006 at 02:21:20PM +0100, sf wrote: > I just had a short (really short) look at your work. My impression is > that your repository setup is much too complicated. Well, I'm not really satisfied with the UI part. What exactly do you find complicated? > As I proposed elsewhere: For submodules to work you only need to allow > commits in tree objects (that is what your implementation requires as > well). Everything else is in the tools. Much simpler. I do not quite get your point. The core of my work allows to put commits into tree objects. Then there is some more (but not quite finished) work to make the tools work together with submodules. So no, not everything is there yet. -- Martin Waitz ``` ## Andreas Ericsson, 2006-12-01 13:43 Subject: Re: [RFC] Submodules in GIT Message-ID: <45703174.8000609@op5.se> URL: https://gitlist.dev/e/45703174.8000609%40op5.se In-Reply-To: <20061201133558.GU18810@admingilde.org> ``` Martin Waitz wrote: > hoi :) > > On Fri, Dec 01, 2006 at 02:05:33PM +0100, sf wrote: >>> On Fri, Dec 01, 2006 at 01:09:49PM +0100, sf wrote: >>>> Martin Waitz wrote: >>>>> So you not only store your submodule HEAD commit in the supermodule >>>>> when you do commit to the supermodule, it also means that your >>>>> submodule HEAD will be updated when you update your supermodule. >>>> Why the magic? The typical workflow in git is >>>> >>>> 1. You work on a branch, i.e. edit and commit and so on. >>>> 2. At some point, you decide to share the work you did on that branch >>>> (e-mail a patch, merge into another branch, push upstream or let it by >>>> pulled by upstream) >>> 3. Other people want to use your new work. >> Sorry, if that was not obvious: You actually procceed with one of the >> options I listed in Step 2. What I wanted to state is that with git you >> do not mix up committing (which is local to your repository and your >> branch) and publishing. > > I guess you are refering to not mix up committing to the submodule and > updating the supermodule index. > These are really two separate steps, I just combined them above because I > wanted to put emphasis on the other part: it is not a one-way flow, it > is bidirectional, so your HEAD would have to changed if the supermodule > gets updated. > And I consider changing HEAD, without looking at the branch it points > to, to be a bad thing. > So a commit in the supermodule turns into a commit in the submodule? That's just plain wrong. If it doesn't, why would the submodule HEAD have to change? -- Andreas Ericsson andreas.ericsson@op5.se OP5 AB www.op5.se ``` ## Martin Waitz, 2006-12-01 13:46 Subject: Re: [RFC] Submodules in GIT Message-ID: <20061201134610.GW18810@admingilde.org> URL: https://gitlist.dev/e/20061201134610.GW18810%40admingilde.org In-Reply-To: <45703174.8000609@op5.se> ``` hoi :) On Fri, Dec 01, 2006 at 02:43:16PM +0100, Andreas Ericsson wrote: > So a commit in the supermodule turns into a commit in the submodule? no. > If it doesn't, why would the submodule HEAD have to change? So how do you update your submodule? Remember: if you git-pull in the supermodule, you want to update the whole thing, including all submodules. -- Martin Waitz ``` ## Stephan Feder, 2006-12-01 13:51 Subject: Re: [RFC] Submodules in GIT Message-ID: <45703375.4050500@b-i-t.de> URL: https://gitlist.dev/e/45703375.4050500%40b-i-t.de In-Reply-To: <20061201133558.GU18810@admingilde.org> ``` Martin Waitz wrote: > hoi :) > > On Fri, Dec 01, 2006 at 02:05:33PM +0100, sf wrote: >> >On Fri, Dec 01, 2006 at 01:09:49PM +0100, sf wrote: >> >>Martin Waitz wrote: >> >>>So you not only store your submodule HEAD commit in the supermodule >> >>>when you do commit to the supermodule, it also means that your >> >>>submodule HEAD will be updated when you update your supermodule. >> >> >> >>Why the magic? The typical workflow in git is >> >> >> >>1. You work on a branch, i.e. edit and commit and so on. >> >>2. At some point, you decide to share the work you did on that branch >> >>(e-mail a patch, merge into another branch, push upstream or let it by >> >>pulled by upstream) >> > >> >3. Other people want to use your new work. >> >> Sorry, if that was not obvious: You actually procceed with one of the >> options I listed in Step 2. What I wanted to state is that with git you >> do not mix up committing (which is local to your repository and your >> branch) and publishing. > > I guess you are refering to not mix up committing to the submodule and > updating the supermodule index. The opposite: If you work in the supermodule, even if it is in the code of the submodule, you only commit to the supermodule. The submodule does not "know" about these changes after step 1. > These are really two separate steps, I just combined them above because I > wanted to put emphasis on the other part: it is not a one-way flow, it > is bidirectional, so your HEAD would have to changed if the supermodule > gets updated. Why do you mix up supermodule and submodule? The way I see your proposal you cannot change submodule and supermodule independently. That is a huge drawback. Regards ``` ## Stephan Feder, 2006-12-01 14:23 Subject: Re: [RFC] Submodules in GIT Message-ID: <45703ACB.6050007@b-i-t.de> URL: https://gitlist.dev/e/45703ACB.6050007%40b-i-t.de In-Reply-To: <20061201134311.GV18810@admingilde.org> ``` Martin Waitz wrote: > hoi :) > > On Fri, Dec 01, 2006 at 02:21:20PM +0100, sf wrote: >> I just had a short (really short) look at your work. My impression is >> that your repository setup is much too complicated. > > Well, I'm not really satisfied with the UI part. > What exactly do you find complicated? I am not talking about the UI. On the contrary, I am talking about the repository, i.e. the directories and files on your disk and their contents. You add a lot of additional information to the repository that is not needed at all. >> As I proposed elsewhere: For submodules to work you only need to allow >> commits in tree objects (that is what your implementation requires as >> well). Everything else is in the tools. Much simpler. > > I do not quite get your point. > The core of my work allows to put commits into tree objects. That is fine. > Then there is some more (but not quite finished) work to make the tools > work together with submodules. So no, not everything is there yet. And what is already there is a lot of meta information (see above). You do not need that. For example, in the index, if it is a commit (i.e. a subproject), store the commit id (not the commit's tree id ). Make the tools handle this case (as yet, all code expects only trees and blobs when they parse trees). Especially, extend update-index to be able to store a commit instead of the tree. Or else, do not change what is recorded in the index. Then, at commit time, you not only commit the superproject but also all subprojects. Or allow both. Anyway, you can create commits in tree objects. See, you did not need to store any additional information in the repository. To push and pull you have to extend the tools as well. That is the next step. Regards Stephan -- b.i.t. beratungsgesellschaft für informations-technologie mbh Stephan Feder elisabethenstr. 62 fon: +49(0)6151/827575 64283 darmstadt fax: +49(0)6151/827576 mailto:sf@b-i-t.de www: http://www.b-i-t.de ``` ## Andreas Ericsson, 2006-12-01 14:52 Subject: Re: [RFC] Submodules in GIT Message-ID: <457041AD.4010601@op5.se> URL: https://gitlist.dev/e/457041AD.4010601%40op5.se In-Reply-To: <20061201134610.GW18810@admingilde.org> ``` Martin Waitz wrote: > hoi :) > > On Fri, Dec 01, 2006 at 02:43:16PM +0100, Andreas Ericsson wrote: >> So a commit in the supermodule turns into a commit in the submodule? > > no. > >> If it doesn't, why would the submodule HEAD have to change? > > So how do you update your submodule? > By committing to it separately, or by getting changes from the upstream project (openssl, libcurl, ...). > Remember: if you git-pull in the supermodule, you want to update the > whole thing, including all submodules. > Only if the new commits I pull into the supermodule DAG has commits which includes a new shapshot of the submodule, otherwise it wouldn't be necessary. -- Andreas Ericsson andreas.ericsson@op5.se OP5 AB www.op5.se ``` ## Martin Waitz, 2006-12-01 14:58 Subject: Re: [RFC] Submodules in GIT Message-ID: <20061201145817.GY18810@admingilde.org> URL: https://gitlist.dev/e/20061201145817.GY18810%40admingilde.org In-Reply-To: <45703375.4050500@b-i-t.de> ``` hoi :) On Fri, Dec 01, 2006 at 02:51:49PM +0100, Stephan Feder wrote: > If you work in the supermodule, even if it is in the code of the > submodule, you only commit to the supermodule. The submodule does not > "know" about these changes after step 1. I think we are using totally different definitions of "submodule". For me a submodule is responsible for everything in or below a certain directory. So by definition when you change something in this directory, you have to change it in the submodule. You can't change the submodule contents in the supermodule without also changing the submodule. This is just like you can't commit a change to a file without also changing the file. Then the supermodule just records the current content of the entire tree. The only new thing is that instead of simple files there are now submodules and that are also recorded. > Why do you mix up supermodule and submodule? The way I see your proposal > you cannot change submodule and supermodule independently. That is a > huge drawback. No, this is the benefit you get by introducing submodules. Why would you want to introduce a submodule when it is not linked to the supermodule? -- Martin Waitz ``` ## Martin Waitz, 2006-12-01 15:00 Subject: Re: [RFC] Submodules in GIT Message-ID: <20061201150045.GZ18810@admingilde.org> URL: https://gitlist.dev/e/20061201150045.GZ18810%40admingilde.org In-Reply-To: <457041AD.4010601@op5.se> ``` hoi :) On Fri, Dec 01, 2006 at 03:52:29PM +0100, Andreas Ericsson wrote: > >Remember: if you git-pull in the supermodule, you want to update the > >whole thing, including all submodules. > > > > Only if the new commits I pull into the supermodule DAG has commits > which includes a new shapshot of the submodule, otherwise it wouldn't be > necessary. Of course. But if the supermodule contains changes to the submodule, you still have to change the submodule. And this implies changing the submodule HEAD or some branch. -- Martin Waitz ``` ## Martin Waitz, 2006-12-01 15:07 Subject: Re: [RFC] Submodules in GIT Message-ID: <20061201150746.GA18810@admingilde.org> URL: https://gitlist.dev/e/20061201150746.GA18810%40admingilde.org In-Reply-To: <45703ACB.6050007@b-i-t.de> ``` hoi :) On Fri, Dec 01, 2006 at 03:23:07PM +0100, Stephan Feder wrote: > And what is already there is a lot of meta information (see above). You > do not need that. What information are you refering to? Perhaps you have looked into my old branch? The current implementation is in "module2". > For example, in the index, if it is a commit (i.e. a subproject), store > the commit id (not the commit's tree id ). This is exactly what I have done. > Especially, extend update-index to be able to store a commit > instead of the tree. Done, except that update-index never stores trees ;-) > Or else, do not change what is recorded in the index. Then, at commit > time, you not only commit the superproject but also all subprojects. But then submodules would be handled differently to files which I wanted to avoid. > To push and pull you have to extend the tools as well. That is the next > step. Also done. -- Martin Waitz ``` ## Stephan Feder, 2006-12-01 15:47 Subject: Re: [RFC] Submodules in GIT Message-ID: <45704EA3.40203@b-i-t.de> URL: https://gitlist.dev/e/45704EA3.40203%40b-i-t.de In-Reply-To: <20061201145817.GY18810@admingilde.org> ``` Martin Waitz wrote: > hoi :) > > On Fri, Dec 01, 2006 at 02:51:49PM +0100, Stephan Feder wrote: >> If you work in the supermodule, even if it is in the code of the >> submodule, you only commit to the supermodule. The submodule does not >> "know" about these changes after step 1. > > I think we are using totally different definitions of "submodule". No so different. The way I see it is that "I" (meaning with submodules implemented as I proposed) could pull regularly from "your" repositories (implemented as you proposed) and work with the result (including submodules). Could you do the same? > For me a submodule is responsible for everything in or below a certain > directory. So by definition when you change something in this > directory, you have to change it in the submodule. But you do not consider the case where you cannot change the submodule because you do not own it. For example, git has the subproject xdiff. If git had been able to work with subprojects as I envision, and if xdiff had been published as a git repository (not necessarily subproject enabled), it could have been pulled in git's subdirectory xdiff as a subproject. There would not have been a separate branch or even repository for xdiff in the git repository. All changes to xdiff in git could have been committed to the git repository only. Independently, they could have been published to upstream and be put into the xdiff repository by its author. But the last part is what only the owner of the xdiff repository is able to decide. (Ok, ok... the example sucks badly because xdiff has been massively changed for its usage in git so the changes would not be integrated by upstream. But you can imagine where you use a library essentially as is, only if you discover bugs you fix them immediately in your repository and keep those fixes in your version of the library, even on upgrade, until the bugs have been fixed by upstream.) > You can't change the submodule contents in the supermodule without also > changing the submodule. > This is just like you can't commit a change to a file without also > changing the file. There is a difference. I would say: If you commit a change to a file in one branch, it need not be changed in all branches. > Then the supermodule just records the current content of the entire > tree. The only new thing is that instead of simple files there are now > submodules and that are also recorded. Yes, and that is all you need. If the changes are to be part of a branch of the submodule, they have to be pulled. That is an independent operation. >> Why do you mix up supermodule and submodule? The way I see your proposal >> you cannot change submodule and supermodule independently. That is a >> huge drawback. > > No, this is the benefit you get by introducing submodules. > Why would you want to introduce a submodule when it is not linked to the > supermodule? Because the submodule must be independent of the supermodule. I see where you are coming from. You have one project that is divided into subprojects but the subprojects themselves are not independent. What I would like to solve is the followng: You have a project X, an this project is made part of two other projects Y and Z (as a submodule or subproject or whatever you want to call it). The project X need not, must not or cannot care that it was made a subproject. But in projects Y and Z, you must be able to bugfix or extend or modify the code of projectX, and you must be able to push and pull changes between all three projects (of course we are only talking about the code part of project X). Do you see where your solution makes that impossible, and that with more changes to the repository layout? Regards Stephan ``` ## Stephan Feder, 2006-12-01 16:04 Subject: Re: [RFC] Submodules in GIT Message-ID: <45705283.5080900@b-i-t.de> URL: https://gitlist.dev/e/45705283.5080900%40b-i-t.de In-Reply-To: <20061201150746.GA18810@admingilde.org> ``` Martin Waitz wrote: > hoi :) > > On Fri, Dec 01, 2006 at 03:23:07PM +0100, Stephan Feder wrote: >> And what is already there is a lot of meta information (see above). You >> do not need that. > > What information are you refering to? > Perhaps you have looked into my old branch? > The current implementation is in "module2". I was looking into git-init-module.sh (branch module2). There you set up a separate git repository for the submodule and store references to it into the supermodules's repository. > >> For example, in the index, if it is a commit (i.e. a subproject), store >> the commit id (not the commit's tree id ). > > This is exactly what I have done. > >> Especially, extend update-index to be able to store a commit >> instead of the tree. > > Done, except that update-index never stores trees ;-) Yes, I forgot. >> Or else, do not change what is recorded in the index. Then, at commit >> time, you not only commit the superproject but also all subprojects. > > But then submodules would be handled differently to files which I wanted > to avoid. On the other hand, it feels more naturally to only commit at the end of your work. So both alternatives have their merits. > >> To push and pull you have to extend the tools as well. That is the next >> step. > > Also done. I hope I have time to give your solution a try. Regards Stephan -- b.i.t. beratungsgesellschaft für informations-technologie mbh Stephan Feder elisabethenstr. 62 fon: +49(0)6151/827575 64283 darmstadt fax: +49(0)6151/827576 mailto:sf@b-i-t.de www: http://www.b-i-t.de ``` ## Martin Waitz, 2006-12-01 16:15 Subject: Re: [RFC] Submodules in GIT Message-ID: <20061201161558.GC18810@admingilde.org> URL: https://gitlist.dev/e/20061201161558.GC18810%40admingilde.org In-Reply-To: <45705283.5080900@b-i-t.de> ``` hoi :) On Fri, Dec 01, 2006 at 05:04:19PM +0100, Stephan Feder wrote: > I was looking into git-init-module.sh (branch module2). There you set up > a separate git repository for the submodule and store references to it > into the supermodules's repository. yes. This is to be able to call git-fsck-objects and git-prune in the toplevel supermodule. When traversing the object tree, it already knows about all submodule, but only about those versions that are really part of their supermodule. So I have to teach git about separate submodule branches which may be used, in order not to prune them away. -- Martin Waitz ``` ## Andreas Ericsson, 2006-12-01 16:38 Subject: Re: [RFC] Submodules in GIT Message-ID: <45705A94.2070509@op5.se> URL: https://gitlist.dev/e/45705A94.2070509%40op5.se In-Reply-To: <20061201150045.GZ18810@admingilde.org> ``` Martin Waitz wrote: > hoi :) > > On Fri, Dec 01, 2006 at 03:52:29PM +0100, Andreas Ericsson wrote: >>> Remember: if you git-pull in the supermodule, you want to update the >>> whole thing, including all submodules. >>> >> Only if the new commits I pull into the supermodule DAG has commits >> which includes a new shapshot of the submodule, otherwise it wouldn't be >> necessary. > > Of course. > > But if the supermodule contains changes to the submodule, you still > have to change the submodule. And this implies changing the submodule > HEAD or some branch. > Not really. I fail to see why HEAD needs to be changed so long as the commit is in the submodule's odb. -- Andreas Ericsson andreas.ericsson@op5.se OP5 AB www.op5.se ``` ## Linus Torvalds, 2006-12-01 16:49 Subject: Re: [RFC] Submodules in GIT Message-ID: URL: https://gitlist.dev/e/Pine.LNX.4.64.0612010844380.3695%40woody.osdl.org In-Reply-To: <45705A94.2070509@op5.se> ``` On Fri, 1 Dec 2006, Andreas Ericsson wrote: > Martin Waitz wrote: > > > > But if the supermodule contains changes to the submodule, you still > > have to change the submodule. And this implies changing the submodule > > HEAD or some branch. > > > > Not really. I fail to see why HEAD needs to be changed so long as the commit > is in the submodule's odb. Right. A commit in the supermodule should _not_ imply a commit in the submodule. Maybe I should take a look at the code, but it sounds like people are still trying to "mix" submodules too much. Think of it this way: one common use for submodules is really to just (occasionally) track somebody elses code. The submodule should be a totally pristine copy from somebody else (ie it might be the "intel driver for X.org" submodule, maintained within intel), and the supermodule just refers to it indirectly (ie the supermodule might be the "Fedora Core X group" which contains all the different drivers from different people). So anything that mixes super-modules and sub-modules too much will always break this kind of model. A supermodule can never "contain changes" to a submodule. A supermodule would always just point to the submodule, and not have any changes what-so-ever of its own. The submodule is self-sufficient, and always contains all its _own_ changes. ``` ## Martin Waitz, 2006-12-01 16:54 Subject: Re: [RFC] Submodules in GIT Message-ID: <20061201165418.GD18810@admingilde.org> URL: https://gitlist.dev/e/20061201165418.GD18810%40admingilde.org In-Reply-To: <45704EA3.40203@b-i-t.de> ``` hoi :) On Fri, Dec 01, 2006 at 04:47:47PM +0100, Stephan Feder wrote: > No so different. The way I see it is that "I" (meaning with submodules > implemented as I proposed) could pull regularly from "your" repositories > (implemented as you proposed) and work with the result (including > submodules). Could you do the same? Sorry, but with all that many people proposing things I am a bit lost now. Sometimes I thought you want exactly the same thing as I do, sometimes I think we are talking in totally different directions. > >For me a submodule is responsible for everything in or below a certain > >directory. So by definition when you change something in this > >directory, you have to change it in the submodule. > > But you do not consider the case where you cannot change the submodule > because you do not own it. I do not understand you here. The submodule is part of the supermodule, and the one who sets up the repository owns the whole thing, including all submodules, just like all the files which are part of the project. If you mean the upstream repository of the submodule, then yes, this is of course completely separated from the submodule and may be owned by someone else. Consequently, this upstream repository of course does not need to change when someone introduces changes in the supermodule. > For example, git has the subproject xdiff. If git had been able to work > with subprojects as I envision, and if xdiff had been published as a git > repository (not necessarily subproject enabled), it could have been > pulled in git's subdirectory xdiff as a subproject. This could have been done if submodule support would have been available at the time xdiff was introduced, yes. > There would not have been a separate branch or even repository for > xdiff in the git repository. What separate branch or repository are you talking about? > All changes to xdiff in git could have been committed to the git > repository only. Yes, but if it would have been integrated as a submodule it obviously would have been committed to the xdiff submodule inside the git repository. So the changes are really part of the git repository, but you could go to the "git/xdiff" directory and only see the changes in the submodule, without the normal supermodule history. > Independently, they could have been published to upstream and be put > into the xdiff repository by its author. But the last part is what > only the owner of the xdiff repository is able to decide. Of course, everything still works like normal git repositories. > >You can't change the submodule contents in the supermodule without also > >changing the submodule. > >This is just like you can't commit a change to a file without also > >changing the file. > > There is a difference. I would say: If you commit a change to a file in > one branch, it need not be changed in all branches. But you need to change _at_least_ one branch. Otherwise you cannot commit to a branch. So if you change something in a submodule, you have to change one branch in the submodule. If you call git-checkout in the supermodule this will result in something like a git-reset in the submodule. > >No, this is the benefit you get by introducing submodules. > >Why would you want to introduce a submodule when it is not linked to the > >supermodule? > > Because the submodule must be independent of the supermodule. > > I see where you are coming from. You have one project that is divided > into subprojects but the subprojects themselves are not independent. > > What I would like to solve is the followng: You have a project X, an > this project is made part of two other projects Y and Z (as a submodule > or subproject or whatever you want to call it). The project X need not, > must not or cannot care that it was made a subproject. But in projects Y > and Z, you must be able to bugfix or extend or modify the code of > projectX, and you must be able to push and pull changes between all > three projects (of course we are only talking about the code part of > project X). Of course. So if you wanted to check out everything, you could have something like ~/src/X, ~/src/Y/X, and ~/src/Z/X. All of these would be GIT repositories, all of them have their independent branches. What I am saying is just that if you update Y, and the new Y contains an updated version of X, then ~/src/Y/X/.git/refs/heads/master will be changed by the pull, resulting in the new version of X being checked out in ~/src/Y/X (alongside all the other updates inside ~/src/Y). This of course is independend from ~/src/X or ~/src/Z/X. > Do you see where your solution makes that impossible, and that with more > changes to the repository layout? No ;-) -- Martin Waitz ``` ## Martin Waitz, 2006-12-01 16:57 Subject: Re: [RFC] Submodules in GIT Message-ID: <20061201165708.GE18810@admingilde.org> URL: https://gitlist.dev/e/20061201165708.GE18810%40admingilde.org In-Reply-To: <45705A94.2070509@op5.se> ``` On Fri, Dec 01, 2006 at 05:38:44PM +0100, Andreas Ericsson wrote: > >But if the supermodule contains changes to the submodule, you still > >have to change the submodule. And this implies changing the submodule > >HEAD or some branch. > > > > Not really. I fail to see why HEAD needs to be changed so long as the > commit is in the submodule's odb. Because I want the submodule to act as a normal git repository. Please note that I also voted against changing HEAD directly, but that the new commit which came from the supermodule is just stored in one branch of the submodule, as part of the supermodule checkout. -- Martin Waitz ``` ## sf, 2006-12-01 17:08 Subject: Re: [RFC] Submodules in GIT Message-ID: <457061A7.2000102@b-i-t.de> URL: https://gitlist.dev/e/457061A7.2000102%40b-i-t.de In-Reply-To: ``` Linus Torvalds wrote: ... > Think of it this way: one common use for submodules is really to just > (occasionally) track somebody elses code. The submodule should be a > totally pristine copy from somebody else (ie it might be the "intel driver > for X.org" submodule, maintained within intel), and the supermodule just > refers to it indirectly (ie the supermodule might be the "Fedora Core X > group" which contains all the different drivers from different people). Could you please be a little bit more specific about how you would store the "pristine copy". There seems to be some agreement to store the commit id of the submodule instead of a plain tree id in the supermodules tree object, and that all objects that are reachable from this commit are made part of the supermodule repository (either fetched or via alternates). Do you agree? ... > A supermodule can never "contain changes" to a submodule. A supermodule > would always just point to the submodule, and not have any changes > what-so-ever of its own. The submodule is self-sufficient, and always > contains all its _own_ changes. That is one of the points Martin Waitz and I are discussing. If I understand you correctly you cannot make any changes to the submodules code _in the supermodule's repository_, no bugfixes, no extensions, no adaptions, nothing. Do you mean that? That would be a third alternative. In my opinion the usefulness of submodules would be unnecessarily restricted if it comes to the choice of either using the code from upstream as is or do not use submodules at all. What is the point of the restriction? Regards ``` ## Martin Waitz, 2006-12-01 17:14 Subject: Re: [RFC] Submodules in GIT Message-ID: <20061201171420.GF18810@admingilde.org> URL: https://gitlist.dev/e/20061201171420.GF18810%40admingilde.org In-Reply-To: ``` hoi :) On Fri, Dec 01, 2006 at 08:49:20AM -0800, Linus Torvalds wrote: > Think of it this way: one common use for submodules is really to just > (occasionally) track somebody elses code. The submodule should be a > totally pristine copy from somebody else (ie it might be the "intel driver > for X.org" submodule, maintained within intel), and the supermodule just > refers to it indirectly (ie the supermodule might be the "Fedora Core X > group" which contains all the different drivers from different people). Yes, but it is not only about tracking, also about distributing submodules. One Fedora X developer fixes a bug in the intel driver, commits that to the submodule and then updates the supermodule to the new version (by calling "git-update-index drivers/intel && git-commit" or something). Then another Feora X developer updates his X repository. By pulling the supermodule he also gets a new version of the submodule. And this new version of the submodule is stored in a branch which can be accessed by the submodule. > A supermodule can never "contain changes" to a submodule. The supermodule always contains _the_entire_ submodule with its complete history, so it also does contain changes. But it does not per-se contain changes, only indirectly (i.e. the commits in the submodule are not part of the supermodule commit chain). > A supermodule would always just point to the submodule, and not have > any changes what-so-ever of its own. The submodule is self-sufficient, > and always contains all its _own_ changes. Yes. -- Martin Waitz ``` ## Stephan Feder, 2006-12-01 17:33 Subject: Re: [RFC] Submodules in GIT Message-ID: <45706758.2020907@b-i-t.de> URL: https://gitlist.dev/e/45706758.2020907%40b-i-t.de In-Reply-To: <20061201165418.GD18810@admingilde.org> ``` Martin Waitz wrote: > hoi :) > > On Fri, Dec 01, 2006 at 04:47:47PM +0100, Stephan Feder wrote: >> No so different. The way I see it is that "I" (meaning with submodules >> implemented as I proposed) could pull regularly from "your" repositories >> (implemented as you proposed) and work with the result (including >> submodules). Could you do the same? > > Sorry, but with all that many people proposing things I am a bit lost > now. Sometimes I thought you want exactly the same thing as I do, > sometimes I think we are talking in totally different directions. We are in agreement about two fundamental parts of the implementation and their meaning: 1. A submodule is stored as a commit id in a tree object. 2. Every object that is reachable from the submodule's commit are reachable from the supermodule's repository. Please confirm. We now argue about how to work with that repository _object_ model. >> >For me a submodule is responsible for everything in or below a certain >> >directory. So by definition when you change something in this >> >directory, you have to change it in the submodule. >> >> But you do not consider the case where you cannot change the submodule >> because you do not own it. > > I do not understand you here. > The submodule is part of the supermodule, and the one who sets up the > repository owns the whole thing, including all submodules, just like all > the files which are part of the project. If you mean by "owns the whole thing" what I stated above in 2. the we agree. > If you mean the upstream repository of the submodule, then yes, this is > of course completely separated from the submodule and may be owned by > someone else. Consequently, this upstream repository of course does not > need to change when someone introduces changes in the supermodule. I think we still agree. >> For example, git has the subproject xdiff. If git had been able to work >> with subprojects as I envision, and if xdiff had been published as a git >> repository (not necessarily subproject enabled), it could have been >> pulled in git's subdirectory xdiff as a subproject. > > This could have been done if submodule support would have been available > at the time xdiff was introduced, yes. > >> There would not have been a separate branch or even repository for >> xdiff in the git repository. > > What separate branch or repository are you talking about? That's it: There is no need for a separate branch or repository. If you have the subproject's commit in the superproject's object database (and we really have that, see 1. and 2. above), why do you _have to_ store it elsewhere? >> All changes to xdiff in git could have been committed to the git >> repository only. > > Yes, but if it would have been integrated as a submodule it obviously > would have been committed to the xdiff submodule inside the git > repository. No. The xdiff submodule would only exist as part of the git repository. You could, f.e., access the xdiff commit in git HEAD as HEAD:xdiff// (again my proposed syntax). HEAD:xdiff//~2:xemit.c would give you the grandparent of xemit.c in the xdiff submodule. And so on. You can even have submodules that have themselves submodules. > So the changes are really part of the git repository, but you could go > to the "git/xdiff" directory and only see the changes in the submodule, > without the normal supermodule history. See above. >> Independently, they could have been published to upstream and be put >> into the xdiff repository by its author. But the last part is what >> only the owner of the xdiff repository is able to decide. > > Of course, everything still works like normal git repositories. OK. >> >You can't change the submodule contents in the supermodule without also >> >changing the submodule. >> >This is just like you can't commit a change to a file without also >> >changing the file. >> >> There is a difference. I would say: If you commit a change to a file in >> one branch, it need not be changed in all branches. > > But you need to change _at_least_ one branch. > Otherwise you cannot commit to a branch. But only the supermodule's branch. > So if you change something in a submodule, you have to change one branch > in the submodule. No. > If you call git-checkout in the supermodule this will result in > something like a git-reset in the submodule. If you mean the submodule repository created by init-module I understand. But why create this "helper repository at all"? >> >No, this is the benefit you get by introducing submodules. >> >Why would you want to introduce a submodule when it is not linked to the >> >supermodule? >> >> Because the submodule must be independent of the supermodule. >> >> I see where you are coming from. You have one project that is divided >> into subprojects but the subprojects themselves are not independent. >> >> What I would like to solve is the followng: You have a project X, an >> this project is made part of two other projects Y and Z (as a submodule >> or subproject or whatever you want to call it). The project X need not, >> must not or cannot care that it was made a subproject. But in projects Y >> and Z, you must be able to bugfix or extend or modify the code of >> projectX, and you must be able to push and pull changes between all >> three projects (of course we are only talking about the code part of >> project X). > > Of course. > > So if you wanted to check out everything, you could have something like > ~/src/X, ~/src/Y/X, and ~/src/Z/X. > All of these would be GIT repositories, all of them have their > independent branches. > > What I am saying is just that if you update Y, and the new Y contains an > updated version of X, then ~/src/Y/X/.git/refs/heads/master will be > changed by the pull, resulting in the new version of X being checked out > in ~/src/Y/X (alongside all the other updates inside ~/src/Y). > This of course is independend from ~/src/X or ~/src/Z/X. > >> Do you see where your solution makes that impossible, and that with more >> changes to the repository layout? > > No ;-) > Sorry, have to leave for home so I must leave that uncommented. Hopefully I can join in during the weekend. Regards ``` ## Andreas Ericsson, 2006-12-01 18:06 Subject: Re: [RFC] Submodules in GIT Message-ID: <45706F19.6090609@op5.se> URL: https://gitlist.dev/e/45706F19.6090609%40op5.se In-Reply-To: <457061A7.2000102@b-i-t.de> ``` sf wrote: > > That is one of the points Martin Waitz and I are discussing. > > If I understand you correctly you cannot make any changes to the > submodules code _in the supermodule's repository_, no bugfixes, no > extensions, no adaptions, nothing. Do you mean that? > > That would be a third alternative. In my opinion the usefulness of > submodules would be unnecessarily restricted if it comes to the choice > of either using the code from upstream as is or do not use submodules at > all. What is the point of the restriction? > That depends on your definition of submodule. In my eyes, a submodule is a separate repo that can be committed to separately (and generally also built separately), although it's usually built into something else. I'm imagining most submodules will contain only library code and its testing routines. Insofar as I've envisioned submodules, it's a separate git repo where you simply record a certain snapshot of the sub-repo with a commit in the super-module, like so: $ git commit ssl-functions/*.[ch] openssl -m "Upgraded openssl with necessary changes to core code" (yes, I know it's horrid to use -m to commit, and I daily advocate against it where I work, but you get the idea, I'm sure) Isn't this how it's supposed to work? Enlighten me, and please remember that I'm drunk atm, so make it obvious ;-) -- Andreas Ericsson andreas.ericsson@op5.se OP5 AB www.op5.se ``` ## Andreas Ericsson, 2006-12-01 18:08 Subject: Re: [RFC] Submodules in GIT Message-ID: <45706F8C.3070000@op5.se> URL: https://gitlist.dev/e/45706F8C.3070000%40op5.se In-Reply-To: <20061201165708.GE18810@admingilde.org> ``` Martin Waitz wrote: > On Fri, Dec 01, 2006 at 05:38:44PM +0100, Andreas Ericsson wrote: >>> But if the supermodule contains changes to the submodule, you still >>> have to change the submodule. And this implies changing the submodule >>> HEAD or some branch. >>> >> Not really. I fail to see why HEAD needs to be changed so long as the >> commit is in the submodule's odb. > > Because I want the submodule to act as a normal git repository. > Please note that I also voted against changing HEAD directly, but that > the new commit which came from the supermodule is just stored in one > branch of the submodule, as part of the supermodule checkout. > You're assuming the super- and sub-module will share HEAD, or at least ODB, I think. I'm not convinced this is necessary. Convince me. I'll go drink bear and get some dancing done while you're at it ;-) -- Andreas Ericsson andreas.ericsson@op5.se OP5 AB www.op5.se ``` ## Martin Waitz, 2006-12-01 18:48 Subject: Re: [RFC] Submodules in GIT Message-ID: <20061201184801.GG18810@admingilde.org> URL: https://gitlist.dev/e/20061201184801.GG18810%40admingilde.org In-Reply-To: <45706758.2020907@b-i-t.de> ``` On Fri, Dec 01, 2006 at 06:33:12PM +0100, Stephan Feder wrote: > We are in agreement about two fundamental parts of the implementation > and their meaning: > > 1. A submodule is stored as a commit id in a tree object. > > 2. Every object that is reachable from the submodule's commit are > reachable from the supermodule's repository. Correct. > >>For example, git has the subproject xdiff. If git had been able to work > >>with subprojects as I envision, and if xdiff had been published as a git > >>repository (not necessarily subproject enabled), it could have been > >>pulled in git's subdirectory xdiff as a subproject. > > > >This could have been done if submodule support would have been available > >at the time xdiff was introduced, yes. > > > >>There would not have been a separate branch or even repository for > >>xdiff in the git repository. > > > >What separate branch or repository are you talking about? > > That's it: There is no need for a separate branch or repository. If you > have the subproject's commit in the superproject's object database (and > we really have that, see 1. and 2. above), why do you _have to_ store it > elsewhere? Let's see if I understand you correctly: You don't want to create an additional .git directory for the submodule and just handle everything with one toplevel .git repository for the whole project. Without the .git directory, you of course do not have refs/heads inside the submodule. So this is a different user-interface approach to submodules when compared to my approach. But the basis is the same and both could inter-operate. Now your submodule is no longer seen as an independent git repository and I think this would cause problems when you want to push/pull between the submodule and its upstream repository. No technical problems, but UI-problems because now your submodule is handled completly different to a "normal" repository. > >Yes, but if it would have been integrated as a submodule it obviously > >would have been committed to the xdiff submodule inside the git > >repository. > > No. The xdiff submodule would only exist as part of the git repository. But you could still call the "xdiff" part of the git repository a submodule. And then changes to the xdiff directory result in a new submodule commit, even when there is no direct reference to it. So you'd still "commit to the xdiff submodule". > You could, f.e., access the xdiff commit in git HEAD as HEAD:xdiff// > (again my proposed syntax). HEAD:xdiff//~2:xemit.c would give you the > grandparent of xemit.c in the xdiff submodule. git-cat-file commit HEAD:xdiff already works out of the box (even cat-file tree to get the submodule tree). But up to now revision parsing follows the file name only once. What about just separating things with "/"? commit HEAD tree HEAD/ blob HEAD/Makefile commit HEAD/xdiff tree HEAD/xdiff/ blob HEAD/xdiff~2/xemit.c this may add some confusion when used with hierarchical branches, but it's still unique: refs/heads/master/xdiff/xemit.c Just use as many path components until a matching reference is found, then start peeling. Or just use / between super and submodule: refs/heads/master:xdiff/xemit.c I think this is easier to read then refs/heads/master:xdiff//:xemit.c > If you mean the submodule repository created by init-module I > understand. But why create this "helper repository at all"? Because it helps "normal" git operations ;-) -- Martin Waitz ``` ## Martin Waitz, 2006-12-01 18:51 Subject: Re: [RFC] Submodules in GIT Message-ID: <20061201185100.GH18810@admingilde.org> URL: https://gitlist.dev/e/20061201185100.GH18810%40admingilde.org In-Reply-To: <45706F8C.3070000@op5.se> ``` hoi :) On Fri, Dec 01, 2006 at 07:08:12PM +0100, Andreas Ericsson wrote: > You're assuming the super- and sub-module will share HEAD, or at least > ODB, I think. not HEAD, only ODB. > I'm not convinced this is necessary. Convince me. I'll go > drink bear and get some dancing done while you're at it ;-) Get me a beer and I will convince you :-) -- Martin Waitz ``` ## Andy Parkins, 2006-12-01 19:17 Subject: Re: [RFC] Submodules in GIT Message-ID: <200612011917.19252.andyparkins@gmail.com> URL: https://gitlist.dev/e/200612011917.19252.andyparkins%40gmail.com In-Reply-To: <45706758.2020907@b-i-t.de> ``` On Friday 2006, December 01 17:33, Stephan Feder wrote: > 1. A submodule is stored as a commit id in a tree object. > > 2. Every object that is reachable from the submodule's commit are > reachable from the supermodule's repository. I'm still not convinced about 2. Why should any of the submodule commits be in the supermodule repository? I know that is what you've implemented, but it still feels like too much of a blending of the submodule into the supermodule. In fact, why should the submodule commits be even visible in the supermodule? That tree->submodule commit is sufficient; there isn't any need to view submodule history in the supermodule. Andy -- Dr Andrew Parkins, M Eng (Hons), AMIEE ``` ## Martin Waitz, 2006-12-01 19:38 Subject: Re: [RFC] Submodules in GIT Message-ID: <20061201193802.GI18810@admingilde.org> URL: https://gitlist.dev/e/20061201193802.GI18810%40admingilde.org In-Reply-To: <200612011917.19252.andyparkins@gmail.com> ``` hoi :) On Fri, Dec 01, 2006 at 07:17:17PM +0000, Andy Parkins wrote: > In fact, why should the submodule commits be even visible in the > supermodule? That tree->submodule commit is sufficient; there isn't > any need to view submodule history in the supermodule. Well, but there is a need for a common object traversal. You need that when sending all objects between two supermodule versions and also when you determine which objects are still reachable. The easiest way to implement the common object traversal is to have all objects in one object repository. It may be possible to use two object stores and still do the common object traversal but I do not think that gives you any benefits. You still don't have a totally separated repository then, because you can't do a reachability analysis in the submodule repository alone. -- Martin Waitz ``` ## Linus Torvalds, 2006-12-01 20:13 Subject: Re: [RFC] Submodules in GIT Message-ID: URL: https://gitlist.dev/e/Pine.LNX.4.64.0612011134080.3695%40woody.osdl.org In-Reply-To: <457061A7.2000102@b-i-t.de> ``` On Fri, 1 Dec 2006, sf wrote: > > Linus Torvalds wrote: > ... > > Think of it this way: one common use for submodules is really to just > > (occasionally) track somebody elses code. The submodule should be a > > totally pristine copy from somebody else (ie it might be the "intel driver > > for X.org" submodule, maintained within intel), and the supermodule just > > refers to it indirectly (ie the supermodule might be the "Fedora Core X > > group" which contains all the different drivers from different people). > > Could you please be a little bit more specific about how you would store the > "pristine copy". Note that it's not necessarily "pristine", since the submodule clearly is a local git repository in its own right. So like _any_ git repository, you can (and may well end up) having your own local branches in the submodule, with your own local modifications. So I'm not claiming that a submodule must always match some external git tree 100%, and that it must be read-only or anything like that. I'm just saying that I suspect that quite often, one of the MOST IMPORTANT parts is that the submodule is really something that somebody else technically maintains, and that this is actually one of the _reasons_ why it is a submodule in the first place. For example, a lot of projects end up having some kind of "library component" as a submodule. Take something like a video player project, which would have something like ffmpeg as a submodule, not because you'd maintain ffmpeg yourself, but simply because (let's say) the library interface changes enough, or you need a specific version with some of your own fixes that haven't been released widely yet, so you want to carry all the libraries you need _with_ you, even though you don't really maintain that submodule. You at most have some small extensions of your own. Now, in this situation, it's relaly really _important_ that the submodule really is totally independent of the supermodule, for several reasons. For example, since you don't "really" own that project, carrying around your own fixes is really really painful. We know it happens all the time, and a lot of projects end up needing their own version, but the _last_ thing you want is to be in merge hell all the time. So as a supermodule maintainer, the best possible thing for you is to be able to push back those local changes to the original project maintainer, so that you _don't_ have to maintain your own changes. But you need to realize that the real maintainer of the submodule is TOTALLY UNINTERESTED in your supermodule. He's not going to maintain it, and in fact, if you have anything in the submodule that ends up talking about your supermodule, that's just going to make it a lot less likely that the upstream maintainer will ever pull your changes. He might take a diff from you, but in a perfect world, you'd actually be able to tell him: "Hey, I've got a git repository with a few fixes to your ffmpeg git tree, please pull from git://myhost.com/submodule.git to get these fixes: ... explanation of fixes and commits that are relevant to ffmpeg, and have nothing to do with the supermodule, except that you need those bug-fixes because you _use_ ffmpeg ... Thanks" See? So this is why it's really important that the submodule really is a git repository in its own right, and why committing stuff in the supermodule NEVER affect the submodule itself directly (it might _cause_ you to also do a commit in the submodule indirectly, but the submodule commit MUST be totally independent, and stand on its own). Now, you don't _have_ to push things upstream, of course. You can always just maintain your own submodule branch, and every once in a while, inside the submodule, you do # fetch the development in the origin/master branch git fetch submodule-origin origin/master # rebase our own special magic sauce on top of that git rebase origin/master to update your submodule, and _then_ you do a commit in the supermodule (after testing that the update is all ok, of course) which will update the "commit" pointer in the supermodule. Notice? In this example, we really maintained the submodule AS a submodule. It was independent, but tied into the supermodule, so that when we clone the supermodule, or do things like bisection on a supermodule, we always end up cloning the submodule too (and in the case of bisection, we really only bisect the supermodule, but the submodule always gets "tracked" in the sense that we would always check out the state of the submodule that was appropriate for that particular commit in the supermodule). > There seems to be some agreement to store the commit id of > the submodule instead of a plain tree id in the supermodules tree object, and > that all objects that are reachable from this commit are made part of the > supermodule repository (either fetched or via alternates). Do you agree? Well, I would actually argue that you may often want to have a supermodule and then at least have the _option_ to decide to not fetch all the submodules. For an example of this kind of usage, let me tell you how we operated at Transmeta a few years ago, which I'm not saying is the _only_ way to operate, but it's ONE way to do it, and I'll also explain _why_ we did it, and why we had submodules. In the case of transmeta, we had our own tools, our own programs, and we "owned" all of those. We _also_ used a lot of external tools, like gcc etc. However, different people worked on different parts, and if you worked on the actual x86 JIT part, you probably didn't want to have all of the gcc stuff in your tree _too_. That just took a lot of space, and you really didn't want to compile the whole toolchain (which took hours), since there were precompiled binaries readily available. Still, from a _release_ standpoint, when we released a new binary, that binary very much depended not just on the actual JIT sources, but on the whole toolchain. So if you wanted to be able to re-create a release, you really needed _everything_. You couldn't just take the "current version" of the toolchain, you needed to have the toolchain that was used AT THE TIME OF THE RELEASE. And this is a _classic_ example of when you'd want to use submodules. Notice how everybody wanted _some_ of the submodules, but really only the release people wanted them _all_. The higher up the chain you were, the less likely you were to really want to muck around with the compiler and the linker, for example. And nobody really owned all modules. So what you really want is: - a supermodule maintainer that is not really the maintainer of _any_ of the submodules, but that does the main "build world" infrastructure (and generally would tend to also maintain the source control infrastructure itself) - submodules that had their own maintainers, and where the maintainers may or may not have wanted the supermodule, but even when they wanted the supermodule, they might not want _all_ of the submodules, simply because they just didn't care. - some of the submodules then have _upstream_ sources that were totally independent, and that you would want to track, but you had zero power AT ALL over them, and yet you migt well want to push back at least some of the fixes you did - at least the ones that made sense even outside your own project - just to avoid having to maintain a _huge_ set of internal patches. So no, I don't think the supermodule should even _force_ people to always get all the submodules. It migth be the default case, but at the same time, it's just being polite to let users decide on their own whether they really want _all_ of the build infrastructure sources. > If I understand you correctly you cannot make any changes to the submodules > code _in the supermodule's repository_, no bugfixes, no extensions, no > adaptions, nothing. Do you mean that? Yes. I think you should make all changes _within_ the submodule, because the submodule should still be an independent git tree in its own right. But obviously, you'd often use a private _branch_ in the submodule beause you end up having whatever private extensions. That's always true: we always have the "master" branch that is kind of the default "private branch" for any repository, but obviously that is often extended upon, and you may have several private branches. For example, after you've done a big update (from some external upstream source) in the submodule that you are using, you migth decide that you do all the work on that new big update in a _new_ private branch within the submodule - and get the submodule changes all squared away on its own _before_ you then decide to commit the end result (the tip of that new private branch) within the supermodule. Ie, you very much should be able to to do git clone supermodule/that/one/submodule my-own-version-of-submodule to clone a submodule _without_ getting anything else (but still get all the work you did within he submodule - very much including your own private branch work). And the importance of keeping the submodule independent is partly just stability and sanity, but partly also scalability. For example, the "index" in a supermodule should NOT include the indexes of all the submodules. That's really important, because the index doesn't really scale. Things do slow down with large indexes. For example, git can handle tens of thousands of files easily. I suspect it scales well to hundreds of thousands of filenames. But with supermodules, you really can end up in the situation where you have _tens_ of these submodules, maybe even hundreds. And if you try to maintain one unified index for the _whole_ thing, I guarantee you that you'll start feeling the pain. Indexing millions of files is just not going to be pretty. So just from a git stability and scalability point, it's important to keep subprojects _separate_. There is obviously integration stuff, but they should still be seen as truly independent projects. Even the supermodule should have clearly its own life even _regardless_ of submodules, because (as I said) quite often you may want the supermodule, but you don't want to have _all_ of the submodules. But it's more than that stability and scalability thing too - keeping them separate is what allows you to do pulls and pushes on an individual subproject basis, and have people really work at that level. For example, if you're the compiler guy at a company, you really do want to work with other compiler people _outside_ the company, but you sure as hell may not be able to give them access to your supermodule. But you may want to work on _just_ the compiler parts (or at least share some branches in public), which means that the subproject really has to be able to work _independently_ of the supermodule. So "independent" here is really key, for several reasons. And that all means, for example, that here must NEVER be any "backpointers". A subproject really can _never_ have backpointers to the superproject, because that fundamentally means that the above kind of "compiler guy works on the compiler subproject in public" cannot work, if your supermodule isn't public. ``` ## Martin Waitz, 2006-12-01 20:30 Subject: Re: [RFC] Submodules in GIT Message-ID: <20061201203002.GJ18810@admingilde.org> URL: https://gitlist.dev/e/20061201203002.GJ18810%40admingilde.org In-Reply-To: ``` hoi :) Linus, you are a lot better in describing all my thoughts than I myself. ;-) -- Martin Waitz ``` ## Andy Parkins, 2006-12-01 21:04 Subject: Re: [RFC] Submodules in GIT Message-ID: <200612012104.39897.andyparkins@gmail.com> URL: https://gitlist.dev/e/200612012104.39897.andyparkins%40gmail.com In-Reply-To: <20061201193802.GI18810@admingilde.org> ``` On Friday 2006, December 01 19:38, Martin Waitz wrote: > On Fri, Dec 01, 2006 at 07:17:17PM +0000, Andy Parkins wrote: > > In fact, why should the submodule commits be even visible in the > > supermodule? That tree->submodule commit is sufficient; there isn't > > any need to view submodule history in the supermodule. > > Well, but there is a need for a common object traversal. > You need that when sending all objects between two supermodule versions > and also when you determine which objects are still reachable. No you don't; when traversing the supermodule history you will come across trees that have submodule commit hashes in them, that is all the other end needs to know. If it wants it can then connect to the submodule and clone submodule to submodule. The whole operation doesn't have to be done in the supermodule though. > The easiest way to implement the common object traversal is to have all > objects in one object repository. That's true; but is it the right way? I really really think the submodule objects should be in the submodule itself. > It may be possible to use two object stores and still do the common > object traversal but I do not think that gives you any benefits. There is one benefit - you can git-clone the submodule just as you would if it were not a submodule. In fact, from the submodule's point of view it knows nothing about the supermodule. > You still don't have a totally separated repository then, because > you can't do a reachability analysis in the submodule repository alone. I'm going to guess by reachability analysis, you mean that the submodule doesn't know that some of it's commits are referenced by the supermodule. As I suggested elsewhere in the thread, that's easily fixed by making a refs/supermodule/commitXXXX file for each supermodule commit that references as particular submodule commit. Then you can git-prune, git-fsck whenever you want. Andy -- Dr Andrew Parkins, M Eng (Hons), AMIEE ``` ## Martin Waitz, 2006-12-01 21:37 Subject: Re: [RFC] Submodules in GIT Message-ID: <20061201213722.GK18810@admingilde.org> URL: https://gitlist.dev/e/20061201213722.GK18810%40admingilde.org In-Reply-To: <200612012104.39897.andyparkins@gmail.com> ``` hoi :) On Fri, Dec 01, 2006 at 09:04:37PM +0000, Andy Parkins wrote: > > It may be possible to use two object stores and still do the common > > object traversal but I do not think that gives you any benefits. > > There is one benefit - you can git-clone the submodule just as you > would if it were not a submodule. In fact, from the submodule's point > of view it knows nothing about the supermodule. The submodule repository obviously has to able to reach all its objects. This is easily doable with the shared object database. So you can already clone the submodule standalone. > I'm going to guess by reachability analysis, you mean that the > submodule doesn't know that some of it's commits are referenced by the > supermodule. As I suggested elsewhere in the thread, that's easily > fixed by making a refs/supermodule/commitXXXX file for each > supermodule commit that references as particular submodule commit. I wouldn't call this "easily". -- Martin Waitz ``` ## Andy Parkins, 2006-12-01 21:54 Subject: Re: [RFC] Submodules in GIT Message-ID: <200612012154.33834.andyparkins@gmail.com> URL: https://gitlist.dev/e/200612012154.33834.andyparkins%40gmail.com In-Reply-To: <20061201213722.GK18810@admingilde.org> ``` On Friday 2006, December 01 21:37, Martin Waitz wrote: > > I'm going to guess by reachability analysis, you mean that the > > submodule doesn't know that some of it's commits are referenced by the > > supermodule. As I suggested elsewhere in the thread, that's easily > > fixed by making a refs/supermodule/commitXXXX file for each > > supermodule commit that references as particular submodule commit. > > I wouldn't call this "easily". Of course it is; when you write a supermodule commit you have it's hash, $SUPERMODULE_HASH, you have the commit-hash of the submodule commit you're referencing, $SUBMODULE_HASH. It's not really hard to do echo $SUBMODULE_HASH > submodule/.git/refs/supermodules/commit$SUPERMODULE_HASH Is it? Andy -- Dr Andrew Parkins, M Eng (Hons), AMIEE ``` ## Josef Weidendorfer, 2006-12-01 22:06 Subject: Re: [RFC] Submodules in GIT Message-ID: <200612012306.41410.Josef.Weidendorfer@gmx.de> URL: https://gitlist.dev/e/200612012306.41410.Josef.Weidendorfer%40gmx.de In-Reply-To: ``` On Friday 01 December 2006 21:13, Linus Torvalds wrote: > > There seems to be some agreement to store the commit id of > > the submodule instead of a plain tree id in the supermodules tree object, and > > that all objects that are reachable from this commit are made part of the > > supermodule repository (either fetched or via alternates). Do you agree? > > Well, I would actually argue that you may often want to have a supermodule > and then at least have the _option_ to decide to not fetch all the > submodules. If you want to allow this, you have to be able to cut off fetching the objects of the supermodule at borders to given submodules, the ones you do not want to track. With "border" I mean the submodule commit in some tree of the supermodule. This looks a little bit like a shallow clone, where you introduce graft points at the border to some of the submodule's object DAGs. But I am not sure that this is scalable: for supermodules with a large number of submodules you are not interested in, your graft file would grow very fast, as there will be new borders with every change in some submodule, which happens to be tracked in the supermodule. So IMHO, instead of a huge graft file, you want to have a fast way to check at a submodule border which submodule this given border is going into. Then, at fetch time, you easily can decide that you do not want to fetch any object from the submodule. Otherwise, you would have to ask the remote end at cloning time: "Is this commit from some submodule I am locally not interested in?" So I think we should introduce a submodule namespaces in supermodules. And at every border from super- to submodules, the name of the submodule we are going into should be specified. Which actually means that we need to introduce a "submodule" object, and trees of a supermodule can have such submodule objects as borders into a submodule. In a submodule object, of course we have the SHA1 of the commit into the submodule DAG, and there would be the global unique name we have choosen for this submodule in this supermodule. Something like submodule: gcc commit: 6287376... Before cloning a supermodule, you should be able to list the names of the submodules available, and select the submodules you want to have cloned together with the supermodule. > Ie, you very much should be able to to do > > git clone supermodule/that/one/submodule > my-own-version-of-submodule > > to clone a submodule _without_ getting anything else (but still get all > the work you did within he submodule - very much including your own > private branch work). So in the example, "that/one/submodule" is _not_ the path of the working tree which happens to be the root of the submodule at current supermodule HEAD, but the unique name from the submodule namespace. This is important, as you should be able to move the root of a submodule inside of your supermodule like moving any other file or directory. I.e. for every supermodule commit, the path to the root directory of a given submodule can change, making it useless as a name for a submodule selection at clone time. ``` ## Martin Waitz, 2006-12-01 22:08 Subject: Re: [RFC] Submodules in GIT Message-ID: <20061201220821.GL18810@admingilde.org> URL: https://gitlist.dev/e/20061201220821.GL18810%40admingilde.org In-Reply-To: <200612012154.33834.andyparkins@gmail.com> ``` hoi :) On Fri, Dec 01, 2006 at 09:54:32PM +0000, Andy Parkins wrote: > On Friday 2006, December 01 21:37, Martin Waitz wrote: > > > > I'm going to guess by reachability analysis, you mean that the > > > submodule doesn't know that some of it's commits are referenced by the > > > supermodule. As I suggested elsewhere in the thread, that's easily > > > fixed by making a refs/supermodule/commitXXXX file for each > > > supermodule commit that references as particular submodule commit. > > > > I wouldn't call this "easily". > > Of course it is; when you write a supermodule commit you have it's hash, > $SUPERMODULE_HASH, you have the commit-hash of the submodule commit you're > referencing, $SUBMODULE_HASH. It's not really hard to do > > echo $SUBMODULE_HASH > > submodule/.git/refs/supermodules/commit$SUPERMODULE_HASH I guess you are aware that you have to scan _all_ trees inside _all_ supermodule commits for possible references. So what do you do with deleted submodules? You wouldn't want them to still sit around in your working directory, but you still have to preserve them. -- Martin Waitz ``` ## Martin Waitz, 2006-12-01 22:12 Subject: Re: [RFC] Submodules in GIT Message-ID: <20061201221230.GM18810@admingilde.org> URL: https://gitlist.dev/e/20061201221230.GM18810%40admingilde.org In-Reply-To: <200612012306.41410.Josef.Weidendorfer@gmx.de> ``` hoi :) On Fri, Dec 01, 2006 at 11:06:40PM +0100, Josef Weidendorfer wrote: > > Well, I would actually argue that you may often want to have a > > supermodule and then at least have the _option_ to decide to not > > fetch all the submodules. > > If you want to allow this, you have to be able to cut off fetching the > objects of the supermodule at borders to given submodules, the ones you > do not want to track. With "border" I mean the submodule commit in some > tree of the supermodule. I don't think this is something special to submodules. There has been interest in checking out only a part of the tree even before talking about submodules and I really think this feature should be independent to submodules. -- Martin Waitz ``` ## Josef Weidendorfer, 2006-12-01 22:26 Subject: Re: [RFC] Submodules in GIT Message-ID: <200612012326.22779.Josef.Weidendorfer@gmx.de> URL: https://gitlist.dev/e/200612012326.22779.Josef.Weidendorfer%40gmx.de In-Reply-To: <20061201221230.GM18810@admingilde.org> ``` On Friday 01 December 2006 23:12, Martin Waitz wrote: > hoi :) > > On Fri, Dec 01, 2006 at 11:06:40PM +0100, Josef Weidendorfer wrote: > > > Well, I would actually argue that you may often want to have a > > > supermodule and then at least have the _option_ to decide to not > > > fetch all the submodules. > > > > If you want to allow this, you have to be able to cut off fetching the > > objects of the supermodule at borders to given submodules, the ones you > > do not want to track. With "border" I mean the submodule commit in some > > tree of the supermodule. > > I don't think this is something special to submodules. There has been > interest in checking out only a part of the tree even before talking > about submodules and I really think this feature should be independent > to submodules. It's not about checking out part of the tree, it's about fetching only part of the objects: If you have a slow modem and want to clone a supermodule, you are not interested in fetching all the objects from some submodules. So it is more like a shallow clone. But even here, submodules are special as you have defined borders between supermodule and submodules. This gives you the freedom the introduce a submodule namespace, and allows you to point to a submodule: "I do not want you!". With shallow clone, there you do not have this option, so there, you need to use something like grafting. BTW: In your submodule implementation, is the user allowed to change the relative path of the root of some submodule, e.g. with "git-mv" ? Josef ``` ## Linus Torvalds, 2006-12-01 22:26 Subject: Re: [RFC] Submodules in GIT Message-ID: URL: https://gitlist.dev/e/Pine.LNX.4.64.0612011423100.3695%40woody.osdl.org In-Reply-To: <200612012306.41410.Josef.Weidendorfer@gmx.de> ``` On Fri, 1 Dec 2006, Josef Weidendorfer wrote: > > > > Well, I would actually argue that you may often want to have a supermodule > > and then at least have the _option_ to decide to not fetch all the > > submodules. > > If you want to allow this, you have to be able to cut off fetching the > objects of the supermodule at borders to given submodules, the ones you > do not want to track. With "border" I mean the submodule commit in some > tree of the supermodule. > > This looks a little bit like a shallow clone No. I would say that it looks more like a "partial checkout" than a shallow clone. A shallow clone limits the data in "time" - we have _some_ data, but we don't have all of the history of that data. In contrast, a submodule that we don't fetch is an all-or-nothing situation: we simply don't have the data at all, and it's really a matter of simply not recursing into that submodule at all - much more like not checking out a particular part of the tree. So if a shallow clone is a "limit in time", a lack of a module (or a lack of a checkout for a subtree in general - you could certainly imagine doing the same thing even _within_ a git repository, and indeed, we did discuss exactly that at one point in time) is more of a "limit in space". ``` ## sf, 2006-12-01 22:35 Subject: Re: [RFC] Submodules in GIT Message-ID: <4570AE4E.8000600@stephan-feder.de> URL: https://gitlist.dev/e/4570AE4E.8000600%40stephan-feder.de In-Reply-To: ``` Linus Torvalds wrote: > > On Fri, 1 Dec 2006, sf wrote: >> Linus Torvalds wrote: >> ... >>> Think of it this way: one common use for submodules is really to just >>> (occasionally) track somebody elses code. The submodule should be a >>> totally pristine copy from somebody else (ie it might be the "intel driver >>> for X.org" submodule, maintained within intel), and the supermodule just >>> refers to it indirectly (ie the supermodule might be the "Fedora Core X >>> group" which contains all the different drivers from different people). >> Could you please be a little bit more specific about how you would store the >> "pristine copy". > > Note that it's not necessarily "pristine", since the submodule clearly is > a local git repository in its own right. So like _any_ git repository, you > can (and may well end up) having your own local branches in the submodule, > with your own local modifications. > > So I'm not claiming that a submodule must always match some external git > tree 100%, and that it must be read-only or anything like that. I'm just > saying that I suspect that quite often, one of the MOST IMPORTANT parts is > that the submodule is really something that somebody else technically > maintains, and that this is actually one of the _reasons_ why it is a > submodule in the first place. > > For example, a lot of projects end up having some kind of "library > component" as a submodule. Take something like a video player project, > which would have something like ffmpeg as a submodule, not because you'd > maintain ffmpeg yourself, but simply because (let's say) the library > interface changes enough, or you need a specific version with some of your > own fixes that haven't been released widely yet, so you want to carry all > the libraries you need _with_ you, even though you don't really maintain > that submodule. You at most have some small extensions of your own. > > Now, in this situation, it's relaly really _important_ that the submodule > really is totally independent of the supermodule, for several reasons. > > For example, since you don't "really" own that project, carrying around > your own fixes is really really painful. We know it happens all the time, > and a lot of projects end up needing their own version, but the _last_ > thing you want is to be in merge hell all the time. So as a supermodule > maintainer, the best possible thing for you is to be able to push back > those local changes to the original project maintainer, so that you > _don't_ have to maintain your own changes. True. But if you need the changes to the submodule for your supermodule to function, and upstream either does not want to merge your changes or the merge will be available only after a long time, then what is the alternative? You must be able to keep local changes, and you must be able to keep pulling from upstream. Of course, what you describe is the ideal case: You find a bug, push the fix upstream, and in no time at all your fix is merged and you can just pull a new version into your superproject, but that might be wishful thinking. > But you need to realize that the real maintainer of the submodule is > TOTALLY UNINTERESTED in your supermodule. He's not going to maintain it, > and in fact, if you have anything in the submodule that ends up talking > about your supermodule, that's just going to make it a lot less likely > that the upstream maintainer will ever pull your changes. He might take a > diff from you, but in a perfect world, you'd actually be able to tell him: > > "Hey, I've got a git repository with a few fixes to your ffmpeg git tree, > please pull from git://myhost.com/submodule.git to get these fixes: > > ... explanation of fixes and commits that are relevant to > ffmpeg, and have nothing to do with the supermodule, except > that you need those bug-fixes because you _use_ ffmpeg ... > > Thanks" > > See? No! All you need is a naming scheme to address the commit of the subproject that should be pulled. The extreme case would be to just address it with its id (well, currently you cannot do that with git pull, but that is fixable). But I already proposed a syntax for naming commits which are "hidden" in a superproject: Just name the path as described in git-rev-parse and append double slashes (to indicate that you mean the commit, not the tree it contains). So no manual work needs be done by upstream. [snipped: about independence of submodule branches] >> There seems to be some agreement to store the commit id of >> the submodule instead of a plain tree id in the supermodules tree object, and >> that all objects that are reachable from this commit are made part of the >> supermodule repository (either fetched or via alternates). Do you agree? > > Well, I would actually argue that you may often want to have a supermodule > and then at least have the _option_ to decide to not fetch all the > submodules. [transmeta example snipped] > So no, I don't think the supermodule should even _force_ people to always > get all the submodules. It migth be the default case, but at the same > time, it's just being polite to let users decide on their own whether they > really want _all_ of the build infrastructure sources. If you want to track some chosen submodules there are two easy solutions: 1. If you want to track their state as it appears from the supermodule's view, pull from master:// 2. If you want to track their state from their own development branches, pull from /master Can you see the difference? >> If I understand you correctly you cannot make any changes to the submodules >> code _in the supermodule's repository_, no bugfixes, no extensions, no >> adaptions, nothing. Do you mean that? > > Yes. I think you should make all changes _within_ the submodule, because > the submodule should still be an independent git tree in its own right. Every commit is a git tree in its own right, is it not? [description of independent submodule development snipped] > And the importance of keeping the submodule independent is partly just > stability and sanity, but partly also scalability. For example, the > "index" in a supermodule should NOT include the indexes of all the > submodules. That's really important, because the index doesn't really > scale. Things do slow down with large indexes. > > For example, git can handle tens of thousands of files easily. I suspect > it scales well to hundreds of thousands of filenames. But with > supermodules, you really can end up in the situation where you have _tens_ > of these submodules, maybe even hundreds. And if you try to maintain one > unified index for the _whole_ thing, I guarantee you that you'll start > feeling the pain. Indexing millions of files is just not going to be > pretty. I am not sure I understand what you say. 1. If you are working on a submodule, then the supermodule never enters the picture. You are working independently. So far, so good. 2. If you are working on the supermodule, git will not be able to function? How would you work without submodules, in which case you would have simply one large project? > So just from a git stability and scalability point, it's important to keep > subprojects _separate_. There is obviously integration stuff, but they > should still be seen as truly independent projects. Even the supermodule > should have clearly its own life even _regardless_ of submodules, because > (as I said) quite often you may want the supermodule, but you don't want > to have _all_ of the submodules. > > But it's more than that stability and scalability thing too - keeping them > separate is what allows you to do pulls and pushes on an individual > subproject basis, and have people really work at that level. For example, > if you're the compiler guy at a company, you really do want to work with > other compiler people _outside_ the company, but you sure as hell may not > be able to give them access to your supermodule. But you may want to work > on _just_ the compiler parts (or at least share some branches in public), > which means that the subproject really has to be able to work > _independently_ of the supermodule. I totally agree. When I try to explain why submodules work that only exist as part of one or more supermodules, I do not mean to say that you cannot or should not have independent branches or repositories for the submodules' code. > So "independent" here is really key, for several reasons. And that all > means, for example, that here must NEVER be any "backpointers". A > subproject really can _never_ have backpointers to the superproject, > because that fundamentally means that the above kind of "compiler guy > works on the compiler subproject in public" cannot work, if your > supermodule isn't public. I took that for granted: from a commit you only ever look backwards (in time/history dimension) or downwards (in content dimension). Regards ``` ## Martin Waitz, 2006-12-01 22:40 Subject: Re: [RFC] Submodules in GIT Message-ID: <20061201224021.GN18810@admingilde.org> URL: https://gitlist.dev/e/20061201224021.GN18810%40admingilde.org In-Reply-To: <200612012326.22779.Josef.Weidendorfer@gmx.de> ``` hoi :) On Fri, Dec 01, 2006 at 11:26:22PM +0100, Josef Weidendorfer wrote: > It's not about checking out part of the tree, it's about fetching only > part of the objects: If you have a slow modem and want to clone a > supermodule, you are not interested in fetching all the objects from > some submodules. So when you want to suppress one submodule, how is this not about only checking out part of the tree? Ok, you also want to avoid downloading the submodule, but you first have to solve the partial checkout. > BTW: In your submodule implementation, is the user allowed to change the > relative path of the root of some submodule, e.g. with "git-mv" ? In principle: yes. However there are some links between both repositories that have to be updated manually (for the shared object repository and for ignoring submodule files in the supermodule). But I expect that much of this configuration stuff will vanish when submodules are better integrated in git. Rename detection for submodules would be another interesting thing to have. It should be much easier as for files because we can simply check for common ancestors and do not have to guess based on the diff. -- Martin Waitz ``` ## sf, 2006-12-01 22:41 Subject: Re: [RFC] Submodules in GIT Message-ID: <4570AF8F.1000801@stephan-feder.de> URL: https://gitlist.dev/e/4570AF8F.1000801%40stephan-feder.de In-Reply-To: ``` Linus Torvalds wrote: ... > In contrast, a submodule that we don't fetch is an all-or-nothing > situation: we simply don't have the data at all, and it's really a matter > of simply not recursing into that submodule at all - much more like not > checking out a particular part of the tree. If you do not want to fetch all of the supermodule then do not fetch the supermodule. Instead fetch only the submodules you are interested in. You do not have to fetch the whole repository. Regards ``` ## Josef Weidendorfer, 2006-12-01 22:55 Subject: Re: [RFC] Submodules in GIT Message-ID: <200612012355.03493.Josef.Weidendorfer@gmx.de> URL: https://gitlist.dev/e/200612012355.03493.Josef.Weidendorfer%40gmx.de In-Reply-To: ``` On Friday 01 December 2006 23:26, Linus Torvalds wrote: > > On Fri, 1 Dec 2006, Josef Weidendorfer wrote: > > > > > > Well, I would actually argue that you may often want to have a supermodule > > > and then at least have the _option_ to decide to not fetch all the > > > submodules. > > > > If you want to allow this, you have to be able to cut off fetching the > > objects of the supermodule at borders to given submodules, the ones you > > do not want to track. With "border" I mean the submodule commit in some > > tree of the supermodule. > > > > This looks a little bit like a shallow clone > > No. > > I would say that it looks more like a "partial checkout" than a shallow > clone. > > A shallow clone limits the data in "time" - we have _some_ data, but we > don't have all of the history of that data. > > In contrast, a submodule that we don't fetch is an all-or-nothing > situation: we simply don't have the data at all, and it's really a matter > of simply not recursing into that submodule at all - much more like not > checking out a particular part of the tree. OK. I still think it should be about "limit in space" regarding the objects in the local repository. For a project containing "gcc" as submodule, and I am not interested in this submodule, there should be a way to not need to fetch all the objects from the gcc submodule at clone time. What about my other argument for a submodule namespace: You want to be able to move the relative root path of a submodule inside of your supermodule, but yet want to have a unique name for the submodule: - to be able to just clone a submodule without having to know the current position in HEAD - more practically, e.g. to be able to name a submodule independent from any current commit you are on in the supermodule, e.g. to be able to store some meta information about a submodule: - "Where is the official upstream of this submodule?" - "Should git allow to commit rewind actions of this submodule in the supermodule?" (which, AFAICS, exactly has the same problems as publishing a rewound branch: you will get into merge hell when you want to pull upstream changes into the supermodule) - "Should this submodule be checked out?" and so on. ``` ## Josef Weidendorfer, 2006-12-01 23:03 Subject: Re: [RFC] Submodules in GIT Message-ID: <200612020003.14377.Josef.Weidendorfer@gmx.de> URL: https://gitlist.dev/e/200612020003.14377.Josef.Weidendorfer%40gmx.de In-Reply-To: <4570AF8F.1000801@stephan-feder.de> ``` On Friday 01 December 2006 23:41, sf wrote: > Linus Torvalds wrote: > ... > > In contrast, a submodule that we don't fetch is an all-or-nothing > > situation: we simply don't have the data at all, and it's really a matter > > of simply not recursing into that submodule at all - much more like not > > checking out a particular part of the tree. > > If you do not want to fetch all of the supermodule then do not fetch the > supermodule. Instead fetch only the submodules you are interested in. > You do not have to fetch the whole repository. But what, when I *want* to fetch the supermodule because of the source tree which is only available in the supermodule? Of course, you can argue that the only objects in trees of a supermodule should be submodule commits, but this is quite restricting the usage of supermodules. See further arguments for a submodule namespace in my other mail. You probably want to specify policies about submodule handling. This information has to be indexed by some name independent from a supermodule commit. Josef ``` ## Martin Waitz, 2006-12-01 23:07 Subject: Re: [RFC] Submodules in GIT Message-ID: <20061201230732.GO18810@admingilde.org> URL: https://gitlist.dev/e/20061201230732.GO18810%40admingilde.org In-Reply-To: <200612012355.03493.Josef.Weidendorfer@gmx.de> ``` hoi :) On Fri, Dec 01, 2006 at 11:55:03PM +0100, Josef Weidendorfer wrote: > What about my other argument for a submodule namespace: > You want to be able to move the relative root path of a submodule > inside of your supermodule, but yet want to have a unique name > for the submodule: > - to be able to just clone a submodule without having to know > the current position in HEAD > - more practically, e.g. to be able to name a submodule > independent from any current commit you are on in the supermodule, > e.g. to be able to store some meta information about a submodule: > - "Where is the official upstream of this submodule?" you can always have a bare repository for all used modules lying around in some defined location. There is no need for a unique submodule-name. -- Martin Waitz ``` ## Linus Torvalds, 2006-12-01 23:09 Subject: Re: [RFC] Submodules in GIT Message-ID: URL: https://gitlist.dev/e/Pine.LNX.4.64.0612011505190.3695%40woody.osdl.org In-Reply-To: <4570AF8F.1000801@stephan-feder.de> ``` On Fri, 1 Dec 2006, sf wrote: > Linus Torvalds wrote: > ... > > In contrast, a submodule that we don't fetch is an all-or-nothing > > situation: we simply don't have the data at all, and it's really a matter > > of simply not recursing into that submodule at all - much more like not > > checking out a particular part of the tree. > > If you do not want to fetch all of the supermodule then do not fetch the > supermodule. So why do you want to limit it? There's absolutely no cost to saying "I want to see all the common shared infrastructure, but I'm actually only interested in this one submodule that I work with". Also, anybody who works on just the build infrastructure simply may not care about all the submodules. The submodules may add up to hundreds of gigs of stuff. Not everybody wants them. But you may still want to get the common build infrastructure. In other words, your "all or nothing" approach is (a) not friendly and (b) has no real advantages anyway, since modules have to be independent enough that you _can_ split them off for other reasons anyway. So forcing that "you have to take everything" mentality onyl has negatives, and no positives. Why do it? ``` ## Josef Weidendorfer, 2006-12-01 23:17 Subject: Re: [RFC] Submodules in GIT Message-ID: <200612020017.44275.Josef.Weidendorfer@gmx.de> URL: https://gitlist.dev/e/200612020017.44275.Josef.Weidendorfer%40gmx.de In-Reply-To: <20061201221230.GM18810@admingilde.org> ``` On Friday 01 December 2006 23:12, Martin Waitz wrote: > hoi :) > > On Fri, Dec 01, 2006 at 11:06:40PM +0100, Josef Weidendorfer wrote: > > > Well, I would actually argue that you may often want to have a > > > supermodule and then at least have the _option_ to decide to not > > > fetch all the submodules. > > > > If you want to allow this, you have to be able to cut off fetching the > > objects of the supermodule at borders to given submodules, the ones you > > do not want to track. With "border" I mean the submodule commit in some > > tree of the supermodule. > > I don't think this is something special to submodules. There has been > interest in checking out only a part of the tree even before talking > about submodules and I really think this feature should be independent > to submodules. After some thinking, a submodule namespace even is important for checking out only parts of a supermodule, exactly because the root of a submodule potentially can change at every commit. When checking out some arbitrary supermodule commit, how do you check that at some submodule border, the user did not want to check out the submodule at all? You need a way to check the DAG identity you are diving into at this border: lets say by going to the root commit of this DAG (!). And via this identity, you have to check whether the user had specified that he wants the submodule to be check out. Without any further meta information (indexed by a submodule name!), this information is only available from the checkout the user switched from, as there would be no file in the working tree from this submodule? Quite a pain. ``` ## Alan Chandler, 2006-12-01 23:23 Subject: Re: [RFC] Submodules in GIT Message-ID: <200612012323.54403.alan@chandlerfamily.org.uk> URL: https://gitlist.dev/e/200612012323.54403.alan%40chandlerfamily.org.uk In-Reply-To: <20061201203002.GJ18810@admingilde.org> ``` On Friday 01 December 2006 20:30, Martin Waitz wrote: > hoi :) > > Linus, you are a lot better in describing all my thoughts than I myself. > ;-) Yes Some of the clearest explanations described from a strategic point of view come from posts such as these. And he keeps saying he can't write documentation. -- Alan Chandler ``` ## Linus Torvalds, 2006-12-01 23:30 Subject: Re: [RFC] Submodules in GIT Message-ID: URL: https://gitlist.dev/e/Pine.LNX.4.64.0612011510290.3695%40woody.osdl.org In-Reply-To: <200612012355.03493.Josef.Weidendorfer@gmx.de> ``` On Fri, 1 Dec 2006, Josef Weidendorfer wrote: > > What about my other argument for a submodule namespace: > You want to be able to move the relative root path of a submodule > inside of your supermodule, but yet want to have a unique name > for the submodule: > - to be able to just clone a submodule without having to know > the current position in HEAD Umm? I don't get the issue. A submodule is a git repo in its own right, and you clone it exactly like you'd clone any other repo. It _does_ have a HEAD. It has it's own branches. It has everything. So when you clone a submodule, you always get all those branches. The supermodule will not _point_ to them all (the branches are local to the submodule, and _will_ depend on things like "which upstreams module am I tracking"), but they'll have to be there, exactly _because_ the submodule has an existence and is tracked on its own. In the trivial case where the submodule doesn't even _have_ any external existence at all (ie it's always maintained as _just_ a submodule, it would probably tend to have just one branch, and a clone would get whatever that branch is), but that's just a degenerate special case of the much richer "this submodule actually has a life of its own" case. > - more practically, e.g. to be able to name a submodule > independent from any current commit you are on in the supermodule, > e.g. to be able to store some meta information about a submodule: The current commit within the supermodule would be _totally_ invisible to the submodule. Of course, if HEAD _differs_ from that commit within the supermodule, then a "git diff" (when done from within the supermodule) should show that, but again, that's actually only as seen from the _supermodule_. > - "Where is the official upstream of this submodule?" That's entirely a question for the submodule. You cannot ask that question within the confines of the supermodule, because it's not even a relevant question in that context. Two different supermodule repositories may well decide to get their submodules from difference places, just because they got cloned from different places (or even just for practical reasons like "that other site is closer to me"). So the official upstream of a submodule must NOT be encoded inside the supermodule (or at least not within its _objects_). Exactly because the upstream location is not a "global" thing - it's per-repository, and thus must not be encoded in the global data (ie the objects). It should be be encoded in some _ephemeral_ place, eg in the ".git/config" file or in a ".git/remotes/origin"-like file (either in the supermodule or the submodule, and I would seriously suggest you do it within in the submodule itself, because you'll want it exactly when you decide to work on the submodule and upgrade _that_). > - "Should git allow to commit rewind actions of this submodule > in the supermodule?" (which, AFAICS, exactly has the same > problems as publishing a rewound branch: you will get into > merge hell when you want to pull upstream changes into the > supermodule) The only thing that a submodule must NOT be allowed to do on its own is pruning (and it's distant cousin "git repack -d"). You must always prune from the supermodule, because the submodule cannot really know on its own what references point into it. (There are alternatives. One alternative is to never allow rewinding - or deletion - of branches in a submodule, and thus solve the problem that way. That is the easier solution, because it also means that a "clone" of a supermodule can just recursively clone the submodules independently _without_ having to worry about reachability, but it's really _really_ draconian). > - "Should this submodule be checked out?" This, I think, requires too much configuration to say separately for every possible submodule, so I would suggest that the way to make that decision is: - "git clone" by default will fetch and check out all submodules (and obviously they have to be described some way outside of the object database, just so that you don't have to parse the _whole_ history of the _whole_ supermodule just to find all possible submodules. So the supermodule _will_ need some "list of submodules and where to get them" in a config file or other). - add a flag (possibly just re-use the current "-n" flag) that disables that recursive fetching of submodules entirely. - have a way to fetch individual submodules one-by-one (that capacity obviously has to be there anyway, since the "recursive" git clone has to be able to do it, so this is likely just "git clone" again, with just logic added to say "when you clone something and are _already_ within a superproject, the clonee becomes a subproject automatically" I dunno. And I'd also like to point out that things don't have to all work fully before we can do at least some cases of this. For example, if the initial version just always clones everything, big deal. I'm not saying that we have to have support for things like this on "Day 1", I'm just saying that I think people will want to be able to not fetch and check out everything, so the design should _allow_ for it. (But I also think that as long as submodules are independent enough, the "design" part should fall out on its own, and it just becomes a "small matter of programming" to actually get it to work). ``` ## sf, 2006-12-01 23:34 Subject: Re: [RFC] Submodules in GIT Message-ID: <4570BC07.4080203@stephan-feder.de> URL: https://gitlist.dev/e/4570BC07.4080203%40stephan-feder.de In-Reply-To: <20061201184801.GG18810@admingilde.org> ``` Martin Waitz wrote: > On Fri, Dec 01, 2006 at 06:33:12PM +0100, Stephan Feder wrote: >> We are in agreement about two fundamental parts of the implementation >> and their meaning: >> >> 1. A submodule is stored as a commit id in a tree object. >> >> 2. Every object that is reachable from the submodule's commit are >> reachable from the supermodule's repository. > > Correct. Good. For me that is the main point. As I said before the user interface is not so important because it can be changed anytime, but to change the object database later is close to impossible. ... > Let's see if I understand you correctly: > > You don't want to create an additional .git directory for the submodule > and just handle everything with one toplevel .git repository for the > whole project. Yes. > Without the .git directory, you of course do not have refs/heads inside > the submodule. Correct.. > So this is a different user-interface approach to submodules when > compared to my approach. But the basis is the same and both could > inter-operate. Big YES. > Now your submodule is no longer seen as an independent git repository > and I think this would cause problems when you want to push/pull between > the submodule and its upstream repository. You can always pick a single commit or several commits out of a larger repository and have a complete git repository. And I already explained how to push and pull even from within superprojects. > No technical problems, but UI-problems because now your submodule is > handled completly different to a "normal" repository. Yes and no. You can always have branches that are only concerned with submodules' code, say, in refs/heads/submodules//. "submodules" here is simply an example and has not deeper meaning. You could call it foo or whatever you like. Or you could use refs/heads// if it suits you. But if you mean the submodule as seen from the supermodule, then there is a difference. Naturally, because the concept of submodules is new to git. >>> Yes, but if it would have been integrated as a submodule it obviously >>> would have been committed to the xdiff submodule inside the git >>> repository. >> No. The xdiff submodule would only exist as part of the git repository. > > But you could still call the "xdiff" part of the git repository a > submodule. And then changes to the xdiff directory result in a new > submodule commit, even when there is no direct reference to it. > So you'd still "commit to the xdiff submodule". Let's make certain that we understand each other. I see a clear distinction between the submodule code in a supermodule branch (commits in the supermodule's tree and nothing else) and submodule branches which are independent of the superproject. Supermodule branches and submodule branches do not interact, only if I want them to. > >> You could, f.e., access the xdiff commit in git HEAD as HEAD:xdiff// >> (again my proposed syntax). HEAD:xdiff//~2:xemit.c would give you the >> grandparent of xemit.c in the xdiff submodule. > > git-cat-file commit HEAD:xdiff already works out of the box (even > cat-file tree to get the submodule tree). But up to now revision > parsing follows the file name only once. > > What about just separating things with "/"? > > commit HEAD > tree HEAD/ > blob HEAD/Makefile > commit HEAD/xdiff > tree HEAD/xdiff/ > blob HEAD/xdiff~2/xemit.c > > this may add some confusion when used with hierarchical branches, but > it's still unique: > > refs/heads/master/xdiff/xemit.c > > Just use as many path components until a matching reference is found, > then start peeling. > Or just use / between super and submodule: > > refs/heads/master:xdiff/xemit.c > > I think this is easier to read then > > refs/heads/master:xdiff//:xemit.c The double slashes is the only way I can think of that clearly indicates that I do not mean the contents named by the path, but the commit that you find there. Once you have named a commit in that way, you can continue to apply other revision naming suffixes, paths, and so on. Let's try. What does git cat-file -p master:dir/sub//^^^:sub/dir/sub//^:dir/file mean? Explanation: Take branch master and go to path dir/sub. There you will find a commit. Take its grand-grandparent and go to path sub/dir/sub (the first sub is a subproject as well but we do not care). There you will, again, find a commit. Take its parent and go to path dir/file which happens to be a blob the contents of which you want to cat. In reality you will never see these kinds of complex paths. Have you ever seen something like git cat-file -p bd2c39f58f915af532b488c5bda753314f0db603~12^{commit}^2^5~8^2~308:README ? >> If you mean the submodule repository created by init-module I >> understand. But why create this "helper repository at all"? > > Because it helps "normal" git operations ;-) Let's see. I still have to try. Regards ``` ## Josef Weidendorfer, 2006-12-01 23:36 Subject: Re: [RFC] Submodules in GIT Message-ID: <200612020036.08826.Josef.Weidendorfer@gmx.de> URL: https://gitlist.dev/e/200612020036.08826.Josef.Weidendorfer%40gmx.de In-Reply-To: ``` On Saturday 02 December 2006 00:09, Linus Torvalds wrote: > > On Fri, 1 Dec 2006, sf wrote: > > Linus Torvalds wrote: > > ... > > > In contrast, a submodule that we don't fetch is an all-or-nothing > > > situation: we simply don't have the data at all, and it's really a matter > > > of simply not recursing into that submodule at all - much more like not > > > checking out a particular part of the tree. > > > > If you do not want to fetch all of the supermodule then do not fetch the > > supermodule. > > So why do you want to limit it? There's absolutely no cost to saying "I > want to see all the common shared infrastructure, but I'm actually only > interested in this one submodule that I work with". So you are for a global submodule namespace in supermodule repositories, do I understand correctly? Otherwise, how would you specify the submodules at clone time given the ability that submodule roots can have relative path changed arbitrarily between commits? ``` ## sf, 2006-12-01 23:49 Subject: Re: [RFC] Submodules in GIT Message-ID: <4570BFA4.8070903@stephan-feder.de> URL: https://gitlist.dev/e/4570BFA4.8070903%40stephan-feder.de In-Reply-To: ``` Linus Torvalds wrote: > > On Fri, 1 Dec 2006, sf wrote: >> Linus Torvalds wrote: >> ... >>> In contrast, a submodule that we don't fetch is an all-or-nothing >>> situation: we simply don't have the data at all, and it's really a matter >>> of simply not recursing into that submodule at all - much more like not >>> checking out a particular part of the tree. >> If you do not want to fetch all of the supermodule then do not fetch the >> supermodule. > > So why do you want to limit it? There's absolutely no cost to saying "I > want to see all the common shared infrastructure, but I'm actually only > interested in this one submodule that I work with". If you need a common infrastructure to be able to work with the submodule, then the submodule is not independent of of the supermodule. I see a contradiction in your requirements. > Also, anybody who works on just the build infrastructure simply may not > care about all the submodules. The submodules may add up to hundreds of > gigs of stuff. Not everybody wants them. But you may still want to get the > common build infrastructure. See above. > In other words, your "all or nothing" approach is > (a) not friendly > and > (b) has no real advantages anyway, since modules have to be independent > enough that you _can_ split them off for other reasons anyway. > > So forcing that "you have to take everything" mentality onyl has > negatives, and no positives. Why do it? (There have been lots of use cases for shallow clones but for a long time git did not support them). If you can extend this partial fetch feature to the non-subproject case I would agree with your reasoning. What makes the subprojects so special in this regard. Do I have to turn a plain tree into a subproject to be able to ignore it? Once you can restrict fetches to parts of the contents you get the ability to restrict fetches to the "common infrastructure" and selected submodules for free. Regards Stephan ``` ## Linus Torvalds, 2006-12-02 00:12 Subject: Re: [RFC] Submodules in GIT Message-ID: URL: https://gitlist.dev/e/Pine.LNX.4.64.0612011540010.3695%40woody.osdl.org In-Reply-To: <200612020036.08826.Josef.Weidendorfer@gmx.de> ``` On Sat, 2 Dec 2006, Josef Weidendorfer wrote: > > So you are for a global submodule namespace in supermodule repositories, > do I understand correctly? > > Otherwise, how would you specify the submodules at clone time given the > ability that submodule roots can have relative path changed arbitrarily > between commits? The only _true_ namespace would be the SHA1 of the commit (and maybe allow a pointer to a tag too, but the namespace ends up being the same). How to _find_ a repository that contains that SHA1 must be left to higher levels. After all, repositories move around, and the place you found them originally is not a stable name. So within the supermodule, on a "git object" level, a submodule should just be named by the SHA1 that was it's HEAD when it was committed within the supermodule. So in the "tree object", you'd see something like the following when you go "git ls-tree HEAD" on the superproject: ... 100644 blob 08602f522183dc43787616f37cba9b8af4e3dade xdiff-interface.c 100644 blob 1346908bea31319aabeabdfd955e2ea9aab37456 xdiff-interface.h 040000 tree 959dd5d97e665998eb26c764d3a889ae7903d9c2 xdiff 050000 link 0215ffb08ce99e2bb59eca114a99499a4d06e704 xyzzy where that 050000 is the new magic type (I picked one out of my *ss: it's not a valid type for a file mode, so it's a godo choice, but it could be anythign that cannot conflict with a real file), which just specifies the "link" part. The SHA1 is the SHA1 of the commit, and the "xyzzy" is obviously just the name within the directory of the submodule. That's all that is actually required for a lot of git commands that already expect all objects to be available (ie "git checkout", "git diff" etc). It only gets interesting for commands that fetch new objects, ie do a "pull/fetch" op, and you'd need to know where/how to fetch new objects for the xyzzy subproject, so that's a "naming" issue. You have a few choices: - get all the objects directly from the subproject as if it was one big project. I actually think this sucks. Why? Because it puts an insane load on the server side, which basically needs to traverse the object list of the _sum_ of all projects. An initial clone (or a really big pull, which comes to the same thing) would be absolutely horrendous So I'd strongly argue against that approach, for scalability reasons. So instead, you should really try to do pulls etc one git repo at a time: - take the "list of subprojects" from the supermodule, and pull them all one by one. This again makes subprojects "less seamless", and makes each subproject more of a separate thing, with the project list gotten from the superproject and parsed separately. But it means you have none of the scalability problems, since you never see things as one huge project with millions of files and even more objects. The second approach also means that you can see the "supermodule" support in git as less of a "plumbing" thing, and it's largely just a thin veneer around the core plumbing that really doesn't understand about multiple repositories at all (apart from the single "link" extension in the tree object), and it's really just scripting to get the subprojects to "look" like one thing, when they really are pretty much independent. ``` ## Josef Weidendorfer, 2006-12-02 00:14 Subject: Re: [RFC] Submodules in GIT Message-ID: <200612020114.42858.Josef.Weidendorfer@gmx.de> URL: https://gitlist.dev/e/200612020114.42858.Josef.Weidendorfer%40gmx.de In-Reply-To: ``` On Saturday 02 December 2006 00:30, Linus Torvalds wrote: > On Fri, 1 Dec 2006, Josef Weidendorfer wrote: > > > > What about my other argument for a submodule namespace: > > You want to be able to move the relative root path of a submodule > > inside of your supermodule, but yet want to have a unique name > > for the submodule: > > - to be able to just clone a submodule without having to know > > the current position in HEAD > > Umm? I don't get the issue. A submodule is a git repo in its own right, > and you clone it exactly like you'd clone any other repo. It _does_ have a > HEAD. It has it's own branches. It has everything. I just thought about the case when you want to clone a submodule directly out of the supermodule repository, at a given realive path. And that can be changing. Of course, every project which happens to be submodule of some supermodule, also can have its own repository, as it is fully independent. And then, you of course can clone from without any knowledge of its relative position in the supermodule. > In the trivial case where the submodule doesn't even _have_ any external > existence at all (ie it's always maintained as _just_ a submodule, it > would probably tend to have just one branch, and a clone would get > whatever that branch is), but that's just a degenerate special case of the > much richer "this submodule actually has a life of its own" case. Yes. > > - more practically, e.g. to be able to name a submodule > > independent from any current commit you are on in the supermodule, > > e.g. to be able to store some meta information about a submodule: > > The current commit within the supermodule would be _totally_ invisible to > the submodule. Of course. Yet, you need some name to store meta information of submodules into some config file of the supermodule, like whether you want to have it checked out (see below). In that case, such a name for a submodule does not have to be global in the supermodule project... > > - "Should git allow to commit rewind actions of this submodule > > in the supermodule?" (which, AFAICS, exactly has the same > > problems as publishing a rewound branch: you will get into > > merge hell when you want to pull upstream changes into the > > supermodule) > > The only thing that a submodule must NOT be allowed to do on its own is > pruning (and it's distant cousin "git repack -d"). You must always prune > >from the supermodule, because the submodule cannot really know on its own > what references point into it. Yes. I just gave an example of a policy some project may want for submodule handling. > > - "Should this submodule be checked out?" > > This, I think, requires too much configuration to say separately for every > possible submodule, so I would suggest that the way to make that decision > is: > > - "git clone" by default will fetch and check out all submodules (and > obviously they have to be described some way outside of the object > database, just so that you don't have to parse the _whole_ history of > the _whole_ supermodule just to find all possible submodules. So the > supermodule _will_ need some "list of submodules and where to get them" > in a config file or other). Exactly. And in this list, you have to specify names. The thing I wanted to discuss is whether such names would need to be globally unique in the project containing submodles, or not. If yes, it IMHO makes a lot of sense to introduce "submodule objects" which contain these submodule names, and which are used as pointers to submodule commits in supermodule trees. ``` ## Linus Torvalds, 2006-12-02 00:33 Subject: Re: [RFC] Submodules in GIT Message-ID: URL: https://gitlist.dev/e/Pine.LNX.4.64.0612011621380.3695%40woody.osdl.org In-Reply-To: <200612020114.42858.Josef.Weidendorfer@gmx.de> ``` On Sat, 2 Dec 2006, Josef Weidendorfer wrote: > > > > The current commit within the supermodule would be _totally_ invisible to > > the submodule. > > Of course. > > Yet, you need some name to store meta information of submodules > into some config file of the supermodule, like whether you want to have > it checked out (see below). Yes, you do need to have a list of submodules somewhere, and you'd need to maintain that separately. One of the results of having the submodules be independent from the supermodule is that it's not all "automatically integrated", and thus the supermodule does end up having to have things like that maintained separately. And yes, if you screw that up, you wouldn't be able to fetch submodules properly etc, even if you see the supermodule, and yes, this sounds more like the CVS "Entries" kind of file that is more "tacked on" than really deeply integrated. But I think the separation is _more_ than worth the fact that you can see things being separate. In fact, I'm very much arguing for keeping things as separate as possible, while just integrating to the smallest possible degree (just _barely_ enough that you can do things like "git clone" and it will fetch multiple repositories and put them all in the right places, and "git diff" and friends will do reasonably sane things). Keep it simple, stupid. > > - "git clone" by default will fetch and check out all submodules (and > > obviously they have to be described some way outside of the object > > database, just so that you don't have to parse the _whole_ history of > > the _whole_ supermodule just to find all possible submodules. So the > > supermodule _will_ need some "list of submodules and where to get them" > > in a config file or other). > > Exactly. And in this list, you have to specify names. Yes. > The thing I wanted to discuss is whether such names would need to be globally > unique in the project containing submodles, or not. My preference would be for it to be "local", just because (as I mentioned), with mirroring etc, it might well be that you want to fetch things from the _closest_ repository. That's really not a global decision, it's a local one. > If yes, it IMHO makes a lot of sense to introduce "submodule objects" which contain > these submodule names, and which are used as pointers to submodule commits in > supermodule trees. You could do it that way, and then it would be global. It would work, and in many ways it would probably be "simpler" on a supermodule level. The advantage of a global namespace is that you can much more easily update it - "git fetch" will just fetch the new file(s) that describe the subprojects very naturally if they are all global. Putting them in a local .git/config file has it's advantages (see above), but it also makes it very hard to version them, and to update the list - it would have to become manual. There are possibly combinations of the two approaches: have a "global namespace" that describes the canonical place to get the subprojects, but have some way to add local "translation" of the canonical names into locally preferred versions (eg you could just have a way to say "this is the local mirror for that global canonical place") Maybe that would work? ``` ## Andy Parkins, 2006-12-02 09:22 Subject: Re: [RFC] Submodules in GIT Message-ID: <200612020922.43832.andyparkins@gmail.com> URL: https://gitlist.dev/e/200612020922.43832.andyparkins%40gmail.com In-Reply-To: ``` On Saturday 2006, December 02 00:12, Linus Torvalds wrote: > 100644 blob 08602f522183dc43787616f37cba9b8af4e3dade xdiff-interface.c > 100644 blob 1346908bea31319aabeabdfd955e2ea9aab37456 xdiff-interface.h > 040000 tree 959dd5d97e665998eb26c764d3a889ae7903d9c2 xdiff > 050000 link 0215ffb08ce99e2bb59eca114a99499a4d06e704 xyzzy > > where that 050000 is the new magic type (I picked one out of my *ss: it's > not a valid type for a file mode, so it's a godo choice, but it could be > anythign that cannot conflict with a real file), which just specifies the > "link" part. The SHA1 is the SHA1 of the commit, and the "xyzzy" is > obviously just the name within the directory of the submodule. Can I argue that the hash in that object should actually be to a real object in the supermodule repository rather than a link? Then THAT object would contain the hash? So in your above example: 100644 blob 08602f522183dc43787616f37cba9b8af4e3dade xdiff-interface.c 100644 blob 1346908bea31319aabeabdfd955e2ea9aab37456 xdiff-interface.h 040000 tree 959dd5d97e665998eb26c764d3a889ae7903d9c2 xdiff 050000 link a7f26495b7b7e32bf949efbd91ee32267b792cba xyzzy And then the local object a7f26495b7b7e32bf949efbd91ee32267b792cba would contain your original hash 0215ffb08ce99e2bb59eca114a99499a4d06e704. The reason I suggest this as without out it the "link" object is the only hash in the tree that doesn't point to a valid object. The contents of objects is entirely arbitrary so it's perfectly okay for that to contain a hash that won't dereference to a real object in the supermodule. The main advantage of this is (I think) that git-prune, git-fsck, and whatever else relies on tree objects all being real, don't need to be modified at all. It also gives you scope to later add fields to the "link" object if you wanted. Andy -- Dr Andrew Parkins, M Eng (Hons), AMIEE ``` ## Andy Parkins, 2006-12-02 09:27 Subject: Re: [RFC] Submodules in GIT Message-ID: <200612020927.50787.andyparkins@gmail.com> URL: https://gitlist.dev/e/200612020927.50787.andyparkins%40gmail.com In-Reply-To: ``` On Saturday 2006, December 02 00:33, Linus Torvalds wrote: > Yes, you do need to have a list of submodules somewhere, and you'd need to > maintain that separately. One of the results of having the submodules be Why? You just recursively search for every "link" object in the supermodule. That tells you which submodules you need and where they should be. During a supermodule clone, it can tell the client end to start a new clone with the correct path because it knows what the local path is at that moment. Andy -- Dr Andrew Parkins, M Eng (Hons), AMIEE ``` ## Andy Parkins, 2006-12-02 10:04 Subject: Re: [RFC] Submodules in GIT Message-ID: <200612021004.22236.andyparkins@gmail.com> URL: https://gitlist.dev/e/200612021004.22236.andyparkins%40gmail.com In-Reply-To: <20061201220821.GL18810@admingilde.org> ``` On Friday 2006, December 01 22:08, Martin Waitz wrote: > > echo $SUBMODULE_HASH > > > submodule/.git/refs/supermodules/commit$SUPERMODULE_HASH > > I guess you are aware that you have to scan _all_ trees inside _all_ > supermodule commits for possible references. No you don't; you do it as part of the appropriate normal operations. * supermodule commit - scan the current tree for "link" objects in the tree. If you find one write the reference in the submodule. * adding a new submodule - if this is a new submodule there can't be any references in the supermodule already. * cloning a supermodule, every new commit that gets written in the supermodule gets checked from "link" objects. > So what do you do with deleted submodules? > You wouldn't want them to still sit around in your working directory, > but you still have to preserve them. Now that is a tricky one. Mind you, I think that problem exists for any implementation. I haven't got a good answer for that. Andy -- Dr Andrew Parkins, M Eng (Hons), AMIEE ``` ## Josef Weidendorfer, 2006-12-02 11:32 Subject: Re: [RFC] Submodules in GIT Message-ID: <200612021232.08699.Josef.Weidendorfer@gmx.de> URL: https://gitlist.dev/e/200612021232.08699.Josef.Weidendorfer%40gmx.de In-Reply-To: ``` On Saturday 02 December 2006 01:12, Linus Torvalds wrote: > On Sat, 2 Dec 2006, Josef Weidendorfer wrote: > > > > So you are for a global submodule namespace in supermodule repositories, > > do I understand correctly? > > > > Otherwise, how would you specify the submodules at clone time given the > > ability that submodule roots can have relative path changed arbitrarily > > between commits? > > The only _true_ namespace would be the SHA1 of the commit (and maybe allow > a pointer to a tag too, but the namespace ends up being the same). I am not so sure about this. Perhaps we want the namespace to be more than the space of commit ids. Suppose you have some superproject which uses two compiler major versions (GCC 3 and GCC 4) as submodules because you want to have your regression test suite run with both major versions. So you would have a submodule at path "gcc3/" and "gcc4/" in your supermodule. As both the gcc 3 and gcc 4 are branches from the same project, the submodule links will go into a connected DAG (suppose GCC uses git). Alone from the commit link, it is not easy to see to what submodule it belongs to (at least from a practical point of view). So it actually _is_ more information if the proposed link objects in the supermodule contain some submodule ID they belong to. They only need be unique in the scope of the superproject (not really globally unique). So another argument for submodule names: Merging. Otherwise, how do you decide to which submodule a link belongs to, especially in the scope of above example? > How to _find_ a repository that contains that SHA1 must be left to higher > levels. After all, repositories move around, and the place you found them > originally is not a stable name. I did not talk about a special format for these submodule IDs yet. We could use an URL, but with such a value a user automatically associates some semantic which can be confusing, as repository URLs can change. We can use some symbolic name which has some meaning in the scope of the superproject, and is specified at submodule creation, like "gcc3" or "gcc4". However, this is a local decision of the person which is importing the submodule. So if two developers of the same project using supermodules independently decide that the import of "gcc" as submodule is the right thing, but use slightly different submodule IDs, you will get 2 different submodules when merging. I argue that this is even the correct thing, and they should decide about the name before both are doing the import, or only one imports and the other pulls. Another option for a submodule ID could be the root commit of the submodule commit DAG. This looks nice as such an ID really is globally unique for projects (more or less: the first commit always contains the time stamp at creation, and the author/commiter email address, even if the tree happens to be the same because you start with the same dummy file). But my example above (with the 2 different submodules from the same GCC project) shows that this is not working. A superproject never could create different submodules from the same (e.g. GCC) project. So I just vote for a symbolic name choosen at submodule creation time. ``` ## Jakub Narebski, 2006-12-02 12:48 Subject: Re: [RFC] Submodules in GIT Message-ID: URL: https://gitlist.dev/e/ekrsio%241qe%241%40sea.gmane.org In-Reply-To: <20061201110032.GL18810@admingilde.org> ``` [cut] From this discussion I think it follows that supermodule should track HEAD version of submodule. Perhaps the supermodule index should have sha1 of submodule commit, so (as usual) you have to update-index in supermodule to record changes in submodule; the difference being that you update to HEAD version, not to working directory version. Or you can just git-commit -a in supermodule which would take working directory version of files, and HEAD version of submodules. -- Jakub Narebski Warsaw, Poland ShadeHawk on #git ``` ## Jakub Narebski, 2006-12-02 13:08 Subject: Re: [RFC] Submodules in GIT Message-ID: URL: https://gitlist.dev/e/ekrtph%246nh%241%40sea.gmane.org In-Reply-To: <45706758.2020907@b-i-t.de> ``` Stephan Feder wrote: > That's it: There is no need for a separate branch or repository. If you > have the subproject's commit in the superproject's object database (and > we really have that, see 1. and 2. above), why do you _have to_ store it > elsewhere? It would be much simpler to have subproject's commit in subproject object database, and have it available in superproject's object database by the way of alternates. Otherwise when commiting new submodule state in supermodule you would have to fetch all the needed objects (submodule mighe have evolved few commits in history inbetween) into superproject's object database. -- Jakub Narebski Warsaw, Poland ShadeHawk on #git ``` ## Jakub Narebski, 2006-12-02 13:14 Subject: Re: [RFC] Submodules in GIT Message-ID: URL: https://gitlist.dev/e/ekru4l%246nh%242%40sea.gmane.org In-Reply-To: <200612012104.39897.andyparkins@gmail.com> ``` Andy Parkins wrote: >> You still don't have a totally separated repository then, because >> you can't do a reachability analysis in the submodule repository alone. > > I'm going to guess by reachability analysis, you mean that the submodule > doesn't know that some of it's commits are referenced by the supermodule.  As > I suggested elsewhere in the thread, that's easily fixed by making a > refs/supermodule/commitXXXX file for each supermodule commit that references > as particular submodule commit.  Then you can git-prune, git-fsck whenever > you want. I think it would be better resolve this in universal way by adding to git repository layout the optional "borrowers" file, which would protect against pruning objects that are referenced by repositories which have given repository as one of the "alternates". By the way, how to slurp all the objects from alternates into repo object repository? -- Jakub Narebski Warsaw, Poland ShadeHawk on #git ``` ## Josef Weidendorfer, 2006-12-02 13:50 Subject: Re: [RFC] Submodules in GIT Message-ID: <200612021450.46005.Josef.Weidendorfer@gmx.de> URL: https://gitlist.dev/e/200612021450.46005.Josef.Weidendorfer%40gmx.de In-Reply-To: <200612021004.22236.andyparkins@gmail.com> ``` On Saturday 02 December 2006 11:04, Andy Parkins wrote: > > So what do you do with deleted submodules? > > You wouldn't want them to still sit around in your working directory, > > but you still have to preserve them. > > Now that is a tricky one. Mind you, I think that problem exists for any > implementation. I haven't got a good answer for that. That suggests that it is probably better to separate submodule repositories from their checked out working trees. Why not put the GITDIRs of the submodules in subdirectories of the supermodules GITDIR instead? ``` ## Torgil Svensson, 2006-12-02 18:57 Subject: Re: [RFC] Submodules in GIT Message-ID: URL: https://gitlist.dev/e/e7bda7770612021057mc9f3eb9q7fc047dd1b5c235f%40mail.gmail.com In-Reply-To: <4570BFA4.8070903@stephan-feder.de> ``` > If you need a common infrastructure to be able to work with the > submodule, then the submodule is not independent of of the supermodule. > I see a contradiction in your requirements. Here's an real-world example that doesn't contradict: http://amarok.kde.org/wiki/Installation_HowTo#From_Anonymous_SVN "svn co -N svn://anonsvn.kde.org/home/kde/trunk/extragear/multimedia cd multimedia svn co svn://anonsvn.kde.org/home/kde/branches/KDE/3.5/kde-common/admin svn up amarok To compile the sources (from the multimedia directory):" and there's probably very few people that want to clone the entire KDE multimedia sub&super-module in this case. //Torgil On 12/2/06, sf wrote: > Linus Torvalds wrote: > > > > On Fri, 1 Dec 2006, sf wrote: > >> Linus Torvalds wrote: > >> ... > >>> In contrast, a submodule that we don't fetch is an all-or-nothing > >>> situation: we simply don't have the data at all, and it's really a matter > >>> of simply not recursing into that submodule at all - much more like not > >>> checking out a particular part of the tree. > >> If you do not want to fetch all of the supermodule then do not fetch the > >> supermodule. > > > > So why do you want to limit it? There's absolutely no cost to saying "I > > want to see all the common shared infrastructure, but I'm actually only > > interested in this one submodule that I work with". > > If you need a common infrastructure to be able to work with the > submodule, then the submodule is not independent of of the supermodule. > I see a contradiction in your requirements. > > > Also, anybody who works on just the build infrastructure simply may not > > care about all the submodules. The submodules may add up to hundreds of > > gigs of stuff. Not everybody wants them. But you may still want to get the > > common build infrastructure. > > See above. > > > In other words, your "all or nothing" approach is > > (a) not friendly > > and > > (b) has no real advantages anyway, since modules have to be independent > > enough that you _can_ split them off for other reasons anyway. > > > > So forcing that "you have to take everything" mentality onyl has > > negatives, and no positives. Why do it? > > (There have been lots of use cases for shallow clones but for a long > time git did not support them). > > If you can extend this partial fetch feature to the non-subproject case > I would agree with your reasoning. What makes the subprojects so special > in this regard. Do I have to turn a plain tree into a subproject to be > able to ignore it? Once you can restrict fetches to parts of the > contents you get the ability to restrict fetches to the "common > infrastructure" and selected submodules for free. > > Regards > > Stephan > > - > To unsubscribe from this list: send the line "unsubscribe git" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ``` ## Linus Torvalds, 2006-12-02 19:41 Subject: Re: [RFC] Submodules in GIT Message-ID: URL: https://gitlist.dev/e/Pine.LNX.4.64.0612021114270.3476%40woody.osdl.org In-Reply-To: ``` On Sat, 2 Dec 2006, Torgil Svensson wrote: > > Here's an real-world example that doesn't contradict: And I'll add the note that people who do things like submodules aren't generally even _used_ to them being "seamless", and most of the time probably don't even want complete seamlessness. As the example that Torgil points to shows, people are quite used to actually even naming the submodules separately, and things like having the "default" set of submodules not equal the "complete" set. In other words, I don't think people expect or want something hugely more complicated than the CVS/modules kind of file. What people _do_ want (and that CVS in general is horribly bad at, and this is not a module-specific issue) is to have the _versioning_ work well. When you check out a specific version of a module, you want any _linked_ modules to follow along too. This is the same reason why CVS users use tags a lot: because even _within_ a single project (no modules, no nothing), it's often hard to re-create the exact state of a version any other way. So you tag every single file and do insane things like that, because CVS just isn't very good at guaranteeing consistency across the whole project. The exact same thing is true about subprojects. I don't think that people who have used CVS subprojects a lot really mind the CVS/modules file itself (but hey, maybe I'm wrong - I've seen _other_ people maintain modules in CVS, but I've never done it myself), but they do mind the fact that it's hard as hell to do something as simple as "get all modules back to version X" without lots and lots of careful crud (ie tagging every singl emodule, things like that). Now, I'm not exactly sure who wants to use git modules, so this is the time to ask: did you hate the CVS/modules file? Or was it something you set up once, and then basically forgot about? People clearly use the ability to mark certain modules as depending on each other, and aliases to say "if you ask for this module, you actually get a set of _these_ modules". _I_ suspect that that isn't the problem people had, and isn't what they have any problems with. What CVS didn't do very well (or at all, afaik) is to say "I want supermodule version XYZ", and then got all the submodules automatically to that (reliable) state. And THAT is something I think is really important for submodules, and it's why I think the most important part isn't actually all the veneer to make "git clone" and "git pull" work (which is really about the CVS/modules kind of wrapper parsing), but actually about the supermodule "tree" object pointing to a very specific version, so that you get the exact same "atomic snapshotting" of multiple trees that you get within a single git tree. In other words, I _suspect_ that that is really what module users are all about. They want the ability to specify an arbitrary collection of these atomic snapshots (for releases etc), and just want a way to copy and move those things around, and are less interested in making everything else very seamless (because most people are happy to do the actual _development_ entirely within the submodules, so the "development" part is actually not that important for the supermodule, the supermodule is mostly for aggregation and snapshots, and tying different versions of different submodules together). So that's where I come from. And maybe I'm totally wrong. I'd like to hear what people who actually _use_ submodules think. ``` ## Martin Waitz, 2006-12-02 19:46 Subject: Re: [RFC] Submodules in GIT Message-ID: <20061202194602.GP18810@admingilde.org> URL: https://gitlist.dev/e/20061202194602.GP18810%40admingilde.org In-Reply-To: <4570BC07.4080203@stephan-feder.de> ``` hoi :) On Sat, Dec 02, 2006 at 12:34:31AM +0100, sf wrote: > > Now your submodule is no longer seen as an independent git repository > > and I think this would cause problems when you want to push/pull between > > the submodule and its upstream repository. > > You can always pick a single commit or several commits out of a larger > repository and have a complete git repository. > > And I already explained how to push and pull even from within superprojects. Sure it you are able to make it work, but it needs more work on the UI part. How do you handle the index? How do you allow to clone only the submodule? I really thought about such a setup too, but then decided that it is much easier to work with submodules when you can really see it as a repository of its own. > > But you could still call the "xdiff" part of the git repository a > > submodule. And then changes to the xdiff directory result in a new > > submodule commit, even when there is no direct reference to it. > > So you'd still "commit to the xdiff submodule". > > Let's make certain that we understand each other. I see a clear > distinction between the submodule code in a supermodule branch (commits > in the supermodule's tree and nothing else) and submodule branches which > are independent of the superproject. Supermodule branches and submodule > branches do not interact, only if I want them to. Agreed. I think the thing which caused some discussion is that I make the current submodule commit which is used by the supermodule available in a refs/head in the submodule. So there is one "branch" in the submodule which corresponds to the version used by the supermodule, but this is just for user interface. It's most important purpose is to give this special commit a name, so that it can be used in merges, etc. By selecting another refs/heads "branch" in the submodule you can also easily detach the submodule from the supermodule. It is really important to understand that you can't branch the submodule alone and still have it connected to the supermodule, because the supermodule always tracks only one commit for each submodule. So every branch that affects the project has to be done on project (topmost supermodule) level. But of course the submodule can have other branches which are not tracked by the supermodule. So by checking out refs/heads/master (as it is used in my implementation) you can attach the submodule to the supermodule (attach as in: bring the working directory in sync with the whole project), and you can detach it by selecting another refs/heads (the submodule is still part of the supermodule, but not in the state which is currently visible in the working directory). This may sound confusing, but it really is the only semantic for submodule branches that makes sense. There are fears that you may commit something that does not match your current working directory. Sure, but you explicitly asked for it and I think it won't be a problem if git-status tells about this fact. > The double slashes is the only way I can think of that clearly indicates > that I do not mean the contents named by the path, but the commit that > you find there. Once you have named a commit in that way, you can > continue to apply other revision naming suffixes, paths, and so on. With the current semantics, you can already get to the submodule commit (just leave out your double slashes), but what is missing is simply to apply all the modifiers again on this submodule commit. So I think we can do without the double slashes. -- Martin Waitz ``` ## Linus Torvalds, 2006-12-02 19:52 Subject: Re: [RFC] Submodules in GIT Message-ID: URL: https://gitlist.dev/e/Pine.LNX.4.64.0612021144520.3476%40woody.osdl.org In-Reply-To: <200612021232.08699.Josef.Weidendorfer@gmx.de> ``` On Sat, 2 Dec 2006, Josef Weidendorfer wrote: > > > > The only _true_ namespace would be the SHA1 of the commit (and maybe allow > > a pointer to a tag too, but the namespace ends up being the same). > > I am not so sure about this. > Perhaps we want the namespace to be more than the space of commit ids. I don't think it would be wrong at all to have a "link object" type, and have the "link" tree entry actually point to that "link object" instead of pointing directly to the commit in the submodule. And yes, that extra indirection would allow for more flexibility (the "link object" can contain comments about the particular version used, pointers to where you can get it - whether human-readable or strictly meant for automation - etc etc). So I agree with Andy Parkins' comment about the link object allowing not only extended namespaces, but also allowing a certain amount of flexibility (ie there's some built-in extensibility and ability to perhaps add future fields if there's a new object type). I just want the naming of the links themselves to use all the same SHA1 hashes etc, so that you always have a very explicit, and very trustworthy version - and never end up in the situation that you know which repository you want at that position, but you don't know exactly which commit in that repo was supposed to be checked out with that particular version of the super-module. ``` ## Martin Waitz, 2006-12-02 20:12 Subject: Re: [RFC] Submodules in GIT Message-ID: <20061202201242.GQ18810@admingilde.org> URL: https://gitlist.dev/e/20061202201242.GQ18810%40admingilde.org In-Reply-To: ``` hoi :) On Fri, Dec 01, 2006 at 03:09:40PM -0800, Linus Torvalds wrote: > On Fri, 1 Dec 2006, sf wrote: > > If you do not want to fetch all of the supermodule then do not fetch the > > supermodule. > > So why do you want to limit it? There's absolutely no cost to saying "I > want to see all the common shared infrastructure, but I'm actually only > interested in this one submodule that I work with". An interesting way to support this "only fetch some modules" use-case is to use several supermodules. So you could have one supermodule which is geared towards developers and only contains the modules they use. Another supermodule contails all the toolchain sources. And then there is the supermodule used for releases which is just a merge of all the other supermodules. The concept is so flexible that you don't have to introduce lots of other things as module namespaces. Just use the tools you have in a creative way ;-) -- Martin Waitz ``` ## Martin Waitz, 2006-12-02 20:18 Subject: Re: [RFC] Submodules in GIT Message-ID: <20061202201826.GR18810@admingilde.org> URL: https://gitlist.dev/e/20061202201826.GR18810%40admingilde.org In-Reply-To: ``` hoi :) On Fri, Dec 01, 2006 at 04:12:10PM -0800, Linus Torvalds wrote: > It only gets interesting for commands that fetch new objects, ie do a > "pull/fetch" op, and you'd need to know where/how to fetch new objects for > the xyzzy subproject, so that's a "naming" issue. You have a few choices: > > - get all the objects directly from the subproject as if it was one big > project. > > I actually think this sucks. Why? Because it puts an insane load on the > server side, which basically needs to traverse the object list of the > _sum_ of all projects. An initial clone (or a really big pull, which > comes to the same thing) would be absolutely horrendous I don't buy your scalability argument. By dividing the object traversal in separate steps you do not win anything. The complexity of the operation still stays the same, as you still have to traverse the exact same amount of objects. By separating the repositories you just make reachability analyis be totally awkward, without winning anything. -- Martin Waitz ``` ## Martin Waitz, 2006-12-02 20:21 Subject: Re: [RFC] Submodules in GIT Message-ID: <20061202202103.GS18810@admingilde.org> URL: https://gitlist.dev/e/20061202202103.GS18810%40admingilde.org In-Reply-To: ``` hoi :) On Sat, Dec 02, 2006 at 11:52:13AM -0800, Linus Torvalds wrote: > I don't think it would be wrong at all to have a "link object" type, and > have the "link" tree entry actually point to that "link object" instead of > pointing directly to the commit in the submodule. > > And yes, that extra indirection would allow for more flexibility (the > "link object" can contain comments about the particular version used, > pointers to where you can get it - whether human-readable or strictly > meant for automation - etc etc). What makes a submodule so special that now we suddenly have to store those stuff in the object database? Storing a fetch location would grossly contradict the distributed nature of git. I really do not see _any_ reason to store more information than the commit sha1 of the submodule. -- Martin Waitz ``` ## Martin Waitz, 2006-12-02 20:24 Subject: Re: [RFC] Submodules in GIT Message-ID: <20061202202428.GT18810@admingilde.org> URL: https://gitlist.dev/e/20061202202428.GT18810%40admingilde.org In-Reply-To: <200612020017.44275.Josef.Weidendorfer@gmx.de> ``` hoi :) On Sat, Dec 02, 2006 at 12:17:44AM +0100, Josef Weidendorfer wrote: > After some thinking, a submodule namespace even is important for checking > out only parts of a supermodule, exactly because the root of a submodule > potentially can change at every commit. have you ever thought about the idea that the location may be an important thing to consider for your decision. Perhaps the submodule is now used for something else (this is why it was moved) and that now you'd like to keep it? Anyway, you can just create several supermodules or implement generic partial tree support for git. I do not see any reason to special case submodules here. -- Martin Waitz ``` ## Martin Waitz, 2006-12-02 20:40 Subject: Re: [RFC] Submodules in GIT Message-ID: <20061202204012.GU18810@admingilde.org> URL: https://gitlist.dev/e/20061202204012.GU18810%40admingilde.org In-Reply-To: <200612021004.22236.andyparkins@gmail.com> ``` hoi :) On Sat, Dec 02, 2006 at 10:04:20AM +0000, Andy Parkins wrote: > On Friday 2006, December 01 22:08, Martin Waitz wrote: > > > > echo $SUBMODULE_HASH > > > > submodule/.git/refs/supermodules/commit$SUPERMODULE_HASH > > > > I guess you are aware that you have to scan _all_ trees inside _all_ > > supermodule commits for possible references. > > No you don't; you do it as part of the appropriate normal operations. > > * supermodule commit - scan the current tree for "link" objects in the > tree. If you find one write the reference in the submodule. > * adding a new submodule - if this is a new submodule there can't be any > references in the supermodule already. > * cloning a supermodule, every new commit that gets written in the > supermodule gets checked from "link" objects. * removing a branch from the supermodule. OK, this is an infrequent operation and it can be handled by redoing everything. I just don't like to duplicate information which is already available easily. We'd need much to many special cases, just to correctly support reachablility analysis. KISS. > > So what do you do with deleted submodules? > > You wouldn't want them to still sit around in your working directory, > > but you still have to preserve them. > > Now that is a tricky one. Mind you, I think that problem exists for any > implementation. I haven't got a good answer for that. If you just keep it in a shared object repository you don't have any problems. Please note that it is not required to keep it in one physical location. You can still use alternates/whatever to store some objects in another repository, but you need to be able to access all objects from the supermodule. -- Martin Waitz ``` ## Martin Waitz, 2006-12-02 20:43 Subject: Re: [RFC] Submodules in GIT Message-ID: <20061202204350.GV18810@admingilde.org> URL: https://gitlist.dev/e/20061202204350.GV18810%40admingilde.org In-Reply-To: <200612021450.46005.Josef.Weidendorfer@gmx.de> ``` hoi :) On Sat, Dec 02, 2006 at 02:50:45PM +0100, Josef Weidendorfer wrote: > On Saturday 02 December 2006 11:04, Andy Parkins wrote: > > > So what do you do with deleted submodules? > > > You wouldn't want them to still sit around in your working directory, > > > but you still have to preserve them. > > > > Now that is a tricky one. Mind you, I think that problem exists for any > > implementation. I haven't got a good answer for that. > > That suggests that it is probably better to separate submodule repositories > from their checked out working trees. Why not put the GITDIRs of the submodules > in subdirectories of the supermodules GITDIR instead? Why not simply use a shared object database instead? You can still have an alternative to some standalone bare repository of the submodule if you do not like to store submodule objects in the supermodule repository. -- Martin Waitz ``` ## Linus Torvalds, 2006-12-02 20:44 Subject: Re: [RFC] Submodules in GIT Message-ID: URL: https://gitlist.dev/e/Pine.LNX.4.64.0612021242080.3476%40woody.osdl.org In-Reply-To: <20061202201826.GR18810@admingilde.org> ``` On Sat, 2 Dec 2006, Martin Waitz wrote: > > I don't buy your scalability argument. Try it. Really. Get the mozilla import (450MB project), and clone it on a machine with half a gig of RAM or less. Then, clone a couple of smaller archives that end up being 450MB _combined_, but clone them separately. And watch the memory usage. > By separating the repositories you just make reachability analyis be > totally awkward, without winning anything. Trust me. Try it out. ``` ## Linus Torvalds, 2006-12-02 20:46 Subject: Re: [RFC] Submodules in GIT Message-ID: URL: https://gitlist.dev/e/Pine.LNX.4.64.0612021245081.3476%40woody.osdl.org In-Reply-To: <20061202202103.GS18810@admingilde.org> ``` On Sat, 2 Dec 2006, Martin Waitz wrote: > > What makes a submodule so special that now we suddenly have to store > those stuff in the object database? I'm not sure it is. I suspect a pure commit link with just a CVS-style "modules" file is sufficient. I'm just saying that I don't think it is _wrong_ to possibly want to expand it. ``` ## Martin Waitz, 2006-12-02 20:58 Subject: Re: [RFC] Submodules in GIT Message-ID: <20061202205853.GW18810@admingilde.org> URL: https://gitlist.dev/e/20061202205853.GW18810%40admingilde.org In-Reply-To: ``` hoi :) On Sat, Dec 02, 2006 at 12:46:01PM -0800, Linus Torvalds wrote: > On Sat, 2 Dec 2006, Martin Waitz wrote: > > > > What makes a submodule so special that now we suddenly have to store > > those stuff in the object database? > > I'm not sure it is. I suspect a pure commit link with just a CVS-style > "modules" file is sufficient. I'm just saying that I don't think it is > _wrong_ to possibly want to expand it. If we later see that we really want to have it we can always introduce it later. I don't think we should do it now if we don't see clear benefits _now_. So I was not against the link object itself (initially I wanted to do it this way, too), only agains the information which was proposed to be stored there. Up to now I haven't found anything which makes sense to store next to the submodule commit to define the identity of the submodule. -- Martin Waitz ``` ## Martin Waitz, 2006-12-02 21:06 Subject: Re: [RFC] Submodules in GIT Message-ID: <20061202210640.GX18810@admingilde.org> URL: https://gitlist.dev/e/20061202210640.GX18810%40admingilde.org In-Reply-To: ``` hoi :) On Sat, Dec 02, 2006 at 12:44:20PM -0800, Linus Torvalds wrote: > On Sat, 2 Dec 2006, Martin Waitz wrote: > > > > I don't buy your scalability argument. > > Try it. > > Really. Get the mozilla import (450MB project), and clone it on a machine > with half a gig of RAM or less. > > Then, clone a couple of smaller archives that end up being 450MB > _combined_, but clone them separately. > > And watch the memory usage. Do I understand you correctly that the problem is not the algorithmic complexity but that you have to map the objects at once instead of map them in small parts one after the other? -- Martin Waitz ``` ## Linus Torvalds, 2006-12-02 21:22 Subject: Re: [RFC] Submodules in GIT Message-ID: URL: https://gitlist.dev/e/Pine.LNX.4.64.0612021252380.3476%40woody.osdl.org In-Reply-To: ``` On Sat, 2 Dec 2006, Linus Torvalds wrote: > > And watch the memory usage. Btw, just in case you don't understand _why_ this is true, the fact is, in a git repository, quite fundamentally, because we don't have "backlinks" at any stage at all, we don't know - and fundamentally _cannot_ know - whether we're goign to see the same object in the future. So operations like "git-rev-list --objects" (or, these days, more commonly anything that just does the equivalent of that internally using the library interfaces - ie "git pack-objects" and friends) VERY FUNDAMENTALLY have to hold on to the object flags for the whole lifetime of the whole operation. And you should realize that this is really really fundamental. You can't fix it with "smarter memory management". You can't fix it with "garbage collection". This is _not_ a result of the fact that we use C and malloc, and we don't free those objects, like some people sometimes seem to believe. So garbage collection will never help this kind of situation. It flows _directly_ from the fact that our objects are immutable: because they are immutable, they don't have any backpointers, because we cannot (and must not) add backpointers to an old existing object when a new object is created that points to it. So this really isn't a memory management issue. You could somewhat work around it by adding a "caching layer" on top of git, and allow that caching layer to modify their cache of old objects (so that they can contain back-pointers), but for 99% of all users that would actually make performance MUCH WORSE, and it would also be a serious problem for coherency issues (one of the things that immutable objects cause is that there are basically never any race conditions, while a "caching layer" like this would have some serious issues about serialization). So: the very fundamental nature and choices that were made in git also means that when you have something like git-pack-objects that wants to walk the whole repo, you will end up with something that remembers EVERY SINGLE OBJECT it walked. And while I've worked very hard to make the memory footprint of individual objects as small as possible, and this means that this all works fine even for fairly large databases (especially since very few operations actually do this "traverse the whole friggin tree" thing), it does mean that there's a very fundamental limit to scalability. You can't just make a whole repository a hundred times bigger - because the operations that traverse the whole thing will require a hundred times more memory! Now, in "real" projects, this is not a problem. I can pretty much _guarantee_ that memory sizes and hardware will grow faster than projects grow. I'm not AT ALL worried about the fact that in ten years, the linux kernel repository will likely be two or three times the size it is now. Because I'm absolutely convinced that in ten years, the machines we have now will be obsolete. So on any "individual project" basis, the fact that memory requirements scale roughly as O(n) in the total repository size is simply not a problem. In fact, O(n) is pretty damn good, especially since the constant is pretty small (basically 28 bytes per object - and 20 of those bytes are the SHA1 that you simply cannot avoid). But it does mean that supermodules really should NOT be so seamless that doing a "git clone" on a supermodule does one _large_ clone. Because it's simply going to be better to: - when you clone the supermodule, track the commits you need on all submodules (this _may_ be a reason in itself for the "link" object, just so that you can traverse the supermodule object dependencies and know what subobject you are looking at even _without_ having to look at the path you got there from) - clone submodules one-by-one, using the list of objects you gathered. Maybe there are other solutions, but quite frankly, I doubt it. Yes, you'll end up "traversing" exactly as many objects either way, but the "globe subobjects one by one" is going to be a _hell_ of a lot more memory-efficient, and quite frankly, "memory usage" == "performance" under many loads (notably, any load that uses too much memory will _suck_ performance-wise, either because of swapping or simply because it will throw out caches that "many small invocations" would not have thrown out). So I guarantee that it's going to be better to do five clones of five small repositories over one clone of one big one. If only because you need less memory to do the five smaller clones. ``` ## Linus Torvalds, 2006-12-02 21:29 Subject: Re: [RFC] Submodules in GIT Message-ID: URL: https://gitlist.dev/e/Pine.LNX.4.64.0612021322560.3476%40woody.osdl.org In-Reply-To: <20061202210640.GX18810@admingilde.org> ``` On Sat, 2 Dec 2006, Martin Waitz wrote: > > Do I understand you correctly that the problem is not the algorithmic > complexity but that you have to map the objects at once instead of map > them in small parts one after the other? Not map them, but track their "used" flag. Yes. You can unmap objects any time at all (since you can just always re-create them at any time very easily and cheaply), but the one thing you CANNOT recreate is the object flags. See "struct object", and the "used" and FLAG_BITS in particular. Almost all git programs need the FLAG_BITS. Something as simple as just traversing the commit history needs at a minimum one _single_ bit for each object: "Have I already seen this". In reality, you tend to need two or three more (ie the UNINTERESTING bit ends up being as important as the SEEN bit, because it's what determines whether it's reachable from some commit we're _not_ interested in, and in the end that's what allows us to not traverse the whole history). So you need at a MINIMUM to track the bits #define SEEN (1u<<0) #define UNINTERESTING (1u<<1) and in practice almost everything needs #define SHOWN (1u<<3) too (SEEN is for deciding whether to _traverse_ something, SHOWN is for deciding whether you've already output the data for this, and the difference is crucial for any depth-first DAG algorithm, since you need to test-and-set the one bit when you first encounter the object, and test-and-set the other bit when you "leave" the object). So three bits are minimal to _any_ git traversal algorithm. Many specific issues want more bits (eg the TREECHANGE bit may not be quite as fundamnetal, but it sure ends up being critical for the "track subtree" case). ``` ## Josef Weidendorfer, 2006-12-03 00:55 Subject: Re: [RFC] Submodules in GIT Message-ID: <200612030155.09630.Josef.Weidendorfer@gmx.de> URL: https://gitlist.dev/e/200612030155.09630.Josef.Weidendorfer%40gmx.de In-Reply-To: <20061202202428.GT18810@admingilde.org> ``` On Saturday 02 December 2006 21:24, Martin Waitz wrote: > On Sat, Dec 02, 2006 at 12:17:44AM +0100, Josef Weidendorfer wrote: > > After some thinking, a submodule namespace even is important for checking > > out only parts of a supermodule, exactly because the root of a submodule > > potentially can change at every commit. > > have you ever thought about the idea that the location may be an > important thing to consider for your decision. Which decision, for what? Sorry, I do not understand. Do you want to say that relative submodule root paths should be kept fix the whole lifetime of a supermodule? Ie. a submodule "identity" is bound to its relative path, and when we move it, it should be seen as deleting at and creating a totally new, different submodule? That's fine. But you have to handle submodule creation/deletion neverless. And while you are at a commit which has a given submodule deleted, you have to keep the submodule data somewhere - referencing it with a name. I do not speak here about the object database, that could be combined; but about all the other files in .git/ of the currently not checked out submodule. > Perhaps the submodule is now used for something else (this is why it was > moved) and that now you'd like to keep it? Can you give a usage szenario? What do you mean here? > Anyway, you can just create several supermodules or implement generic > partial tree support for git. I do not see any reason to special case > submodules here. What should such a general partial tree support look like? I suppose you want to configure paths which should not be checked out. As long as you say that a given submodule always has to exist at a given path, you are right: then, you can say: "Please, do not check out this submodule" which is the same as "Do not check out this path". But I think it is quite restrictive to not allow to move submodules around. When the supermodule upstream decides to move a submodule, your partial tree config to not check out a submodule will be lost. But more important, if you made changes to a given submodule, and pull from upstream which changed the submodule position in-between, your changes will be not taken over to the new position, as the move is seen as creation of a totally independent submodule. ``` ## Josef Weidendorfer, 2006-12-03 01:02 Subject: Re: [RFC] Submodules in GIT Message-ID: <200612030202.34990.Josef.Weidendorfer@gmx.de> URL: https://gitlist.dev/e/200612030202.34990.Josef.Weidendorfer%40gmx.de In-Reply-To: <20061202204350.GV18810@admingilde.org> ``` On Saturday 02 December 2006 21:43, Martin Waitz wrote: > On Sat, Dec 02, 2006 at 02:50:45PM +0100, Josef Weidendorfer wrote: > > On Saturday 02 December 2006 11:04, Andy Parkins wrote: > > > > So what do you do with deleted submodules? > > > > You wouldn't want them to still sit around in your working directory, > > > > but you still have to preserve them. > > > > > > Now that is a tricky one. Mind you, I think that problem exists for any > > > implementation. I haven't got a good answer for that. > > > > That suggests that it is probably better to separate submodule repositories > > from their checked out working trees. Why not put the GITDIRs of the submodules > > in subdirectories of the supermodules GITDIR instead? > > Why not simply use a shared object database instead? Sure. I have no problem with this. But can we go one step further? AFAICS your submodules store the .git/ directories of submodules directly at submodule position in the working tree - but you have a link .git/objects into the object database of the supermodule. When the user wants to delete the submodule, he would remove this .git/ directory, too. So you loose the .git/refs of the submodule etc. I would suggest to put the submodule .git dirs into the .git dir of the supermodule. ``` ## Josef Weidendorfer, 2006-12-03 01:11 Subject: Re: [RFC] Submodules in GIT Message-ID: <200612030211.17159.Josef.Weidendorfer@gmx.de> URL: https://gitlist.dev/e/200612030211.17159.Josef.Weidendorfer%40gmx.de In-Reply-To: <20061202205853.GW18810@admingilde.org> ``` On Saturday 02 December 2006 21:58, Martin Waitz wrote: > So I was not against the link object itself (initially I wanted to do it > this way, too), only agains the information which was proposed to be > stored there. Up to now I haven't found anything which makes sense to > store next to the submodule commit to define the identity of the > submodule. Isn't it enough reason that a porcelain probably wants to store meta information for a given submodule, giving the need to put a name/identity to it? Josef ``` ## Josef Weidendorfer, 2006-12-03 02:07 Subject: Thoughts about memory requirements in traversals [Was: Re: [RFC] Submodules in GIT] Message-ID: <200612030307.26429.Josef.Weidendorfer@gmx.de> URL: https://gitlist.dev/e/200612030307.26429.Josef.Weidendorfer%40gmx.de In-Reply-To: ``` On Saturday 02 December 2006 22:22, Linus Torvalds wrote: > So operations like "git-rev-list --objects" (or, these days, more commonly > anything that just does the equivalent of that internally using the > library interfaces - ie "git pack-objects" and friends) VERY FUNDAMENTALLY > have to hold on to the object flags for the whole lifetime of the whole > operation. > > [...] > > So this really isn't a memory management issue. You could somewhat work > around it by adding a "caching layer" on top of git, and allow that > caching layer to modify their cache of old objects (so that they can > contain back-pointers), but for 99% of all users that would actually make > performance MUCH WORSE, and it would also be a serious problem for > coherency issues (one of the things that immutable objects cause is that > there are basically never any race conditions, while a "caching layer" > like this would have some serious issues about serialization). Thinking about this... You have to make very sure to always update the caching layer containing the backlinks on every addition of a further object. You can do this because you always reached this new object by some other object, which exactly is the backpointer. Now let us suppose we are able to do this. What does this give us? Take a look at object traversal: We have to store the flag "already visited" for objects we could reach again in the traversal. But with the backlinks, we can see that most of the objects can only be reached via one path, and therefore, there is no need to store the flag, as it never will be queried in the further traversal. (Similar for objects with two paths: When you have visited the object two times, you can throw away the flag, as it is not queried any more). Regarding the caching layer and object traversal, it would have been enough to only store "is this object reachable via more than 1 path?". For this, the "cache" could be the set of objects reachable with more than one path. And such a set stored in a file should be quite managable, and be quite small, relative to the size of the object database. In fact, this "cache" can be created with a usual object traversal (which has the original memory requirement), but as long as we do not add objects to the database, further traversals would only need a fraction of memory. When only adding a small number of objects, it should be easy to update the cache; while with big actions like fetching/pulling, we simply should remove the file with the backlink information. > problem. In fact, O(n) is pretty damn good, especially since the constant > is pretty small (basically 28 bytes per object - and 20 of those bytes > are the SHA1 that you simply cannot avoid). Again only some thoughts... Pack files are fully self-contained object stores, yes? So in the scope of a single pack file, the offset of this object is enough as object identification. If we could make sure that in any given algorithm touching objects, like commit traversal, we always have the offset available when we need to do an object lookup, then, it should be enough to store object flags only indexed by the offset of this object in the pack. The translation SHA1 -> offset can be done with the pack index. As you usually have multiple packs, a (pack number / offset) tuple should be enough as object ID. Thinking even one step further: Would it make sense to define an encoding format for the content of commit and tree objects inside of packs, where the SHA1 is replaced by the offset of the object in this pack? As exactly the SHA1 is the least compressable thing, this could promise quite a benefit. AFAIK, we currently only use these offsets for referencing objects in delta chains. More about the original topic of this thread (and off-topic to the new subject): > But it does mean that supermodules really should NOT be so seamless that > doing a "git clone" on a supermodule does one _large_ clone. Because it's > simply going to be better to: > > - when you clone the supermodule, track the commits you need on all > submodules (this _may_ be a reason in itself for the "link" object, > just so that you can traverse the supermodule object dependencies and > know what subobject you are looking at even _without_ having to look at > the path you got there from) > > - clone submodules one-by-one, using the list of objects you gathered. Without submodule identities, we would have to clone path-by-path, as we can not distinguish different submodules apart from there location. ``` ## Linus Torvalds, 2006-12-03 02:25 Subject: Re: Thoughts about memory requirements in traversals [Was: Re: [RFC] Submodules in GIT] Message-ID: URL: https://gitlist.dev/e/Pine.LNX.4.64.0612021814530.3476%40woody.osdl.org In-Reply-To: <200612030307.26429.Josef.Weidendorfer@gmx.de> ``` On Sun, 3 Dec 2006, Josef Weidendorfer wrote: > > Thinking about this... > You have to make very sure to always update the caching layer containing > the backlinks on every addition of a further object. You can do this > because you always reached this new object by some other object, which > exactly is the backpointer. You're missing the big issue. The issue is that a cache like that would ABSOLUTELY SUCK. You could speed up the non-common operations with it, but: - any changes would become a LOT more expensive to do, because they all need to update every single object they add (ie a "commit" would now have to add backpointers TO EVERY SINGLE BLOB). Imagine what this does to something like the kernel, where a commit reaches 22,000 files! You can do it at a finer granularity (ie do just the direct backlinks and only do the "tree->blob" and "tree->tree" things rather than the full commit reachability, but it's still going to be MUCH more painful than what we do now. - the cache would be a lot bigger than the current pack-files, and it would be fragile as hell to boot. Because it needs to get rewritten for every operation, it gets corrupted much more easily, and that's ignoring things like race conditions, so it would now need a ton of locking that git simply doesn't do at all. - everything would basically slow down. - you couldn't do shared object databases AT ALL, because backpointers wouldn't work. The whole _reason_ you can share object databases is the same reason we can't have backpointers: objects are immutable and never change depending on circustances. The _only_ downside of the current situation is literally the 24 or 28 bytes per object that we look at. For most operations, we don't even look at that many objects, so it's really the worst-case things. > In fact, this "cache" can be created with a usual object traversal > (which has the original memory requirement), but as long as we do > not add objects to the database, further traversals would only need > a fraction of memory. Right. If the project is totally read-only, the cache would work well. For real development, it would SUCK. It would make things like "git reset" very expensive indeed, for example (you'd have to unwind the whole cache: either regenerating it - which would take minutes - or being very careful indeed and being able to always remove objects properly and keeping track of them 100%). IOW, it's nasty nasty nasty. And it doesn't really even help anything but a case that we actually already handle really well (I spent a lot of effort on making the memory footprint minimal). But it does mean that you do NOT want to traverse a hundred different project "as if" they were one. That's really the only thing it means. And since you can do submodules as independent projects, and you SHOULD do them that way for tons of other reasons _anyway_, even that isn't a reason to screw up all the _wonderful_ properties of the git object database. So what I'm trying to say is that the immutable non-backpointer nature of the git database is what makes it so WONDERFUL. It's efficient, it's dense, it's stable, and it allows us all the clever things we do. But it means that we do end up alway spending 28 bytes per object, and we can never throw those 28 bytes away during a single "traversal" run. ``` ## Shawn Pearce, 2006-12-03 02:46 Subject: Re: Thoughts about memory requirements in traversals [Was: Re: [RFC] Submodules in GIT] Message-ID: <20061203024655.GD26668@spearce.org> URL: https://gitlist.dev/e/20061203024655.GD26668%40spearce.org In-Reply-To: <200612030307.26429.Josef.Weidendorfer@gmx.de> ``` Josef Weidendorfer wrote: > Thinking even one step further: > Would it make sense to define an encoding format for the content of > commit and tree objects inside of packs, where the SHA1 is replaced by the > offset of the object in this pack? > As exactly the SHA1 is the least compressable thing, this could promise > quite a benefit. I actually had the same idea the other day. I discarded it after thinking about it for a minute. Here's the problem: Lets say we do this for the tree and parent IDs in a commit, because these are the most commonly needed part of a commit during revision traversal. So we want to put the offset to the tree and the offset to each parent at the front of the commit somehow to make them very cheap to access. This means that when we start to write out a commit we need to know the offset to the tree that commit references. But git-pack-objects sorts object by type: commit, tree, blob (I forget where tags go, but they aren't important in this context). So generally *all* commits appear before the first tree. So when we write out the first commit we need to know exactly how many bytes every commit will need (compressed mind you) in this pack so we can determine the position of the first tree. Now do this for every commit and every tree that those commits use... yes, its a lot of work to precompute and store all offsets before you even write out the first byte. Its even worse with parent commits because ancestors tend to appear behind the commit (newest->oldest) so that "git log" can benefit from OS read-ahead. So you also have to keep track of your parent commmit offsets. Not pretty. Extending that idea to tree objects (store the offset of the entry) makes the issue even uglier. Oh, and packs aren't entirely self-contained. A pack is only self contained in the sense that no object in the pack deltafies against an object outside of the pack[1]. However by design an object (e.g. a commit or a tree) can reference an object which is either loose or which is in another pack. This is especially important for every large projects where not every commit/tree/tag/blob will fit into one 4 giB file. **1** Except in the case of thin packs, which are used only on the network and only to save bandwidth. > AFAIK, we currently only use these offsets for referencing objects in > delta chains. Yes, that's a recent feature to reference a delta base. -- ``` ## Josef Weidendorfer, 2006-12-03 03:21 Subject: Re: Thoughts about memory requirements in traversals [Was: Re: [RFC] Submodules in GIT] Message-ID: <200612030421.18662.Josef.Weidendorfer@gmx.de> URL: https://gitlist.dev/e/200612030421.18662.Josef.Weidendorfer%40gmx.de In-Reply-To: <20061203024655.GD26668@spearce.org> ``` On Sunday 03 December 2006 03:46, Shawn Pearce wrote: > Josef Weidendorfer wrote: > > Thinking even one step further: > > Would it make sense to define an encoding format for the content of > > commit and tree objects inside of packs, where the SHA1 is replaced by the > > offset of the object in this pack? > > As exactly the SHA1 is the least compressable thing, this could promise > > quite a benefit. > [...] > > This means that when we start to write out a commit we need to know > the offset to the tree that commit references. But git-pack-objects > sorts object by type: commit, tree, blob (I forget where tags go, > but they aren't important in this context). So generally *all* > commits appear before the first tree. So when we write out the first > commit we need to know exactly how many bytes every commit will need > (compressed mind you) in this pack so we can determine the position > of the first tree. Now do this for every commit and every tree > that those commits use... yes, its a lot of work to precompute > and store all offsets before you even write out the first byte. Yes, it looks like a hen-and-egg problem, but IMHO you can handle it nicely with another redirection, i.e. a table you build up while repacking the file, and storing this table at the end. You simply sequentially renumber any object SHA, starting from 0 in the order you see them. You can do two renumberings, one for the objects contained in the original pack (1), and one for the external ones (2). Put these new numbers (with a bit distinguishing (1) and (2)) as replacement into commit/tree objects. At the end, you have the new offsets for objects in (1). Put redirection tables for (1) [new number -> new offset] and (2) [other new number->SHA1 of external object] at the end of the new pack. This way, you effectivly have removed all incompressable SHAs from the pack file aside from one entry in the redirection tables for each external object. The only problem I see is how to decode the objects, i.e. how to get the original SHA1 from an offset: we can not recalculate the SHA1 from the object content as we changed the content itself. But there should be a way to store the SHA1 in front of the object somehow, perhaps it is already given by the current format? Am I missing something here? ``` ## Martin Waitz, 2006-12-03 06:29 Subject: Re: [RFC] Submodules in GIT Message-ID: <20061203062916.GY18810@admingilde.org> URL: https://gitlist.dev/e/20061203062916.GY18810%40admingilde.org In-Reply-To: <200612030155.09630.Josef.Weidendorfer@gmx.de> ``` hoi :) On Sun, Dec 03, 2006 at 01:55:08AM +0100, Josef Weidendorfer wrote: > On Saturday 02 December 2006 21:24, Martin Waitz wrote: > > On Sat, Dec 02, 2006 at 12:17:44AM +0100, Josef Weidendorfer wrote: > > > After some thinking, a submodule namespace even is important for checking > > > out only parts of a supermodule, exactly because the root of a submodule > > > potentially can change at every commit. > > > > have you ever thought about the idea that the location may be an > > important thing to consider for your decision. > > Which decision, for what? Sorry, I do not understand. to check out, or not to check out. > What should such a general partial tree support look like? I suppose you > want to configure paths which should not be checked out. As long as you > say that a given submodule always has to exist at a given path, you are > right: then, you can say: "Please, do not check out this submodule" which > is the same as "Do not check out this path". You could say something like "do not check out anything below "test/". If then some submodule moves from "test/foo" to "build/foo", it will be checked out, because this module is now not only used for testing, but is needed for building in the new version of the supermodule. -- Martin Waitz ``` ## Torgil Svensson, 2006-12-03 09:19 Subject: Re: [RFC] Submodules in GIT Message-ID: URL: https://gitlist.dev/e/e7bda7770612030119v197cbc95h6b3fa9e22b78c058%40mail.gmail.com In-Reply-To: ``` On 12/2/06, Linus Torvalds wrote: > > In other words, I don't think people expect or want something hugely more > complicated than the CVS/modules kind of file. What about the case when you want _everything_, do you then have to know the names of all submodules, present and past? If you have an old irrelevant submodule in the history that happens to have the same name as one of them you are interested in, do you get this as well? During a debugging session it might be convenient to do a "all but X" kind of fetch if you have a project dependent on several small modules and one of them is the big black sheep. For simple cases, I think it's sufficient to have the "everyone or no-one" option. If git enforces sending submodules one by one and requires the fetching side to specify links explicitly couldn't the selection be left to the user to decide with "hooks" or plumbing? Default hook could implement a simple white- or black-list. ``` ## Andy Parkins, 2006-12-03 09:42 Subject: Re: [RFC] Submodules in GIT Message-ID: <200612030942.20946.andyparkins@gmail.com> URL: https://gitlist.dev/e/200612030942.20946.andyparkins%40gmail.com In-Reply-To: <200612021255.59972.Josef.Weidendorfer@gmx.de> ``` On Saturday 2006, December 02 11:55, Josef Weidendorfer wrote: > > > 100644 blob 08602f522183dc43787616f37cba9b8af4e3dade xdiff-interface.c > > > 100644 blob 1346908bea31319aabeabdfd955e2ea9aab37456 xdiff-interface.h > > > 040000 tree 959dd5d97e665998eb26c764d3a889ae7903d9c2 xdiff > > > 050000 link 0215ffb08ce99e2bb59eca114a99499a4d06e704 xyzzy > > > > > > where that 050000 is the new magic type (I picked one out of my *ss: > > > it's not a valid type for a file mode, so it's a godo choice, but it > > > could be anythign that cannot conflict with a real file), which just > > > specifies the "link" part. The SHA1 is the SHA1 of the commit, and the > > > "xyzzy" is obviously just the name within the directory of the > > > submodule. > > > > Can I argue that the hash in that object should actually be to a real > > object in the supermodule repository rather than a link? > > That is the thing we already are discussing here :-) > IMHO, submodule IDs make a lot of sense, and this needs to specify the > submodule ID at every link. Which would force us to use seperate objects. I wasn't really going as deep as a submodule ID. Just moving the submodule commit hash from the supermodule tree, to a supermodule "link" object. What goes in that object is a separate problem I believe. The primary reason I think it's a good idea is that it is consistent with every other hash in the tree. It seems to be inconsistent to say blob objects have a hash that points to an object in this repo tree objects have a hash that points to an object in this repo link objects have a hash the points to an object in a different repo > However, I am not speaking about some separation issue, but more about a > design decision. For fetching/pulling/merging, you want be able to > distinguish submodules not only by the commit id into the submodule: > multiple > submodules could link into the same DAG (but different branches) of another > repository which would make unique fetching/pulling/merge decisions > difficult, especially when you think about the possibility that the > relative root path of a submodule in a supermodule should be able to change > at any supermodule commit. I can't say I've understood what you mean here. There is no difference in facilities if there is a link object in the local repository as well. It's merely an extra layer of indirection. Apart from the tiny cost of dereferencing that link object, there is no disadvantage. Andy -- Dr Andrew Parkins, M Eng (Hons), AMIEE ``` ## Jakub Narebski, 2006-12-03 11:10 Subject: Re: Thoughts about memory requirements in traversals [Was: Re: [RFC] Submodules in GIT] Message-ID: URL: https://gitlist.dev/e/ekubag%246km%242%40sea.gmane.org In-Reply-To: <200612030421.18662.Josef.Weidendorfer@gmx.de> ``` Josef Weidendorfer wrote: > On Sunday 03 December 2006 03:46, Shawn Pearce wrote: >> Josef Weidendorfer wrote: >>> Thinking even one step further: >>> Would it make sense to define an encoding format for the content of >>> commit and tree objects inside of packs, where the SHA1 is replaced by >>> the offset of the object in this pack? >>> As exactly the SHA1 is the least compressable thing, this could promise >>> quite a benefit. >> [...] >> >> This means that when we start to write out a commit we need to know >> the offset to the tree that commit references. But git-pack-objects >> sorts object by type: commit, tree, blob (I forget where tags go, >> but they aren't important in this context). So generally *all* >> commits appear before the first tree. So when we write out the first >> commit we need to know exactly how many bytes every commit will need >> (compressed mind you) in this pack so we can determine the position >> of the first tree. Now do this for every commit and every tree >> that those commits use... yes, its a lot of work to precompute >> and store all offsets before you even write out the first byte. > > Yes, it looks like a hen-and-egg problem, but IMHO you can > handle it nicely with another redirection, i.e. a table you build > up while repacking the file, and storing this table at the end. > > You simply sequentially renumber any object SHA, starting from 0 > in the order you see them. You can do two renumberings, one for > the objects contained in the original pack (1), and one for the > external ones (2). Put these new numbers (with a bit distinguishing > (1) and (2)) as replacement into commit/tree objects. > At the end, you have the new offsets for objects in (1). Put > redirection tables for (1) [new number -> new offset] > and (2) [other new number->SHA1 of external object] at the end > of the new pack. > This way, you effectivly have removed all incompressable SHAs from > the pack file aside from one entry in the redirection tables for > each external object. > > The only problem I see is how to decode the objects, i.e. how to > get the original SHA1 from an offset: we can not recalculate the > SHA1 from the object content as we changed the content itself. > But there should be a way to store the SHA1 in front of the object > somehow, perhaps it is already given by the current format? > > Am I missing something here? Doesn't this idea clash with the object and delta reusing for repack? Hmmm... perhaps with the two indirect tables it wouldn't, only the tables would need to be recalculated... or perhaps it would because of offset clashes. -- Jakub Narebski Warsaw, Poland ShadeHawk on #git ``` ## Josef Weidendorfer, 2006-12-03 11:47 Subject: Re: Thoughts about memory requirements in traversals [Was: Re: [RFC] Submodules in GIT] Message-ID: <200612031247.12957.Josef.Weidendorfer@gmx.de> URL: https://gitlist.dev/e/200612031247.12957.Josef.Weidendorfer%40gmx.de In-Reply-To: ``` On Sunday 03 December 2006 12:10, Jakub Narebski wrote: > > You simply sequentially renumber any object SHA, starting from 0 > > in the order you see them. You can do two renumberings, one for > > the objects contained in the original pack (1), and one for the > > external ones (2). Put these new numbers (with a bit distinguishing > > (1) and (2)) as replacement into commit/tree objects. > > At the end, you have the new offsets for objects in (1). Put > > redirection tables for (1) [new number -> new offset] > > and (2) [other new number->SHA1 of external object] at the end > > of the new pack. > > Doesn't this idea clash with the object and delta reusing for repack? In general, yes: you modify object content by encoding, and if you want to reuse the objects without decompression, and need to keep the info to be able to decode, ie. the redirection table. This gets problematic if you want to join multiple packs, or fetch objects and want to reuse the compressed representation, as the object renumbering is only local to one pack, and numbers can be reused between packs. So this idea is probably for archival packs only. ``` ## Linus Torvalds, 2006-12-03 17:54 Subject: Re: [RFC] Submodules in GIT Message-ID: URL: https://gitlist.dev/e/Pine.LNX.4.64.0612030946150.3476%40woody.osdl.org In-Reply-To: ``` On Sun, 3 Dec 2006, Torgil Svensson wrote: > > On 12/2/06, Linus Torvalds wrote: > > > > In other words, I don't think people expect or want something hugely more > > complicated than the CVS/modules kind of file. > > What about the case when you want _everything_, do you then have to > know the names of all submodules, present and past? Afaik, the way people do this historically is simply: - often have an alias for "everything" (eg "all" or "src" or "world"), and if you want everything, you basically ask for it by checking out the "src" module. Ie this is the "upstream" way to let downstream check out everything. - if you're downstream, and you have a partial repo, and you realize that you want everything else, you just look at gitweb (assuming it is extended to show module information, of course ;) or the .gitmodules (or whatever it would be called) file to get the other pieces manually. But hey, I also think it would be fine to have "git clone --allmodules" or something ("fetch" too). I think this whole question will depend more on how people end up _using_ module support than on any technical issues per se. Again, I suspect the people who now set up modules in CVS are likely to have a better idea than I do about how they usually do it (and why). > If you have an old irrelevant submodule in the history that happens to > have the same name as one of them you are interested in, do you get > this as well? I dunno. Details, details. I'm also not sure this is hugely important. It could be "solved" by simply having the requirement that all modules need to be named differently (notice that "module name" is _not_ the same thing as "the directory name where the module shows up". That's not the case even in CVS modules, and with a "link" type in the git tree object, the directory where a module shows up would basically be totally independent of the "name" of the module). > During a debugging session it might be convenient to do a "all but X" > kind of fetch if you have a project dependent on several small modules > and one of them is the big black sheep. I suspect it's more common to name the modules you want to fetch explicitly, rather than make it a "negative" choice, but that sounds largely like just an interface issue. ``` ## Andy Parkins, 2006-12-03 19:33 Subject: Re: [RFC] Submodules in GIT Message-ID: <200612031933.42968.andyparkins@gmail.com> URL: https://gitlist.dev/e/200612031933.42968.andyparkins%40gmail.com In-Reply-To: ``` On Saturday 2006, December 02 19:41, Linus Torvalds wrote: > Now, I'm not exactly sure who wants to use git modules, so this is the > time to ask: did you hate the CVS/modules file? Or was it something you > set up once, and then basically forgot about? People clearly use the > ability to mark certain modules as depending on each other, and aliases to > say "if you ask for this module, you actually get a set of _these_ > modules". Never used CVS/modules, but I used svn:externals. I have a few projects that are libraries that I use in many other projects. So, my directory tree looks like this: projects/ libX/ projectP/ libX/ projectQ/ libX/ The nightmare I had was that I would add a feature to projectP/libX, and commit it. Great. Then later I'd do "svn update" in projectQ - HAVING MADE NO CHANGES TO IT - and libX would update to the latest version, which turns out to be incompatible with projectQ, and I can no longer even build projectQ. If only libX would stay where it was put. The worst of it is if you check out an older version, say "stable-release" that you tagged last year, the svn:external would always just check out the latest version, so you'd have to go back through the logs to find out what approximate submodule revision you should really check out, check it out and then remember not to do svn update, because that would just reset the external to the latest version. AHHHHHHH! Maddening to say the least. This fits exactly with what you have described as the primary reason for wanting submodules. I didn't want seamless integration, I was happy to change into projectP/libX to make libX commits. All I actually wanted was the particular checkout of libX for a particular checkin of projectP to be remembered. That's it. Anything else is just gravy. I'm doing exactly the same sort of thing now but with git. git hasn't fixed the problem (yet) but certainly hasn't made it any worse than it was. svn:externals were nothing more than a way of storing a URL in the repository - who cares, I wish now I'd never bothered, they serve no version control purpose and are merely a UI convenience. Andy -- Dr Andrew Parkins, M Eng (Hons), AMIEE ``` ## Martin Waitz, 2006-12-03 20:46 Subject: Re: [RFC] Submodules in GIT Message-ID: <20061203204644.GZ18810@admingilde.org> URL: https://gitlist.dev/e/20061203204644.GZ18810%40admingilde.org In-Reply-To: ``` hoi :) On Sat, Dec 02, 2006 at 12:44:20PM -0800, Linus Torvalds wrote: > And watch the memory usage. hmm, really sad, it was such a nice concept until now... You are right, I have to think more about scalability. O(N) anywhere is really bad for submodules. They really should be able to bundle the kernel, mozilla, qt and whatnot into one project and that will get huge. -- Martin Waitz ``` ## Sven Verdoolaege, 2006-12-03 22:16 Subject: Re: [RFC] Submodules in GIT Message-ID: <20061203221630.GA940MdfPADPa@greensroom.kotnet.org> URL: https://gitlist.dev/e/20061203221630.GA940MdfPADPa%40greensroom.kotnet.org In-Reply-To: ``` On Fri, Dec 01, 2006 at 04:12:10PM -0800, Linus Torvalds wrote: > So within the supermodule, on a "git object" level, a submodule should > just be named by the SHA1 that was it's HEAD when it was committed within > the supermodule. So in the "tree object", you'd see something like the > following when you go "git ls-tree HEAD" on the superproject: > > ... > 100644 blob 08602f522183dc43787616f37cba9b8af4e3dade xdiff-interface.c > 100644 blob 1346908bea31319aabeabdfd955e2ea9aab37456 xdiff-interface.h > 040000 tree 959dd5d97e665998eb26c764d3a889ae7903d9c2 xdiff > 050000 link 0215ffb08ce99e2bb59eca114a99499a4d06e704 xyzzy > > where that 050000 is the new magic type (I picked one out of my *ss: it's > not a valid type for a file mode, so it's a godo choice, but it could be > anythign that cannot conflict with a real file), which just specifies the > "link" part. The SHA1 is the SHA1 of the commit, and the "xyzzy" is > obviously just the name within the directory of the submodule. > > That's all that is actually required for a lot of git commands that > already expect all objects to be available (ie "git checkout", "git diff" > etc). But is this object (and all the objects it points to) going to be available (in the superproject) ? The following seems to suggest that you think they shouldn't. How is fsck-objects then going to check that such an object is valid ? Is it going to call fsck-objects recursively on the (available) submodules ? On Fri, Dec 01, 2006 at 03:30:32PM -0800, Linus Torvalds wrote: > The only thing that a submodule must NOT be allowed to do on its own is > pruning (and it's distant cousin "git repack -d"). How are you going to enforce this if the submodule isn't supposed to know that it is being used as a submodule ? > You must always prune > from the supermodule, because the submodule cannot really know on its own > what references point into it. How is one of the supermodules going to know what references from other supermodules containing the submodule point into the submodule ? ``` ## Linus Torvalds, 2006-12-03 22:32 Subject: Re: [RFC] Submodules in GIT Message-ID: URL: https://gitlist.dev/e/Pine.LNX.4.64.0612031421030.3476%40woody.osdl.org In-Reply-To: <20061203221630.GA940MdfPADPa@greensroom.kotnet.org> ``` On Sun, 3 Dec 2006, Sven Verdoolaege wrote: > > On Fri, Dec 01, 2006 at 03:30:32PM -0800, Linus Torvalds wrote: > > The only thing that a submodule must NOT be allowed to do on its own is > > pruning (and it's distant cousin "git repack -d"). > > How are you going to enforce this if the submodule isn't supposed > to know that it is being used as a submodule ? Note that there's actually two "submodules": - there's the submodule "project" itself. This one must be totally unaware of the supermodule, because this one might be cloned and copied _independently_ of the supermodule. - there's the PARTICULAR CHECKED-OUT COPY of the submodule that is actually checked out in a supermodule. This is just a specific _instance_ of the particular submodule. So a particular instance of a submodule might be "aware" of the fact that it's a submodule of a supermodule. For example, the "awareness" migth be as simple as just a magic flag file inside it's .git/ directory. And that awareness would be what simply disabled pruning or "repack -d" within that particular instance. But this magic flag doesn't affect the bigger-picture git repository. It's a _private_ flag. So it doesn't affect the git part, any more than it really affects the git repository that you may have a [user] name = Myname email = myemail in your .git/config file. See? You can have private data in a git repository, but that doesn't mean that it's visible as _repository_ data. But it can still affect how git commands act (eg the "user" definitions above will affect the default user information that "git commit" uses, of course, without actually affecting the git archive in any other way) > How is one of the supermodules going to know what references from other > supermodules containing the submodule point into the submodule ? Why would it care? They are other supermodules. It doesn't matter, the same way it doesn't matter that _my_ "git" tree may not have all the same references that _your_ "git" repo has. If I want to get the same references, I'd need to fetch them from you, and at that point, I'd need to get all the objects that are pointed to by those refs too. But only on "git fetch" do you actually start caring. ``` ## Jakub Narebski, 2006-12-03 22:49 Subject: Re: [RFC] Submodules in GIT Message-ID: URL: https://gitlist.dev/e/ekvk8s%24c3d%241%40sea.gmane.org In-Reply-To: ``` Linus Torvalds wrote: > On Sun, 3 Dec 2006, Sven Verdoolaege wrote: >> >> On Fri, Dec 01, 2006 at 03:30:32PM -0800, Linus Torvalds wrote: >>> The only thing that a submodule must NOT be allowed to do on its own is >>> pruning (and it's distant cousin "git repack -d"). >> >> How are you going to enforce this if the submodule isn't supposed >> to know that it is being used as a submodule ? > > Note that there's actually two "submodules": > > - there's the submodule "project" itself. > > This one must be totally unaware of the supermodule, because this one > might be cloned and copied _independently_ of the supermodule. > > - there's the PARTICULAR CHECKED-OUT COPY of the submodule that is > actually checked out in a supermodule. > > This is just a specific _instance_ of the particular submodule. > > So a particular instance of a submodule might be "aware" of the fact that > it's a submodule of a supermodule. For example, the "awareness" migth be > as simple as just a magic flag file inside it's .git/ directory. And that > awareness would be what simply disabled pruning or "repack -d" within that > particular instance. If we use objects/info/alternates (or equivalent, e.g. objects/info/modules, or modules file) in superproject to refer to submodule repository object database (so superproject has access to all the objects including submodule), I'd prefer to have in submodule objects/info/borrowers file, which would point to superproject (and to other repositories which have submodule as one of alternate object databases) for git-prune and friends to check which parts are truly unreachable. This would be generic solution to the problem with alternates, not only specific to submodule support. -- Jakub Narebski Warsaw, Poland ShadeHawk on #git ``` ## Josef Weidendorfer, 2006-12-04 11:12 Subject: Re: [RFC] Submodules in GIT Message-ID: <200612041212.55911.Josef.Weidendorfer@gmx.de> URL: https://gitlist.dev/e/200612041212.55911.Josef.Weidendorfer%40gmx.de In-Reply-To: ``` On Sunday 03 December 2006 23:32, Linus Torvalds wrote: > So a particular instance of a submodule might be "aware" of the fact that > it's a submodule of a supermodule. For example, the "awareness" migth be > as simple as just a magic flag file inside it's .git/ directory. And that > awareness would be what simply disabled pruning or "repack -d" within that > particular instance. That prohibits the problem in your supermodule and your instance of the given submodule. But IMHO, using a submodule commit which could be removed by pruning in another instance of the submodule is really not the thing you ever want. If you start your own branch in a submodule, and start to rely on it in the supermodule, you _will_ want to push this to the submodule upstream. And if you find that you have to rebase in the submodule, you simply have to rewrite your branch commits in the supermodule too. Otherwise, you effectively fork the submodule project purely for your superproject. So I suppose that in practical use, pruning in submodules probably would not have any negative effect. If it has, you made something wrong. So you probably should only a submodule commit if it has "publishing quality" (unless being on a temporary supermodule branch). Ie. any "borrowers" file should be empty. ``` ## Michael K. Edwards, 2006-12-04 18:56 Subject: Re: [RFC] Submodules in GIT Message-ID: URL: https://gitlist.dev/e/f2b55d220612041056w68db5891t105054c0d35efe45%40mail.gmail.com In-Reply-To: ``` (I wrote most of this a couple of days ago, so it's not at the tip of the conversational tree, so to speak. But it's effectively a response to Linus's "what do you want to do with submodules" question, with some thoughts on implementation. Sorry it's so long; like Blaise Pascal, "I would have written a shorter letter, but I did not have the time.") The supermodule concept, implemented right, could really improve cooperation among embedded platform integrators, boutique distro publishers, and other editorial contributors to sprawling metaprojects who don't want to run kernel.org-scale mirrors. To make this work, you need sparse repositories (conserving resources when fetching, by omitting the bulk of currently un-needed submodules that can reliably be obtained later from elsewhere) and shallow cloning (conserving resources when publishing, by referring cloners to a third-party repository for universally available content). For instance, it would be a wonderful thing if the pile-o-patches nightmare that is PTXdist (and crosstool and buildtool and every other approach I have seen for ongoing maintenance of embedded toolchains and userlands) were obsoleted by a git supermodule. Its submodules would mostly track external projects, but would also logically contain the fix-up patches worked out during platform integration, checked in to branches anchored at each upstream release point. The supermodule would contain all of the build automation, log auditing, and remote unit testing stuff, as well as the metadata for each submodule involved in this platform build cycle. At a content level, the sparsely populated / shallowly published supermodule wouldn't be much different from today's PTXdist. But the pay-off comes when you merge forward to a new release of some base component (compiler, library, etc.) and discover that some of your fix-ups have been adopted or obsoleted upstream, and new fix-ups are needed for components that depend on the updated bit, and the set of configurables has changed (for which you need to compensate in the meta-configurator). Instead of piling up versioned patch directories, you commit fix-ups to the sub-modules, which other integration branches can ignore (if they aren't affected), merge, or cherry-pick. As I understand it, in today's git, every content object is a patch to the _data_ of one and only one git repository, containing the label of the preceding _data_ state plus a diff of file contents and attributes. Assuming this model is retained, any clean state of a "leaf" module (one with no submodules) can be reached by replaying a series of patches, starting from the repository's root node (an empty directory with the hopefully unique label generated by init-db). The label (SHA1) of the last patch is therefore a perfectly good label for this _data_ state. If all we were trying to do with supermodules was to capture and track various states of the submodules' data, we could extend the format of content objects to include "state X of submodule with init-db label Y". That would have the effect of capturing submodule states as _data_ in non-"leaf" modules. We would have to help cloners find a place from which to pull these states, of course; and it's easy to get sidetracked onto that part of the problem. But that's not where the bang for the buck is in supermodules. The whole model of distributed supermodules, with references to slightly diverging submodules whose content should mostly be fetched from external sources, smells to me just like LVM. The external sources (like an LVM volume of which you have taken a "snapshot") make up the bulk of the content pool. They also give you a window into developments on the submodule's own branches (like being able to peek forward and merge changes from the original volume). The supermodule (the snapshot volume) provides most of the interesting refs (submodule commits referenced by supermodule tags and branch heads), along with enough "journaled" content to replay forward from some checkpoint guaranteed to be available in each external source to any of these refs. The implication here is that submodule states are not just SHA1 labels to be embedded within supermodule data diffs. One ought to be able to clone a supermodule without immediately cloning full copies of any of its submodules. This ought to populate the clone's content database with all of the quanta of submodule content that aren't guaranteed to be available from any not-too-stale submodule mirror. When cloning, you don't want to have to inspect every supermodule state for submodule states that are outside the global subset. So the supermodule needs to maintain a set of supplemental refs from which all referenced submodule states can be reached. This allows you to traverse the portion of the pool of submodule content that can't be reached from true submodule branch heads. On 12/1/06, Linus Torvalds wrote: > Yes, you do need to have a list of submodules somewhere, and you'd need to > maintain that separately. One of the results of having the submodules be > independent from the supermodule is that it's not all "automatically > integrated", and thus the supermodule does end up having to have things > like that maintained separately. This is not a defect; it's a virtue. It's important for every commit to the supermodule to contain the information of which submodule branches you're currently on and how far along them you've crawled. Any particular supermodule commit point is likely to reflect an integration milestone visible only to the person working at the supermodule level. No content object should ever cross a submodule boundary, because then you wouldn't be able to apply it to the submodule in isolation (or in another supermodule state) or identify it when it is applied upstream and propagates back to you in a pull. But the supermodule can also contain supplemental refs (heads and tags) that don't exist in the submodule (and shouldn't necessarily be pushed to it); the commits they refer to are localized to the submodule but may not be reachable from any of the submodule's branch heads. > And yes, if you screw that up, you wouldn't be able to fetch submodules > properly etc, even if you see the supermodule, and yes, this sounds more > like the CVS "Entries" kind of file that is more "tacked on" than really > deeply integrated. But I think the separation is _more_ than worth the > fact that you can see things being separate. There is an opportunity for useful deep integration here. The same algorithm that does reachability analysis for "git prune" can dig from supermodule down to submodules, copying objects into the supermodule database until it hits a commit that is advertised as "global" by the submodule. "git clone" of the supermodule can then pull the bulk of the submodules (a superset of the "global" subset) from (a mirror of) the canonical place for each, and use the supermodule object database as an alternate source for commits that don't exist in the "canonical" submodule. > In fact, I'm very much arguing for keeping things as separate as possible, > while just integrating to the smallest possible degree (just _barely_ > enough that you can do things like "git clone" and it will fetch multiple > repositories and put them all in the right places, and "git diff" and > friends will do reasonably sane things). > > Keep it simple, stupid. As simple as possible; but no simpler. The "alternates" / "git clone --reference" model is already almost powerful enough for the supermodule to contain a "journal" of submodule commits that haven't yet been retired to the canonical subset (guaranteed present in each mirror). The only difference is that the supermodule should be considered a "weak alternates" source. Commit objects in the supermodule's database should be visible to submodule-level operations (so that commits which are accepted upstream get flowed in nicely during "git pull"). But if a commit becomes reachable from a ref that is really in the submodule (not just one of the supermodule's "supplemental refs", which should _not_ be visible to submodule operations), then it should be copied into the submodule's object database. (The refs internal to the submodule should retain their integrity even if the supermodule is inaccessible.) The existing "strong alternates" mechanism should be reserved for repos which are at least as public and persistent as the submodule, and supermodules don't qualify (e. g., Linus's transmeta scenario). > On Sat, 2 Dec 2006, Josef Weidendorfer wrote: > > The thing I wanted to discuss is whether such names would need to be globally > > unique in the project containing submodles, or not. > > My preference would be for it to be "local", just because (as I > mentioned), with mirroring etc, it might well be that you want to fetch > things from the _closest_ repository. That's really not a global decision, > it's a local one. I think "global resource, local provider" is the way to go, with each provider advertising what checkpoints of what resources it can supply. When I clone or pull, I should be able to consult a local mapping of submodule URIs to "mirrors" (which may well be local repositories containing content and branches that aren't in the "official" upstream). The only thing that may need "global" agreement is the boundaries of the "global" subset for each submodule, i. e., the set of commit objects that can reliably be obtained from any mirror of the "official" upstream repository. That doesn't need to be terribly clever; "at least three days old on a globally published branch" would probably be a perfectly good heuristic. > > If yes, it IMHO makes a lot of sense to introduce "submodule objects" which contain > > these submodule names, and which are used as pointers to submodule commits in > > supermodule trees. > > You could do it that way, and then it would be global. It would work, and > in many ways it would probably be "simpler" on a supermodule level. I think the implication of "submodule objects" is that supermodule diffs would say "roll submodule X from commit-id A to commit-id B". I don't think that would work very well for pulls/merges in the sparsely populated scenario, because you want to be able to pull the non-canonical subset of the individual diffs between states A and B into the supermodule's object pool. When you decide later to flesh out submodule X, you should only have to clone some canonical mirror and then fast-forward to state B using objects you already have in the supermodule pool. The merge case is even clearer. Suppose I pull updates from two remote branches of the supermodule onto my master branch. Each remote branch has added the same submodule, cloned from third-party repositories whose clone history goes back to the same origin. (The example I have in mind is when some project switches to git from some other SCM, and the maintainers of the remote branches port their integration patches over from their git-svn tracker submodule to a clone of upstream's new git repo.) I should be able to postpone the merge effort, come back later and clone the upstream repo, then merge the non-canonical commits that were pulled earlier. I might want to decide at supermodule pull time to postpone pulling the bodies of the submodule commits; but I want the full sequence of submodule commit IDs in the supermodule commit object. So it's not so much the supermodule _state_ that has a hierarchical structure; it's the supermodule _diffs_ and _object_pool_ that become hierarchical. > The advantage of a global namespace is that you can much more easily > update it - "git fetch" will just fetch the new file(s) that describe the > subprojects very naturally if they are all global. Putting them in a local > .git/config file has it's advantages (see above), but it also makes it > very hard to version them, and to update the list - it would have to > become manual. I think the only global-to-local-namespace mapping applies to the different labels for the "empty repository" state generated at init-db time. Given the init-db SHA1 of the linux kernel repository, I should be able to choose any mirror or clone of that repository as a source for objects in its "global set". I expect this provider not to scribble on globally published branches, but that isn't even all that critical; anything outside the canonical set is kept in the supermodule's object pool, so I can always blow the submodule away and regenerate it from a different mirror. > There are possibly combinations of the two approaches: have a "global > namespace" that describes the canonical place to get the subprojects, but > have some way to add local "translation" of the canonical names into > locally preferred versions (eg you could just have a way to say "this is > the local mirror for that global canonical place") > > Maybe that would work? Sure. But all you really need from the canonical place is its init-db SHA1 (permanent) and its list of globally published branches (monotonically expanding). A URL for it is a convenient shorthand but doesn't have to be persistent. Cheers, ``` ## Torgil Svensson, 2006-12-04 20:26 Subject: Re: [RFC] Submodules in GIT Message-ID: URL: https://gitlist.dev/e/e7bda7770612041226j4d4a5584m279afa9a2d7dfe74%40mail.gmail.com In-Reply-To: ``` On 12/3/06, Linus Torvalds wrote: > > > If you have an old irrelevant submodule in the history that happens to > > have the same name as one of them you are interested in, do you get > > this as well? > > It could be "solved" by simply having the requirement that all modules > need to be named differently (notice that "module name" is _not_ the same > thing as "the directory name where the module shows up". Okay, missed that part. I wasn't familiar with contents of the CVS modules files and misinterpreted your suggestion. MODULE [OPTIONS] [&OTHERMODULE...] [DIR] [FILES] So all this is UI only and the "normal" operations on the supermodule ``` ## Linus Torvalds, 2006-12-04 20:41 Subject: Re: [RFC] Submodules in GIT Message-ID: URL: https://gitlist.dev/e/Pine.LNX.4.64.0612041234390.3476%40woody.osdl.org In-Reply-To: ``` On Mon, 4 Dec 2006, Torgil Svensson wrote: > > Okay, missed that part. I wasn't familiar with contents of the CVS > modules files and misinterpreted your suggestion. > > MODULE [OPTIONS] [&OTHERMODULE...] [DIR] [FILES] > > So all this is UI only and the "normal" operations on the supermodule > will just ignore what's behind the commit-links? Right. That's how CVS modules work (although in the case of CVS modules, the "dir" thing is obviously there in the "modules" file, so it's not _purely_ UI in CVS - this would likely be different in a git implementation, because the "tree" object ends up telling not just the exact version, but the location too). So my suggestion basically boils down to: - "fetch" and "clone" etc will just look at the "modules" file, and recursively fetch/clone whatever the module files talks about. This is the "thin veneer to make it _look_ like git actually understands submodules" part. It woudln't really - they're very much tacked on. - the tree entries are what makes the "once you have all the submodule objects, this is how you can do 'diff' and 'checkout' on them, and this is what tells you the exact version that goes along with a particular supermodule version". In other words, the simple and stupid way to do this is to just consider these two things two totally independent issues, and have different mechanisms for telling different operations what to do. Is it "pretty"? No. The whole sub-module thing wouldn't be a tightly integrated low-level thing, it would very much be all about tracking multiple _separate_ git repositories, and just make them work well together. They'd very much still be separate, with just some simple infrastructure glue to make them look somewhat integrated. So yeah, it's a bit hacky, but for the reasons I've tried to outline, I actually think that users _want_ hacky. Exactly because "deep integration" ends up having so many _bad_ features, so it's better to have a thin and simple layer that you can actually see past if you want to. ``` ## Torgil Svensson, 2006-12-04 21:36 Subject: Re: [RFC] Submodules in GIT Message-ID: URL: https://gitlist.dev/e/e7bda7770612041336s73e677ebh758b030f9f75c1d8%40mail.gmail.com In-Reply-To: ``` On 12/4/06, Linus Torvalds wrote: > > So yeah, it's a bit hacky, but for the reasons I've tried to outline, I > actually think that users _want_ hacky. Exactly because "deep integration" > ends up having so many _bad_ features, so it's better to have a thin and > simple layer that you can actually see past if you want to. Thin and simple sounds very good. Let's try it with an example. Lets say we have one apllication App1 and three librarys (Lib1, Lib2, Lib3) with the following dependency-graph: App1 /\ / \ Lib1 Lib2 \ / \ / Lib3 (don't really needed for this example but looks nice) All components can be used individually and have their own upstream, maintainer etc. To compile App1 however, I need some files from both Lib1 and Lib2 specifying it's API. To satisfy these dependencies, It sounds reasonable to link Lib2 and Lib3 submodules from App1. In your concept, can I construct a modules file to fetch the API files and ``` ## Sam Vilain, 2006-12-05 01:31 Subject: Re: [RFC] Submodules in GIT Message-ID: <4574CBFF.8040708@vilain.net> URL: https://gitlist.dev/e/4574CBFF.8040708%40vilain.net In-Reply-To: ``` Michael K. Edwards wrote: > who don't want to run kernel.org-scale mirrors. To make this work, > you need sparse repositories (conserving resources when fetching, by > omitting the bulk of currently un-needed submodules that can reliably > be obtained later from elsewhere) and shallow cloning (conserving > resources when publishing, by referring cloners to a third-party > repository for universally available content). Did you see GitTorrent? http://gittorrent.utsl.gen.nz/ A lot of similar ideas to what you mention. Sorry, still no prototype :) I'd see the submodules thing as a good way to glue together a whole bunch of repositories, so that the core mirror servers only have to mirror a small-ish number of repositories. Sam. ``` ## Daniel Barkalow, 2006-12-05 02:33 Subject: Re: [RFC] Submodules in GIT Message-ID: URL: https://gitlist.dev/e/Pine.LNX.4.64.0612042053080.20138%40iabervon.org In-Reply-To: ``` On Sat, 2 Dec 2006, Linus Torvalds wrote: > So that's where I come from. And maybe I'm totally wrong. I'd like to hear > what people who actually _use_ submodules think. I think you'd rather hear from people who _would_ use submodules; I've worked on a number of projects that would have benefitted from that general functionality, but nobody trusted the implementation enough to actually use it. At my work, we're doing a bunch of stuff with microcontrollers. We've got about a dozen different boards with microcontrollers, and each of them has different firmware. We also have a bunch of code that can go on any of the boards. The way things are organized currently is that each board has its own project, and there's a "common-micro" project with the common code. This sort of works, but it means that when you change things in common-micro, you never know what effect this will have on boards other than the one you're actually working on. What I'd like to have is that each project has a "common-micro" subdirectory, and changes to each of these can be merged into each other, but that doesn't happen automaticly, and each board's revisions include the common-micro revision they were created with. A few notes: I'd never work on common-micro in isolation. Nothing in there even compiles by itself, because the compiler needs to know the target microcontroller type, which depends on the board it's for. It only makes sense to prepare a new revision of "common-micro" in the context of some particular board, at least if you want to test it at all. I'd sometimes want to include temporary hacks in the common-micro for a particular board, when things are late and I need to change some library behavior in a way that I know works for the board I'm working on, but I don't have time to think about all of the other boards (each of which is special in some way). I'd often make some change that I know improves the cleanliness of common-micro, but which requires changes to every board to compensate. I don't want to make all of the changes at once; I'll update each board appropriately the next time I work on it. But, of course, until I update each board, that board needs to keep using the version of common-micro without the changes. I don't want to have repository states where stuff doesn't work, which means I can't do it as one big tree; I need to be able to make a commit with board1 and a common-micro change without having a version of board2 that would use the changed common-micro, because I haven't come up with a board2 version that works with it yet. -Daniel ``` ## Uwe Kleine-Koenig, 2006-12-05 09:01 Subject: Re: [RFC] Submodules in GIT Message-ID: <20061205090125.GA2428@cepheus> URL: https://gitlist.dev/e/20061205090125.GA2428%40cepheus In-Reply-To: <456F29A2.1050205@op5.se> ``` Hello, Andreas Ericsson wrote: > The only problem I'm seeing atm is that the supermodule somehow has to > mark whatever commits it's using from the submodule inside the submodule > repo so that they effectively become un-prunable, otherwise the > supermodule may some day find itself with a history that it can't restore. One could circumvent that by creating a separate repo for the submodule at checkout time and pull the needed objects in the supermodule's odb when commiting the supermodule. This way prune in the submodule cannot do any harm, because in it's odb are no objects that are important for the supermodule. Uwe -- Uwe Kleine-Koenig ``` ## Andreas Ericsson, 2006-12-05 10:33 Subject: Re: [RFC] Submodules in GIT Message-ID: <45754AFE.1070207@op5.se> URL: https://gitlist.dev/e/45754AFE.1070207%40op5.se In-Reply-To: <20061205090125.GA2428@cepheus> ``` Uwe Kleine-Koenig wrote: > Hello, > > Andreas Ericsson wrote: >> The only problem I'm seeing atm is that the supermodule somehow has to >> mark whatever commits it's using from the submodule inside the submodule >> repo so that they effectively become un-prunable, otherwise the >> supermodule may some day find itself with a history that it can't restore. > One could circumvent that by creating a separate repo for the submodule > at checkout time and pull the needed objects in the supermodule's odb > when commiting the supermodule. This way prune in the submodule cannot > do any harm, because in it's odb are no objects that are important for > the supermodule. > Yes, but then you'd lose history connectivity (I'm assuming you'd only pull in the tree and blob objects from the submodule, and prefix the tree-entrys with whatever directory you're storing the submodul in). -- Andreas Ericsson andreas.ericsson@op5.se OP5 AB www.op5.se ``` ## Andreas Ericsson, 2006-12-05 10:38 Subject: Re: [RFC] Submodules in GIT Message-ID: <45754C0E.3070904@op5.se> URL: https://gitlist.dev/e/45754C0E.3070904%40op5.se In-Reply-To: ``` Linus Torvalds wrote: > > On Mon, 4 Dec 2006, Torgil Svensson wrote: >> Okay, missed that part. I wasn't familiar with contents of the CVS >> modules files and misinterpreted your suggestion. >> >> MODULE [OPTIONS] [&OTHERMODULE...] [DIR] [FILES] >> >> So all this is UI only and the "normal" operations on the supermodule >> will just ignore what's behind the commit-links? > > Right. That's how CVS modules work (although in the case of CVS modules, > the "dir" thing is obviously there in the "modules" file, so it's not > _purely_ UI in CVS - this would likely be different in a git > implementation, because the "tree" object ends up telling not just the > exact version, but the location too). > > So my suggestion basically boils down to: > > - "fetch" and "clone" etc will just look at the "modules" file, and > recursively fetch/clone whatever the module files talks about. This is > the "thin veneer to make it _look_ like git actually understands > submodules" part. It woudln't really - they're very much tacked on. > > - the tree entries are what makes the "once you have all the submodule > objects, this is how you can do 'diff' and 'checkout' on them, and this > is what tells you the exact version that goes along with a particular > supermodule version". > > In other words, the simple and stupid way to do this is to just consider > these two things two totally independent issues, and have different > mechanisms for telling different operations what to do. > > Is it "pretty"? No. The whole sub-module thing wouldn't be a tightly > integrated low-level thing, it would very much be all about tracking > multiple _separate_ git repositories, and just make them work well > together. They'd very much still be separate, with just some simple > infrastructure glue to make them look somewhat integrated. > > So yeah, it's a bit hacky, but for the reasons I've tried to outline, I > actually think that users _want_ hacky. Exactly because "deep integration" > ends up having so many _bad_ features, so it's better to have a thin and > simple layer that you can actually see past if you want to. > Indeed. With the "tight" integration option we'd also have to have the mechanism to rewrite the tree-entries with the location where the submodule is located in the working tree. This might be needed anyways, but it sure as hell seems a lot easier to just tack that part on when doing a checkout and actually creating all the files. -- Andreas Ericsson andreas.ericsson@op5.se OP5 AB www.op5.se ``` ## Andreas Ericsson, 2006-12-05 10:42 Subject: Re: [RFC] Submodules in GIT Message-ID: <45754D27.9070701@op5.se> URL: https://gitlist.dev/e/45754D27.9070701%40op5.se In-Reply-To: ``` Torgil Svensson wrote: > On 12/4/06, Linus Torvalds wrote: >> >> So yeah, it's a bit hacky, but for the reasons I've tried to outline, I >> actually think that users _want_ hacky. Exactly because "deep >> integration" >> ends up having so many _bad_ features, so it's better to have a thin and >> simple layer that you can actually see past if you want to. > > Thin and simple sounds very good. Let's try it with an example. Lets > say we have one apllication App1 and three librarys (Lib1, Lib2, Lib3) > with the following dependency-graph: > > App1 > /\ > / \ > Lib1 Lib2 > \ / > \ / > Lib3 (don't really needed for this example but looks nice) > > All components can be used individually and have their own upstream, > maintainer etc. > > To compile App1 however, I need some files from both Lib1 and Lib2 > specifying it's API. To satisfy these dependencies, It sounds > reasonable to link Lib2 and Lib3 submodules from App1. In your > concept, can I construct a modules file to fetch the API files and > their history without checking out the whole Lib1 and Lib2 source? I think not. Then it wouldn't be a submodule anymore, but just some random sources from an upstream project. Not that it's an uncommon workflow or anything, but it's sort of akin to just importing the SHA1 implementation (a few source-files with no real interest in the history of those source-files) from openssl into a different project rather than actually using the entire openssl lib (which would be nice to have as a submodule). -- Andreas Ericsson andreas.ericsson@op5.se OP5 AB www.op5.se ``` ## Jakub Narebski, 2006-12-05 11:01 Subject: Re: [RFC] Submodules in GIT Message-ID: URL: https://gitlist.dev/e/el3jeb%24gkr%241%40sea.gmane.org In-Reply-To: <45754C0E.3070904@op5.se> ``` Andreas Ericsson wrote: > Indeed. With the "tight" integration option we'd also have to have the > mechanism to rewrite the tree-entries with the location where the > submodule is located in the working tree. This might be needed anyways, > but it sure as hell seems a lot easier to just tack that part on when > doing a checkout and actually creating all the files. Excellent idea! This way most of the concerns for "separate repositories for submodules" layout about ability to rename directory the submodule resides in, or move submodule are resolved. The other part would be to use submodule-aware git-mv to move submodule(s). Perhaps the following solution would work best: * refs/submodules/ holds sha1 of top commit in submodule * objects/info/submodules is a file which can be automatically generated (or at least automatically updated) on checkout, with the following contents: TAB or SPC with the usual rule that # and ; means comment, \ at end of line is used for continuations, empty lines doesn't matter etc. -- Jakub Narebski Warsaw, Poland ShadeHawk on #git ``` ## Jakub Narebski, 2006-12-05 11:09 Subject: Re: [RFC] Submodules in GIT Message-ID: URL: https://gitlist.dev/e/el3jsh%24i62%241%40sea.gmane.org In-Reply-To: <45754D27.9070701@op5.se> ``` Andreas Ericsson wrote: > Torgil Svensson wrote: >> On 12/4/06, Linus Torvalds wrote: >>> >>> So yeah, it's a bit hacky, but for the reasons I've tried to outline, I >>> actually think that users _want_ hacky. Exactly because "deep >>> integration" >>> ends up having so many _bad_ features, so it's better to have a thin and >>> simple layer that you can actually see past if you want to. >> >> Thin and simple sounds very good. Let's try it with an example. Lets >> say we have one apllication App1 and three librarys (Lib1, Lib2, Lib3) >> with the following dependency-graph: >> >> App1 >> / \ >> / \ >> Lib1 Lib2 >> \ / >> \ / >> Lib3 (don't really needed for this example but looks nice) >> >> All components can be used individually and have their own upstream, >> maintainer etc. >> >> To compile App1 however, I need some files from both Lib1 and Lib2 >> specifying it's API. To satisfy these dependencies, It sounds >> reasonable to link Lib2 and Lib3 submodules from App1. In your >> concept, can I construct a modules file to fetch the API files and >> their history without checking out the whole Lib1 and Lib2 source? > > I think not. Then it wouldn't be a submodule anymore, but just some > random sources from an upstream project. Not that it's an uncommon > workflow or anything, but it's sort of akin to just importing the SHA1 > implementation (a few source-files with no real interest in the history > of those source-files) from openssl into a different project rather than > actually using the entire openssl lib (which would be nice to have as a > submodule). Note that this is what partial checkouts (another great idea nobody implemented yet[*1*]; you can do partial checkout but there is no UI for this, and working with partial checkouts is bit hard) is about, although it would buy you only working area space, and not repository (object database storage) space. For now, you can imitate this by having in in Lib1 and Lib2 the 'includes' branch which would contain only the API (and which you would have to keep up to date with 'master', but it should be fairly easy: just merge changes into 'includes', perhaps with help of git-rerere, or [nonexisting] git-rerere2). [*1*] Although with our track[*2*] I guess it is reasonable to think it would get implemented soon. [*2*] Out of four "great ideas": shallow clone / sparse clone, submodules support, lazy clone / remote alternates, two are in example-implementation (submodules support) and beta work (shallow clone is in 'next'). -- Jakub Narebski Warsaw, Poland ShadeHawk on #git ``` ## Jakub Narebski, 2006-12-05 11:11 Subject: Re: [RFC] Submodules in GIT Message-ID: URL: https://gitlist.dev/e/el3k0a%24i62%242%40sea.gmane.org In-Reply-To: <45754AFE.1070207@op5.se> ``` Andreas Ericsson wrote: > Uwe Kleine-Koenig wrote: > >> Andreas Ericsson wrote: >>> The only problem I'm seeing atm is that the supermodule somehow has to >>> mark whatever commits it's using from the submodule inside the submodule >>> repo so that they effectively become un-prunable, otherwise the >>> supermodule may some day find itself with a history that it can't restore. >>> >> One could circumvent that by creating a separate repo for the submodule >> at checkout time and pull the needed objects in the supermodule's odb >> when commiting the supermodule. This way prune in the submodule cannot >> do any harm, because in it's odb are no objects that are important for >> the supermodule. > > Yes, but then you'd lose history connectivity (I'm assuming you'd only > pull in the tree and blob objects from the submodule, and prefix the > tree-entrys with whatever directory you're storing the submodul in). I thought that Uwe meant pulling (getting) _all_ the needed objects from submodule object repository into supermodule object repository: commits, trees and blobs, full history. -- Jakub Narebski Warsaw, Poland ShadeHawk on #git ``` ## Uwe Kleine-Koenig, 2006-12-05 15:02 Subject: Re: [RFC] Submodules in GIT Message-ID: <20061205150217.GA5573@cepheus> URL: https://gitlist.dev/e/20061205150217.GA5573%40cepheus In-Reply-To: <45754AFE.1070207@op5.se> ``` Hella Andreas, Andreas Ericsson wrote: > >>The only problem I'm seeing atm is that the supermodule somehow has to > >>mark whatever commits it's using from the submodule inside the submodule > >>repo so that they effectively become un-prunable, otherwise the > >>supermodule may some day find itself with a history that it can't restore. > >One could circumvent that by creating a separate repo for the submodule > >at checkout time and pull the needed objects in the supermodule's odb > >when commiting the supermodule. This way prune in the submodule cannot > >do any harm, because in it's odb are no objects that are important for > >the supermodule. > > Yes, but then you'd lose history connectivity (I'm assuming you'd only > pull in the tree and blob objects from the submodule, and prefix the > tree-entrys with whatever directory you're storing the submodul in). That's the reason for me prefering to pull in the complete commit. I don't understand what you mean with "prefix the tree-entrys with whatever directory you're storing the submodul in". Maybe one of us doesn't understand tree objects correctly. AFAICT they don't store the location where they occur, so there is no need to store a prefix. E.g. zeisberg@cepheus:/tmp$ mkdir test-repo zeisberg@cepheus:/tmp$ cd test-repo/ zeisberg@cepheus:/tmp/test-repo$ git-init-db defaulting to local storage area zeisberg@cepheus:/tmp/test-repo$ echo LD_FLAGS=-ltest > Makefile zeisberg@cepheus:/tmp/test-repo$ git add Makefile zeisberg@cepheus:/tmp/test-repo$ git commit -m 'test1' Committing initial tree 754eadab39642175748bb02155d2959176bcf014 zeisberg@cepheus:/tmp/test-repo$ mkdir subdir zeisberg@cepheus:/tmp/test-repo$ cp Makefile subdir/ zeisberg@cepheus:/tmp/test-repo$ git add subdir/ zeisberg@cepheus:/tmp/test-repo$ git commit -m 'test2' zeisberg@cepheus:/tmp/test-repo$ git ls-tree HEAD 100644 blob 610bafd79f92c7e546b104d5b22795df1f099723 Makefile 040000 tree 754eadab39642175748bb02155d2959176bcf014 subdir So the tree that only contains the Makefile specifing LD_FLAGS has the sha1id 754eadab39642175748bb02155d2959176bcf014 independent of being the root of my project or a subtree. But maybe I misunderstood you? Best regards Uwe -- Uwe Kleine-Koenig If a lawyer and an IRS agent were both drowning, and you could only save ``` ## Andreas Ericsson, 2006-12-05 15:30 Subject: Re: [RFC] Submodules in GIT Message-ID: <457590AD.4000806@op5.se> URL: https://gitlist.dev/e/457590AD.4000806%40op5.se In-Reply-To: <20061205150217.GA5573@cepheus> ``` Uwe Kleine-Koenig wrote: > Hella Andreas, > > Andreas Ericsson wrote: >>>> The only problem I'm seeing atm is that the supermodule somehow has to >>>> mark whatever commits it's using from the submodule inside the submodule >>>> repo so that they effectively become un-prunable, otherwise the >>>> supermodule may some day find itself with a history that it can't restore. >>> One could circumvent that by creating a separate repo for the submodule >>> at checkout time and pull the needed objects in the supermodule's odb >>> when commiting the supermodule. This way prune in the submodule cannot >>> do any harm, because in it's odb are no objects that are important for >>> the supermodule. >> Yes, but then you'd lose history connectivity (I'm assuming you'd only >> pull in the tree and blob objects from the submodule, and prefix the >> tree-entrys with whatever directory you're storing the submodul in). > That's the reason for me prefering to pull in the complete commit. > > I don't understand what you mean with "prefix the tree-entrys with > whatever directory you're storing the submodul in". > Maybe one of us doesn't understand tree objects correctly. AFAICT they > don't store the location where they occur, so there is no need to store > a prefix. E.g. > > 100644 blob 610bafd79f92c7e546b104d5b22795df1f099723 Makefile > 040000 tree 754eadab39642175748bb02155d2959176bcf014 subdir > > So the tree that only contains the Makefile specifing LD_FLAGS has the > sha1id 754eadab39642175748bb02155d2959176bcf014 independent of being the > root of my project or a subtree. > > But maybe I misunderstood you? > Nopes. I just didn't think of the fact that subtrees are trees and never store any path-info no matter what. So basically the supermodule can store all trees of all submodules for each commit adding a new submodule revision (which is neat, since "casuals" never have to bother with getting all the submodules if they want to see all the code used in any particular revision), while we invent the new tree object "subm" that points to a commit in the submodule repo. We then teach the tools to recognize when the *real* submodule repo is present and just don't check out trees from the supermodule odb that lead us to directories where submodules reside. Simple and beautiful. Me likes. *IF* we teach the history viewers about submodules is a different matter though. I'm not sure it would make much sense to have simple text-mode browsers show the submodule history, although I can imagine qgit and gitk wanting to take advantage of their nice side-by-side DAG displaying code to show all the repos in parallell, or link between them in some point-and-click kind of way. -- Andreas Ericsson andreas.ericsson@op5.se OP5 AB www.op5.se ``` ## Sven Verdoolaege, 2006-12-05 16:00 Subject: Re: [RFC] Submodules in GIT Message-ID: <20061205160019.GR940MdfPADPa@greensroom.kotnet.org> URL: https://gitlist.dev/e/20061205160019.GR940MdfPADPa%40greensroom.kotnet.org In-Reply-To: <20061205090125.GA2428@cepheus> ``` On Tue, Dec 05, 2006 at 10:01:25AM +0100, Uwe Kleine-Koenig wrote: > Hello, > > Andreas Ericsson wrote: > > The only problem I'm seeing atm is that the supermodule somehow has to > > mark whatever commits it's using from the submodule inside the submodule > > repo so that they effectively become un-prunable, otherwise the > > supermodule may some day find itself with a history that it can't restore. > One could circumvent that by creating a separate repo for the submodule > at checkout time and pull the needed objects in the supermodule's odb > when commiting the supermodule. This way prune in the submodule cannot > do any harm, because in it's odb are no objects that are important for > the supermodule. I _think_ Linus argued against doing this (for scalability reasons), although he didn't actually answer my question when I asked him directly. In his proposal you wouldn't need to do this, because the particular checked-out copy of the submodule that is located in a subdirectory of a superproject would not be allowed to be pruned and it seems that Martin has also implemented it like this. ``` ## sf, 2006-12-05 22:07 Subject: Re: [RFC] Submodules in GIT Message-ID: <4575EDA7.2090705@stephan-feder.de> URL: https://gitlist.dev/e/4575EDA7.2090705%40stephan-feder.de In-Reply-To: ``` Daniel Barkalow wrote: > On Sat, 2 Dec 2006, Linus Torvalds wrote: > >> So that's where I come from. And maybe I'm totally wrong. I'd like to hear >> what people who actually _use_ submodules think. > > I think you'd rather hear from people who _would_ use submodules; I've > worked on a number of projects that would have benefitted from that > general functionality, but nobody trusted the implementation enough to > actually use it. > > At my work, we're doing a bunch of stuff with microcontrollers. We've got > about a dozen different boards with microcontrollers, and each of them has > different firmware. We also have a bunch of code that can go on any of the > boards. > > The way things are organized currently is that each board has its own > project, and there's a "common-micro" project with the common code. This > sort of works, but it means that when you change things in common-micro, > you never know what effect this will have on boards other than the one > you're actually working on. What I'd like to have is that each project has > a "common-micro" subdirectory, and changes to each of these can be merged > into each other, but that doesn't happen automaticly, and each board's > revisions include the common-micro revision they were created with. Our setup and requirements at work are exactly the same: We have a few main projects that are developed independently and we have one "helper" project for code that is general enough to be reused. So work on the helper project is only done while working on one of the main projects. When we switch to another main project we integrate the changes to the "helper" project. That's the theory, at least. Regards Stephan ``` ## Jon Loeliger, 2006-12-08 18:29 Subject: Re: [RFC] Submodules in GIT Message-ID: <1165602554.19135.309.camel@cashmere.sps.mot.com> URL: https://gitlist.dev/e/1165602554.19135.309.camel%40cashmere.sps.mot.com In-Reply-To: ``` On Fri, 2006-12-01 at 14:13, Linus Torvalds wrote: > So this is why it's really important that the submodule really is a git > repository in its own right, and why committing stuff in the supermodule > NEVER affect the submodule itself directly (it might _cause_ you to also > do a commit in the submodule indirectly, but the submodule commit MUST be > totally independent, and stand on its own). An implication of this is that the entire administrative responsibility for having some super-sub module interaction lies entirely with the supermodule. Why not have a "glue" object at the "stub"-interface of the supermodule tree that provides policy mappings to the sub-modules. Perhaps indicating git URL location, mappings of branch names between super- and sub- modules, special commit SHA1s, user policy or config choices at the boundary, and things like that. Is that the sort of direction we are headed? jdl ``` ## Sven Verdoolaege, 2006-12-08 18:45 Subject: Re: [RFC] Submodules in GIT Message-ID: <20061208184532.GE940MdfPADPa@greensroom.kotnet.org> URL: https://gitlist.dev/e/20061208184532.GE940MdfPADPa%40greensroom.kotnet.org In-Reply-To: <1165602554.19135.309.camel@cashmere.sps.mot.com> ``` On Fri, Dec 08, 2006 at 12:29:14PM -0600, Jon Loeliger wrote: > Why not have a "glue" object at the "stub"-interface of > the supermodule tree that provides policy mappings to > the sub-modules. Perhaps indicating git URL location, > mappings of branch names between super- and sub- modules, > special commit SHA1s, user policy or config choices at > the boundary, and things like that. > > Is that the sort of direction we are headed? Not unless you have something useful in mind that could be put in these glue objects. URLs and branch names, in particular, should not be stored in the repository itself, but in configuration files, since they will be different for different copies of the repo. ``` ## R. Steve McKown, 2006-12-09 21:34 Subject: Re: [RFC] Submodules in GIT Message-ID: <200612091434.15001.rsmckown@yahoo.com> URL: https://gitlist.dev/e/200612091434.15001.rsmckown%40yahoo.com In-Reply-To: ``` On Saturday 02 December 2006 12:41 pm, Linus Torvalds wrote: > In other words, I _suspect_ that that is really what module users are all > about. They want the ability to specify an arbitrary collection of these > atomic snapshots (for releases etc), and just want a way to copy and move > those things around, and are less interested in making everything else > very seamless (because most people are happy to do the actual > _development_ entirely within the submodules, so the "development" part > is actually not that important for the supermodule, the supermodule is > mostly for aggregation and snapshots, and tying different versions of > different submodules together). > > So that's where I come from. And maybe I'm totally wrong. I'd like to hear > what people who actually _use_ submodules think. Here's some thoughts on subprojects from my company's perspective. I apologize for the long message. Abstract: We use submodules heavily in CVS and SVN. I like what I've read from Linus about the "thin veneer" approach of integrating subprojects. It seems conceptually to provide the support we desire. For us, it's important that the mandated linkage between a master project and a subproject is minimal to maximize our flexibility in building our processes. We develop and maintain a lot of embedded applications. Both for higher level systems (ex: 32MB RAM/32MB storage) running the Linux kernel and a customized set of libs/app support code and more deeply embedded environments (ex: 8KB of RAM and 32KB of storage). Even though these two cases are very different in many repects, the version management issues are the same. - We (mostly) track everything needed to build historical versions of code with 100% fidelity. This includes all of the tools used to compile, build, test, deploy, debug, etc. the actual build results themselves. I initially looked at Vesta several years ago. I love their conceptual approach to this problem (integrated build system that caches mid-level build results within the repository itself), but it's too unwieldy, very hard to set up (lots of up-front effort), and lacks many useful features. - Most of our "applications" are a relatively small amount of app-specific code with references to several/many shared modules. Shared modules can contain support tools, like build/test/debug/deploy support for a given embedded platform, in-house developed shared app code, or shared code developed by third parties. - We use CVS to manage our larger system development projects. The repo is about 2GB and has several dozen application-code submodules. We use the "third party sources" approach to tracking submodules as outlined in Ch.13 of the CVS manual. Additionally, we manage our "buildox" (similar to buildroot in concept) in another CVS repo. All prior interesting versions of the buildroot can be built from source (toolchains, everything), if necessary. Applications contain metadata (a file...) in the repo so the app-level build system can ensure it is being ran under the correct version of buildbox; clunky but serviceable. CVS is a nightmare because of its poor branch/tagging facilities, and many of the things we *ought* to be doing with revision control we don't because of the complexity. - We use SVN to manage our deeply embedded system projects. The repo is about 250MB in size. Applications use the svn:externals property to reference needed modules. We aren't using a buildbox in this environment yet (bad!). SVN's simple branching and svn:externals are a giant leap forward in comparison to CVS's capabilities. Below are some common use case scenarios that are to varying degrees unweildy in CVS and/or SVN. Many of these involving non-trivial branching and merging operations are nearly impractical in CVS, and the lack of merge tracking (to support repeated safe merging from one branch to another) makes some of these a bit tricky in SVN too. Of course neither repo supports disconnected/distributed operation, which would make a number of activities that much simpler as well. - Round trip module management. A specific app requires a change to a shared module, so it makes a local branch to develop the change. The "diff" is presented to the maintainer (who may be inhouse). The next interesting maintainer version of the module gets imported into our repo (if in house, it's already there), where the app can reference it. This merge process may leave changes not yet implemented (or never to be implemented) by the module maintainer in the local branch used by the apps. Other apps are unaffected, as they are linking to a prior version in the local branch. - Pragmatic development. It's typical that in developing an application, a developer will need to simultaneously make changes to one or more submodules. If more than trivial, he/she should branch the submodules and continually tracking the HEAD of those branches in the relevant app. This is so complex and fraught with problems in CVS that it doesn't get done, and developers house too much change over time in their working directories. With SVN and svn:externals, the process is workable. It is nice that an svn:external can point to (the HEAD of) a branch when making changes. - An application implements a new feature internally (say support for a new digital chipset in the embedded world) which later needs to be "promoted" to a subproject for use by others. Pretty easy in SVN. A challenge in CVS; it's really not possible to "convert" app code into a "third party source" and retain an historical link. - Updating build tools. In concept no different than updating a shared code module. In practice, due to the buildbox strategy, it's a bit convoluted. I don't expect this to get much smoother. Getting Vesta-like features, where integrated build suport can cache lower-level build results in a version-safe manner (like the binary code built when the cross toolchain was built) would be killer, but that's surely OT for the submodules discussion. Thanks, ``` ## Torgil Svensson, 2006-12-10 11:47 Subject: Re: [RFC] Submodules in GIT Message-ID: URL: https://gitlist.dev/e/e7bda7770612100347j78854d79x547084972ed14e99%40mail.gmail.com In-Reply-To: <200612091434.15001.rsmckown@yahoo.com> ``` What if we use linus "module" file concept and allow the link objects to track subtrees? An object may look like this: commit: link: /path/to/remote/tree/or/blob Tracking upstream library: -------------------------- clone as usual Inhouse libraries/applications: ------------------------------- To satisfy versioning of build-dependencies - make links of type "external/lib1_header.h" -> "/headers/lib1_header.h" (blob) "external/lib1_interface" -> "/api" (tree) If git supports "sparse fetching" of subtrees we can follow the history in the submodule only concerning the files we want without fetching the whole subtree. "modules" file could specify something like "always clone on fetch" Build environment ------------------ First make links to all tools, applications, etc ... "buildtools/random_app1" -> "/" "buildtools/random_app2" -> "/" "sub_build_projects/user_interface" -> "/" "sub_build_projects/kernel" -> "/" "apps/special_app1" -> "/" "libs/special_lib1" -> "/special/lib/binary/path" Here we can have a build system that for example creates a "i386" folder and the repo itself Documentation release ---------------------- "Lib1/" -> "/docs" "Lib2/" -> "/docs" "App1/" -> "/docs" Special customer release for a specific HW platform --------------------------------------------------- "Lib1/lib1.h" -> "/headers/lib1.h" "Lib1/lib1.so" -> "/i386/Lib1/lib1.so" "Lib1/docs" -> "/docs" "App1_binary" -> "/i386/App1/App1_binary" "docs" -> "/docs" commit&tag&bag this and send to customer. If the customer says something is broken, we can make an SHA1 of the customers tree and immediately see if there's changes not belonging to us. Now this can be broken in so many ways that I can't even count, so I appreciate some feedback to correct my head. On 12/9/06, R. Steve McKown wrote: > On Saturday 02 December 2006 12:41 pm, Linus Torvalds wrote: > > In other words, I _suspect_ that that is really what module users are all > > about. They want the ability to specify an arbitrary collection of these > > atomic snapshots (for releases etc), and just want a way to copy and move > > those things around, and are less interested in making everything else > > very seamless (because most people are happy to do the actual > > _development_ entirely within the submodules, so the "development" part > > is actually not that important for the supermodule, the supermodule is > > mostly for aggregation and snapshots, and tying different versions of > > different submodules together). > > > > So that's where I come from. And maybe I'm totally wrong. I'd like to hear > > what people who actually _use_ submodules think. > > Here's some thoughts on subprojects from my company's perspective. I > apologize for the long message. > > Abstract: We use submodules heavily in CVS and SVN. I like what I've read > from Linus about the "thin veneer" approach of integrating subprojects. It > seems conceptually to provide the support we desire. For us, it's important > that the mandated linkage between a master project and a subproject is > minimal to maximize our flexibility in building our processes. > > > We develop and maintain a lot of embedded applications. Both for higher level > systems (ex: 32MB RAM/32MB storage) running the Linux kernel and a customized > set of libs/app support code and more deeply embedded environments (ex: 8KB > of RAM and 32KB of storage). Even though these two cases are very different > in many repects, the version management issues are the same. > > - We (mostly) track everything needed to build historical versions of code > with 100% fidelity. This includes all of the tools used to compile, build, > test, deploy, debug, etc. the actual build results themselves. I initially > looked at Vesta several years ago. I love their conceptual approach to this > problem (integrated build system that caches mid-level build results within > the repository itself), but it's too unwieldy, very hard to set up (lots of > up-front effort), and lacks many useful features. > > - Most of our "applications" are a relatively small amount of app-specific > code with references to several/many shared modules. Shared modules can > contain support tools, like build/test/debug/deploy support for a given > embedded platform, in-house developed shared app code, or shared code > developed by third parties. > > - We use CVS to manage our larger system development projects. The repo is > about 2GB and has several dozen application-code submodules. We use the > "third party sources" approach to tracking submodules as outlined in Ch.13 of > the CVS manual. Additionally, we manage our "buildox" (similar to buildroot > in concept) in another CVS repo. All prior interesting versions of the > buildroot can be built from source (toolchains, everything), if necessary. > Applications contain metadata (a file...) in the repo so the app-level build > system can ensure it is being ran under the correct version of buildbox; > clunky but serviceable. CVS is a nightmare because of its poor > branch/tagging facilities, and many of the things we *ought* to be doing with > revision control we don't because of the complexity. > > - We use SVN to manage our deeply embedded system projects. The repo is about > 250MB in size. Applications use the svn:externals property to reference > needed modules. We aren't using a buildbox in this environment yet (bad!). > SVN's simple branching and svn:externals are a giant leap forward in > comparison to CVS's capabilities. > > > Below are some common use case scenarios that are to varying degrees unweildy > in CVS and/or SVN. Many of these involving non-trivial branching and merging > operations are nearly impractical in CVS, and the lack of merge tracking (to > support repeated safe merging from one branch to another) makes some of these > a bit tricky in SVN too. Of course neither repo supports > disconnected/distributed operation, which would make a number of activities > that much simpler as well. > > - Round trip module management. A specific app requires a change to a shared > module, so it makes a local branch to develop the change. The "diff" is > presented to the maintainer (who may be inhouse). The next interesting > maintainer version of the module gets imported into our repo (if in house, > it's already there), where the app can reference it. This merge process may > leave changes not yet implemented (or never to be implemented) by the module > maintainer in the local branch used by the apps. Other apps are unaffected, > as they are linking to a prior version in the local branch. > > - Pragmatic development. It's typical that in developing an application, a > developer will need to simultaneously make changes to one or more submodules. > If more than trivial, he/she should branch the submodules and continually > tracking the HEAD of those branches in the relevant app. This is so complex > and fraught with problems in CVS that it doesn't get done, and developers > house too much change over time in their working directories. With SVN and > svn:externals, the process is workable. It is nice that an svn:external can > point to (the HEAD of) a branch when making changes. > > - An application implements a new feature internally (say support for a new > digital chipset in the embedded world) which later needs to be "promoted" to > a subproject for use by others. Pretty easy in SVN. A challenge in CVS; > it's really not possible to "convert" app code into a "third party source" > and retain an historical link. > > - Updating build tools. In concept no different than updating a shared code > module. In practice, due to the buildbox strategy, it's a bit convoluted. I > don't expect this to get much smoother. Getting Vesta-like features, where > integrated build suport can cache lower-level build results in a version-safe > manner (like the binary code built when the cross toolchain was built) would > be killer, but that's surely OT for the submodules discussion. > > Thanks, > Steve > - > To unsubscribe from this list: send the line "unsubscribe git" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ``` ## Andreas Ericsson, 2006-12-12 08:32 Subject: Re: [RFC] Submodules in GIT Message-ID: <457E692E.7060708@op5.se> URL: https://gitlist.dev/e/457E692E.7060708%40op5.se In-Reply-To: <1165602554.19135.309.camel@cashmere.sps.mot.com> ``` Jon Loeliger wrote: > On Fri, 2006-12-01 at 14:13, Linus Torvalds wrote: > >> So this is why it's really important that the submodule really is a git >> repository in its own right, and why committing stuff in the supermodule >> NEVER affect the submodule itself directly (it might _cause_ you to also >> do a commit in the submodule indirectly, but the submodule commit MUST be >> totally independent, and stand on its own). > > An implication of this is that the entire administrative > responsibility for having some super-sub module interaction > lies entirely with the supermodule. > That's a good thing. I wouldn't want the openssl maintainers to have to bother with every project that uses their code, and I'm fairly certain they feel the same. -- Andreas Ericsson andreas.ericsson@op5.se OP5 AB www.op5.se ``` ## Torgil Svensson, 2006-12-14 21:27 Subject: Re: [RFC] Submodules in GIT Message-ID: URL: https://gitlist.dev/e/e7bda7770612141327r11368e4dtabe8077e96545040%40mail.gmail.com In-Reply-To: ``` On 12/10/06, Torgil Svensson wrote: > What if we use linus "module" file concept and allow the link objects > to track subtrees? An object may look like this: > > commit: > link: /path/to/remote/tree/or/blob > Special customer release for a specific HW platform > --------------------------------------------------- > "Lib1/lib1.h" -> "/headers/lib1.h" > "Lib1/lib1.so" -> "/i386/Lib1/lib1.so" > "App1_binary" -> "/i386/App1/App1_binary" This example is somewhat complex since the build for lib1.so and the header-file might not has gone through the same commit on the lib1 subproject. Consider this example: lib1 - library project (source tracking) ------------------------------------------ Blob: /src/lib1.h app1 - application project (source tracking) ----------------------------------------- Link: /headers/lib1.h -> /src/lib1.h build1 - Build project (binary build tracking) ------------------------------------ Link: /src/lib1 -> / Link: /src/app1 -> / Blob: /i386/lib1/lib1.so Blob: /i386/app1/app1 Release Project (file compilation tracking) ----------------------------------- Link: /headers/lib1.h -> /src/lib1.h Link: /bin/lib1.so -> /i386/lib1/lib1.so Link: /bin/app1 -> /i386/app1/app1 , and should be the same, dictated by the app1 project. Can we enforce this in the modules file or should the different supermodules fix this somehow using scripts/hooks? How do the super-projects in this case get access to the blobs pointed by the links - transparent or explicit in the build-process? ``` ## Josef Weidendorfer, 2006-12-14 23:07 Subject: Re: [RFC] Submodules in GIT Message-ID: <200612150007.44331.Josef.Weidendorfer@gmx.de> URL: https://gitlist.dev/e/200612150007.44331.Josef.Weidendorfer%40gmx.de In-Reply-To: ``` On Thursday 14 December 2006 22:27, Torgil Svensson wrote: > This example is somewhat complex since the build for lib1.so and the > header-file might not has gone through the same commit on the lib1 > subproject. Consider this example: If you want to track build results for some source, why would you ever want these builds go out of sync with the source? As the built files depend on the source (and other things), the source should be a submodule of the build project. Hmm... I think I see a problem / wish for submodules here. With the current submodule proposal, we force submodules to be subdirectories inside of a supermodule. Your example has the folling submodule dependence ("X ==> Y" means Y being a submodule of X): App ==> Lib ^ ^ | | AppBuild ==> LibBuild If we force submodules to be subdirectories of supermodules, Lib needlessly will have to appear two times in a checkout of AppBuild. However, there is nothing wrong with it. Yet, you perhaps want the 2 Lib submodules not to go out of sync. This easily can be done with symlinking the Lib checkouts. As they are submodules, everything should work fine. Perhaps an option you want to have is to force a checkout of AppBuild to make these symlinking itself when it detects identical submodules links. Hmmm... the only problem with a symlink is that it can go wrong when moved. Unfortunately, I do not have a good solution for this. We can not make UNIX symlinks smart in any way. Hardlinking directories would be a solution, but that is not possible. Another thing: With normal "$buildroot != $srcroot" environments, the source can not be a subdirectory of the build directory. Yet, we want to specify submodule/supermodule relation. This is difficult to do with a submodule object, as it needs to appear in trees in the supermodule. Actually, the best workaround for this is to make Lib a direct submodule of AppBuild, and specify the relationship of LibBuild ==> Lib only in AppBuild. BTW, build project commits probably should not depend on any history of other build commits. So you actually want all build commits to be root commits, and have a tag name which could include the source commit id from which the build was done. This gives some loose coupling. > Link: /headers/lib1.h -> /src/lib1.h > Link: /bin/lib1.so -> /i386/lib1/lib1.so > Link: /bin/app1 -> /i386/app1/app1 > > > , and should be the same, > dictated by the app1 project. I do not see any problem here. Symlinks are stored in the git repository. As the AppBuild commit depends on App and LibBuild submodule commits, the symlinks always should be correct. > Can we enforce this in the modules file > or should the different supermodules fix this somehow using > scripts/hooks? I do not see any need for an hook. But of course, a checkout hook should be able to generate files/links. However, IMHO this should be not done with hooks but with Makefile targets. > How do the super-projects in this case get access to the blobs pointed > by the links - transparent or explicit in the build-process? Submodules should automatically be checked out when checking out the supermodule. So the blobs should already be there. Or do I miss something? ``` ## Torgil Svensson, 2006-12-15 17:43 Subject: Re: [RFC] Submodules in GIT Message-ID: URL: https://gitlist.dev/e/e7bda7770612150943j71a7362bmb509cea3b7756003%40mail.gmail.com In-Reply-To: <200612150007.44331.Josef.Weidendorfer@gmx.de> ``` On 12/15/06, Josef Weidendorfer wrote: > If you want to track build results for some source, > why would you ever want these builds go out of sync with the source? I don't, bad wording by me. That was the problem I wanted to address. > Your example has the folling submodule dependence > ("X ==> Y" means Y being a submodule of X): > > App ==> Lib > ^ ^ > | | > AppBuild ==> LibBuild In my example "AppBuild" and "LibBuild" were the same project but this scenario is relevant as well. > If we force submodules to be subdirectories of supermodules, > Lib needlessly will have to appear two times in a checkout of > AppBuild. > > However, there is nothing wrong with it. Yet, you perhaps want > the 2 Lib submodules not to go out of sync. This easily > can be done with symlinking the Lib checkouts. As they are submodules, > everything should work fine. This is interesting. In my notation: /path/to/link/name -> /path/to/subtree means that there is a link named "name" in the tree object for "path/to/link". The link points to a "link object" specifying a subtree or blob of the tree that is pointed to in a submodule commit. This is not currently implemented but has at least the following advantages: 1. You can access files in a submodule without fetching the whole submodule (which may be very large). (App1 is only interested in lib1.h, the rest is toally irrelevant) 2. Superproject can access referenced (linked) files in it's own folder-structure without being forced a structure by the subproject. If you do a symlink instead, doesn't you loose versioning information? What happens with the symlinks if someone clones the superproject? > > Perhaps an option you want to have is to force a checkout > of AppBuild to make these symlinking itself when it detects > identical submodules links. > > Hmmm... the only problem with a symlink is that it can go wrong > when moved. Unfortunately, I do not have a good solution for > this. We can not make UNIX symlinks smart in any way. > Hardlinking directories would be a solution, but that is not > possible. > Wouldn't specifying the submodule path in the link object fit in well here? Then each "link object" can represent a checked out tree from the subproject in the superproject directory-structure. > Another thing: > With normal "$buildroot != $srcroot" environments, the source > can not be a subdirectory of the build directory. This is true for symlinks and would also be corrected if we have a (sparse) submodule checkout there in it's place. > BTW, build project commits probably should not depend on any > history of other build commits. Why? Can you give an example here. > > Link: /headers/lib1.h -> /src/lib1.h > > Link: /bin/lib1.so -> /i386/lib1/lib1.so > > Link: /bin/app1 -> /i386/app1/app1 > > > > > > , and should be the same, > > dictated by the app1 project. > > I do not see any problem here. Symlinks are stored in the git repository. > As the AppBuild commit depends on App and LibBuild submodule commits, the > symlinks always should be correct. The main reason for these "links" are for versioning purposes: the uniqe SHA1 of the "link" representing a tree/blob in a version of the submodule should be "included" in the supermodules commit. Symlinks won't give that at all. > > How do the super-projects in this case get access to the blobs pointed > > by the links - transparent or explicit in the build-process? > > Submodules should automatically be checked out when checking out the > supermodule. So the blobs should already be there. > Or do I miss something? Probably not as that was a piece of the puzzle that I was missing. ``` ## Josef Weidendorfer, 2006-12-15 21:42 Subject: Re: [RFC] Submodules in GIT Message-ID: <200612152242.50472.Josef.Weidendorfer@gmx.de> URL: https://gitlist.dev/e/200612152242.50472.Josef.Weidendorfer%40gmx.de In-Reply-To: ``` On Friday 15 December 2006 18:43, Torgil Svensson wrote: > On 12/15/06, Josef Weidendorfer wrote: > > However, there is nothing wrong with it. Yet, you perhaps want > > the 2 Lib submodules not to go out of sync. This easily > > can be done with symlinking the Lib checkouts. As they are submodules, > > everything should work fine. > > This is interesting. In my notation: > > /path/to/link/name -> /path/to/subtree > > means that there is a link named "name" in the tree object for > "path/to/link". The link points to a "link object" specifying a > subtree or blob of the tree that is pointed to in a submodule commit. Ah, now I understand. I somehow missed this notation. > This is not currently implemented but has at least the following > advantages: > > 1. You can access files in a submodule without fetching the whole > submodule (which may be very large). (App1 is only interested in > lib1.h, the rest is toally irrelevant) > 2. Superproject can access referenced (linked) files in it's own > folder-structure without being forced a structure by the subproject. That all sounds fine, but how do you create such symlinks in practice? Do you want to introduce special porcelain commands to create them? Especially, what is the SCM user supposed to do to change the link target, ie. from /path/to/subtree to /path2/to2/subtree2 ? Should this do a re-checkout at the other point? By linking a file from a submodule, such a link seems to force that this file has to be at a fixed position in the submodule. Otherwise, some magic has to happen when the file is moved in the submodule, possibly leading to a dangling link, eg. if the whole subdirectory specified in the link is removed. IMHO this is getting way to complex. Much simpler is to include the full submodule at some path in the supermodule, and create normal symlinks from the supermodule into the submodule. If you only want to check out part of a submodule, this should be done with path-limiting checkouts, which should be a feature totally independent from submodules. And if you want to limit the number of objects transferred in cloning of a subproject, it is better to further split this subproject into multiple subprojects itself. > If you do a symlink instead, doesn't you loose versioning information? Of course, you need the submodule fully checked out somewhere in the supermodule, and the link goes into the submodule directory. The versioning is given by the supermodule/submodule link. > What happens with the symlinks if someone clones the superproject? As already said: the link has to go into a submodule directory, which will be checked out automatically with the clone of the supermodule. > > Perhaps an option you want to have is to force a checkout > > of AppBuild to make these symlinking itself when it detects > > identical submodules links. > > > > Hmmm... the only problem with a symlink is that it can go wrong > > when moved. Unfortunately, I do not have a good solution for > > this. We can not make UNIX symlinks smart in any way. > > Hardlinking directories would be a solution, but that is not > > possible. > > > > Wouldn't specifying the submodule path in the link object fit in well > here? Then each "link object" can represent a checked out tree from > the subproject in the superproject directory-structure. The problem is not the representation in the git repository, but the checked out module/submodule, where you need to use normal UNIX file semantics. To move submodules around, the user should be able to just use the normal UNIX "mv" commands, and git should be able to detect move actions after the fact. The simple thing here is that currently, git does not have this problem as it tracks content, and does not even try to detect any moves at commit time. This is different with submodules, as there, you want to be able to track moves of any submodule root directories. This now becomes a problem if you use symlinks to "unify" multiple checkouts of the same submodule at multiple places in the supermodule, and move the symlink around, as it easily can get dangling this way. Thus, you would not have a way to see what submodule this link was talking about. And for this thing, I do not see how your link object could help. So it is better to use a simple submodule concept, and for this corner cases, we perhaps could expect the user to fix e.g. a dangling symlink to a previous submodule checkout himself, using a meaningful error message. > > BTW, build project commits probably should not depend on any > > history of other build commits. > > Why? Can you give an example here. If you have a source commit chain A => B => C => D, you want to make any build commits totally independent: you first only are interested in a build commit for source versions A and D, and later find out that a build commit for B and C would be nice, too. If you force build commits into some history order, this order now would be A => D => B => C, which makes no sense. Build commit independence can easily be achieved by making every commit parentless, without further history. You still have the link to the source version via the submodule link in the tree. But to not loose any such build commits, they have to appear as tags or refs (unless integrated in another superproject build commit). > > > Link: /headers/lib1.h -> /src/lib1.h > > > Link: /bin/lib1.so -> /i386/lib1/lib1.so > > > Link: /bin/app1 -> /i386/app1/app1 > > > > > > > > > , and should be the same, > > > dictated by the app1 project. > > > > I do not see any problem here. Symlinks are stored in the git repository. > > As the AppBuild commit depends on App and LibBuild submodule commits, the > > symlinks always should be correct. > > The main reason for these "links" are for versioning purposes: the > uniqe SHA1 of the "link" representing a tree/blob in a version of the > submodule should be "included" in the supermodules commit. Symlinks > won't give that at all. The version coupling will be there if the whole submodule is available at some path in the supermodule checkout, as said above. ``` ## Torgil Svensson, 2006-12-15 23:43 Subject: Re: [RFC] Submodules in GIT Message-ID: URL: https://gitlist.dev/e/e7bda7770612151543o39c9d233q91ea643a134196d3%40mail.gmail.com In-Reply-To: <200612152242.50472.Josef.Weidendorfer@gmx.de> ``` On 12/15/06, Josef Weidendorfer wrote: > That all sounds fine, but how do you create such symlinks in practice? I'm very open to suggestions here, but the concept growing in my head is based around Linus 'module'-file and keep things simple. A git configuration file that specifies: * link name for reference * local path to link * submodule source * submodule path to tree/blob * submodule commit / HEAD / branch * options (depth-limit , ...) I'm reconsidering having the path-name in the link, it should be sufficient to have two SHA1's, one for the commit and one for the tree/blob. Super-module should have the tree/blob in it's database so that the link part only is there for version information and reference (checking dirty state or history on the submodule). This way it easy to clone the super-project and use it without having to map up all sub-project sources. Sub-project sources is not important for version information and could always be specified in the project in a README-type of file. > Especially, what is the SCM user supposed to do to change the link > target, ie. from > /path/to/subtree > to > /path2/to2/subtree2 > ? > Should this do a re-checkout at the other point? That would be a change in the modules file, maybe through a command that also fixes the link. The link will have to be updated in the index and commited as normal. > By linking a file from a submodule, such a link seems to force that > this file has to be at a fixed position in the submodule. Otherwise, > some magic has to happen when the file is moved in the submodule, > possibly leading to a dangling link, eg. if the whole subdirectory > specified in the link is removed. Since we have the SHA1 (this is what we're using) and tree/blob information in the super-modules database the change itself is not a problem. The problem is to track renames/moves and your remove case in the submodule. The tool that tracks the submodule should probably warn/exit here and we would fix up the modules file manually. > IMHO this is getting way to complex. One of complex situation here as I see it is the ability to handle to track/checkout only a subset (tree/blob) of the submodule. This is also quite an important feature - in my example it means the difference of tracking one header file versus the whole source. > If you only want to check out part of a submodule, this should be > done with path-limiting checkouts, which should be a feature totally > independent from submodules. If we can do path-limiting checkouts on a repo (module) we also can do it on a sub-module since they are exactly the same. This is a very powerful feature and it'd be a huge waste if it wasn't allowed for a super-module to do on submodules. > And if you want to limit the number of objects transferred in cloning > of a subproject, it is better to further split this subproject into > multiple subprojects itself. What if we have no control of the submodule? This can be tracked from upstream, sourceforge, another company, etc. The submodule will often live their own life and could be X, kernel, gcc, cairo, whatever, ... > The problem is not the representation in the git repository, but the > checked out module/submodule, where you need to use normal UNIX file semantics. > To move submodules around, the user should be able to just use > the normal UNIX "mv" commands, and git should be able to detect move > actions after the fact. If we disregard the commit info, the link will act exactly as a normal tree/blob. Git can know we're moving a subproject by watching the module file. The main problem is to keep modules file up-to-date with reality. We could enforce module file validity by disallowing such operations and let the user do a "force" operation which also alters the modules file. > This now becomes a problem if you use symlinks to "unify" multiple checkouts > of the same submodule at multiple places in the supermodule, and move > the symlink around, as it easily can get dangling this way. Thus, you would > not have a way to see what submodule this link was talking about. The symlink only exists in the modules file. We only have the SHA1's at the tree-level and there we have everything underneath the tree/blob SHA1 in our database. We will only know if the modules symlink file is dangling next time we fetch from the submodule - here we would notify the user but our database is still consistent. > If you have a source commit chain A => B => C => D, you want > to make any build commits totally independent: you first only > are interested in a build commit for source versions A and D, > and later find out that a build commit for B and C would be nice, > too. If you force build commits into some history order, this > order now would be A => D => B => C, which makes no sense. It makes no sense because the user seem to have act irrationally. The commit-chain is completely valid as it has tracked the correct history of the builds. I can't see any problems here, the build-project is independent of the source-project with it's own history. We can hope the user has given good explanations for his/her actions in the commit messages though. ``` ## Torgil Svensson, 2006-12-16 01:13 Subject: Re: [RFC] Submodules in GIT Message-ID: URL: https://gitlist.dev/e/e7bda7770612151713k418434e6gd8d565e49a766477%40mail.gmail.com In-Reply-To: ``` On 12/16/06, Torgil Svensson wrote: > I'm very open to suggestions here, but the concept growing in my head > is based around Linus 'module'-file and keep things simple. A git > configuration file that specifies: > * link name for reference > * local path to link > * submodule source > * submodule path to tree/blob > * submodule commit / HEAD / branch > * options (depth-limit , ...) > > I'm reconsidering having the path-name in the link, it should be > sufficient to have two SHA1's, one for the commit and one for the > tree/blob. Super-module should have the tree/blob in it's database so > that the link part only is there for version information and reference > (checking dirty state or history on the submodule). This way it easy > to clone the super-project and use it without having to map up all > sub-project sources. Sub-project sources is not important for version > information and could always be specified in the project in a > README-type of file. > See it as the link only is there for the version handling between different modules and it's the module file that give an UI to the the link (which project, branch, ....). Many users will not care whats behind those links, but if they want to edit the link they have to create the modules file or fetch it somewhere - it may even be provided and version controlled in the project itself. example tree object: 100644 blob README 100644 blob REPORTING-BUGS 100644 link 040000 tree arch 040000 tree block 040000 link Note that the links functions exactly as the blobs and trees in the database. The difference is that they origin from _a_subproject_ (we don't care which in this stage) with the specified commit SHA1. If the link isn't represented in the modules file, it's no big deal, it can be added later on if needed. If the blame-game begins or if we want to check what we're using on a submodule level we can always pinpoint the exact file/tree content and ``` ## Torgil Svensson, 2006-12-16 01:20 Subject: Re: [RFC] Submodules in GIT Message-ID: URL: https://gitlist.dev/e/e7bda7770612151720w2e65fe83s9942e1ec1f7092a2%40mail.gmail.com In-Reply-To: ``` On 12/16/06, Torgil Svensson wrote: > > example tree object: > > 100644 blob README > 100644 blob REPORTING-BUGS > 100644 link > 040000 tree arch > 040000 tree block > 040000 link > Sorry, I was sloppy and forgot the names: 100644 blob README 100644 blob REPORTING-BUGS 100644 link AUTHORS 040000 tree arch 040000 tree block 040000 link misc Now it doesn't looks like trees/blobs anymore so maybe a link object is handy: 100644 blob README 100644 blob REPORTING-BUGS 100644 link AUTHORS 040000 tree arch 040000 tree block 040000 link misc link-object: ``` ## Jakub Narebski, 2006-12-16 01:34 Subject: Re: [RFC] Submodules in GIT Message-ID: URL: https://gitlist.dev/e/elviac%2463t%241%40sea.gmane.org In-Reply-To: ``` Torgil Svensson wrote: > On 12/16/06, Torgil Svensson wrote: >> >> example tree object: >> >> 100644 blob README >> 100644 blob REPORTING-BUGS >> 100644 link >> 040000 tree arch >> 040000 tree block >> 040000 link >> > > Sorry, I was sloppy and forgot the names: > > 100644 blob README > 100644 blob REPORTING-BUGS > 100644 link AUTHORS > 040000 tree arch > 040000 tree block > 040000 link misc > > Now it doesn't looks like trees/blobs anymore so maybe a link object > is handy: > > 100644 blob README > 100644 blob REPORTING-BUGS > 100644 link AUTHORS > 040000 tree arch > 040000 tree block > 040000 link misc > > link-object: > > What do you need for in link-object? Wouldn't you use usually the sha1 of top tree of a commit, which is uniquely defined by commit object, so you need only ? -- Jakub Narebski Warsaw, Poland ShadeHawk on #git ``` ## Linus Torvalds, 2006-12-16 01:49 Subject: Re: [RFC] Submodules in GIT Message-ID: URL: https://gitlist.dev/e/Pine.LNX.4.64.0612151747470.3849%40woody.osdl.org In-Reply-To: ``` On Sat, 16 Dec 2006, Torgil Svensson wrote: > > 100644 blob README > 100644 blob REPORTING-BUGS > 100644 link > 040000 tree arch > 040000 tree block > 040000 link That 040000 needs to be something else. In order for something like a git-fsck-objects to know that it's a link, it needs to be marked as such. In git, we never just randomly open an object by SHA1, and then figure out its type. We always open things by explicitly knowing both the type and the SHA1, and if the object we find has the wrong type, that's a consistency error in the database (or the user). ``` ## Linus Torvalds, 2006-12-16 02:12 Subject: Re: [RFC] Submodules in GIT Message-ID: URL: https://gitlist.dev/e/Pine.LNX.4.64.0612151809510.3557%40woody.osdl.org In-Reply-To: ``` On Fri, 15 Dec 2006, Linus Torvalds wrote: > > On Sat, 16 Dec 2006, Torgil Svensson wrote: > > > > 100644 blob README > > 100644 blob REPORTING-BUGS > > 100644 link > > 040000 tree arch > > 040000 tree block > > 040000 link > > That 040000 needs to be something else. Side note: that's not to say that I would really see why you'd want to have both the tree and the commit SHA1's, and why you seemingly think that the links don't need a filename. Hmm? If you require the tree objects to be in the database, you might as well require that the commit object be there. But you could make rules that say that subprojects don't need the whole commit history, for example (which is just a shallow clone in the subproject). ``` ## Torgil Svensson, 2006-12-16 08:40 Subject: Re: [RFC] Submodules in GIT Message-ID: URL: https://gitlist.dev/e/e7bda7770612160040v1a769153p909a8cd40e5ea991%40mail.gmail.com In-Reply-To: ``` On 12/16/06, Jakub Narebski wrote: > > 100644 blob > > >> > > > > Sorry, I was sloppy and forgot the names: > > > > 100644 blob README > > 100644 blob REPORTING-BUGS > > 100644 link AUTHORS > > 040000 tree arch > > 040000 tree block > > 040000 link misc > > > > Now it doesn't looks like trees/blobs anymore so maybe a link object > > is handy: > > README > > 100644 blob REPORTING-BUGS > > 100644 link AUTHORS > > 040000 tree arch > > 040000 tree block > > 040000 link misc > > > > link-object: > > > > > > What do you need for in link-object? Wouldn't you > use usually the sha1 of top tree of a commit, which is uniquely defined > by commit object, so you need only ? > 1. "Sparse" repository's - In my example, I want to cherry-pick header-files or binary-files from different projects without fetching all, potentially huge, submodules in their entirety. Imaging having X, kernel, gcc, gtk and libc6 as sub-projects and you really only care about some header files. 2. Super-module directory-hierarchy independent from submodules. Super-project want to have the header-files and binaries it's own way. This also gives version controlled file-collections, the "release case" in my example - collecting different binaries and header-files from different submodules together in a new directory-structure, add some documentation and configuration files and get the whole thing under strong version-control down to the beginning of time for each little component. 3. Super-module development independent of submodules - If we have the tree/blob-object with all it contents in the database many git-operations can act as the link (commit) wasn't there since we have access to all relevant data to work with. This makes it easy to clone the super-project and work on it seamlessly without having to care about submodules or mapping up submodule repository's (unless you want to modify the links or the data underneath it of course). ``` ## Torgil Svensson, 2006-12-16 08:50 Subject: Re: [RFC] Submodules in GIT Message-ID: URL: https://gitlist.dev/e/e7bda7770612160050j526c2b86gbabeae13f2ff114a%40mail.gmail.com In-Reply-To: ``` On 12/16/06, Linus Torvalds wrote: > Side note: that's not to say that I would really see why you'd want to > have both the tree and the commit SHA1's, and why you seemingly think that > the links don't need a filename. Hmm? I really want that file-name back - we can call it a mind short-circuit. > If you require the tree objects to be in the database, you might as well > require that the commit object be there. But you could make rules that say > that subprojects don't need the whole commit history, for example (which > is just a shallow clone in the subproject). You have a very good point here, this would give us the history of the ``` ## Jakub Narebski, 2006-12-16 09:57 Subject: Re: [RFC] Submodules in GIT Message-ID: URL: https://gitlist.dev/e/em0fpq%2445b%241%40sea.gmane.org In-Reply-To: ``` Torgil Svensson wrote: > On 12/16/06, Jakub Narebski wrote: >>> Now it doesn't looks like trees/blobs anymore so maybe a link object >>> is handy: >>> README >>> 100644 blob REPORTING-BUGS >>> 100644 link AUTHORS >>> 040000 tree arch >>> 040000 tree block >>> 040000 link misc This would be (using the submodule original proposal) 140000 link misc >>> link-object: >>> >>> >> >> What do you need for in link-object? Wouldn't you >> use usually the sha1 of top tree of a commit, which is uniquely defined >> by commit object, so you need only ? >> > > 1. "Sparse" repository's - In my example, I want to cherry-pick > header-files or binary-files from different projects without fetching > all, potentially huge, submodules in their entirety. Imaging having X, > kernel, gcc, gtk and libc6 as sub-projects and you really only care > about some header files. > > 2. Super-module directory-hierarchy independent from submodules. > Super-project want to have the header-files and binaries it's own way. > This also gives version controlled file-collections, the "release > case" in my example - collecting different binaries and header-files > from different submodules together in a new directory-structure, add > some documentation and configuration files and get the whole thing > under strong version-control down to the beginning of time for each > little component. All fine, but this does not and I think cannot protect us from the fact that we can have which doesn't match . I think it would be better to have sparse/partial checkout first. But that is just my idea. Because with which is not sha1 of commit tree you might loose (I think) the ability to merge, for example your changes to submodule with upstream. > 3. Super-module development independent of submodules - If we have the > tree/blob-object with all it contents in the database many > git-operations can act as the link (commit) wasn't there since we have > access to all relevant data to work with. This makes it easy to clone > the super-project and work on it seamlessly without having to care > about submodules or mapping up submodule repository's (unless you want > to modify the links or the data underneath it of course). This is I think irrelevant to the fact if we have only , or link object and also -- Jakub Narebski Warsaw, Poland ShadeHawk on #git ``` ## Junio C Hamano, 2006-12-16 10:25 Subject: Re: [RFC] Submodules in GIT Message-ID: <7vac1oi1ce.fsf@assigned-by-dhcp.cox.net> URL: https://gitlist.dev/e/7vac1oi1ce.fsf%40assigned-by-dhcp.cox.net In-Reply-To: ``` [jc: adding back people on CC list while feeling sick of having to do so...] Jakub Narebski writes: >>>> Now it doesn't looks like trees/blobs anymore so maybe a link object >>>> is handy: >>>> README >>>> 100644 blob REPORTING-BUGS >>>> 100644 link AUTHORS >>>> 040000 tree arch >>>> 040000 tree block >>>> 040000 link misc > > This would be (using the submodule original proposal) > > 140000 link misc If I recall correctly, the original original proposal used 160000 because that is not a bitpattern used in stat.h, and a link behaves like a directory and a symbolic link at the same time. ``` ## Torgil Svensson, 2006-12-16 15:05 Subject: Re: [RFC] Submodules in GIT Message-ID: URL: https://gitlist.dev/e/e7bda7770612160705l61d1f350n70a8ba91754491c9%40mail.gmail.com In-Reply-To: ``` On 12/16/06, Jakub Narebski wrote: > All fine, but this does not and I think cannot protect us from the > fact that we can have which doesn't match > . True, that will be a real problem. Unless we have a bug in git, do you see a scenario in which this is likely to happen? > I think it would be better to have sparse/partial checkout first. > But that is just my idea. Because with which > is not sha1 of commit tree you might loose (I think) the ability > to merge, for example your changes to submodule with upstream. That's correct. I also want a sparse/partial checkout but I don't want the full submodule path. I'm also perfectly fine (for my current use-cases) with not being able to merge upstream unless we're tracking the commit tree (here, we might not want to specify the tree SHA1). I'm not trying to impose a technically fragile solution here [I don't believe it is, but I'm not the most competent to say that either], I'm trying to find solutions for my use cases and I had problems adapting them to the current suggestion. > > 3. Super-module development independent of submodules - If we have the > > tree/blob-object with all it contents in the database many > > git-operations can act as the link (commit) wasn't there since we have > > access to all relevant data to work with. This makes it easy to clone > > the super-project and work on it seamlessly without having to care > > about submodules or mapping up submodule repository's (unless you want > > to modify the links or the data underneath it of course). > > This is I think irrelevant to the fact if we have only , > or link object and also ``` ## Torgil Svensson, 2006-12-16 15:38 Subject: Re: [RFC] Submodules in GIT Message-ID: URL: https://gitlist.dev/e/e7bda7770612160738w68a47790vef922804efd56c76%40mail.gmail.com In-Reply-To: ``` On 12/16/06, Torgil Svensson wrote: > On 12/16/06, Jakub Narebski wrote: > > All fine, but this does not and I think cannot protect us from the > > fact that we can have which doesn't match > > . > > True, that will be a real problem. Unless we have a bug in git, do you > see a scenario in which this is likely to happen? > > I also want a sparse/partial checkout but I don't want > the full submodule path. This might not be as problematic as we think. If we do the same sparse/partial checkout (what's the definition here?) with the as we do with the only case and consider the to be a _local_ (to the super-project) shortcut. Then we only track the submodules using the commit - local conflicts are easier to handle, git would refuse to commit a not present in the commit tree. We might even consider two object types: module: name link: name ``` ## Jakub Narebski, 2006-12-16 16:32 Subject: Re: [RFC] Submodules in GIT Message-ID: <200612161732.11746.jnareb@gmail.com> URL: https://gitlist.dev/e/200612161732.11746.jnareb%40gmail.com In-Reply-To: ``` Torgil Svensson wrote: > On 12/16/06, Jakub Narebski wrote: >> All fine, but this does not and I think cannot protect us from the >> fact that we can have which doesn't match >> . > > True, that will be a real problem. Unless we have a bug in git, do you > see a scenario in which this is likely to happen? Well, I just rather have than the definition of sparse checkout (for example subdirectory name, or file name, or glob pattern). Besides you need the name of directory (for tree) or file (for blob), otherwise you would have no way to update it when submodule advances version, and you want to use new submodule version. And if you have that, you don't need in repository, in link object. You might want it in the index, for performance reasons, though. >> I think it would be better to have sparse/partial checkout first. >> But that is just my idea. Because with which >> is not sha1 of commit tree you might loose (I think) the ability >> to merge, for example your changes to submodule with upstream. > > That's correct. I also want a sparse/partial checkout but I don't want > the full submodule path. I'm also perfectly fine (for my current > use-cases) with not being able to merge upstream unless we're tracking > the commit tree (here, we might not want to specify the tree SHA1). With sparse (for example defined by 'src/*.h') or partial (for example defined by 'Documentation/') checkout you should be able to merge upstream... unless conflicts are in the not checked out part. > I'm not trying to impose a technically fragile solution here [I don't > believe it is, but I'm not the most competent to say that either], I'm > trying to find solutions for my use cases and I had problems adapting > them to the current suggestion. Have you read http://git.or.cz/gitwiki/SubprojectSupport on GitWiki? Have you tested the experimental submodule support (proof of concept) http://git.admingilde.org/tali/git.git/module2 by Martin Waitz? -- Jakub Narebski ``` ## Torgil Svensson, 2006-12-17 00:21 Subject: Re: [RFC] Submodules in GIT Message-ID: URL: https://gitlist.dev/e/e7bda7770612161621p324ba357x883ed28c46597750%40mail.gmail.com In-Reply-To: <200612161732.11746.jnareb@gmail.com> ``` On 12/16/06, Jakub Narebski wrote: > Well, I just rather have than the definition > of sparse checkout (for example subdirectory name, or file name, > or glob pattern). This is entirely an UI issue: On 12/16/06, Torgil Svensson wrote: > is based around Linus 'module'-file and keep things simple. A git > configuration file that specifies: > * link name for reference > * local path to link > * submodule source > * submodule path to tree/blob > * submodule commit / HEAD / branch > * options (depth-limit , ...) On 12/16/06, Jakub Narebski wrote: > And if you have that, you don't need in repository, in link object. Correct. Since the commit contains all the version information, the following combinations should give the same information iff we keep the commit in the database: 1. + 2. + I used the sha1 because I wanted them to behave exactly like trees/blobs in the database for operations that can disregard the commit info. Now, if we keep the commit in the database as Linus suggests we can reach the target from there with a symlink. This would be more readable but also cost a few object lookups extra iterating over the symlink. > With sparse (for example defined by 'src/*.h') or partial (for example > defined by 'Documentation/') checkout you should be able to merge > upstream... unless conflicts are in the not checked out part. This would be a great feature! Will this conflict with path shortcuts? If so, we might consider two types of objects: "link" which cannot merge upstream and "module" which can merge upstream and contains a .git object repository. IMHO, "module" is a more intuitive name for specifying a (functionality wise fully fledged) submodule with a repository inside. "link" could be used for just mirroring a tree/blob. I'm not sure if a separation is needed on a technical level. > Have you read http://git.or.cz/gitwiki/SubprojectSupport on GitWiki? Yes > Have you tested the experimental submodule support (proof of concept) Not yet ```