Re: [RFC] Submodules in GIT

On Tuesday 2006 November 28 09:36, Andreas Ericsson wrote:

I'd agree. However, is there a need to require user intervention? Can we not make the following analogies to normal git operation:

file in working directory -> submodule working directory file in index -> submodule repository

It's perfectly possible to make a commit with different contents in the index and the working directory - it shows up in the git-status output very nicely. Why not deal with submodules in the same way?

Now imagine the following repository:
  file1
  file2
  submodule1/file3
Make changes to file2 and file3, but don't update-index or commit.  git-status 
would show:

# Changed but not updated: # (use git-update-index to mark for commit) # # modified: file2 # dirty: submodule1 # nothing to commit

Now "git-update-index file2" and git-status

# Updated but not checked in: # (will commit) # # modified: file2 # # Changed but not updated: # (use git-update-index to mark for commit) # # dirty: submodule1/ #

Now do a commit in submodule1/ and git-status in the supermodule.

# Updated but not checked in: # (will commit) # # modified: file2 # submodule: submodule1/ #

Obviously the detail would be different, but you get the idea. There is almost no difference between git-with-submodules and git-as-normal.

I suppose there would actually need to be an extra step were the submodule is added to the supermodule index. So really there would be three states from git-status:

# Updated but not checked in: # (will commit) # # modified: file2 # # Changed but not updated: # (use git-update-index to mark for commit) # # modified: file1 # submodule: submodule1/ # # Dirty submodules: # (commit changes in the submodule to clean) # # dirty: submodule1/ #

Which means: since the last supermodule commit there has been
 * a change to file2, which is in the index and would be committed.
 * a change to file1, which is not in the index and won't be committed.
 * a commit to submodule1, which won't be committed
 * changes to the submodule working directory

This really reinforces Linus's interpretation that submodules are directories - they would presumably just get a new object type and be referenced in the tree object. git-update-index would be blind to dirty submodules with no new commit, just as git-update-index on an unchanged file has no effect.

Has this question been answered yet? How does the supermodule know which branch to track in the submodule? Does it simply track HEAD or when the submodule is added to the supermodule is it told which branch to track? I suppose it's got to be HEAD really hasn't it?

Andy

Andy Parkins wrote:

I think that the proper place for that would be supermodule _index_. The supermodule tree would have commit entry, and the index would have symbolic branch (and perhaps some infor about where to find refs for submodule).

This I guess breaks index abstraction slightly, but on the other hand allows for tracking non-HEAD branch of submodule, and for submodule to not know about supermodule at all...

On Tuesday 2006 November 28 10:50, Jakub Narebski wrote:

The reason I thought it would have to be HEAD at all times, is to prevent situations where the supermodule commit doesn't reflect the state of the current tree.

Let's imagine that we're doing non-HEAD tracking in the supermodule.
  supermodule
   +-------- libsubmodule1
   +-------- libsubmodule2
So, you do a "make" in supermodule; this of course will call make in each of 
the submodules.  You test the output and find that it's all working nicely.  
Time for a supermodule commit.  We want to freeze this working state.  You 
commit and tag "supermodule-rc1"

Unfortunately, during development, you've switched libsubmodule1 to branch "development", but supermodule isn't tracking libsubmodule1/HEAD it's tracking libsubmodule1/master. Your supermodule commit doesn't capture a snapshot of the tree you're using.

Now you say to the mailing list "hey guys, can you test "supermodule-rc1"? They check it out, and find that everything is broken. Why? Because what you wanted to check in was libsubmodule@development, but what actually went in was libsubmodule@master.

I think I've talked myself into the position where it definitely has to be HEAD being tracked in the submodules; anything else is a disaster waiting to happen because commit doesn't check in your current tree.

Andy

Andy Parkins <andyparkins@gmail.com> wrote:

Yes, but not only that, HEAD is the only thing that fits with the rest of the git repository/index/working directory model.

Lets review...

What's HEAD? Its the commit which matches the index state as closely as possible, with the only differences being the changes in progress that are being prepared for the next commit (whose parent will be HEAD). If the index and working directory are both clean (no changes) then its also the current content of this directory, right?

What's the index?  Its what you are about to commit.

What's the working directory? Its the current content, which may also be partially checked out or dirty.

So HEAD in a submodule is the current content of that submodule. Therefore any update-index call on a submodule should load HEAD (totally ignoring whatever branch it refers to) into the supermodule index.

On Tuesday 2006 November 28 15:44, Shawn Pearce wrote:

I was with you right up until here.

Why should a submodule do anything to the supermodule? This is like saying, when I edit a working tree file, it should automatically call update-index. The supermodule index should only be updated in response to a manual update-index (or commit -a I suppose).

Worse, if you allow that to happen, the supermodule can commit a state that cannot be retrieved from the submodule's repository. The ONLY thing a supermodule can record about a submodule is a commit. Changing the index doesn't create a commit, so it can't change anything in the supermodule.

If you change the submodule index then that submodule is "dirty", this state has no parallel with normal git operation. The nearest thing is that you've changed a file but not saved it. Apart from showing the "dirty" state in the supermodule's git-status, I don't see that there is anything that the supermodule can do - it can't go around committing in a repository that it not itself.

IMO, it should always be possible to take a submodule and work on it in isolation - in an extreme case, by moving it out of the supermodule tree entirely.

In summary, from the supermodule's point of view:
 * A submodule with changed working directory is "dirty-wd"
 * A submodule with changed index is "dirty-idx" from the supermodule's
 * A submodule with changed HEAD (since the last supermodule commit) 
   is "changed but not updated" and can hence be "update-index"ed into the
   supermodule
 * A submodule with changed HEAD that has been added to the supermodule index
   is "updated but not checked in"
 * A submodule with changed HEAD (since the last supermodule update-index) is
   both "changed but not updated" _and_ "updated but not checked in", just 
   like any normal file.

What's needed then:
 * A way of telling git to treat a particular directory as a submodule instead
   of a directory
 * git-status gets knowledge of how to check for "dirty" submodules
 * git-commit-tree learns about how to store "submodule" object types in
   trees.  The submodule object type will be nothing more than the hash of the
   current HEAD commit.  (This might be my ignorance, perhaps it's just 
   update-index that needs to know this)

I don't know enough about the plumbing to know if my description above is using the right nomenclature - I'm sure someone will correct me.

In my head, it would look something like this:

$ mkdir supermodule; cd supermodule $ git init-db $ git clone proto://host/submodule.git $ git add --submodule submodule $ git update-index submodule $ git commit -m "Added submodule to supermodule" [ edit submodule ] $ git status submodule is dirty, the working directory has changed [ update-index in submodule ] $ git status submodule is dirty, the index has changed [ commit in submodule ] $ git status submodule is changed but not updated $ git update-index submodule $ git status submodule is updated but not checked in $ git commit -m "Record submodule change in supermodule"

Am I crazy?

Andy

Andy Parkins <andyparkins@gmail.com> wrote:

You misread my poorly written statement.  :-)

What I meant to say was that update-index run in the supermodule
would load the submodule content into the supermodule index; much
as an update-index on a file would load the content of that file
into the index.
 

Aside from sharing object directories, yes.
 

Err, uhm, more like git-write-tree. git-commit-tree doesn't care about the tree content. And all of the tree reading code. And all object traversal code (e.g. rev-list --objects). Martin Waitz's submodule prototype has been working on those details. Its non-trivial due to the number of locations affected.

Yes, exactly my thoughts on the matter.
 

Maybe, but I'm not a shrink.  Your email looked sane.  :-)

On Tue, 2006-11-28 at 10:29, Andy Parkins wrote:

This seems to me to be tantamount to saying something like:

    We need a "recursively defined git repository" that
    is representable as a git repository.

That is, can the tree object be changed from containing just "blob" and "tree" references to also having a new "git" reference as well?

jdl

hoi :)

On Tue, Nov 28, 2006 at 04:29:05PM +0000, Andy Parkins wrote:

when tracking refs/heads/master instead of HEAD, you also get:
   * A submodule where HEAD is not pointing to refs/heads/master is
     "dirty-branch" or something.

This is handled by creating a GIT repository in this directory. My current implementation needs some more magic by the user to add it to the index, but I plan to change this to the way that GIT repositories will be recognized as possible submodules.

This is on top of my TODO.

it's only update-index that has to know this. Otherwise it would be implicitly updated and you would never get your "changed but not updated" status as above.

Andy Parkins wrote: ...

So what? You have a submodule commit that only exists in the supermodule. I fail to see the problem. The changes you made to the submodule _in the supermodule_ can later be pulled from wherever you want.

Regards

Stephan

Andy Parkins wrote:

How about if the supermodule commit errors out by default if you commit a different submodule branch than the one you committed the previous time? Require the user to explicitly acknowledge that yes, they want to check in the contents of "development" now, even though the supermodule was tracking "master" before.

Otherwise I think you could easily end up with just the opposite situation, where you forget you've checked out "development" for a moment to look at something, and end up inadvertently committing a bunch of stuff that's not ready for prime time yet. In a standalone git setting, that's no big deal since the commit only updates the current branch and doesn't touch the master branch, but (as I understand the proposal) in a supermodule setting you'd actually end up essentially doing a merge between your development branch and the previously committed master. Or maybe not a merge, but worse, you'd *replace* the previously committed master with what's in your dev branch.

I think wanting to commit a submodule on a different branch than last time is probably not a typical day-to-day use case, so we should make sure the user really wants to do it (but allow it if so.)

On a related note, it would be great from a usability point of view if there were a way to say "I always want to be on the same branch in all submodules and the supermodule." I think a common scenario will be that you are doing development that touches a couple of different applications and your development effort is really a single set of changes even though it happens to cross submodule boundaries. If this branches-in-sync option is turned on, I'd want "git checkout development" to check out the development branch in the entire set of repositories.

More generally, while I 100% agree that it's very useful to be able to operate independently on each submodule, I think it's also going to be common to use submodules to selectively clone different pieces of a larger project. Say your current development effort needs server A, library B, and documentation C, and you want to have *just* those pieces in your environment. You don't particularly care about the details of how the system has assembled the pieces you want; you want to be able to make your changes and push them when you're done. They are really just pieces of a larger code base, not independent entities that happen to be pulled together into a composite workspace temporarily.

For that use case, I don't want the system to act differently depending on whether server A and library B are in the same submodule or separate ones; I want to treat the supermodule as the repository, and the system should take care of the details of managing the submodules. When I do "git commit -a" I want it to give me one editor to write one commit comment that covers all of the changes I've made, and when I do "git checkout -b" I want a new branch to apply across all the files I'm working with.

It is entirely possible that the above is a matter best left to the porcelain layer, and that's fine with me. But I think the Perforce-style "compose a single workspace out of different bits of a larger project" model is hugely useful and whatever submodule system Git ends up with, it should be able to emulate as much of that feature as possible.

Steven Grimm <koreth@midwinter.com> wrote:

Right, you would be replacing the prior branch of that submodule with the new submodule branch.

I think the safety valve you are looking for here is two things:

  * don't automatically update the submodule's HEAD into the
    supermodule's index.

  * make sure the submodule's HEAD is a fast-forward of the
    supermodule's index, with a --force option to force it
	anyway.

Otherwise the developer just has to know what he/she is doing. Today you can put stuff that isn't ready for prime-time into a repository on the wrong branch just by applying the wrong patch, or cherry-picking the wrong commit, etc... the user can (and will) make mistakes. But they can also easily recover from them by rewinding history and redoing it.

That's not really an issue.

A branch doesn't exist just because you checked-out the branch, or because you created it. A branch exists because there were two or more commits (B and C) which use the same parent (A) and two or more of those commits survive, e.g. they have refs which point to them (directly or indirectly) or they were merged into another commit which itself survives.

Therefore if the supermodule is on the "development branch" the submodules are also immediately on the same branch, because their HEADs are derived from whatever is stored in the supermodule's tree. And that tree is derived from whatever "development branch" means.

Really what you want/need is a special head in the submodule which acts as the "branch that corresponds to the supermodule". This probably should just be a naked SHA1 stored in HEAD, which is committable only because a supermodule exists in a higher level directory.

The fact that the submodule project has branches *at all* is totally irrelevant once you start to speak about that submodule within the supermodule, as its the supermodule which determines the branch of the submodule.

That's a mess.

You start to get into weird cases where the directory structure expected by the build process is no longer intact, because the user has sliced it apart in weird ways. And there's no single version which corresponds to that workspace as (if I recall correctly) you can pick different tags or branches at will. I believe that ClearCase has the same bug.

You also can't version that now spliced workspace, aside from taking the configuration file and putting that under version control too.

However I think the proposal on the table will support that to some degree, in that you can take any version of any repository and embed it at any directory of any other repository. This means you can for example embed the Linux kernel, glibc and gcc projects into a larger "embedded device" repository, but you cannot alter the structure of any of those three projects without making your own locally developed branch of them. Which is actually the correct thing to do as any subslicing of a repository is exactly that: a locally developed branch of that repository.

hoi :)

On Tue, Nov 28, 2006 at 01:35:37PM +0000, Andy Parkins wrote:

The way I wanted to address this is to show in the supermodule git-status that the submodule is using another branch. That way you are warned and can decide not to commit the supermodule.

I implemented tracking of refs/heads/master (not HEAD) without much thinking, and only recently began to think about possible problems with this approach.

But I think it is an important design decision to take, so I'd like to have consensus here.

Pro HEAD:
 - update-index on submodule really updates the supermodule index with
   a commit that resembles the working directory.
Contra HEAD:
 - HEAD is not garanteed to be equal to the working directory anyway,
   you may have uncommitted changes.
 - when updating the supermodule, you have to take care that your
   submodules are on the right branch.
   You might for example have some testing-throwawy branch in one
   submodule and don't want to merge it with other changes yet.

Pro refs/heads/master:
 - the supermodule really tracks one defined branch of development.
 - you can easily overwrite one submodule by changing to another branch,
   without fearing that changes in the supermodule change anything
   there.
Contra refs/heads/master:
 - after updating the supermodule, you may not have the correct working
   directory checked out everywhere, because some submodules may be on a
   different branch.
 - there is one branch in the submodule which is special to all the other.

I think that most of the disadvantages of refs/heads/master can be solved by printing the above-mentioned warning in git-status when the submodule is using another branch (similiar to the planned-but-not-implemented warn if the submodule has uncommited changes).

I don't yet know how to cope with tracking HEAD directly, so I'm still in favor of tracking refs/heads/master, as already implemented.

On Wednesday 2006, November 29 16:03, Martin Waitz wrote:

The problem I see with tracking a particular branch is that it makes it less convenient to use git's quick-branching features in the submodules. Let's say I want to try something out quickly in a submodule, I make a branch, commit, commit, "hmm, looks good, let's snapshot it in the supermodule", make a supermodule branch, "oh no, I've got to tell the supermodule to track the new (but temporary) branch in the submodule do a commit, switch the submodule branch back to master, delete the temporary branch, remember that the supermodule is tracking that branch and tell the supermodule to track something else instead... It all seems too complicated to me.

Ouch. Why does the submodule need to update the supermodule index? That should be done by update-index in the supermodule. Further, how is the supermodule index going to represent working directory changes in the submodule? The only link between the two is a commit hash. It has to be like that otherwise you haven't made a supermodule-submodule, you've just made one super-repository. Also, if you don't store submodule commit hashes, then there is no way to guarantee that you're going to be able get back the state of the submodule again.

That's the case for every file in a repository, so isn't really a worry. It's the equivalent of changing a file and not updating the index - who cares? As long as update-index tells you that the submodule is dirty and what to do to clean it, everything is great.

What is the "right" branch though? As I said above, if you're tracking one branch in the submodule then you've effectively locked that submodule to that branch for all supermodule uses. Or you've made yourself a big rod to beat yourself with everytime you want to do some development on an "off" branch on the submodule.

Why is this a pro?

You can always do that anyway by simply not running update-index for the submodule in the supermodule.

This seems like the biggest problem to me - doesn't this negate all the advantages of a submodule system? After a check in, you have no idea if what you checked in was what was in your working tree.

Andy

Andy Parkins wrote:

  >>  - when updating the supermodule, you have to take care that your

Perhaps I'm just daft, but I fail to see how you can safely track a topic-branch that might get rewinded or rebased in the submodule without crippling the supermodule. Wasn't the intention that the supermodule has a new tree object (called "submodule") that points to a commit in the submodule from where it gets its tree and stuff? Is the intention that the supermodule pulls all of the submodules history into its own ODB? If so, what's the difference between just having one large repository. If not, how can you make it not break in case the commit it references in the submodule is pruned away?

One possible way would ofcourse be to add something like this to the supermodule commit: submodule directory/commit-sha1 tree submodule-tree-sha1

but then you're in trouble because the supermodule will have the same files as all the submodules stored in its own tree. I'm confused. Could someone shed some light on how this sub-/super-module connection is supposed to work in the supermodule's commit objects?

On Thursday 2006 November 30 12:16, Andreas Ericsson wrote:

Who said anything but rebase/rewind? As it happens though, I don't see why you can't (it wouldn't be pleasant though). A rebase or rewind still leaves the original commit in the object database, so provided no one runs git-prune, there is no catastrophic failure.

I certainly never suggested anything /but/ storing a submodule type that stores the commit. The current debate is about whether the supermodule should track HEAD or some defined branch in the submodule.

I don't really know, I only joined in to stand up against commit in the supermodule triggering commits in the submodule. That lead to me trying to get an understanding of how it would work.

As far as I can see, the only way a submodule is any use is if it is always a submodule-commit-hash that is noted in the supermodule tree object. That means that the supermodule will only commit clean submodules. The rest is just UI to show something useful in the difficult cases when the submodule tree is dirty.

Andy

hoi :)

On Wed, Nov 29, 2006 at 08:00:22PM +0000, Andy Parkins wrote:

What about: You decide to try something out quickly and create a new branch in the submodule. After you have verified that it works, you merge it to the submodules master branch and commit that to the supermodule. Not that complicated, isn't it? In fact, my current implementation does not even allow to change the branch name of the submodule which is tracked by the supermodule ;-).

Please excuse that I am not an native english speaker and I may have caused some confusion here.

That is exactly what I wanted to say. In the supermoduel you call update-index (with the submodule path as argument) to update the index of the supermodule. Just like normal files. Nothing new.

This is handled in the next paragraph. The argument really is: HEAD always points to the checked out branch, so it really has a relationship to the working directory.

Yes, it's not a real counter-argument, but it relativates the previous pro-argument.

yes, but luckily GIT branches are very flexible.

I don't think it is that bad.

You always know which branch in the submodule is the "upstream" branch which is managed by the supermodule. You can easily have several topic-branches and merge updates from the master branch. otherwise you always have to remember which branch holds your current contents from the supermodule.

When viewed from the supermodule, you are storing one branch per submodule in your tree.

Suppose you are working on a complicated feature in one submodule. You create your own branch for that feature and work on it. Now you want to update your project, so you pull a new supermodule version. Now this pull also included one (for you unimportant) change in the submodule. I think it is more clear to update the master branch with the new version coming from the supermodule, while leaving your work intact (you haven't commited it to the supermodule yet, so the supermodule should not care about your changes, it's just some dirty tree). Then you can freely merge between your branch and master as you like and are not forced to merge at once. And perhaps you even do not want to merge at all, because you are on an experimental branch which really is mutually exclusive with the current supermodule contents.

Of course you know: git-status will tell it. This is no different to today, where you can commit while still leaving a part of the tree dirty.

Martin Waitz wrote:

There's no real technical reason for locking it to a single branch though, and in case of a fork in the upstream submodule project, you might suddenly decide that "the other team" is heading in a much more interesting direction and you want to use their work in your module instead. Will you now have to maintain a separate branch just to keep the same name as the branch the original team used?

It could be, and as has already been stated, there's no real reason to limit this to a particular branch, so I don't see why we would want to impose such non-real restrictions.

No you don't. The branch-name might be moved to some other tip of the DAG, and that's exactly the same as changing the branch you're tracking.

No you don't. The only thing you need is the commit-sha.

Wrong again. You're storing one particular point in the revision history.

git reset to the rescue.

This is all just policy though. Tools that enforce a certain policy are not good tools.

The only problem I'm seeing atm is that the supermodule somehow has to mark whatever commits it's using from the submodule inside the submodule repo so that they effectively become un-prunable, otherwise the supermodule may some day find itself with a history that it can't restore.

The really major problem with this is that now you'll have one repository of the submodule that is actually special, so it's not certain you can go and use any repository at all of the submodule code, since the upstream repo most likely won't be all that interested in having all of that meta-data inside it. In reality, I'm sure this will be a small problem though, as submodules that are in reality projects which the supermodule's maintainer isn't the owner of will most likely never rewind their history beyond the supermodules stored commit. It's something fsck will have to be taught to watch for though.

Andreas Ericsson wrote: ...

That has nothing to do with submodules. What you state here is the problem of alternate repositories.

There are two solutions:

1. Do not use alternates.

2. Do not prune a repository that is used as an alternate repository by 
other repositories.

For the submodule discussion that would mean:

1. Only fetch and work on branches of submodules you are interested in. 
It does not matter that the origin repository contains (probably orders 
of magnitude) more data. You will never touch that.

2. You can never prune the main (the supermodule's) repository, at least 
not with what git provides today.

That is why the sanest approach to subprojects is to put commits into tree objects, define a way to name these commits and make git understand these new commit names. Done. Works.

Regards

Stephan

hoi :)

On Fri, Dec 01, 2006 at 01:03:48PM +0100, sf wrote:

It even already works (well, not with what git provides today, but with my implementation). git-prune simply walks all the submodules, too, when doing it's reachability analysis.

What does not work is a prune inside the submodule, because it does not know about all the commits used by the supermodule.

Martin Waitz wrote:

I just had a short (really short) look at your work. My impression is that your repository setup is much too complicated.

As I proposed elsewhere: For submodules to work you only need to allow commits in tree objects (that is what your implementation requires as well). Everything else is in the tools. Much simpler.

Regards

Stephan

hoi :)

On Fri, Dec 01, 2006 at 02:21:20PM +0100, sf wrote:

Well, I'm not really satisfied with the UI part. What exactly do you find complicated?

I do not quite get your point. The core of my work allows to put commits into tree objects. Then there is some more (but not quite finished) work to make the tools work together with submodules. So no, not everything is there yet.

Martin Waitz wrote:

I am not talking about the UI. On the contrary, I am talking about the repository, i.e. the directories and files on your disk and their contents. You add a lot of additional information to the repository that is not needed at all.

That is fine.

And what is already there is a lot of meta information (see above). You do not need that.

For example, in the index, if it is a commit (i.e. a subproject), store the commit id (not the commit's tree id ). Make the tools handle this case (as yet, all code expects only trees and blobs when they parse trees). Especially, extend update-index to be able to store a commit instead of the tree.

Or else, do not change what is recorded in the index. Then, at commit time, you not only commit the superproject but also all subprojects.

Or allow both.

Anyway, you can create commits in tree objects. See, you did not need to 
  store any additional information in the repository.

To push and pull you have to extend the tools as well. That is the next step.

Regards

Stephan

hoi :)

On Fri, Dec 01, 2006 at 03:23:07PM +0100, Stephan Feder wrote:

What information are you refering to? Perhaps you have looked into my old branch? The current implementation is in "module2".

This is exactly what I have done.

Done, except that update-index never stores trees ;-)

But then submodules would be handled differently to files which I wanted to avoid.

Also done.

Martin Waitz wrote:

I was looking into git-init-module.sh (branch module2). There you set up a separate git repository for the submodule and store references to it into the supermodules's repository.

Yes, I forgot.

On the other hand, it feels more naturally to only commit at the end of your work. So both alternatives have their merits.

I hope I have time to give your solution a try.

Regards

Stephan

hoi :)

On Fri, Dec 01, 2006 at 05:04:19PM +0100, Stephan Feder wrote:

yes.

This is to be able to call git-fsck-objects and git-prune in the toplevel supermodule. When traversing the object tree, it already knows about all submodule, but only about those versions that are really part of their supermodule. So I have to teach git about separate submodule branches which may be used, in order not to prune them away.

Hello,

Andreas Ericsson wrote:

One could circumvent that by creating a separate repo for the submodule at checkout time and pull the needed objects in the supermodule's odb when commiting the supermodule. This way prune in the submodule cannot do any harm, because in it's odb are no objects that are important for the supermodule.

Uwe

Uwe Kleine-Koenig wrote:

Yes, but then you'd lose history connectivity (I'm assuming you'd only pull in the tree and blob objects from the submodule, and prefix the tree-entrys with whatever directory you're storing the submodul in).

Andreas Ericsson wrote:

I thought that Uwe meant pulling (getting) _all_ the needed objects from submodule object repository into supermodule object repository: commits, trees and blobs, full history.

Hella Andreas,

Andreas Ericsson wrote:

That's the reason for me prefering to pull in the complete commit.

I don't understand what you mean with "prefix the tree-entrys with whatever directory you're storing the submodul in". Maybe one of us doesn't understand tree objects correctly. AFAICT they don't store the location where they occur, so there is no need to store a prefix. E.g.

	zeisberg@cepheus:/tmp$ mkdir test-repo
	zeisberg@cepheus:/tmp$ cd test-repo/
	zeisberg@cepheus:/tmp/test-repo$ git-init-db 
	defaulting to local storage area
	zeisberg@cepheus:/tmp/test-repo$ echo LD_FLAGS=-ltest > Makefile
	zeisberg@cepheus:/tmp/test-repo$ git add Makefile
	zeisberg@cepheus:/tmp/test-repo$ git commit -m 'test1'
	Committing initial tree 754eadab39642175748bb02155d2959176bcf014
	zeisberg@cepheus:/tmp/test-repo$ mkdir subdir
	zeisberg@cepheus:/tmp/test-repo$ cp Makefile subdir/
	zeisberg@cepheus:/tmp/test-repo$ git add subdir/
	zeisberg@cepheus:/tmp/test-repo$ git commit -m 'test2'
	zeisberg@cepheus:/tmp/test-repo$ git ls-tree HEAD
	100644 blob 610bafd79f92c7e546b104d5b22795df1f099723    Makefile
	040000 tree 754eadab39642175748bb02155d2959176bcf014    subdir

So the tree that only contains the Makefile specifing LD_FLAGS has the sha1id 754eadab39642175748bb02155d2959176bcf014 independent of being the root of my project or a subtree.

But maybe I misunderstood you?

Best regards Uwe

Uwe Kleine-Koenig wrote:

Nopes. I just didn't think of the fact that subtrees are trees and never store any path-info no matter what. So basically the supermodule can store all trees of all submodules for each commit adding a new submodule revision (which is neat, since "casuals" never have to bother with getting all the submodules if they want to see all the code used in any particular revision), while we invent the new tree object "subm" that points to a commit in the submodule repo. We then teach the tools to recognize when the *real* submodule repo is present and just don't check out trees from the supermodule odb that lead us to directories where submodules reside. Simple and beautiful. Me likes.

*IF* we teach the history viewers about submodules is a different matter though. I'm not sure it would make much sense to have simple text-mode browsers show the submodule history, although I can imagine qgit and gitk wanting to take advantage of their nice side-by-side DAG displaying code to show all the repos in parallell, or link between them in some point-and-click kind of way.

On Tue, Dec 05, 2006 at 10:01:25AM +0100, Uwe Kleine-Koenig wrote:

I _think_ Linus argued against doing this (for scalability reasons), although he didn't actually answer my question when I asked him directly. In his proposal you wouldn't need to do this, because the particular checked-out copy of the submodule that is located in a subdirectory of a superproject would not be allowed to be pruned and it seems that Martin has also implemented it like this.

On Thursday 2006 November 30 17:06, Martin Waitz wrote:

WHAT? I've got to make merges (that I don't necessarily want) in order to commit in the supermodule? This completely negates any useful functioning of branches in the submodule. I want to be able to make a quick development branch in the submodule and NOT merge that code into master and then be able to still commit that in the supermodule.

I think you're imagining the binding between the super and sub is very much tighter than it should be. What if I'm working on a development version of the supermodule, which includes a stable version of the submodule? Vice versa?

That prevents me "trying something out" on a topic branch in the submodule. Here's a scenario using my suggested "supermodule tracks submodule HEAD" method.

 * You're developerA
 * Make a development branch in the supermodule
 * In the submodule, make a whole load of topic branches
 * Make a development branch in the submodule
 * Merge the topic branches into the development branch of the submodule
 * Commit in the supermodule.  This capture
 * Tag that commit "my-tested-arrangement-of-submodule-features"
 * Push that tag to the central repository - tell the world.
 * DeveloperB checks out that tag and tries it.  Great stuff.

Now: here's the secret fact that I didn't tell you that will break 
your "supermodule tracks submodule branch" method.  DeveloperB has decided to 
have this in his remote:
  Pull: refs/heads/master:refs/heads/upstream/master
Oops. The supermodule, which has been told to track the "master" branch in the 
submodule is tracking different things in developerA's repository from 
developerB's repository.  Worse, what if developerB did this:
  Pull: refs/heads/master:refs/heads/development
  Pull: refs/heads/development:refs/heads/master

Branches are completely arbitrary per-repository. You cannot rely on them being consistent between different repositories. If you store the name of a submodule branch in a supermodule - that supermodule is only valid for that one special case of your particular version of the submodule.

Andy

hoi :)

On Fri, Dec 01, 2006 at 09:02:48AM +0000, Andy Parkins wrote:

exactly!

Please think about it.

If you track HEAD, then this means that you track HEAD. In _both_ directions!

So you not only store your submodule HEAD commit in the supermodule when you do commit to the supermodule, it also means that your submodule HEAD will be updated when you update your supermodule. And what happens if you already commited something to HEAD in the mean time? Exactly: a merge is needed.

And you are right: you might not want to do this now, because you branched off, because you _wanted_ to have some development which is _independent_ to the current supermodule work.

So tracking HEAD really makes branching in the submodule hard to work with.

What does the supermodule provide to the submodule? It stores one reference to a commit sha1. Just like a reference inside refs/heads inside the submodule. There really is not much difference between the sha1 stored inside the supermodules tree and one stored inside refs/. So from the submodules point of view, the supermodule is not much more then one special branch. But it is not possible to use the supermodule index directly as one "magic" branch for several reasons. So we need synchronization methods between the index entry for the submodule which is stored in the supermodule and the references in the submodule. These are git-update-index/git-commit and git-checkout, both called explicitly or implicitly in the supermodule. And I really think it makes sense to have a one-to-one relationship between the submodule "branch" stored in the supermodule and the branchname used in the submodule.

I don't see your problem here.

This is still supposed to be a distributed system. DeveloperB does not only check out the whole project including several modules. He is also supposed to _work_ with it.

What if DeveloperB also has several topic branches? When he checks out the new supermodule, only his current HEAD in the submodule will be updated. So he first has to change to some supermodule-tracking branch inside the submodule, then pull the supermodule updates, then eventually merge the new contents of his supermodule-tracking branch into his topic branches. So why not make this "let's update one supermodule-tracking-branch" automatic?

So what? He can do to the repository whatever he wants? He wants to change one submodule to a different branch? He can do so! But please do not expect the system to magically be able to resolve problems. If you _by intent_ changed the submodule to another branch which is incompatible to the one used in the submodule you can't expect that this is magically merged. This is the same as with normal files. Sure you can replace one file with new contents that are different to the one used by someone else. Don't expect this can be merged automatically. So now you have two forks/branches of the project. So what?

Same for a system including submodules: If you change one submodule to a totally different branch, then you effectivley forked/branched the entire project. (Nomenclature is a bit difficult here: what I mean by totally different branch is: the submodule commit tracked by the supermodule is not directly connected to the one tracked by an old version of the supermodule).

So whenever you introduce conflicting changes somewhere in the project (be it in a submodule or in a file) you _always_ fork/branch the entire project (i.e. the topmost supermodule). You can't circumvent that.

So what are submodule branches good for then? To store other lines of development which are not yet / not any more tracked by the supermodule. Perhaps you store references to branches stored in another supermodule, or another standalone repository. Or a temporary branch which is only used for testing. There are really many possiblilities. But they all have one thing in common: they are not meant to be tracked by the supermodule.

Yes, but a submodule is special here: it really has one special branch. The module is not independent any more. That is the _nature_ of a submodule.

Sure, we are in a distributed system. But the supermodule always has to know which branch in the submodule has to be tracked. The easiest thing is to always use the default refs/heads/master. Surely this could be changed if there is a need.

Martin Waitz wrote: ...

Why the magic? The typical workflow in git is

1. You work on a branch, i.e. edit and commit and so on.
2. At some point, you decide to share the work you did on that branch 
(e-mail a patch, merge into another branch, push upstream or let it by 
pulled by upstream)

I fail to understand why these two steps have to be mixed up. Someone care to explain?

Regards

Stephan

hoi :)

On Fri, Dec 01, 2006 at 01:09:49PM +0100, sf wrote:

3. Other people want to use your new work.

Martin Waitz wrote:

Sorry, if that was not obvious: You actually procceed with one of the options I listed in Step 2. What I wanted to state is that with git you do not mix up committing (which is local to your repository and your branch) and publishing.

Regards

Stephan

hoi :)

On Fri, Dec 01, 2006 at 02:05:33PM +0100, sf wrote:

I guess you are refering to not mix up committing to the submodule and updating the supermodule index. These are really two separate steps, I just combined them above because I wanted to put emphasis on the other part: it is not a one-way flow, it is bidirectional, so your HEAD would have to changed if the supermodule gets updated. And I consider changing HEAD, without looking at the branch it points to, to be a bad thing.

Martin Waitz wrote:

So a commit in the supermodule turns into a commit in the submodule? That's just plain wrong. If it doesn't, why would the submodule HEAD have to change?

hoi :)

On Fri, Dec 01, 2006 at 02:43:16PM +0100, Andreas Ericsson wrote:

no.

So how do you update your submodule?

Remember: if you git-pull in the supermodule, you want to update the
whole thing, including all submodules.

Martin Waitz wrote:

By committing to it separately, or by getting changes from the upstream project (openssl, libcurl, ...).

Only if the new commits I pull into the supermodule DAG has commits which includes a new shapshot of the submodule, otherwise it wouldn't be necessary.

hoi :)

On Fri, Dec 01, 2006 at 03:52:29PM +0100, Andreas Ericsson wrote:

Of course.

But if the supermodule contains changes to the submodule, you still have to change the submodule. And this implies changing the submodule HEAD or some branch.

Martin Waitz wrote:

Not really. I fail to see why HEAD needs to be changed so long as the commit is in the submodule's odb.

On Fri, 1 Dec 2006, Andreas Ericsson wrote:

Right. A commit in the supermodule should _not_ imply a commit in the submodule.

Maybe I should take a look at the code, but it sounds like people are still trying to "mix" submodules too much.

Think of it this way: one common use for submodules is really to just (occasionally) track somebody elses code. The submodule should be a totally pristine copy from somebody else (ie it might be the "intel driver for X.org" submodule, maintained within intel), and the supermodule just refers to it indirectly (ie the supermodule might be the "Fedora Core X group" which contains all the different drivers from different people).

So anything that mixes super-modules and sub-modules too much will always break this kind of model.

A supermodule can never "contain changes" to a submodule. A supermodule would always just point to the submodule, and not have any changes what-so-ever of its own. The submodule is self-sufficient, and always contains all its _own_ changes.

Linus Torvalds wrote:
...
 > Think of it this way: one common use for submodules is really to just
 > (occasionally) track somebody elses code. The submodule should be a
 > totally pristine copy from somebody else (ie it might be the "intel 
driver
 > for X.org" submodule, maintained within intel), and the supermodule just
 > refers to it indirectly (ie the supermodule might be the "Fedora Core X
 > group" which contains all the different drivers from different people).

Could you please be a little bit more specific about how you would store the "pristine copy". There seems to be some agreement to store the commit id of the submodule instead of a plain tree id in the supermodules tree object, and that all objects that are reachable from this commit are made part of the supermodule repository (either fetched or via alternates). Do you agree?

...
 > A supermodule can never "contain changes" to a submodule. A supermodule
 > would always just point to the submodule, and not have any changes
 > what-so-ever of its own. The submodule is self-sufficient, and always
 > contains all its _own_ changes.

That is one of the points Martin Waitz and I are discussing.

If I understand you correctly you cannot make any changes to the submodules code _in the supermodule's repository_, no bugfixes, no extensions, no adaptions, nothing. Do you mean that?

That would be a third alternative. In my opinion the usefulness of submodules would be unnecessarily restricted if it comes to the choice of either using the code from upstream as is or do not use submodules at all. What is the point of the restriction?

Regards

sf wrote:

That depends on your definition of submodule. In my eyes, a submodule is a separate repo that can be committed to separately (and generally also built separately), although it's usually built into something else. I'm imagining most submodules will contain only library code and its testing routines.

Insofar as I've envisioned submodules, it's a separate git repo where you simply record a certain snapshot of the sub-repo with a commit in the super-module, like so:

$ git commit ssl-functions/*.[ch] openssl -m "Upgraded openssl with necessary changes to core code"

(yes, I know it's horrid to use -m to commit, and I daily advocate against it where I work, but you get the idea, I'm sure)

Isn't this how it's supposed to work? Enlighten me, and please remember that I'm drunk atm, so make it obvious ;-)

On Fri, 1 Dec 2006, sf wrote:

Note that it's not necessarily "pristine", since the submodule clearly is a local git repository in its own right. So like _any_ git repository, you can (and may well end up) having your own local branches in the submodule, with your own local modifications.

So I'm not claiming that a submodule must always match some external git tree 100%, and that it must be read-only or anything like that. I'm just saying that I suspect that quite often, one of the MOST IMPORTANT parts is that the submodule is really something that somebody else technically maintains, and that this is actually one of the _reasons_ why it is a submodule in the first place.

For example, a lot of projects end up having some kind of "library component" as a submodule. Take something like a video player project, which would have something like ffmpeg as a submodule, not because you'd maintain ffmpeg yourself, but simply because (let's say) the library interface changes enough, or you need a specific version with some of your own fixes that haven't been released widely yet, so you want to carry all the libraries you need _with_ you, even though you don't really maintain that submodule. You at most have some small extensions of your own.

Now, in this situation, it's relaly really _important_ that the submodule really is totally independent of the supermodule, for several reasons.

For example, since you don't "really" own that project, carrying around your own fixes is really really painful. We know it happens all the time, and a lot of projects end up needing their own version, but the _last_ thing you want is to be in merge hell all the time. So as a supermodule maintainer, the best possible thing for you is to be able to push back those local changes to the original project maintainer, so that you _don't_ have to maintain your own changes.

But you need to realize that the real maintainer of the submodule is TOTALLY UNINTERESTED in your supermodule. He's not going to maintain it, and in fact, if you have anything in the submodule that ends up talking about your supermodule, that's just going to make it a lot less likely that the upstream maintainer will ever pull your changes. He might take a diff from you, but in a perfect world, you'd actually be able to tell him:

 "Hey, I've got a git repository with a few fixes to your ffmpeg git tree, 
  please pull from git://myhost.com/submodule.git to get these fixes:

	... explanation of fixes and commits that are relevant to
	ffmpeg, and have nothing to do with the supermodule, except
	that you need those bug-fixes because you _use_ ffmpeg ...

  Thanks"

See?

So this is why it's really important that the submodule really is a git repository in its own right, and why committing stuff in the supermodule NEVER affect the submodule itself directly (it might _cause_ you to also do a commit in the submodule indirectly, but the submodule commit MUST be totally independent, and stand on its own).

Now, you don't _have_ to push things upstream, of course. You can always just maintain your own submodule branch, and every once in a while, inside the submodule, you do

	# fetch the development in the origin/master branch
	git fetch submodule-origin origin/master

	# rebase our own special magic sauce on top of that
	git rebase origin/master

to update your submodule, and _then_ you do a commit in the supermodule (after testing that the update is all ok, of course) which will update the "commit" pointer in the supermodule.

Notice? In this example, we really maintained the submodule AS a submodule. It was independent, but tied into the supermodule, so that when we clone the supermodule, or do things like bisection on a supermodule, we always end up cloning the submodule too (and in the case of bisection, we really only bisect the supermodule, but the submodule always gets "tracked" in the sense that we would always check out the state of the submodule that was appropriate for that particular commit in the supermodule).

Well, I would actually argue that you may often want to have a supermodule and then at least have the _option_ to decide to not fetch all the submodules.

For an example of this kind of usage, let me tell you how we operated at Transmeta a few years ago, which I'm not saying is the _only_ way to operate, but it's ONE way to do it, and I'll also explain _why_ we did it, and why we had submodules.

In the case of transmeta, we had our own tools, our own programs, and we "owned" all of those. We _also_ used a lot of external tools, like gcc etc. However, different people worked on different parts, and if you worked on the actual x86 JIT part, you probably didn't want to have all of the gcc stuff in your tree _too_. That just took a lot of space, and you really didn't want to compile the whole toolchain (which took hours), since there were precompiled binaries readily available.

Still, from a _release_ standpoint, when we released a new binary, that binary very much depended not just on the actual JIT sources, but on the whole toolchain. So if you wanted to be able to re-create a release, you really needed _everything_. You couldn't just take the "current version" of the toolchain, you needed to have the toolchain that was used AT THE TIME OF THE RELEASE.

And this is a _classic_ example of when you'd want to use submodules. Notice how everybody wanted _some_ of the submodules, but really only the release people wanted them _all_. The higher up the chain you were, the less likely you were to really want to muck around with the compiler and the linker, for example.

And nobody really owned all modules. 

So what you really want is:

 - a supermodule maintainer that is not really the maintainer of _any_ of 
   the submodules, but that does the main "build world" infrastructure 
   (and generally would tend to also maintain the source control 
   infrastructure itself)

 - submodules that had their own maintainers, and where the maintainers 
   may or may not have wanted the supermodule, but even when they wanted 
   the supermodule, they might not want _all_ of the submodules, simply 
   because they just didn't care.

 - some of the submodules then have _upstream_ sources that were totally 
   independent, and that you would want to track, but you had zero power 
   AT ALL over them, and yet you migt well want to push back at least some 
   of the fixes you did - at least the ones that made sense even outside 
   your own project - just to avoid having to maintain a _huge_ set of 
   internal patches.

So no, I don't think the supermodule should even _force_ people to always get all the submodules. It migth be the default case, but at the same time, it's just being polite to let users decide on their own whether they really want _all_ of the build infrastructure sources.

Yes. I think you should make all changes _within_ the submodule, because the submodule should still be an independent git tree in its own right.

But obviously, you'd often use a private _branch_ in the submodule beause you end up having whatever private extensions. That's always true: we always have the "master" branch that is kind of the default "private branch" for any repository, but obviously that is often extended upon, and you may have several private branches.

For example, after you've done a big update (from some external upstream source) in the submodule that you are using, you migth decide that you do all the work on that new big update in a _new_ private branch within the submodule - and get the submodule changes all squared away on its own _before_ you then decide to commit the end result (the tip of that new private branch) within the supermodule.

Ie, you very much should be able to to do

	git clone supermodule/that/one/submodule my-own-version-of-submodule

to clone a submodule _without_ getting anything else (but still get all the work you did within he submodule - very much including your own private branch work).

And the importance of keeping the submodule independent is partly just stability and sanity, but partly also scalability. For example, the "index" in a supermodule should NOT include the indexes of all the submodules. That's really important, because the index doesn't really scale. Things do slow down with large indexes.

For example, git can handle tens of thousands of files easily. I suspect it scales well to hundreds of thousands of filenames. But with supermodules, you really can end up in the situation where you have _tens_ of these submodules, maybe even hundreds. And if you try to maintain one unified index for the _whole_ thing, I guarantee you that you'll start feeling the pain. Indexing millions of files is just not going to be pretty.

So just from a git stability and scalability point, it's important to keep subprojects _separate_. There is obviously integration stuff, but they should still be seen as truly independent projects. Even the supermodule should have clearly its own life even _regardless_ of submodules, because (as I said) quite often you may want the supermodule, but you don't want to have _all_ of the submodules.

But it's more than that stability and scalability thing too - keeping them separate is what allows you to do pulls and pushes on an individual subproject basis, and have people really work at that level. For example, if you're the compiler guy at a company, you really do want to work with other compiler people _outside_ the company, but you sure as hell may not be able to give them access to your supermodule. But you may want to work on _just_ the compiler parts (or at least share some branches in public), which means that the subproject really has to be able to work _independently_ of the supermodule.

So "independent" here is really key, for several reasons. And that all means, for example, that here must NEVER be any "backpointers". A subproject really can _never_ have backpointers to the superproject, because that fundamentally means that the above kind of "compiler guy works on the compiler subproject in public" cannot work, if your supermodule isn't public.

hoi :)

Linus, you are a lot better in describing all my thoughts than I myself. ;-)

On Friday 01 December 2006 20:30, Martin Waitz wrote:

Yes

Some of the clearest explanations described from a strategic point of view come from posts such as these.

And he keeps saying he can't write documentation.

On Friday 01 December 2006 21:13, Linus Torvalds wrote: 

If you want to allow this, you have to be able to cut off fetching the objects of the supermodule at borders to given submodules, the ones you do not want to track. With "border" I mean the submodule commit in some tree of the supermodule.

This looks a little bit like a shallow clone, where you introduce graft points at the border to some of the submodule's object DAGs. But I am not sure that this is scalable: for supermodules with a large number of submodules you are not interested in, your graft file would grow very fast, as there will be new borders with every change in some submodule, which happens to be tracked in the supermodule.

So IMHO, instead of a huge graft file, you want to have a fast way to check at a submodule border which submodule this given border is going into. Then, at fetch time, you easily can decide that you do not want to fetch any object from the submodule. Otherwise, you would have to ask the remote end at cloning time: "Is this commit from some submodule I am locally not interested in?"

So I think we should introduce a submodule namespaces in supermodules. And at every border from super- to submodules, the name of the submodule we are going into should be specified. Which actually means that we need to introduce a "submodule" object, and trees of a supermodule can have such submodule objects as borders into a submodule. In a submodule object, of course we have the SHA1 of the commit into the submodule DAG, and there would be the global unique name we have choosen for this submodule in this supermodule. Something like

 submodule: gcc
 commit: 6287376...

Before cloning a supermodule, you should be able to list the names of the submodules available, and select the submodules you want to have cloned together with the supermodule.

So in the example, "that/one/submodule" is _not_ the path of the working tree which happens to be the root of the submodule at current supermodule HEAD, but the unique name from the submodule namespace.

This is important, as you should be able to move the root of a submodule inside of your supermodule like moving any other file or directory. I.e. for every supermodule commit, the path to the root directory of a given submodule can change, making it useless as a name for a submodule selection at clone time.

hoi :)

On Fri, Dec 01, 2006 at 11:06:40PM +0100, Josef Weidendorfer wrote:

I don't think this is something special to submodules. There has been interest in checking out only a part of the tree even before talking about submodules and I really think this feature should be independent to submodules.

On Friday 01 December 2006 23:12, Martin Waitz wrote:

It's not about checking out part of the tree, it's about fetching only part of the objects: If you have a slow modem and want to clone a supermodule, you are not interested in fetching all the objects from some submodules.

So it is more like a shallow clone. But even here, submodules are special as you have defined borders between supermodule and submodules. This gives you the freedom the introduce a submodule namespace, and allows you to point to a submodule: "I do not want you!".

With shallow clone, there you do not have this option, so there, you need to use something like grafting.

BTW: In your submodule implementation, is the user allowed to change the
relative path of the root of some submodule, e.g. with "git-mv" ?

Josef

hoi :)

On Fri, Dec 01, 2006 at 11:26:22PM +0100, Josef Weidendorfer wrote:

So when you want to suppress one submodule, how is this not about only checking out part of the tree? Ok, you also want to avoid downloading the submodule, but you first have to solve the partial checkout.

In principle: yes. However there are some links between both repositories that have to be updated manually (for the shared object repository and for ignoring submodule files in the supermodule). But I expect that much of this configuration stuff will vanish when submodules are better integrated in git.

Rename detection for submodules would be another interesting thing to have. It should be much easier as for files because we can simply check for common ancestors and do not have to guess based on the diff.

On Friday 01 December 2006 23:12, Martin Waitz wrote:

After some thinking, a submodule namespace even is important for checking out only parts of a supermodule, exactly because the root of a submodule potentially can change at every commit.

When checking out some arbitrary supermodule commit, how do you check that at some submodule border, the user did not want to check out the submodule at all? You need a way to check the DAG identity you are diving into at this border: lets say by going to the root commit of this DAG (!). And via this identity, you have to check whether the user had specified that he wants the submodule to be check out. Without any further meta information (indexed by a submodule name!), this information is only available from the checkout the user switched from, as there would be no file in the working tree from this submodule?

Quite a pain.

hoi :)

On Sat, Dec 02, 2006 at 12:17:44AM +0100, Josef Weidendorfer wrote:

have you ever thought about the idea that the location may be an important thing to consider for your decision.

Perhaps the submodule is now used for something else (this is why it was moved) and that now you'd like to keep it?

Anyway, you can just create several supermodules or implement generic partial tree support for git. I do not see any reason to special case submodules here.

On Saturday 02 December 2006 21:24, Martin Waitz wrote:

Which decision, for what? Sorry, I do not understand.

Do you want to say that relative submodule root paths should be kept fix the whole lifetime of a supermodule? Ie. a submodule "identity" is bound to its relative path, and when we move it, it should be seen as deleting at and creating a totally new, different submodule?

That's fine. But you have to handle submodule creation/deletion neverless. And while you are at a commit which has a given submodule deleted, you have to keep the submodule data somewhere - referencing it with a name. I do not speak here about the object database, that could be combined; but about all the other files in .git/ of the currently not checked out submodule.

Can you give a usage szenario? What do you mean here?

What should such a general partial tree support look like? I suppose you want to configure paths which should not be checked out. As long as you say that a given submodule always has to exist at a given path, you are right: then, you can say: "Please, do not check out this submodule" which is the same as "Do not check out this path".

But I think it is quite restrictive to not allow to move submodules around. When the supermodule upstream decides to move a submodule, your partial tree config to not check out a submodule will be lost. But more important, if you made changes to a given submodule, and pull from upstream which changed the submodule position in-between, your changes will be not taken over to the new position, as the move is seen as creation of a totally independent submodule.

hoi :)

On Sun, Dec 03, 2006 at 01:55:08AM +0100, Josef Weidendorfer wrote:

to check out, or not to check out.

You could say something like "do not check out anything below "test/". If then some submodule moves from "test/foo" to "build/foo", it will be checked out, because this module is now not only used for testing, but is needed for building in the new version of the supermodule.

On Fri, 1 Dec 2006, Josef Weidendorfer wrote:

No. 

I would say that it looks more like a "partial checkout" than a shallow clone.

A shallow clone limits the data in "time" - we have _some_ data, but we don't have all of the history of that data.

In contrast, a submodule that we don't fetch is an all-or-nothing situation: we simply don't have the data at all, and it's really a matter of simply not recursing into that submodule at all - much more like not checking out a particular part of the tree.

So if a shallow clone is a "limit in time", a lack of a module (or a lack of a checkout for a subtree in general - you could certainly imagine doing the same thing even _within_ a git repository, and indeed, we did discuss exactly that at one point in time) is more of a "limit in space".

Linus Torvalds wrote: ...

If you do not want to fetch all of the supermodule then do not fetch the supermodule. Instead fetch only the submodules you are interested in. You do not have to fetch the whole repository.

Regards

On Friday 01 December 2006 23:41, sf wrote:

But what, when I *want* to fetch the supermodule because of the source tree which is only available in the supermodule?

Of course, you can argue that the only objects in trees of a supermodule should be submodule commits, but this is quite restricting the usage of supermodules.

See further arguments for a submodule namespace in my other mail. You probably want to specify policies about submodule handling. This information has to be indexed by some name independent from a supermodule commit.

Josef

On Fri, 1 Dec 2006, sf wrote:

So why do you want to limit it? There's absolutely no cost to saying "I want to see all the common shared infrastructure, but I'm actually only interested in this one submodule that I work with".

Also, anybody who works on just the build infrastructure simply may not care about all the submodules. The submodules may add up to hundreds of gigs of stuff. Not everybody wants them. But you may still want to get the common build infrastructure.

In other words, your "all or nothing" approach is
 (a) not friendly
and
 (b) has no real advantages anyway, since modules have to be independent 
     enough that you _can_ split them off for other reasons anyway.

So forcing that "you have to take everything" mentality onyl has negatives, and no positives. Why do it?

On Saturday 02 December 2006 00:09, Linus Torvalds wrote:

So you are for a global submodule namespace in supermodule repositories, do I understand correctly?

Otherwise, how would you specify the submodules at clone time given the ability that submodule roots can have relative path changed arbitrarily between commits?

On Sat, 2 Dec 2006, Josef Weidendorfer wrote:

The only _true_ namespace would be the SHA1 of the commit (and maybe allow a pointer to a tag too, but the namespace ends up being the same).

How to _find_ a repository that contains that SHA1 must be left to higher levels. After all, repositories move around, and the place you found them originally is not a stable name.

So within the supermodule, on a "git object" level, a submodule should just be named by the SHA1 that was it's HEAD when it was committed within the supermodule. So in the "tree object", you'd see something like the following when you go "git ls-tree HEAD" on the superproject:

	...
	100644 blob 08602f522183dc43787616f37cba9b8af4e3dade	xdiff-interface.c
	100644 blob 1346908bea31319aabeabdfd955e2ea9aab37456	xdiff-interface.h
	040000 tree 959dd5d97e665998eb26c764d3a889ae7903d9c2	xdiff
	050000 link 0215ffb08ce99e2bb59eca114a99499a4d06e704	xyzzy

where that 050000 is the new magic type (I picked one out of my *ss: it's not a valid type for a file mode, so it's a godo choice, but it could be anythign that cannot conflict with a real file), which just specifies the "link" part. The SHA1 is the SHA1 of the commit, and the "xyzzy" is obviously just the name within the directory of the submodule.

That's all that is actually required for a lot of git commands that already expect all objects to be available (ie "git checkout", "git diff" etc).

It only gets interesting for commands that fetch new objects, ie do a "pull/fetch" op, and you'd need to know where/how to fetch new objects for the xyzzy subproject, so that's a "naming" issue. You have a few choices:

 - get all the objects directly from the subproject as if it was one big 
   project.

   I actually think this sucks. Why? Because it puts an insane load on the 
   server side, which basically needs to traverse the object list of the 
   _sum_ of all projects. An initial clone (or a really big pull, which 
   comes to the same thing) would be absolutely horrendous

So I'd strongly argue against that approach, for scalability reasons. So instead, you should really try to do pulls etc one git repo at a time:

 - take the "list of subprojects" from the supermodule, and pull them all 
   one by one.

   This again makes subprojects "less seamless", and makes each subproject 
   more of a separate thing, with the project list gotten from the 
   superproject and parsed separately. But it means you have none of the 
   scalability problems, since you never see things as one huge project 
   with millions of files and even more objects.

The second approach also means that you can see the "supermodule" support in git as less of a "plumbing" thing, and it's largely just a thin veneer around the core plumbing that really doesn't understand about multiple repositories at all (apart from the single "link" extension in the tree object), and it's really just scripting to get the subprojects to "look" like one thing, when they really are pretty much independent.

On Saturday 2006, December 02 00:12, Linus Torvalds wrote:

Can I argue that the hash in that object should actually be to a real object in the supermodule repository rather than a link? Then THAT object would contain the hash? So in your above example:

  100644 blob 08602f522183dc43787616f37cba9b8af4e3dade	xdiff-interface.c
  100644 blob 1346908bea31319aabeabdfd955e2ea9aab37456	xdiff-interface.h
  040000 tree 959dd5d97e665998eb26c764d3a889ae7903d9c2	xdiff
  050000 link a7f26495b7b7e32bf949efbd91ee32267b792cba	xyzzy

And then the local object a7f26495b7b7e32bf949efbd91ee32267b792cba would contain your original hash 0215ffb08ce99e2bb59eca114a99499a4d06e704.

The reason I suggest this as without out it the "link" object is the only hash in the tree that doesn't point to a valid object. The contents of objects is entirely arbitrary so it's perfectly okay for that to contain a hash that won't dereference to a real object in the supermodule.

The main advantage of this is (I think) that git-prune, git-fsck, and whatever else relies on tree objects all being real, don't need to be modified at all.

It also gives you scope to later add fields to the "link" object if you wanted.

Andy

On Saturday 02 December 2006 01:12, Linus Torvalds wrote:

I am not so sure about this. Perhaps we want the namespace to be more than the space of commit ids.

Suppose you have some superproject which uses two compiler major versions (GCC 3 and GCC 4) as submodules because you want to have your regression test suite run with both major versions. So you would have a submodule at path "gcc3/" and "gcc4/" in your supermodule. As both the gcc 3 and gcc 4 are branches from the same project, the submodule links will go into a connected DAG (suppose GCC uses git).

Alone from the commit link, it is not easy to see to what submodule it belongs to (at least from a practical point of view).

So it actually _is_ more information if the proposed link objects in the supermodule contain some submodule ID they belong to. They only need be unique in the scope of the superproject (not really globally unique).

So another argument for submodule names: Merging. Otherwise, how do you decide to which submodule a link belongs to, especially in the scope of above example?

I did not talk about a special format for these submodule IDs yet. We could use an URL, but with such a value a user automatically associates some semantic which can be confusing, as repository URLs can change.

We can use some symbolic name which has some meaning in the scope of the superproject, and is specified at submodule creation, like "gcc3" or "gcc4". However, this is a local decision of the person which is importing the submodule. So if two developers of the same project using supermodules independently decide that the import of "gcc" as submodule is the right thing, but use slightly different submodule IDs, you will get 2 different submodules when merging.

I argue that this is even the correct thing, and they should decide about the name before both are doing the import, or only one imports and the other pulls.

Another option for a submodule ID could be the root commit of the submodule commit DAG. This looks nice as such an ID really is globally unique for projects (more or less: the first commit always contains the time stamp at creation, and the author/commiter email address, even if the tree happens to be the same because you start with the same dummy file).

But my example above (with the 2 different submodules from the same GCC project) shows that this is not working. A superproject never could create different submodules from the same (e.g. GCC) project.

So I just vote for a symbolic name choosen at submodule creation time.

On Sat, 2 Dec 2006, Josef Weidendorfer wrote:

I don't think it would be wrong at all to have a "link object" type, and have the "link" tree entry actually point to that "link object" instead of pointing directly to the commit in the submodule.

And yes, that extra indirection would allow for more flexibility (the "link object" can contain comments about the particular version used, pointers to where you can get it - whether human-readable or strictly meant for automation - etc etc).

So I agree with Andy Parkins' comment about the link object allowing not only extended namespaces, but also allowing a certain amount of flexibility (ie there's some built-in extensibility and ability to perhaps add future fields if there's a new object type).

I just want the naming of the links themselves to use all the same SHA1 hashes etc, so that you always have a very explicit, and very trustworthy version - and never end up in the situation that you know which repository you want at that position, but you don't know exactly which commit in that repo was supposed to be checked out with that particular version of the super-module.

hoi :)

On Sat, Dec 02, 2006 at 11:52:13AM -0800, Linus Torvalds wrote:

What makes a submodule so special that now we suddenly have to store those stuff in the object database?

Storing a fetch location would grossly contradict the distributed nature of git. I really do not see _any_ reason to store more information than the commit sha1 of the submodule.

On Sat, 2 Dec 2006, Martin Waitz wrote:

I'm not sure it is. I suspect a pure commit link with just a CVS-style "modules" file is sufficient. I'm just saying that I don't think it is _wrong_ to possibly want to expand it.

hoi :)

On Sat, Dec 02, 2006 at 12:46:01PM -0800, Linus Torvalds wrote:

If we later see that we really want to have it we can always introduce it later. I don't think we should do it now if we don't see clear benefits _now_.

So I was not against the link object itself (initially I wanted to do it this way, too), only agains the information which was proposed to be stored there. Up to now I haven't found anything which makes sense to store next to the submodule commit to define the identity of the submodule.

On Saturday 02 December 2006 21:58, Martin Waitz wrote:

Isn't it enough reason that a porcelain probably wants to store meta information for a given submodule, giving the need to put a name/identity to it?

Josef

hoi :)

On Fri, Dec 01, 2006 at 04:12:10PM -0800, Linus Torvalds wrote:

I don't buy your scalability argument. By dividing the object traversal in separate steps you do not win anything. The complexity of the operation still stays the same, as you still have to traverse the exact same amount of objects.

By separating the repositories you just make reachability analyis be totally awkward, without winning anything.

On Sat, 2 Dec 2006, Martin Waitz wrote:

Try it.

Really. Get the mozilla import (450MB project), and clone it on a machine with half a gig of RAM or less.

Then, clone a couple of smaller archives that end up being 450MB _combined_, but clone them separately.

And watch the memory usage.

Trust me. Try it out.

hoi :)

On Sat, Dec 02, 2006 at 12:44:20PM -0800, Linus Torvalds wrote:

Do I understand you correctly that the problem is not the algorithmic complexity but that you have to map the objects at once instead of map them in small parts one after the other?

On Sat, 2 Dec 2006, Martin Waitz wrote:

Not map them, but track their "used" flag. Yes. You can unmap objects any time at all (since you can just always re-create them at any time very easily and cheaply), but the one thing you CANNOT recreate is the object flags. See "struct object", and the "used" and FLAG_BITS in particular.

Almost all git programs need the FLAG_BITS. Something as simple as just traversing the commit history needs at a minimum one _single_ bit for each object: "Have I already seen this". In reality, you tend to need two or three more (ie the UNINTERESTING bit ends up being as important as the SEEN bit, because it's what determines whether it's reachable from some commit we're _not_ interested in, and in the end that's what allows us to not traverse the whole history).

So you need at a MINIMUM to track the bits

	#define SEEN            (1u<<0)
	#define UNINTERESTING   (1u<<1)

and in practice almost everything needs

	#define SHOWN           (1u<<3)

too (SEEN is for deciding whether to _traverse_ something, SHOWN is for deciding whether you've already output the data for this, and the difference is crucial for any depth-first DAG algorithm, since you need to test-and-set the one bit when you first encounter the object, and test-and-set the other bit when you "leave" the object).

So three bits are minimal to _any_ git traversal algorithm. Many specific issues want more bits (eg the TREECHANGE bit may not be quite as fundamnetal, but it sure ends up being critical for the "track subtree" case).

On Sat, 2 Dec 2006, Linus Torvalds wrote:

Btw, just in case you don't understand _why_ this is true, the fact is, in a git repository, quite fundamentally, because we don't have "backlinks" at any stage at all, we don't know - and fundamentally _cannot_ know - whether we're goign to see the same object in the future.

So operations like "git-rev-list --objects" (or, these days, more commonly anything that just does the equivalent of that internally using the library interfaces - ie "git pack-objects" and friends) VERY FUNDAMENTALLY have to hold on to the object flags for the whole lifetime of the whole operation.

And you should realize that this is really really fundamental. You can't fix it with "smarter memory management". You can't fix it with "garbage collection". This is _not_ a result of the fact that we use C and malloc, and we don't free those objects, like some people sometimes seem to believe.

So garbage collection will never help this kind of situation. It flows _directly_ from the fact that our objects are immutable: because they are immutable, they don't have any backpointers, because we cannot (and must not) add backpointers to an old existing object when a new object is created that points to it.

So this really isn't a memory management issue. You could somewhat work around it by adding a "caching layer" on top of git, and allow that caching layer to modify their cache of old objects (so that they can contain back-pointers), but for 99% of all users that would actually make performance MUCH WORSE, and it would also be a serious problem for coherency issues (one of the things that immutable objects cause is that there are basically never any race conditions, while a "caching layer" like this would have some serious issues about serialization).

So: the very fundamental nature and choices that were made in git also 
means that when you have something like git-pack-objects that wants to 
walk the whole repo, you will end up with something that remembers EVERY 
SINGLE OBJECT it walked. 

And while I've worked very hard to make the memory footprint of individual objects as small as possible, and this means that this all works fine even for fairly large databases (especially since very few operations actually do this "traverse the whole friggin tree" thing), it does mean that there's a very fundamental limit to scalability. You can't just make a whole repository a hundred times bigger - because the operations that traverse the whole thing will require a hundred times more memory!

Now, in "real" projects, this is not a problem. I can pretty much _guarantee_ that memory sizes and hardware will grow faster than projects grow. I'm not AT ALL worried about the fact that in ten years, the linux kernel repository will likely be two or three times the size it is now. Because I'm absolutely convinced that in ten years, the machines we have now will be obsolete.

So on any "individual project" basis, the fact that memory requirements scale roughly as O(n) in the total repository size is simply not a problem. In fact, O(n) is pretty damn good, especially since the constant is pretty small (basically 28 bytes per object - and 20 of those bytes are the SHA1 that you simply cannot avoid).

But it does mean that supermodules really should NOT be so seamless that doing a "git clone" on a supermodule does one _large_ clone. Because it's simply going to be better to:

 - when you clone the supermodule, track the commits you need on all 
   submodules (this _may_ be a reason in itself for the "link" object, 
   just so that you can traverse the supermodule object dependencies and 
   know what subobject you are looking at even _without_ having to look at 
   the path you got there from)

 - clone submodules one-by-one, using the list of objects you gathered.

Maybe there are other solutions, but quite frankly, I doubt it. Yes, you'll end up "traversing" exactly as many objects either way, but the "globe subobjects one by one" is going to be a _hell_ of a lot more memory-efficient, and quite frankly, "memory usage" == "performance" under many loads (notably, any load that uses too much memory will _suck_ performance-wise, either because of swapping or simply because it will throw out caches that "many small invocations" would not have thrown out).

So I guarantee that it's going to be better to do five clones of five small repositories over one clone of one big one. If only because you need less memory to do the five smaller clones.

On Saturday 02 December 2006 22:22, Linus Torvalds wrote:

Thinking about this... You have to make very sure to always update the caching layer containing the backlinks on every addition of a further object. You can do this because you always reached this new object by some other object, which exactly is the backpointer.

Now let us suppose we are able to do this. What does this give us?

Take a look at object traversal: We have to store the flag "already visited" for objects we could reach again in the traversal. But with the backlinks, we can see that most of the objects can only be reached via one path, and therefore, there is no need to store the flag, as it never will be queried in the further traversal. (Similar for objects with two paths: When you have visited the object two times, you can throw away the flag, as it is not queried any more).

Regarding the caching layer and object traversal, it would have been enough to only store "is this object reachable via more than 1 path?". For this, the "cache" could be the set of objects reachable with more than one path. And such a set stored in a file should be quite managable, and be quite small, relative to the size of the object database.

In fact, this "cache" can be created with a usual object traversal (which has the original memory requirement), but as long as we do not add objects to the database, further traversals would only need a fraction of memory.

When only adding a small number of objects, it should be easy to update the cache; while with big actions like fetching/pulling, we simply should remove the file with the backlink information.

Again only some thoughts... Pack files are fully self-contained object stores, yes? So in the scope of a single pack file, the offset of this object is enough as object identification. If we could make sure that in any given algorithm touching objects, like commit traversal, we always have the offset available when we need to do an object lookup, then, it should be enough to store object flags only indexed by the offset of this object in the pack. The translation SHA1 -> offset can be done with the pack index. As you usually have multiple packs, a (pack number / offset) tuple should be enough as object ID.

Thinking even one step further: Would it make sense to define an encoding format for the content of commit and tree objects inside of packs, where the SHA1 is replaced by the offset of the object in this pack? As exactly the SHA1 is the least compressable thing, this could promise quite a benefit. AFAIK, we currently only use these offsets for referencing objects in delta chains.

More about the original topic of this thread (and off-topic to the new subject):

Without submodule identities, we would have to clone path-by-path, as we can not distinguish different submodules apart from there location.

On Sun, 3 Dec 2006, Josef Weidendorfer wrote:

You're missing the big issue.

The issue is that a cache like that would ABSOLUTELY SUCK.

You could speed up the non-common operations with it, but:

 - any changes would become a LOT more expensive to do, because they all 
   need to update every single object they add (ie a "commit" would now 
   have to add backpointers TO EVERY SINGLE BLOB).

   Imagine what this does to something like the kernel, where a commit 
   reaches 22,000 files!

   You can do it at a finer granularity (ie do just the direct backlinks 
   and only do the "tree->blob" and "tree->tree" things rather than the 
   full commit reachability, but it's still going to be MUCH more painful 
   than what we do now.

 - the cache would be a lot bigger than the current pack-files, and it 
   would be fragile as hell to boot. Because it needs to get rewritten for 
   every operation, it gets corrupted much more easily, and that's 
   ignoring things like race conditions, so it would now need a ton of 
   locking that git simply doesn't do at all.

 - everything would basically slow down.

 - you couldn't do shared object databases AT ALL, because backpointers 
   wouldn't work. The whole _reason_ you can share object databases is the 
   same reason we can't have backpointers: objects are immutable and never 
   change depending on circustances.

The _only_ downside of the current situation is literally the 24 or 28 bytes per object that we look at. For most operations, we don't even look at that many objects, so it's really the worst-case things.

Right. If the project is totally read-only, the cache would work well.

For real development, it would SUCK. It would make things like "git reset" very expensive indeed, for example (you'd have to unwind the whole cache: either regenerating it - which would take minutes - or being very careful indeed and being able to always remove objects properly and keeping track of them 100%).

IOW, it's nasty nasty nasty. And it doesn't really even help anything but a case that we actually already handle really well (I spent a lot of effort on making the memory footprint minimal).

But it does mean that you do NOT want to traverse a hundred different project "as if" they were one. That's really the only thing it means.

And since you can do submodules as independent projects, and you SHOULD do them that way for tons of other reasons _anyway_, even that isn't a reason to screw up all the _wonderful_ properties of the git object database.

So what I'm trying to say is that the immutable non-backpointer nature of the git database is what makes it so WONDERFUL. It's efficient, it's dense, it's stable, and it allows us all the clever things we do. But it means that we do end up alway spending 28 bytes per object, and we can never throw those 28 bytes away during a single "traversal" run.

Josef Weidendorfer <Josef.Weidendorfer@gmx.de> wrote:

I actually had the same idea the other day. I discarded it after thinking about it for a minute. Here's the problem:

Lets say we do this for the tree and parent IDs in a commit, because these are the most commonly needed part of a commit during revision traversal. So we want to put the offset to the tree and the offset to each parent at the front of the commit somehow to make them very cheap to access.

This means that when we start to write out a commit we need to know the offset to the tree that commit references. But git-pack-objects sorts object by type: commit, tree, blob (I forget where tags go, but they aren't important in this context). So generally *all* commits appear before the first tree. So when we write out the first commit we need to know exactly how many bytes every commit will need (compressed mind you) in this pack so we can determine the position of the first tree. Now do this for every commit and every tree that those commits use... yes, its a lot of work to precompute and store all offsets before you even write out the first byte.

Its even worse with parent commits because ancestors tend to appear behind the commit (newest->oldest) so that "git log" can benefit from OS read-ahead. So you also have to keep track of your parent commmit offsets. Not pretty.

Extending that idea to tree objects (store the offset of the entry) makes the issue even uglier.

Oh, and packs aren't entirely self-contained. A pack is only self contained in the sense that no object in the pack deltafies against an object outside of the pack[1]. However by design an object (e.g. a commit or a tree) can reference an object which is either loose or which is in another pack. This is especially important for every large projects where not every commit/tree/tag/blob will fit into one 4 giB file.

**1** Except in the case of thin packs, which are used only on the network and only to save bandwidth.

Yes, that's a recent feature to reference a delta base.

On Sunday 03 December 2006 03:46, Shawn Pearce wrote:

Yes, it looks like a hen-and-egg problem, but IMHO you can handle it nicely with another redirection, i.e. a table you build up while repacking the file, and storing this table at the end.

You simply sequentially renumber any object SHA, starting from 0 in the order you see them. You can do two renumberings, one for the objects contained in the original pack (1), and one for the external ones (2). Put these new numbers (with a bit distinguishing (1) and (2)) as replacement into commit/tree objects. At the end, you have the new offsets for objects in (1). Put redirection tables for (1) [new number -> new offset] and (2) [other new number->SHA1 of external object] at the end of the new pack. This way, you effectivly have removed all incompressable SHAs from the pack file aside from one entry in the redirection tables for each external object.

The only problem I see is how to decode the objects, i.e. how to get the original SHA1 from an offset: we can not recalculate the SHA1 from the object content as we changed the content itself. But there should be a way to store the SHA1 in front of the object somehow, perhaps it is already given by the current format?

Am I missing something here?

Josef Weidendorfer wrote:

Doesn't this idea clash with the object and delta reusing for repack? Hmmm... perhaps with the two indirect tables it wouldn't, only the tables would need to be recalculated... or perhaps it would because of offset clashes.

On Sunday 03 December 2006 12:10, Jakub Narebski wrote:

In general, yes: you modify object content by encoding, and if you want to reuse the objects without decompression, and need to keep the info to be able to decode, ie. the redirection table.

This gets problematic if you want to join multiple packs, or fetch objects and want to reuse the compressed representation, as the object renumbering is only local to one pack, and numbers can be reused between packs.

So this idea is probably for archival packs only.

hoi :)

On Sat, Dec 02, 2006 at 12:44:20PM -0800, Linus Torvalds wrote:

hmm, really sad, it was such a nice concept until now... You are right, I have to think more about scalability. O(N) anywhere is really bad for submodules. They really should be able to bundle the kernel, mozilla, qt and whatnot into one project and that will get huge.

Linus Torvalds wrote:

If you need a common infrastructure to be able to work with the submodule, then the submodule is not independent of of the supermodule. I see a contradiction in your requirements.

See above.

(There have been lots of use cases for shallow clones but for a long time git did not support them).

If you can extend this partial fetch feature to the non-subproject case I would agree with your reasoning. What makes the subprojects so special in this regard. Do I have to turn a plain tree into a subproject to be able to ignore it? Once you can restrict fetches to parts of the contents you get the ability to restrict fetches to the "common infrastructure" and selected submodules for free.

Regards

Stephan

Here's an real-world example that doesn't contradict:

http://amarok.kde.org/wiki/Installation_HowTo#From_Anonymous_SVN

"svn co -N svn://anonsvn.kde.org/home/kde/trunk/extragear/multimedia cd multimedia svn co svn://anonsvn.kde.org/home/kde/branches/KDE/3.5/kde-common/admin svn up amarok

To compile the sources (from the multimedia directory):"

and there's probably very few people that want to clone the entire KDE multimedia sub&super-module in this case.

//Torgil

On 12/2/06, sf <sf-gmane@stephan-feder.de> wrote:

On Sat, 2 Dec 2006, Torgil Svensson wrote:

And I'll add the note that people who do things like submodules aren't generally even _used_ to them being "seamless", and most of the time probably don't even want complete seamlessness.

As the example that Torgil points to shows, people are quite used to actually even naming the submodules separately, and things like having the "default" set of submodules not equal the "complete" set.

In other words, I don't think people expect or want something hugely more complicated than the CVS/modules kind of file.

What people _do_ want (and that CVS in general is horribly bad at, and this is not a module-specific issue) is to have the _versioning_ work well. When you check out a specific version of a module, you want any _linked_ modules to follow along too.

This is the same reason why CVS users use tags a lot: because even _within_ a single project (no modules, no nothing), it's often hard to re-create the exact state of a version any other way. So you tag every single file and do insane things like that, because CVS just isn't very good at guaranteeing consistency across the whole project.

The exact same thing is true about subprojects. I don't think that people who have used CVS subprojects a lot really mind the CVS/modules file itself (but hey, maybe I'm wrong - I've seen _other_ people maintain modules in CVS, but I've never done it myself), but they do mind the fact that it's hard as hell to do something as simple as "get all modules back to version X" without lots and lots of careful crud (ie tagging every singl emodule, things like that).

Now, I'm not exactly sure who wants to use git modules, so this is the time to ask: did you hate the CVS/modules file? Or was it something you set up once, and then basically forgot about? People clearly use the ability to mark certain modules as depending on each other, and aliases to say "if you ask for this module, you actually get a set of _these_ modules".

_I_ suspect that that isn't the problem people had, and isn't what they have any problems with. What CVS didn't do very well (or at all, afaik) is to say "I want supermodule version XYZ", and then got all the submodules automatically to that (reliable) state. And THAT is something I think is really important for submodules, and it's why I think the most important part isn't actually all the veneer to make "git clone" and "git pull" work (which is really about the CVS/modules kind of wrapper parsing), but actually about the supermodule "tree" object pointing to a very specific version, so that you get the exact same "atomic snapshotting" of multiple trees that you get within a single git tree.

In other words, I _suspect_ that that is really what module users are all about. They want the ability to specify an arbitrary collection of these atomic snapshots (for releases etc), and just want a way to copy and move those things around, and are less interested in making everything else very seamless (because most people are happy to do the actual _development_ entirely within the submodules, so the "development" part is actually not that important for the supermodule, the supermodule is mostly for aggregation and snapshots, and tying different versions of different submodules together).

So that's where I come from. And maybe I'm totally wrong. I'd like to hear what people who actually _use_ submodules think.

On 12/2/06, Linus Torvalds <torvalds@osdl.org> wrote:

What about the case when you want _everything_, do you then have to know the names of all submodules, present and past?

If you have an old irrelevant submodule in the history that happens to have the same name as one of them you are interested in, do you get this as well?

During a debugging session it might be convenient to do a "all but X" kind of fetch if you have a project dependent on several small modules and one of them is the big black sheep.

For simple cases, I think it's sufficient to have the "everyone or no-one" option. If git enforces sending submodules one by one and requires the fetching side to specify links explicitly couldn't the selection be left to the user to decide with "hooks" or plumbing? Default hook could implement a simple white- or black-list.

On Sun, 3 Dec 2006, Torgil Svensson wrote:

Afaik, the way people do this historically is simply:

 - often have an alias for "everything" (eg "all" or "src" or "world"), 
   and if you want everything, you basically ask for it by checking out 
   the "src" module.

   Ie this is the "upstream" way to let downstream check out everything.

 - if you're downstream, and you have a partial repo, and you realize that 
   you want everything else, you just look at gitweb (assuming it is 
   extended to show module information, of course ;) or the .gitmodules 
   (or whatever it would be called) file to get the other pieces manually.

But hey, I also think it would be fine to have "git clone --allmodules" or something ("fetch" too). I think this whole question will depend more on how people end up _using_ module support than on any technical issues per se. Again, I suspect the people who now set up modules in CVS are likely to have a better idea than I do about how they usually do it (and why).

I dunno. Details, details. I'm also not sure this is hugely important.

It could be "solved" by simply having the requirement that all modules need to be named differently (notice that "module name" is _not_ the same thing as "the directory name where the module shows up". That's not the case even in CVS modules, and with a "link" type in the git tree object, the directory where a module shows up would basically be totally independent of the "name" of the module).

I suspect it's more common to name the modules you want to fetch explicitly, rather than make it a "negative" choice, but that sounds largely like just an interface issue.

On 12/3/06, Linus Torvalds <torvalds@osdl.org> wrote:

Okay, missed that part. I wasn't familiar with contents of the CVS modules files and misinterpreted your suggestion.

MODULE [OPTIONS] [&OTHERMODULE...] [DIR] [FILES]

So all this is UI only and the "normal" operations on the supermodule

On Mon, 4 Dec 2006, Torgil Svensson wrote:

Right. That's how CVS modules work (although in the case of CVS modules, the "dir" thing is obviously there in the "modules" file, so it's not _purely_ UI in CVS - this would likely be different in a git implementation, because the "tree" object ends up telling not just the exact version, but the location too).

So my suggestion basically boils down to:

 - "fetch" and "clone" etc will just look at the "modules" file, and 
   recursively fetch/clone whatever the module files talks about. This is 
   the "thin veneer to make it _look_ like git actually understands 
   submodules" part. It woudln't really - they're very much tacked on.

 - the tree entries are what makes the "once you have all the submodule 
   objects, this is how you can do 'diff' and 'checkout' on them, and this 
   is what tells you the exact version that goes along with a particular 
   supermodule version".

In other words, the simple and stupid way to do this is to just consider these two things two totally independent issues, and have different mechanisms for telling different operations what to do.

Is it "pretty"? No. The whole sub-module thing wouldn't be a tightly integrated low-level thing, it would very much be all about tracking multiple _separate_ git repositories, and just make them work well together. They'd very much still be separate, with just some simple infrastructure glue to make them look somewhat integrated.

So yeah, it's a bit hacky, but for the reasons I've tried to outline, I actually think that users _want_ hacky. Exactly because "deep integration" ends up having so many _bad_ features, so it's better to have a thin and simple layer that you can actually see past if you want to.

On 12/4/06, Linus Torvalds <torvalds@osdl.org> wrote:

Thin and simple sounds very good. Let's try it with an example. Lets say we have one apllication App1 and three librarys (Lib1, Lib2, Lib3) with the following dependency-graph:

        App1
          /\
         /  \
   Lib1   Lib2
       \     /
        \   /
        Lib3 (don't really needed for this example but looks nice)

All components can be used individually and have their own upstream, maintainer etc.

To compile App1 however, I need some files from both Lib1 and Lib2 specifying it's API. To satisfy these dependencies, It sounds reasonable to link Lib2 and Lib3 submodules from App1. In your concept, can I construct a modules file to fetch the API files and

Torgil Svensson wrote:

I think not. Then it wouldn't be a submodule anymore, but just some random sources from an upstream project. Not that it's an uncommon workflow or anything, but it's sort of akin to just importing the SHA1 implementation (a few source-files with no real interest in the history of those source-files) from openssl into a different project rather than actually using the entire openssl lib (which would be nice to have as a submodule).

Andreas Ericsson wrote:

Note that this is what partial checkouts (another great idea nobody implemented yet[*1*]; you can do partial checkout but there is no UI for this, and working with partial checkouts is bit hard) is about, although it would buy you only working area space, and not repository (object database storage) space.

For now, you can imitate this by having in in Lib1 and Lib2 the 'includes' branch which would contain only the API (and which you would have to keep up to date with 'master', but it should be fairly easy: just merge changes into 'includes', perhaps with help of git-rerere, or [nonexisting] git-rerere2).

[*1*] Although with our track[*2*] I guess it is reasonable to think it would get implemented soon. [*2*] Out of four "great ideas": shallow clone / sparse clone, submodules support, lazy clone / remote alternates, two are in example-implementation (submodules support) and beta work (shallow clone is in 'next').

Linus Torvalds wrote:

Indeed. With the "tight" integration option we'd also have to have the mechanism to rewrite the tree-entries with the location where the submodule is located in the working tree. This might be needed anyways, but it sure as hell seems a lot easier to just tack that part on when doing a checkout and actually creating all the files.

Andreas Ericsson wrote:

Excellent idea! This way most of the concerns for "separate repositories for submodules" layout about ability to rename directory the submodule resides in, or move submodule are resolved. The other part would be to use submodule-aware git-mv to move submodule(s).

Perhaps the following solution would work best:
 * refs/submodules/<module> holds sha1 of top commit in submodule
 * objects/info/submodules is a file which can be automatically generated
   (or at least automatically updated) on checkout, with the following
   contents:

   <module> TAB or SPC <path to submodule, or GIT_DIR of submodule, or
                        GIT_OBJECT_DIRECTORY of submodule>

   with the usual rule that # and ; means comment, \ at end of line is used
   for continuations, empty lines doesn't matter etc.

On Saturday 2006, December 02 19:41, Linus Torvalds wrote:

Never used CVS/modules, but I used svn:externals. I have a few projects that are libraries that I use in many other projects. So, my directory tree looks like this:

 projects/
   libX/
   projectP/
    libX/
   projectQ/
    libX/

The nightmare I had was that I would add a feature to projectP/libX, and commit it. Great. Then later I'd do "svn update" in projectQ - HAVING MADE NO CHANGES TO IT - and libX would update to the latest version, which turns out to be incompatible with projectQ, and I can no longer even build projectQ. If only libX would stay where it was put. The worst of it is if you check out an older version, say "stable-release" that you tagged last year, the svn:external would always just check out the latest version, so you'd have to go back through the logs to find out what approximate submodule revision you should really check out, check it out and then remember not to do svn update, because that would just reset the external to the latest version. AHHHHHHH! Maddening to say the least.

This fits exactly with what you have described as the primary reason for wanting submodules. I didn't want seamless integration, I was happy to change into projectP/libX to make libX commits. All I actually wanted was the particular checkout of libX for a particular checkin of projectP to be remembered. That's it. Anything else is just gravy.

I'm doing exactly the same sort of thing now but with git. git hasn't fixed the problem (yet) but certainly hasn't made it any worse than it was. svn:externals were nothing more than a way of storing a URL in the repository - who cares, I wish now I'd never bothered, they serve no version control purpose and are merely a UI convenience.

Andy

On Sat, 2 Dec 2006, Linus Torvalds wrote:

I think you'd rather hear from people who _would_ use submodules; I've worked on a number of projects that would have benefitted from that general functionality, but nobody trusted the implementation enough to actually use it.

At my work, we're doing a bunch of stuff with microcontrollers. We've got about a dozen different boards with microcontrollers, and each of them has different firmware. We also have a bunch of code that can go on any of the boards.

The way things are organized currently is that each board has its own project, and there's a "common-micro" project with the common code. This sort of works, but it means that when you change things in common-micro, you never know what effect this will have on boards other than the one you're actually working on. What I'd like to have is that each project has a "common-micro" subdirectory, and changes to each of these can be merged into each other, but that doesn't happen automaticly, and each board's revisions include the common-micro revision they were created with.

A few notes: 

I'd never work on common-micro in isolation. Nothing in there even compiles by itself, because the compiler needs to know the target microcontroller type, which depends on the board it's for. It only makes sense to prepare a new revision of "common-micro" in the context of some particular board, at least if you want to test it at all.

I'd sometimes want to include temporary hacks in the common-micro for a particular board, when things are late and I need to change some library behavior in a way that I know works for the board I'm working on, but I don't have time to think about all of the other boards (each of which is special in some way).

I'd often make some change that I know improves the cleanliness of common-micro, but which requires changes to every board to compensate. I don't want to make all of the changes at once; I'll update each board appropriately the next time I work on it. But, of course, until I update each board, that board needs to keep using the version of common-micro without the changes.

I don't want to have repository states where stuff doesn't work, which means I can't do it as one big tree; I need to be able to make a commit with board1 and a common-micro change without having a version of board2 that would use the changed common-micro, because I haven't come up with a board2 version that works with it yet.

	-Daniel

Daniel Barkalow wrote:

Our setup and requirements at work are exactly the same: We have a few main projects that are developed independently and we have one "helper" project for code that is general enough to be reused. So work on the helper project is only done while working on one of the main projects. When we switch to another main project we integrate the changes to the "helper" project.

That's the theory, at least.

Regards

Stephan

On Saturday 02 December 2006 12:41 pm, Linus Torvalds wrote:

Here's some thoughts on subprojects from my company's perspective. I apologize for the long message.

Abstract: We use submodules heavily in CVS and SVN.  I like what I've read 
from Linus about the "thin veneer" approach of integrating subprojects.  It 
seems conceptually to provide the support we desire.  For us, it's important 
that the mandated linkage between a master project and a subproject is 
minimal to maximize our flexibility in building our processes.

We develop and maintain a lot of embedded applications. Both for higher level systems (ex: 32MB RAM/32MB storage) running the Linux kernel and a customized set of libs/app support code and more deeply embedded environments (ex: 8KB of RAM and 32KB of storage). Even though these two cases are very different in many repects, the version management issues are the same.

- We (mostly) track everything needed to build historical versions of code 
with 100% fidelity.  This includes all of the tools used to compile, build, 
test, deploy, debug, etc. the actual build results themselves.  I initially 
looked at Vesta several years ago.  I love their conceptual approach to this 
problem (integrated build system that caches mid-level build results within 
the repository itself), but it's too unwieldy, very hard to set up (lots of 
up-front effort), and lacks many useful features.

- Most of our "applications" are a relatively small amount of app-specific 
code with references to several/many shared modules.  Shared modules can 
contain support tools, like build/test/debug/deploy support for a given 
embedded platform, in-house developed shared app code, or shared code 
developed by third parties.

- We use CVS to manage our larger system development projects.  The repo is 
about 2GB and has several dozen application-code submodules.  We use the 
"third party sources" approach to tracking submodules as outlined in Ch.13 of 
the CVS manual.  Additionally, we manage our "buildox" (similar to buildroot 
in concept) in another CVS repo.  All prior interesting versions of the 
buildroot can be built from source (toolchains, everything), if necessary.  
Applications contain metadata (a file...) in the repo so the app-level build 
system can ensure it is being ran under the correct version of buildbox; 
clunky but serviceable.  CVS is a nightmare because of its poor 
branch/tagging facilities, and many of the things we *ought* to be doing with 
revision control we don't because of the complexity.

- We use SVN to manage our deeply embedded system projects.  The repo is about 
250MB in size.  Applications use the svn:externals property to reference 
needed modules.  We aren't using a buildbox in this environment yet (bad!).  
SVN's simple branching and svn:externals are a giant leap forward in 
comparison to CVS's capabilities.

Below are some common use case scenarios that are to varying degrees unweildy in CVS and/or SVN. Many of these involving non-trivial branching and merging operations are nearly impractical in CVS, and the lack of merge tracking (to support repeated safe merging from one branch to another) makes some of these a bit tricky in SVN too. Of course neither repo supports disconnected/distributed operation, which would make a number of activities that much simpler as well.

- Round trip module management.  A specific app requires a change to a shared 
module, so it makes a local branch to develop the change.  The "diff" is 
presented to the maintainer (who may be inhouse).  The next interesting 
maintainer version of the module gets imported into our repo (if in house, 
it's already there), where the app can reference it.  This merge process may 
leave changes not yet implemented (or never to be implemented) by the module 
maintainer in the local branch used by the apps.  Other apps are unaffected, 
as they are linking to a prior version in the local branch.

- Pragmatic development.  It's typical that in developing an application, a 
developer will need to simultaneously make changes to one or more submodules.  
If more than trivial, he/she should branch the submodules and continually 
tracking the HEAD of those branches in the relevant app.  This is so complex 
and fraught with problems in CVS that it doesn't get done, and developers 
house too much change over time in their working directories.  With SVN and 
svn:externals, the process is workable.  It is nice that an svn:external can 
point to (the HEAD of) a branch when making changes.

- An application implements a new feature internally (say support for a new 
digital chipset in the embedded world) which later needs to be "promoted" to 
a subproject for use by others.  Pretty easy in SVN.  A challenge in CVS; 
it's really not possible to "convert" app code into a "third party source" 
and retain an historical link.

- Updating build tools.  In concept no different than updating a shared code 
module.  In practice, due to the buildbox strategy, it's a bit convoluted.  I 
don't expect this to get much smoother.  Getting Vesta-like features, where 
integrated build suport can cache lower-level build results in a version-safe 
manner (like the binary code built when the cross toolchain was built) would 
be killer, but that's surely OT for the submodules discussion.

Thanks,

What if we use linus "module" file concept and allow the link objects to track subtrees? An object may look like this:

commit: <SHA1> link: <SHA1> /path/to/remote/tree/or/blob

Tracking upstream library: -------------------------- clone as usual

Inhouse libraries/applications: ------------------------------- To satisfy versioning of build-dependencies - make links of type "external/lib1_header.h" -> "<commit>/headers/lib1_header.h" (blob) "external/lib1_interface" -> "<commit>/api" (tree)

If git supports "sparse fetching" of subtrees we can follow the history in the submodule only concerning the files we want without fetching the whole subtree. "modules" file could specify something like "always clone on fetch"

Build environment ------------------ First make links to all tools, applications, etc ... "buildtools/random_app1" -> "<commit>/" "buildtools/random_app2" -> "<commit>/" "sub_build_projects/user_interface" -> "<commit>/" "sub_build_projects/kernel" -> "<commit>/" "apps/special_app1" -> "<commit>/" "libs/special_lib1" -> "<commit-from-another-build-project>/special/lib/binary/path"

Here we can have a build system that for example creates a "i386" folder and the repo itself

Documentation release ---------------------- "Lib1/" -> "<lib1 commit>/docs" "Lib2/" -> "<lib2 commit>/docs" "App1/" -> "<app1 commit>/docs"

Special customer release for a specific HW platform --------------------------------------------------- "Lib1/lib1.h" -> "<lib1-commit>/headers/lib1.h" "Lib1/lib1.so" -> "<build-environment-commit>/i386/Lib1/lib1.so" "Lib1/docs" -> "<lib1-commit>/docs" "App1_binary" -> "<build-environment-commit>/i386/App1/App1_binary" "docs" -> "<app1-commit>/docs"

commit&tag&bag this and send to customer. If the customer says something is broken, we can make an SHA1 of the customers tree and immediately see if there's changes not belonging to us.

Now this can be broken in so many ways that I can't even count, so I appreciate some feedback to correct my head.

On 12/9/06, R. Steve McKown <rsmckown@yahoo.com> wrote:

On 12/10/06, Torgil Svensson <torgil.svensson@gmail.com> wrote:

This example is somewhat complex since the build for lib1.so and the header-file might not has gone through the same commit on the lib1 subproject. Consider this example:

lib1 - library project (source tracking)
------------------------------------------
Blob: /src/lib1.h

app1 - application project (source tracking)
-----------------------------------------
Link: /headers/lib1.h -> <lib1-commit1>/src/lib1.h

build1 - Build project (binary build tracking)
------------------------------------
Link: /src/lib1 -> <lib1-commit2>/
Link: /src/app1 -> <app1-commit>/
Blob: /i386/lib1/lib1.so
Blob: /i386/app1/app1

Release Project (file compilation tracking)
-----------------------------------
Link: /headers/lib1.h -> <lib1-commit3>/src/lib1.h
Link: /bin/lib1.so -> <build1-commit>/i386/lib1/lib1.so
Link: /bin/app1 -> <build1-commit>/i386/app1/app1

<lib1-commit1>, <lib1-commit2> and <lib1-commit3> should be the same, dictated by the app1 project. Can we enforce this in the modules file or should the different supermodules fix this somehow using scripts/hooks?

How do the super-projects in this case get access to the blobs pointed by the links - transparent or explicit in the build-process?

On Thursday 14 December 2006 22:27, Torgil Svensson wrote:

If you want to track build results for some source, why would you ever want these builds go out of sync with the source? As the built files depend on the source (and other things), the source should be a submodule of the build project.

Hmm... I think I see a problem / wish for submodules here.

With the current submodule proposal, we force submodules to be subdirectories inside of a supermodule.

Your example has the folling submodule dependence ("X ==> Y" means Y being a submodule of X):

  App     ==>   Lib
   ^             ^
   |             |
 AppBuild ==> LibBuild

If we force submodules to be subdirectories of supermodules, Lib needlessly will have to appear two times in a checkout of AppBuild.

However, there is nothing wrong with it. Yet, you perhaps want the 2 Lib submodules not to go out of sync. This easily can be done with symlinking the Lib checkouts. As they are submodules, everything should work fine.

Perhaps an option you want to have is to force a checkout of AppBuild to make these symlinking itself when it detects identical submodules links.

Hmmm... the only problem with a symlink is that it can go wrong when moved. Unfortunately, I do not have a good solution for this. We can not make UNIX symlinks smart in any way. Hardlinking directories would be a solution, but that is not possible.

Another thing: With normal "$buildroot != $srcroot" environments, the source can not be a subdirectory of the build directory. Yet, we want to specify submodule/supermodule relation. This is difficult to do with a submodule object, as it needs to appear in trees in the supermodule.

Actually, the best workaround for this is to make Lib a direct submodule of AppBuild, and specify the relationship of LibBuild ==> Lib only in AppBuild.

BTW, build project commits probably should not depend on any history of other build commits. So you actually want all build commits to be root commits, and have a tag name which could include the source commit id from which the build was done. This gives some loose coupling.

I do not see any problem here. Symlinks are stored in the git repository. As the AppBuild commit depends on App and LibBuild submodule commits, the symlinks always should be correct.

I do not see any need for an hook. But of course, a checkout hook should
be able to generate files/links. However, IMHO this should be not
done with hooks but with Makefile targets.
 

Submodules should automatically be checked out when checking out the supermodule. So the blobs should already be there. Or do I miss something?

On 12/15/06, Josef Weidendorfer <Josef.Weidendorfer@gmx.de> wrote:

I don't, bad wording by me. That was the problem I wanted to address.

In my example "AppBuild" and "LibBuild" were the same project but this scenario is relevant as well.

This is interesting. In my notation:

/path/to/link/name -> <commit>/path/to/subtree

means that there is a link named "name" in the tree object for "path/to/link". The link points to a "link object" specifying a subtree or blob of the tree that is pointed to in a submodule commit. This is not currently implemented but has at least the following advantages:

1. You can access files in a submodule without fetching the whole
submodule (which may be very large). (App1 is only interested in
lib1.h, the rest is toally irrelevant)
2. Superproject can access referenced (linked) files in it's own
folder-structure without being forced a structure by the subproject.

If you do a symlink instead, doesn't you loose versioning information? What happens with the symlinks if someone clones the superproject?

Wouldn't specifying the submodule path in the link object fit in well here? Then each "link object" can represent a checked out tree from the subproject in the superproject directory-structure.

This is true for symlinks and would also be corrected if we have a (sparse) submodule checkout there in it's place.

Why? Can you give an example here.

The main reason for these "links" are for versioning purposes: the uniqe SHA1 of the "link" representing a tree/blob in a version of the submodule should be "included" in the supermodules commit. Symlinks won't give that at all.

Probably not as that was a piece of the puzzle that I was missing.

On Friday 15 December 2006 18:43, Torgil Svensson wrote:

Ah, now I understand. I somehow missed this notation.

That all sounds fine, but how do you create such symlinks in practice?
Do you want to introduce special porcelain commands to create them?
Especially, what is the SCM user supposed to do to change the link
target, ie. from
 <commit>/path/to/subtree
to 
 <commit>/path2/to2/subtree2
?
Should this do a re-checkout at the other point?

By linking a file from a submodule, such a link seems to force that this file has to be at a fixed position in the submodule. Otherwise, some magic has to happen when the file is moved in the submodule, possibly leading to a dangling link, eg. if the whole subdirectory specified in the link is removed.

IMHO this is getting way to complex. Much simpler is to include the full submodule at some path in the supermodule, and create normal symlinks from the supermodule into the submodule.

If you only want to check out part of a submodule, this should be done with path-limiting checkouts, which should be a feature totally independent from submodules.

And if you want to limit the number of objects transferred in cloning
of a subproject, it is better to further split this subproject into
multiple subprojects itself.
 

Of course, you need the submodule fully checked out somewhere in the supermodule, and the link goes into the submodule directory. The versioning is given by the supermodule/submodule link.

As already said: the link has to go into a submodule directory, which will be checked out automatically with the clone of the supermodule.

The problem is not the representation in the git repository, but the checked out module/submodule, where you need to use normal UNIX file semantics. To move submodules around, the user should be able to just use the normal UNIX "mv" commands, and git should be able to detect move actions after the fact. The simple thing here is that currently, git does not have this problem as it tracks content, and does not even try to detect any moves at commit time. This is different with submodules, as there, you want to be able to track moves of any submodule root directories.

This now becomes a problem if you use symlinks to "unify" multiple checkouts of the same submodule at multiple places in the supermodule, and move the symlink around, as it easily can get dangling this way. Thus, you would not have a way to see what submodule this link was talking about.

And for this thing, I do not see how your link object could help.

So it is better to use a simple submodule concept, and for this corner cases, we perhaps could expect the user to fix e.g. a dangling symlink to a previous submodule checkout himself, using a meaningful error message.

If you have a source commit chain A => B => C => D, you want to make any build commits totally independent: you first only are interested in a build commit for source versions A and D, and later find out that a build commit for B and C would be nice, too. If you force build commits into some history order, this order now would be A => D => B => C, which makes no sense.

Build commit independence can easily be achieved by making every commit parentless, without further history. You still have the link to the source version via the submodule link in the tree. But to not loose any such build commits, they have to appear as tags or refs (unless integrated in another superproject build commit).

The version coupling will be there if the whole submodule is available at some path in the supermodule checkout, as said above.

On 12/15/06, Josef Weidendorfer <Josef.Weidendorfer@gmx.de> wrote:

I'm very open to suggestions here, but the concept growing in my head
is based around Linus 'module'-file and keep things simple. A git
configuration file that specifies:
* link name for reference
* local path to link
* submodule source
* submodule path to tree/blob
* submodule commit / HEAD / branch
* options (depth-limit , ...)

I'm reconsidering having the path-name in the link, it should be sufficient to have two SHA1's, one for the commit and one for the tree/blob. Super-module should have the tree/blob in it's database so that the link part only is there for version information and reference (checking dirty state or history on the submodule). This way it easy to clone the super-project and use it without having to map up all sub-project sources. Sub-project sources is not important for version information and could always be specified in the project in a README-type of file.

That would be a change in the modules file, maybe through a command that also fixes the link. The link will have to be updated in the index and commited as normal.

Since we have the SHA1 (this is what we're using) and tree/blob information in the super-modules database the change itself is not a problem. The problem is to track renames/moves and your remove case in the submodule. The tool that tracks the submodule should probably warn/exit here and we would fix up the modules file manually.

One of complex situation here as I see it is the ability to handle to track/checkout only a subset (tree/blob) of the submodule. This is also quite an important feature - in my example it means the difference of tracking one header file versus the whole source.

If we can do path-limiting checkouts on a repo (module) we also can do it on a sub-module since they are exactly the same. This is a very powerful feature and it'd be a huge waste if it wasn't allowed for a super-module to do on submodules.

What if we have no control of the submodule? This can be tracked from upstream, sourceforge, another company, etc. The submodule will often live their own life and could be X, kernel, gcc, cairo, whatever, ...

If we disregard the commit info, the link will act exactly as a normal tree/blob. Git can know we're moving a subproject by watching the module file. The main problem is to keep modules file up-to-date with reality. We could enforce module file validity by disallowing such operations and let the user do a "force" operation which also alters the modules file.

The symlink only exists in the modules file. We only have the SHA1's at the tree-level and there we have everything underneath the tree/blob SHA1 in our database. We will only know if the modules symlink file is dangling next time we fetch from the submodule - here we would notify the user but our database is still consistent.

It makes no sense because the user seem to have act irrationally. The commit-chain is completely valid as it has tracked the correct history of the builds. I can't see any problems here, the build-project is independent of the source-project with it's own history. We can hope the user has given good explanations for his/her actions in the commit messages though.

On 12/16/06, Torgil Svensson <torgil.svensson@gmail.com> wrote:

See it as the link only is there for the version handling between different modules and it's the module file that give an UI to the the link (which project, branch, ....). Many users will not care whats behind those links, but if they want to edit the link they have to create the modules file or fetch it somewhere - it may even be provided and version controlled in the project itself.

example tree object:

100644 blob <sha1 of blob> README 100644 blob <sha1 of blob> REPORTING-BUGS 100644 link <sha1 of blob> <sha1 of commit> 040000 tree <sha1 of tree> arch 040000 tree <sha1 of tree> block 040000 link <sha1 of tree> <sha1 of commit>

Note that the links functions exactly as the blobs and trees in the database. The difference is that they origin from _a_subproject_ (we don't care which in this stage) with the specified commit SHA1. If the link isn't represented in the modules file, it's no big deal, it can be added later on if needed.

If the blame-game begins or if we want to check what we're using on a submodule level we can always pinpoint the exact file/tree content and

On 12/16/06, Torgil Svensson <torgil.svensson@gmail.com> wrote:

Sorry, I was sloppy and forgot the names:

100644 blob <sha1 of blob> README 100644 blob <sha1 of blob> REPORTING-BUGS 100644 link <sha1 of blob> <sha1 of commit> AUTHORS 040000 tree <sha1 of tree> arch 040000 tree <sha1 of tree> block 040000 link <sha1 of tree> <sha1 of commit> misc

Now it doesn't looks like trees/blobs anymore so maybe a link object is handy:

100644 blob <sha1 of blob> README 100644 blob <sha1 of blob> REPORTING-BUGS 100644 link <sha1 of link> AUTHORS 040000 tree <sha1 of tree> arch 040000 tree <sha1 of tree> block 040000 link <sha1 of link> misc

link-object: <sha1 of commit>

Torgil Svensson wrote:

What do you need <sha1 of tree/blob> for in link-object? Wouldn't you use usually the sha1 of top tree of a commit, which is uniquely defined by commit object, so you need only <ahs1 of commit>?

On 12/16/06, Jakub Narebski <jnareb@gmail.com> wrote:

1. "Sparse" repository's - In my example, I want to cherry-pick
header-files or binary-files from different projects without fetching
all, potentially huge, submodules in their entirety. Imaging having X,
kernel, gcc, gtk and libc6 as sub-projects and you really only care
about some header files.

2. Super-module directory-hierarchy independent from submodules.
Super-project want to have the header-files and binaries it's own way.
This also gives version controlled file-collections, the "release
case" in my example - collecting different binaries and header-files
from different submodules together in a new directory-structure, add
some documentation and configuration files and get the whole thing
under strong version-control down to the beginning of time for each
little component.

3. Super-module development independent of submodules - If we have the
tree/blob-object with all it contents in the database many
git-operations can act as the link (commit) wasn't there since we have
access to all relevant data to work with. This makes it easy to clone
the super-project and work on it seamlessly without having to care
about submodules or mapping up submodule repository's (unless you want
to modify the links or the data underneath it of course).

<opublikowany i wysłany>

Torgil Svensson wrote:

This would be (using the submodule original proposal)

    140000 link <sha1 of link>     misc

All fine, but this does not and I think cannot protect us from the fact that we can have <sha1 of tree/blob> which doesn't match <sha1 of commit>.

I think it would be better to have sparse/partial checkout first. But that is just my idea. Because with <sha1 of tree/blob> which is not sha1 of commit tree you might loose (I think) the ability to merge, for example your changes to submodule with upstream.

This is I think irrelevant to the fact if we have only <sha1 of commit>, or link object and also <sha1 of tree/blob>

[jc: adding back people on CC list while feeling sick of having to do so...]

Jakub Narebski <jnareb@gmail.com> writes:

If I recall correctly, the original original proposal used 160000 because that is not a bitpattern used in stat.h, and a link behaves like a directory and a symbolic link at the same time.

On 12/16/06, Jakub Narebski <jnareb@gmail.com> wrote:

True, that will be a real problem. Unless we have a bug in git, do you see a scenario in which this is likely to happen?

That's correct. I also want a sparse/partial checkout but I don't want the full submodule path. I'm also perfectly fine (for my current use-cases) with not being able to merge upstream unless we're tracking the commit tree (here, we might not want to specify the tree SHA1).

I'm not trying to impose a technically fragile solution here [I don't believe it is, but I'm not the most competent to say that either], I'm trying to find solutions for my use cases and I had problems adapting them to the current suggestion.

On 12/16/06, Torgil Svensson <torgil.svensson@gmail.com> wrote:

This might not be as problematic as we think. If we do the same sparse/partial checkout (what's the definition here?) with the <sha1 of tree/blob> as we do with the only <sha1 of commit> case and consider the <sha1 of tree/blob> to be a _local_ (to the super-project) shortcut. Then we only track the submodules using the commit - local conflicts are easier to handle, git would refuse to commit a <sha1 of tree/blob> not present in the commit tree.

We might even consider two object types: module: <sha1 of commit> name link: <sha1 of commit> <sha1 of tree/blob> name

Torgil Svensson wrote:

Well, I just rather have than <sha1 of tree/blob> the definition of sparse checkout (for example subdirectory name, or file name, or glob pattern).

Besides you need the name of directory (for tree) or file (for blob),
otherwise you would have no way to update it when submodule advances
version, and you want to use new submodule version. And if you have
that, you don't need <sha1 of tree/blob> in repository, in link object.
You might want it in the index, for performance reasons, though.
 

With sparse (for example defined by 'src/*.h') or partial (for example defined by 'Documentation/') checkout you should be able to merge upstream... unless conflicts are in the not checked out part.

Have you read  http://git.or.cz/gitwiki/SubprojectSupport on GitWiki?
Have you tested the experimental submodule support (proof of concept)
  http://git.admingilde.org/tali/git.git/module2
by Martin Waitz?

On 12/16/06, Jakub Narebski <jnareb@gmail.com> wrote:

This is entirely an UI issue:

On 12/16/06, Torgil Svensson <torgil.svensson@gmail.com> wrote:

On 12/16/06, Jakub Narebski <jnareb@gmail.com> wrote:

Correct. Since the commit contains all the version information, the
following combinations should give the same information iff we keep
the commit in the database:
1. <sha1 of commit> + <sha1 of tree/blob>
2. <sha1 of commit> + <symlink to tree/blob>

I used the sha1 because I wanted them to behave exactly like trees/blobs in the database for operations that can disregard the commit info. Now, if we keep the commit in the database as Linus suggests we can reach the target from there with a symlink. This would be more readable but also cost a few object lookups extra iterating over the symlink.

This would be a great feature! Will this conflict with path shortcuts? If so, we might consider two types of objects: "link" which cannot merge upstream and "module" which can merge upstream and contains a .git object repository.

IMHO, "module" is a more intuitive name for specifying a (functionality wise fully fledged) submodule with a repository inside. "link" could be used for just mirroring a tree/blob. I'm not sure if a separation is needed on a technical level.

Yes

Not yet

On Sat, 16 Dec 2006, Torgil Svensson wrote:

That 040000 needs to be something else.

In order for something like a git-fsck-objects to know that it's a link, it needs to be marked as such.

In git, we never just randomly open an object by SHA1, and then figure out its type. We always open things by explicitly knowing both the type and the SHA1, and if the object we find has the wrong type, that's a consistency error in the database (or the user).

On Fri, 15 Dec 2006, Linus Torvalds wrote:

Side note: that's not to say that I would really see why you'd want to have both the tree and the commit SHA1's, and why you seemingly think that the links don't need a filename. Hmm?

If you require the tree objects to be in the database, you might as well require that the commit object be there. But you could make rules that say that subprojects don't need the whole commit history, for example (which is just a shallow clone in the subproject).

On 12/16/06, Linus Torvalds <torvalds@osdl.org> wrote:

I really want that file-name back - we can call it a mind short-circuit.

You have a very good point here, this would give us the history of the

hoi :)

On Fri, Dec 01, 2006 at 03:09:40PM -0800, Linus Torvalds wrote:

An interesting way to support this "only fetch some modules" use-case is to use several supermodules.

So you could have one supermodule which is geared towards developers and only contains the modules they use. Another supermodule contails all the toolchain sources. And then there is the supermodule used for releases which is just a merge of all the other supermodules.

The concept is so flexible that you don't have to introduce lots of other things as module namespaces. Just use the tools you have in a creative way ;-)

On Friday 01 December 2006 23:26, Linus Torvalds wrote:

OK.

I still think it should be about "limit in space" regarding the objects in the local repository.

For a project containing "gcc" as submodule, and I am not interested in this submodule, there should be a way to not need to fetch all the objects from the gcc submodule at clone time.

What about my other argument for a submodule namespace:
You want to be able to move the relative root path of a submodule
inside of your supermodule, but yet want to have a unique name
for the submodule:
- to be able to just clone a submodule without having to know
the current position in HEAD
- more practically, e.g. to be able to name a submodule
independent from any current commit you are on in the supermodule,
e.g. to be able to store some meta information about a submodule:
- "Where is the official upstream of this submodule?"
- "Should git allow to commit rewind actions of this submodule
   in the supermodule?" (which, AFAICS, exactly has the same
   problems as publishing a rewound branch: you will get into
   merge hell when you want to pull upstream changes into the
   supermodule)
- "Should this submodule be checked out?"
and so on.

hoi :)

On Fri, Dec 01, 2006 at 11:55:03PM +0100, Josef Weidendorfer wrote:

you can always have a bare repository for all used modules lying around in some defined location. There is no need for a unique submodule-name.

On Fri, 1 Dec 2006, Josef Weidendorfer wrote:

Umm? I don't get the issue. A submodule is a git repo in its own right, and you clone it exactly like you'd clone any other repo. It _does_ have a HEAD. It has it's own branches. It has everything.

So when you clone a submodule, you always get all those branches. The supermodule will not _point_ to them all (the branches are local to the submodule, and _will_ depend on things like "which upstreams module am I tracking"), but they'll have to be there, exactly _because_ the submodule has an existence and is tracked on its own.

In the trivial case where the submodule doesn't even _have_ any external existence at all (ie it's always maintained as _just_ a submodule, it would probably tend to have just one branch, and a clone would get whatever that branch is), but that's just a degenerate special case of the much richer "this submodule actually has a life of its own" case.

The current commit within the supermodule would be _totally_ invisible to the submodule.

Of course, if HEAD _differs_ from that commit within the supermodule, then a "git diff" (when done from within the supermodule) should show that, but again, that's actually only as seen from the _supermodule_.

That's entirely a question for the submodule. You cannot ask that question within the confines of the supermodule, because it's not even a relevant question in that context. Two different supermodule repositories may well decide to get their submodules from difference places, just because they got cloned from different places (or even just for practical reasons like "that other site is closer to me").

So the official upstream of a submodule must NOT be encoded inside the supermodule (or at least not within its _objects_). Exactly because the upstream location is not a "global" thing - it's per-repository, and thus must not be encoded in the global data (ie the objects).

It should be be encoded in some _ephemeral_ place, eg in the ".git/config" file or in a ".git/remotes/origin"-like file (either in the supermodule or the submodule, and I would seriously suggest you do it within in the submodule itself, because you'll want it exactly when you decide to work on the submodule and upgrade _that_).

The only thing that a submodule must NOT be allowed to do on its own is pruning (and it's distant cousin "git repack -d"). You must always prune from the supermodule, because the submodule cannot really know on its own what references point into it.

(There are alternatives. One alternative is to never allow rewinding - or deletion - of branches in a submodule, and thus solve the problem that way. That is the easier solution, because it also means that a "clone" of a supermodule can just recursively clone the submodules independently _without_ having to worry about reachability, but it's really _really_ draconian).

This, I think, requires too much configuration to say separately for every possible submodule, so I would suggest that the way to make that decision is:

 - "git clone" by default will fetch and check out all submodules (and 
   obviously they have to be described some way outside of the object 
   database, just so that you don't have to parse the _whole_ history of 
   the _whole_ supermodule just to find all possible submodules. So the 
   supermodule _will_ need some "list of submodules and where to get them" 
   in a config file or other).

 - add a flag (possibly just re-use the current "-n" flag) that disables 
   that recursive fetching of submodules entirely.

 - have a way to fetch individual submodules one-by-one (that capacity 
   obviously has to be there anyway, since the "recursive" git clone has 
   to be able to do it, so this is likely just "git clone" again, with 
   just logic added to say "when you clone something and are _already_ 
   within a superproject, the clonee becomes a subproject automatically"

I dunno. And I'd also like to point out that things don't have to all work fully before we can do at least some cases of this. For example, if the initial version just always clones everything, big deal. I'm not saying that we have to have support for things like this on "Day 1", I'm just saying that I think people will want to be able to not fetch and check out everything, so the design should _allow_ for it.

(But I also think that as long as submodules are independent enough, the "design" part should fall out on its own, and it just becomes a "small matter of programming" to actually get it to work).

On Saturday 02 December 2006 00:30, Linus Torvalds wrote:

I just thought about the case when you want to clone a submodule directly out of the supermodule repository, at a given realive path. And that can be changing.

Of course, every project which happens to be submodule of some supermodule, also can have its own repository, as it is fully independent. And then, you of course can clone from without any knowledge of its relative position in the supermodule.

Yes.

Of course.

Yet, you need some name to store meta information of submodules into some config file of the supermodule, like whether you want to have it checked out (see below).

In that case, such a name for a submodule does not have to be global in the supermodule project...

Yes. I just gave an example of a policy some project may want for submodule handling.

Exactly. And in this list, you have to specify names.

The thing I wanted to discuss is whether such names would need to be globally unique in the project containing submodles, or not.

If yes, it IMHO makes a lot of sense to introduce "submodule objects" which contain these submodule names, and which are used as pointers to submodule commits in supermodule trees.

On Sat, 2 Dec 2006, Josef Weidendorfer wrote:

Yes, you do need to have a list of submodules somewhere, and you'd need to maintain that separately. One of the results of having the submodules be independent from the supermodule is that it's not all "automatically integrated", and thus the supermodule does end up having to have things like that maintained separately.

And yes, if you screw that up, you wouldn't be able to fetch submodules properly etc, even if you see the supermodule, and yes, this sounds more like the CVS "Entries" kind of file that is more "tacked on" than really deeply integrated. But I think the separation is _more_ than worth the fact that you can see things being separate.

In fact, I'm very much arguing for keeping things as separate as possible, while just integrating to the smallest possible degree (just _barely_ enough that you can do things like "git clone" and it will fetch multiple repositories and put them all in the right places, and "git diff" and friends will do reasonably sane things).

Keep it simple, stupid. 

Yes. 

My preference would be for it to be "local", just because (as I mentioned), with mirroring etc, it might well be that you want to fetch things from the _closest_ repository. That's really not a global decision, it's a local one.

You could do it that way, and then it would be global. It would work, and in many ways it would probably be "simpler" on a supermodule level.

The advantage of a global namespace is that you can much more easily update it - "git fetch" will just fetch the new file(s) that describe the subprojects very naturally if they are all global. Putting them in a local .git/config file has it's advantages (see above), but it also makes it very hard to version them, and to update the list - it would have to become manual.

There are possibly combinations of the two approaches: have a "global namespace" that describes the canonical place to get the subprojects, but have some way to add local "translation" of the canonical names into locally preferred versions (eg you could just have a way to say "this is the local mirror for that global canonical place")

Maybe that would work?

On Saturday 2006, December 02 00:33, Linus Torvalds wrote:

Why? You just recursively search for every "link" object in the supermodule. That tells you which submodules you need and where they should be.

During a supermodule clone, it can tell the client end to start a new clone with the correct path because it knows what the local path is at that moment.

Andy

(I wrote most of this a couple of days ago, so it's not at the tip of the conversational tree, so to speak. But it's effectively a response to Linus's "what do you want to do with submodules" question, with some thoughts on implementation. Sorry it's so long; like Blaise Pascal, "I would have written a shorter letter, but I did not have the time.")

The supermodule concept, implemented right, could really improve cooperation among embedded platform integrators, boutique distro publishers, and other editorial contributors to sprawling metaprojects who don't want to run kernel.org-scale mirrors. To make this work, you need sparse repositories (conserving resources when fetching, by omitting the bulk of currently un-needed submodules that can reliably be obtained later from elsewhere) and shallow cloning (conserving resources when publishing, by referring cloners to a third-party repository for universally available content).

For instance, it would be a wonderful thing if the pile-o-patches nightmare that is PTXdist (and crosstool and buildtool and every other approach I have seen for ongoing maintenance of embedded toolchains and userlands) were obsoleted by a git supermodule. Its submodules would mostly track external projects, but would also logically contain the fix-up patches worked out during platform integration, checked in to branches anchored at each upstream release point. The supermodule would contain all of the build automation, log auditing, and remote unit testing stuff, as well as the metadata for each submodule involved in this platform build cycle.

At a content level, the sparsely populated / shallowly published supermodule wouldn't be much different from today's PTXdist. But the pay-off comes when you merge forward to a new release of some base component (compiler, library, etc.) and discover that some of your fix-ups have been adopted or obsoleted upstream, and new fix-ups are needed for components that depend on the updated bit, and the set of configurables has changed (for which you need to compensate in the meta-configurator). Instead of piling up versioned patch directories, you commit fix-ups to the sub-modules, which other integration branches can ignore (if they aren't affected), merge, or cherry-pick.

As I understand it, in today's git, every content object is a patch to the _data_ of one and only one git repository, containing the label of the preceding _data_ state plus a diff of file contents and attributes. Assuming this model is retained, any clean state of a "leaf" module (one with no submodules) can be reached by replaying a series of patches, starting from the repository's root node (an empty directory with the hopefully unique label generated by init-db). The label (SHA1) of the last patch is therefore a perfectly good label for this _data_ state.

If all we were trying to do with supermodules was to capture and track various states of the submodules' data, we could extend the format of content objects to include "state X of submodule with init-db label Y". That would have the effect of capturing submodule states as _data_ in non-"leaf" modules. We would have to help cloners find a place from which to pull these states, of course; and it's easy to get sidetracked onto that part of the problem. But that's not where the bang for the buck is in supermodules.

The whole model of distributed supermodules, with references to slightly diverging submodules whose content should mostly be fetched from external sources, smells to me just like LVM. The external sources (like an LVM volume of which you have taken a "snapshot") make up the bulk of the content pool. They also give you a window into developments on the submodule's own branches (like being able to peek forward and merge changes from the original volume). The supermodule (the snapshot volume) provides most of the interesting refs (submodule commits referenced by supermodule tags and branch heads), along with enough "journaled" content to replay forward from some checkpoint guaranteed to be available in each external source to any of these refs.

The implication here is that submodule states are not just SHA1 labels to be embedded within supermodule data diffs. One ought to be able to clone a supermodule without immediately cloning full copies of any of its submodules. This ought to populate the clone's content database with all of the quanta of submodule content that aren't guaranteed to be available from any not-too-stale submodule mirror. When cloning, you don't want to have to inspect every supermodule state for submodule states that are outside the global subset. So the supermodule needs to maintain a set of supplemental refs from which all referenced submodule states can be reached. This allows you to traverse the portion of the pool of submodule content that can't be reached from true submodule branch heads.

On 12/1/06, Linus Torvalds <torvalds@osdl.org> wrote:

This is not a defect; it's a virtue.  It's important for every commit
to the supermodule to contain the information of which submodule
branches you're currently on and how far along them you've crawled.
Any particular supermodule commit point is likely to reflect an
integration milestone visible only to the person working at the
supermodule level.  No content object should ever cross a submodule
boundary, because then you wouldn't be able to apply it to the
submodule in isolation (or in another supermodule state) or identify
it when it is applied upstream and propagates back to you in a pull.
 But the supermodule can also contain supplemental refs (heads and
tags) that don't exist in the submodule (and shouldn't necessarily be
pushed to it); the commits they refer to are localized to the
submodule but may not be reachable from any of the submodule's branch
heads.

There is an opportunity for useful deep integration here. The same algorithm that does reachability analysis for "git prune" can dig from supermodule down to submodules, copying objects into the supermodule database until it hits a commit that is advertised as "global" by the submodule. "git clone" of the supermodule can then pull the bulk of the submodules (a superset of the "global" subset) from (a mirror of) the canonical place for each, and use the supermodule object database as an alternate source for commits that don't exist in the "canonical" submodule.

As simple as possible; but no simpler. The "alternates" / "git clone --reference" model is already almost powerful enough for the supermodule to contain a "journal" of submodule commits that haven't yet been retired to the canonical subset (guaranteed present in each mirror). The only difference is that the supermodule should be considered a "weak alternates" source. Commit objects in the supermodule's database should be visible to submodule-level operations (so that commits which are accepted upstream get flowed in nicely during "git pull").

But if a commit becomes reachable from a ref that is really in the submodule (not just one of the supermodule's "supplemental refs", which should _not_ be visible to submodule operations), then it should be copied into the submodule's object database. (The refs internal to the submodule should retain their integrity even if the supermodule is inaccessible.) The existing "strong alternates" mechanism should be reserved for repos which are at least as public and persistent as the submodule, and supermodules don't qualify (e. g., Linus's transmeta scenario).

I think "global resource, local provider" is the way to go, with each
provider advertising what checkpoints of what resources it can supply.
 When I clone or pull, I should be able to consult a local mapping of
submodule URIs to "mirrors" (which may well be local repositories
containing content and branches that aren't in the "official"
upstream).  The only thing that may need "global" agreement is the
boundaries of the "global" subset for each submodule, i. e., the set
of commit objects that can reliably be obtained from any mirror of the
"official" upstream repository.  That doesn't need to be terribly
clever; "at least three days old on a globally published branch" would
probably be a perfectly good heuristic.

I think the implication of "submodule objects" is that supermodule diffs would say "roll submodule X from commit-id A to commit-id B". I don't think that would work very well for pulls/merges in the sparsely populated scenario, because you want to be able to pull the non-canonical subset of the individual diffs between states A and B into the supermodule's object pool. When you decide later to flesh out submodule X, you should only have to clone some canonical mirror and then fast-forward to state B using objects you already have in the supermodule pool.

The merge case is even clearer. Suppose I pull updates from two remote branches of the supermodule onto my master branch. Each remote branch has added the same submodule, cloned from third-party repositories whose clone history goes back to the same origin. (The example I have in mind is when some project switches to git from some other SCM, and the maintainers of the remote branches port their integration patches over from their git-svn tracker submodule to a clone of upstream's new git repo.) I should be able to postpone the merge effort, come back later and clone the upstream repo, then merge the non-canonical commits that were pulled earlier.

I might want to decide at supermodule pull time to postpone pulling the bodies of the submodule commits; but I want the full sequence of submodule commit IDs in the supermodule commit object. So it's not so much the supermodule _state_ that has a hierarchical structure; it's the supermodule _diffs_ and _object_pool_ that become hierarchical.

I think the only global-to-local-namespace mapping applies to the different labels for the "empty repository" state generated at init-db time. Given the init-db SHA1 of the linux kernel repository, I should be able to choose any mirror or clone of that repository as a source for objects in its "global set". I expect this provider not to scribble on globally published branches, but that isn't even all that critical; anything outside the canonical set is kept in the supermodule's object pool, so I can always blow the submodule away and regenerate it from a different mirror.

Sure. But all you really need from the canonical place is its init-db SHA1 (permanent) and its list of globally published branches (monotonically expanding). A URL for it is a convenient shorthand but doesn't have to be persistent.

Cheers,

Michael K. Edwards wrote:

Did you see GitTorrent? http://gittorrent.utsl.gen.nz/ A lot of similar ideas to what you mention. Sorry, still no prototype :)

I'd see the submodules thing as a good way to glue together a whole bunch of repositories, so that the core mirror servers only have to mirror a small-ish number of repositories.

Sam.

On Fri, Dec 01, 2006 at 04:12:10PM -0800, Linus Torvalds wrote:

But is this object (and all the objects it points to) going to be available (in the superproject) ? The following seems to suggest that you think they shouldn't. How is fsck-objects then going to check that such an object is valid ? Is it going to call fsck-objects recursively on the (available) submodules ?

On Fri, Dec 01, 2006 at 03:30:32PM -0800, Linus Torvalds wrote:

How are you going to enforce this if the submodule isn't supposed to know that it is being used as a submodule ?

How is one of the supermodules going to know what references from other supermodules containing the submodule point into the submodule ?

On Sun, 3 Dec 2006, Sven Verdoolaege wrote:

Note that there's actually two "submodules":

 - there's the submodule "project" itself.

   This one must be totally unaware of the supermodule, because this one 
   might be cloned and copied _independently_ of the supermodule.

 - there's the PARTICULAR CHECKED-OUT COPY of the submodule that is 
   actually checked out in a supermodule.

   This is just a specific _instance_ of the particular submodule.

So a particular instance of a submodule might be "aware" of the fact that it's a submodule of a supermodule. For example, the "awareness" migth be as simple as just a magic flag file inside it's .git/ directory. And that awareness would be what simply disabled pruning or "repack -d" within that particular instance.

But this magic flag doesn't affect the bigger-picture git repository. It's a _private_ flag. So it doesn't affect the git part, any more than it really affects the git repository that you may have a

	[user]
		name = Myname
		email = myemail

in your .git/config file.

See? You can have private data in a git repository, but that doesn't mean that it's visible as _repository_ data. But it can still affect how git commands act (eg the "user" definitions above will affect the default user information that "git commit" uses, of course, without actually affecting the git archive in any other way)

Why would it care? They are other supermodules. It doesn't matter, the same way it doesn't matter that _my_ "git" tree may not have all the same references that _your_ "git" repo has. If I want to get the same references, I'd need to fetch them from you, and at that point, I'd need to get all the objects that are pointed to by those refs too. But only on "git fetch" do you actually start caring.

Linus Torvalds wrote:

If we use objects/info/alternates (or equivalent, e.g. objects/info/modules, or modules file) in superproject to refer to submodule repository object database (so superproject has access to all the objects including submodule), I'd prefer to have in submodule objects/info/borrowers file, which would point to superproject (and to other repositories which have submodule as one of alternate object databases) for git-prune and friends to check which parts are truly unreachable.

This would be generic solution to the problem with alternates, not only specific to submodule support.

On Sunday 03 December 2006 23:32, Linus Torvalds wrote:

That prohibits the problem in your supermodule and your instance of the given submodule.

But IMHO, using a submodule commit which could be removed by pruning in another instance of the submodule is really not the thing you ever want. If you start your own branch in a submodule, and start to rely on it in the supermodule, you _will_ want to push this to the submodule upstream.

And if you find that you have to rebase in the submodule, you simply have to rewrite your branch commits in the supermodule too. Otherwise, you effectively fork the submodule project purely for your superproject.

So I suppose that in practical use, pruning in submodules probably would not have any negative effect. If it has, you made something wrong. So you probably should only a submodule commit if it has "publishing quality" (unless being on a temporary supermodule branch).

Ie. any "borrowers" file should be empty.

Linus Torvalds wrote:

True. But if you need the changes to the submodule for your supermodule to function, and upstream either does not want to merge your changes or the merge will be available only after a long time, then what is the alternative? You must be able to keep local changes, and you must be able to keep pulling from upstream. Of course, what you describe is the ideal case: You find a bug, push the fix upstream, and in no time at all your fix is merged and you can just pull a new version into your superproject, but that might be wishful thinking.

No! All you need is a naming scheme to address the commit of the subproject that should be pulled. The extreme case would be to just address it with its id (well, currently you cannot do that with git pull, but that is fixable). But I already proposed a syntax for naming commits which are "hidden" in a superproject: Just name the path as described in git-rev-parse and append double slashes (to indicate that you mean the commit, not the tree it contains). So no manual work needs be done by upstream.

[snipped: about independence of submodule branches]

[transmeta example snipped]

If you want to track some chosen submodules there are two easy solutions:

1. If you want to track their state as it appears from the supermodule's
view, pull from master:<submodule>//
2. If you want to track their state from their own development branches,
 pull from <submodule>/master

Can you see the difference?

Every commit is a git tree in its own right, is it not?

[description of independent submodule development snipped]

I am not sure I understand what you say.

1. If you are working on a submodule, then the supermodule never enters
the picture. You are working independently. So far, so good.

2. If you are working on the supermodule, git will not be able to
function? How would you work without submodules, in which case you would
 have simply one large project?

I totally agree. When I try to explain why submodules work that only exist as part of one or more supermodules, I do not mean to say that you cannot or should not have independent branches or repositories for the submodules' code.

I took that for granted: from a commit you only ever look backwards (in time/history dimension) or downwards (in content dimension).

Regards

On Fri, 2006-12-01 at 14:13, Linus Torvalds wrote:

An implication of this is that the entire administrative responsibility for having some super-sub module interaction lies entirely with the supermodule.

Why not have a "glue" object at the "stub"-interface of the supermodule tree that provides policy mappings to the sub-modules. Perhaps indicating git URL location, mappings of branch names between super- and sub- modules, special commit SHA1s, user policy or config choices at the boundary, and things like that.

Is that the sort of direction we are headed?

jdl

On Fri, Dec 08, 2006 at 12:29:14PM -0600, Jon Loeliger wrote:

Not unless you have something useful in mind that could be put in these glue objects. URLs and branch names, in particular, should not be stored in the repository itself, but in configuration files, since they will be different for different copies of the repo.

Jon Loeliger wrote:

That's a good thing. I wouldn't want the openssl maintainers to have to bother with every project that uses their code, and I'm fairly certain they feel the same.

hoi :)

On Fri, Dec 01, 2006 at 08:49:20AM -0800, Linus Torvalds wrote:

Yes, but it is not only about tracking, also about distributing submodules.

One Fedora X developer fixes a bug in the intel driver, commits that to the submodule and then updates the supermodule to the new version (by calling "git-update-index drivers/intel && git-commit" or something). Then another Feora X developer updates his X repository. By pulling the supermodule he also gets a new version of the submodule. And this new version of the submodule is stored in a branch which can be accessed by the submodule.

The supermodule always contains _the_entire_ submodule with its complete history, so it also does contain changes. But it does not per-se contain changes, only indirectly (i.e. the commits in the submodule are not part of the supermodule commit chain).

Yes.

On Fri, Dec 01, 2006 at 05:38:44PM +0100, Andreas Ericsson wrote:

Because I want the submodule to act as a normal git repository. Please note that I also voted against changing HEAD directly, but that the new commit which came from the supermodule is just stored in one branch of the submodule, as part of the supermodule checkout.

Martin Waitz wrote:

You're assuming the super- and sub-module will share HEAD, or at least ODB, I think. I'm not convinced this is necessary. Convince me. I'll go drink bear and get some dancing done while you're at it ;-)

hoi :)

On Fri, Dec 01, 2006 at 07:08:12PM +0100, Andreas Ericsson wrote:

not HEAD, only ODB.

Get me a beer and I will convince you :-)

Martin Waitz wrote:

The opposite: If you work in the supermodule, even if it is in the code of the submodule, you only commit to the supermodule. The submodule does not "know" about these changes after step 1.

Why do you mix up supermodule and submodule? The way I see your proposal you cannot change submodule and supermodule independently. That is a huge drawback.

Regards

hoi :)

On Fri, Dec 01, 2006 at 02:51:49PM +0100, Stephan Feder wrote:

I think we are using totally different definitions of "submodule".

For me a submodule is responsible for everything in or below a certain directory. So by definition when you change something in this directory, you have to change it in the submodule. You can't change the submodule contents in the supermodule without also changing the submodule. This is just like you can't commit a change to a file without also changing the file.

Then the supermodule just records the current content of the entire tree. The only new thing is that instead of simple files there are now submodules and that are also recorded.

No, this is the benefit you get by introducing submodules. Why would you want to introduce a submodule when it is not linked to the supermodule?

Martin Waitz wrote:

No so different. The way I see it is that "I" (meaning with submodules implemented as I proposed) could pull regularly from "your" repositories (implemented as you proposed) and work with the result (including submodules). Could you do the same?

But you do not consider the case where you cannot change the submodule because you do not own it.

For example, git has the subproject xdiff. If git had been able to work with subprojects as I envision, and if xdiff had been published as a git repository (not necessarily subproject enabled), it could have been pulled in git's subdirectory xdiff as a subproject. There would not have been a separate branch or even repository for xdiff in the git repository.

All changes to xdiff in git could have been committed to the git repository only. Independently, they could have been published to upstream and be put into the xdiff repository by its author. But the last part is what only the owner of the xdiff repository is able to decide.

(Ok, ok... the example sucks badly because xdiff has been massively changed for its usage in git so the changes would not be integrated by upstream. But you can imagine where you use a library essentially as is, only if you discover bugs you fix them immediately in your repository and keep those fixes in your version of the library, even on upgrade, until the bugs have been fixed by upstream.)

There is a difference. I would say: If you commit a change to a file in one branch, it need not be changed in all branches.

Yes, and that is all you need. If the changes are to be part of a branch of the submodule, they have to be pulled. That is an independent operation.

Because the submodule must be independent of the supermodule.

I see where you are coming from. You have one project that is divided into subprojects but the subprojects themselves are not independent.

What I would like to solve is the followng: You have a project X, an this project is made part of two other projects Y and Z (as a submodule or subproject or whatever you want to call it). The project X need not, must not or cannot care that it was made a subproject. But in projects Y and Z, you must be able to bugfix or extend or modify the code of projectX, and you must be able to push and pull changes between all three projects (of course we are only talking about the code part of project X).

Do you see where your solution makes that impossible, and that with more changes to the repository layout?

Regards

Stephan

hoi :)

On Fri, Dec 01, 2006 at 04:47:47PM +0100, Stephan Feder wrote:

Sorry, but with all that many people proposing things I am a bit lost now. Sometimes I thought you want exactly the same thing as I do, sometimes I think we are talking in totally different directions.

I do not understand you here. The submodule is part of the supermodule, and the one who sets up the repository owns the whole thing, including all submodules, just like all the files which are part of the project.

If you mean the upstream repository of the submodule, then yes, this is of course completely separated from the submodule and may be owned by someone else. Consequently, this upstream repository of course does not need to change when someone introduces changes in the supermodule.

This could have been done if submodule support would have been available at the time xdiff was introduced, yes.

What separate branch or repository are you talking about?

Yes, but if it would have been integrated as a submodule it obviously would have been committed to the xdiff submodule inside the git repository. So the changes are really part of the git repository, but you could go to the "git/xdiff" directory and only see the changes in the submodule, without the normal supermodule history.

Of course, everything still works like normal git repositories.

But you need to change _at_least_ one branch. Otherwise you cannot commit to a branch.

So if you change something in a submodule, you have to change one branch in the submodule. If you call git-checkout in the supermodule this will result in something like a git-reset in the submodule.

Of course.

So if you wanted to check out everything, you could have something like ~/src/X, ~/src/Y/X, and ~/src/Z/X. All of these would be GIT repositories, all of them have their independent branches.

What I am saying is just that if you update Y, and the new Y contains an updated version of X, then ~/src/Y/X/.git/refs/heads/master will be changed by the pull, resulting in the new version of X being checked out in ~/src/Y/X (alongside all the other updates inside ~/src/Y). This of course is independend from ~/src/X or ~/src/Z/X.

No ;-)

Martin Waitz wrote:

We are in agreement about two fundamental parts of the implementation and their meaning:

1. A submodule is stored as a commit id in a tree object.

2. Every object that is reachable from the submodule's commit are 
reachable from the supermodule's repository.

Please confirm.

We now argue about how to work with that repository _object_ model.

If you mean by "owns the whole thing" what I stated above in 2. the we agree.

I think we still agree.

That's it: There is no need for a separate branch or repository. If you have the subproject's commit in the superproject's object database (and we really have that, see 1. and 2. above), why do you _have to_ store it elsewhere?

No. The xdiff submodule would only exist as part of the git repository. You could, f.e., access the xdiff commit in git HEAD as HEAD:xdiff// (again my proposed syntax). HEAD:xdiff//~2:xemit.c would give you the grandparent of xemit.c in the xdiff submodule. And so on. You can even have submodules that have themselves submodules.

See above.

OK.

But only the supermodule's branch.

No.

If you mean the submodule repository created by init-module I understand. But why create this "helper repository at all"?

Sorry, have to leave for home so I must leave that uncommented. Hopefully I can join in during the weekend.

Regards

On Fri, Dec 01, 2006 at 06:33:12PM +0100, Stephan Feder wrote:

Correct.

Let's see if I understand you correctly:

You don't want to create an additional .git directory for the submodule and just handle everything with one toplevel .git repository for the whole project. Without the .git directory, you of course do not have refs/heads inside the submodule.

So this is a different user-interface approach to submodules when compared to my approach. But the basis is the same and both could inter-operate.

Now your submodule is no longer seen as an independent git repository and I think this would cause problems when you want to push/pull between the submodule and its upstream repository. No technical problems, but UI-problems because now your submodule is handled completly different to a "normal" repository.

But you could still call the "xdiff" part of the git repository a submodule. And then changes to the xdiff directory result in a new submodule commit, even when there is no direct reference to it. So you'd still "commit to the xdiff submodule".

git-cat-file commit HEAD:xdiff already works out of the box (even cat-file tree to get the submodule tree). But up to now revision parsing follows the file name only once.

What about just separating things with "/"?

commit HEAD tree HEAD/ blob HEAD/Makefile commit HEAD/xdiff tree HEAD/xdiff/ blob HEAD/xdiff~2/xemit.c

this may add some confusion when used with hierarchical branches, but it's still unique:

	refs/heads/master/xdiff/xemit.c

Just use as many path components until a matching reference is found, then start peeling. Or just use / between super and submodule:

	refs/heads/master:xdiff/xemit.c

I think this is easier to read then

	refs/heads/master:xdiff//:xemit.c

Because it helps "normal" git operations ;-)

Martin Waitz wrote:

Good. For me that is the main point. As I said before the user interface is not so important because it can be changed anytime, but to change the object database later is close to impossible.

...

Yes.

Correct..

Big YES.

You can always pick a single commit or several commits out of a larger repository and have a complete git repository.

And I already explained how to push and pull even from within superprojects.

Yes and no. You can always have branches that are only concerned with submodules' code, say, in refs/heads/submodules/<submodule>/. "submodules" here is simply an example and has not deeper meaning. You could call it foo or whatever you like. Or you could use refs/heads/<submodule>/ if it suits you.

But if you mean the submodule as seen from the supermodule, then there is a difference. Naturally, because the concept of submodules is new to git.

Let's make certain that we understand each other. I see a clear distinction between the submodule code in a supermodule branch (commits in the supermodule's tree and nothing else) and submodule branches which are independent of the superproject. Supermodule branches and submodule branches do not interact, only if I want them to.

The double slashes is the only way I can think of that clearly indicates that I do not mean the contents named by the path, but the commit that you find there. Once you have named a commit in that way, you can continue to apply other revision naming suffixes, paths, and so on.

Let's try. What does git cat-file -p master:dir/sub//^^^:sub/dir/sub//^:dir/file mean?

Explanation: Take branch master and go to path dir/sub. There you will
find a commit. Take its grand-grandparent and go to path sub/dir/sub
(the first sub is a subproject as well but we do not care). There you
will, again, find a commit. Take its parent and go to path dir/file
which happens to be a blob the contents of which you want to cat.

In reality you will never see these kinds of complex paths. Have you ever seen something like git cat-file -p bd2c39f58f915af532b488c5bda753314f0db603~12^{commit}^2^5~8^2~308:README ?

Let's see. I still have to try.

Regards

hoi :)

On Sat, Dec 02, 2006 at 12:34:31AM +0100, sf wrote:

Sure it you are able to make it work, but it needs more work on the UI part. How do you handle the index? How do you allow to clone only the submodule?

I really thought about such a setup too, but then decided that it is much easier to work with submodules when you can really see it as a repository of its own.

Agreed. I think the thing which caused some discussion is that I make the current submodule commit which is used by the supermodule available in a refs/head in the submodule. So there is one "branch" in the submodule which corresponds to the version used by the supermodule, but this is just for user interface. It's most important purpose is to give this special commit a name, so that it can be used in merges, etc.

By selecting another refs/heads "branch" in the submodule you can also easily detach the submodule from the supermodule. It is really important to understand that you can't branch the submodule alone and still have it connected to the supermodule, because the supermodule always tracks only one commit for each submodule. So every branch that affects the project has to be done on project (topmost supermodule) level. But of course the submodule can have other branches which are not tracked by the supermodule. So by checking out refs/heads/master (as it is used in my implementation) you can attach the submodule to the supermodule (attach as in: bring the working directory in sync with the whole project), and you can detach it by selecting another refs/heads (the submodule is still part of the supermodule, but not in the state which is currently visible in the working directory). This may sound confusing, but it really is the only semantic for submodule branches that makes sense. There are fears that you may commit something that does not match your current working directory. Sure, but you explicitly asked for it and I think it won't be a problem if git-status tells about this fact.

With the current semantics, you can already get to the submodule commit (just leave out your double slashes), but what is missing is simply to apply all the modifiers again on this submodule commit. So I think we can do without the double slashes.

On Friday 2006, December 01 17:33, Stephan Feder wrote:

I'm still not convinced about 2. Why should any of the submodule commits be in the supermodule repository? I know that is what you've implemented, but it still feels like too much of a blending of the submodule into the supermodule.

In fact, why should the submodule commits be even visible in the supermodule? That tree->submodule commit is sufficient; there isn't any need to view submodule history in the supermodule.

Andy

hoi :)

On Fri, Dec 01, 2006 at 07:17:17PM +0000, Andy Parkins wrote:

Well, but there is a need for a common object traversal. You need that when sending all objects between two supermodule versions and also when you determine which objects are still reachable.

The easiest way to implement the common object traversal is to have all objects in one object repository.

It may be possible to use two object stores and still do the common object traversal but I do not think that gives you any benefits. You still don't have a totally separated repository then, because you can't do a reachability analysis in the submodule repository alone.

On Friday 2006, December 01 19:38, Martin Waitz wrote:

No you don't; when traversing the supermodule history you will come across trees that have submodule commit hashes in them, that is all the other end needs to know. If it wants it can then connect to the submodule and clone submodule to submodule. The whole operation doesn't have to be done in the supermodule though.

That's true; but is it the right way? I really really think the submodule objects should be in the submodule itself.

There is one benefit - you can git-clone the submodule just as you would if it were not a submodule. In fact, from the submodule's point of view it knows nothing about the supermodule.

I'm going to guess by reachability analysis, you mean that the submodule doesn't know that some of it's commits are referenced by the supermodule. As I suggested elsewhere in the thread, that's easily fixed by making a refs/supermodule/commitXXXX file for each supermodule commit that references as particular submodule commit. Then you can git-prune, git-fsck whenever you want.

Andy

hoi :)

On Fri, Dec 01, 2006 at 09:04:37PM +0000, Andy Parkins wrote:

The submodule repository obviously has to able to reach all its objects. This is easily doable with the shared object database.

So you can already clone the submodule standalone.

I wouldn't call this "easily".

On Friday 2006, December 01 21:37, Martin Waitz wrote:

Of course it is; when you write a supermodule commit you have it's hash, $SUPERMODULE_HASH, you have the commit-hash of the submodule commit you're referencing, $SUBMODULE_HASH. It's not really hard to do

echo $SUBMODULE_HASH > submodule/.git/refs/supermodules/commit$SUPERMODULE_HASH

Is it?

Andy

hoi :)

On Fri, Dec 01, 2006 at 09:54:32PM +0000, Andy Parkins wrote:

I guess you are aware that you have to scan _all_ trees inside _all_ supermodule commits for possible references.

So what do you do with deleted submodules? You wouldn't want them to still sit around in your working directory, but you still have to preserve them.

On Friday 2006, December 01 22:08, Martin Waitz wrote:

No you don't; you do it as part of the appropriate normal operations.

 * supermodule commit - scan the current tree for "link" objects in the
   tree.  If you find one write the reference in the submodule.
 * adding a new submodule - if this is a new submodule there can't be any
   references in the supermodule already.
 * cloning a supermodule, every new commit that gets written in the 
   supermodule gets checked from "link" objects.

Now that is a tricky one. Mind you, I think that problem exists for any implementation. I haven't got a good answer for that.

Andy

On Saturday 02 December 2006 11:04, Andy Parkins wrote:

That suggests that it is probably better to separate submodule repositories from their checked out working trees. Why not put the GITDIRs of the submodules in subdirectories of the supermodules GITDIR instead?

hoi :)

On Sat, Dec 02, 2006 at 02:50:45PM +0100, Josef Weidendorfer wrote:

Why not simply use a shared object database instead?

You can still have an alternative to some standalone bare repository of the submodule if you do not like to store submodule objects in the supermodule repository.

On Saturday 02 December 2006 21:43, Martin Waitz wrote:

Sure. I have no problem with this.

But can we go one step further? AFAICS your submodules store the .git/ directories of submodules directly at submodule position in the working tree - but you have a link .git/objects into the object database of the supermodule. When the user wants to delete the submodule, he would remove this .git/ directory, too. So you loose the .git/refs of the submodule etc. I would suggest to put the submodule .git dirs into the .git dir of the supermodule.

hoi :)

On Sat, Dec 02, 2006 at 10:04:20AM +0000, Andy Parkins wrote:

 * removing a branch from the supermodule.
   OK, this is an infrequent operation and it can be handled by redoing
   everything.

I just don't like to duplicate information which is already available easily. We'd need much to many special cases, just to correctly support reachablility analysis. KISS.

If you just keep it in a shared object repository you don't have any problems.

Please note that it is not required to keep it in one physical location. You can still use alternates/whatever to store some objects in another repository, but you need to be able to access all objects from the supermodule.

Andy Parkins wrote:

I think it would be better resolve this in universal way by adding to git repository layout the optional "borrowers" file, which would protect against pruning objects that are referenced by repositories which have given repository as one of the "alternates".

By the way, how to slurp all the objects from alternates into repo object repository?

Stephan Feder wrote:

It would be much simpler to have subproject's commit in subproject object database, and have it available in superproject's object database by the way of alternates.

Otherwise when commiting new submodule state in supermodule you would have to fetch all the needed objects (submodule mighe have evolved few commits in history inbetween) into superproject's object database.

[cut]

From this discussion I think it follows that supermodule should track HEAD version of submodule. Perhaps the supermodule index should have sha1 of submodule commit, so (as usual) you have to update-index in supermodule to record changes in submodule; the difference being that you update to HEAD version, not to working directory version. Or you can just git-commit -a in supermodule which would take working directory version of files, and HEAD version of submodules.

On Saturday 2006, December 02 11:55, Josef Weidendorfer wrote:

I wasn't really going as deep as a submodule ID. Just moving the submodule commit hash from the supermodule tree, to a supermodule "link" object. What goes in that object is a separate problem I believe.

The primary reason I think it's a good idea is that it is consistent with every other hash in the tree. It seems to be inconsistent to say

blob objects have a hash that points to an object in this repo tree objects have a hash that points to an object in this repo link objects have a hash the points to an object in a different repo

I can't say I've understood what you mean here. There is no difference in facilities if there is a link object in the local repository as well. It's merely an extra layer of indirection. Apart from the tiny cost of dereferencing that link object, there is no disadvantage.

Andy