Re: RFC: Subprojects

Linus Torvalds <torvalds@osdl.org> writes:

There are two positive properties about this setup, and one negative:

 + The contained project is kept totally independent and does
   not have to know it is contained.

 + The tree for the contained project can be rooted anywhere in
   the containing project's tree.

 - The contained project cannot be rooted at the same level or
   higher than the containing project; the containing project
   can only delegate a whole subdirectory to the contained
   project.

The "embedded software" example Simon originally suggested can be represented with the above. I'll think aloud for a while here, because I am of a slow kind who needs a more-or-less concrete illustration to understand what is being discussed (that is primarily why I have not said anything on this topic so far).

The "containing" project would have a handful "gitlink" objects among other things. The toplevel tree object from a commit in such a project might look like this (mode bits 0160000 is S_IFDIR|S_IFLNK, which is what this thing is):

	$ git ls-tree HEAD
        0100644 blob 012345... Makefile
        0100644 blob 123456... README
        0160000 link 234567... gcc-4.0
        0160000 link 345678... linux-2.6
	0040000 tree 456789... src
	$ git cat-file -t 345678
        link
        $ git cat-file link 345678
        commit 87530db5ec7d519c7ba334e414307c5130ae2da8
	url git://...torvalds/linux-2.6.git/

        The upstream Linux 2.6 repository.
	$ cd linux-2.6 && git-rev-parse --verify HEAD
        87530db5ec7d519c7ba334e414307c5130ae2da8

URL will be used as a suggestion for people who cloned this tree to set up their repository. The place and method you clone from Linus tree might be different, so this has to stay suggestion and should be overridable by the repository owner. And to help people at an unusual location you could have textual comment at the end, just like tags.

How would this get set up initially?  Here is one way.

        $ git init-db
        $ edit Makefile README src/*
	$ git clone git://...torvalds/linux-2.6.git/ linux-2.6
        $ git clone git://.../gcc-4.0.git/ gcc-4.0
        $ link=$(echo 'The upstream Linux 2.6 repository.' |
                 git-mklink linux-2.6)
	$ git update-index --add --cacheinfo 0160000 $link linux-2.6
        $ : ;# same for gcc-4.0
        $ git add . ;# add the rest as usual
	$ git commit

I presume that the index file have the "gitlink" object just like in a tree object. The usual merge rules would apply to those index entries; we should be able to treat gitlinks just like we handle symlinks.

Interesting would be "git checkout-index linux-2.6" (or what "git read-tree -u" does in this "containing" project for linux-2.6 subdirectory). After descending into linux-2.6, it should not just do "git reset --hard $commit" for the commit recorded in the gitlink (the user may have local modifications in the subtree). Doing "git update-ref HEAD $commit" there is not quite right either because the index there would then need to be adjusted as well. Perhaps the real core level commands such as "checkout-index" and "read-tree -u" should fail when the subproject tree is dirty, just like "read-tree -m old new" does not always have to succeed.

What does "git-diff-index/git-diff-tree/git-diff-files" would do with them?

	$ git-diff-files linux-2.6

would compare the commit recorded in the link and what is checked out in the linux-2.6/.git/HEAD and report that difference. So do other git-diff-* siblings. At the core level we do not have to recurse and look at linux-2.6/.git/index (we may end up doing so at the end, I dunno; initially we said at the core level we do not have to generate patches but we ended up having -p option go all of git-diff-* siblings).

Fetching/cloning at the core level is easy. "git-fetch-pack" would just need to do one level, but Porcelains need to address how to actually arrange the subprojects cloning to happen, which is harder.

"git clone" would say: "Ah, now I see these gitlinks; we need to clone them. linux-2.6 directory needs to be populated with commit 87530d from git://...torvalds/linux-2.6.git/ repository. Would this work for you, or would you use different mirror?" and then it clones the repository and sets linux-2.6/.git/HEAD to the named commit and does a checkout. The URL used for this actual subcloning would need to be stored somewhere in $GIT_DIR/, perhaps in config as you suggested. I do not think we need a separate name for it -- we can probably say "linux-2.6" for this (i.e. use the pathname itself as the key).

What happens if the containing project wants to move these gitlinks (or remove them)? When checking out such a commit with "git-read-tree -u", would the subproject directory be wiped out (again, such a "read-tree" would be prevented if it would result in information loss)?

All of this sounds quite a lot of change with brittleness.

Now I'll think aloud about a completely different design.

We could simply overlay the projects. I think this is what Johannes suggested earlier.

You keep one branch for each "subproject", and make commits into each branch (i.e. if you modified files for the upstream kernel, the change is committed to the branch for linux-2.6 subproject), but when checking things out, you do an equivalent of octopus merge across subprojects.

One downside of this approach is we cannot re-root the subprojects until we update read-tree and write-tree, but I suspect that would be a lot smaller change. Once that is done, we could:

 $ git init-db
 $ mkdir linux-2.6
 $ H=$(git-fetch-pack -k git://...torvalds/linux-2.6.git/ master)
 $ echo $H >.git/refs/heads/kernel
 $ : ;# same for gcc-4.0
 $ cat .git/config <<EOF
 [core]
 	branchroot = linux-2.6 for kernel
 	branchroot = gcc-4.0 for gcc
 EOF
 $ git add . ;# add src and stuff
 $ git commit ;# commits only the scaffolding into "master"

So far, we fetched the kernel and gcc HEAD with needed objects and stored them into separate branches. Then:

 $ git setup-overlay embed master kernel gcc ;# works like an octopus

The setup-overlay command would create a new branch "embed" to hold an octopus merge across named branches "master", "kernel", and "gcc", and mark that the repository is in a funny "overlay" mode, in which various commands work differently from usual:

 $ edit linux-2.6/CREDITS gcc-4.0/COPYING Makefile
 $ git commit -a

The "commit" needs to be taught to look at what setup-overlay left for us, pick out paths that belong to each constituent branch and do a re-rooting write-tree, for each branch.

This would keep changes to subprojects independent painlessly, but we would also need a way to tie the versions of subprojects together (i.e. "this version of src was done with this particular version of linux-2.6"). This can be done by committing the octopus to "embed" branch. Probably easiest would be to make one commit each to modified constituent branch, and after that make another commit to "embed" to commit the octopus to keep track of the aggregation --- the commit would have the parents set to the previous embed and top commit of each constituent branch.

If we do not need re-rooting (e.g. redo your slurping gitk into git.git), I think all of the above can be done without any core changes. It would be a lot of Porcelainish work, but I suspect the core impact would be smaller.