RFC: Subprojects

Hello,

one thing that I have been missing so far in all SCM systems apart from CVS (and there it's just coincidence) is the ability to include a project as part of a bigger project. Developing software for embedded systems, I need that feature fairly often, for example the source tree for a particular device almost always contains one or more Linux trees, some binutils, gcc and gdb stuff and so on.

The changes necessary here would be fairly simple: "tree" objects would point to a "commit" or a "tag" object when a subproject is used.

In the working directory, this would be represented by a .git directory that contains a symref to the embedding project instead of the objects directory. Head pointers are only required if you intend to push changes upstream to the maintainer of the embedded project. Each subproject has its own index.

Would such a feature make sense, and what behaviour would make the most sense for the various operations (e.g. shall commits in the inner project propagate to the outer?)?

    Simon

Hi,

On Wed, 11 Jan 2006, Simon Richter wrote:

What I do: I call it a branch. While this might seem technically incorrect, it is not.

And since the subprojects are really independent, you can connect them by an octopus.

Sorry, we discussed similar things already. It is not necessary to change the structure. Even more: it makes no sense. Why would you want to have two or more commit messages for the same revision?

Remember: trees, commits and tags (objects in general) are immutable. You 
may think that you just commit a new revision of the subproject, and it is 
picked up by the overall project, but that is not the case!

You can do this like I said: use branches (and possibly a common GIT_OBJECT_DIRECTORY to save on disk space).

Hth, Dscho

Hello,

Johannes Schindelin wrote:

The important thing for me is that I need to be able to transfer them easily, or turn a subdirectory into a subproject or vice versa.

Because the commit affects both the subproject and the master project.

This is why I asked for intended behaviour on commit in a subproject. It is pretty obvious that the master project would need a new tree object to reference the new version of the subproject, and hence, a new commit to keep it all together (and correctly so, since I would like my master project to refer to that particular version of the subproject that is known to work).

Yes, however that wouldn't cover consistency between the subprojects, would it?

    Simon

On Wed, 11 Jan 2006, Simon Richter wrote:

Turning a _snapshot_ of a subproject into a subdirectory is easy: you can literally just create a subdirectory, copy it there, and it will re-use all the objects that the subproject uses (ie the top-level project will have a "tree" entry that just points to the same tree entry as the top-level commit in the sub-project).

However, while that works as a way to import snapshots, it doesn't work in any other way. It allows you to share objects with the "real project", and it's space-efficient etc, but there's no shared history, and you cannot merge back-and-forth, which is probably what you really want to do.

Quite frankly, you really probably want more of a "git-aware symlink" kind of thing. I'd really hesitate (in fact, I'd object) to re-use the existing "tree" type for it, but you're not the only one to have asked for subproject support, so this is clearly not a odd request.

What we _could_ do is for you to first do a commit in the "independent" subproject (it really would be a totally independent git repository in all ways: you could continue to merge it with other subprojects of the same type), and then you could commit a new pointer to that subproject in the master project.

The two would really be fundamentally independent: they'd be two different git projects, one would just have a strange kind of "symlink" to the other, which would include a name and the top commit SHA1 of the other project.

Getting everything to work reasonably seamlessly would be potentially painful (getting "git diff" to recurse into the subdirectory correctly is non-trivial: you'd have a separate ".git/index" file for it), but it sounds doable.

I'd suggest adding a new kind of object ("gitlink") which has some well-specified format (20-byte SHA1 + ASCII C string "name" - the name translation to external repository would be done in the .git/config file of the "outer" project). Then a special file mode to indicate that in the "struct tree", and support for "git-update-cache" to understand how such an object is really tied into the "<pathname>/.git/HEAD" file rather than the rest of the directory contents.

Then a "git fetch" would have to be taught to recursively fetch the other subproject when the "gitlink" changes.

It should be doable: somebody could try to implement a rough first draft (maybe not very seamless at first).

		Linus

Hi,

Linus Torvalds wrote:

Exactly. My proposal is to allow the tree object to point to the toplevel commit object directly, thus importing the entire project[1].

Well, the history cannot be really shared, as turning subtrees to projects and vice versa is a valid use case (and in fact something I do pretty often in my projects, as various "helper" classes evolve into "utility" libraries). Since the subproject needs to be self-contained, the history before it became a separate project will be difficult to represent, to say the least.

[...]

Exactly. The questions I posed in the last paragraph of the initial 
mail, rewritten for clarity, would be
  - "should cg-commit automatically create a commit in the master 
project when a change in the subproject is committed?", and
  - "should cg-commit automatically commit all changes to subprojects 
when a path that has been listed on the command line contains a 
subproject?".

There are three cases, basically:

  - change to subproject, part of a larger set of changes, not ready for 
prime time: A commit in the subproject, master left alone (obviously, 
the directory would show as "modified".
  - change to subproject, fixing a bug that affects the master project. 
You'd expect these to happen often, as I could fix stuff that the master 
doesn't care about in another tree. In this case, you'd want a new 
commit to happen in the master as well, for everyone to enjoy.
  - change to subproject and master that need to go in sync, like 
renaming a configure option. Obviously, this can also happen after an 
update of the subproject, so creating a new commit on the master after 
an update is bad, but normal behaviour for update would be to merge and 
create a commit instantly if there were no conflicts.

Well, that would be exactly what the tree contains. One could do an additional level of indirection with another object that just points to an sha1 (because its name is given by the tree referencing it).

Having such an object would mainly have the advantage of being extensible.

Well, most people would likely not care about the external repository of the subproject, as they can get all the objects from the master project. I'm not even sure the subproject needs an "origin" branch by default, as you can push changes to the master project's maintainers who would then eventually push them on to the subproject's maintainers (in fact I believe it's vital to be able to keep changes to the subproject in the master project so development can go on while the subproject maintainers review the changes.

That sounds pretty simple actually.

I think that would be mostly implicit if we were to use a direct tree->commit reference, but can be implemented trivially even for the link objects.

Indeed. I just wanted to have rough use cases thrown at me before I even think of implementing something like it.

    Simon

[1] There is a slight problem with that approach: When you cherry-pick changes into a subproject (like I did today with the asm-arm/uaccess.h constness fix), the subproject will have a separate branch whose head is known to the outer project only. When the subproject gets merged with its origin later, that branch is no longer needed, and it makes a lot of sense to have the master project reference origin again. This means that if you look at the history of the inner project from the POV of the outer, the new commit is no longer a descendant of the old, but in fact it may be a good idea to attempt a fast-forward merge nevertheless as going through the common ancestor is very likely to cause conflicts here, and those conflicts have already been resolved, or you wouldn't be seeing an updated subproject link (i.e. you just want to preserve local changes and stuff committed on top of the old reference).

On Wed, 11 Jan 2006, Simon Richter wrote:

No.

Don't commit in the "parent" thing automatically. The sub-project should be as independent as possible, and you should see a commit to that to be nothing more than "editing" a regular file in the top-level project.

In many ways, if you decide to look into doing a "gitlink" kind of object, that object really _does_ conceptually point to the .git/HEAD file in the subproject. So when you do a commit in the subproject, conceptually that is no different from editing the .git/HEAD.

Then, when you want to commit the _dependency_ of the top-level project on the sub-project, you commit in the top level. That commit probably does other things too: it probably also commits the code in the top level that now depends on the sub-project changes.

So don't tie the two together any more than necessary. My suggested usage case has the big advantage that the sub-project is much less tightly coupled, so you can do things like "git pull" _inside_ the subproject, to update it, and then do a big compile in the top-level project to reflect the changes (and perhaps update stuff at the top level to conform to changes in the sub-project), and then commit in the top independently (which will now automatically pick up the changes to .git/HEAD that "git pull" did on the subproject).

Again, I'd really suggest not. If you keep the "gitlink" really meaning the ".git/HEAD" contents of the subproject (which is a good semantic rule), then if there is any dirty state in the sub-project, it is totally irrelevant to a "git commit". Because it's dirty, it's not part of of .git/HEAD yet.

Now, obviously you should make "git status" _talk_ about the fact that the sub-project is dirty, so that the committer sees that the sub-project needs committing first (the same way "git status" now informs git commit about dirty files that haven't been updated).

(I'm saying "git" here all the time rather than cg-, because not only don't I know cogito very well, I think you want to do most of the core at the git level, and just teach cg about the new capability).

			Linus

Linus Torvalds <torvalds@osdl.org> writes:

There are two positive properties about this setup, and one negative:

 + The contained project is kept totally independent and does
   not have to know it is contained.

 + The tree for the contained project can be rooted anywhere in
   the containing project's tree.

 - The contained project cannot be rooted at the same level or
   higher than the containing project; the containing project
   can only delegate a whole subdirectory to the contained
   project.

The "embedded software" example Simon originally suggested can be represented with the above. I'll think aloud for a while here, because I am of a slow kind who needs a more-or-less concrete illustration to understand what is being discussed (that is primarily why I have not said anything on this topic so far).

The "containing" project would have a handful "gitlink" objects among other things. The toplevel tree object from a commit in such a project might look like this (mode bits 0160000 is S_IFDIR|S_IFLNK, which is what this thing is):

	$ git ls-tree HEAD
        0100644 blob 012345... Makefile
        0100644 blob 123456... README
        0160000 link 234567... gcc-4.0
        0160000 link 345678... linux-2.6
	0040000 tree 456789... src
	$ git cat-file -t 345678
        link
        $ git cat-file link 345678
        commit 87530db5ec7d519c7ba334e414307c5130ae2da8
	url git://...torvalds/linux-2.6.git/

        The upstream Linux 2.6 repository.
	$ cd linux-2.6 && git-rev-parse --verify HEAD
        87530db5ec7d519c7ba334e414307c5130ae2da8

URL will be used as a suggestion for people who cloned this tree to set up their repository. The place and method you clone from Linus tree might be different, so this has to stay suggestion and should be overridable by the repository owner. And to help people at an unusual location you could have textual comment at the end, just like tags.

How would this get set up initially?  Here is one way.

        $ git init-db
        $ edit Makefile README src/*
	$ git clone git://...torvalds/linux-2.6.git/ linux-2.6
        $ git clone git://.../gcc-4.0.git/ gcc-4.0
        $ link=$(echo 'The upstream Linux 2.6 repository.' |
                 git-mklink linux-2.6)
	$ git update-index --add --cacheinfo 0160000 $link linux-2.6
        $ : ;# same for gcc-4.0
        $ git add . ;# add the rest as usual
	$ git commit

I presume that the index file have the "gitlink" object just like in a tree object. The usual merge rules would apply to those index entries; we should be able to treat gitlinks just like we handle symlinks.

Interesting would be "git checkout-index linux-2.6" (or what "git read-tree -u" does in this "containing" project for linux-2.6 subdirectory). After descending into linux-2.6, it should not just do "git reset --hard $commit" for the commit recorded in the gitlink (the user may have local modifications in the subtree). Doing "git update-ref HEAD $commit" there is not quite right either because the index there would then need to be adjusted as well. Perhaps the real core level commands such as "checkout-index" and "read-tree -u" should fail when the subproject tree is dirty, just like "read-tree -m old new" does not always have to succeed.

What does "git-diff-index/git-diff-tree/git-diff-files" would do with them?

	$ git-diff-files linux-2.6

would compare the commit recorded in the link and what is checked out in the linux-2.6/.git/HEAD and report that difference. So do other git-diff-* siblings. At the core level we do not have to recurse and look at linux-2.6/.git/index (we may end up doing so at the end, I dunno; initially we said at the core level we do not have to generate patches but we ended up having -p option go all of git-diff-* siblings).

Fetching/cloning at the core level is easy. "git-fetch-pack" would just need to do one level, but Porcelains need to address how to actually arrange the subprojects cloning to happen, which is harder.

"git clone" would say: "Ah, now I see these gitlinks; we need to clone them. linux-2.6 directory needs to be populated with commit 87530d from git://...torvalds/linux-2.6.git/ repository. Would this work for you, or would you use different mirror?" and then it clones the repository and sets linux-2.6/.git/HEAD to the named commit and does a checkout. The URL used for this actual subcloning would need to be stored somewhere in $GIT_DIR/, perhaps in config as you suggested. I do not think we need a separate name for it -- we can probably say "linux-2.6" for this (i.e. use the pathname itself as the key).

What happens if the containing project wants to move these gitlinks (or remove them)? When checking out such a commit with "git-read-tree -u", would the subproject directory be wiped out (again, such a "read-tree" would be prevented if it would result in information loss)?

All of this sounds quite a lot of change with brittleness.

Now I'll think aloud about a completely different design.

We could simply overlay the projects. I think this is what Johannes suggested earlier.

You keep one branch for each "subproject", and make commits into each branch (i.e. if you modified files for the upstream kernel, the change is committed to the branch for linux-2.6 subproject), but when checking things out, you do an equivalent of octopus merge across subprojects.

One downside of this approach is we cannot re-root the subprojects until we update read-tree and write-tree, but I suspect that would be a lot smaller change. Once that is done, we could:

 $ git init-db
 $ mkdir linux-2.6
 $ H=$(git-fetch-pack -k git://...torvalds/linux-2.6.git/ master)
 $ echo $H >.git/refs/heads/kernel
 $ : ;# same for gcc-4.0
 $ cat .git/config <<EOF
 [core]
 	branchroot = linux-2.6 for kernel
 	branchroot = gcc-4.0 for gcc
 EOF
 $ git add . ;# add src and stuff
 $ git commit ;# commits only the scaffolding into "master"

So far, we fetched the kernel and gcc HEAD with needed objects and stored them into separate branches. Then:

 $ git setup-overlay embed master kernel gcc ;# works like an octopus

The setup-overlay command would create a new branch "embed" to hold an octopus merge across named branches "master", "kernel", and "gcc", and mark that the repository is in a funny "overlay" mode, in which various commands work differently from usual:

 $ edit linux-2.6/CREDITS gcc-4.0/COPYING Makefile
 $ git commit -a

The "commit" needs to be taught to look at what setup-overlay left for us, pick out paths that belong to each constituent branch and do a re-rooting write-tree, for each branch.

This would keep changes to subprojects independent painlessly, but we would also need a way to tie the versions of subprojects together (i.e. "this version of src was done with this particular version of linux-2.6"). This can be done by committing the octopus to "embed" branch. Probably easiest would be to make one commit each to modified constituent branch, and after that make another commit to "embed" to commit the octopus to keep track of the aggregation --- the commit would have the parents set to the previous embed and top commit of each constituent branch.

If we do not need re-rooting (e.g. redo your slurping gitk into git.git), I think all of the above can be done without any core changes. It would be a lot of Porcelainish work, but I suspect the core impact would be smaller.

On Sat, 14 Jan 2006, Junio C Hamano wrote:

Right.

Yes.

However, I think this is actually a _huge_ advantage.

The thing is, if you do the contained projects as "union projects" as you suggest, I will bet that it will really really suck, because it ends up losing the two positives above.

In particular, any real independent project will have it's own "Makefile" or "configure-in", and often its own "src" subdirectory or other pseudo-standard names.

And the "contained project as a link" approach has zero problems with that at all, exactly because it keeps the projects clearly separate - just linked (one way).

I would actually argue that git itself wouldn't do a whole lot with them. There are real advantages to seeing only the diffs wrt _one_ of the projects, and I'd argue that

	git-diff-*

would actually act like they now act for directories that they don't recurse into, ie you'd see something like

	:160000 160000 5eb57670... 3f1a42aa... M	sub-project

and it would be up to higher-level porcelain to recurse.

Why? Partly because that's actually likely enough for a lot of users: you _can_ use just the raw git programs by just doing

	cd sub-project
	git diff
	..
	git commit

and so technically you aren't really missing a lot. The capabilities are there, you just have to do some more by hand (but in many ways that is _good_: it makes it obvious that you're really committing a _different_ subproject).

The other reason? A lot of the git infrastructure really does only work on the "one project" level. The programs work with _one_ index, not two. Reading two trees is perfectly possible, but unless you keep them in separate stages, you can't separate them afterwards. IOW, trying to be recursive really does end up being a big change, for very little gain (and for a lot of potential bugs and instability).

In contrast, doing it at a higher level means that you have a simple and reliable lower level that you can trust. Layering is good.

Actually, I would say no - that's actually not a "clone" operation so much as a "checkout" operation. There are strong arguments that you should _not_ clone sub-projects when you clone the top-level project: there's no reason to. Anybody else who clones it will have all the information you have, so cloning th esub-project is just extra work.

So only if you actually check it out (which is often in practice the second stage of the cloning, of course) do you want to fetch the subproject too. But even then you might want to ask the user (he may have a local repository for that sub-project somewhere else, so going to the "canonical name" might be the wrong thing to do - and he might not even care, because he might want to work _just_ on the top-level project).

I think this one has serious disadvantages:

 - it's much less obvious when there are common names and especially 
   common subdirectories.
 - in _practice_, almost all sub-projects are kept in sub-directories. Are 
   you doing to change the sub-project git tree? How are you going to 
   merge back to the original sub-project?
 - iow, I think this only works for sub-projects that are totally 
   controlled by the top-level project - in which case they might as well 
   just be totally merged into the top level (the way we did with the 
   "tools" project, and largely with "gitk").

in the "gitk" case, we could actually continue to keep gitk a separate project, but that was really fortunate: it's purely because gitk ends up being a single file, with no Makefile at all to build it independently etc. The moment we integrated the "tools" sub-project into git, we lost the ability to do that, exactly because they now needed to share Makefiles etc, making all further development very inter-twined.

Put another way: the moment you have linkages going both ways between the subproject and the top-level project, it's no longer two separate projects. At that point, it in practice becomes one, since the sub-project can no longer do independent development without merging becoming a big issue.

The advantage of having a "git link" is exactly the fact that the dependency goes only one way. The subproject remains truly independent.

			Linus

So far I've not seen any convincing arguments why the sub-projects can not be managed by the Makefile, or equivalent, of the super-project. Particularly when the sub-projects have a life of their own.

On Sat, 14 Jan 2006, A Large Angry SCM wrote:

Now, from a developer standpoint I actually agree with you. I find sub-projects totally useless - I'm much happier just having separate trees.

The advantage (as far as I can tell) of sub-projects is not that they are easier to develop in, but that it's a total nightmare for the technical _user_ to download ten different projects from ten different sites, and configure them properly and install them in the right order, and keep them up-to-date.

There are projects that I simply gave up even trying to track: I wasn't interested in being a developer per se, but I _was_ interested in trying to test and give feedback to the current development tree - but it was just too damn confusing to get it working.

If I could have just done a "git clone <top-level>" to get it all, I'd have been a much more productive user.

This is why I think sub-projects are more about "git checkout" and an automated "git fetch" than anything else. Doing actual development etc you can easily do one project at a time. "git diff" and "git commit" wouldn't need any real ability to recurse into subprojects and try to make it seamless. And if you do a "git pull" that needs to do anything but fast-forward, you might as well resolve the sub-projects one by one.

		Linus

Linus Torvalds wrote:

$ make get_sub_components

This can work with most any SCM (depending on your environment), is amazingly flexible, and does not require special support in the SCM.

The "get" rule for each sub-project could be something like:

	git_sub-project:
		mkdir sub-project
		cd sub-project
		git-init-db
		git-fetch <fetch-options> <repository> <refspec>
		git-checkout <branch>
		$(MAKE) get_sub_components

And all of this can be done today, without changing git, with more flexibility, with Make rules.

A Large Angry SCM <gitzilla@gmail.com> writes:

I am with you two on this one, in principle, as a developer.

There lies a drake here --- <repository> is not the same for everybody. It is not a big showstopper dragon, though.

On 1/15/06, Junio C Hamano <junkio@cox.net> wrote:

Well, that /little complication/ applies to doing it in git too ;-) There's no way to tell how the dev doing the top level checkout has access to the subproject repos.

I am with gitzilla on this one. Let the projects have their own bootstraping mechanisms, using make, ant or whatever catches their fancy. One of the great things about git is that it doesn't assume that it's being used by all the projects in the world -- thanks to Linus' disregard for arbitrary metadata and to your git-cherry implementation, it's all about the content -- and so it interoperates great with Arch, SVN, CVS, etc.

Having intra-git subproject support assumes that the subprojects are all in git. Heh! That covers about 0.001% of reality out there. Per-project bootstraping scripts will use whatever tools they need for the checkout.

Automating the 'checkout' stage for git subprojects is trivial, and I'd argue not interesting enough to try and solve within git, specially when most subprojects are going to be using a different SCM anyway. And all the *interesting* operations (branch, commit, tag) are perhaps indeed interesting problems to solve, but definite misfeatures in a tool that tries to be sane and minimalistic.

IOWs, adding some repo metadata describing subprojects is the wrong thing to do, just like tracking patches via metadata would be the wrong thing to do. It's all about files -- which git handles masterfully.

cheers,

martin

Martin Langhoff <martin.langhoff@gmail.com> writes:

I had the exactly the same reaction when I saw the project bundling facility of Arch (tla 1.0 -- I do not know what the newer versions use). It probably was a great way to tie two or more Arch projects together, but it would quickly become less useful once the component project is outside Arch space and the toplevel project would end up with doing some Makefile targets like ALASCM described.

I hope this settles this issue and nobody would bring up "Wee want subprojects" ever again ;-).

On Sat, Jan 14, 2006 at 04:49:15PM -0800, Junio C Hamano wrote:

But since we can import everything into a GIT repository, and have (some) tools for pushing changes back, we can pretend that it is being used for every project in the world.

  Tom

On Sun, 15 Jan 2006, Martin Langhoff wrote:

But most of the content of the project that started this thread is the revisions of the subprojects. Sure, it could all be done in the build system, but then it becomes impractical to manage. Git could refuse to support tracking the executable bit on files, or what directories things are in, and we could tell people to use their build systems to set these things, but it would make the tool impractical to use. We want to track some metadata, because it's actually important; what we don't want to track is the metadata that is local to the particular working tree. That's why we track only one executable bit, not a full set of permissions; it's a matter of local policy who can interact with the files in a working tree, but it's part of the content whether a file is executable.

So the problem with handling subprojects with the build system is that it is too tempting to use the revision control system directly on the subproject, at which point the thing you're developing and testing isn't at all what other people will get if they check out your commit. You want "git status" to report it as an uncommitted change if you have a different revision of the subproject than your previous commit had, and it can't tell if this information is buried in the build system.

I like Linus's proposal: which revision of which project goes where is part of the content, while how you manipulate data for that project is a matter of local policy, and is not tracked, although it might be a good idea to let project provide overridable defaults (so that, if you're a random member of the general public and don't have a special method for accessing the repository, you don't have to track it down yourself).

The tricky question is whether we should permit the "subproject" objects to specify a revision that isn't a hash, for use in identifying revisions of subprojects in other systems.

	-Daniel
*This .sig left intentionally blank*

Daniel Barkalow wrote: [...]

Using "git-status" is the wrong tool to use there. What you should be using is "make project_status". Claiming "that it is too tempting to use the revision control system on the subproject" is wrong; you should use the SCM (of the subproject) to manage the subproject. You use the build system to manage the _entire_ project.

I think Linus' proposal is an attempt to solve the problem in the wrong place; it encumbers the SCM with features of limited applicability, that impose a specific methodology on how to handle subprojects, and requires that the SCM of the subproject be Git.

Why would you want to limit how required versions of subprojects are specified? Your project policies and procedures may require that subprojects be specified by a subproject SCM specific immutable revision but the policies and procedures of other projects may not be so restrictive and could accept a tag identifying the latest "stable" (or something) revision.

--

On Mon, 16 Jan 2006, A Large Angry SCM wrote:

I'm talking about using "git status" on the main project, in case you're misunderstanding me. If you can manage the entire project with the build system, then you don't need git or any version control at all, aside from your build system. But you'd also lose the ability to use webgit, bisect, gitk, git log, and so forth on the project as a whole.

If you accept a tag identifying the latest stable revision, then you might as well not bother. The point of revision controlling a project is to be able to reconstruct previous states. If you allow any event, especially outside, unrelated, events to change the reconstructed state for a revision, then this is not the case. Your normal debugging situation will be "It's broken, and I didn't change anything." because someone somewhere else changed something, and you have no record of what last worked. And you can obviously forget any hope of "git bisect" working.

	-Daniel
*This .sig left intentionally blank*

Daniel Barkalow wrote:

When you say "main project", do you mean the top level project and all of its subprojects? Or just the top level project without any of its subprojects?

A build system and a SCM working together can be a powerful very tool, even if one or both of the components is not. Since managing a project with a build system and no SCM means that you don't have any history and since you seem to want access to the project's history, I'll ignore you statement. (Backups and directory snapshots are primitive SCMs.)

Consider the following:

1) For a project to be a sub-project, it must also be a project.

2) So the standard SCM tools will work on the project.

3) The super-project is also a project and the standard SCM tools will 
work on it.

4) Projects managed by an SCM are only considered consistent and usable 
at specific points in that project's history; it may be every recorded 
point in it's history or it may be just a few specific recorded points.

5) A super-project only cares about specific states of its sub-projects, 
corresponding to points in the sub-project's history.

6) For each sub-project, the super-project needs to record the 
sub-project's state identifier for each recorded point in the 
super-project's history.

7) The super-project can record each sub-project's state identifier 
somewhere in the build system.

8) If the super-project's SCM is Git then webgit, bisect, gitk, git log, 
and so forth all work and will identify when and where the recorded 
state identifier of each sub-project is changed.

9) The information is available to the build system to permit using 
git-bisect in the super-project and notice that the breakage occurred 
when the recorded state identifier of a sub-project changed. If the 
sub-project used Git, the build system can automatically start 
git-bisect'ing in the sub-project.

10) The information is available to the build system to permit doing 
similar things for git-log and so forth.

11) Webgit and gitk are a little more work but by creating a git 
repository that contains all of the history and refs of each project in 
the entire project, you can navigate and view the history of any project 
in the entire project.

12) If the SCM of some of the sub-projects is not Git, the build system 
can still do git-bisect, git-log, etc. equivalents for the entire 
project (subject to the limitations of the SCMs involved).

CVS does not have the concept of "hash" for use in identifying the state of a particular set of files but it does have tags and CVS tags work like Git tags. The form of identifier that is used to identify the particular state of interest of a sub-project is dependent on the policies and procedures of the super-project, and the SCM of the sub-project.

The need to reproduce a particular state at sometime in the future is well understood, even in organizations that use tools that only support tags. Implying that it's not possible without using Git's immutable hashes as the state identifier is just wrong.

--

On 1/14/06, Linus Torvalds <torvalds@osdl.org> wrote:

That is exactly how subprojects are used in Perforce- and ClearCase-like SCM: the working tree is "configured" to contain the super-project (build configuration) and the actual work happens in the subproject and _only_ there. The mentioned systems even have heavily used permission system just to prevent either checkout or commit anywhere outside the area of responsibility of a developer. (The "permissions" are somehow pointless in git context, just mentioned them to underline the main point).

Linus Torvalds <torvalds@osdl.org> writes:

After a good night's sleep, I agree. I have not thought things through and still have a feeling that (feasibilities aside) it would be interesting if we can do a "union projects" a la "union mounts" (or translucent filesystem). But that "interesting" thing would probably not be very useful in practice.

This I agree with.

Yup. BTW, I think with a couple of minor tweaking and giving it the same restriction ("two pluses and one negative") as the gitlink proposal, the "union" approach would work equally well, perhaps with a simpler implementation. I'll think aloud about this at the end.

We are in complete agreement here.

Yes, I agree to the above 100%; the serious disadvantages come from the fact that we do not have clear separation between subprojects -- which new files belong to what subproject. I think re-rooting read-tree and write-tree would help solving that. After I wrote the message you are replying to, I came up with a couple of tweaks.

 - Do the octopus-like thing, but always give subprojects a
   separate directories to work in.

 - Extend "commit" objects for the toplevel project to record
   what subprojects with what head commits are contained at
   which subdirectory.  I wrote in the previous message to make
   subprojects heads parents of aggregate commits, but I think
   that one without "where to" information has a serious
   disadvantage when computing a merge.

In the "embedded linux" example that has "linux-2.6" and "gcc-4.0" projects as an externally controlled subprojects, and has all the rest (including the toplevel Makefile) in "master" branch:

     $ tar xf embed.tar embed && cd embed && git init-db
     $ git add . ;# toplevel Makefile and stuff
     $ git commit -a -m 'embedded repo - initial'

After doing "git-fetch-pack -k git://.../linux-2.6.git/ master" and "echo $H >.git/refs/heads/kernel" (similar for gcc-4.0) to set up the branch heads (but we do not have any working tree files for these subprojects yet):

	$ git bind -m 'Bind kernel and gcc into us' \
        	kernel=linux-2.6 gcc=gcc-4.0

would prepare the subprojects binding (I am just looking for a better word --- I called it "setup-overlay" in the previous message). This would:

 - append the tree object in "kernel" commit object to the
   current index, rerooted at linux-2.6/; similar for "gcc" at
   gcc-4.0/. We may need a new mode and option for read-tree for
   this, or we may not.  Internally this step would be scripted
   in "git bind" wrapper like this:

	git read-tree --bind --prefix=linux-2.6 kernel
	git read-tree --bind --prefix=gcc-4.0 gcc

   and would result in an index file that has these trees
   "mounted" at specified places.  If you look at only the index
   file, you cannot tell this is an overlay, unlike gitlink
   scheme.

 - make a commit that records the tree object (the whole thing
   including the subproject trees), with the initial commit we
   made earlier as the sole parent commit, and additionally
   records the two subproject heads with bind points.  This
   happens in the same "git bind" wrapper, and produces
   something like:

	$ git cat-file commit HEAD
        tree e9de76f2e141824439caa00a65e3b91d05d125c9
        parent bfca932434cc65e7aa90794e7c4d66f75d00b16a
        bind a8fe7257b8427d31cfcca0aa336335bb43689fc9 linux-2.6
        bind b3b2df23226634f42c9646bd7961fbea8b00f914 gcc-4.0
        author Junio C Hamano <junkio@cox.net> 1137205528 -0800
        committer Junio C Hamano <junkio@cox.net> 1137205528 -0800

	Bind kernel and gcc into us.

   "bind" line needs to be taught to fsck-objects.  The format
   is the object name of the commit followed by (c-style quoted)
   subdirectory name.

 - record the branch name vs subproject directory binding in
   $GIT_DIR/ somewhere, say $GIT_DIR/mtab ;-).

	$ cat .git/mtab
	kernel	linux-2.6
        gcc	gcc-4.0

After this, "git checkout-index -f -q -u -a" would populate the whole thing. Instead of linux-2.6/.git/HEAD as in gitlink example, I am using .git/refs/heads/kernel; this would not make a semantic difference. One big difference however is I have only one index file that controls the whole tree, without using a separate linux-2.6/.git/index.

After mucking with a file in linux-2.6/ subdirectory and nowhere else, committing the result from the whole tree would work like this:

 - Look at the current commit and notice the bind for two
   subdirectories; then look them up in $GIT_DIR/mtab to see
   which branches keep track of them.

 - Notice that there are modified paths in the index vs tree
   from the last commit under linux-2.6/ directory.

 - Write out only that part, re-rooted, into a tree.

	git write-tree --prefix=linux-2.6

 - Make a commit to record that tree, with a parent set to the
   "kernel" branch head; update the "kernel" branch head at that
   commit.

 - Make another commit to record the tree made from the whole
   index (obviously linux-2.6 subdirectory would result in the
   same tree object we just committed in the subproject) with
   parent set to .git/HEAD and bind adjusted accordingly; update
   the "HEAD".

Now I have to think about clones and merges but this is getting too long so I'll leave it to a separate message.

Continuing with the "union" approach...

Junio C Hamano <junkio@cox.net> writes:

Clarification. By "mounted", I mean 'without affecting existing index entries, create index entries from the tree, with all the paths have "linux-2.6/" prefixed to them'.

I now realize this needs to be something like:

	master	kernel=linux-2.6/ gcc=gcc-4.0/

that is, "when on branch master, bind these two heads at these directories", to allow switching to another branch and switching back to this branch. And the file should probably be called $GIT_DIR/modules, to parallel CVSROOT/modules file.

The core-level cloning would just "clone" the objects, treating "bind" line in the commit just like "parent" to pull necessary objects.

Checkout would involve the usual read-tree -u which extracts the tree (which is the whole tree, with files of the subprojects in it), and notices "bind" lines are there but there are no matching $GIT_DIR/modules entries for those directories. Probably it would create $GIT_DIR/refs/heads/bind/a8fe725 for the linux-2.6 subproject (what the original committer called "kernel" branch), and similarly for the gcc-4.0 subproject, add an appropriate entry to $GIT_DIR/modules file. The user would then rename the branch names and optionally arrange remotes/ files to update the bound branches appropriately:

	$ mv .git/refs/heads/bind/a8fe725 .git/refs/heads/kernel

Now, let's say this "master" branch is checked out, and somehow the "kernel" branch gets updated. That is, the commit recorded on the "bind" line of the HEAD commit does not match the branch head that can be found out via $GIT_DIR/modules file. This will not happen if you are committing into the "master" branch using the "commit to subprojects and then to the toplevel project" mechanism yourself, but it would happen if the "kernel" branch was moved by "git fetch" fast-forwarding, or if you switched to the "kernel" branch (which would essentially remove everything from your tree, and checkout the kernel source at the root level, not in linux-2.6/ subdirectory), did an upstream merge yourself, and switched back to the "master" branch.

To keep the problem simpler, let's say we only deal with the case where "kernel" branch head is a fast-forward of what is on "bind" in the HEAD commit of "master" branch. Then "checkout" needs to notice it, and check out the subdirectory from the "kernel" branch head (*not* using the object name on "bind" line).

So the outline of the "checkout" would be like this:

 * Read commit object from new HEAD.

 * For each "bind" line:

   If the subdirectory does not have a corresponding branch,
   create one in $GIT_DIR/refs/heads/bind/; record it in
   $GIT_DIR/modules for the new branch (otherwise leave branch
   as is).

   Make sure the commit recorded on "bind" line is an ancestor
   of the branch head.  Otherwise it is an error and checkout is
   prevented until the "kernel" branch is resolved to be a
   descendant of it.

   Run "read-tree -u --prefix=" to merge in the subtree into the
   index, and update the working tree.

At this point, there may be mismatch between the tree in the HEAD and the working tree files and index, when subproject commit recorded on the "bind" line is different from the corresponding subproject branch head, and "git diff" would show it. When making a commit here, the "subproject and then toplevel" commit scheme I described earlier would record the current "kernel" branch head on the "bind" line in the new commit, along with the tree object that contains the tree from "kernel" branch head commit as a subtree.

About "merge", we should be able to do this:

	$ git checkout master ;# the whole mess
        $ git pull -b kernel git://..torvalds/linux-2.6.git/

that is, 'pull from this URL but into "kernel" branch not to the current branch'. Independent of this "subprojects" topic, merging in a separate temporary directory into non-current branch is something we have talked about some time ago, and in this particular case, instead of using a throw-away temporary directory, we have a pre-made directory to do the merge already, so let's say that is solved elsewhere first. Once we have that, the above "checkout" would be able to integrate the result into the "master" project.

On Saturday 14 January 2006 21:16, you wrote:

The suggested "bind" info in commit objects has the same problem as the original overlay: if the superproject already has a subdirectory kernel/, and there is an additional "bind" specification in commits also for kernel/, what should be done?

So the gitlink object seems to be the only solution if we want to bind git versions of subprojects into a superproject.

But as this seems to make everything quite complex and not-obvious for a user, I am with Paskys simple subproject idea.

Josef

Josef Weidendorfer <Josef.Weidendorfer@gmx.de> writes:

In "pu", I have some of the necessary basic pieces for "bind" approach, barely enough so that anybody interested could start prototyping using them as building blocks. It still has very rough edges; the missing includes rev-list and fsck-objects, so you cannot do a send-pack or fetch-pack yet.

Yesterday I was working on "gitlink" approach to have similar core-side support for prototyping. I haven't finished it into a buildable state yet (it is not in "pu"), and I am pessimistic if I ever will X-<.

I think the updated "bind" thing makes the two approaches semantically equivalent (i.e. it does not allow an arbitrary overlayed setup anymore). We simply do not allow the conflicting "bind". So neither is the _only_ solution. We probably could make both to work, but the details differ.

 * With "gitlink", the index of containing project never has
   subprojects parts of the tree, which I see it as an advantage
   compared to what "bind" does.  It only has one "gitlink"
   entry per each subproject.  update-index, read-tree,
   ls-files, diff-*, etc. needs to be aware of "gitlink" object.
   Especially tricky is read-tree.  It needs to treat a
   "gitlink" object as a directory for D/F conflict detection
   purposes, but treat it similar to blobs in most other aspects
   (e.g.  results in one entry in the index).  The stat
   information update-index and diff-files uses for quick
   up-to-date check needs to be taught not to worry about the
   stat information of the subdirectory a "gitlink" object
   points at (e.g. if you do a whole-tree build, the timestamp
   of the directory would change, but that does not mean the
   subtree is dirty).  tree/directory traversal code needs to be
   aware of "gitlink" and stop there.  This approach involves
   quite a lot of code changes, mostly because what is in the
   current index never correspond to a directory on the
   filesystem but "gitlink" quacks like a directory.

 * With "bind", the index of containing project keeps the entire
   tree structure, including subproject part.  In fact, there is
   no other separate index for the subproject part.

   An updated write-tree in "pu" can write a tree for only the
   subproject part with "write-tree --prefix=<path>/" from such
   an index file, and read-tree can read with "read-tree
   --prefix=<path>/" to graft a subproject tree on top of the
   current index contents.  Without the --prefix, write-tree
   writes out the whole thing for a commit for the containing
   project, so if somebody cloned that superproject, getting the
   whole tree out in order to "make" is just the matter of doing
   a regular "read-tree && checkout-index".

   We could introduce "bind the rest" to make write-tree write
   out a tree that contains only the containing project part and
   not any of the subproject part (e.g. Makefile, README and
   src/ but not linux-2.6/ nor gcc-4.0/ in the earlier example).
   Essentially the contents of such a tree object would be the
   same as what "gitlink" approach would have had for the
   containing project in the index file, minus "gitlink" entries
   themselves).  This is not so surprising, because the missing
   information "gitlink" approach recorded in the tree object
   itself is expressed on "bind" lines in the commit object with
   this approach.

An advantage with the "bind" approach, from the implementation point of view, is that none of the "index vs working tree" part of the core needs to be modified (you would notice that many issues I had with trying "gitlink" I listed above are "index vs working tree" issues). "tree object vs index" part needed to be enhanced somewhat (e.g. the re-rooting read-tree/write-tree with the --prefix option) but it was not too painful.

On Mon, 16 Jan 2006, Junio C Hamano wrote:

So why not use the "bind" approach for the "index vs working tree" part, but write out "gitlink"-style tree objects? I think putting the info in the tree objects in the location the subproject would appear is nicer than having tree objects that tell only part of the story, and you don't have to worry about commits that stick a subproject on top of something in the tree.

In any case, I think it would be good to track where the subprojects are in some core state, and probably the right solution is to have special index entries for them, in addition to having their contents in the index. I'm not seeing a clear way to get from commit objects with "bind" lines to an index with the appropriate things read and back otherwise.

One idea I toyed with a while ago for the index/working tree implementation is having an index file per bound project, such that each project has a completely ordinary index file, and you just need to tell checkout-index where to write. This is especially cute because the index file for the superproject doesn't need to know about the subprojects at all; they're not in that index, and the working tree is just directories of untracked files. Not sure if this is a useful idea at this point or not.

	-Daniel
*This .sig left intentionally blank*

Daniel Barkalow <barkalow@iabervon.org> writes:

I said "index vs working tree" as a mere example, and never said "gitlink" is easier (or at least as easy as "bind") for "tree object vs index" or "tree object vs working tree through index". In fact I suspect those parts also need to be changed fairly heavily, and to be honest, I am not very much looking forward to investigating the details.

Actually, the "special entry" was what I found out to be quite a pain, if you mean to have "linux-2.6/" in the index and have it used in some meaningful way. Further hacking and prototyping _might_ convince me otherwise, but I am not so optimistic at this moment.

Here again I am thinking aloud, remembering the earlier example of an embedded linux project that ships with linux-2.6 and gcc-4.0, along with its own README and Makefile at the toplevel and src/ for its own sources. The tools at the tip of "pu" should be able to let you do the following:

	$ git cat-file commit $such_toplevel_commit
	tree $tree
        parent $parent
        bind $primarysub /
        bind $linuxsub linux-2.6/
        bind $gccsub gcc-4.0/
	author A U Thor <author@example.com> 1137392543 -0800
	commmitter A U Thor <author@example.com> 1137392543 -0800

        An example.

where $tree is the object name of the whole tree (no "gitlink" object), $primarysub and $linuxsub are the object names of commit objects for the primary subproject (which sits at the rootlevel) and another subproject (which sits at linux-2.6/ subdirectory).

To make sure there is no misunderstanding:

	* "git-ls-tree $tree" would show the object name of
          $linuxsub^{tree} at path "linux-2.6/" because
          "tree" line of a commit describes the whole tree,
          including subprojects.

	* "git-ls-tree $primarysub" would show README,
          Makefile and src/ directories but not linux-2.6/ nor
          gcc-4.0/.

	* "git-ls-tree $linuxsub" would show COPYING, Makefile
          etc., not linux-2.6/COPYING.

Reading such a commit is easy:

	$ git-read-tree $tree ;# ;-)

But that is cheating.  Constructing such an index can be done by:

	$ git-read-tree $primarysub
        $ git-read-tree --prefix=linux-2.6/ $linuxsub
        $ git-read-tree --prefix=gcc-4.0/ $gccsub

When you have such an index, writing out various trees are:

	$ git-write-tree ;# $tree
	$ git-write-tree --prefix=linux-2.6/ ;# $linuxsub^{tree}
	$ git-write-tree --prefix=gcc-4.0/ ;# $gccsub^{tree}
	$ git-write-tree \
          --bound=linux-2.6/ --bound=gcc-4.0/ ;# $primarysub^{tree}

The decision to use what --prefix and --bound and what tree(s) to write out must come from somewhere, and as you say it would be nice if we _could_ stick them in the index as "special entries", but for the purpose of prototyping I am assuming I keep that somewhere in $GIT_DIR/ (the "mtab" in the previous message. Maybe "$GIT_DIR/bind" is a good name?).

Dear diary, on Tue, Jan 17, 2006 at 07:18:47AM CET, I got a letter where Junio C Hamano <junkio@cox.net> said that...

I perhaps missed this in the thread, but is it really so useful to bind the subprojects to specific commits? If you care about reproducing specific configuration, all you have to do is tag and seek recursively - and even having a separate tiny git branch tracking just a single file listing the commit ids of subprojects seems more elegant to me than just forcing the specific commit ids. In the general case, I think it most usually goes "this project#branch, the latest commit you can get", so I'm not really convinced that you are optimizing for the right case at all.

On Tue, 17 Jan 2006, Petr Baudis wrote:

Think from a debugging standpoint. You know that the main project worked with a particular commit of the superproject. The bug you've found is related to the behavior of one of the subprojects in the the context of your superproject, but you don't know this. In order to reproduce the working version and search for the change that broken things, you need to be able to identify which commits of subprojects were used in each commit of the superproject; these are almost certainly not the latest commits on any branch of the subproject. And if you're going to want to debug things later, no commit of the superproject can just say to use the latest in the subproject. You don't know what you make a commit whether it will turn out to be a configuration that you'll want to recreate later.

Now it may be useful to have a tool to update all of the subprojects to the latest versions, similar in end-user usage to pulling repositories, but you still need to generate new commits when you do this, rather than reinterpreting the old commits with new content, so that you keep the history immutable. You also want to know when you've done this, so that you don't clone a tree that's working fine and build it only to find that the clone has fetched a new kernel version with different behavior without letting you know that anything has changed.

	-Daniel
*This .sig left intentionally blank*

Hi

For reference, here is what subversion does:

http://svnbook.red-bean.com/nightly/en/svn.advanced.externals.html

--Craig

On Tue, 17 Jan 2006, Daniel Barkalow wrote:

Yes, there are real advantages to being able to tag a very specific version of a tree.

You can do it manually (ie tag the versions of everything used), but there's a real convenience to being able to say "I want the tree to look exactly as it looked for our internal test-release that we shipped as a pre-view to customer so-and-so".

You can do it with ad-hoc build rules inside a company, but the likelihood that they don't work all the time is pretty high. Somebody forgot to follow the right procedure, and had updated a sub-tree without marking it, and now you can't reproduce the problem that a customer has with a debug build, because you have no way to reproduce the exact binary...

It's why people tag every file for huge trees under CVS for a release, and accept why building a release may take hours. It's crazy, yes, but there are other projects than just the BSD's that have that "World" mentality, where they want every single program under _one_ umbrella, so that they can tag them all together.

Me, I think it's crazy engineering ("if you can't reproduce it with individual projects, you're not doing programming, you're doing Voodoo"), but it's something that some organizations simply require.

Now, it might be enough with a cogito approach of ".git/subprojects", and just _version-control_ it in the top-level project, but then you'd need to make sure that all the tools automatically update the version when they do a "pull" or a "commit" on a subproject. But then it almost boils down to "gitlink"s after all.

		Linus

On Mon, 16 Jan 2006, Junio C Hamano wrote:

I suspect that it's just as easy, except that you get confronted immediately with the issues that you haven't dealt with in the bind approach (mentioned below). If you had parse_tree_buffer() just ignore them, and had write-tree take a list of bind lines, that would match the status of your "bind" implementation, I think (except for the part you say is cheating).

Incidentally, I don't think we'd want "gitlink" objects with the "gitlink" approach; we'd want trees to contain commit objects for subprojects. The "gitlink" thing that corresponds to ".git/HEAD" isn't an object, it's a tree entry, which, like ".git/HEAD" (or, more appropriately, ".git/refs/heads/something") maps a name to the hash of a commit object.

Hmm... maybe libification should go ahead of subprojects. If access to the index weren't so often open-coded, it would just be a matter of having these entries in the data structure, but not actually returned by any current call, and it would be just like they were in some other structure.

Actually, it should be easy to have them in the index file but not in the main index data structure, by skipping over them in the for loop near the end of read_cache(). Put them in a separate structure, and write them back to the file in write_cache(), and have a different method entirely for changing them, and they shouldn't affect the normal use of the index.

Side issue here: this implies that the kernel objects are in the superproject's repository, or at least accessible from it. So prune has to not remove them. So, if you've committed changes to a subproject but not yet committed the fact that you want to use the changed subproject into the superproject, fsck-objects has to find them somewhere.

This is for backwards compatibility, I assume?

The hard thing here is getting the commits for the trees. The bind lines need commits, which means either identifying that we already have the correct commit object, because we didn't change anything in the subproject, or generating a new commit object with some message and the right parent. And we want to use commit objects, not tree objects, in the bind lines, so that, once we track a problem to the change of which commit is bound, we can treat the subproject as a project and debug it with bisect, rather than just having one tree that works and one that doesn't.

	-Daniel
*This .sig left intentionally blank*

Daniel Barkalow <barkalow@iabervon.org> writes:

And libification has been waiting for the core to settle ;-) We have to start somewhere.

Yeah, I guess I was vaguely thinking along those lines while I was driving to work this morning. I appreciate your spelling it out to make things clearer.

Yes.  I was planning to have "$GIT_DIR/bind" that says:

	master kernel=linux-2.6/ gcc=gcc-4.0/

meaning:

	The project kept track by "master" branch binds the
	project kept track by "kernel" branch as its subproject
	at its linux-2.6/ subdirectory.

or something like that, so when you make a commit, you update those other branches as needed. You already raised that issue at the end of your message, and I will explain how I think that can/should be done as a response to that part later.

This is done more for not having to touch *anything* that does "index vs working file", "tree vs index" and "tree vs working file via index". It also is the easiest way to keep the "a commit object name can be used in place of the tree object name of the tree it contains" invariant. Also I suspect this organization might help recursive subprojects, but if it is the case, that is just a byproduct, not a design goal.

Your wording "get the commit" is a bit misleading. Even when the tree for a subproject happens to match a commit in the subproject in a distant past, we would not want to use it unless the user explicitly asked for it. IOW, we do not actively go and look for a commit.

Our subproject tree either matches the subproject branch head, in which case we just reuse it, or we make a new commit on top of that ourselves.

Let's say my project breaks with the latest kernel, and I suspect that it would work with v2.6.13 sources. To test that theory, I could:

        $ git branch -f kernel v2.6.13 ;# rewind

	$ git ls-files linux-2.6/ |
          xargs git update-index --force-remove
        $ git read-tree --prefix=linux-2.6/ -u kernel

to construct such a tree. Maybe the latter two-command sequence "ls-files & read-tree --prefix" sequence deserves to become a command, "git update-subproject kernel" [*1*].

The result may work as-is, or I may need to do some further futzing in linux-2.6/ directory before the result works. Once the result starts working, I'd want to make a commit:

 - I compare the result of write-tree for linux-2.6/ portion and
   the tree object name contained in the head commit of the
   "kernel" branch.  If they match, then the current "kernel"
   branch head commit is what I'll place on the "bind" line in
   my commit; I do not have to make a new commit in the "kernel"
   subproject in this case.

 - If the tree object does not match the "kernel" head, that
   means I have tweaked the kernel part further, on top of
   v2.6.13.  So I make a commit for the kernel subproject (whose
   parent is obviously v2.6.13), update the kernel branch head
   with that commit, and then record that tip-of-the-tree commit
   for the subproject on the "bind" line in my commit for the
   toplevel.

Or let's say my project builds with the latest kernel (IOW, I did not do the branch -f kernel in the above), and I made some custom tweaks in the kernel area. The above precedure would result in a new commit on top of the latest kernel, update the "kernel" branch head, and make a commit for the toplevel that records the updated "kernel" branch head on its "bind" line.

Note that the above procedure did not use the commit object name recorded on the "bind" line at all in either case. From the mechanism point of view, it is the right thing to do. From the usability point of view, however, we may want to take notice that "bind" line commit and the bound branch head do not match, and remind/warn the user about it. If the reason why they are different is because the user rewound the bound branch to use a known working version, or made fixes in the subproject and pulled the result into the bound branch (in which case there is no funny rewinding involved), then this warning is extraneous. But in the normal case of keep reusing the same vintage of subprojects (and maybe making necessary adjustments to subprojects while working on the main project), the commit object on the "bind" line of the HEAD commit and bound branch head should match.

[Footnote]

*1* One could also do a forward development on the kernel branch in a separate working tree and fetch from there. For example, if our example "superproject" is in embed/ directory, and there is a linux/ directory next to it to house a kernel repository, we could:

        $ cd ../linux/
        $ edit && compile && test 
        $ git commit -m 'Fix for upstream, not just for embed'

to make an upstream fix, and then:

        $ cd ../embed/
        $ git fetch ../linux/ master:kernel

to update the "kernel" subproject branch head.  In such a case:

	$ git update-subproject kernel

would bring the subproject working tree and index up to date with respect to the updated kernel branch.

Junio C Hamano <junkio@cox.net> writes:

I started this "bind" design as a thought experiment, but I started to like it more and more.

One interesting outcome of keeping the whole tree in the index and the tree object recorded in the commit object of the toplevel project is that a merge in the toplevel project "just works".

To preserve our sanity, let's say we refuse to merge two commits that have different sets of subprojects. That is, they must have the "bind" lines for the same set of subdirectories. The commits bound at these subdirectories do not need to match.

Before starting a merge, we require that the index is in sync with the tree object recorded in the top commit, just like we do for a normal merge[*1*]. Then we use the current merge machinery that does not know anything about "bind" to perform the merge, using the merge base of the toplevel project and usual three-way merge. From the mechanism point of view, there is no need to look at commits on "bind" line of either side to come up with the resulting tree.

We could notice that the commit bound at linux-2.6/ subdirectory of one side is v2.6.15 and the other side is v2.6.16-rc1, and because one is a fast-forward of the other, choose to pick the tree associated with v2.6.16-rc1 commit without actually doing the 3-way resolve of linux-2.6/ subtree part, but that is purely a performance optimization [*2*].

When writing out the merge result as a commit, we would create (this is the fun part) a commit for linux-2.6/ part that has two parents: the commits bound to linux-2.6/ tree from the two toplevel commits being merged are the parents of such a subproject commit. And the resulting toplevel merge commit would have that commit object name on its "bind" line. Obviously, when the bound subproject head of one side is a fast-forwad of the other, we do not create such a merge commit for the subproject; instead, we just record the one that is ahead on the "bind" line of the resulting toplevel merge commit.

[Footnote]

*1* As a side effect, this also ensures the index is in sync with the bound commits of the subprojects. As an additional requirement, we may want to enforce that the bound commits must match the branch heads that keep track of subprojects.

*2* Of course, from the usability, safety and confusion avoidance point of view, it _might_ make sense to require that bound commits are in such fast-forward relationship. But that is a policy issue; at the mechanism level, there is no need to impose such a requirement.

Is there a version of git with this to try it ?

Alexander Litvinov wrote:

The pu branch of the official git repo.

Alexander Litvinov <lan@ac-sw.com> writes:

	http://article.gmane.org/gmane.comp.version-control.git/14760

As I warned there, it still has quite rough edges, so do not use it on your production repositories.

On Tue, 17 Jan 2006, Junio C Hamano wrote:

Well, we could do a pass at cleaning, organizing, and documenting the internals, which is sort of the start to each of them.

Okay, so you're using additional branch heads in the superproject to track the current state of the subprojects. That makes sense, although I think it would confuse people less if they were held separately. IIRC, refs/subprojects/kernel/heads/master is a perfectly good ref name these days, so that might be a good idea. That would also mean that refs/tags/v2.6.14 and refs/tags/v2.7.2.3 wouldn't get confused (being linux and gcc tags, respectively), because they'd be under the appropriate subprojects.

I assume these get updated by checkout when you check out the commit with them as bind lines?

Ah, okay, so it's cheating for checkout, because checkout is supposed to understand everything, but not cheating for other things. I thought we decided that the stuff that doesn't know about subprojects sees them as opaque, rather than as their contents, so your toplevel git diff doesn't show you a millions lines when you switch from linux-2.6.14 to 15.

We don't search the history for just any commit, but we have to look somewhere...

I hadn't realized that the subprojects had branch heads. It makes much more sense that you'd expect to be able to just write out bind lines if you've got that information.

I thought we decided that committing the superproject wouldn't commit the subprojects. If this is what we decided, then the subproject tree is required to match the branch head, because we must have committed the subproject already (which is good, because otherwise the user will get confused about which commit message to write when).

Shouldn't "git read-tree --prefix=linux-2.6/ -u kernel" remove everything else in the index in linux-2.6 itself, making the "git update-index --force-remove" unnecessary?

Equivalently, you make a commit for the kernel subproject, update the branch head, and start over; the first case should apply now.

I hope people will want to prepare their commits to the kernel subproject as would be suitable for pushing to Linus, which would suggest that they'd tend to do a commit in the kernel subproject embedded in their superproject separately from doing the commit in the superproject, and so the branch head would match the index but not the bind line when they got to committing the superproject.

	-Daniel
*This .sig left intentionally blank*

Daniel Barkalow <barkalow@iabervon.org> writes:

I did not think of that when I wrote that message, but you are right.

Yeah; to put it another way, read-tree of the toplevel tree and checking it out is equivalent to do the skelton read-tree followed by --prefix read-tree of all the bound projects, so checkout can optimize.

It was discussed in the context of "gitlink" approach as a way to keep things simple. In the "bind" approach, I am doing things a bit differently, and this "toplevel has everything" is one big difference.

I see it as a policy. We can forbid the modification of the subproject part of the index (i.e. detect and refuse to commit and/or do "git reset --mixed" only for the subproject part) so that the commit outlined in the "bind" approach does not _have_ to make a new commit, if you want to work that way. But if somebody else wants to make a related set of changes to the superproject and bound subprojects, we _could_ allow a commit per subproject.

I agree that "-u" should imply that. The current "read-tree --prefix=linux-2.6/" in proposed updates refuses if linux-2.6/ appears in the original index.

Yes, that is the workflow I outlined in the footnote part you did not quote. I think it is cleaner to do things that way: to have a separate, kernel-only repository+worktree and do pure kernel work there, and fetch into the superproject branch that keeps track of the kernel subproject in that superproject.

Having more than one working tree with .git/, everything except HEAD and index undef which are symlinked to one copy, like you do, would be a natural way to work.

	embed/.git/HEAD -> refs/heads/master

	embed/linux-2.6/.git/HEAD -> refs/heads/kernel
	embed/linux-2.6/.git/refs -> ../.git/refs
	embed/linux-2.6/.git/objects -> ../.git/objects

Then, after hacking on the collective whole to make the whole thing work in "embed" directory, you would:

	$ cd linux-2.6
        $ git commit

to make commit that can be sent Linus, at the same time updating the "kernel" branch. Then come back to the toplevel, tell git that you updated the "kernel" branch so it does not complain that the "bind" in the HEAD commit does not match "kernel" head, and make a toplevel commit.

On Wed, 18 Jan 2006, Junio C Hamano wrote:

I thought that had been a question of what is best as an interface, but either is plausible.

I think it makes most sense, for the purpose of consolidating code paths, if the superproject may only be committed with clean subprojects; the porcelain has the option of responding to unclean subprojects by committing them to make them clean, and then there is only a single case for how the superproject commit happens. I think it makes most sense as a command line option, like -a is; if you want to commit dirty suprojects, you use --subprojects, and it does that. If you're not expecting to need it, you won't start doing the wrong commit.

I actually meant that I expected people to go into superproject/linux-2.6, make changes, and commit there, using the place it appears in their superproject working tree as a working tree for the subproject, so the opposite of your footnote, but still doing the subproject commit as a step before the superproject commit.

For example, they might want to send the subproject changes upstream as a patch, get feedback, reset the subproject, do revised changes, commit that, get it merged upstream, and then commit the changes to the superproject, including in the message the fact that the changes have been pushed upstream. But they may still want to do this all within the working tree of the superproject, so that they can test their changes in context.

I'm not sure having a .git directory for a subproject inside a subdirectory of the superproejct's working tree is all that good an idea, and I don't think it should be necessary in any case, because the toplevel index has all the information from the subproject index. The only think would be having "git commit" notice what you're doing when you run it from a directory that's a subproject.

	-Daniel
*This .sig left intentionally blank*

Dear diary, on Wed, Jan 18, 2006 at 07:21:59PM CET, I got a letter where Daniel Barkalow <barkalow@iabervon.org> said that...

I passionately agree - this is the only thing I do not like on the current Junio's proposal (besides that top-level subproject confusion). The way it is proposed, you are mixing different projects in a single refs namespace and I think that's *really* confusing.

Besides, you are going to get a lot of complications since to do merging properly you need two heads per subproject (its 'master' and 'origin' heads; and it's useful to have e.g. all the upstream heads called 'origin' since then you can say cg-fetch -r origin in the superproject and have all the subproject origins fetched as well) and you might want to have other subproject heads as well. Now, for different superproject heads, you want separate set of subproject heads. You can see the downward spiral from here, I guess... And multiply all that by two since you also have tags.

It actually took me a short while to realize that keeping separate subproject/.git/refs makes no sense precisely because for different superproject heads, you want a different set of subproject refs. So in line with Daniel's proposal, I'd propose:

	refs/subprojects/<superhead>/<subid>/heads/master

<superhead> is the name of the current HEAD (${#refs/heads/}). <subid> is a little more tricky - this should be the part after the equal sign in .git/mtab (or .git/binds or .git/subprojects or whichever is the name of the day). Obviously, you can just figure out something, but I'd like to assign this automagically.

OTOH, in Cogito I might as well just default to sha1 of something random (e.g. the path+commitid+time()) since I do not expect this to be normally referenced by a human; I just intend to switch from refs/ to refs/subprojects/<superhead>/<subid>/ when dealing with the subproject exclusively. ($GIT_REF_DIR (by default $GIT_DIR/refs) would come useful; I'll probably whip up a patch when I get to finally need it.)

FWIW, my idea is that it should be "a seamless experience for the user" (tm) to do development in a subproject of another project, and I can see no reason why should that be hard to do in any way.

Dear diary, on Mon, Jan 16, 2006 at 09:49:48PM CET, I got a letter where Junio C Hamano <junkio@cox.net> said that...

Now, I must have missed the obvious again, but what is the point in having the write-tree --exclude stuff? My impression (also from your later mail in this thread) is that now the moment you introduce any binds, your top-level development changes to "two-tiered" - the top-level project and the meta-project holding it all together. I'd say that's pretty confusing and I don't see big gain in this; the simplicity of the original proposal was a lot more appealing.

On Saturday 14 January 2006 14:59, Junio C Hamano wrote:

If I cleary understand this idea it is NOT that I dreaming about. Almost all our sub-projects are used in more than one project (imaging network layer library). So variant with gitlink is that I willing.

Alexander Litvinov wrote:

Then it isn't so much a subproject as a separate project of its own. Otherwise glibc would be a subproject of pretty much everything and that's hardly a sane setup.

Hello,

Junio C Hamano wrote:

I'd prefer to have the objects needed to get the linux-2.6 tree in the object db of the containing project. Then "url" is not needed, and you could directly use the commit as value for the link. i.e.

       $ git ls-tree HEAD
         0100644 blob 012345... Makefile
         0100644 blob 123456... README
         0160000 link 435363... gcc-4.0
         0160000 link 87530d... linux-2.6
         0040000 tree 456789... src

(You could now rename "link" to "commit", but it would break the layout.)

Moreover I prefer the the link approach over the bind method. The reason is, that binds use information from the commit object to build the wc other than the tree. Moreover the condition that the "containing" tree must not have an entry named linux-2.6 is handled implicitly with links.

Please correct me if I'm wrong somewhere. It's some time ago I read the patches and this thread. This mail is the result of some thoughts in my vacation.

Best regards Uwe

Uwe Zeisberger <zeisberg@informatik.uni-freiburg.de> writes:

... which is actually closer to what bind commit approach gives you. The tree object in a commit of the containing project has the full tree object at path linux-2.6/. The "bind" lines in the commit object are just notes that tell you where those trees happen to came from.

I had an impression that two approaches were more or less equivalent, especially the last round of bound commit approach. It does not let anything to exist at the bound path in the containing project either ("read-tree --prefix" rejects it).

I have to admit that I haven't thought about the issues involved for a long time, having no great need nor desire for subprojects myself, and especially with more generally useful stuff like performance enhancement for pack generation to occupy me. I am not sure I am much more qualified to comment than you are at this point.

The bound commit lowlevel changes have been sitting in "pu" for about a month by now, but nobody seems to be interested enough to start prototyping Porcelain around it. Neither the gitlink approach. After seeing not much interest on the list, I was hoping that I could retire both WIPs.

On Wednesday 11 January 2006 21:58, Simon Richter wrote:

I really miss this feature. This is the last stopper for moving from CVS to git for out project.

On 1/12/06, Alexander Litvinov <lan@ac-sw.com> wrote:

What about using nested checkouts? They work great with git as-is, just add an .gitignore file.

As Linus points out, there are many good reasons why a top-level commit should _not_ commit the nested subproject. And once you are observing that rule, what's left then? git status and git diff <HEAD> can show an aggregate of top-level and nested subprojects, but that's ease-of-use -- not something only.

What is your show stopper?

cheers,

martin

On Thursday 12 January 2006 10:46, Martin Langhoff wrote:

I would agree to make separate commits for each sub project.

1. I need to have ability to make tags, branches thru all subprojects.
2. Update (pull) sould update each subproject, it is hard to update them by 
hands.
3. The need of some sort of checkout script (can be solved by storing this 
script in base project, but it would be much nicer allow git fetch all 
subprojects)

Nothing else I can imagine.

On 1/12/06, Alexander Litvinov <lan@ac-sw.com> wrote:

I suspect that this is a bad idea -- for the same reason as committing to a subproject is a bad idea. The subprojects most likely have their own external repositories -- and lifecycles of their own. The same headname/branchname won't do.

A simple shellscript can help you here.

As you say, a bootstrapping shellscript can sort this out.

Sounds quite doable ;-)

(have to warn you though -- git is quite addictive. there's no going back...)

cheers,

martin

This is one main idea of supporing subprojects. Everything else I already can do. I want to be able to make tag over composite project and be able to fetch tagged files later. The same with branches.

I cleary understand if I made tag/branch on subproject outside my composite project I will not be able to work with it - this is ok.

But tag/branches on whole composite project is "the must".

I hope it is possible to teach git (or may be something else) to scan all subprojects and fetch common tags/branches and work with them.

On 1/12/06, Alexander Litvinov <lan@ac-sw.com> wrote:

The Linus' proposal of gitlink will probably help you here: gitlink will be tagged as well, so you just have to teach git-checkout about checking out subprojects.

On Thu, 12 Jan 2006 11:25:33 +0600, Alexander Litvinov wrote:

It sounds like you want 'config-manager', http://packages.debian.org/unstable/devel/config-manager, it doesn't support git (yet) but I can't imagine it is hard to add that support in.

Cheers, Anand

On Thu, 12 Jan 2006, Martin Langhoff wrote:

The core structural thing (which I'm not sure CVS handles) is having each commit of the outer project specify the commit of the inner project that it contains in some way. This would be good with CVS, but is vital with git, because there's no way of estimating it when you don't have a linear history. (With CVS, you could say that the inner project version for a given outer project version should be the version that was current when the outer project was committed. But that isn't well-defined for git.) If you try to debug anything involving the history, you'd have problems with choosing versions of the two projects that don't actually match.

	-Daniel
*This .sig left intentionally blank*

  Hello,

  I've tried to take a different approach - KISS and don't make the
subprojects part of the git-tracked tree but a thing purely local to
your particular checkout. Subprojects are simply listed in
.git/subprojects and various commands are called recursively on them.

  - No auto-cloning of subprojects is possible.
  - Switching between branches and merging is troublesome in case the
    required subprojects arrangement changes inbetween. Now, this is
    a matter of taste - I don't see this as a huge problem since you
    can make special provisions for that and this should be going to be
    so rare that it's not worth optimizing for in my eyes.
  + It's simple.
  + It's flexible. You can have _optional_ subprojects - IIRC e.g.
    mplayer can use ffmpeg if it's checked out as a subproject but
    will use own copy if it's not there. You can clone subprojects
    based e.g. on selected features before compilation. If you do so,
    your GIT won't bother you with uncommitted local changes, and you
    will not have to filter this out from any other changes you are
    going to commit.

  The main goal of this is simply to be able to check out bunch of
stuff to subdirectories and make cg-update update all of it, without
any special scripts. ;-)

  This patch is just trivial proof-of-concept thing which makes only
cg-update and cg-fetch aware of the subprojects; many commands still
need to be taught about subprojects but some don't - I currently don't
think a recursive cg-merge is a particularily good idea, for one.
I think the good default is to make all read-only commands by default
recursive and all modifying commands by default non-recursive. (And
it might be useful to be able to mark some subprojects read-only.)

  How to create a subproject? Simply cg-clone inside a working copy,
it will register it automagically. So far there are no tools to further
maintain the subprojects, though, therefore a mv or rm needs to be
followed by an appropriate modification of parent .git/subprojects.

  This is not committed yet - I'm curious about your opinions.

On Sun, 15 Jan 2006, Petr Baudis wrote:

This seems very sane. Goodie.

		Linus

Petr Baudis <pasky@suse.cz> writes:

All sounds sane and simple.  Good job!

Especially because this does not even try to let the project express its version dependencies on its subprojects, I like it for its simplicity (which makes it very easy to explain, to my mind that is the biggest plus). However, I fear that others might complain to say that the contained things do not deserve to be called "subprojects" if there is no version linkage [*1*].

I think something like this is greatly helpful for people (like me) as "an end user who builds from source out of SCM not from tarballs."

If you go this route, one minor concern is what the format of the $GIT_DIR/subproject should be (and if they do not deserve to be called "subproject" then what the name of the file be), because this may likely be a "nice if they were compatible" item across Porcelains. The list of names separated with LF has two very big pluses:

 - it is certainly the easiest to parse.

 - it can readily be used as --exclude-from file, as long as you
   do not care about another directory with a same name
   somewhere other than the subproject itself.

Even if the limitation to the latter becomes a real issue, a separate exclude-from file could easily be generated on the fly (prefix them with '/' to force "not anywhere in the subtree but only here" matching, perhaps with escaping shell glob pattern while you are at it), which does not sound so bad.

[Footnote]

*1* I do not personally care about version linkage; this is me being lazy to avoid core side support ;-). Yesterday I was mucking with rev-list code to see how gitlink and/or bind commit would affect what it needs to do (especially wrt its --objects flag), and I did not like the potential code impact I saw there.