Mixing and matching multiple projects

Imagine that there is an open source embedded appliance project "xyzzy" that consists of two submodules and the top-level superproject that binds them and serves as the scaffolding, perhaps organized like this:

    /s/xyzzy/.git       -- the superproject
    /s/xyzzy/os/.git    -- the "os" submodule
    /s/xyzzy/u/.git     -- the "userspace" submodule

The "os" submodule may contain various subdirectories, among which there may be a device drivers directory. Further imagine that the Makefile in that directory, /s/xyzzy/os/drivers/Makefile, is written in such a way that it can run make in any subdirectory of drivers/ and can link with resulting *.o files in these subdirectories.

Now suppose somebody wanted to add an add-on device driver for frotz device that does not appear in the open source tree. Further suppose that the open source folks do not want to accept support for "frotz" driver in their tree for whatever reason.

At the working tree level, he can add /s/xyzzy/os/drivers/frotz/ directory and populate its sources there. Everything else is already set up to build and link to it properly. But how would he add that directory to git?

He could fork the /s/xyzzy/os/.git submodule and add "drivers/frotz/" directory, and maintain his own fork of the submodule, which in turn means that he has to fork the top-level superproject /s/xyzzy/.git because its tree has to point at a commit in /s/xyzzy/os/.git repository that has the "drivers/frotz" directory in it, which is different from the open source version. A proprietary fork has to happen somewhere; that is a given.

But I wonder if we can do without forking the /s/xyzzy/os/.git submodule?

If we *could* do this:

	cd /s/xyzzy
        mkdir os/drivers/frotz && populate the directory with sources
        git add os/drivers/frotz ;# to the top-level superproject!!

to add the "frotz driver" submodule directly to the superproject, then we could leave /s/xyzzy/os/.git intact, letting it follow the open source world. Because the superproject must be forked in order to keep track of what happens in the appliance that supports the "frotz" driver anyway, this could result in the minimum amount of forking from the end user's point of view.

People who know the Git data structure of course can immediately see that this is not supported (and I strongly suspect unsupportable, but my suspicion has a history of being wrong from time to time, and that is why I am sending this message). The index at /s/xyzzy/.git/index has to be able to record "os" as a gitlink and "os/drivers/frotz/frotz.c" as a regular blob at the same time, which is a D/F conflict.

Even if you modify the index D/F conflict check to allow this, you cannot write the top-level tree object for the superproject, as "os" needs to be recorded as a gitlink to bind the /s/xyzzy/os/.git submodule, while you also have to have another "os" in the same top-level tree object as a tree object that has a single entry "drivers" (which is a tree) in it, under which all the "os/drivers/frotz" subdirectory hang. All tree traversal code would be very unhappy to see such a tree with "duplicate" entry, including the unpack-trees machinery used for merging and switching branches, connectivity machinery used for fsck, object enumeration, and object transfer. I imagine that it *could* be done, but it won't be a mere two-weekend hack; it has to first destroy pretty much everything in the current codebase and rebuild it to add such a support.

The "frotz" submodule could be added as /s/xyzzy/frotz/.git and the build infrastructure at /s/xyzzy/.git (which we are going to fork anyway) could be tweaked so that it creates a symbolic link in /s/xyzzy/os/drivers to point there, pretending that there is a 'frotz' subdirectory. While that would certainly be a workaround, I am wondering if people who have used submodules and/or nested projects more than I have better solutions to this puzzle.

Comments?

Junio C Hamano <gitster@pobox.com> writes:

A crucial step was missing here for my story to make *any* sense. Sorry. A step to create a repository at /s/xyzzy/os/drivers/frotz/.git must be here, like this:

        (cd os/drivers/frotz && git init && git add . && git commit)

And then that is added as a submodule to the superproject.

Junio C Hamano <gitster@pobox.com> writes:

submodule?

Sorry.

be

then we

source

track of

anyway,

user's

How about 
- creating a "driver" project in the superproject, 
- add a .gitignore or .git/exclude entry for the
/s/xyzzy/os/drivers/frotz folder into the os project
- symlinking the /s/xyzzy/os/drivers/frotz folder to the
/s/xyzzy/driver/frotz folder

some of this can be automated in the post checkout script of the superproject.

"Holding, Lawrence" <Lawrence.Holding@cubic.com> writes:

Thanks.

I think that is essentially the same as what I wrote with "symlink" in my last paragraph as a known-to-work workaround. The real location of the frotz driver, whether it is in a directory directly in the superproject, or a separate submodule that is bound to the superproject outside the "os" submodule, does not make an essential difference, as long as it is outside "os".

I was asking if people who have used submodules have better solutions than that "symlink" solution.

In message <7vhayptght.fsf@alter.siamese.dyndns.org>, Junio C Hamano writes:

        /s/xyzzy/.git       -- the superproject
        /s/xyzzy/os/.git    -- the "os" submodule
        /s/xyzzy/u/.git     -- the "userspace" submodule

    At the working tree level, he can add /s/xyzzy/os/drivers/frotz/ directory
    and populate its sources there. Everything else is already set up to build
    and link to it properly. But how would he add that directory to git?

    But I wonder if we can do without forking the /s/xyzzy/os/.git submodule?

    I am wondering if people who have used submodules and/or nested
    projects more than I have better solutions to this puzzle.

Using gitslave [http://gitslave.sf.net] instead of git-submodules, you can *almost* get there.

Specifically, gitslave allows the superproject /s/xyzzy to attach a subproject at /s/xyzzy/os/drivers/frotz (either recursively through /os which you don't want to do or directly).

However, the problem is the .gitignore in the os subproject. If we didn't have some way to ignore drivers/frotz, various git commands would get annoyed. Adding it in the os subproject requires forking which you said you didn't want to do. However, you could use core.excludesfile in the os subproject to ignore the necessary file.

gitslave doesn't support automatically using core.excludesfile (and indeed would automatically make an entry in os/.gitignore when you set up the hierarchy). However, it isn't a lot of work to revert that .gitignore commit and set up core.excludesfile; but setting up core.excludesfile on every clone might get annoying. It probably wouldn't be too difficult to tearch gitslave how to do that automagically.

Is using this third party tool and making the necessary changes or manually doing the work after every clone worth it? Do you need the git-submodule semantics of freezing the subprojects at a particular SHA instead of floating at the tip of a branch? Are the branches used between xyzzy and os desynchronized? Only you can judge. Actually it seems possible that you could use gitslave for os/drivers/frotz and git-submodules for os. Whether that is wise…

I would be open to moving gitslave into contrib similar to what git-subtree is under the process of doing, but gitslave would certainly need some cleanup since it grew rather organically.

					-Seth Robertson