{"thread":{"id":"29657","subject":"Mixing and matching multiple projects","startedAt":"2012-02-17T19:19:26Z","lastAt":"2012-02-20T00:36:57Z","messageCount":5,"participants":["Junio C Hamano","Seth Robertson","Holding, Lawrence"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"184900","messageId":"7vhayptght.fsf@alter.siamese.dyndns.org","threadId":"29657","inReplyTo":null,"subject":"Mixing and matching multiple projects","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-02-17T19:19:26Z","receivedAt":"2012-02-17T19:19:26Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Imagine that there is an open source embedded appliance project \"xyzzy\"\nthat consists of two submodules and the top-level superproject that binds\nthem and serves as the scaffolding, perhaps organized like this:\n\n    /s/xyzzy/.git       -- the superproject\n    /s/xyzzy/os/.git    -- the \"os\" submodule\n    /s/xyzzy/u/.git     -- the \"userspace\" submodule\n\nThe \"os\" submodule may contain various subdirectories, among which there\nmay be a device drivers directory.  Further imagine that the Makefile in\nthat directory, /s/xyzzy/os/drivers/Makefile, is written in such a way\nthat it can run make in any subdirectory of drivers/ and can link with\nresulting *.o files in these subdirectories.\n\nNow suppose somebody wanted to add an add-on device driver for frotz\ndevice that does not appear in the open source tree.  Further suppose that\nthe open source folks do not want to accept support for \"frotz\" driver in\ntheir tree for whatever reason.\n\nAt the working tree level, he can add /s/xyzzy/os/drivers/frotz/ directory\nand populate its sources there. Everything else is already set up to build\nand link to it properly. But how would he add that directory to git?\n\nHe could fork the /s/xyzzy/os/.git submodule and add \"drivers/frotz/\"\ndirectory, and maintain his own fork of the submodule, which in turn means\nthat he has to fork the top-level superproject /s/xyzzy/.git because its\ntree has to point at a commit in /s/xyzzy/os/.git repository that has the\n\"drivers/frotz\" directory in it, which is different from the open source\nversion. A proprietary fork has to happen somewhere; that is a given.\n\nBut I wonder if we can do without forking the /s/xyzzy/os/.git submodule?\n\nIf we *could* do this:\n\n\tcd /s/xyzzy\n        mkdir os/drivers/frotz && populate the directory with sources\n        git add os/drivers/frotz ;# to the top-level superproject!!\n\nto add the \"frotz driver\" submodule directly to the superproject, then we\ncould leave /s/xyzzy/os/.git intact, letting it follow the open source\nworld. Because the superproject must be forked in order to keep track of\nwhat happens in the appliance that supports the \"frotz\" driver anyway,\nthis could result in the minimum amount of forking from the end user's\npoint of view.\n\nPeople who know the Git data structure of course can immediately see that\nthis is not supported (and I strongly suspect unsupportable, but my\nsuspicion has a history of being wrong from time to time, and that is why\nI am sending this message).  The index at /s/xyzzy/.git/index has to be\nable to record \"os\" as a gitlink and \"os/drivers/frotz/frotz.c\" as a\nregular blob at the same time, which is a D/F conflict.\n\nEven if you modify the index D/F conflict check to allow this, you cannot\nwrite the top-level tree object for the superproject, as \"os\" needs to be\nrecorded as a gitlink to bind the /s/xyzzy/os/.git submodule, while you\nalso have to have another \"os\" in the same top-level tree object as a tree\nobject that has a single entry \"drivers\" (which is a tree) in it, under\nwhich all the \"os/drivers/frotz\" subdirectory hang.  All tree traversal\ncode would be very unhappy to see such a tree with \"duplicate\" entry,\nincluding the unpack-trees machinery used for merging and switching\nbranches, connectivity machinery used for fsck, object enumeration, and\nobject transfer.  I imagine that it *could* be done, but it won't be a\nmere two-weekend hack; it has to first destroy pretty much everything in\nthe current codebase and rebuild it to add such a support.\n\nThe \"frotz\" submodule could be added as /s/xyzzy/frotz/.git and the build\ninfrastructure at /s/xyzzy/.git (which we are going to fork anyway) could\nbe tweaked so that it creates a symbolic link in /s/xyzzy/os/drivers to\npoint there, pretending that there is a 'frotz' subdirectory. While that\nwould certainly be a workaround, I am wondering if people who have used\nsubmodules and/or nested projects more than I have better solutions to\nthis puzzle.\n\nComments?\n"},{"id":"184902","messageId":"7v39a9tf45.fsf@alter.siamese.dyndns.org","threadId":"29657","inReplyTo":"7vhayptght.fsf@alter.siamese.dyndns.org","subject":"Re: Mixing and matching multiple projects","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-02-17T19:49:14Z","receivedAt":"2012-02-17T19:49:14Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano <gitster@pobox.com> writes:\n\n> But I wonder if we can do without forking the /s/xyzzy/os/.git submodule?\n>\n> If we *could* do this:\n>\n> \tcd /s/xyzzy\n>       mkdir os/drivers/frotz && populate the directory with sources\n\nA crucial step was missing here for my story to make *any* sense. Sorry.\nA step to create a repository at /s/xyzzy/os/drivers/frotz/.git must be\nhere, like this:\n\n        (cd os/drivers/frotz && git init && git add . && git commit)\n\nAnd then that is added as a submodule to the superproject.\n\n>       git add os/drivers/frotz ;# to the top-level superproject!!\n>\n> to add the \"frotz driver\" submodule directly to the superproject, then we\n> could leave /s/xyzzy/os/.git intact, letting it follow the open source\n> world. Because the superproject must be forked in order to keep track of\n> what happens in the appliance that supports the \"frotz\" driver anyway,\n> this could result in the minimum amount of forking from the end user's\n> point of view.\n>\n> People who know the Git data structure of course can immediately see that\n> this is not supported (and I strongly suspect unsupportable, but my\n> suspicion has a history of being wrong from time to time, and that is why\n> I am sending this message).  The index at /s/xyzzy/.git/index has to be\n> able to record \"os\" as a gitlink and \"os/drivers/frotz/frotz.c\" as a\n> regular blob at the same time, which is a D/F conflict.\n>\n> Even if you modify the index D/F conflict check to allow this, you cannot\n> write the top-level tree object for the superproject, as \"os\" needs to be\n> recorded as a gitlink to bind the /s/xyzzy/os/.git submodule, while you\n> also have to have another \"os\" in the same top-level tree object as a tree\n> object that has a single entry \"drivers\" (which is a tree) in it, under\n> which all the \"os/drivers/frotz\" subdirectory hang.  All tree traversal\n> code would be very unhappy to see such a tree with \"duplicate\" entry,\n> including the unpack-trees machinery used for merging and switching\n> branches, connectivity machinery used for fsck, object enumeration, and\n> object transfer.  I imagine that it *could* be done, but it won't be a\n> mere two-weekend hack; it has to first destroy pretty much everything in\n> the current codebase and rebuild it to add such a support.\n>\n> The \"frotz\" submodule could be added as /s/xyzzy/frotz/.git and the build\n> infrastructure at /s/xyzzy/.git (which we are going to fork anyway) could\n> be tweaked so that it creates a symbolic link in /s/xyzzy/os/drivers to\n> point there, pretending that there is a 'frotz' subdirectory. While that\n> would certainly be a workaround, I am wondering if people who have used\n> submodules and/or nested projects more than I have better solutions to\n> this puzzle.\n>\n> Comments?\n"},{"id":"184904","messageId":"201202172006.q1HK6MJg030775@no.baka.org","threadId":"29657","inReplyTo":"7vhayptght.fsf@alter.siamese.dyndns.org","subject":"Re: Mixing and matching multiple projects","fromName":"Seth Robertson","fromEmail":"in-gitvger@baka.org","sentAt":"2012-02-17T20:06:22Z","receivedAt":"2012-02-17T20:06:22Z","isPatch":false,"sender":{"key":"in-gitvger@baka.org","avatar":null},"body":"\nIn message <7vhayptght.fsf@alter.siamese.dyndns.org>, Junio C Hamano writes:\n\n        /s/xyzzy/.git       -- the superproject\n        /s/xyzzy/os/.git    -- the \"os\" submodule\n        /s/xyzzy/u/.git     -- the \"userspace\" submodule\n\n    At the working tree level, he can add /s/xyzzy/os/drivers/frotz/ directory\n    and populate its sources there. Everything else is already set up to build\n    and link to it properly. But how would he add that directory to git?\n\n    But I wonder if we can do without forking the /s/xyzzy/os/.git submodule?\n\n    I am wondering if people who have used submodules and/or nested\n    projects more than I have better solutions to this puzzle.\n\nUsing gitslave [http://gitslave.sf.net] instead of git-submodules, you\ncan *almost* get there.\n\nSpecifically, gitslave allows the superproject /s/xyzzy to attach a\nsubproject at /s/xyzzy/os/drivers/frotz (either recursively through\n/os which you don't want to do or directly).\n\nHowever, the problem is the .gitignore in the os subproject.  If we\ndidn't have some way to ignore drivers/frotz, various git commands\nwould get annoyed.  Adding it in the os subproject requires forking\nwhich you said you didn't want to do.  However, you could use\ncore.excludesfile in the os subproject to ignore the necessary file.\n\ngitslave doesn't support automatically using core.excludesfile (and\nindeed would automatically make an entry in os/.gitignore when you set\nup the hierarchy).  However, it isn't a lot of work to revert that\n.gitignore commit and set up core.excludesfile; but setting up\ncore.excludesfile on every clone might get annoying.  It probably\nwouldn't be too difficult to tearch gitslave how to do that\nautomagically.\n\nIs using this third party tool and making the necessary changes or\nmanually doing the work after every clone worth it?  Do you need the\ngit-submodule semantics of freezing the subprojects at a particular\nSHA instead of floating at the tip of a branch?  Are the branches used\nbetween xyzzy and os desynchronized?  Only you can judge.  Actually it\nseems possible that you could use gitslave for os/drivers/frotz and\ngit-submodules for os.  Whether that is wise…\n\nI would be open to moving gitslave into contrib similar to what\ngit-subtree is under the process of doing, but gitslave would\ncertainly need some cleanup since it grew rather organically.\n\n\t\t\t\t\t-Seth Robertson\n"},{"id":"184954","messageId":"A5E8E180685CEF45AB9E737A010799802D972E@cdnz-ex1.corp.cubic.cub","threadId":"29657","inReplyTo":"7v39a9tf45.fsf@alter.siamese.dyndns.org","subject":"RE: Mixing and matching multiple projects","fromName":"Holding, Lawrence","fromEmail":"lawrence.holding@cubic.com","sentAt":"2012-02-20T00:17:23Z","receivedAt":"2012-02-20T00:17:23Z","isPatch":false,"sender":{"key":"lawrence.holding@cubic.com","avatar":null},"body":"Junio C Hamano <gitster@pobox.com> writes:\n> Junio C Hamano <gitster@pobox.com> writes:\n> \n> > But I wonder if we can do without forking the /s/xyzzy/os/.git\nsubmodule?\n> >\n> > If we *could* do this:\n> >\n> > \tcd /s/xyzzy\n> >       mkdir os/drivers/frotz && populate the directory with sources\n> \n> A crucial step was missing here for my story to make *any* sense.\nSorry.\n> A step to create a repository at /s/xyzzy/os/drivers/frotz/.git must\nbe\n> here, like this:\n> \n>         (cd os/drivers/frotz && git init && git add . && git commit)\n> \n> And then that is added as a submodule to the superproject.\n> \n> >       git add os/drivers/frotz ;# to the top-level superproject!!\n> >\n> > to add the \"frotz driver\" submodule directly to the superproject,\nthen we\n> > could leave /s/xyzzy/os/.git intact, letting it follow the open\nsource\n> > world. Because the superproject must be forked in order to keep\ntrack of\n> > what happens in the appliance that supports the \"frotz\" driver\nanyway,\n> > this could result in the minimum amount of forking from the end\nuser's\n> > point of view.\n> >\n> ........\n> >\nHow about \n- creating a \"driver\" project in the superproject, \n- add a .gitignore or .git/exclude entry for the\n/s/xyzzy/os/drivers/frotz folder into the os project\n- symlinking the /s/xyzzy/os/drivers/frotz folder to the\n/s/xyzzy/driver/frotz folder\n\nsome of this can be automated in the post checkout script of the\nsuperproject.\n"},{"id":"184955","messageId":"7vy5rynxw6.fsf@alter.siamese.dyndns.org","threadId":"29657","inReplyTo":"A5E8E180685CEF45AB9E737A010799802D972E@cdnz-ex1.corp.cubic.cub","subject":"Re: Mixing and matching multiple projects","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-02-20T00:36:57Z","receivedAt":"2012-02-20T00:36:57Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Holding, Lawrence\" <Lawrence.Holding@cubic.com> writes:\n\n> How about \n> - creating a \"driver\" project in the superproject, \n> - add a .gitignore or .git/exclude entry for the\n> /s/xyzzy/os/drivers/frotz folder into the os project\n> - symlinking the /s/xyzzy/os/drivers/frotz folder to the\n> /s/xyzzy/driver/frotz folder\n\nThanks.\n\nI think that is essentially the same as what I wrote with \"symlink\" in my\nlast paragraph as a known-to-work workaround.  The real location of the\nfrotz driver, whether it is in a directory directly in the superproject,\nor a separate submodule that is bound to the superproject outside the \"os\"\nsubmodule, does not make an essential difference, as long as it is outside\n\"os\".\n\nI was asking if people who have used submodules have better solutions than\nthat \"symlink\" solution.\n"}]}