{"thread":{"id":"5704","subject":"Notes on Using Git with Subprojects","startedAt":"2006-09-26T17:40:56Z","lastAt":"2006-10-01T05:19:06Z","messageCount":39,"participants":["A Large Angry SCM","Johannes Schindelin","Daniel Barkalow","Shawn Pearce","Martin Waitz","Sven Verdoolaege","Jeff King","Junio C Hamano","Rogan Dawes","Michael S. Tsirkin"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"27687","messageId":"45196628.9010107@gmail.com","threadId":"5704","inReplyTo":null,"subject":"Notes on Using Git with Subprojects","fromName":"A Large Angry SCM","fromEmail":"gitzilla@gmail.com","sentAt":"2006-09-26T17:40:56Z","receivedAt":"2006-09-26T17:40:56Z","isPatch":false,"sender":{"key":"gitzilla@gmail.com","avatar":"https://gravatar.com/avatar/354625c442439908ff3dd99757dee330e29e9df7847472384faf7a00add247fb?d=mp&s=160"},"body":"\t\t\t\t\t\t\t20060926.1715\n\nNotes on Using Git with Subprojects\n===================================\nCopyright (C) 2006 by Raymond S Brand\n\n\nGit does not have native support for subprojects and this is a good\nthing because managing subprojects is better handled by the project\nbuild machinery. Managing subprojects with the project build machinery\nis more flexible than the native support from an SCM and allows the use\nof different SCMs by different subprojects. However, there is a lot of\ninterest in using Git for the SCM of a project with subprojects.\n\nGit, unfortunately, does not make it easy. What is wanted is to put all\nof the subprojects in one repository and be able to checkout the various\nparts from a local copy of the repository. The problem is, with Git, a\nrepository can have at most one working directory associated with it at\na time. This is because Git stores a lot of information about the\ncontents of the working directory in the repository. In fact, the usual\nsituation is that the repository, itself, is in the working directory.\n\nThis note describes a method to use Git with subprojects; other methods\nare also possible.\n\n\nDefinitions\n-----------\nParent Project:\n\tA project that logically contains one or more subprojects.\n\nProject:\n\tA set of files that is (relatively) self contained with respect\n\tto changes and treated as a unit by the SCM.\n\nRoot Project:\n\tA project that is not a subproject.\n\nSubproject:\n\tA project logically contained in another project.\n\n\nSetup\n-----\nAll subprojects are contained in a single repository and referred to by\nseparate branches in $GIT_DIR/refs/heads/. Developers create a local\ncopy of the project repository; hereafter, referred to as the \"local\nmaster repository\" or just \"local master\". All fetches, pulls and pushes\nby the developers with the \"local master\" of other developers should be\nto and from their own \"local master\".\n\n\nRoot Project Checkout\n---------------------\nThe root project is the project that all of the subprojects are a part\nof. It is a parent project of one or more subprojects; each of which can\nalso be parent projects to other subprojects.\n\nTo checkout the root project, choose an name for the project working\ndirectory (the working directory must not already exist) and perform the\nequivalent of the following commands.\n\n\tgit-clone -s -n $LOCAL_MASTER $ROOT_DIR \\\n\t\t&& cd $ROOT_DIR \\\n\t\t&& git-checkout -b $ROOT_BRANCH--local $ROOT_BRANCH\n\nWhere:\n\t$LOCAL_MASTER is the path to the \"local master\"\n\t$ROOT_DIR is the name of the working directory to use\n\t$ROOT_BRANCH is the branch name of the root project\n\t$ROOT_BRANCH--local is a branch for local changes\n\nThis will leave the current working directory of the shell in the\nproject working directory.\n\nThis also creates a branch with the suffix of \"--local\" to hold all of\nthe local working directory commits and modifications. The $ROOT_BRANCH\nis used as a tracking branch so that upstream changes can be fetched\ninto the working repository without affecting the checked out files.\n\nOnce the root project is checked out, the subprojects are checked out.\n\n\nSubproject Checkout\n-------------------\nEach project that is a parent project needs to checkout all of the\nsubprojects of the project. Each subproject is checked out with the\nequivalent of the following bash commands:\n\n\tgit-clone -s -n $LOCAL_MASTER $SUBPROJECT_DIR \\\n\t\t&& ( cd $SUBPROJECT_DIR \\\n\t\t\t&& git-checkout -b $SUBPROJECT_BRANCH--local \\\n\t\t\t\t$SUBPROJECT_BRANCH\n\t\t)\n\nWhere:\n\t$LOCAL_MASTER is the path to the \"local master\"\n\t$SUBPROJECT_DIR is the directory name of the subproject\n\t$SUBPROJECT_BRANCH is the branch name of the subproject\n\t$SUBPROJECT_BRANCH--local is a branch for local changes\n\t\nIf a subproject has subprojects, then the checkouts need to done\nrecursively. With suitable project/subproject/branch naming conventions\nthis can easily automated.\n\n\nProject Development\n-------------------\nChanges to a project are performed in the working directory of the\nproject and are recorded in the repository in the working directory on\nthe $PROJECT--local branch.\n\n\nReceiving Project Upstream Changes\n----------------------------------\nUpstream project changes are first fetched into the project tracking\nbranch of the local master repository and are then fetched into the\nproject tracking branch of the working directory repositories. To merge\nupstream changes into the working directory, a pull from the project\ntracking branch of the working directory repository executed.\n\n\t# Fetch project branch from upstream to local master\n\t(cd $LOCAL_MASTER && git-fetch $UPSTREAM $PROJECT_BRANCH)\n\n\t# Fetch project branch from local master to working repo\n\tgit-fetch $LOCAL_MASTER $PROJECT_BRANCH\n\n\t# Merge upstream changes in to working directory\n\tgit-pull --no-commit . $PROJECT_BRANCH\n\nWhere:\n\t$LOCAL_MASTER is the path to the \"local master\"\n\t$UPSTREAM is the Git URL of the upstream repository\n\t$PROJECT_BRANCH is the branch name of the (sub)project\n\t\n\nSending Project Changes Upstream\n--------------------------------\nTo send project changes upstream from a working directory repository,\nthe changes are first pushed to a branch in the local master repository,\n$PROJECT--$IDENT. The changes can then be pushed or pulled from the\nlocal master repository.\n\n\t# Push project changes to local master\n\tgit-push $LOCAL_MASTER \\\n\t\t$PROJECT_BRANCH--local:$PROJECT_BRANCH--$IDENT\n\n\t# Push project changes from local master to upstream\n\t(cd $LOCAL_MASTER && git-push $UPSTREAM \\\n\t\t$PROJECT_BRANCH--$IDENT:$PROJECT_BRANCH--$NICK--$IDENT)\n\nWhere:\n\t$LOCAL_MASTER is the path to the \"local master\"\n\t$UPSTREAM is the Git URL of the upstream repository\n\t$PROJECT_BRANCH is the branch name of the (sub)project\n\t$PROJECT_BRANCH--local is a branch for local changes\n\t$IDENT is a label unique for this set of working directories\n\t$NICK is a (branch name safe) identifier of the developer\n\t\n\nAutomation\n----------\nAdding the following example code to the makefiles of the root project\nand subprojects with the appropriate information can automate most of\nthe operations needed to support subprojects with Git. The ProjectSetup\ntarget needs the REPOSITORY and IDENT make variables to be set but the\nother targets will use the values saved by the ProjectSetup target.\n\n ---->8----  ---->8----  ---->8----  ---->8----  ---->8----  ---->8----\n# Add this to the makefiles\nPROJECT := branch/name          # Relative to $GIT_DIR/refs/heads/\nSUBPROJECTS := a/b c/d          # Each relative to $GIT_DIR/refs/heads/\ninclude Git_machinery.make\n\n# Rest of the Makefile\n ---->8----  ---->8----  ---->8----  ---->8----  ---->8----  ---->8----\n# Git_machinery.make\n\ninclude GIT-SUBPROJECT-VARS.make\n\nREFS := /refs/heads/$(REFPREFIX)\n\nGIT-SUBPROJECT-VARS.make:\n\trm -rf GIT-SUBPROJECT-VARS.make\n\techo \"REPOSITORY := $(REPOSITORY)\" > GIT-SUBPROJECT-VARS.make\n\techo \"REFPREFIX := $(REFPREFIX)\" >> GIT-SUBPROJECT-VARS.make\n\techo \"IDENT := $(IDENT)\" >> GIT-SUBPROJECT-VARS.make\n\nProjectSetup:: GIT-SUBPROJECT-VARS.make\n\tfor SUBPROJECT in $(SUBPROJECTS) ; \\\n\tdo \\\n\t    git-clone -s -n $(REPOSITORY) $$(basename $$SUBPROJECT) \\\n\t    && ( cd $$(basename $$SUBPROJECT) \\\n\t\t&& git-checkout -b \\\n\t\t    $(REFS)$$SUBPROJECT--local $(REFS)$$SUBPROJECT \\\n\t\t&& $(MAKE) ProjectSetup \\\n\t    ) \\\n\tdone\n\nFetchMaster::\n\tgit-fetch $(REPOSITORY) $(REFS)$(PROJECT)\n\tfor SUBPROJECT in $(SUBPROJECTS) ; \\\n\tdo \\\n\t    $(MAKE) -C $$(basename $$SUBPROJECT) FetchMaster ; \\\n\tdone\n\nPullMaster::\n\tgit-fetch $(REPOSITORY) $(REFS)$(PROJECT)\n\tgit-pull --no-commit .  $(REFS)$(PROJECT)\n\tfor SUBPROJECT in $(SUBPROJECTS) ; \\\n\tdo \\\n\t    $(MAKE) -C $$(basename $$SUBPROJECT) PullMaster ; \\\n\tdone\n\nPushMaster::\n\tgit-push $(REPOSITORY) \\\n\t    $(REFS)$(PROJECT)--local:$(REFS)$(PROJECT)--$(IDENT)\n\tfor SUBPROJECT in $(SUBPROJECTS) ; \\\n\tdo \\\n\t    $(MAKE) -C $$(basename $$SUBPROJECT) PushMaster ; \\\n\tdone\n\n ---->8----  ---->8----  ---->8----  ---->8----  ---->8----  ---->8----\n\nThe example Makefile code has some limitations: There is no error\nchecking. It does not handle subprojects rooted in a non top level\ndirectory of the parent project. There are other Git commands that can\nbe usefully applied recursively to all subprojects.\n"},{"id":"27701","messageId":"Pine.LNX.4.63.0609262203510.25371@wbgn013.biozentrum.uni-wuerzburg.de","threadId":"5704","inReplyTo":"45196628.9010107@gmail.com","subject":"Re: Notes on Using Git with Subprojects","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2006-09-26T20:25:19Z","receivedAt":"2006-09-26T20:25:19Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Tue, 26 Sep 2006, A Large Angry SCM wrote:\n\n> \t\t\t\t\t\t\t20060926.1715\n\nYou forgot the time zone ;-)\n\n> Notes on Using Git with Subprojects\n> ===================================\n> Copyright (C) 2006 by Raymond S Brand\n> \n> \n> Git does not have native support for subprojects and this is a good\n> thing because managing subprojects is better handled by the project\n> build machinery. Managing subprojects with the project build machinery\n> is more flexible than the native support from an SCM and allows the use\n> of different SCMs by different subprojects. However, there is a lot of\n> interest in using Git for the SCM of a project with subprojects.\n> \n> Git, unfortunately, does not make it easy.\n\nAfter skimming your text, I imagine that it should be possible (read: \neasy) to write a really simple script which does what you describe, \nstoring the relevant information about root or sub projects in the config.\n\nHowever, you left out one of the most important aspects of subprojects: \nthe ability to manage the state of the root project: you can add, update \nand remove subprojects.\n\nA while ago, Junio started playing with a new object type for subprojects \nso that you could have tree objects containing subprojects in addition \nto tree objects and blobs.\n\nOf course, the difficult thing about this is to teach all tools to behave \nsensibly with the new object type.\n\nNow, your approach of having multiple clones (sharing the object pool) is \nmore simple than Junio's approach: no need to introduce a new object type, \nor adapt existing tools.\n\nTaking this a step further, how about managing the root project in this \nmanner:\n\nA root project is a branch containing just one special file, \n\"root-project\". This file consists of lines like these:\n\n-- snip --\nf80a17bf3da1e24ac904f9078f68c3bf935ff250 next\n03adf42c988195b50e1a1935ba5fcbc39b2b029b todo\n-- snap --\n\nThe meaning: subdirectory \"next\" contains subproject \"next\" which is also \ntracked in the branch \"next\" of the root project. Likewise for \"todo\". The \nroot project could even contain some administrative files like a Makefile, \na license, README, etc.\n\nYou could even handle the update of root-project with each commit in a \nsubproject by a hook in that subproject's .git/hooks/post-commit, so that \nyou'd only need a script \"git-checkout-root-project.sh\" to initialize them \nall, and probably a script \"git-update-root-project.sh\".\n\nThoughts?\n\nCiao,\nDscho\n\nP.S.: Is it just me, or do other people also find it confusing that \n--shared means different things for git-init-db and git-clone? (I know I \nam the sinner...)\n"},{"id":"27707","messageId":"Pine.LNX.4.64.0609261629160.9789@iabervon.org","threadId":"5704","inReplyTo":"45196628.9010107@gmail.com","subject":"Re: Notes on Using Git with Subprojects","fromName":"Daniel Barkalow","fromEmail":"barkalow@iabervon.org","sentAt":"2006-09-26T21:23:15Z","receivedAt":"2006-09-26T21:23:15Z","isPatch":false,"sender":{"key":"barkalow@iabervon.org","avatar":"https://avatars.githubusercontent.com/u/55364219?v=4"},"body":"On Tue, 26 Sep 2006, A Large Angry SCM wrote:\n\n> Git, unfortunately, does not make it easy. What is wanted is to put all\n> of the subprojects in one repository and be able to checkout the various\n> parts from a local copy of the repository. The problem is, with Git, a\n> repository can have at most one working directory associated with it at\n> a time. This is because Git stores a lot of information about the\n> contents of the working directory in the repository. In fact, the usual\n> situation is that the repository, itself, is in the working directory.\n\nThere are a bunch of use cases which people see as subprojects, with \nslightly different desires. For example, I personally don't think there's \nany point to subprojects if a commit of the parent project doesn't specify \nthe embedded commits of each subproject (so, for example, you can use \nbisect on the parent project to figure out which act of updating a \nsubproject broke the resulting system). AFAICT, your design doesn't handle \nthat, but uses the most recently fetched versions of all subprojects, with \nthe revision control of the parent only handling revisions in the \narrangement and membership of subprojects in the parent.\n\n\t-Daniel\n*This .sig left intentionally blank*\n"},{"id":"27708","messageId":"20060926213003.GA8177@spearce.org","threadId":"5704","inReplyTo":"Pine.LNX.4.64.0609261629160.9789@iabervon.org","subject":"Re: Notes on Using Git with Subprojects","fromName":"Shawn Pearce","fromEmail":"spearce@spearce.org","sentAt":"2006-09-26T21:30:03Z","receivedAt":"2006-09-26T21:30:03Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Daniel Barkalow <barkalow@iabervon.org> wrote:\n> On Tue, 26 Sep 2006, A Large Angry SCM wrote:\n> \n> > Git, unfortunately, does not make it easy. What is wanted is to put all\n> > of the subprojects in one repository and be able to checkout the various\n> > parts from a local copy of the repository. The problem is, with Git, a\n> > repository can have at most one working directory associated with it at\n> > a time. This is because Git stores a lot of information about the\n> > contents of the working directory in the repository. In fact, the usual\n> > situation is that the repository, itself, is in the working directory.\n> \n> There are a bunch of use cases which people see as subprojects, with \n> slightly different desires. For example, I personally don't think there's \n> any point to subprojects if a commit of the parent project doesn't specify \n> the embedded commits of each subproject (so, for example, you can use \n> bisect on the parent project to figure out which act of updating a \n> subproject broke the resulting system). AFAICT, your design doesn't handle \n> that, but uses the most recently fetched versions of all subprojects, with \n> the revision control of the parent only handling revisions in the \n> arrangement and membership of subprojects in the parent.\n\nI agree entirely.\n\nI have about 30 \"subprojects\" tacked into one large Git repository\nfor this exact reason.  In at least 5 of these cases they shouldn't\nbe sharing a Git repository as by all rights they are different\nprojects.\n\nWhat I'm doing is sort of like tacking both the Linux kernel and\nglibc into the same Git repository because you might need to change\nand bisect over updates to the system call layer.  Insane, yes.\nProbably shouldn't be done; but right now that interface layer\nbetween several subprojects is still in flux and it makes it rather\neasy to keep everything in sync.\n\nIts annoying to perform commits to the \"root project\" every time the\nsubproject changes.  And it brings some complexity when you want to\ntalk about merging that root project.  But if its automated as part\nof \"git commit\" and \"git merge\" (either directly in those tools or\nby hooks users can install) then its probbaly a non issue.\n\n-- \nShawn.\n"},{"id":"27712","messageId":"4519A321.4010507@gmail.com","threadId":"5704","inReplyTo":"Pine.LNX.4.63.0609262203510.25371@wbgn013.biozentrum.uni-wuerzburg.de","subject":"Re: Notes on Using Git with Subprojects","fromName":"A Large Angry SCM","fromEmail":"gitzilla@gmail.com","sentAt":"2006-09-26T22:01:05Z","receivedAt":"2006-09-26T22:01:05Z","isPatch":false,"sender":{"key":"gitzilla@gmail.com","avatar":"https://gravatar.com/avatar/354625c442439908ff3dd99757dee330e29e9df7847472384faf7a00add247fb?d=mp&s=160"},"body":"Johannes Schindelin wrote:\n> Hi,\n> \n> On Tue, 26 Sep 2006, A Large Angry SCM wrote:\n> \n>> \t\t\t\t\t\t\t20060926.1715\n> \n> You forgot the time zone ;-)\n\nUTC of course. ;-)\n\n[...]\n> \n> After skimming your text, I imagine that it should be possible (read: \n> easy) to write a really simple script which does what you describe, \n> storing the relevant information about root or sub projects in the config.\n> \n> However, you left out one of the most important aspects of subprojects: \n> the ability to manage the state of the root project: you can add, update \n> and remove subprojects.\n\nSince each project can have subprojects, the root project is special \n_only_ in that the initial checkout is _not_ handled by the build \nmachinery (make in my example). This means that adding and removing \nsubprojects should be no different for the root project. A subproject \nshould be able to stand on its own (with its subprojects); otherwise, \nit's not a project but instead a directory tree versioned separately.\n\n> A while ago, Junio started playing with a new object type for subprojects \n> so that you could have tree objects containing subprojects in addition \n> to tree objects and blobs.\n\nWhich I though was awful. Subprojects are really much better managed by \nthe build machinery; it's more flexible and doesn't require all the \nseparate projects to use the same VCS.\n\n> Of course, the difficult thing about this is to teach all tools to behave \n> sensibly with the new object type.\n\nAnd teach all the tools to gracefully handle subprojects using a \ndifferent VCS ...\n\n> Now, your approach of having multiple clones (sharing the object pool) is \n> more simple than Junio's approach: no need to introduce a new object type, \n> or adapt existing tools.\n> \n> Taking this a step further, how about managing the root project in this \n> manner:\n> \n> A root project is a branch containing just one special file, \n> \"root-project\". This file consists of lines like these:\n> \n> -- snip --\n> f80a17bf3da1e24ac904f9078f68c3bf935ff250 next\n> 03adf42c988195b50e1a1935ba5fcbc39b2b029b todo\n> -- snap --\n> \n> The meaning: subdirectory \"next\" contains subproject \"next\" which is also \n> tracked in the branch \"next\" of the root project. Likewise for \"todo\". The \n> root project could even contain some administrative files like a Makefile, \n> a license, README, etc.\n\nHow the state (subproject list, branch names, etc.) is recorded in a \nparent project is only important to the parent project. The parent \nproject must also know how to interact with with each of its subprojects.\n\nFor instance, if you were building some kind of internet appliance, it \ncould have a very large number of subprojects (kernel, various servers \nand daemons, etc) with no common build tool or target set used by all of \nthem. What each parent project does in this situation is to \"interface\" \nto the build machinery of each subproject so that when you command \"make \nappliance image\" at the top level project, all the subprojects _and_ the \ntop level project do what is need to create the appliance image. The \nappliance can depend on projects using many VCS'; Git for the kernel, \nSVN for the web server, CVS for the SMTP daemon, Monotone for some other \npart, etc.\n\n> You could even handle the update of root-project with each commit in a \n> subproject by a hook in that subproject's .git/hooks/post-commit, so that \n> you'd only need a script \"git-checkout-root-project.sh\" to initialize them \n> all, and probably a script \"git-update-root-project.sh\".\n\nFor _full_ subproject support, handling a tree of projects is required. \nSo treating the root project differently that any other parent project \n(except of the initial checkout of the root project), means that you \ncan't work a subproject independent of its parent projects.\n"},{"id":"27713","messageId":"4519A49F.6000206@gmail.com","threadId":"5704","inReplyTo":"Pine.LNX.4.64.0609261629160.9789@iabervon.org","subject":"Re: Notes on Using Git with Subprojects","fromName":"A Large Angry SCM","fromEmail":"gitzilla@gmail.com","sentAt":"2006-09-26T22:07:27Z","receivedAt":"2006-09-26T22:07:27Z","isPatch":false,"sender":{"key":"gitzilla@gmail.com","avatar":"https://gravatar.com/avatar/354625c442439908ff3dd99757dee330e29e9df7847472384faf7a00add247fb?d=mp&s=160"},"body":"Daniel Barkalow wrote:\n> There are a bunch of use cases which people see as subprojects, with \n> slightly different desires. For example, I personally don't think there's \n> any point to subprojects if a commit of the parent project doesn't specify \n> the embedded commits of each subproject (so, for example, you can use \n> bisect on the parent project to figure out which act of updating a \n> subproject broke the resulting system). AFAICT, your design doesn't handle \n> that, but uses the most recently fetched versions of all subprojects, with \n> the revision control of the parent only handling revisions in the \n> arrangement and membership of subprojects in the parent.\n\nThat isn't that much different than what I outlined. Instead of \nrecording the branch name and directory in the parent project, you could \nrecord the commit SHA1 ID for each subproject and directory. The \nmachinery changes but the idea is the same.\n"},{"id":"27714","messageId":"Pine.LNX.4.63.0609270009400.25371@wbgn013.biozentrum.uni-wuerzburg.de","threadId":"5704","inReplyTo":"4519A321.4010507@gmail.com","subject":"Re: Notes on Using Git with Subprojects","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2006-09-26T22:13:05Z","receivedAt":"2006-09-26T22:13:05Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Tue, 26 Sep 2006, A Large Angry SCM wrote:\n\n> How the state (subproject list, branch names, etc.) is recorded in a \n> parent project is only important to the parent project. The parent \n> project must also know how to interact with with each of its \n> subprojects.\n\nGranted, if you mix VCSes, this is most pragmatic.\n\nBut it is also wrong: The whole point in bundling the subprojects together \nis (IMHO) to get the benefits of a VCS for the root project, i.e. for the \ncombined states of the subprojects. After all, you want to say \"I know \nthat this collection of projects at these states compiled and worked \nfine.\"\n\nAnd if you let a build system handle the stitching of the subprojects, you \ncompletely lose these benefits.\n\nCiao,\nDscho\n"},{"id":"27717","messageId":"4519AACD.7020508@gmail.com","threadId":"5704","inReplyTo":"20060926213003.GA8177@spearce.org","subject":"Re: Notes on Using Git with Subprojects","fromName":"A Large Angry SCM","fromEmail":"gitzilla@gmail.com","sentAt":"2006-09-26T22:33:49Z","receivedAt":"2006-09-26T22:33:49Z","isPatch":false,"sender":{"key":"gitzilla@gmail.com","avatar":"https://gravatar.com/avatar/354625c442439908ff3dd99757dee330e29e9df7847472384faf7a00add247fb?d=mp&s=160"},"body":"Shawn Pearce wrote:\n> Daniel Barkalow <barkalow@iabervon.org> wrote:\n[...]\n> I agree entirely.\n> \n> I have about 30 \"subprojects\" tacked into one large Git repository\n> for this exact reason.  In at least 5 of these cases they shouldn't\n> be sharing a Git repository as by all rights they are different\n> projects.\n> \n> What I'm doing is sort of like tacking both the Linux kernel and\n> glibc into the same Git repository because you might need to change\n> and bisect over updates to the system call layer.  Insane, yes.\n> Probably shouldn't be done; but right now that interface layer\n> between several subprojects is still in flux and it makes it rather\n> easy to keep everything in sync.\n> \n> Its annoying to perform commits to the \"root project\" every time the\n> subproject changes.  And it brings some complexity when you want to\n> talk about merging that root project.  But if its automated as part\n> of \"git commit\" and \"git merge\" (either directly in those tools or\n> by hooks users can install) then its probbaly a non issue.\n\nSo, for each subproject of a parent project, you want to record branch, \nversion (commit ID), and directory location. Not quite as easy to do in \na makefile but do-able.\n\nAn operation that _needs_ to change more than one project's versioned \nstate a time should be rare. If you have to do it often, then instead of \nsubprojects you probably have a partitioning of one project. A \nsubproject should be independent of its parent projects. A merge of a \nparent project should not affect a subproject other than to pick a \nparticular subproject version.\n\nYour example of the kernel and glibc is an example of sibling projects. \nEach one in independent and (some) versions of each project should work \n(better or worse) with the other. The root project here shouldn't really \ndo more than specify which version of the kernel and glibc to use.\n"},{"id":"27718","messageId":"4519AD9F.7020303@gmail.com","threadId":"5704","inReplyTo":"Pine.LNX.4.63.0609270009400.25371@wbgn013.biozentrum.uni-wuerzburg.de","subject":"Re: Notes on Using Git with Subprojects","fromName":"A Large Angry SCM","fromEmail":"gitzilla@gmail.com","sentAt":"2006-09-26T22:45:51Z","receivedAt":"2006-09-26T22:45:51Z","isPatch":false,"sender":{"key":"gitzilla@gmail.com","avatar":"https://gravatar.com/avatar/354625c442439908ff3dd99757dee330e29e9df7847472384faf7a00add247fb?d=mp&s=160"},"body":"Johannes Schindelin wrote:\n> Hi,\n> \n> On Tue, 26 Sep 2006, A Large Angry SCM wrote:\n> \n>> How the state (subproject list, branch names, etc.) is recorded in a \n>> parent project is only important to the parent project. The parent \n>> project must also know how to interact with with each of its \n>> subprojects.\n> \n> Granted, if you mix VCSes, this is most pragmatic.\n> \n> But it is also wrong: The whole point in bundling the subprojects together \n> is (IMHO) to get the benefits of a VCS for the root project, i.e. for the \n> combined states of the subprojects. After all, you want to say \"I know \n> that this collection of projects at these states compiled and worked \n> fine.\"\n> \n> And if you let a build system handle the stitching of the subprojects, you \n> completely lose these benefits.\n\nBundling and subproject support are two different things. Bundling is \nfor convenience. Subprojects are usually the result of a dependency on a \nproject managed or controlled by some other entity or on some part of \nthe larger project with radically different development requirements.\n\nRecording which version of a subproject to use is important and my note \nfailed to discuss it. That I'll remedy over the next several days.\n"},{"id":"27773","messageId":"20060927080652.GA8056@admingilde.org","threadId":"5704","inReplyTo":"4519AACD.7020508@gmail.com","subject":"Re: Notes on Using Git with Subprojects","fromName":"Martin Waitz","fromEmail":"tali@admingilde.org","sentAt":"2006-09-27T08:06:52Z","receivedAt":"2006-09-27T08:06:52Z","isPatch":false,"sender":{"key":"tali@admingilde.org","avatar":"https://gravatar.com/avatar/3f89b03eee362187effabe257898735b475673a12265c398ea9161259ae91553?d=mp&s=160"},"body":"hoi :)\n\nOn Tue, Sep 26, 2006 at 03:33:49PM -0700, A Large Angry SCM wrote:\n> So, for each subproject of a parent project, you want to record branch, \n> version (commit ID), and directory location. Not quite as easy to do in \n> a makefile but do-able.\n\nI've been playing with this kind of subprojects a little bit.\n\nMy current approach is like this:\n\n * create a .gitmodules file which lists all the directories\n   which contain a submodule.\n * the .git/refs/heads directory of the submodule gets stored in\n   .gitmodule/<modulename> inside the parent project\n * both things above should be tracked in the parent project.\n   This way you always store the current state of each submodule\n   in each commit of the parent project.  And you don't have to\n   create a new parent commit for each change.  You can commit\n   to the parent project when you think that all your modules are\n   in a good state.\n * When checking out a project, all submodules listen in .gitmodules\n   get checked out, too.\n * If there is a merge conflict in the module list or its refs/heads,\n   this is handled specially, e.g. by triggering a new merge inside\n   the submodule.\n * The object directory is shared between the parent and all modules.\n   To make fsck-objects happy, the parent gets a refs/module link\n   pointing to .gitmodule/ and all submodules get a refs/parent\n   link pointing to the refs directory of the parent.\n\nThe concept is similiar to the gitlink objects which have been floating\naround, but it is easier to prototype as no new git object type has to\nbe created.  If it works well we can later move the information stored\nin .gitmodule* into an object type of its own.\n\nBy storing the complete refs/heads directory for each submodule instead\nof only one head, it is possible to track multiple branches of a\nsubproject.  I'm don't know yet how this works out in praktice but I\nthink that it can be nice to be able to atomically commit to several\nbranches of one submodule (perhaps one branch per customer, per\nhardware platform, whatever).\n\nSo, what have I done up to now? Not much. I created a little\nscript to set up a submodule as described above:\nhttp://git.admingilde.org/?p=tali/git.git;a=blob;f=git-init-module.sh;h=0108873fd3aa8a42035039b19e8555513c075fca;hb=module\n\nNext steps would be to modify clone and checkout to actually be able\nto work in such a setup.  If this works then merging of subprojects\nhas to be done (the most complex part I guess).\n\n-- \nMartin Waitz\n"},{"id":"27779","messageId":"Pine.LNX.4.63.0609271152270.14200@wbgn013.biozentrum.uni-wuerzburg.de","threadId":"5704","inReplyTo":"20060927080652.GA8056@admingilde.org","subject":"Re: Notes on Using Git with Subprojects","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2006-09-27T09:55:22Z","receivedAt":"2006-09-27T09:55:22Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Wed, 27 Sep 2006, Martin Waitz wrote:\n\n> On Tue, Sep 26, 2006 at 03:33:49PM -0700, A Large Angry SCM wrote:\n> > So, for each subproject of a parent project, you want to record branch, \n> > version (commit ID), and directory location. Not quite as easy to do in \n> > a makefile but do-able.\n> \n> I've been playing with this kind of subprojects a little bit.\n> \n> My current approach is like this:\n> \n>  * create a .gitmodules file which lists all the directories\n>    which contain a submodule.\n>  * the .git/refs/heads directory of the submodule gets stored in\n>    .gitmodule/<modulename> inside the parent project\n\nTaking this a step further, you could make subproject/.git/refs/heads a \nsymbolic link to .git/refs/heads/subproject, with the benefit that fsck \nJust Works.\n\nNevertheless, you have to take care of the fact that you need to commit \nthe state of the root project just after committing to any subproject.\n\nCiao,\nDscho\n"},{"id":"27783","messageId":"20060927113813.GC8056@admingilde.org","threadId":"5704","inReplyTo":"Pine.LNX.4.63.0609271152270.14200@wbgn013.biozentrum.uni-wuerzburg.de","subject":"Re: Notes on Using Git with Subprojects","fromName":"Martin Waitz","fromEmail":"tali@admingilde.org","sentAt":"2006-09-27T11:38:13Z","receivedAt":"2006-09-27T11:38:13Z","isPatch":false,"sender":{"key":"tali@admingilde.org","avatar":"https://gravatar.com/avatar/3f89b03eee362187effabe257898735b475673a12265c398ea9161259ae91553?d=mp&s=160"},"body":"hoi :)\n\nOn Wed, Sep 27, 2006 at 11:55:22AM +0200, Johannes Schindelin wrote:\n> > My current approach is like this:\n> > \n> >  * create a .gitmodules file which lists all the directories\n> >    which contain a submodule.\n> >  * the .git/refs/heads directory of the submodule gets stored in\n> >    .gitmodule/<modulename> inside the parent project\n> \n> Taking this a step further, you could make subproject/.git/refs/heads a \n> symbolic link to .git/refs/heads/subproject, with the benefit that fsck \n> Just Works.\n\nin fact it is done this way (more or less).\n\n> Nevertheless, you have to take care of the fact that you need to commit \n> the state of the root project just after committing to any subproject.\n\nwhy?\n\nYou can accumulate as many changes in different subprojects until you\nget to a state that is worth committing in the parent project.\nAll these changes are then seen as one atomic change to the whole\nproject.\n\n-- \nMartin Waitz\n"},{"id":"27785","messageId":"Pine.LNX.4.63.0609271358100.14200@wbgn013.biozentrum.uni-wuerzburg.de","threadId":"5704","inReplyTo":"20060927113813.GC8056@admingilde.org","subject":"Re: Notes on Using Git with Subprojects","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2006-09-27T12:01:11Z","receivedAt":"2006-09-27T12:01:11Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi.\n\nOn Wed, 27 Sep 2006, Martin Waitz wrote:\n\n> On Wed, Sep 27, 2006 at 11:55:22AM +0200, Johannes Schindelin wrote:\n> > > My current approach is like this:\n> > > \n> > >  * create a .gitmodules file which lists all the directories\n> > >    which contain a submodule.\n> > >  * the .git/refs/heads directory of the submodule gets stored in\n> > >    .gitmodule/<modulename> inside the parent project\n> > \n> > Taking this a step further, you could make subproject/.git/refs/heads a \n> > symbolic link to .git/refs/heads/subproject, with the benefit that fsck \n> > Just Works.\n> \n> in fact it is done this way (more or less).\n\nWith the difference, that if you store the refs outside of \n<root>/.git/refs, you have to take extra care that prune does not delete \nthe corresponding objects.\n\n> > Nevertheless, you have to take care of the fact that you need to commit \n> > the state of the root project just after committing to any subproject.\n> \n> why?\n> \n> You can accumulate as many changes in different subprojects until you\n> get to a state that is worth committing in the parent project.\n> All these changes are then seen as one atomic change to the whole\n> project.\n\nAFAICT this is not the idea of subprojects-in-git. If you have to track \nthe subprojects in the root project manually anyway, you don't need _any_ \nadditional tool (you _can_ track files in a subdirectory containing a .git \nsubdirectory).\n\nCiao,\nDscho\n"},{"id":"27786","messageId":"20060927124413.GN1221MdfPADPa@greensroom.kotnet.org","threadId":"5704","inReplyTo":"Pine.LNX.4.63.0609271358100.14200@wbgn013.biozentrum.uni-wuerzburg.de","subject":"Re: Notes on Using Git with Subprojects","fromName":"Sven Verdoolaege","fromEmail":"skimo@kotnet.org","sentAt":"2006-09-27T12:44:13Z","receivedAt":"2006-09-27T12:44:13Z","isPatch":false,"sender":{"key":"skimo@kotnet.org","avatar":null},"body":"On Wed, Sep 27, 2006 at 02:01:11PM +0200, Johannes Schindelin wrote:\n> On Wed, 27 Sep 2006, Martin Waitz wrote:\n> > On Wed, Sep 27, 2006 at 11:55:22AM +0200, Johannes Schindelin wrote:\n> > > Nevertheless, you have to take care of the fact that you need to commit \n> > > the state of the root project just after committing to any subproject.\n\nSo what happens if you pull some changes into a subproject?\nAre you going to create a commit in the root project for each\nintermediate commit that you pulled into your subproject?\nIf no, then why should you do so if you happen to do these change\nin your local repo?\n\n> AFAICT this is not the idea of subprojects-in-git.\n\nAs already pointed out by Daniel, there is no such thing as\n\"the idea of subprojects-in-git\".  There are many ideas of\nsubprojects-in-git.\n\nI, for one, would want to commit the changed state of a subproject\nto the superproject explicitly.\n\n> If you have to track \n> the subprojects in the root project manually anyway, you don't need _any_ \n> additional tool (you _can_ track files in a subdirectory containing a .git \n> subdirectory).\n\nIf I switch to a different branch or bisect in the superproject,\nthen the states of the subprojects should be changed accordingly.\n\nskimo\n"},{"id":"27787","messageId":"20060927124604.GD8056@admingilde.org","threadId":"5704","inReplyTo":"Pine.LNX.4.63.0609271358100.14200@wbgn013.biozentrum.uni-wuerzburg.de","subject":"Re: Notes on Using Git with Subprojects","fromName":"Martin Waitz","fromEmail":"tali@admingilde.org","sentAt":"2006-09-27T12:46:04Z","receivedAt":"2006-09-27T12:46:04Z","isPatch":false,"sender":{"key":"tali@admingilde.org","avatar":"https://gravatar.com/avatar/3f89b03eee362187effabe257898735b475673a12265c398ea9161259ae91553?d=mp&s=160"},"body":"hoi :)\n\nOn Wed, Sep 27, 2006 at 02:01:11PM +0200, Johannes Schindelin wrote:\n> On Wed, 27 Sep 2006, Martin Waitz wrote:\n> > On Wed, Sep 27, 2006 at 11:55:22AM +0200, Johannes Schindelin wrote:\n> > > > My current approach is like this:\n> > > > \n> > > >  * create a .gitmodules file which lists all the directories\n> > > >    which contain a submodule.\n> > > >  * the .git/refs/heads directory of the submodule gets stored in\n> > > >    .gitmodule/<modulename> inside the parent project\n> > > \n> > > Taking this a step further, you could make subproject/.git/refs/heads a \n> > > symbolic link to .git/refs/heads/subproject, with the benefit that fsck \n> > > Just Works.\n> > \n> > in fact it is done this way (more or less).\n> \n> With the difference, that if you store the refs outside of \n> <root>/.git/refs, you have to take extra care that prune does not delete \n> the corresponding objects.\n\nthat's why there is .git/refs/module/modulname -> .gitmodule/modulename.\n\n> > You can accumulate as many changes in different subprojects until you\n> > get to a state that is worth committing in the parent project.\n> > All these changes are then seen as one atomic change to the whole\n> > project.\n> \n> AFAICT this is not the idea of subprojects-in-git. If you have to track \n> the subprojects in the root project manually anyway, you don't need _any_ \n> additional tool (you _can_ track files in a subdirectory containing a .git \n> subdirectory).\n\nBut then you loose the fine grained commits of your subprojects.\nYou only store the tree of the subproject when committing to the parent,\nnot the entire history.\n\nI think having the \"commit subproject changes to parent\" step as a\nmanual action makes sense in the same way as you have to trigger a\ncommit to a repository by hand, too. You are not storing every little\nchange to your filesystem in the database.\n\n-- \nMartin Waitz\n"},{"id":"27788","messageId":"Pine.LNX.4.63.0609271509240.14200@wbgn013.biozentrum.uni-wuerzburg.de","threadId":"5704","inReplyTo":"20060927124604.GD8056@admingilde.org","subject":"Re: Notes on Using Git with Subprojects","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2006-09-27T13:13:38Z","receivedAt":"2006-09-27T13:13:38Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Wed, 27 Sep 2006, Martin Waitz wrote:\n\n> On Wed, Sep 27, 2006 at 02:01:11PM +0200, Johannes Schindelin wrote:\n>\n> > AFAICT this is not the idea of subprojects-in-git. If you have to track \n> > the subprojects in the root project manually anyway, you don't need _any_ \n> > additional tool (you _can_ track files in a subdirectory containing a .git \n> > subdirectory).\n> \n> But then you loose the fine grained commits of your subprojects. You \n> only store the tree of the subproject when committing to the parent, not \n> the entire history.\n\nThe more I get told about subprojects (I don't have a use for them \nmyself), the more I think you are right: subprojects should not be \nintegrated deeply into git.\n\nCiao,\nDscho\n"},{"id":"27797","messageId":"451AADC3.40201@gmail.com","threadId":"5704","inReplyTo":"20060927080652.GA8056@admingilde.org","subject":"Re: Notes on Using Git with Subprojects","fromName":"A Large Angry SCM","fromEmail":"gitzilla@gmail.com","sentAt":"2006-09-27T16:58:43Z","receivedAt":"2006-09-27T16:58:43Z","isPatch":false,"sender":{"key":"gitzilla@gmail.com","avatar":"https://gravatar.com/avatar/354625c442439908ff3dd99757dee330e29e9df7847472384faf7a00add247fb?d=mp&s=160"},"body":"Martin Waitz wrote:\n[...]\n> My current approach is like this:\n> \n>  * create a .gitmodules file which lists all the directories\n>    which contain a submodule.\n>  * the .git/refs/heads directory of the submodule gets stored in\n>    .gitmodule/<modulename> inside the parent project\n>  * both things above should be tracked in the parent project.\n>    This way you always store the current state of each submodule\n>    in each commit of the parent project.  And you don't have to\n>    create a new parent commit for each change.  You can commit\n>    to the parent project when you think that all your modules are\n>    in a good state.\n\nThis means that modules are not separate, stand alone projects but, \nrather, just a sub part of your bigger project. Very useful and \napplicable in some situations but other situations want/need separate, \nstand alone subprojects.\n\n>  * When checking out a project, all submodules listen in .gitmodules\n>    get checked out, too.\n>  * If there is a merge conflict in the module list or its refs/heads,\n>    this is handled specially, e.g. by triggering a new merge inside\n>    the submodule.\n>  * The object directory is shared between the parent and all modules.\n>    To make fsck-objects happy, the parent gets a refs/module link\n>    pointing to .gitmodule/ and all submodules get a refs/parent\n>    link pointing to the refs directory of the parent.\n> \n[...]\n> By storing the complete refs/heads directory for each submodule instead\n> of only one head, it is possible to track multiple branches of a\n> subproject.  I'm don't know yet how this works out in praktice but I\n> think that it can be nice to be able to atomically commit to several\n> branches of one submodule (perhaps one branch per customer, per\n> hardware platform, whatever).\n\nIt's not immediately clear to me if tracking several long term \n(globally) visible branches in a checkout sub module is generally useful \nor only useful in special situations. I need to think about this...\n\n[...]\n\nYou solved a similar problem to the one I'm working on; and one that is \napplicable to a number of projects. Namely, projects where all the parts \nare under the control of the same entity. For projects looking to escape \n  CVS, that use CVS modules, this looks like a Git solution.\n\nThe problem I'm working on is how to deal with the sub parts of a larger \nproject when those sub parts are controlled by different entity. Silly \nexample: the kernel is controlled by Linus; glibc is controlled by the \nGNU folks, gcc is controlled by some other GNU folks, the web server is \ncontrolled by the Apache Foundation, the X server is controlled by Xorg, \netc.\n"},{"id":"27798","messageId":"451AB122.9020600@gmail.com","threadId":"5704","inReplyTo":"Pine.LNX.4.63.0609271152270.14200@wbgn013.biozentrum.uni-wuerzburg.de","subject":"Re: Notes on Using Git with Subprojects","fromName":"A Large Angry SCM","fromEmail":"gitzilla@gmail.com","sentAt":"2006-09-27T17:13:06Z","receivedAt":"2006-09-27T17:13:06Z","isPatch":false,"sender":{"key":"gitzilla@gmail.com","avatar":"https://gravatar.com/avatar/354625c442439908ff3dd99757dee330e29e9df7847472384faf7a00add247fb?d=mp&s=160"},"body":"Johannes Schindelin wrote:\n> Hi,\n> \n> On Wed, 27 Sep 2006, Martin Waitz wrote:\n> \n>> On Tue, Sep 26, 2006 at 03:33:49PM -0700, A Large Angry SCM wrote:\n>>> So, for each subproject of a parent project, you want to record branch, \n>>> version (commit ID), and directory location. Not quite as easy to do in \n>>> a makefile but do-able.\n>> I've been playing with this kind of subprojects a little bit.\n>>\n>> My current approach is like this:\n>>\n>>  * create a .gitmodules file which lists all the directories\n>>    which contain a submodule.\n>>  * the .git/refs/heads directory of the submodule gets stored in\n>>    .gitmodule/<modulename> inside the parent project\n> \n> Taking this a step further, you could make subproject/.git/refs/heads a \n> symbolic link to .git/refs/heads/subproject, with the benefit that fsck \n> Just Works.\n\nWouldn't an fsck in the parent complain about missing objects?\n"},{"id":"27799","messageId":"20060927173335.GC2807@coredump.intra.peff.net","threadId":"5704","inReplyTo":"451AADC3.40201@gmail.com","subject":"Re: Notes on Using Git with Subprojects","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2006-09-27T17:33:35Z","receivedAt":"2006-09-27T17:33:35Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Sep 27, 2006 at 09:58:43AM -0700, A Large Angry SCM wrote:\n\n> This means that modules are not separate, stand alone projects but, \n> rather, just a sub part of your bigger project. Very useful and \n> applicable in some situations but other situations want/need separate, \n> stand alone subprojects.\n\nOne thing that I believe some people have requested for subprojects is\nto avoid downloading files/history for subprojects you're not interested\nin.  I think this could be faciliated in this scheme by only cloning the\nheads of the subprojects you're interested in (there would need to be\nspecial machinery to handle this at the root level if we want to allow\nmaking root commits without necessarily having all of the subprojects).\n\nA first step to this would be an argument to git-clone to allow cloning\nonly a subset of refs.\n\n> The problem I'm working on is how to deal with the sub parts of a larger \n> project when those sub parts are controlled by different entity. Silly \n> example: the kernel is controlled by Linus; glibc is controlled by the \n> GNU folks, gcc is controlled by some other GNU folks, the web server is \n> controlled by the Apache Foundation, the X server is controlled by Xorg, \n> etc.\n\nThe nice thing about this approach is that I believe the two systems\nneed only differ at clone time. Instead of creating a remotes file with\nall of the subproject branches, you would just create multiple remotes\nfiles. The root fetching porcelain needs to be smart enough to fetch all\nremotes.\n\n-Peff\n"},{"id":"27817","messageId":"7vhcyt81gn.fsf@assigned-by-dhcp.cox.net","threadId":"5704","inReplyTo":"20060927124413.GN1221MdfPADPa@greensroom.kotnet.org","subject":"Re: Notes on Using Git with Subprojects","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2006-09-27T21:05:28Z","receivedAt":"2006-09-27T21:05:28Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Sven Verdoolaege <skimo@kotnet.org> writes:\n\n> On Wed, Sep 27, 2006 at 02:01:11PM +0200, Johannes Schindelin wrote:\n>\n>> AFAICT this is not the idea of subprojects-in-git.\n>\n> As already pointed out by Daniel, there is no such thing as\n> \"the idea of subprojects-in-git\".  There are many ideas of\n> subprojects-in-git.\n>\n> I, for one, would want to commit the changed state of a subproject\n> to the superproject explicitly.\n\nI think this is a very good summary of the point in this entire\nthread.  Different workflows call for different granularity, and\nif something deserves to be called \"subproject\", not just \"a\nsubdirectory of a single project\", it is not unreasonable to\nthink that it would want to track its own state at different\npace from the other parts of the project, and at finer grain\nthan the project taken as the whole.\n\nNot allowing subprojects to do so means every little change\nanywhere in the project tree results in a tree-wide new commit\nobject; in that case, the whole thing is a single large project\nfrom core-git's point of view.\n\nAvoiding checking out parts of the project tree that you do not\ncare about while you work on such a single large project is\nanother interesting and useful area to think about, but I would\nsay at that point it is not about subproject at all -- it is\nabout working in a sparsely populated working tree of a single\nproject.\n\nXCB team recently told us that they started from such a single\nlarge project and now they are splitting that into separate\npieces.  Their experiences may be valuable to be shared to\ndiscuss pros-and-cons of these two approaches.\n"},{"id":"27830","messageId":"Pine.LNX.4.63.0609280113490.14200@wbgn013.biozentrum.uni-wuerzburg.de","threadId":"5704","inReplyTo":"451AB122.9020600@gmail.com","subject":"Re: Notes on Using Git with Subprojects","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2006-09-27T23:14:33Z","receivedAt":"2006-09-27T23:14:33Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Wed, 27 Sep 2006, A Large Angry SCM wrote:\n\n> Johannes Schindelin wrote:\n> > Hi,\n> > \n> > On Wed, 27 Sep 2006, Martin Waitz wrote:\n> > \n> > > On Tue, Sep 26, 2006 at 03:33:49PM -0700, A Large Angry SCM wrote:\n> > > > So, for each subproject of a parent project, you want to record branch,\n> > > > version (commit ID), and directory location. Not quite as easy to do in\n> > > > a makefile but do-able.\n> > > I've been playing with this kind of subprojects a little bit.\n> > > \n> > > My current approach is like this:\n> > > \n> > >  * create a .gitmodules file which lists all the directories\n> > >    which contain a submodule.\n> > >  * the .git/refs/heads directory of the submodule gets stored in\n> > >    .gitmodule/<modulename> inside the parent project\n> > \n> > Taking this a step further, you could make subproject/.git/refs/heads a\n> > symbolic link to .git/refs/heads/subproject, with the benefit that fsck Just\n> > Works.\n> \n> Wouldn't an fsck in the parent complain about missing objects?\n\n... not if my original idea (which I might have forgotten to mention ;-) \nwas implemented: symlinking subproject/.git/objects to .git/objects.\n\nCiao,\nDscho\n"},{"id":"27831","messageId":"20060927233639.GE21839@spearce.org","threadId":"5704","inReplyTo":"Pine.LNX.4.63.0609280113490.14200@wbgn013.biozentrum.uni-wuerzburg.de","subject":"Re: Notes on Using Git with Subprojects","fromName":"Shawn Pearce","fromEmail":"spearce@spearce.org","sentAt":"2006-09-27T23:36:39Z","receivedAt":"2006-09-27T23:36:39Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:\n> On Wed, 27 Sep 2006, A Large Angry SCM wrote:\n> > Wouldn't an fsck in the parent complain about missing objects?\n> \n> ... not if my original idea (which I might have forgotten to mention ;-) \n> was implemented: symlinking subproject/.git/objects to .git/objects.\n\nRight.  Which is one of the truely wonderful things about symlinks\nin .git/refs and symlinking .git/objects. :-)\n\n\nI don't know about anyone else but this thread has certainly helped\nme rationalize a few thoughts about \"subproject\" support.\n\nThe major things I've taken away from it are:\n\n - Subprojects of any reasonable SCM should be supported.\n\n   Although this is Git we sometimes want to play nice with other\n   people working in the same pond.  We have a historical track\n   record of doing this when it makes sense (git-svn, git-cvsserver,\n   etc.) but clearly doing it for every SCM out there is not\n   possible.\n\n   But that said having \"out of the box\" support for Git subprojects\n   within a larger Git project should Just Work.  It doesn't\n   really yet.\n\n - Higher level projects should drive subprojects.\n\n   Higher level projects tend to be composed of specific revisions or\n   specific generations of subprojects.\n\n   Part of the content of the higher level project is just what\n   those subproject specifications are and how those subprojects\n   should appear in a working directory.\n\n - Git Porcelain should help the user.\n\n   Git operations should translate down through lower level projects\n   when possible, and lower level project changes should push up\n   when possible.\n\n   E.g. git-fetch in a higher level repository should percolate\n   down into the lower level repositories automatically.  Ditto with\n   git-checkout and probably git-push.  git-commit in a lower level\n   repository probably should update the specification file(s)\n   in the higher level repository but not commit the higher level\n   repository.\n\n - The subproject SCM interface needs to be modular.\n\n   Users need to include many different subprojects and not all of\n   them use Git.  Ideally Git would be able to at least be easily\n   taught by the user how to invoke a particular subproject's SCM\n   for the purpose of an initial checkout, if not for additional\n   operations such as pull, push, commit and tag.\n\n\nI used the term \"generation of subprojects\" as not everyone wants\nto bind their root project to a specific revision of a subproject.\nIndeed that may not be entirely practical.  Instead just a particular\nlineage of development (e.g. \"Version 1.0\" vs. \"Version 1.2\")\nmay be all that is needed.\n\nFor example including a CVS hosted subproject into a larger Git\nproject means you can't use a specific SHA1 to reference a single\nversion of that CVS subproject.  Yet you can use a CVS branch\nor label name.  But both of those are moving targets in a CVS\nrepository.  But in a sane CVS project a label or a branch will\nbe relatively stable over time, meaning that its good enough given\nthat its all we got (without importing everything into Git anyway).\n\nLikewise a Git project including a Git subproject should be able\nto reference a named tag of the subproject.  If the subproject\nchanges its tag and the fetcher agrees to the change (with a\n--force) then its OK for the subproject to follow that tag change.\nLikewise it should be acceptable for a subproject to reference a\nspecific branch head.  Although this is a moving target that may be\nacceptable while the higher level project is under rapid development.\n\nHowever an annoted tag probably should not be able to be created\non the higher level project unless all lower-level subprojects\nare referenced by tags (or the equivilant) in their SCM.  Which\nimplies using a \"stable tag\" in CVS, a \"/tags/foo@rev\" in SVN,\nor an annotated tag in Git and updating the specification file(s)\nto reflect that.\n\n-- \nShawn.\n"},{"id":"27832","messageId":"451B0F59.6070901@dawes.za.net","threadId":"5704","inReplyTo":"20060927233639.GE21839@spearce.org","subject":"Re: Notes on Using Git with Subprojects","fromName":"Rogan Dawes","fromEmail":"lists@dawes.za.net","sentAt":"2006-09-27T23:55:05Z","receivedAt":"2006-09-27T23:55:05Z","isPatch":false,"sender":{"key":"lists@dawes.za.net","avatar":null},"body":"Shawn Pearce wrote:\n\n> \n>  - Higher level projects should drive subprojects.\n> \n>    Higher level projects tend to be composed of specific revisions or\n>    specific generations of subprojects.\n> \n>    Part of the content of the higher level project is just what\n>    those subproject specifications are and how those subprojects\n>    should appear in a working directory.\n> \n\n> \n> I used the term \"generation of subprojects\" as not everyone wants\n> to bind their root project to a specific revision of a subproject.\n> Indeed that may not be entirely practical.  Instead just a particular\n> lineage of development (e.g. \"Version 1.0\" vs. \"Version 1.2\")\n> may be all that is needed.\n> \n\nDoes it not make sense that a commit of the higher level project should \ninclude the contents of its subprojects at that particular moment in time?\n\ne.g. using the previous example of a kernel, apache, glibc, etc\n\nYou may track the subprojects using whatever scm applies to THAT \nsubproject. But when you want to record the state of the entire project, \nyou want to include the state of the subprojects. So, your super-project \ncommit would actually recurse down into the working directories of the \nsubprojects and record the state/contents of each file that makes up \neach of the subprojects.\n\nSo, if someone is tracking the overall project, and they do a pull of \nv1.1 (tag), they will see exactly what v1.1 looked like in your repo.\n\nWhat this makes me think is that it might be useful to have a mechanism \nfor recalculating the tree-ish of a subdirectory and finding an \nassociated commit, for the case where a subproject is also managed by git.\n\ni.e. given a super-project in this state, and knowing that this \nsubproject is managed by git, which revision of the subproject are we \ntalking about, and can we find a commit that matches this tree-ish? \n(assuming we have the history of the subproject available, of course)\n\nRegards,\n\nRogan\n"},{"id":"27842","messageId":"20060928003658.GB22593@spearce.org","threadId":"5704","inReplyTo":"451B0F59.6070901@dawes.za.net","subject":"Re: Notes on Using Git with Subprojects","fromName":"Shawn Pearce","fromEmail":"spearce@spearce.org","sentAt":"2006-09-28T00:36:58Z","receivedAt":"2006-09-28T00:36:58Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Rogan Dawes <lists@dawes.za.net> wrote:\n> Shawn Pearce wrote:\n> \n> >\n> > - Higher level projects should drive subprojects.\n> >\n> >   Higher level projects tend to be composed of specific revisions or\n> >   specific generations of subprojects.\n> >\n> >   Part of the content of the higher level project is just what\n> >   those subproject specifications are and how those subprojects\n> >   should appear in a working directory.\n> >\n> \n> >\n> >I used the term \"generation of subprojects\" as not everyone wants\n> >to bind their root project to a specific revision of a subproject.\n> >Indeed that may not be entirely practical.  Instead just a particular\n> >lineage of development (e.g. \"Version 1.0\" vs. \"Version 1.2\")\n> >may be all that is needed.\n> >\n> \n> Does it not make sense that a commit of the higher level project should \n> include the contents of its subprojects at that particular moment in time?\n\nIf the subproject is a Git repository we can simply record a commit\nSHA1 or a tag SHA1 and assuming we can locate the objects for that\nrepository in the future we can fully recover that subproject.\nNo subtree tracking necessary.\n\nIf the subproject is an SVN repository we can simply record\nthe branch/tag path and the specific revision number and fully\nrecover that subproject, assuming we have access to the central\nSVN repository.  If we don't or we are concerned about that then we\nwould want to use git-svn/git-svnimport to mirror the SVN repository\nand treat it as a proper Git subproject.\n\nIf the subproject is a Monotone or Darcs project I believe we could\nalso store some small token for a unique directory state of that\nsubproject and recover that from Monotone or Darcs without importing\nthe entire subproject into Git.  If you don't trust Monotone or Darcs\nthen maybe you should be importing/mirroring that project into Git.\nAt which point its a proper Git subproject.\n\nIf the subproject is CVS then you either need to trust the CVS\ntag or import the tree into Git.  If you are going to import the\ntree into Git then you might as well just treat the subproject as\na proper Git subproject and use some sort of CVS import or mirror\nto keep that in sync.\n\nEtc...\n\nMy basic point here is that a subproject already has its own SCM\nand if its not Git you probably either are going to convert it to\nGit via a Git bridge (git-svn anyone?) or you are going to keep it\nin its \"native\" SCM as you probably don't want to incur the disk\ncosts of storing it twice (once in its native SCM and once in Git).\n\nSo subtree tracking is probably unncessary.\n\n-- \nShawn.\n"},{"id":"27861","messageId":"451B45D6.1010006@gmail.com","threadId":"5704","inReplyTo":"20060927173335.GC2807@coredump.intra.peff.net","subject":"Re: Notes on Using Git with Subprojects","fromName":"A Large Angry SCM","fromEmail":"gitzilla@gmail.com","sentAt":"2006-09-28T03:47:34Z","receivedAt":"2006-09-28T03:47:34Z","isPatch":false,"sender":{"key":"gitzilla@gmail.com","avatar":"https://gravatar.com/avatar/354625c442439908ff3dd99757dee330e29e9df7847472384faf7a00add247fb?d=mp&s=160"},"body":"Jeff King wrote:\n[...]\n> One thing that I believe some people have requested for subprojects is\n> to avoid downloading files/history for subprojects you're not interested\n> in.  I think this could be faciliated in this scheme by only cloning the\n> heads of the subprojects you're interested in (there would need to be\n> special machinery to handle this at the root level if we want to allow\n> making root commits without necessarily having all of the subprojects).\n\nIn what I'm suggesting, commits are local to a project's working \ndirectory repository and are pushed somewhere else to be recorded long \nterm. Since projects are stand alone, possibly with dependencies, \nworking on a (sub)project without having other associated (sub)projects \nis accomplished by checking it out.\n\n> A first step to this would be an argument to git-clone to allow cloning\n> only a subset of refs.\n\nSomething like this?\n\n\tgit-init-db\n\tgit-fetch <repository> <refspecs>\n"},{"id":"27862","messageId":"20060928035225.GA22006@coredump.intra.peff.net","threadId":"5704","inReplyTo":"451B45D6.1010006@gmail.com","subject":"Re: Notes on Using Git with Subprojects","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2006-09-28T03:52:25Z","receivedAt":"2006-09-28T03:52:25Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Sep 27, 2006 at 08:47:34PM -0700, A Large Angry SCM wrote:\n\n> >A first step to this would be an argument to git-clone to allow cloning\n> >only a subset of refs.\n> Something like this?\n> \n> \tgit-init-db\n> \tgit-fetch <repository> <refspecs>\n\nExactly, but I was suggesting something more user-friendly (e.g., it's\nnice to use git-clone because it creates the remotes file). I was going\nto hack up a quick change to git-clone, but I think some thought needs\nto be given to semantics, especially with respect to tags (should it\nimply no tags? Only tags which point to refs we're already fetching?).\n\n-Peff\n"},{"id":"27863","messageId":"20060928035238.GC22897@spearce.org","threadId":"5704","inReplyTo":"451B45D6.1010006@gmail.com","subject":"Re: Notes on Using Git with Subprojects","fromName":"Shawn Pearce","fromEmail":"spearce@spearce.org","sentAt":"2006-09-28T03:52:38Z","receivedAt":"2006-09-28T03:52:38Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"A Large Angry SCM <gitzilla@gmail.com> wrote:\n> Jeff King wrote:\n> [...]\n> >One thing that I believe some people have requested for subprojects is\n> >to avoid downloading files/history for subprojects you're not interested\n> >in.  I think this could be faciliated in this scheme by only cloning the\n> >heads of the subprojects you're interested in (there would need to be\n> >special machinery to handle this at the root level if we want to allow\n> >making root commits without necessarily having all of the subprojects).\n> \n> In what I'm suggesting, commits are local to a project's working \n> directory repository and are pushed somewhere else to be recorded long \n> term. Since projects are stand alone, possibly with dependencies, \n> working on a (sub)project without having other associated (sub)projects \n> is accomplished by checking it out.\n> \n> >A first step to this would be an argument to git-clone to allow cloning\n> >only a subset of refs.\n> \n> Something like this?\n> \n> \tgit-init-db\n> \tgit-fetch <repository> <refspecs>\n\nMore like:\n\n \tgit-init-db\n \tgit-fetch --keep <repository> <refspecs>\n\nbut yes.  :-)\n\n-- \nShawn.\n"},{"id":"27864","messageId":"20060928035855.GD22897@spearce.org","threadId":"5704","inReplyTo":"20060928035225.GA22006@coredump.intra.peff.net","subject":"Re: Notes on Using Git with Subprojects","fromName":"Shawn Pearce","fromEmail":"spearce@spearce.org","sentAt":"2006-09-28T03:58:55Z","receivedAt":"2006-09-28T03:58:55Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Jeff King <peff@peff.net> wrote:\n> On Wed, Sep 27, 2006 at 08:47:34PM -0700, A Large Angry SCM wrote:\n> \n> > >A first step to this would be an argument to git-clone to allow cloning\n> > >only a subset of refs.\n> > Something like this?\n> > \n> > \tgit-init-db\n> > \tgit-fetch <repository> <refspecs>\n> \n> Exactly, but I was suggesting something more user-friendly (e.g., it's\n> nice to use git-clone because it creates the remotes file). I was going\n> to hack up a quick change to git-clone, but I think some thought needs\n> to be given to semantics, especially with respect to tags (should it\n> imply no tags? Only tags which point to refs we're already fetching?).\n\nIf you are fetching a set of commits from a repository you probably\nshould be fetching any tags that point at the commits you've fetched.\nThey tend to be few compared to the commits, they tend to be small,\nand they tend to be important milestones in the tracked project.\n\nI think that's why the native Git protocol sends tags for any\ncommits that were also sent.  :)\n\n-- \nShawn.\n"},{"id":"27865","messageId":"20060928040027.GA22489@coredump.intra.peff.net","threadId":"5704","inReplyTo":"20060928035855.GD22897@spearce.org","subject":"Re: Notes on Using Git with Subprojects","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2006-09-28T04:00:27Z","receivedAt":"2006-09-28T04:00:27Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Sep 27, 2006 at 11:58:55PM -0400, Shawn Pearce wrote:\n\n> If you are fetching a set of commits from a repository you probably\n> should be fetching any tags that point at the commits you've fetched.\n> They tend to be few compared to the commits, they tend to be small,\n> and they tend to be important milestones in the tracked project.\n> \n> I think that's why the native Git protocol sends tags for any\n> commits that were also sent.  :)\n\nOh, that's clever. :)\n\nDo we do the right thing for non-git transports?\n\n-Peff\n"},{"id":"27868","messageId":"20060928040931.GE22897@spearce.org","threadId":"5704","inReplyTo":"20060928040027.GA22489@coredump.intra.peff.net","subject":"Re: Notes on Using Git with Subprojects","fromName":"Shawn Pearce","fromEmail":"spearce@spearce.org","sentAt":"2006-09-28T04:09:31Z","receivedAt":"2006-09-28T04:09:31Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Jeff King <peff@peff.net> wrote:\n> On Wed, Sep 27, 2006 at 11:58:55PM -0400, Shawn Pearce wrote:\n> \n> > If you are fetching a set of commits from a repository you probably\n> > should be fetching any tags that point at the commits you've fetched.\n> > They tend to be few compared to the commits, they tend to be small,\n> > and they tend to be important milestones in the tracked project.\n> > \n> > I think that's why the native Git protocol sends tags for any\n> > commits that were also sent.  :)\n> \n> Oh, that's clever. :)\n> \n> Do we do the right thing for non-git transports?\n\nYes, I think we do.\n\nOnly its not quite as clever as the HTTP/FTP commit walker first\nneeds to get a list of available refs (which includes tag and\ntag^{}) and compares each obtained commit to the ^{} entries.\nIf there's a match it gets the tag.\n\nAnd rsync being as dumb as it is should be fetching everything. :)\n\n-- \nShawn.\n"},{"id":"27870","messageId":"451B542F.8060205@gmail.com","threadId":"5704","inReplyTo":"20060927233639.GE21839@spearce.org","subject":"Re: Notes on Using Git with Subprojects","fromName":"A Large Angry SCM","fromEmail":"gitzilla@gmail.com","sentAt":"2006-09-28T04:48:47Z","receivedAt":"2006-09-28T04:48:47Z","isPatch":false,"sender":{"key":"gitzilla@gmail.com","avatar":"https://gravatar.com/avatar/354625c442439908ff3dd99757dee330e29e9df7847472384faf7a00add247fb?d=mp&s=160"},"body":"Shawn Pearce wrote:\n[...]\n>  - Git Porcelain should help the user.\n> \n>    Git operations should translate down through lower level projects\n>    when possible, and lower level project changes should push up\n>    when possible.\n> \n>    E.g. git-fetch in a higher level repository should percolate\n>    down into the lower level repositories automatically.  Ditto with\n>    git-checkout and probably git-push.  git-commit in a lower level\n>    repository probably should update the specification file(s)\n>    in the higher level repository but not commit the higher level\n>    repository.\n\nI think recursing through all subprojects for most Git commands is \nactually the exception. Plus, porcelains aren't going to help much past \nthe first subproject that isn't Git managed.\n\n[...]\n\n> However an annoted tag probably should not be able to be created\n> on the higher level project unless all lower-level subprojects\n> are referenced by tags (or the equivilant) in their SCM.  Which\n> implies using a \"stable tag\" in CVS, a \"/tags/foo@rev\" in SVN,\n> or an annotated tag in Git and updating the specification file(s)\n> to reflect that.\n\nI fail to see a reason for this restriction. Each project should be \nmanaged separately. Also, how do you enforce the restriction on other VCSs?\n"},{"id":"27874","messageId":"451B5783.5080107@gmail.com","threadId":"5704","inReplyTo":"451B0F59.6070901@dawes.za.net","subject":"Re: Notes on Using Git with Subprojects","fromName":"A Large Angry SCM","fromEmail":"gitzilla@gmail.com","sentAt":"2006-09-28T05:02:59Z","receivedAt":"2006-09-28T05:02:59Z","isPatch":false,"sender":{"key":"gitzilla@gmail.com","avatar":"https://gravatar.com/avatar/354625c442439908ff3dd99757dee330e29e9df7847472384faf7a00add247fb?d=mp&s=160"},"body":"Rogan Dawes wrote:\n[...]\n> Does it not make sense that a commit of the higher level project should \n> include the contents of its subprojects at that particular moment in time?\n> \n> e.g. using the previous example of a kernel, apache, glibc, etc\n> \n> You may track the subprojects using whatever scm applies to THAT \n> subproject. But when you want to record the state of the entire project, \n> you want to include the state of the subprojects. So, your super-project \n> commit would actually recurse down into the working directories of the \n> subprojects and record the state/contents of each file that makes up \n> each of the subprojects.\n> \n> So, if someone is tracking the overall project, and they do a pull of \n> v1.1 (tag), they will see exactly what v1.1 looked like in your repo.\n> \n> What this makes me think is that it might be useful to have a mechanism \n> for recalculating the tree-ish of a subdirectory and finding an \n> associated commit, for the case where a subproject is also managed by git.\n> \n> i.e. given a super-project in this state, and knowing that this \n> subproject is managed by git, which revision of the subproject are we \n> talking about, and can we find a commit that matches this tree-ish? \n> (assuming we have the history of the subproject available, of course)\n\nSome development environments will require that all the (used) code is \nimported into the local VCS of choice. But not all environments. For \nsome development environments, recording the version of the subproject \nis sufficient. Assuming it's possible at some future time to get the \nstate associated with the version.\n\nAlso keep in mind, to effectively participate in a project, you will \nlikely need to use the VCS of the project. So importing everything into \nanother VCS (Git) will just cause _you_ more work.\n"},{"id":"27877","messageId":"20060928073706.GE8056@admingilde.org","threadId":"5704","inReplyTo":"451AADC3.40201@gmail.com","subject":"Re: Notes on Using Git with Subprojects","fromName":"Martin Waitz","fromEmail":"tali@admingilde.org","sentAt":"2006-09-28T07:37:06Z","receivedAt":"2006-09-28T07:37:06Z","isPatch":false,"sender":{"key":"tali@admingilde.org","avatar":"https://gravatar.com/avatar/3f89b03eee362187effabe257898735b475673a12265c398ea9161259ae91553?d=mp&s=160"},"body":"hoi :)\n\nOn Wed, Sep 27, 2006 at 09:58:43AM -0700, A Large Angry SCM wrote:\n> This means that modules are not separate, stand alone projects but, \n> rather, just a sub part of your bigger project. Very useful and \n> applicable in some situations but other situations want/need separate, \n> stand alone subprojects.\n\nyou can do everything with the submodule which would be possible with\na normal GIT repository.  And you can always clone it into an directory\nwhich is not controlled by a parent project.\n\nI really think that this is an very important property of a submodule.\n\n> >By storing the complete refs/heads directory for each submodule instead\n> >of only one head, it is possible to track multiple branches of a\n> >subproject.  I'm don't know yet how this works out in praktice but I\n> >think that it can be nice to be able to atomically commit to several\n> >branches of one submodule (perhaps one branch per customer, per\n> >hardware platform, whatever).\n> \n> It's not immediately clear to me if tracking several long term \n> (globally) visible branches in a checkout sub module is generally useful \n> or only useful in special situations. I need to think about this...\n\nOne use-case which may be important here:\n\nThe submodule has two different branches which got forked and are not\nintended to be merged again.  At some point in time the parent project\nwants to switch from one branch of the submodule to another branch.\nIf a user still has modifications in the old branch and wants to\nupdate the parent project then it is important to know if the local\nmodifications and those coming from the parent have to be merged or\nshould stay in different branches.\nIf the parent is switching branches there should only be some warning\nif the user still has modifications in the old branch, giving him the\nchance to port the modifications to the other branch.\n\n-- \nMartin Waitz\n"},{"id":"27889","messageId":"20060928150224.GE28790@mellanox.co.il","threadId":"5704","inReplyTo":"7vhcyt81gn.fsf@assigned-by-dhcp.cox.net","subject":"Re: Notes on Using Git with Subprojects","fromName":"Michael S. Tsirkin","fromEmail":"mst@mellanox.co.il","sentAt":"2006-09-28T15:02:25Z","receivedAt":"2006-09-28T15:02:25Z","isPatch":false,"sender":{"key":"mst@kernel.org","avatar":null},"body":"Quoting r. Junio C Hamano <junkio@cox.net>:\n> Avoiding checking out parts of the project tree that you do not\n> care about while you work on such a single large project is\n> another interesting and useful area to think about, but I would\n> say at that point it is not about subproject at all -- it is\n> about working in a sparsely populated working tree of a single\n> project.\n\nI agree completely - at least as far as I'm concerned, working in\na sparsely populated working tree is what it's all about.\nFor example, sometimes I am just editing documentation and\nit would be nice \n\nIt's easy to check out just a subdirectory the first time:\n>git checkout master `git-ls-tree -r --name-only master subdirectory`\n>echo ref: refs/heads/master > .git/HEAD\nbut when you try a pull/rebase git will check out all of the tree.\n\nIs there some way to avoid this?\n\n-- \nMST\n"},{"id":"27892","messageId":"Pine.LNX.4.63.0609281738120.14200@wbgn013.biozentrum.uni-wuerzburg.de","threadId":"5704","inReplyTo":"20060928035238.GC22897@spearce.org","subject":"Re: Notes on Using Git with Subprojects","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2006-09-28T15:39:15Z","receivedAt":"2006-09-28T15:39:15Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Wed, 27 Sep 2006, Shawn Pearce wrote:\n\n> A Large Angry SCM <gitzilla@gmail.com> wrote:\n> > Jeff King wrote:\n> > [...]\n> > >One thing that I believe some people have requested for subprojects is\n> > >to avoid downloading files/history for subprojects you're not interested\n> > >in.  I think this could be faciliated in this scheme by only cloning the\n> > >heads of the subprojects you're interested in (there would need to be\n> > >special machinery to handle this at the root level if we want to allow\n> > >making root commits without necessarily having all of the subprojects).\n> > \n> > In what I'm suggesting, commits are local to a project's working \n> > directory repository and are pushed somewhere else to be recorded long \n> > term. Since projects are stand alone, possibly with dependencies, \n> > working on a (sub)project without having other associated (sub)projects \n> > is accomplished by checking it out.\n> > \n> > >A first step to this would be an argument to git-clone to allow cloning\n> > >only a subset of refs.\n> > \n> > Something like this?\n> > \n> > \tgit-init-db\n> > \tgit-fetch <repository> <refspecs>\n> \n> More like:\n> \n>  \tgit-init-db\n>  \tgit-fetch --keep <repository> <refspecs>\n> \n> but yes.  :-)\n\nYou are missing the remotes/ information:\n\ngit-repo-config remote.origin.url <repository>\nfor spec in <refspecs>; do\n\tgit-repo-config remote.origin.fetch $spec ^$\ndone\n\nCiao,\nDscho\n"},{"id":"27929","messageId":"20060928201636.GA4386@coredump.intra.peff.net","threadId":"5704","inReplyTo":"7vhcyt81gn.fsf@assigned-by-dhcp.cox.net","subject":"Re: Notes on Using Git with Subprojects","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2006-09-28T20:16:36Z","receivedAt":"2006-09-28T20:16:36Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Sep 27, 2006 at 02:05:28PM -0700, Junio C Hamano wrote:\n\n> Avoiding checking out parts of the project tree that you do not\n> care about while you work on such a single large project is\n> another interesting and useful area to think about, but I would\n> say at that point it is not about subproject at all -- it is\n> about working in a sparsely populated working tree of a single\n> project.\n\nKeep in mind that it might not be an attempt to avoid checking out part\nof the tree, but rather importing part of the tree (the subproject) into\nyour repository at all (to save space, download time, etc). So unless\nyou're also proposing sparse repos, I think this still might be a\nsubproject issue.\n\n-Peff\n"},{"id":"27930","messageId":"451C30EF.8050305@gmail.com","threadId":"5704","inReplyTo":"20060928073706.GE8056@admingilde.org","subject":"Re: Notes on Using Git with Subprojects","fromName":"A Large Angry SCM","fromEmail":"gitzilla@gmail.com","sentAt":"2006-09-28T20:30:39Z","receivedAt":"2006-09-28T20:30:39Z","isPatch":false,"sender":{"key":"gitzilla@gmail.com","avatar":"https://gravatar.com/avatar/354625c442439908ff3dd99757dee330e29e9df7847472384faf7a00add247fb?d=mp&s=160"},"body":"Martin Waitz wrote:\n> On Wed, Sep 27, 2006 at 09:58:43AM -0700, A Large Angry SCM wrote:\n>> This means that modules are not separate, stand alone projects but, \n>> rather, just a sub part of your bigger project. Very useful and \n>> applicable in some situations but other situations want/need separate, \n>> stand alone subprojects.\n> \n> you can do everything with the submodule which would be possible with\n> a normal GIT repository.  And you can always clone it into an directory\n> which is not controlled by a parent project.\n> \n> I really think that this is an very important property of a submodule.\n\nI must be missing something.\n\nI just read you original message in the (sub)thread again and you said:\n\n\t * the .git/refs/heads directory of the submodule gets stored in\n\t   .gitmodule/<modulename> inside the parent project\n\nIf the submodule refs in the parent are a _copy_, then work performed in \nthe submodule outside of the parent will be lost when the parent is in \ncontrol of the submodule again.\n\nIf the submodule refs in the parent are the actual submodule refs then \nthe submodule is not independent of the parent.\n\nIf the submodule refs in the parent are a symlink to the refs in the \nsubmodule, then the parent has no control over which version of the \nsubmodule it gets on the next checkout since the submodule can update \nthe ref.\n\n[...]\n> One use-case which may be important here:\n> \n> The submodule has two different branches which got forked and are not\n> intended to be merged again.  At some point in time the parent project\n> wants to switch from one branch of the submodule to another branch.\n> If a user still has modifications in the old branch and wants to\n> update the parent project then it is important to know if the local\n> modifications and those coming from the parent have to be merged or\n> should stay in different branches.\n> If the parent is switching branches there should only be some warning\n> if the user still has modifications in the old branch, giving him the\n> chance to port the modifications to the other branch.\n\nAgain, this is leading me to believe that the submodule is not \nindependent of the parent.\n"},{"id":"27981","messageId":"20060929070435.GA20011@admingilde.org","threadId":"5704","inReplyTo":"451C30EF.8050305@gmail.com","subject":"Re: Notes on Using Git with Subprojects","fromName":"Martin Waitz","fromEmail":"tali@admingilde.org","sentAt":"2006-09-29T07:04:35Z","receivedAt":"2006-09-29T07:04:35Z","isPatch":false,"sender":{"key":"tali@admingilde.org","avatar":"https://gravatar.com/avatar/3f89b03eee362187effabe257898735b475673a12265c398ea9161259ae91553?d=mp&s=160"},"body":"hoi :)\n\nOn Thu, Sep 28, 2006 at 01:30:39PM -0700, A Large Angry SCM wrote:\n> If the submodule refs in the parent are the actual submodule refs then \n> the submodule is not independent of the parent.\n\nok, it's not independent in that sense that you can move the directory\naway and expect the submodule to work even when the parent does not\nexist any more.\n\nBut you can do normal GIT work as before.\nYou can create new branches (they will be stored in the parent\nbut you have to \"git add .gitmodule/...\" explicitly in order\nto track the branch in the parent), fetch/pull from other sides,\ncreate commits, etc.\nSo in your normal workflow you do not have to do anything in the\nparent while you work in the submodule.  That's what I've called\n\"independent\".\n\nI hope the intention is clear now, perhaps I've been sloppy\nwith words.\n\n-- \nMartin Waitz\n"},{"id":"28054","messageId":"451F4FCA.7060804@gmail.com","threadId":"5704","inReplyTo":"45196628.9010107@gmail.com","subject":"Re: Notes on Using Git with Subprojects","fromName":"A Large Angry SCM","fromEmail":"gitzilla@gmail.com","sentAt":"2006-10-01T05:19:06Z","receivedAt":"2006-10-01T05:19:06Z","isPatch":false,"sender":{"key":"gitzilla@gmail.com","avatar":"https://gravatar.com/avatar/354625c442439908ff3dd99757dee330e29e9df7847472384faf7a00add247fb?d=mp&s=160"},"body":"This updates the note based on the discussion the original note \ngenerated. The most significant difference is the ability to specify \na particular version of a subproject.\n\n\n\t\t\t\t\t\t\t20061001.0400\n\nNotes on Using Git with Subprojects\n===================================\nCopyright (C) 2006 by Raymond S Brand\n\n\nGit does not have native support for subprojects and this is a good\nthing because managing subprojects is better handled by the project\nbuild machinery. Managing subprojects with the project build machinery\nis more flexible than the native support from an SCM and allows the use\nof different SCMs by different subprojects. However, there is a lot of\ninterest in using Git for the SCM of a project with subprojects.\n\nGit, unfortunately, does not make it easy. What is wanted is to put all\nof the subprojects in one repository and be able to checkout the various\nparts from a local copy of the repository. The problem is, with Git, a\nrepository can have at most one working directory associated with it at\na time. This is because Git stores a lot of information about the\ncontents of the working directory in the repository. In fact, the usual\nsituation is that the repository, itself, is in the working directory.\n\nAn important criteria for supporting subprojects is that a subproject\nmay be developed and controlled by an entity or organization different\nthan that of the parent project of the subproject. As a consequence of\nthis, the build machinery needed to manage a project controlled by\nanother entity as a subproject must be in the parent project. \n\nThis note describes a method to use Git with subprojects; other methods\nare also possible.\n\n\nDefinitions\n-----------\nParent Project:\n\tA project that logically contains one or more subprojects.\n\nProject:\n\tA set of files that is (relatively) self contained with respect\n\tto changes and treated as a unit by the SCM.\n\nRoot Project:\n\tA project that is not a subproject.\n\nSubproject:\n\tA project logically contained in another project.\n\nSuper project:\n\tA project composed of subprojects.\n\n\nSetup\n-----\nAll subprojects are contained in a single repository and referred to by\nseparate heads or tags. Developers create a local copy of the project\nrepository; hereafter, referred to as the \"local master repository\" or\njust \"local master\". All fetches, pulls and pushes by the developers\nwith the \"local master\" of other developers should be to and from their\nown \"local master\".\n\n\nRoot Project Checkout\n---------------------\nThe root project is the project that all of the subprojects are a part\nof. It is a parent project of one or more subprojects; each of which may\nalso be parent projects to other subprojects.\n\nTo checkout the root project, choose an name for the project working\ndirectory (the working directory must not already exist) and perform the\nequivalent of the following commands.\n\n\tgit-clone -s -n $LOCAL_MASTER $ROOT_DIR \\\n\t\t&& cd $ROOT_DIR \\\n\t\t&& git-checkout -b $ROOT_BRANCH--local $ROOT_VERSION\n\nWhere:\n\t$LOCAL_MASTER is the path to the \"local master\"\n\t$ROOT_DIR is the name of the working directory to use\n\t$ROOT_VERSION is the ref name of the root project state\n\t$ROOT_BRANCH--local is a branch for local changes\n\nThis will leave the current working directory of the shell in the\nproject working directory.\n\nThis also creates a branch with the suffix of \"--local\" to hold all of\nthe local working directory commits and modifications. The $ROOT_BRANCH\nis used as a tracking branch so that upstream changes can be fetched\ninto the working repository without affecting the checked out files.\n\nOnce the root project is checked out, the subprojects are checked out.\n\n\nSubproject Checkout\n-------------------\nEach project that is a parent project needs to checkout all of the\nsubprojects of the project. Each subproject is checked out with the\nequivalent of the following bash commands:\n\n\tgit-clone -s -n $LOCAL_MASTER $SUBPROJECT_DIR \\\n\t\t&& ( cd $SUBPROJECT_DIR \\\n\t\t\t&& git-checkout -b $SUBPROJECT_BRANCH--local \\\n\t\t\t\t$SUBPROJECT_VERSION\n\t\t)\n\nWhere:\n\t$LOCAL_MASTER is the path to the \"local master\"\n\t$SUBPROJECT_DIR is the directory name of the subproject\n\t$SUBPROJECT_VERSION is the ref name of the subproject state\n\t$SUBPROJECT_BRANCH--local is a branch for local changes\n\t\n\nProject Development\n-------------------\nChanges to a project are performed in the working directory of the\nproject and are recorded in the repository in the working directory on\nthe $PROJECT--local branch.\n\n\nReceiving Project Upstream Changes\n----------------------------------\nUpstream project changes are first fetched into the project tracking\nbranch of the local master repository and are then fetched into the\nproject tracking branch of the working directory repositories. To merge\nupstream changes into the working directory, a pull from the project\ntracking branch of the working directory repository executed.\n\n\t# Fetch project branch from upstream to local master\n\t(cd $LOCAL_MASTER && git-fetch $UPSTREAM $PROJECT_BRANCH)\n\n\t# Fetch project branch from local master to working repo\n\tgit-fetch $LOCAL_MASTER $PROJECT_BRANCH\n\n\t# Merge upstream changes in to working directory\n\tgit-pull --no-commit . $PROJECT_BRANCH\n\nWhere:\n\t$LOCAL_MASTER is the path to the \"local master\"\n\t$UPSTREAM is the Git URL of the upstream repository\n\t$PROJECT_BRANCH is the branch name of the (sub)project\n\n\nSending Project Changes Upstream\n--------------------------------\nTo send project changes upstream from a working directory repository,\nthe changes are first pushed to a branch in the local master repository,\n$PROJECT--$IDENT. The changes can then be pushed or pulled from the\nlocal master repository.\n\n\t# Push project changes to local master\n\tgit-push $LOCAL_MASTER \\\n\t\t$PROJECT_BRANCH--local:$PROJECT_BRANCH--$IDENT\n\n\t# Push project changes from local master to upstream\n\t(cd $LOCAL_MASTER && git-push $UPSTREAM \\\n\t\t$PROJECT_BRANCH--$IDENT:$PROJECT_BRANCH--$NICK--$IDENT)\n\nWhere:\n\t$LOCAL_MASTER is the path to the \"local master\"\n\t$UPSTREAM is the Git URL of the upstream repository\n\t$PROJECT_BRANCH is the tracking branch of the (sub)project\n\t$PROJECT_BRANCH--local is a branch for local changes\n\t$IDENT is a label unique for this set of working directories\n\t$NICK is a (branch name safe) identifier of the developer\n\n\nAutomation\n----------\nThe following proof of concept code can be added to parent project\nmakefiles to automate most of the operations needed to support\nsubprojects with Git.\n\n ---->8----  ---->8----  ---->8----  ---->8----  ---->8----  ---->8----\n# Add this to the makefiles\n\nSUBPROJECTLIST := GIT:s1_ref:s1_dev:s1 \\ \n\t\t  GIT:s2_ref:s2_dev:s2 \\ \n\t\t  GIT:s3_ref:s3_dev:s3\n\ninclude Git_machinery.make\n\n# Rest of the Makefile\n ---->8----  ---->8----  ---->8----  ---->8----  ---->8----  ---->8----\n# Git_machinery.make\n\nGIT-CLONE := gitclone\nGIT-CHECKOUT := gitcheckout\nGIT_FETCH := gitfetch\nGIT-PULL := gitpull\nGIT-PUSH := gitpush\n\ninclude GIT-SUBPROJECT-VARS.mak\n\nGIT-SUBPROJECT-VARS.mak: Makefile\n\t@rm -rf GIT-SUBPROJECT-VARS.mak\n\t@echo \"REPOSITORY := $(REPOSITORY)\" > GIT-SUBPROJECT-VARS.mak\n\t@echo \"IDENT := $(IDENT)\" >> GIT-SUBPROJECT-VARS.mak\n\n# SUBPROJECTLIST has the following format:\n#\tEach \"word\" in the list starts with a scheme identifier followed\n#\tby a ':'. The reainder if the \"word\" is the scheme specific\n#\tsubproject details.\n#   Scheme: GIT\n#\tThe scheme specific subproject details are 3 fields separated by\n#\t':'s.\n#\tField 1: Version reference - reference to checkout and fetch\n#\t\tfrom.\n#\tField 2: Development branch ref name.\n#\tField 3: Sub directory for the subproject.\n\nGIT--subproject--setup::\n\tfor SUBPROJECT in $(filter GIT:%,$(SUBPROJECTLIST)) ; do \\\n\t  PARAM=($$(echo $$SUBPROJECT | tr ':' ' ')) ; \\\n\t  $(GIT-CLONE) -s -n $(REPOSITORY) $${PARAM[3]} \\\n\t  && ( cd $${PARAM[3]} \\\n\t    && $(GIT-CHECKOUT) -b $${PARAM[2]}--local $${PARAM[1]} ; \\\n\t  ) ; \\\n\tdone\n\nGIT--subproject--fetch::\n\t@for SUBPROJECT in $(filter GIT:%,$(SUBPROJECTLIST)) ; do \\\n\t  PARAM=($$(echo $$SUBPROJECT | tr ':' ' ')) ; \\\n\t  ( cd $${PARAM[3]} \\\n\t    && $(GIT-FETCH) $(REPOSITORY) $${PARAM[1]} ; \\\n\t  ) ; \\\n\tdone\n\nGIT--subproject--pull::\n\t@for SUBPROJECT in $(filter GIT:%,$(SUBPROJECTLIST)) ; do \\\n\t  PARAM=($$(echo $$SUBPROJECT | tr ':' ' ')) ; \\\n\t  ( cd $${PARAM[3]} \\\n\t    && $(GIT-FETCH) $(REPOSITORY) $${PARAM[1]} ; \\\n\t    && $(GIT-PULL) --no-commit . $${PARAM[1]} ; \\\n\t  ) ; \\\n\tdone\n\nGIT--subproject--push::\n\t@for SUBPROJECT in $(filter GIT:%,$(SUBPROJECTLIST)) ; do \\\n\t  PARAM=($$(echo $$SUBPROJECT | tr ':' ' ')) ; \\\n\t  ( cd $${PARAM[3]} \\\n\t    && $(GIT-PUSH) $(REPOSITORY) \\\n\t      $${PARAM[2]}--local:$${PARAM[2]}--$(IDENT) ; \\\n\t  ) ; \\\n\tdone\n\n ---->8----  ---->8----  ---->8----  ---->8----  ---->8----  ---->8----\n\nThe example Makefile code has a number of limitations: There is no error\nhandling. It only handles Git managed subprojects. There are other Git\ncommands that can be usefully applied to subprojects.\n"}]}