Re: Notes on Using Git with Subprojects

This updates the note based on the discussion the original note generated. The most significant difference is the ability to specify a particular version of a subproject.

							20061001.0400

Notes on Using Git with Subprojects =================================== Copyright (C) 2006 by Raymond S Brand

Git does not have native support for subprojects and this is a good thing because managing subprojects is better handled by the project build machinery. Managing subprojects with the project build machinery is more flexible than the native support from an SCM and allows the use of different SCMs by different subprojects. However, there is a lot of interest in using Git for the SCM of a project with subprojects.

Git, unfortunately, does not make it easy. What is wanted is to put all of the subprojects in one repository and be able to checkout the various parts from a local copy of the repository. The problem is, with Git, a repository can have at most one working directory associated with it at a time. This is because Git stores a lot of information about the contents of the working directory in the repository. In fact, the usual situation is that the repository, itself, is in the working directory.

An important criteria for supporting subprojects is that a subproject may be developed and controlled by an entity or organization different than that of the parent project of the subproject. As a consequence of this, the build machinery needed to manage a project controlled by another entity as a subproject must be in the parent project.

This note describes a method to use Git with subprojects; other methods are also possible.

Definitions
-----------
Parent Project:
	A project that logically contains one or more subprojects.

Project:
	A set of files that is (relatively) self contained with respect
	to changes and treated as a unit by the SCM.

Root Project:
	A project that is not a subproject.

Subproject:
	A project logically contained in another project.

Super project:
	A project composed of subprojects.

Setup ----- All subprojects are contained in a single repository and referred to by separate heads or tags. Developers create a local copy of the project repository; hereafter, referred to as the "local master repository" or just "local master". All fetches, pulls and pushes by the developers with the "local master" of other developers should be to and from their own "local master".

Root Project Checkout --------------------- The root project is the project that all of the subprojects are a part of. It is a parent project of one or more subprojects; each of which may also be parent projects to other subprojects.

To checkout the root project, choose an name for the project working directory (the working directory must not already exist) and perform the equivalent of the following commands.

	git-clone -s -n $LOCAL_MASTER $ROOT_DIR \
		&& cd $ROOT_DIR \
		&& git-checkout -b $ROOT_BRANCH--local $ROOT_VERSION

Where:
	$LOCAL_MASTER is the path to the "local master"
	$ROOT_DIR is the name of the working directory to use
	$ROOT_VERSION is the ref name of the root project state
	$ROOT_BRANCH--local is a branch for local changes

This will leave the current working directory of the shell in the project working directory.

This also creates a branch with the suffix of "--local" to hold all of the local working directory commits and modifications. The $ROOT_BRANCH is used as a tracking branch so that upstream changes can be fetched into the working repository without affecting the checked out files.

Once the root project is checked out, the subprojects are checked out.

Subproject Checkout ------------------- Each project that is a parent project needs to checkout all of the subprojects of the project. Each subproject is checked out with the equivalent of the following bash commands:

	git-clone -s -n $LOCAL_MASTER $SUBPROJECT_DIR \
		&& ( cd $SUBPROJECT_DIR \
			&& git-checkout -b $SUBPROJECT_BRANCH--local \
				$SUBPROJECT_VERSION
		)

Where:
	$LOCAL_MASTER is the path to the "local master"
	$SUBPROJECT_DIR is the directory name of the subproject
	$SUBPROJECT_VERSION is the ref name of the subproject state
	$SUBPROJECT_BRANCH--local is a branch for local changes
	

Project Development ------------------- Changes to a project are performed in the working directory of the project and are recorded in the repository in the working directory on the $PROJECT--local branch.

Receiving Project Upstream Changes ---------------------------------- Upstream project changes are first fetched into the project tracking branch of the local master repository and are then fetched into the project tracking branch of the working directory repositories. To merge upstream changes into the working directory, a pull from the project tracking branch of the working directory repository executed.

	# Fetch project branch from upstream to local master
	(cd $LOCAL_MASTER && git-fetch $UPSTREAM $PROJECT_BRANCH)

	# Fetch project branch from local master to working repo
	git-fetch $LOCAL_MASTER $PROJECT_BRANCH

	# Merge upstream changes in to working directory
	git-pull --no-commit . $PROJECT_BRANCH

Where:
	$LOCAL_MASTER is the path to the "local master"
	$UPSTREAM is the Git URL of the upstream repository
	$PROJECT_BRANCH is the branch name of the (sub)project

Sending Project Changes Upstream -------------------------------- To send project changes upstream from a working directory repository, the changes are first pushed to a branch in the local master repository, $PROJECT--$IDENT. The changes can then be pushed or pulled from the local master repository.

	# Push project changes to local master
	git-push $LOCAL_MASTER \
		$PROJECT_BRANCH--local:$PROJECT_BRANCH--$IDENT

	# Push project changes from local master to upstream
	(cd $LOCAL_MASTER && git-push $UPSTREAM \
		$PROJECT_BRANCH--$IDENT:$PROJECT_BRANCH--$NICK--$IDENT)

Where:
	$LOCAL_MASTER is the path to the "local master"
	$UPSTREAM is the Git URL of the upstream repository
	$PROJECT_BRANCH is the tracking branch of the (sub)project
	$PROJECT_BRANCH--local is a branch for local changes
	$IDENT is a label unique for this set of working directories
	$NICK is a (branch name safe) identifier of the developer

Automation ---------- The following proof of concept code can be added to parent project makefiles to automate most of the operations needed to support subprojects with Git.

 ---->8----  ---->8----  ---->8----  ---->8----  ---->8----  ---->8----
# Add this to the makefiles

SUBPROJECTLIST := GIT:s1_ref:s1_dev:s1 \ 
		  GIT:s2_ref:s2_dev:s2 \ 
		  GIT:s3_ref:s3_dev:s3

include Git_machinery.make

# Rest of the Makefile
 ---->8----  ---->8----  ---->8----  ---->8----  ---->8----  ---->8----
# Git_machinery.make

GIT-CLONE := gitclone GIT-CHECKOUT := gitcheckout GIT_FETCH := gitfetch GIT-PULL := gitpull GIT-PUSH := gitpush

include GIT-SUBPROJECT-VARS.mak

GIT-SUBPROJECT-VARS.mak: Makefile
	@rm -rf GIT-SUBPROJECT-VARS.mak
	@echo "REPOSITORY := $(REPOSITORY)" > GIT-SUBPROJECT-VARS.mak
	@echo "IDENT := $(IDENT)" >> GIT-SUBPROJECT-VARS.mak

# SUBPROJECTLIST has the following format: # Each "word" in the list starts with a scheme identifier followed # by a ':'. The reainder if the "word" is the scheme specific # subproject details. # Scheme: GIT # The scheme specific subproject details are 3 fields separated by # ':'s. # Field 1: Version reference - reference to checkout and fetch # from. # Field 2: Development branch ref name. # Field 3: Sub directory for the subproject.

GIT--subproject--setup::
	for SUBPROJECT in $(filter GIT:%,$(SUBPROJECTLIST)) ; do \
	  PARAM=($$(echo $$SUBPROJECT | tr ':' ' ')) ; \
	  $(GIT-CLONE) -s -n $(REPOSITORY) $${PARAM[3]} \
	  && ( cd $${PARAM[3]} \
	    && $(GIT-CHECKOUT) -b $${PARAM[2]}--local $${PARAM[1]} ; \
	  ) ; \
	done

GIT--subproject--fetch::
	@for SUBPROJECT in $(filter GIT:%,$(SUBPROJECTLIST)) ; do \
	  PARAM=($$(echo $$SUBPROJECT | tr ':' ' ')) ; \
	  ( cd $${PARAM[3]} \
	    && $(GIT-FETCH) $(REPOSITORY) $${PARAM[1]} ; \
	  ) ; \
	done

GIT--subproject--pull::
	@for SUBPROJECT in $(filter GIT:%,$(SUBPROJECTLIST)) ; do \
	  PARAM=($$(echo $$SUBPROJECT | tr ':' ' ')) ; \
	  ( cd $${PARAM[3]} \
	    && $(GIT-FETCH) $(REPOSITORY) $${PARAM[1]} ; \
	    && $(GIT-PULL) --no-commit . $${PARAM[1]} ; \
	  ) ; \
	done

GIT--subproject--push::
	@for SUBPROJECT in $(filter GIT:%,$(SUBPROJECTLIST)) ; do \
	  PARAM=($$(echo $$SUBPROJECT | tr ':' ' ')) ; \
	  ( cd $${PARAM[3]} \
	    && $(GIT-PUSH) $(REPOSITORY) \
	      $${PARAM[2]}--local:$${PARAM[2]}--$(IDENT) ; \
	  ) ; \
	done

 ---->8----  ---->8----  ---->8----  ---->8----  ---->8----  ---->8----

The example Makefile code has a number of limitations: There is no error handling. It only handles Git managed subprojects. There are other Git commands that can be usefully applied to subprojects.