Re: [RFC] Submodules in GIT

From: Torgil Svensson <torgil.svensson@gmail.com>
Date: Dec 15, 2006, 23:43 UTC
Message-ID: <e7bda7770612151543o39c9d233q91ea643a134196d3@mail.gmail.com>
In-Reply-To: <200612152242.50472.Josef.Weidendorfer@gmx.de>

On 12/15/06, Josef Weidendorfer <Josef.Weidendorfer@gmx.de> wrote:

> That all sounds fine, but how do you create such symlinks in practice?

I'm very open to suggestions here, but the concept growing in my head
is based around Linus 'module'-file and keep things simple. A git
configuration file that specifies:
* link name for reference
* local path to link
* submodule source
* submodule path to tree/blob
* submodule commit / HEAD / branch
* options (depth-limit , ...)

I'm reconsidering having the path-name in the link, it should be sufficient to have two SHA1's, one for the commit and one for the tree/blob. Super-module should have the tree/blob in it's database so that the link part only is there for version information and reference (checking dirty state or history on the submodule). This way it easy to clone the super-project and use it without having to map up all sub-project sources. Sub-project sources is not important for version information and could always be specified in the project in a README-type of file.

Show 7 quoted lines

> Especially, what is the SCM user supposed to do to change the link
> target, ie. from
>  <commit>/path/to/subtree
> to
>  <commit>/path2/to2/subtree2
> ?
> Should this do a re-checkout at the other point?

That would be a change in the modules file, maybe through a command that also fixes the link. The link will have to be updated in the index and commited as normal.

Show 5 quoted lines

> By linking a file from a submodule, such a link seems to force that
> this file has to be at a fixed position in the submodule. Otherwise,
> some magic has to happen when the file is moved in the submodule,
> possibly leading to a dangling link, eg. if the whole subdirectory
> specified in the link is removed.

Since we have the SHA1 (this is what we're using) and tree/blob information in the super-modules database the change itself is not a problem. The problem is to track renames/moves and your remove case in the submodule. The tool that tracks the submodule should probably warn/exit here and we would fix up the modules file manually.

> IMHO this is getting way to complex.

One of complex situation here as I see it is the ability to handle to track/checkout only a subset (tree/blob) of the submodule. This is also quite an important feature - in my example it means the difference of tracking one header file versus the whole source.

> If you only want to check out part of a submodule, this should be
> done with path-limiting checkouts, which should be a feature totally
> independent from submodules.

If we can do path-limiting checkouts on a repo (module) we also can do it on a sub-module since they are exactly the same. This is a very powerful feature and it'd be a huge waste if it wasn't allowed for a super-module to do on submodules.

> And if you want to limit the number of objects transferred in cloning
> of a subproject, it is better to further split this subproject into
> multiple subprojects itself.

What if we have no control of the submodule? This can be tracked from upstream, sourceforge, another company, etc. The submodule will often live their own life and could be X, kernel, gcc, cairo, whatever, ...

Show 5 quoted lines

> The problem is not the representation in the git repository, but the
> checked out module/submodule, where you need to use normal UNIX file semantics.
> To move submodules around, the user should be able to just use
> the normal UNIX "mv" commands, and git should be able to detect move
> actions after the fact.

If we disregard the commit info, the link will act exactly as a normal tree/blob. Git can know we're moving a subproject by watching the module file. The main problem is to keep modules file up-to-date with reality. We could enforce module file validity by disallowing such operations and let the user do a "force" operation which also alters the modules file.

> This now becomes a problem if you use symlinks to "unify" multiple checkouts
> of the same submodule at multiple places in the supermodule, and move
> the symlink around, as it easily can get dangling this way. Thus, you would
> not have a way to see what submodule this link was talking about.

The symlink only exists in the modules file. We only have the SHA1's at the tree-level and there we have everything underneath the tree/blob SHA1 in our database. We will only know if the modules symlink file is dangling next time we fetch from the submodule - here we would notify the user but our database is still consistent.

Show 6 quoted lines

> If you have a source commit chain A => B => C => D, you want
> to make any build commits totally independent: you first only
> are interested in a build commit for source versions A and D,
> and later find out that a build commit for B and C would be nice,
> too. If you force build commits into some history order, this
> order now would be A => D => B => C, which makes no sense.

It makes no sense because the user seem to have act irrationally. The commit-chain is completely valid as it has tracked the correct history of the builds. I can't see any problems here, the build-project is independent of the source-project with it's own history. We can hope the user has given good explanations for his/her actions in the commit messages though.