threads / discuss / 11754

Submodules use case: web development based on modular CMS

Subject: Submodules use case: web development based on modular CMS

## tl;dr

4 messages between Jan 27, 2008 and Jan 28, 2008.

replies: 3people: 2as markdown or json

Jakub Narebski· Jan 27, 2008, 00:39 UTC · lore
A bit of time ago I have stumbled upon the following blog entry 
(question): "Agaric wants version control that lets Drupal core and 
contrib replace entire directories within our checkouts"[1]
  http://tinyurl.com/yv3jp4
Here the author want to use version control system for customized 
version of CMS system, namely:
 * Drupal core
 * selected Drupal contributed modules ("inside" Drupal core)
 * own work (theme, custom modules) (also "inside" Drupal core)
And of course he wants to have development branches: dev, test and live, 
and want to always checkout correct versions of Drupal, Drupal modules 
and own work.

Somebody (in comments) proposed using Subversion externals; I have proposed using Git submodules (subproject) support. Nevertheless while I have at one time manage PHPNuke portal using CVS and vendor branches, and even had half-written automatic fast checkout/fast export script, I haven't worked with submodules in Git, nor with using Git to manage web site.

Comments (here, and as comments to mentioned blog article) about how such thing should be done with Git very appreciated. (And for example how Git submodules differ from svn:externals and why git way is better ;-)).

P.S. BTW. Drupal is considering moving from CVS to some other SCM, see http://groups.drupal.org/revision-control-systems . Perhaps Git would be the choice? Is anybody here active in Drupal development? Is for example their Version Control API too CVS-centric (single file history centric)?

[1]http://agaricdesign.com/note/agaric-wants-version-control-lets-drupal-core-and-contrib-replace-entire-directories-within-our-checkouts
-- 
Jakub Narebski
Poland
Martin Langhoff· Jan 27, 2008, 03:00 UTC · re: Jakub Narebski · lore

Re: Submodules use case: web development based on modular CMS

On Jan 27, 2008 1:39 PM, Jakub Narebski <jnareb@gmail.com> wrote:
> A bit of time ago I have stumbled upon the following blog entry
> (question): "Agaric wants version control that lets Drupal core and
> contrib replace entire directories within our checkouts"[1]
>   http://tinyurl.com/yv3jp4

While it is possible to use submodules, I tend to think that you can just use your own repo / branch "off" the main project, just like unofficial kernel modules do.

It does have a number of advantages -- including being explicit wrt to what version of the core project the module is developed against. Drupal and similar projects (like Moodle) do have a module API, but it's not *that* stable -- in spite of the best intentions, those APIs get subtle changes all the time. You should have your "release" script package only the module subdirectories -- and perhaps any delta in the core (of Drupal) that needs to be applied. In many "contrib" projects I've seen module files and little patch files that you are supposed to apply - and maintenance of those is a pain. It makes more sense to use git over the whole tree.

So it's a tradeoff -- IMHO, at first blush it looks more "natural" to use submodules, but as things progress over time I think it forces a lot of additional and ill-fitted work like maintaining patches, or adding tags that indicate "developed against Drupal v1.2.3". And the day the contrib module becomes official, "merging it in" is a bit of a mess if you want to preserve history. So in the long haul, it makes a lot more sense to use the "branch off the full project" approach that has served the kernel modules so well.

In that sense, I think that gitk's tree getting merged in a subdirectory is good as an example of what git can do, but a bit pointless as gitk depend quite tightly on the behaviour of git commands. So distributing gitk separately would be a big pain as each gitk version is usable against a narrow set of git revisions. Luckily, it gets merged into git and that's how it gets distributed.

My .2c anyway ;-)
m
Jakub Narebski· Jan 27, 2008, 18:23 UTC · re: Martin Langhoff · lore

Re: Submodules use case: web development based on modular CMS

I CC'ed drupal mainling list...
On Sun, 27 Jan 2008, Martin Langhoff wrote:
Show 10 quoted lines
> On Jan 27, 2008 1:39 PM, Jakub Narebski <jnareb@gmail.com> wrote:
>>
>> A bit of time ago I have stumbled upon the following blog entry
>> (question): "Agaric wants version control that lets Drupal core and
>> contrib replace entire directories within our checkouts"[1]
>>   http://tinyurl.com/yv3jp4
> 
> While it is possible to use submodules, I tend to think that you can
> just use your own repo / branch "off" the main project, just like
> unofficial kernel modules do.
I think it is why they currently do, and they want to change it...
<blockquote href="http://tinyurl.com/yv3jp4">
  Right now we have each of these three areas checked out independently,
  and while they do happily reside inside one another, there's some
  inconvenience and risk of not keeping everything in synch.
  The solution we're looking at now is shell scripts but if there's
  a way for a version control system to just understand this need and
  accept it, that would be best.
</quote>

...but I guess they use CVS just like Drupal does, which might make it painful.

Show 19 quoted lines
> It does have a number of advantages -- including being explicit wrt to
> what version of the core project the module is developed against.
> Drupal and similar projects (like Moodle) do have a module API, but
> it's not *that* stable -- in spite of the best intentions, those APIs
> get subtle changes all the time. You should have your "release" script
> package only the module subdirectories -- and perhaps any delta in the
> core (of Drupal) that needs to be applied. In many "contrib" projects
> I've seen module files and little patch files that you are supposed to
> apply - and maintenance of those is a pain. It makes more sense to use
> git over the whole tree.
> 
> So it's a tradeoff -- IMHO, at first blush it looks more "natural" to
> use submodules, but as things progress over time I think it forces a
> lot of additional and ill-fitted work like maintaining patches, or
> adding tags that indicate "developed against Drupal v1.2.3". And the
> day the contrib module becomes official, "merging it in" is a bit of a
> mess if you want to preserve history. So in the long haul, it makes a
> lot more sense to use the "branch off the full project" approach that
> has served the kernel modules so well.
I think that choice between developing "in core" / on branch like Linux 
kernel modules and subsystems vs using submodules support like msysGit, 
cgit and (perhaps) in the future git-gui and gitk, depends on a few 
things:
 1. How tightly coupled are modules with core / how independent
    development of modules is
 2. How tightly coupled / how independent modules are from each other,
    both from the development point of view, but also from testing point
    of view
 3. How much work has to be done to add / enable module

How coupled modules are with the core depends among others on the stability of modular API. The dependency of modules on core development might be measured by [relative] number of commits which change core and module, or which change module responding to change in core (bugfixes and workaround for core bugs excluded). And of course if module development _is_ done independently.

If modules are not developed in isolation (e.g. code movement), or cannot be tested in isolation (unit tests for modules alone, whole tests for core + single module) then of course it doesn't make much sense to use submodules support. (For example it would be hard to test for example networking module without filesystem module, for Linux kernel modules.)

If you have to run some install script to install / start using modules (for example phpBB modules are custom files + specialized patch-like file, which allows modifying not only files but also databases), then that's of no avail to use submodules because you have to use [custom] scripts to update nevertheless...

Show 6 quoted lines
> In that sense, I think that gitk's tree getting merged in a
> subdirectory is good as an example of what git can do, but a bit
> pointless as gitk depend quite tightly on the behaviour of git
> commands. So distributing gitk separately would be a big pain as each
> gitk version is usable against a narrow set of git revisions. Luckily,
> it gets merged into git and that's how it gets distributed.

Errr... AFAICT it is not true. It is true that gitk and git-gui require sufficiently new git version to work, but IIRC both have feature autodetection, which allows to use gitk and git-gui with older git (and of course the reverse: old gitk or git-gui with new git). I have used gitk from 'master' occassionaly, while having as default some older git version installed from RPM (released version).

gitk and git-gui development is managed independently from git, and can be done using only gitk and git-gui sources. So using submodules is a good idea. Additionally this allows to not checkout gitk and git-gui for example if you don't have Tcl/Tk installed or don't use windowed environment.

gitweb could also be managed as module... but currently there is no maintainer who would manage gitweb.git repository...

On the other hand having gitk, git-gui, gitweb in git proper allow to use just introduced features, and not only features available in last released version.

BTW. if some tool is tightly coupled with git, it can use git.git repository (or modification of thereof) as submodule, as shown in the cases of msysGit and cgit.

-- 
Jakub Narebski
Poland
Jakub Narebski· Jan 28, 2008, 10:44 UTC · re: Jakub Narebski · lore

Re: Submodules use case: web development based on modular CMS

On Sun, 27 Jan 2008, Jakub Narebski wrote:
> Somebody (in comments) proposed using Subversion externals; I have 
> proposed using Git submodules (subproject) support. 
BTW. Mercurial supports something similar by so called (unofficial) 
"forest extension":
  http://www.selenic.com/mercurial/wiki/index.cgi/ForestExtension

If I understand it corectly the mechanism follow idea of manifest-like file Mercurial uses, and is similar to one of the test implementations of submodule support before we had current "in core" version.

I wonder if Bazaar-NG has similar extension / plugin...

It would be nice to have comparison of submodule / subproject support in different version control systems...

-- 
Jakub Narebski
Poland

← back to recent threads