RFC patch"Recursive Make considered harmful"

Convert build process from recurse Make to a single Make

The old Makefiles in Documentation/ and tools/ still exist until we feel confident that I didn't miss anything on this conversion.

Most of this patch is fixing up the main Makefile to avoid overlapping target names.

Signed-off-by: Ryan Anderson <ryan@michonline.com>
---

 Documentation/Makefile.inc |   50 ++++++++++++++++++++++++++++++++++++++++++++
 Makefile                   |   24 ++++++++++++++-------
 tools/Makefile.inc         |   12 +++++++++++
 3 files changed, 78 insertions(+), 8 deletions(-)
 create mode 100644 Documentation/Makefile.inc
 create mode 100644 tools/Makefile.inc

003afd3ed1f83b4533b628182fa16c9ab0dc0467

Ryan Anderson wrote:

Please explain the rational for this.

On Wed, Jul 27, 2005 at 10:25:10AM -0400, A Large Angry SCM wrote:

I'm new to the list, but given the subject, I'm fairly confident it's this.

http://www.canb.auug.org.au/~millerp/rmch/recu-make-cons-harm.html

I'm a convert. I converted a fairly large code base at work, and it was a huge boon for productivity. Build times dropped dramatically (from 40 seconds to 2-5 for a single file change).

He used the exact wording just about everyone dones when referring to it conceptually. It's easy to google for support and rebuttal:

http://www.google.com/search?hl=en&q=Recursive+Make+considered+harmful&btnG=Google+Search

    Thanks,
        Kirby

Kirby C. Bohling wrote:

...

Thanks for the references.

A quick read of the paper and some of the rebuttals make me think that 
either way (recursive/non-recursive):
	* require about the same amount of makefile/dependency maintenance work 
from developers.
	* allow developers to be lazy in different ways with 
makefiles/dependencies.
	* achieves the same end.

The non-recursive make method may have a small advantage for developers using Git for their SCM because the Git operations are also performed at the top level due to Git's design.

Ryan Anderson <ryan@michonline.com> writes:

While I do not have strong objections to make the build process go faster, it is somewhat disturbing that the Makefile pieces maintained in subdirectories need to name things they touch using paths that include the subdirectory names. I do not have a better alternative to suggest, though...

I'd keep it in the proposed updates branch for now and wait for a bit until discussions on the list die out.

Junio C Hamano wrote:

For a project the size of Git, is there any real benefit to this change?

Besides pathing issues, you also have to aware that all identifiers in the included makefile fragments will be global.

I don't object to the change but I see it as trading one maintenance issue for another.

Dear diary, on Thu, Jul 28, 2005 at 12:07:07AM CEST, I got a letter where A Large Angry SCM <gitzilla@gmail.com> told me that...

I'd also argue that generally, larger files are inherently harder to maintain, and having all the targets for all the subdirectories in a single file sounds nightmarish. (OTOH, by now you probably know that I'm a keep-it-as-local-as-possible junkie.)

Hi, Petr Baudis wrote:

which is why you'd include Makefile[.inc] snippets from subdirectories instead.

Ryan, I am dropping this patch, at least for now, after keeping it in the "pu" (proposed updates) branch and using it myself. There are two complaints from me.

I am used to "make bin=$HOME/bin/i386 install install-tools", which the patch breaks (I do not want to build docs for myself). This is minor; I could say "install-bin install-toolsxx" instead.

I do not deal with RPM packages myself, but guessing by reading git-core.spec.in, I think it relies on the install target not touching the documentation, in order for it to be able to build doc-full and/or doc-less binary packages. The patch makes install target to also build and install docs.

The Debian build is not affected because it does not produce separate git-core and doc-git-core packages[*1*]; probably this was the reason you did not notice this.

I think what is installed from the toplevel and what comes from tools/ subdirectory are divided mostly for historical reasons and nothing else[*2*], and I do not mind the install target depending on install-bin and install-tools, but I suspect that binary packaging folks would appreciate to have a separate doc target that is not done by a normal install.

Speeding up the build procedure by defining dependencies correctly is a worthy goal. Personally I feel a low hanging fruit is in the main Makefile, before worrying about the make recursion. Many things are in libgit.a and when I touch something only relevant to small number of things, say csum-file.c, all "git-%: %.c" programs are recompiled and relinked, even most of them do not link with csum-file.o (this particular one is only used by git-pack-objects, by the way).

[Footnote]

*1* Which, BTW, would be the Debian way, if I am not mistaken.

*2* Although one _could_ argue that tools/ is primarily meant for "project lead" role users who accept and incorporate patches obtained via e-mails.

Hi, Junio C Hamano wrote:

git-core-doc, actually.

Debian does that only if the documentation is substantial. Even then, manpages may not be segregated into -doc.

However, I *would* segregate gitk into its own Debian package, because it requires wish et al., which would pull a large chunk of X11 stuff, which people may not want on their server.

Patch follows separately -- I'll have to pull it from my other mess (which includes yet another Debian package for Cogito ;-).

Matthias Urlichs <smurf@smurf.noris.de> writes:

While I agree gitk should not come as part of git package, this brings up a different issue.

Ideally, I'd want to see gitk packaged from its repository kernel.org:/pub/scm/gitk/git.git/ Paul Mackerras maintains, not from GIT one which _will_ lag behind.

We have a copy of gitk in git repository because Linus "merged" it as "the coolest merge ever" example. While I intend to keep updating from gitk repository from time to time only because I do not want to ship ancient version of it, I see a big problem down the road. What happens if someday Paul wanted to have a toplevel Makefile of his own, or if somebody sends him a patch to add debian/rules file to build a separate gitk package from its own source tree? Pulling/merging from gitk repo to update the copy git has suddenly becomes a nightmere.

While I _do_ rely on gitk in my git work, and I _do_ like its simplicity (just a single file right now), my longer term preference is to drop the copy we have in git tree and treat it just like the other repository browser, qgit. Our documentation should point people at it as part of the Porcelain suite.

Junio C Hamano wrote: ...

...

Then this should happen sooner rather than later.

On Wed, Jul 27, 2005 at 02:50:55PM -0700, Junio C Hamano wrote:

Sorry for taking so long to respond here - I've probably got 2 or 3 general replies to make on this thread, but basically, I truly intended it as a RFC.

I think the best justification for the end goal of the process I was thinking of starting is this:

	$ git clone -l git-linus git-example
	defaulting to local storage area
	0 blocks
	$ cd git-example
	$ git checkout
	$ ls | wc -l
	154

I've been spending some time trying to think out what qualifies as a "tool" and what is "core", etc. I think it wouldn't be a bad idea to think about restructuring things a bit so that all the little "helper" scripts we keep adding don't fill up the top level directory.

I think I'm going to rethink this, a bit more. I'm unhappy with how I had to edit the sub-dir Makefiles to include directory names. Sam, if you happen to be reading this, feel free to help out!

I'm almost thinking that something like:

	PROGS := 
	SCRIPTS :=
	include x/Makefile.inc
	PROGRAMS += $(addprefix x/,$PROGS)
	ALL_SCRIPTS += $(addprefix x/,$SCRIPTS)

in the top-level Makefile might be the cleanest way to keep the subdirectory ones simpler - but that's still somewhat distasteful, and only fixes up one part of the problem.

Anyway, I'll come back to this later when I've got some of the follow-up issues sorted out, like what to do with the directory structure.

If the goal is to speed up the build process the only sane way is to fix the dependencies. In kbuild fixdep is used to parse the .c file and it locates all references to .h files (recursive) and also detects any usage of CONFIG_ symbols. This part should be relative straightforward to include in git.

That is doable for sure. But it hits you hard when you have to create some special rules in a subdirectory - then you need to know in what directory you are placed. You could assing sub := x before including x/Makefile.inc.

On the other hand. The recursive make considered harmful is IMHO a bit overaggregated. See the kernel where it is used extensively. And it works with no hassle. For a small project like git it should be possible to keep the dependencies in proper shape so there is no cross directory boundaries to worry about - or at least only a few.

	Sam

Dear diary, on Fri, Jul 29, 2005 at 09:31:34AM CEST, I got a letter where Sam Ravnborg <sam@ravnborg.org> told me that...

FWIW, I made tiny "build system" (inspired by kconfig) for smaller projects I work on:

http://pasky.or.cz/~pasky/dev/tunneler/co/Makefile http://pasky.or.cz/~pasky/dev/tunneler/co/Makefile.lib http://pasky.or.cz/~pasky/dev/tunneler/co/client/Makefile

Perhaps someone might find that a nice base for further hacking. It generally appears to work pretty well in practice, although the automatic dependency tracking might not be perfect.

On Fri, 29 Jul 2005 09:46:14 +0200 Petr Baudis <pasky@suse.cz> wrote:

Me too! :)

http://onion.dynserv.net/~timo/index.php?page=Projects/build

It also has configuration system written in bash.