git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [RFC/PATCH 0/3] Thinning the git toplevel directory

From
Jeff King <peff@peff.net>
Date
Feb 23, 2011, 08:43 UTC
Message-ID
<20110223084329.GA10738@sigill.intra.peff.net>
In-Reply-To
<20110223082929.GB30485@elie>
On Wed, Feb 23, 2011 at 02:29:29AM -0600, Jonathan Nieder wrote:
>     If the project grows too much, all those "make" calls start to
>     take time.  It is possible for parallel "make" invocations to
>     trample on each other's work if they are not well coordinated.

Aren't there cases where no amount of coordination will help? E.g., let's say I have two subdirs, lib/ and cmds/. From the top-level I do "make -j". This invokes two recursive sub-makes, one per directory. The make in cmds/ sees that we haven't built stuff in lib/ yet. So it invokes a recursive make in lib/. Now we have two parallel makes running in lib/, stomping on each other. The problem is that no single make was allowed to see the whole dependency tree.

I know GNU make does have some magic for communicating between recursive makes. Does it handle this situation?

You can fix this with a rule like "only invoke recursive makes on directories _below_ you, never above or to the side". But then you can't run "make" from inside cmds/.

You could have a dependency in the top-level that says recursing into cmds/ depends on having finished recursing into libs/. But that's not strictly correct. You want to be working on part of what's in cmds/ (building .o files) in parallel with what's going on in lib/, and then wait on lib/ only for the linking portion.

Those are all things that are trivial to handle in a single Makefile.
>  B. top-level Makefile slurps in Makefiles from subdirs.  Other
>     Makefiles would
I like this better, but...
>      - keep careful track of what directory "make" was run from; [*]
> [...]
>     [*] is a little hazy and sounds hackish.

Yeah, you have to be careful with paths. I think a more sane way would be a single top-level Makefile that either contains everything, or sources tidbits from subdirectories. But in either case, assumes it's running from the top-level.

A dummy Makefile in each subdir that cd's to the toplevel and runs a specific target. So from the top-level, "make" would build everything, "make lib" would build stuff in the lib directory, and the Makefile in lib/ would just do "cd .. && make lib".

It's not perfect, but it's simple and predictable.
-Peff
Previous: Jonathan NiederNext: Jonathan Nieder
Message 37 of 54 in “Move test-*.c to test/ subdirectory”
  1. Move test-*.c to test/ subdirectoryNguyễn Thái Ngọc Duy, Feb 9, 2011
  2. Nguyen Thai Ngoc DuyFeb 9, 2011
  3. Junio C HamanoFeb 9, 2011
  4. Move test-* to t/helper/ subdirectoryNguyễn Thái Ngọc Duy, Feb 10, 2011
  5. 0/3 Thinning the git toplevel directoryJonathan Nieder, Feb 18, 2011
  6. 1/3 Move libgit.a sources into a libgit/ subdirectoryJonathan Nieder, Feb 18, 2011
  7. Nguyen Thai Ngoc DuyFeb 18, 2011
  8. Jonathan NiederFeb 18, 2011
  9. Jeff KingFeb 18, 2011
  10. Jonathan NiederFeb 18, 2011
  11. Nguyen Thai Ngoc DuyFeb 18, 2011
  12. 2/3 Move test-* into a test-programs/ subdirectoryJonathan Nieder, Feb 18, 2011
  13. 3/3 Move header files into a include/ subdirectoryJonathan Nieder, Feb 18, 2011
  14. Nguyen Thai Ngoc DuyFeb 18, 2011
  15. 6/3 Move remaining header files into a include/ subdirectoryJonathan Nieder, Feb 18, 2011
  16. 4/3 compat: do not use relative paths to refer to git-compat-util.h et alJonathan Nieder, Feb 18, 2011
  17. 5/3 block-sha1: do not use relative path for git-compat-util.hJonathan Nieder, Feb 18, 2011
  18. 6/3 Move git-compat-util.h, strbuf.h, and cache.h to include/Jonathan Nieder, Feb 18, 2011
  19. Nguyen Thai Ngoc DuyFeb 18, 2011
  20. 9/3 Thinning the git toplevel directoryJonathan Nieder, Feb 18, 2011
  21. 7/3 Move test-sha1.sh to test-programs/Jonathan Nieder, Feb 18, 2011
  22. 8/3 Move build helpers to scripts/ subdirectoryJonathan Nieder, Feb 18, 2011
  23. 9/3 Move non-builtin git commands and script libraries to a subdirectoryJonathan Nieder, Feb 18, 2011
  24. Jonathan NiederFeb 18, 2011
  25. Jeff KingFeb 18, 2011
  26. Jonathan NiederFeb 18, 2011
  27. Nguyen Thai Ngoc DuyFeb 18, 2011
  28. Junio C HamanoFeb 18, 2011
  29. Jonathan NiederFeb 19, 2011
  30. Sverre RabbelierFeb 19, 2011
  31. The git_remote_helpers package (Re: [RFC/PATCH 0/3] Thinning the git toplevel directory)Jonathan Nieder, Feb 19, 2011
  32. Jeff KingFeb 22, 2011
  33. Junio C HamanoFeb 22, 2011
  34. Sverre RabbelierFeb 22, 2011
  35. Jeff KingFeb 23, 2011
  36. Jonathan NiederFeb 23, 2011
  37. Jeff KingFeb 23, 2011
  38. Recursive make and variations on the themeJonathan Nieder, Feb 23, 2011
  39. Junio C HamanoFeb 23, 2011
  40. Nicolas PitreFeb 23, 2011
  41. Drew NorthupFeb 23, 2011
  42. Nicolas PitreFeb 24, 2011
  43. Drew NorthupFeb 24, 2011
  44. Nicolas PitreFeb 24, 2011
  45. Jeff KingFeb 24, 2011
  46. Drew NorthupFeb 24, 2011
  47. Piotr KrukowieckiFeb 19, 2011
  48. Junio C HamanoFeb 19, 2011
  49. Jonathan NiederFeb 19, 2011
  50. Piotr KrukowieckiFeb 19, 2011
  51. Piotr KrukowieckiFeb 19, 2011
  52. Advertising the prebuilt htmldocs and manpagesJonathan Nieder, Feb 19, 2011
  53. Junio C HamanoFeb 20, 2011
  54. Jonathan NiederFeb 20, 2011

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.