git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH 0/2] fix problems with recursive submodule fetching

From
Kevin Ballard <kevin@sb.org>
Date
Sep 19, 2010, 03:54 UTC
Message-ID
<DD3654D9-46B9-4980-9138-38FDC33A383C@sb.org>
In-Reply-To
<4C953DE5.6020900@web.de>
On Sep 18, 2010, at 3:32 PM, Jens Lehmann wrote:
Show 19 quoted lines
>> And of course the performance issue is still relevant.
> 
> Hmm, I think it still the right thing to do to a full fetch of all
> submodule changes, as you can't possibly know at fetch time what you'll
> need later. But I have to admit that "git submodule update" does some
> kind of optimization, as it only fetches submodules where the commit
> recorded in the superproject has changed. But IMO this aims a bit too
> short, as it still can lead to commits missing in submodules (which are
> on branches you didn't check out and ran "git submodule update" on).
> 
> The aim of these changes is to make you able to do a fetch before you go
> onto a plane and then check out every branch of the superproject without
> having a commit missing from a submodule. And even more, you should be
> able to use all new submodule commits not yet committed inside the
> superproject without being able to forget to fetch them before you leave.
> 
> But maybe we can reduce this performance impact by running the submodule
> fetches in threads. Dunno yet, I'll look into that as soon as I have the
> other two patches ready.
In a situation like mine, with 17 nested submodules, you're still going to significantly increase the time for a fetch. I understand the desire to be able to run something like `git fetch` and have everything be there, but I don't agree it should be the default behavior. Perhaps it should be a flag, like `git fetch --update-submodules` (or `git fetch --recursive`). That's friendly enough to let people know about (much friendlier than `git submodule update --recursive`), but wouldn't cause any changes to the common case. And a config variable to control whether recursive is the default would not be amiss (though I would make that config variable default to false).
I should also note that fetching all submodules may be not desired in the slightest. If I use submodules to pull in specific tagged builds of another project, I really have no desire at all to regularly update the branches in that repository. Similarly, if I pull in another project that's out of my control, I have no desire to update any submodules it may have unless the project itself changes. That's certainly the case in my project. Here's the submodule layout:
 .--Root-.
/ / /|\ \ \  
A B C D E F G
    |
    H
   /|\
  I J K
   /|\
  L M N
   /|\
  O P Q
I control the root, and I control most of the first-level of submodules. I would also not mind seeing updates to other submodules at that level, though that doesn't generally concern me. However, I don't control submodule H, and I have absolutely no desire whatsoever to see updates to I-Q unless needed by submodule H. In this layout, regularly running `git submodule update --init --recursive` works perfectly. Having `git fetch` fetch submodules I-Q on every invocation is a complete waste of my time.
-Kevin Ballard
Previous: Jens LehmannNext: Jens Lehmann
Message 24 of 55 in “Teach fetch and pull to recursively fetch submodules too”
  1. 0/2 Teach fetch and pull to recursively fetch submodules tooJens Lehmann, Aug 29, 2010
  2. 1/2 fetch/pull: Recursively fetch populated submodulesJens Lehmann, Aug 29, 2010
  3. 2/2 Submodules: Add the new "fetch" config optionJens Lehmann, Aug 29, 2010
  4. Junio C HamanoAug 30, 2010
  5. 2/2 Submodules: Add the new "fetch" config option for fetch and pullJens Lehmann, Aug 30, 2010
  6. Ævar Arnfjörð BjarmasonAug 29, 2010
  7. Jens LehmannAug 29, 2010
  8. Junio C HamanoAug 30, 2010
  9. Jens LehmannAug 30, 2010
  10. Kevin BallardSep 15, 2010
  11. Kevin BallardSep 15, 2010
  12. fetch: Get submodule paths from index and not from .gitmodulesJens Lehmann, Sep 16, 2010
  13. Kevin BallardSep 16, 2010
  14. Jens LehmannSep 17, 2010
  15. Johannes SixtSep 17, 2010
  16. Jens LehmannSep 17, 2010
  17. Johannes SixtSep 17, 2010
  18. Jens LehmannSep 17, 2010
  19. Johannes SixtSep 17, 2010
  20. Kevin BallardSep 18, 2010
  21. 0/2 fix problems with recursive submodule fetchingJens Lehmann, Sep 18, 2010
  22. 1/2 fetch: Fix a bug swallowing the output of recursive submodule fetchingJens Lehmann, Sep 18, 2010
  23. 2/2 fetch: Get submodule paths from index and not from .gitmodulesJens Lehmann, Sep 18, 2010
  24. Kevin BallardSep 19, 2010
  25. Jens LehmannSep 19, 2010
  26. Kevin BallardSep 20, 2010
  27. 0/3 Teach fetch and pull to recursively fetch submodules tooJens Lehmann, Oct 5, 2010
  28. 1/3 fetch/pull: Recursively fetch populated submodulesJens Lehmann, Oct 5, 2010
  29. 2/3 Submodules: Add the new "fetch" config option for fetch and pullJens Lehmann, Oct 5, 2010
  30. Jon SeymourOct 7, 2010
  31. Jens LehmannOct 9, 2010
  32. Jonathan NiederOct 9, 2010
  33. Jens LehmannOct 9, 2010
  34. 3/3 Add the 'fetch.recursive' config settingJens Lehmann, Oct 5, 2010
  35. Junio C HamanoOct 5, 2010
  36. Kevin BallardOct 6, 2010
  37. Jonathan NiederOct 6, 2010
  38. Jens LehmannOct 9, 2010
  39. Jonathan NiederOct 9, 2010
  40. Jens LehmannOct 9, 2010
  41. Jonathan NiederOct 21, 2010
  42. Jens LehmannOct 21, 2010
  43. Jens LehmannOct 9, 2010
  44. Marc BranchaudOct 13, 2010
  45. Jens LehmannOct 13, 2010
  46. Kevin BallardOct 13, 2010
  47. Jens LehmannOct 13, 2010
  48. Kevin BallardOct 13, 2010
  49. Chris PackhamOct 14, 2010
  50. Jens LehmannOct 14, 2010
  51. Chris PackhamOct 14, 2010
  52. Marc BranchaudOct 13, 2010
  53. Kevin BallardOct 13, 2010
  54. Jens LehmannSep 15, 2010
  55. Kevin BallardSep 15, 2010

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.