git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[RFC PATCHv1 00/12] git submodule update in C with parallel cloning

From
Stefan Beller <sbeller@google.com>
Date
Oct 16, 2015, 01:52 UTC
Message-ID
<1444960333-16003-1-git-send-email-sbeller@google.com>

So eventually we want to have projects with lots of submodules such as Android (which would have O(1000) submodules).

The very first thing a user does is cloning a project, so we want to impress with speed there as well. git clone however is lazy and just calls `git submodule update --init --recursive`. So we need to make that fast.

This series rewrites parts of git submodule update in C and in the second-last patch it separates cloning and doing the other actions(checkout/rebase/merge etc) by doing the cloning first and then the rest.

The last patch (which is broken in the first version of the series), then proceeds to put the cloning of the submodules into the get_next_task callback of the parallel process API.

That said, the first few patches introduce some churn in the behavior and tests of Git, so maybe put your eyes there?

Thanks for any advice, Stefan

Stefan Beller (12):
  git submodule update: Announce skipping submodules on stderr
  git submodule update: Announce uninitialized modules on stderr
  git submodule update: Move branch calculation to where it's needed
  git submodule update: Announce outcome of submodule operation to
    stderr
  git submodule update: Use its own list implementation.
  git submodule update: Handle unmerged submodules in C
  submodule config: keep update strategy around
  git submodule update: check for "none" in C
  git submodule update: Check url in C
  git submodule update: Clone projects from within C
  submodule--helper: Do not emit submodules to process directly.
  WIP/broken Clone all outstanding submodules in parallel
 builtin/submodule--helper.c | 221 ++++++++++++++++++++++++++++++++++++++++++++
 git-submodule.sh            |  38 +++-----
 submodule-config.c          |  11 +++
 submodule-config.h          |   1 +
 t/t7400-submodule-basic.sh  |  12 +--
 t/t7406-submodule-update.sh |  12 +--
 6 files changed, 256 insertions(+), 39 deletions(-)
-- 
2.5.0.277.gfdc362b.dirty
Next: Stefan Beller
Message 1 of 21 in “[RFC PATCHv1 00/12] git submodule update in C with parallel cloning”
  1. Stefan BellerOct 16, 2015
  2. 01/12 git submodule update: Announce skipping submodules on stderrStefan Beller, Oct 16, 2015
  3. Junio C HamanoOct 16, 2015
  4. Stefan BellerOct 16, 2015
  5. 02/12 git submodule update: Announce uninitialized modules on stderrStefan Beller, Oct 16, 2015
  6. Junio C HamanoOct 16, 2015
  7. 03/12 git submodule update: Move branch calculation to where it's neededStefan Beller, Oct 16, 2015
  8. Junio C HamanoOct 16, 2015
  9. 04/12 git submodule update: Announce outcome of submodule operation to stderrStefan Beller, Oct 16, 2015
  10. 05/12 git submodule update: Use its own list implementation.Stefan Beller, Oct 16, 2015
  11. Junio C HamanoOct 16, 2015
  12. Stefan BellerOct 16, 2015
  13. 06/12 git submodule update: Handle unmerged submodules in CStefan Beller, Oct 16, 2015
  14. Junio C HamanoOct 20, 2015
  15. Stefan BellerOct 20, 2015
  16. 07/12 submodule config: keep update strategy aroundStefan Beller, Oct 16, 2015
  17. 08/12 git submodule update: check for "none" in CStefan Beller, Oct 16, 2015
  18. 09/12 git submodule update: Check url in CStefan Beller, Oct 16, 2015
  19. 10/12 git submodule update: Clone projects from within CStefan Beller, Oct 16, 2015
  20. 11/12 submodule--helper: Do not emit submodules to process directly.Stefan Beller, Oct 16, 2015
  21. 12/12 WIP/broken Clone all outstanding submodules in parallelStefan Beller, Oct 16, 2015

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.