git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Parallelism for submodule update

From
ZCZitzmann, Christian <christian.zitzmann@vitesco.com>
Date
Jan 2, 2023, 16:44 UTC
Message-ID
<DB5PR02MB100691E6422F5E94228F0E0EC8AF79@DB5PR02MB10069.eurprd02.prod.outlook.com>

Hello, we are using git since many years with also heavily using submodules.

When updating the submodules, only the fetching part is done in parallel (with config submodule.fetchjobs or --jobs) but the checkout is done sequentially
What I’ve recognized when cloning with
- scalar clone --full-clone --recurse-submodules <URL>
or
- git clone --filter=blob:none --also-filter-submodules --recurse-submodules <URL>
We loose performance, as the fetch of the blobs is done in the sequential checkout part, instead of in the parallel part.
Furthermore, the utilization - without partial clone - of network and harddisk is not always good, as first the network is utilized (fetch) and then the harddisk (checkout)

As the checkout part is local to the submodule (no shared resources to block), it would be great if we could move the checkout into the parallelized part. E.g. by doing fetch and checkout (with blob fetching) in one step with e.g. run_processes_parallel_tr2

I expect that this significantly improves the performance, especially when using partial clones.
Do you think this is possible? Do I miss anything in my thoughts?
Best regards,
Christian Zitzmann
Next: rsbecker@nexbridge.com
Message 1 of 4 in “Parallelism for submodule update”
  1. Zitzmann, ChristianJan 2, 2023
  2. rsbecker@nexbridge.comJan 2, 2023
  3. Zitzmann, ChristianJan 13, 2023
  4. Calvin WanJan 19, 2023

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.