git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCHv2 0/8] Expose the submodule parallelism to the user

From
Stefan Beller <sbeller@google.com>
Date
Nov 3, 2015, 19:41 UTC
Message-ID
<CAGZ79kbWbN_8XSMyYnkxstqV-+fHEixceeGaR4NYGqrvw0ZaUQ@mail.gmail.com>
In-Reply-To
<5632B0E1.8040309@ramsayjones.plus.com>

On Thu, Oct 29, 2015 at 4:50 PM, Ramsay Jones <ramsay@ramsayjones.plus.com> wrote:

Show 27 quoted lines
>
>
> On 29/10/15 15:51, Stefan Beller wrote:
>> On Thu, Oct 29, 2015 at 6:19 AM, Ramsay Jones
>> <ramsay@ramsayjones.plus.com> wrote:
>>
>>> Hmm, is there a way to _not_ fetch in parallel (override the
>>> config) from the command line for a given command?
>>>
>>> ATB,
>>> Ramsay Jones
>>
>> git config submodule.jobs 42
>> git <foo> --jobs 1 # should run just one task, despite having 42 configured
>
> Heh, yes ... I didn't pose the question quite right ...
>>
>> It does use the parallel processing machinery though, but with a maximum of
>> one subcommand being spawned. Is that what you're asking?
>
> ... but, despite that, you correctly inferred what I was really
> asking about! :)
>
> I was just wondering what overhead the parallel processing machinery
> adds to the original 'non-parallel' code path (for the j=1 case).
> I suspect the answer is 'not much', but that's just a guess.
> Have you measured it?
Totally unscientific:
 * Make a copy of my current gerrit repository and time the fetch.
 * That repo contains 5 submodules, one needs fetching

time git fetch --recurse-submodules=yes --jobs=1 # this series real 0m7.150s user 0m3.459s sys 0m1.126s

time git fetch --recurse-submodules=yes # origin/master real 0m7.667s user 0m3.439s sys 0m1.190s

Now let's test a few more times repeatedly to avoid cold caches or network hiccups, (also there is nothing to fetch, so it's more like doing 6 ls-remotes in a row, one for gerrit and 5 submodules)

this series, best out of 5: real 0m3.971s user 0m2.447s sys 0m0.452s

this series, worst out of 5: real 0m4.229s user 0m2.506s sys 0m0.413s

origin/master, best out of 5: real 0m3.968s user 0m2.516s sys 0m0.380s

origin/master, worst out of 5: real 0m4.217s user 0m2.472s sys 0m0.408s

The ratio of real time taken longer is < 1 % in both the best and worst case.

If you really care about 1 % of performance, you'd want to fetch in parallel anyway?

> What happens if there is only a single
> submodule to fetch?

Ok let's see. I created https://github.com/stefanbeller/test-sub-1 to play around with it. However time git fetch --recurse-submodules=yes or time git fetch --recurse-submodules=yes --jobs 100 seems to be lost in the noise.

So I am not sure what the question is w.r.t. having just one submodule.

Show 5 quoted lines
>
> ATB,
> Ramsay Jones
>
>
Previous: Ramsay JonesNext: Junio C Hamano
Message 47 of 48 in “Expose the submodule parallelism to the user”
  1. 0/9 Expose the submodule parallelism to the userStefan Beller, Oct 27, 2015
  2. 1/9 submodule-config: "goto" removal in parse_config()Stefan Beller, Oct 27, 2015
  3. Jonathan NiederOct 27, 2015
  4. Junio C HamanoOct 27, 2015
  5. 2/9 submodule config: keep update strategy aroundStefan Beller, Oct 27, 2015
  6. 3/9 run_processes_parallel: Add output to tracing messagesStefan Beller, Oct 27, 2015
  7. 4/9 git submodule update: have a dedicated helper for cloningStefan Beller, Oct 27, 2015
  8. 5/9 submodule update: expose parallelism to the userStefan Beller, Oct 27, 2015
  9. Junio C HamanoOct 27, 2015
  10. Stefan BellerOct 28, 2015
  11. Junio C HamanoOct 28, 2015
  12. 6/9 clone: allow an explicit argument for parallel submodule clonesStefan Beller, Oct 27, 2015
  13. Junio C HamanoOct 27, 2015
  14. Stefan BellerOct 28, 2015
  15. 7/9 submodule config: remove name_and_item_from_varStefan Beller, Oct 27, 2015
  16. 8/9 submodule-config: parse_configStefan Beller, Oct 27, 2015
  17. 9/9 fetching submodules: Respect `submodule.jobs` config optionStefan Beller, Oct 27, 2015
  18. Junio C HamanoOct 27, 2015
  19. Junio C HamanoOct 27, 2015
  20. 0/8 Expose the submodule parallelism to the userStefan Beller, Oct 28, 2015
  21. 1/8 run_processes_parallel: Add output to tracing messagesStefan Beller, Oct 28, 2015
  22. Eric SunshineOct 30, 2015
  23. Stefan BellerOct 30, 2015
  24. 2/8 submodule config: keep update strategy aroundStefan Beller, Oct 28, 2015
  25. Eric SunshineOct 30, 2015
  26. Stefan BellerOct 30, 2015
  27. Eric SunshineOct 30, 2015
  28. Stefan BellerOct 30, 2015
  29. 3/8 submodule config: remove name_and_item_from_varStefan Beller, Oct 28, 2015
  30. Eric SunshineOct 30, 2015
  31. Stefan BellerOct 30, 2015
  32. 4/8 submodule-config: parse_configStefan Beller, Oct 28, 2015
  33. Eric SunshineOct 30, 2015
  34. Stefan BellerOct 30, 2015
  35. 5/8 fetching submodules: Respect `submodule.jobs` config optionStefan Beller, Oct 28, 2015
  36. Eric SunshineOct 30, 2015
  37. 6/8 git submodule update: have a dedicated helper for cloningStefan Beller, Oct 28, 2015
  38. Junio C HamanoOct 29, 2015
  39. 7/8 submodule update: expose parallelism to the userStefan Beller, Oct 28, 2015
  40. 8/8 clone: allow an explicit argument for parallel submodule clonesStefan Beller, Oct 28, 2015
  41. Eric SunshineNov 1, 2015
  42. Ramsay JonesOct 29, 2015
  43. Stefan BellerOct 29, 2015
  44. Junio C HamanoOct 29, 2015
  45. Stefan BellerOct 29, 2015
  46. Ramsay JonesOct 29, 2015
  47. Stefan BellerNov 3, 2015
  48. Junio C HamanoOct 29, 2015

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.