git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCHv2 5/8] fetching submodules: Respect `submodule.jobs` config option

From
ESEric Sunshine <ericsunshine@gmail.com>
Date
Oct 30, 2015, 02:17 UTC
Message-ID
<CAPig+cS5yCzcz6xNyaMLTBFUzPqmbbE8x2_toFxAvXELcc786A@mail.gmail.com>
In-Reply-To
<1446074504-6014-6-git-send-email-sbeller@google.com>
On Wed, Oct 28, 2015 at 7:21 PM, Stefan Beller <sbeller@google.com> wrote:
Show 25 quoted lines
> This allows to configure fetching and updating in parallel
> without having the command line option.
>
> This moved the responsibility to determine how many parallel processes
> to start from builtin/fetch to submodule.c as we need a way to communicate
> "The user did not specify the number of parallel processes in the command
> line options" in the builtin fetch. The submodule code takes care of
> the precedence (CLI > config > default)
>
> Signed-off-by: Stefan Beller <sbeller@google.com>
> ---
> diff --git a/Documentation/config.txt b/Documentation/config.txt
> index 391a0c3..785721a 100644
> --- a/Documentation/config.txt
> +++ b/Documentation/config.txt
> @@ -2643,6 +2643,13 @@ submodule.<name>.ignore::
>         "--ignore-submodules" option. The 'git submodule' commands are not
>         affected by this setting.
>
> +submodule.jobs::
> +       This is used to determine how many submodules can be operated on in
> +       parallel. Specifying a positive integer allows up to that number
> +       of submodules being fetched in parallel. This is used in fetch
> +       and clone operations only. A value of 0 will give some reasonable
> +       default. The defaults may change with different versions of Git.

I'm not sure that "default" is the correct word here. When you talk about a "default", you're normally explaining what happens when the configuration is not provided. (In fact, the default number of jobs is 1, which you may want to document here).

Show 22 quoted lines
>  tag.sort::
>         This variable controls the sort ordering of tags when displayed by
>         linkgit:git-tag[1]. Without the "--sort=<value>" option provided, the
> diff --git a/submodule-config.c b/submodule-config.c
> index 1cea404..07bdcdf 100644
> --- a/submodule-config.c
> +++ b/submodule-config.c
> @@ -32,6 +32,7 @@ enum lookup_type {
>
>  static struct submodule_cache cache;
>  static int is_cache_init;
> +static int parallel_jobs = -1;
>
>  static int config_path_cmp(const struct submodule_entry *a,
>                            const struct submodule_entry *b,
> @@ -235,6 +236,9 @@ static int parse_generic_submodule_config(const char *var,
>                                           const char *key,
>                                           const char *value)
>  {
> +       if (!strcmp(key, "jobs")) {
> +               parallel_jobs = strtol(value, NULL, 10);
> +       }
Style: unnecessary braces

Why does this allow a negative value? The documentation doesn't mention anything about it.

Show 15 quoted lines
>         return 0;
>  }
>
> diff --git a/submodule.c b/submodule.c
> index 0257ea3..188ba02 100644
> --- a/submodule.c
> +++ b/submodule.c
> @@ -752,6 +752,11 @@ int fetch_populated_submodules(const struct argv_array *options,
>         argv_array_push(&spf.args, "--recurse-submodules-default");
>         /* default value, "--submodule-prefix" and its value are added later */
>
> +       if (max_parallel_jobs < 0)
> +               max_parallel_jobs = config_parallel_submodules();
> +       if (max_parallel_jobs < 0)
> +               max_parallel_jobs = 1;

run_process_parallel() itself specially handles max_parallel_jobs==0, so you don't need to consider it here. Okay.

Show 25 quoted lines
> +
>         calculate_changed_submodule_paths();
>         run_processes_parallel(max_parallel_jobs,
>                                get_next_submodule,
> diff --git a/t/t5526-fetch-submodules.sh b/t/t5526-fetch-submodules.sh
> index 1b4ce69..5c3579c 100755
> --- a/t/t5526-fetch-submodules.sh
> +++ b/t/t5526-fetch-submodules.sh
> @@ -470,4 +470,18 @@ test_expect_success "don't fetch submodule when newly recorded commits are alrea
>         test_i18ncmp expect.err actual.err
>  '
>
> +test_expect_success 'fetching submodules respects parallel settings' '
> +       git config fetch.recurseSubmodules true &&
> +       (
> +               cd downstream &&
> +               GIT_TRACE=$(pwd)/trace.out git fetch --jobs 7 &&
> +               grep "7 children" trace.out &&
> +               git config submodule.jobs 8 &&
> +               GIT_TRACE=$(pwd)/trace.out git fetch &&
> +               grep "8 children" trace.out &&
> +               GIT_TRACE=$(pwd)/trace.out git fetch --jobs 9 &&
> +               grep "9 children" trace.out
> +       )
> +'

Not specifically related to this test, but maybe add tests to check cases when --jobs is not specified, and --jobs=1?

Show 5 quoted lines
> +
>  test_done
> --
> 2.5.0.281.g4ed9cdb
>
Previous: Stefan BellerNext: Stefan Beller
Message 36 of 48 in “Expose the submodule parallelism to the user”
  1. 0/9 Expose the submodule parallelism to the userStefan Beller, Oct 27, 2015
  2. 1/9 submodule-config: "goto" removal in parse_config()Stefan Beller, Oct 27, 2015
  3. Jonathan NiederOct 27, 2015
  4. Junio C HamanoOct 27, 2015
  5. 2/9 submodule config: keep update strategy aroundStefan Beller, Oct 27, 2015
  6. 3/9 run_processes_parallel: Add output to tracing messagesStefan Beller, Oct 27, 2015
  7. 4/9 git submodule update: have a dedicated helper for cloningStefan Beller, Oct 27, 2015
  8. 5/9 submodule update: expose parallelism to the userStefan Beller, Oct 27, 2015
  9. Junio C HamanoOct 27, 2015
  10. Stefan BellerOct 28, 2015
  11. Junio C HamanoOct 28, 2015
  12. 6/9 clone: allow an explicit argument for parallel submodule clonesStefan Beller, Oct 27, 2015
  13. Junio C HamanoOct 27, 2015
  14. Stefan BellerOct 28, 2015
  15. 7/9 submodule config: remove name_and_item_from_varStefan Beller, Oct 27, 2015
  16. 8/9 submodule-config: parse_configStefan Beller, Oct 27, 2015
  17. 9/9 fetching submodules: Respect `submodule.jobs` config optionStefan Beller, Oct 27, 2015
  18. Junio C HamanoOct 27, 2015
  19. Junio C HamanoOct 27, 2015
  20. 0/8 Expose the submodule parallelism to the userStefan Beller, Oct 28, 2015
  21. 1/8 run_processes_parallel: Add output to tracing messagesStefan Beller, Oct 28, 2015
  22. Eric SunshineOct 30, 2015
  23. Stefan BellerOct 30, 2015
  24. 2/8 submodule config: keep update strategy aroundStefan Beller, Oct 28, 2015
  25. Eric SunshineOct 30, 2015
  26. Stefan BellerOct 30, 2015
  27. Eric SunshineOct 30, 2015
  28. Stefan BellerOct 30, 2015
  29. 3/8 submodule config: remove name_and_item_from_varStefan Beller, Oct 28, 2015
  30. Eric SunshineOct 30, 2015
  31. Stefan BellerOct 30, 2015
  32. 4/8 submodule-config: parse_configStefan Beller, Oct 28, 2015
  33. Eric SunshineOct 30, 2015
  34. Stefan BellerOct 30, 2015
  35. 5/8 fetching submodules: Respect `submodule.jobs` config optionStefan Beller, Oct 28, 2015
  36. Eric SunshineOct 30, 2015
  37. 6/8 git submodule update: have a dedicated helper for cloningStefan Beller, Oct 28, 2015
  38. Junio C HamanoOct 29, 2015
  39. 7/8 submodule update: expose parallelism to the userStefan Beller, Oct 28, 2015
  40. 8/8 clone: allow an explicit argument for parallel submodule clonesStefan Beller, Oct 28, 2015
  41. Eric SunshineNov 1, 2015
  42. Ramsay JonesOct 29, 2015
  43. Stefan BellerOct 29, 2015
  44. Junio C HamanoOct 29, 2015
  45. Stefan BellerOct 29, 2015
  46. Ramsay JonesOct 29, 2015
  47. Stefan BellerNov 3, 2015
  48. Junio C HamanoOct 29, 2015

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.