git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCHv3 08/11] fetching submodules: respect `submodule.jobs` config option

From
Jens Lehmann <jens.lehmann@web.de>
Date
Nov 13, 2015, 20:47 UTC
Message-ID
<56464C44.6090902@web.de>
In-Reply-To
<CAGZ79kYiPb-GJ37Zq-2ULpLD8Lh_3qAxa0W+u6+5fMrX6YzJdw@mail.gmail.com>
Am 12.11.2015 um 00:34 schrieb Stefan Beller:
Show 12 quoted lines
> On Wed, Nov 11, 2015 at 11:55 AM, Jens Lehmann <Jens.Lehmann@web.de> wrote:
>>>
>>> TL;DR: checkout is serial, network-related stuff only will be using
>>> submodule.jobs
>>
>>
>> My point being: isn't "jobs" a bit too generic for a config option that
>> is only relevant for network-related stuff? Maybe "submodule.fetchJobs"
>> or similar would be better, as you are already thinking about adding
>> other parallelisms with different constraints later?
>
> Actually I don't think that far ahead.

Maybe I've been bitten once too often by too generic names that became a problem later on ... ;-)

Show 14 quoted lines
> (I assume network to be the bottleneck for clone/fetch operations)
> All I want is a saturated network all the time, and as the native git protocol
> doesn't provide that (tcp startup takes time until full band witdth is reached,
> local operations both on client and server) I added the parallel stuff
> to 'smear' different submodule network traffics along the timeline,
> such that we have a better approximation of an always fully saturated link
> for the whole operation. So in the long term future, we maybe want to
> reuse an http/ssh session for a different submodule, possibly interleaving
> the different submodules on the wire to make it even faster. Though that
> may not be helping much.
>
> So we're back at bike shedding about the name. submodule.fetchJobs
> sounds like it only applies to fetching, do you think it's sufficient for clone
> as well?

Hmm, to me fetching is a part of cloning, so I don't have a problem with that. And documenting it accordingly should make it clear to everyone.

Show 7 quoted lines
> Once upon a time, Junio used  'submodule.fetchParallel' or  'submodule.paralle'
> in a discussion[1] for the distinction of the local and networked things.
> [1] Discussing "[PATCH] Add fetch.recurseSubmoduleParallelism config option"
>
> How about submodules.parallelNetwork for the networking part and
> submodules.parallelLocal for the local part? (I don't implement parallelLocal in
> the next few weeks I'd estimate).

If 'submodules.parallelNetwork' will be used for submodule push too as soon as that learns parallel operation, I'm ok with that. But if we don't have good reason to believe the number of jobs for fetch can simply be reused for push, me thinks we should have one config option containing the term "fetch" now and another that contains "push" when we need it later, just to be on the safe side. Otherwise it might be hard to explain to users why 'submodules.parallelNetwork' is only used for fetch and clone and why they have to set 'submodules.parallelPush' for pushing ...

So either 'submodule.fetchParallel' or 'submodule.fetchJobs' is fine for me, and 'submodules.parallelNetwork' is ok too as long as we have reason to believe this value can be used for push later too.

Previous: Stefan BellerNext: Stefan Beller
Message 27 of 34 in “[PATCHv3 00/11] Expose the submodule parallelism to the user”
  1. Stefan BellerNov 4, 2015
  2. 01/11 run_processes_parallel: delimit intermixed task outputStefan Beller, Nov 4, 2015
  3. 02/11 run-command: report failure for degraded output just onceStefan Beller, Nov 4, 2015
  4. Junio C HamanoNov 4, 2015
  5. Stefan BellerNov 4, 2015
  6. Johannes SixtNov 4, 2015
  7. Junio C HamanoNov 4, 2015
  8. Jeff KingNov 4, 2015
  9. Junio C HamanoNov 5, 2015
  10. Jeff KingNov 5, 2015
  11. Junio C HamanoNov 5, 2015
  12. Stefan BellerNov 5, 2015
  13. Junio C HamanoNov 4, 2015
  14. Stefan BellerNov 4, 2015
  15. Junio C HamanoNov 4, 2015
  16. Stefan BellerNov 4, 2015
  17. 03/11 run-command: omit setting file descriptors to non blocking in WindowsStefan Beller, Nov 4, 2015
  18. 04/11 submodule-config: keep update strategy aroundStefan Beller, Nov 4, 2015
  19. 05/11 submodule-config: drop check against NULLStefan Beller, Nov 4, 2015
  20. 06/11 submodule-config: remove name_and_item_from_varStefan Beller, Nov 4, 2015
  21. 07/11 submodule-config: introduce parse_generic_submodule_configStefan Beller, Nov 4, 2015
  22. 08/11 fetching submodules: respect `submodule.jobs` config optionStefan Beller, Nov 4, 2015
  23. Jens LehmannNov 10, 2015
  24. Stefan BellerNov 10, 2015
  25. Jens LehmannNov 11, 2015
  26. Stefan BellerNov 11, 2015
  27. Jens LehmannNov 13, 2015
  28. Stefan BellerNov 13, 2015
  29. 09/11 git submodule update: have a dedicated helper for cloningStefan Beller, Nov 4, 2015
  30. 10/11 submodule update: expose parallelism to the userStefan Beller, Nov 4, 2015
  31. 11/11 clone: allow an explicit argument for parallel submodule clonesStefan Beller, Nov 4, 2015
  32. Junio C HamanoNov 4, 2015
  33. Stefan BellerNov 4, 2015
  34. Junio C HamanoNov 4, 2015

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.