git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: RFC: auto-enabling parallel-checkout on NFS

From
JHJeff Hostetler <git@jeffhostetler.com>
Date
Nov 19, 2020, 14:04 UTC
Message-ID
<212a2def-6811-b6e4-0550-ecae2fe0c02c@jeffhostetler.com>
In-Reply-To
<20201119040117.67914-1-matheus.bernardino@usp.br>
On 11/18/20 11:01 PM, Matheus Tavares wrote:
Show 12 quoted lines
> Hi, Jeff
> 
> On Mon, Nov 16, 2020 at 12:19 PM Jeff Hostetler <git@jeffhostetler.com> wrote:
>>
>> I can't really speak to NFS performance, but I have to wonder if there's
>> not something else affecting the results -- 4 and/or 8 core results are
>> better than 16+ results in some columns.  And we get diminishing returns
>> after ~16.
> 
> Yeah, that's a good point. I'm not sure yet what's causing the
> diminishing returns, but Geert and I are investigating. Maybe we are
> hitting some limit for parallelism in this scenario.

I seem to recall back when I was working on this problem that the unzip of each blob was a major pain point. Combine this long delta-chains and each worker would need multiple rounds of read/memmap, unzip, and de-delta before it had the complete blob and could then smudge and write.

This makes me wonder if repacking the repo with shorter delta-chains affects the checkout times. And improves the perf when there are more workers. I'm not saying that this is a solution, but rather an experiment to see if it changes anything and maybe adjust our focus.

Show 17 quoted lines
> 
>> I'm wondering if during these test runs, you were IO vs CPU bound and if
>> VM was a problem.
> 
> I would say we are more IO bound during these tests. While a sequential
> linux-v5.8 checkout usually uses 100% of one core in my laptop's SSD,
> in this setup, it only used 5% to 10%. And even with 64 workers (on a
> single core), CPU usage stays around 60% most of the time.
> 
> About memory, the peak PSS was around 1.75GB, with 64 workers, and the
> machine has 10GB of RAM. But are there other numbers that I should keep
> an eye on while running the test?
> 
>> I'm wondering if setting thread affinity would help here.
> 
> Hmm, I only had one core online during the benchmark, so I think thread
> affinity wouldn't impact the runtime.

I wasn't really thinking about the 64 workers on 1 core case. I was more thinking about the 64 workers on 64 cores case and wondering if workers were being randomly bounced from core to core and we were thrashing.

> 
> Thanks,
> Matheus
>
Previous: Matheus TavaresNext: Ævar Arnfjörð Bjarmason
Message 4 of 10 in “RFC: auto-enabling parallel-checkout on NFS”
  1. Matheus TavaresNov 15, 2020
  2. Jeff HostetlerNov 16, 2020
  3. Matheus TavaresNov 19, 2020
  4. Jeff HostetlerNov 19, 2020
  5. Ævar Arnfjörð BjarmasonNov 20, 2020
  6. Geert JansenNov 23, 2020
  7. Ævar Arnfjörð BjarmasonNov 19, 2020
  8. Jeff HostetlerNov 19, 2020
  9. Geert JansenNov 23, 2020
  10. Ævar Arnfjörð BjarmasonNov 24, 2020

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.