{"thread":{"id":"54648","subject":"RFC: auto-enabling parallel-checkout on NFS","startedAt":"2020-11-15T19:44:25Z","lastAt":"2020-11-24T12:58:58Z","messageCount":10,"participants":["Matheus Tavares","Jeff Hostetler","Ævar Arnfjörð Bjarmason","Geert Jansen"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"409974","messageId":"20201115194359.67901-1-matheus.bernardino@usp.br","threadId":"54648","inReplyTo":null,"subject":"RFC: auto-enabling parallel-checkout on NFS","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-11-15T19:43:59Z","receivedAt":"2020-11-15T19:44:25Z","isPatch":false,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"Hi, everyone\n\nI've been testing the parallel checkout code on some different machines,\nto benchmark its performance against the sequential version. As\ndiscussed in [1], the biggest speedups, on Linux, are usually seen on\nSSDs and NFS volumes. (I haven't got the chance to benchmark on\nWindows or OSX yet.)\n\nRegarding NFS, I got some 2~3.4x speedups even when the NFS client and\nserver were both running on single-core machines. Here are some runtimes\nfor a linux-v5.8 clone (means of 5 cold-cache executions):\n\n    nfs 3.0              nfs 4.0              nfs 4.1\n1:  183.708 s ± 3.290 s  205.851 s ± 0.844 s  217.317 s ± 3.047 s\n2:  130.510 s ± 3.917 s  139.124 s ± 0.772 s  142.963 s ± 0.765 s\n4:   89.611 s ± 1.032 s  102.701 s ± 1.665 s   94.728 s ± 1.014 s\n8:   68.097 s ± 0.820 s  104.914 s ± 1.239 s   69.359 s ± 0.619 s\n16:  63.999 s ± 0.820 s  104.808 s ± 2.279 s   64.843 s ± 0.587 s\n32:  62.316 s ± 2.095 s  102.105 s ± 1.537 s   64.122 s ± 0.374 s\n64:  63.699 s ± 0.841 s  103.103 s ± 1.319 s   63.532 s ± 0.734 s\n\nThe parallel version was also faster for some smaller checkouts. For\nexample, the following numbers come from a bat-v0.16.0 clone\n(251 files, ~3MB):\n\n nfs 3.0 nfs 4.0 nfs 4.1\n1: 0.853 s ± 0.080 s 0.814 s ± 0.020 s 0.876 s ± 0.065 s\n2: 0.671 s ± 0.020 s 0.702 s ± 0.030 s 0.705 s ± 0.030 s\n4: 0.530 s ± 0.024 s 0.595 s ± 0.020 s 0.570 s ± 0.030 s\n8: 0.470 s ± 0.033 s 0.609 s ± 0.025 s 0.510 s ± 0.031 s\n16: 0.469 s ± 0.037 s 0.616 s ± 0.022 s 0.513 s ± 0.030 s\n32: 0.487 s ± 0.030 s 0.639 s ± 0.018 s 0.527 s ± 0.028 s\n64: 0.520 s ± 0.022 s 0.680 s ± 0.028 s 0.562 s ± 0.026 s\n\nWhile looking at these numbers with Geert (in CC), he had the idea that\nwe could try to detect when the checkout path is within an NFS mount,\nand auto-enable paralellism for this case. This way, users in NFS would\nget the best performance by default. And it seems that using ~16 workers\nwould produce good results regardless of the NFS version that they might\nbe running.\n\nThe major downside is that detecting the file system type is quite\nplatform-dependent, so there is no simple and portable solution. (Also,\nI'm not sure if the optimal number of workers would be the same on\ndifferent OSes). But we decided to give it a try, so this is a\nrough prototype that would work for Linux:\nhttps://github.com/matheustavares/git/commit/2e2c787e2a1742fed8c35dba185b7cd208603de9\n\nAny thoughts on this idea? Or alternative suggestions?\n\nThanks,\nMatheus\n\n[1]: https://lore.kernel.org/git/815137685ac3e41444201316f537db9797dcacd2.1604521276.git.matheus.bernardino@usp.br/\n"},{"id":"410013","messageId":"9c999e38-34db-84bb-3a91-ae2a62b964b5@jeffhostetler.com","threadId":"54648","inReplyTo":"20201115194359.67901-1-matheus.bernardino@usp.br","subject":"Re: RFC: auto-enabling parallel-checkout on NFS","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2020-11-16T15:19:09Z","receivedAt":"2020-11-16T15:19:13Z","isPatch":false,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 11/15/20 2:43 PM, Matheus Tavares wrote:\n> Hi, everyone\n> \n> I've been testing the parallel checkout code on some different machines,\n> to benchmark its performance against the sequential version. As\n> discussed in [1], the biggest speedups, on Linux, are usually seen on\n> SSDs and NFS volumes. (I haven't got the chance to benchmark on\n> Windows or OSX yet.)\n> \n> Regarding NFS, I got some 2~3.4x speedups even when the NFS client and\n> server were both running on single-core machines. Here are some runtimes\n> for a linux-v5.8 clone (means of 5 cold-cache executions):\n> \n>     nfs 3.0              nfs 4.0              nfs 4.1\n> 1:  183.708 s ± 3.290 s  205.851 s ± 0.844 s  217.317 s ± 3.047 s\n> 2:  130.510 s ± 3.917 s  139.124 s ± 0.772 s  142.963 s ± 0.765 s\n> 4:   89.611 s ± 1.032 s  102.701 s ± 1.665 s   94.728 s ± 1.014 s\n> 8:   68.097 s ± 0.820 s  104.914 s ± 1.239 s   69.359 s ± 0.619 s\n> 16:  63.999 s ± 0.820 s  104.808 s ± 2.279 s   64.843 s ± 0.587 s\n> 32:  62.316 s ± 2.095 s  102.105 s ± 1.537 s   64.122 s ± 0.374 s\n> 64:  63.699 s ± 0.841 s  103.103 s ± 1.319 s   63.532 s ± 0.734 s\n> \n> The parallel version was also faster for some smaller checkouts. For\n> example, the following numbers come from a bat-v0.16.0 clone\n> (251 files, ~3MB):\n> \n> nfs 3.0 nfs 4.0 nfs 4.1\n> 1: 0.853 s ± 0.080 s 0.814 s ± 0.020 s 0.876 s ± 0.065 s\n> 2: 0.671 s ± 0.020 s 0.702 s ± 0.030 s 0.705 s ± 0.030 s\n> 4: 0.530 s ± 0.024 s 0.595 s ± 0.020 s 0.570 s ± 0.030 s\n> 8: 0.470 s ± 0.033 s 0.609 s ± 0.025 s 0.510 s ± 0.031 s\n> 16: 0.469 s ± 0.037 s 0.616 s ± 0.022 s 0.513 s ± 0.030 s\n> 32: 0.487 s ± 0.030 s 0.639 s ± 0.018 s 0.527 s ± 0.028 s\n> 64: 0.520 s ± 0.022 s 0.680 s ± 0.028 s 0.562 s ± 0.026 s\n> \n> While looking at these numbers with Geert (in CC), he had the idea that\n> we could try to detect when the checkout path is within an NFS mount,\n> and auto-enable paralellism for this case. This way, users in NFS would\n> get the best performance by default. And it seems that using ~16 workers\n> would produce good results regardless of the NFS version that they might\n> be running.\n> \n> The major downside is that detecting the file system type is quite\n> platform-dependent, so there is no simple and portable solution. (Also,\n> I'm not sure if the optimal number of workers would be the same on\n> different OSes). But we decided to give it a try, so this is a\n> rough prototype that would work for Linux:\n> https://github.com/matheustavares/git/commit/2e2c787e2a1742fed8c35dba185b7cd208603de9\n> \n> Any thoughts on this idea? Or alternative suggestions?\n> \n> Thanks,\n> Matheus\n> \n> [1]: https://lore.kernel.org/git/815137685ac3e41444201316f537db9797dcacd2.1604521276.git.matheus.bernardino@usp.br/\n> \n\nI can't really speak to NFS performance, but I have to wonder if there's\nnot something else affecting the results -- 4 and/or 8 core results are\nbetter than 16+ results in some columns. And we get diminishing returns\nafter ~16.\n\nI'm wondering if during these test runs, you were IO vs CPU bound and if\nVM was a problem. I'm wondering if setting thread affinity would help\nhere.\n\nJeff\n\n"},{"id":"410332","messageId":"20201119040117.67914-1-matheus.bernardino@usp.br","threadId":"54648","inReplyTo":"9c999e38-34db-84bb-3a91-ae2a62b964b5@jeffhostetler.com","subject":"Re: RFC: auto-enabling parallel-checkout on NFS","fromName":"Matheus Tavares","fromEmail":"matheus.bernardino@usp.br","sentAt":"2020-11-19T04:01:17Z","receivedAt":"2020-11-19T04:01:31Z","isPatch":false,"sender":{"key":"matheus.tavb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/12701583?v=4"},"body":"Hi, Jeff\n\nOn Mon, Nov 16, 2020 at 12:19 PM Jeff Hostetler wrote:\n>\n> I can't really speak to NFS performance, but I have to wonder if there's\n> not something else affecting the results -- 4 and/or 8 core results are\n> better than 16+ results in some columns.  And we get diminishing returns\n> after ~16.\n\nYeah, that's a good point. I'm not sure yet what's causing the\ndiminishing returns, but Geert and I are investigating. Maybe we are\nhitting some limit for parallelism in this scenario.\n\n> I'm wondering if during these test runs, you were IO vs CPU bound and if\n> VM was a problem.\n\nI would say we are more IO bound during these tests. While a sequential\nlinux-v5.8 checkout usually uses 100% of one core in my laptop's SSD,\nin this setup, it only used 5% to 10%. And even with 64 workers (on a\nsingle core), CPU usage stays around 60% most of the time.\n\nAbout memory, the peak PSS was around 1.75GB, with 64 workers, and the\nmachine has 10GB of RAM. But are there other numbers that I should keep\nan eye on while running the test?\n\n> I'm wondering if setting thread affinity would help here.\n\nHmm, I only had one core online during the benchmark, so I think thread\naffinity wouldn't impact the runtime.\n\nThanks,\nMatheus\n"},{"id":"410337","messageId":"87y2ixpvos.fsf@evledraar.gmail.com","threadId":"54648","inReplyTo":"20201115194359.67901-1-matheus.bernardino@usp.br","subject":"Re: RFC: auto-enabling parallel-checkout on NFS","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2020-11-19T09:01:07Z","receivedAt":"2020-11-19T09:01:30Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\n> The major downside is that detecting the file system type is quite\n> platform-dependent, so there is no simple and portable solution. (Also,\n> I'm not sure if the optimal number of workers would be the same on\n> different OSes). But we decided to give it a try, so this is a\n> rough prototype that would work for Linux:\n> https://github.com/matheustavares/git/commit/2e2c787e2a1742fed8c35dba185b7cd208603de9\n\nI'm not intrinsically opposed to hardcoding some \"nr_threads = is_nfs()\n? x : y\" as a stopgap.\n\nI do think we should be thinking about a sustainable way of doing this\nsort of thing, this method of testing once and hardcoding something\nisn't a good approach.\n\nIt doesn't anticipate all sorts of different setups, e.g. in this case\nNFS is not a FS, but a protocol, there's probably going to be some\nimplementations where parallel is much worse due to a quirk of the\nimplementation.\n\nI think integrating an optimization run with the relatively new\ngit-maintenance is a better way forward.\n\nYou'd configure e.g.:\n\n maintenance.performanceTests.enabled=true\n maintenance.performanceTests.writeConfig=true\n\nWhich would run e.g.:\n\n git config --type bool core.untrackedCache $(git update-index --test-untracked-cache && echo true || echo false)\n git config checkout.workers $(git maintenance--helper auto-discover-config checkout.workers)\n\nSuch an implementation can be really basic at first, or even just punt\non the test and use your current \"is it NFS?\" check.\n\nBut I think we should be moving to some helper that does the actual test\nlocally when asked/configured by the user, so we're not making a bunch\nof guesses in advance about the size/shape of the repository, OS/nfs/fs\netc.\n"},{"id":"410354","messageId":"212a2def-6811-b6e4-0550-ecae2fe0c02c@jeffhostetler.com","threadId":"54648","inReplyTo":"20201119040117.67914-1-matheus.bernardino@usp.br","subject":"Re: RFC: auto-enabling parallel-checkout on NFS","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2020-11-19T14:04:34Z","receivedAt":"2020-11-19T14:04:58Z","isPatch":false,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 11/18/20 11:01 PM, Matheus Tavares wrote:\n> Hi, Jeff\n> \n> On Mon, Nov 16, 2020 at 12:19 PM Jeff Hostetler wrote:\n>>\n>> I can't really speak to NFS performance, but I have to wonder if there's\n>> not something else affecting the results -- 4 and/or 8 core results are\n>> better than 16+ results in some columns.  And we get diminishing returns\n>> after ~16.\n> \n> Yeah, that's a good point. I'm not sure yet what's causing the\n> diminishing returns, but Geert and I are investigating. Maybe we are\n> hitting some limit for parallelism in this scenario.\n\nI seem to recall back when I was working on this problem that\nthe unzip of each blob was a major pain point. Combine this\nlong delta-chains and each worker would need multiple rounds of\nread/memmap, unzip, and de-delta before it had the complete blob\nand could then smudge and write.\n\nThis makes me wonder if repacking the repo with shorter delta-chains\naffects the checkout times. And improves the perf when there are\nmore workers. I'm not saying that this is a solution, but rather\nan experiment to see if it changes anything and maybe adjust our\nfocus.\n\n> \n>> I'm wondering if during these test runs, you were IO vs CPU bound and if\n>> VM was a problem.\n> \n> I would say we are more IO bound during these tests. While a sequential\n> linux-v5.8 checkout usually uses 100% of one core in my laptop's SSD,\n> in this setup, it only used 5% to 10%. And even with 64 workers (on a\n> single core), CPU usage stays around 60% most of the time.\n> \n> About memory, the peak PSS was around 1.75GB, with 64 workers, and the\n> machine has 10GB of RAM. But are there other numbers that I should keep\n> an eye on while running the test?\n> \n>> I'm wondering if setting thread affinity would help here.\n> \n> Hmm, I only had one core online during the benchmark, so I think thread\n> affinity wouldn't impact the runtime.\n\nI wasn't really thinking about the 64 workers on 1 core case. I was\nmore thinking about the 64 workers on 64 cores case and wondering\nif workers were being randomly bounced from core to core and we were\nthrashing.\n\n> \n> Thanks,\n> Matheus\n> \n"},{"id":"410355","messageId":"e0ddf00d-785b-9220-0082-8be54fb7a7c6@jeffhostetler.com","threadId":"54648","inReplyTo":"87y2ixpvos.fsf@evledraar.gmail.com","subject":"Re: RFC: auto-enabling parallel-checkout on NFS","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2020-11-19T14:11:14Z","receivedAt":"2020-11-19T14:11:18Z","isPatch":false,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 11/19/20 4:01 AM, Ævar Arnfjörð Bjarmason wrote:\n> \n>> The major downside is that detecting the file system type is quite\n>> platform-dependent, so there is no simple and portable solution. (Also,\n>> I'm not sure if the optimal number of workers would be the same on\n>> different OSes). But we decided to give it a try, so this is a\n>> rough prototype that would work for Linux:\n>> https://github.com/matheustavares/git/commit/2e2c787e2a1742fed8c35dba185b7cd208603de9\n> \n> I'm not intrinsically opposed to hardcoding some \"nr_threads = is_nfs()\n> ? x : y\" as a stopgap.\n> \n> I do think we should be thinking about a sustainable way of doing this\n> sort of thing, this method of testing once and hardcoding something\n> isn't a good approach.\n> \n> It doesn't anticipate all sorts of different setups, e.g. in this case\n> NFS is not a FS, but a protocol, there's probably going to be some\n> implementations where parallel is much worse due to a quirk of the\n> implementation.\n> \n> I think integrating an optimization run with the relatively new\n> git-maintenance is a better way forward.\n> \n> You'd configure e.g.:\n> \n> maintenance.performanceTests.enabled=true\n> maintenance.performanceTests.writeConfig=true\n> \n> Which would run e.g.:\n> \n> git config --type bool core.untrackedCache $(git update-index --test-untracked-cache && echo true || echo false)\n> git config checkout.workers $(git maintenance--helper auto-discover-config checkout.workers)\n> \n> Such an implementation can be really basic at first, or even just punt\n> on the test and use your current \"is it NFS?\" check.\n> \n> But I think we should be moving to some helper that does the actual test\n> locally when asked/configured by the user, so we're not making a bunch\n> of guesses in advance about the size/shape of the repository, OS/nfs/fs\n> etc.\n> \n\nI kinda like this idea. It would give us a chance to let maintenance\nperiodically probe the repo/system and improve some of these tuning\nknobs.\n\nJeff\n"},{"id":"410416","messageId":"87mtzcp6u0.fsf@evledraar.gmail.com","threadId":"54648","inReplyTo":"212a2def-6811-b6e4-0550-ecae2fe0c02c@jeffhostetler.com","subject":"Re: RFC: auto-enabling parallel-checkout on NFS","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2020-11-20T12:10:15Z","receivedAt":"2020-11-20T12:10:40Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Nov 19 2020, Jeff Hostetler wrote:\n\n> On 11/18/20 11:01 PM, Matheus Tavares wrote:\n>> Hi, Jeff\n>>\n>> On Mon, Nov 16, 2020 at 12:19 PM Jeff Hostetler wrote:\n>>>\n>>> I can't really speak to NFS performance, but I have to wonder if there's\n>>> not something else affecting the results -- 4 and/or 8 core results are\n>>> better than 16+ results in some columns. And we get diminishing returns\n>>> after ~16.\n>>\n>> Yeah, that's a good point. I'm not sure yet what's causing the\n>> diminishing returns, but Geert and I are investigating. Maybe we are\n>> hitting some limit for parallelism in this scenario.\n>\n> I seem to recall back when I was working on this problem that\n> the unzip of each blob was a major pain point. Combine this\n> long delta-chains and each worker would need multiple rounds of\n> read/memmap, unzip, and de-delta before it had the complete blob\n> and could then smudge and write.\n>\n> This makes me wonder if repacking the repo with shorter delta-chains\n> affects the checkout times. And improves the perf when there are\n> more workers. I'm not saying that this is a solution, but rather\n> an experiment to see if it changes anything and maybe adjust our\n> focus.\n\nI've had part success with \"git gc --keep-largest-pack\" /\ngc.bigPackThreshold=N where N is at least the size you get from a fresh\n\"git clone\" when on NFS.\n\nIt has the effect of essentially implementing a version of what you're\nsuggesting, but in an arguably better way. Your initial clone will have\nwhatever depth of chains you have, but all new objects pulled down will\ngo into new packs/objects that won't share chains with that old big\npack.\n\nSo your repository will be bigger overall, but your old/new pack/pack(s)\nwill eventually come to mostly reflect a cold/hot object storage. So\nwhat you need from a pack is more likely to already have been fetched\ninto the FS cache, and over an NFS mount those requests may have been\npre-fetched/fetched already. You can also more effectively warm the\nlocal OS cache by cat-ing >/dev/null the pack-files that aren't the big\nlarge pack on-login or whatever.\n\n>>\n>>> I'm wondering if during these test runs, you were IO vs CPU bound and if\n>>> VM was a problem.\n>>\n>> I would say we are more IO bound during these tests. While a sequential\n>> linux-v5.8 checkout usually uses 100% of one core in my laptop's SSD,\n>> in this setup, it only used 5% to 10%. And even with 64 workers (on a\n>> single core), CPU usage stays around 60% most of the time.\n>>\n>> About memory, the peak PSS was around 1.75GB, with 64 workers, and the\n>> machine has 10GB of RAM. But are there other numbers that I should keep\n>> an eye on while running the test?\n>>\n>>> I'm wondering if setting thread affinity would help here.\n>>\n>> Hmm, I only had one core online during the benchmark, so I think thread\n>> affinity wouldn't impact the runtime.\n>\n> I wasn't really thinking about the 64 workers on 1 core case. I was\n> more thinking about the 64 workers on 64 cores case and wondering\n> if workers were being randomly bounced from core to core and we were\n> thrashing.\n"},{"id":"410592","messageId":"20201123231817.GA28189@dev-dsk-gerardu-1d-54592b62.us-east-1.amazon.com","threadId":"54648","inReplyTo":"212a2def-6811-b6e4-0550-ecae2fe0c02c@jeffhostetler.com","subject":"Re: RFC: auto-enabling parallel-checkout on NFS","fromName":"Geert Jansen","fromEmail":"gerardu@amazon.com","sentAt":"2020-11-23T23:18:18Z","receivedAt":"2020-11-23T23:18:28Z","isPatch":false,"sender":{"key":"gerardu@amazon.com","avatar":null},"body":"Hi Jeff,\n\nOn Thu, Nov 19, 2020 at 09:04:34AM -0500, Jeff Hostetler wrote:\n\n> On 11/18/20 11:01 PM, Matheus Tavares wrote:\n> >\n> >On Mon, Nov 16, 2020 at 12:19 PM Jeff Hostetler wrote:\n> >>\n> >>I can't really speak to NFS performance, but I have to wonder if there's\n> >>not something else affecting the results -- 4 and/or 8 core results are\n> >>better than 16+ results in some columns. And we get diminishing returns\n> >>after ~16.\n> >\n> >Yeah, that's a good point. I'm not sure yet what's causing the\n> >diminishing returns, but Geert and I are investigating. Maybe we are\n> >hitting some limit for parallelism in this scenario.\n> \n> I seem to recall back when I was working on this problem that\n> the unzip of each blob was a major pain point. Combine this\n> long delta-chains and each worker would need multiple rounds of\n> read/memmap, unzip, and de-delta before it had the complete blob\n> and could then smudge and write.\n\nI think that there are two cases here:\n\n1) (CPU bound case) On local machines with multiple cores and SSD disks,\n checkout is CPU bound and the parallel checkout works because the unzipping\n can now run on multiple CPUs in parallel. Shorter chains would use less CPU\n time and we'd see a smilar benefit on both paralell and sequential checkout.\n\n2) (IO bound case) On networked file systems, file system IO is pretty much\n always the bottleneck for git and similar applications that use small files.\n On NFS calling open() is always a round trip, and so is close() (in the\n absence of delegations and O_CREAT). The latency of these calls depends on\n the NFS server and network distance, but 1ms is a reasonable order of\n magnitude when thinking about this. Beause this 1ms is a lot more than the\n typical CPU time to process a single blob, checkout will be IO bound.\n Parallel checkout works by allowing the application to maintain an IO depth\n > 1 for these workloads, which amortizes the network latency over multiple\n requests.\n\nRegarding the diminishing returns: I did some initial analysis of Mattheus'\ncode and I'm not sure yet. I see the code achieving a high IO depth in our\nserver logs, which would indicate that the diminishing returns are caused by\nfile system contention. This would have to be some kind of general contention\nsince it happens both on NFS and EFS. I will do a deeper investigation on this\nand will report what I find.\n\nBest regards,\nGeert\n"},{"id":"410602","messageId":"20201123233735.GB28189@dev-dsk-gerardu-1d-54592b62.us-east-1.amazon.com","threadId":"54648","inReplyTo":"87y2ixpvos.fsf@evledraar.gmail.com","subject":"Re: RFC: auto-enabling parallel-checkout on NFS","fromName":"Geert Jansen","fromEmail":"gerardu@amazon.com","sentAt":"2020-11-23T23:37:35Z","receivedAt":"2020-11-23T23:37:49Z","isPatch":false,"sender":{"key":"gerardu@amazon.com","avatar":null},"body":"Hi Ævar,\n\nOn Thu, Nov 19, 2020 at 10:01:07AM +0100, Ævar Arnfjörð Bjarmason wrote:\n> \n> > The major downside is that detecting the file system type is quite\n> > platform-dependent, so there is no simple and portable solution. (Also,\n> > I'm not sure if the optimal number of workers would be the same on\n> > different OSes). But we decided to give it a try, so this is a\n> > rough prototype that would work for Linux:\n> > https://github.com/matheustavares/git/commit/2e2c787e2a1742fed8c35dba185b7cd208603de9\n> \n> I'm not intrinsically opposed to hardcoding some \"nr_threads = is_nfs()\n> ? x : y\" as a stopgap.\n> \n> I do think we should be thinking about a sustainable way of doing this\n> sort of thing, this method of testing once and hardcoding something\n> isn't a good approach.\n> \n> It doesn't anticipate all sorts of different setups, e.g. in this case\n> NFS is not a FS, but a protocol, there's probably going to be some\n> implementations where parallel is much worse due to a quirk of the\n> implementation.\n> \n> I think integrating an optimization run with the relatively new\n> git-maintenance is a better way forward.\n> \n> You'd configure e.g.:\n> \n> maintenance.performanceTests.enabled=true\n> maintenance.performanceTests.writeConfig=true\n> \n> Which would run e.g.:\n> \n> git config --type bool core.untrackedCache $(git update-index --test-untracked-cache && echo true || echo false)\n> git config checkout.workers $(git maintenance--helper auto-discover-config checkout.workers)\n> \n> Such an implementation can be really basic at first, or even just punt\n> on the test and use your current \"is it NFS?\" check.\n> \n> But I think we should be moving to some helper that does the actual test\n> locally when asked/configured by the user, so we're not making a bunch\n> of guesses in advance about the size/shape of the repository, OS/nfs/fs\n> etc.\n\nI like this idea as something that will give the best configuration for a given\nrepository. I also know from working with customers for a long time that most\nusers will use the default settings for almost any application, and that\ndefault configurations therefore matter a lot.\n\nThe ideal experience would be, in my view, that a clone or checkout would\nautomatically benefit from parallel checkout, even if this is the first\ncheckout into a new repository.\n\nMaybe both ideas could be combined? We could have some reasonable heuristic\nbased on file system type (and maybe number of CPUs) that gives most of the\nbenefits of paralell checkout, while still being a reasonable compromise that\nthat works across different NFS servers and file systems. Power users that want\nmore aggressive tuning could run the maintenance command that measures file\nsystem performance and comes up with an optimal value for checkout.workers.\n\nRegards,\nGeert\n"},{"id":"410672","messageId":"878saqq5c8.fsf@evledraar.gmail.com","threadId":"54648","inReplyTo":"20201123233735.GB28189@dev-dsk-gerardu-1d-54592b62.us-east-1.amazon.com","subject":"Re: RFC: auto-enabling parallel-checkout on NFS","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2020-11-24T12:58:31Z","receivedAt":"2020-11-24T12:58:58Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Nov 24 2020, Geert Jansen wrote:\n\n> Hi Ævar,\n>\n> On Thu, Nov 19, 2020 at 10:01:07AM +0100, Ævar Arnfjörð Bjarmason wrote:\n>> \n>> > The major downside is that detecting the file system type is quite\n>> > platform-dependent, so there is no simple and portable solution. (Also,\n>> > I'm not sure if the optimal number of workers would be the same on\n>> > different OSes). But we decided to give it a try, so this is a\n>> > rough prototype that would work for Linux:\n>> > https://github.com/matheustavares/git/commit/2e2c787e2a1742fed8c35dba185b7cd208603de9\n>> \n>> I'm not intrinsically opposed to hardcoding some \"nr_threads = is_nfs()\n>> ? x : y\" as a stopgap.\n>> \n>> I do think we should be thinking about a sustainable way of doing this\n>> sort of thing, this method of testing once and hardcoding something\n>> isn't a good approach.\n>> \n>> It doesn't anticipate all sorts of different setups, e.g. in this case\n>> NFS is not a FS, but a protocol, there's probably going to be some\n>> implementations where parallel is much worse due to a quirk of the\n>> implementation.\n>> \n>> I think integrating an optimization run with the relatively new\n>> git-maintenance is a better way forward.\n>> \n>> You'd configure e.g.:\n>> \n>> maintenance.performanceTests.enabled=true\n>> maintenance.performanceTests.writeConfig=true\n>> \n>> Which would run e.g.:\n>> \n>> git config --type bool core.untrackedCache $(git update-index --test-untracked-cache && echo true || echo false)\n>> git config checkout.workers $(git maintenance--helper auto-discover-config checkout.workers)\n>> \n>> Such an implementation can be really basic at first, or even just punt\n>> on the test and use your current \"is it NFS?\" check.\n>> \n>> But I think we should be moving to some helper that does the actual test\n>> locally when asked/configured by the user, so we're not making a bunch\n>> of guesses in advance about the size/shape of the repository, OS/nfs/fs\n>> etc.\n>\n> I like this idea as something that will give the best configuration for a given\n> repository. I also know from working with customers for a long time that most\n> users will use the default settings for almost any application, and that\n> default configurations therefore matter a lot.\n>\n> The ideal experience would be, in my view, that a clone or checkout would\n> automatically benefit from parallel checkout, even if this is the first\n> checkout into a new repository.\n>\n> Maybe both ideas could be combined? We could have some reasonable heuristic\n> based on file system type (and maybe number of CPUs) that gives most of the\n> benefits of paralell checkout, while still being a reasonable compromise that\n> that works across different NFS servers and file systems. Power users that want\n> more aggressive tuning could run the maintenance command that measures file\n> system performance and comes up with an optimal value for checkout.workers.\n\nYeah, I'm not opposed to it in the least. I just think as a practical\nmatter it may become a non-issue if we had something like\nmaintenance.performanceTests.*\n\nBecause we eventually run a \"gc/maintenance\", and there we detach from\nthe terminal, so we can run something like a find_optimal_nr_threads()\nwithout keeping the user waiting.\n\nIf the only reason we had a\nfind_if_nfs_and_nr_cores_to_guess_nr_threads() was because the more\ngeneral find_optimal_nr_threads() took a tad too long when run\ninteractively then changing where/how it's run would make the\nfind_if_nfs_and_nr_cores_to_guess_nr_threads() codepath unnecessary.\n\nThe \"on clone\" case is something we have in general with other speedups\n& sane defaults. E.g. in making the commit-graph.\n\nI haven't kept up with the latest state of that, but there was\nwork/discussions on generating that there too in a smart way. E.g. you\nclone & we either make it or fork to the background and generate it. So\nin practice the user cloning a big repo has sane performance right away\nor really soon.\n\nBut yeah, fully agreed on that we should ship sane defaults when\npossible.\n\nI do think (and I may be wrong here) that in particular with performance\noptions it's more acceptable to not try as hard by default. A lot of\nthem don't matter except for some really large repos, and users working\nwith those tend to be part of some established ecosystem, where e.g. you\nclone chromium.git and a script could set up your custom config for you.\n\nBut maybe there's common cases I'm not aware of where that assumption\ndoesn't hold, e.g. (I'm just guessing here) people cloning arbitrary\nrepos on some VM on Amazon that happens to use NFS, and then wondering\nwhy things are slow.\n"}]}