{"thread":{"id":"59026","subject":"Parallelism for submodule update","startedAt":"2023-01-02T16:45:02Z","lastAt":"2023-01-19T22:01:36Z","messageCount":4,"participants":["Zitzmann, Christian","rsbecker@nexbridge.com","Calvin Wan"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"469750","messageId":"DB5PR02MB100691E6422F5E94228F0E0EC8AF79@DB5PR02MB10069.eurprd02.prod.outlook.com","threadId":"59026","inReplyTo":null,"subject":"Parallelism for submodule update","fromName":"Zitzmann, Christian","fromEmail":"christian.zitzmann@vitesco.com","sentAt":"2023-01-02T16:44:57Z","receivedAt":"2023-01-02T16:45:02Z","isPatch":false,"sender":{"key":"christian.zitzmann@vitesco.com","avatar":null},"body":"Hello,\nwe are using git since many years with also heavily using submodules. \n\nWhen updating the submodules, only the fetching part is done in parallel (with config submodule.fetchjobs or --jobs) but the checkout is done sequentially\n\nWhat I’ve recognized when cloning with\n- scalar clone --full-clone --recurse-submodules \nor\n- git clone --filter=blob:none --also-filter-submodules --recurse-submodules \n\nWe loose performance, as the fetch of the blobs is done in the sequential checkout part, instead of in the parallel part.\n\nFurthermore, the utilization - without partial clone - of network and harddisk is not always good, as first the network is utilized (fetch) and then the harddisk (checkout)\n\nAs the checkout part is local to the submodule (no shared resources to block), it would be great if we could move the checkout into the parallelized part.\nE.g. by doing fetch and checkout (with blob fetching) in one step with e.g. run_processes_parallel_tr2\n\nI expect that this significantly improves the performance, especially when using partial clones.\n\nDo you think this is possible? Do I miss anything in my thoughts?\n\nBest regards,\n\nChristian Zitzmann\n\n\n"},{"id":"469751","messageId":"009801d91eca$e1646360$a42d2a20$@nexbridge.com","threadId":"59026","inReplyTo":"DB5PR02MB100691E6422F5E94228F0E0EC8AF79@DB5PR02MB10069.eurprd02.prod.outlook.com","subject":"RE: Parallelism for submodule update","fromName":"","fromEmail":"rsbecker@nexbridge.com","sentAt":"2023-01-02T16:54:24Z","receivedAt":"2023-01-02T16:54:40Z","isPatch":false,"sender":{"key":"randall.becker@nexbridge.ca","avatar":"https://avatars.githubusercontent.com/u/28956764?v=4"},"body":"\n\n>-----Original Message-----\n>From: \nOn January 2, 2023 11:45 AM Christian Zitzmann wrote:\n>we are using git since many years with also heavily using submodules.\n>\n>When updating the submodules, only the fetching part is done in parallel (with\n>config submodule.fetchjobs or --jobs) but the checkout is done sequentially\n>\n>What I’ve recognized when cloning with\n>- scalar clone --full-clone --recurse-submodules or\n>- git clone --filter=blob:none --also-filter-submodules --recurse-submodules\n>\n>\n>We loose performance, as the fetch of the blobs is done in the sequential\n>checkout part, instead of in the parallel part.\n>\n>Furthermore, the utilization - without partial clone - of network and harddisk is not\n>always good, as first the network is utilized (fetch) and then the harddisk\n>(checkout)\n>\n>As the checkout part is local to the submodule (no shared resources to block), it\n>would be great if we could move the checkout into the parallelized part.\n>E.g. by doing fetch and checkout (with blob fetching) in one step with e.g.\n>run_processes_parallel_tr2\n>\n>I expect that this significantly improves the performance, especially when using\n>partial clones.\n>\n>Do you think this is possible? Do I miss anything in my thoughts?\n\nSince this is a platform-specific request, if it happens, this should be a configuration switch that defaults off. On my platform, the file system itself is fairly fast, but the name service traversals and resolutions (what happens in the name service) is a performance problem. Doing the checkout/switch in parallel would actually be counter-productive in my case. So I would keep it off, but I get that other platforms could benefit.\n\nRegards,\nRandall\n\n--\nBrief whoami: NonStop&UNIX developer since approximately\nUNIX(421664400)\nNonStop(211288444200000000)\n-- In real life, I talk too much.\n\n\n\n"},{"id":"470252","messageId":"DB5PR02MB10069A5579A25D465DC126F998AC29@DB5PR02MB10069.eurprd02.prod.outlook.com","threadId":"59026","inReplyTo":"009801d91eca$e1646360$a42d2a20$@nexbridge.com","subject":"RE: Parallelism for submodule update","fromName":"Zitzmann, Christian","fromEmail":"christian.zitzmann@vitesco.com","sentAt":"2023-01-13T10:49:34Z","receivedAt":"2023-01-13T10:50:41Z","isPatch":false,"sender":{"key":"christian.zitzmann@vitesco.com","avatar":null},"body":"Hello Randall,\nYes, I guess this is a quite common that the harddisk is much faster than the Network Services.\nWith the scalar strategy (e.g. blobless clones) the checkout phase does not involve mainly harddisk activity anymore, but includes fetching sources from the remote. So it consumes a lot of Network Services.\nEspecially with parallelism we would gain a lot of performance here, as network and harddisk are utilized in parallel.\nThis could be even a general strategy (without using blobless clones) to have fetch and checkout done together, but both in an parallel Scheme\n\nCurrently it's like this:\n\nMultithreading (mainly network utilization, only small amount of data - Commits and Trees -)\n\tThread1: Fetch submodule1 -> NETWORK\n\tThread2: Fetch submodule2 -> NETWORK\n\t---\n\tThread: Fetch Submodule -> NETWORK\n\nSequential (alternating harddisk and network utilization)\n\tLoop1\n\t\tTry to Checkout Submodule1 commit -> HARDDISK\n\t\tFetch missing objects (e.g. Blobs - big amount of Data) -> NETWORK\n\t\tCheckout Submodule commit -> HARDDISK\n\tLoop2\t\t\n\t\tTry to Checkout Submodule1 commit -> HARDDISK\n\t\tFetch missing objects (e.g. Blobs - big amount of Data) -> NETWORK\n\t\tCheckout Submodule commit -> HARDDISK\n\t\t\n\t\t...\n\tLoop\n\t\tTry to Checkout Submodule commit -> HARDDISK\n\t\tFetch missing objects (e.g. Blobs - big amount of Data) -> NETWORK\n\t\tCheckout Submodule commit -> HARDDISK\n\nHere the Network accesses in the sequential part really have significant waiting times (e.g. name service) with low local resources utilization\n\n\nThe proposal is to change it to a full parallel flow!\n\nMultithreading (both network and harddisk are utilized all the time)\n\tThread1:\n\t\tFetch submodule1 (blobless) -> NETWORK\n\t\tTry to Checkout Submodule commit -> HARDDISK\n\t\tFetch missing objects -> NETWORK\n\t\tCheckout Submodule commit -> HARDDISK\n\tThread2:\n\t\tFetch submodule2 (blobless) -> NETWORK\n\t\tTry to Checkout Submodule commit -> HARDDISK\n\t\tFetch missing objects -> NETWORK\n\t\tCheckout Submodule commit... -> HARDDISK\n\t...\n\n\tThread\n\nThe only negative effect I'd see when having very slow harddisks, or disks that suffer significantly from parallel access, the overall performance could also suffer.\n\nIn general in the partial clone, but even in the full clone approach, network and harddisk utilization will be in parallel, and therefore performance can increase.\n\n\nBest regards\n\nChristian\n\n-----Original Message-----\nFrom: rsbecker@nexbridge.com \nSent: Montag, 2. Januar 2023 17:54\nTo: Zitzmann, Christian ; git@vger.kernel.org\nSubject: RE: Parallelism for submodule update \n\n[Sie erhalten nicht häufig E-Mails von rsbecker@nexbridge.com. Weitere Informationen, warum dies wichtig ist, finden Sie unter https://aka.ms/LearnAboutSenderIdentification ]\n\n>-----Original Message-----\n>From: \nOn January 2, 2023 11:45 AM Christian Zitzmann wrote:\n>we are using git since many years with also heavily using submodules.\n>\n>When updating the submodules, only the fetching part is done in \n>parallel (with config submodule.fetchjobs or --jobs) but the checkout \n>is done sequentially\n>\n>What I’ve recognized when cloning with\n>- scalar clone --full-clone --recurse-submodules or\n>- git clone --filter=blob:none --also-filter-submodules \n>--recurse-submodules \n>\n>We loose performance, as the fetch of the blobs is done in the \n>sequential checkout part, instead of in the parallel part.\n>\n>Furthermore, the utilization - without partial clone - of network and \n>harddisk is not always good, as first the network is utilized (fetch) \n>and then the harddisk\n>(checkout)\n>\n>As the checkout part is local to the submodule (no shared resources to \n>block), it would be great if we could move the checkout into the parallelized part.\n>E.g. by doing fetch and checkout (with blob fetching) in one step with e.g.\n>run_processes_parallel_tr2\n>\n>I expect that this significantly improves the performance, especially \n>when using partial clones.\n>\n>Do you think this is possible? Do I miss anything in my thoughts?\n\nSince this is a platform-specific request, if it happens, this should be a configuration switch that defaults off. On my platform, the file system itself is fairly fast, but the name service traversals and resolutions (what happens in the name service) is a performance problem. Doing the checkout/switch in parallel would actually be counter-productive in my case. So I would keep it off, but I get that other platforms could benefit.\n\nRegards,\nRandall\n\n--\nBrief whoami: NonStop&UNIX developer since approximately\nUNIX(421664400)\nNonStop(211288444200000000)\n-- In real life, I talk too much.\n\n\n\n"},{"id":"470737","messageId":"20230119213911.1515188-1-calvinwan@google.com","threadId":"59026","inReplyTo":"DB5PR02MB100691E6422F5E94228F0E0EC8AF79@DB5PR02MB10069.eurprd02.prod.outlook.com","subject":"Re: Parallelism for submodule update","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2023-01-19T21:39:11Z","receivedAt":"2023-01-19T22:01:36Z","isPatch":false,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"Hi Christian,\n\nI investigated this as well about 2 months ago and am happy to share my\nfindings with you :)\n\n> When updating the submodules, only the fetching part is done in parallel (with config submodule.fetchjobs or --jobs) but the checkout is done sequentially\n\nCorrect.\n\n> What I’ve recognized when cloning with\n> - scalar clone --full-clone --recurse-submodules \n> or\n> - git clone --filter=blob:none --also-filter-submodules --recurse-submodules \n> \n> We loose performance, as the fetch of the blobs is done in the sequential checkout part, instead of in the parallel part.\n> \n> Furthermore, the utilization - without partial clone - of network and harddisk is not always good, as first the network is utilized (fetch) and then the harddisk (checkout)\n\nAlso an astute observation that separating out the parallelization of\nfetch and checkout doesn't allow us to fully use our resources.\n\n> As the checkout part is local to the submodule (no shared resources to block), it would be great if we could move the checkout into the parallelized part.\n> E.g. by doing fetch and checkout (with blob fetching) in one step with e.g. run_processes_parallel_tr2\n> \n> I expect that this significantly improves the performance, especially when using partial clones.\n> \n> Do you think this is possible? Do I miss anything in my thoughts?\n\nSort of. The issue with run_processes_parallel_tr2 is that it creates a\nsubprocess with a git command. There is no git command that we can call\nthat lets us do both the correct fetch and checkout command, so first\nyou would have to create a new option/command for that (and what happens\nif we want to add to that parallelization in the future? Create another\noption/command?). I think we can do better than that!\n\n`git submodule update`, when called from clone, essentially does 4\nthings to the submodule: init, clone, checkout, and recursively calls\nitself for child submodules. One idea I had was to separate out the\nindividual tasks that `git submodule update` does and create a new\nsubmodule--helper command (eg. git submodule--helper update-helper) that\ncalls those individual tasks. Then, clone would directly call\nrun_processes_parallel_tr2 with the new submodule--helper command and\neach process separated by submodule.\n\nThis is what I imagine the general idea of what\n`git clone --recurse-submodules` would look like:\nsuperproject cloning\nrun_processes_parallel_tr2(git submodule--helper update-helper)\n Init\n Clone\n Checkout\n Recursive git submdodule update-helper\n\nI'll discuss what I think are the benefits of this approach:\n- The entirety of submodule update would be parallelized so network and\n hard disk resources can be used together\n- There only needs to be one config option that controls how many\n parallel processes to spawn\n- Any new features to submodule update are automatically parallelized\n\nThe drawback is that any new feature that would cause a race condition\nif run in parallel would have to have additional locking code written\nfor it since separating it out would be difficult. In this case, only\nadding lines to .gitmodules in init is at risk of a race condition, but\nfortunately that can be handled first in series before running\neverything else in parallel.\n\nI haven't started implementing this and am not planning to fix this in\nthe near future. This is because we are planning a more long-term\nsolution (2y+) to solve problems like this (notice how much simpler it\nwould've been to add parallelization if we didn't have to create\nsubprocesses for every separate git command and instead could call from\na variety of library functions). So if you need the parallelizations\nsooner or want to scratch your itch, you're more than welcome to\nimplement it. Happy to bounce ideas off of and review any patches for\nthis!\n\nThanks,\nCalvin\n"}]}