{"thread":{"id":"60876","subject":"cloning the linxu kernel repo at a VPS with small RAM","startedAt":"2024-02-08T17:32:27Z","lastAt":"2024-02-10T10:35:20Z","messageCount":5,"participants":["Toralf Förster","Dragan Simic","Eric Wong"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"488243","messageId":"2f773980-70ec-4ad0-a49c-3ac12c294a39@gmx.de","threadId":"60876","inReplyTo":null,"subject":"cloning the linxu kernel repo at a VPS with small RAM","fromName":"Toralf Förster","fromEmail":"toralf.foerster@gmx.de","sentAt":"2024-02-08T17:32:24Z","receivedAt":"2024-02-08T17:32:27Z","isPatch":false,"sender":{"key":"toralf.foerster@gmx.de","avatar":null},"body":"Situation:\n\nThe command\n     git clone\nhttps://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git\nfails at a virtual server with about 2 GiB RAM under a recent Debian\nbookworm with git 2.39.2. What works for me:\n       git clone --depth=1\nhttps://git.kernel.org/pub/scm/linux/kernel/git/stable\n       cd ./linux\n       git config gc.auto 0\n       git config pack.threads 1\n       git fetch --tags\nas seen in [1].\n\nQ:\nI do wonder if Git could automatically try to deal with only 1.5 GiB\navailable RAM?\n\nTIA\n\n[1]\nhttps://github.com/toralf/tor-relays/blob/main/playbooks/roles/setup/tasks/kernel-git.yaml#L22\n--\nToralf\n"},{"id":"488252","messageId":"6c4e309aa77e534cffdba9ae56032b99@manjaro.org","threadId":"60876","inReplyTo":"2f773980-70ec-4ad0-a49c-3ac12c294a39@gmx.de","subject":"Re: cloning the linxu kernel repo at a VPS with small RAM","fromName":"Dragan Simic","fromEmail":"dsimic@manjaro.org","sentAt":"2024-02-08T20:16:31Z","receivedAt":"2024-02-08T20:16:35Z","isPatch":false,"sender":{"key":"dsimic@manjaro.org","avatar":null},"body":"Hello Toralf,\n\nOn 2024-02-08 18:32, Toralf Förster wrote:\n> Situation:\n> \n> The command\n>     git clone\n> https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git\n> fails at a virtual server with about 2 GiB RAM under a recent Debian\n> bookworm with git 2.39.2. What works for me:\n>       git clone --depth=1\n> https://git.kernel.org/pub/scm/linux/kernel/git/stable\n>       cd ./linux\n>       git config gc.auto 0\n>       git config pack.threads 1\n>       git fetch --tags\n> as seen in [1].\n> \n> Q:\n> I do wonder if Git could automatically try to deal with only 1.5 GiB\n> available RAM?\n\nHere's an excerpt from my ~/.bashrc, which sets the things up\non my Pinebook Pro laptop with an RK3399 SoC and 4 GB of RAM,\nwhich is also thermally constrained:\n\n# Missing nproc(1) is handled properly\nREASONABLE_THREADS=$(nproc 2> /dev/null || echo 1)\nREASONABLE_THREADS=$((${REASONABLE_THREADS} / 2))\n((${REASONABLE_THREADS} == 0)) && REASONABLE_THREADS=1\n\nexport GIT_CONFIG_COUNT=3\nexport GIT_CONFIG_KEY_0='grep.threads'\nexport GIT_CONFIG_VALUE_0=${REASONABLE_THREADS}\nexport GIT_CONFIG_KEY_1='index.threads'\nexport GIT_CONFIG_VALUE_1=${REASONABLE_THREADS}\nexport GIT_CONFIG_KEY_2='pack.threads'\nexport GIT_CONFIG_VALUE_2=${REASONABLE_THREADS}\n\nexport ZSTD_NBTHREADS=${REASONABLE_THREADS}\nunset REASONABLE_THREADS\n\nObviously, this does a bit more than configuring git only.\n\nPerhaps modifying this to additionally take the amount of RAM\ninto the calculation of REASONABLE_THREADS could be a solution\nfor your use case?\n"},{"id":"488254","messageId":"b532a9fd7b363b1dd8da2e8d15d995cc@manjaro.org","threadId":"60876","inReplyTo":"6c4e309aa77e534cffdba9ae56032b99@manjaro.org","subject":"Re: cloning the linxu kernel repo at a VPS with small RAM","fromName":"Dragan Simic","fromEmail":"dsimic@manjaro.org","sentAt":"2024-02-08T20:35:52Z","receivedAt":"2024-02-08T20:35:55Z","isPatch":false,"sender":{"key":"dsimic@manjaro.org","avatar":null},"body":"On 2024-02-08 21:16, Dragan Simic wrote:\n> On 2024-02-08 18:32, Toralf Förster wrote:\n>> Situation:\n>> \n>> The command\n>>     git clone\n>> https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git\n>> fails at a virtual server with about 2 GiB RAM under a recent Debian\n>> bookworm with git 2.39.2. What works for me:\n>>       git clone --depth=1\n>> https://git.kernel.org/pub/scm/linux/kernel/git/stable\n>>       cd ./linux\n>>       git config gc.auto 0\n>>       git config pack.threads 1\n>>       git fetch --tags\n>> as seen in [1].\n>> \n>> Q:\n>> I do wonder if Git could automatically try to deal with only 1.5 GiB\n>> available RAM?\n> \n> Here's an excerpt from my ~/.bashrc, which sets the things up\n> on my Pinebook Pro laptop with an RK3399 SoC and 4 GB of RAM,\n> which is also thermally constrained:\n> \n> # Missing nproc(1) is handled properly\n> REASONABLE_THREADS=$(nproc 2> /dev/null || echo 1)\n> REASONABLE_THREADS=$((${REASONABLE_THREADS} / 2))\n> ((${REASONABLE_THREADS} == 0)) && REASONABLE_THREADS=1\n> \n> export GIT_CONFIG_COUNT=3\n> export GIT_CONFIG_KEY_0='grep.threads'\n> export GIT_CONFIG_VALUE_0=${REASONABLE_THREADS}\n> export GIT_CONFIG_KEY_1='index.threads'\n> export GIT_CONFIG_VALUE_1=${REASONABLE_THREADS}\n> export GIT_CONFIG_KEY_2='pack.threads'\n> export GIT_CONFIG_VALUE_2=${REASONABLE_THREADS}\n> \n> export ZSTD_NBTHREADS=${REASONABLE_THREADS}\n> unset REASONABLE_THREADS\n> \n> Obviously, this does a bit more than configuring git only.\n> \n> Perhaps modifying this to additionally take the amount of RAM\n> into the calculation of REASONABLE_THREADS could be a solution\n> for your use case?\n\nActually, perhaps we could provide the following files as part\nof the git project, which would take the above-described approach,\nand let the distributions package them:\n\n  - /etc/profile.d/git.sh\n  - /etc/profile.d/git.csh\n\nIf everyone agrees, I'd we willing to take a crack on these.\nOf course, all suggestions and thoughts are more than welcome.\n"},{"id":"488261","messageId":"20240209012940.M610977@dcvr","threadId":"60876","inReplyTo":"2f773980-70ec-4ad0-a49c-3ac12c294a39@gmx.de","subject":"Re: cloning the linxu kernel repo at a VPS with small RAM","fromName":"Eric Wong","fromEmail":"e@80x24.org","sentAt":"2024-02-09T01:29:40Z","receivedAt":"2024-02-09T01:35:25Z","isPatch":false,"sender":{"key":"e@80x24.org","avatar":null},"body":"Toralf Förster <toralf.foerster@gmx.de> wrote:\n> I do wonder if Git could automatically try to deal with only 1.5 GiB\n> available RAM?\n\nI have to use pack.threads=1 myself on SMP machines, too.\n\n_SC_AVPHYS_PAGES should probably be accounted for in addition\nto _SC_NPROCESSORS_CONF when calculating the default pack.threads value.\nCalculating good defaults is hard, though...\n\nI've also been meaning to try the newer tombstone-free khashl\nto replace khash in our tree for memory savings:\n\nhttps://attractivechaos.wordpress.com/2019/12/28/deletion-from-hash-tables-without-tombstones/\n\nPerhaps you (or someone else) can take the lead, there.\n"},{"id":"488349","messageId":"c936daec-df9f-44ab-8626-5f0813593246@gmx.de","threadId":"60876","inReplyTo":"2f773980-70ec-4ad0-a49c-3ac12c294a39@gmx.de","subject":"Re: cloning the linxu kernel repo at a VPS with small RAM","fromName":"Toralf Förster","fromEmail":"toralf.foerster@gmx.de","sentAt":"2024-02-10T10:35:14Z","receivedAt":"2024-02-10T10:35:20Z","isPatch":false,"sender":{"key":"toralf.foerster@gmx.de","avatar":null},"body":"On 2/8/24 18:32, Toralf Förster wrote:\n>\n> Q:\n> I do wonder if Git could automatically try to deal with only 1.5 GiB\n> available RAM?\n\nGiven that git fails for the linux kernel repo - where Git was used\nfirst in the wild (?) - it should definitely works for that repo even\nwith a small RAM, right?\n\n--\nToralf\n\n"}]}