# cloning the linxu kernel repo at a VPS with small RAM 5 messages from 2024-02-08 to 2024-02-10. Participants: Toralf Förster, Dragan Simic, Eric Wong. Thread: https://gitlist.dev/t/60876 ## Toralf Förster, 2024-02-08 17:32 Subject: cloning the linxu kernel repo at a VPS with small RAM Message-ID: <2f773980-70ec-4ad0-a49c-3ac12c294a39@gmx.de> URL: https://gitlist.dev/e/2f773980-70ec-4ad0-a49c-3ac12c294a39%40gmx.de ``` Situation: The command git clone https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git fails at a virtual server with about 2 GiB RAM under a recent Debian bookworm with git 2.39.2. What works for me: git clone --depth=1 https://git.kernel.org/pub/scm/linux/kernel/git/stable cd ./linux git config gc.auto 0 git config pack.threads 1 git fetch --tags as seen in [1]. Q: I do wonder if Git could automatically try to deal with only 1.5 GiB available RAM? TIA [1] https://github.com/toralf/tor-relays/blob/main/playbooks/roles/setup/tasks/kernel-git.yaml#L22 -- Toralf ``` ## Dragan Simic, 2024-02-08 20:16 Subject: Re: cloning the linxu kernel repo at a VPS with small RAM Message-ID: <6c4e309aa77e534cffdba9ae56032b99@manjaro.org> URL: https://gitlist.dev/e/6c4e309aa77e534cffdba9ae56032b99%40manjaro.org In-Reply-To: <2f773980-70ec-4ad0-a49c-3ac12c294a39@gmx.de> ``` Hello Toralf, On 2024-02-08 18:32, Toralf Förster wrote: > Situation: > > The command > git clone > https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git > fails at a virtual server with about 2 GiB RAM under a recent Debian > bookworm with git 2.39.2. What works for me: > git clone --depth=1 > https://git.kernel.org/pub/scm/linux/kernel/git/stable > cd ./linux > git config gc.auto 0 > git config pack.threads 1 > git fetch --tags > as seen in [1]. > > Q: > I do wonder if Git could automatically try to deal with only 1.5 GiB > available RAM? Here's an excerpt from my ~/.bashrc, which sets the things up on my Pinebook Pro laptop with an RK3399 SoC and 4 GB of RAM, which is also thermally constrained: # Missing nproc(1) is handled properly REASONABLE_THREADS=$(nproc 2> /dev/null || echo 1) REASONABLE_THREADS=$((${REASONABLE_THREADS} / 2)) ((${REASONABLE_THREADS} == 0)) && REASONABLE_THREADS=1 export GIT_CONFIG_COUNT=3 export GIT_CONFIG_KEY_0='grep.threads' export GIT_CONFIG_VALUE_0=${REASONABLE_THREADS} export GIT_CONFIG_KEY_1='index.threads' export GIT_CONFIG_VALUE_1=${REASONABLE_THREADS} export GIT_CONFIG_KEY_2='pack.threads' export GIT_CONFIG_VALUE_2=${REASONABLE_THREADS} export ZSTD_NBTHREADS=${REASONABLE_THREADS} unset REASONABLE_THREADS Obviously, this does a bit more than configuring git only. Perhaps modifying this to additionally take the amount of RAM into the calculation of REASONABLE_THREADS could be a solution for your use case? ``` ## Dragan Simic, 2024-02-08 20:35 Subject: Re: cloning the linxu kernel repo at a VPS with small RAM Message-ID: URL: https://gitlist.dev/e/b532a9fd7b363b1dd8da2e8d15d995cc%40manjaro.org In-Reply-To: <6c4e309aa77e534cffdba9ae56032b99@manjaro.org> ``` On 2024-02-08 21:16, Dragan Simic wrote: > On 2024-02-08 18:32, Toralf Förster wrote: >> Situation: >> >> The command >> git clone >> https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git >> fails at a virtual server with about 2 GiB RAM under a recent Debian >> bookworm with git 2.39.2. What works for me: >> git clone --depth=1 >> https://git.kernel.org/pub/scm/linux/kernel/git/stable >> cd ./linux >> git config gc.auto 0 >> git config pack.threads 1 >> git fetch --tags >> as seen in [1]. >> >> Q: >> I do wonder if Git could automatically try to deal with only 1.5 GiB >> available RAM? > > Here's an excerpt from my ~/.bashrc, which sets the things up > on my Pinebook Pro laptop with an RK3399 SoC and 4 GB of RAM, > which is also thermally constrained: > > # Missing nproc(1) is handled properly > REASONABLE_THREADS=$(nproc 2> /dev/null || echo 1) > REASONABLE_THREADS=$((${REASONABLE_THREADS} / 2)) > ((${REASONABLE_THREADS} == 0)) && REASONABLE_THREADS=1 > > export GIT_CONFIG_COUNT=3 > export GIT_CONFIG_KEY_0='grep.threads' > export GIT_CONFIG_VALUE_0=${REASONABLE_THREADS} > export GIT_CONFIG_KEY_1='index.threads' > export GIT_CONFIG_VALUE_1=${REASONABLE_THREADS} > export GIT_CONFIG_KEY_2='pack.threads' > export GIT_CONFIG_VALUE_2=${REASONABLE_THREADS} > > export ZSTD_NBTHREADS=${REASONABLE_THREADS} > unset REASONABLE_THREADS > > Obviously, this does a bit more than configuring git only. > > Perhaps modifying this to additionally take the amount of RAM > into the calculation of REASONABLE_THREADS could be a solution > for your use case? Actually, perhaps we could provide the following files as part of the git project, which would take the above-described approach, and let the distributions package them: - /etc/profile.d/git.sh - /etc/profile.d/git.csh If everyone agrees, I'd we willing to take a crack on these. Of course, all suggestions and thoughts are more than welcome. ``` ## Eric Wong, 2024-02-09 01:29 Subject: Re: cloning the linxu kernel repo at a VPS with small RAM Message-ID: <20240209012940.M610977@dcvr> URL: https://gitlist.dev/e/20240209012940.M610977%40dcvr In-Reply-To: <2f773980-70ec-4ad0-a49c-3ac12c294a39@gmx.de> ``` Toralf Förster wrote: > I do wonder if Git could automatically try to deal with only 1.5 GiB > available RAM? I have to use pack.threads=1 myself on SMP machines, too. _SC_AVPHYS_PAGES should probably be accounted for in addition to _SC_NPROCESSORS_CONF when calculating the default pack.threads value. Calculating good defaults is hard, though... I've also been meaning to try the newer tombstone-free khashl to replace khash in our tree for memory savings: https://attractivechaos.wordpress.com/2019/12/28/deletion-from-hash-tables-without-tombstones/ Perhaps you (or someone else) can take the lead, there. ``` ## Toralf Förster, 2024-02-10 10:35 Subject: Re: cloning the linxu kernel repo at a VPS with small RAM Message-ID: URL: https://gitlist.dev/e/c936daec-df9f-44ab-8626-5f0813593246%40gmx.de In-Reply-To: <2f773980-70ec-4ad0-a49c-3ac12c294a39@gmx.de> ``` On 2/8/24 18:32, Toralf Förster wrote: > > Q: > I do wonder if Git could automatically try to deal with only 1.5 GiB > available RAM? Given that git fails for the linux kernel repo - where Git was used first in the wild (?) - it should definitely works for that repo even with a small RAM, right? -- Toralf ```