# git status --> Out of memory, realloc failed 7 messages from 2017-02-25 to 2017-03-03. Participants: Carsten Fuchs, René Scharfe, Duy Nguyen. Thread: https://gitlist.dev/t/45223 ## Carsten Fuchs, 2017-02-25 10:13 Subject: git status --> Out of memory, realloc failed Message-ID: <84c02ca1-269e-2f26-c625-476d7f087f5c@cafu.de> URL: https://gitlist.dev/e/84c02ca1-269e-2f26-c625-476d7f087f5c%40cafu.de ``` Dear Git group, I use Git at a web hosting service, where my user account has a memory limit of 768 MB: (uiserver):p7715773:~$ uname -a Linux infongp-de15 3.14.0-ui16322-uiabi1-infong-amd64 #1 SMP Debian 3.14.79-2~ui80+4 (2016-11-17) x86_64 GNU/Linux (uiserver):p7715773:~$ git --version git version 2.1.4 The problem is that `git status` fails with an out of memory error: (uiserver):p7715773:~/cafu$ git status fatal: Out of memory, realloc failed fatal: recursion detected in die handler I talked to their support and their suggestion was to set a couple of memory constraints and to run `git gc`. This seemed to work well – but `git status` still fails: (uiserver):p7715773:~/cafu$ cat ~/.gitconfig [color] ui = auto [user] name = Carsten Fuchs email = carsten.fuchs@cafu.de [core] editor = nano pager = less -M -FRXS packedgitwindowsize = 30m packedgitlimit = 40m [i18n] commitEncoding = ISO-8859-1 [pack] threads = 1 windowMemory = 10m packSizeLimit = 20m deltaCacheSize = 30m (uiserver):p7715773:~/cafu$ git gc Zähle Objekte: 44293, Fertig. Komprimiere Objekte: 100% (24534/24534), Fertig. Schreibe Objekte: 100% (44293/44293), Fertig. Total 44293 (delta 17560), reused 41828 (delta 16708) (uiserver):p7715773:~/cafu$ git status fatal: Out of memory, realloc failed fatal: recursion detected in die handler The repository is tracking about 19000 files which together take 260 MB. The git server version is 2.7.4.1.g5468f9e (Bitbucket) Well, their next response was that they have no solution for me – except, unsurprisingly, coaxing me into a more expensive hosting package. I've read the Git man page about `git config`, but was not able to come up with anything to improve the situation. Any ideas what I could do to reduce the memory consumption of `git status`? Best regards, Carsten PS: Many thanks to Philip Oakley for initial advice at git-users, to which I should have properly subscribed in the first place, as the Google Groups interface seems to lose messages (mine at least, and inadvertently posts them as HTML) and gmane's NNTP interface reports that it is unidirectional/read-only. ``` ## René Scharfe, 2017-03-01 10:02 Subject: Re: git status --> Out of memory, realloc failed Message-ID: URL: https://gitlist.dev/e/ea0722e2-c2bd-bd80-a233-50676efcafda%40web.de In-Reply-To: <84c02ca1-269e-2f26-c625-476d7f087f5c@cafu.de> ``` Am 25.02.2017 um 11:13 schrieb Carsten Fuchs: > Dear Git group, > > I use Git at a web hosting service, where my user account has a memory > limit of 768 MB: > > (uiserver):p7715773:~$ uname -a > Linux infongp-de15 3.14.0-ui16322-uiabi1-infong-amd64 #1 SMP Debian > 3.14.79-2~ui80+4 (2016-11-17) x86_64 GNU/Linux What's the output of "ulimit -a"? > (uiserver):p7715773:~$ git --version > git version 2.1.4 That's quite old. Can you try a more recent version easily (2.12.0 just came out)? I don't remember a specific fix, and memory usage perhaps even increased with newer versions, but ruling out already fixed issues would be nice. > (uiserver):p7715773:~/cafu$ git gc > Zähle Objekte: 44293, Fertig. > Komprimiere Objekte: 100% (24534/24534), Fertig. > Schreibe Objekte: 100% (44293/44293), Fertig. > Total 44293 (delta 17560), reused 41828 (delta 16708) > > (uiserver):p7715773:~/cafu$ git status > fatal: Out of memory, realloc failed > fatal: recursion detected in die handler > > The repository is tracking about 19000 files which together take 260 MB. > The git server version is 2.7.4.1.g5468f9e (Bitbucket) Is your repository publicly accessible? René ``` ## Carsten Fuchs, 2017-03-01 20:12 Subject: Re: git status --> Out of memory, realloc failed Message-ID: URL: https://gitlist.dev/e/cbd281fc-3a4b-b4dc-5dff-145c97cd68d6%40cafu.de In-Reply-To: ``` Hi René, Am 01.03.2017 um 11:02 schrieb René Scharfe: >> I use Git at a web hosting service, where my user account has a memory >> limit of 768 MB: >> >> (uiserver):p7715773:~$ uname -a >> Linux infongp-de15 3.14.0-ui16322-uiabi1-infong-amd64 #1 SMP Debian >> 3.14.79-2~ui80+4 (2016-11-17) x86_64 GNU/Linux > > What's the output of "ulimit -a"? (uiserver):p7715773:~$ ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 1 file size (blocks, -f) unlimited pending signals (-i) 16382 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 512 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) 1800 max user processes (-u) 42 virtual memory (kbytes, -v) 786432 file locks (-x) unlimited >> (uiserver):p7715773:~$ git --version >> git version 2.1.4 > > That's quite old. Can you try a more recent version easily (2.12.0 just came out)? I don't think that they have any newer version available, having just upgraded from Git 1.7 a couple of weeks ago... (1und1) Git 1.7 used to work for me in the same environment, but iirc they also said they switched from 32-bit to 64-bit edition and blame the increased memory consumption on that change. I'll ask for a newer version, but I'd be surprised if this happened. >> The repository is tracking about 19000 files which together take 260 MB. >> The git server version is 2.7.4.1.g5468f9e (Bitbucket) > > Is your repository publicly accessible? Unfortunately, no. There are no big secrets in there, but just a couple of database details so that I cannot make it universally available. I can gladly give you access though. (E.g. by adding your public SSH key?) Best regards, Carsten ``` ## Duy Nguyen, 2017-03-02 09:31 Subject: Re: git status --> Out of memory, realloc failed Message-ID: URL: https://gitlist.dev/e/CACsJy8B3Qy1mra_GwhbYdk5LFrgYx%3D6NRqq%3Dwvw2_wcWCO_yNw%40mail.gmail.com In-Reply-To: ``` On Thu, Mar 2, 2017 at 3:12 AM, Carsten Fuchs wrote: >>> The repository is tracking about 19000 files which together take 260 MB. >>> The git server version is 2.7.4.1.g5468f9e (Bitbucket) >> >> >> Is your repository publicly accessible? > > > Unfortunately, no. There are no big secrets in there, but just a couple of > database details so that I cannot make it universally available. I can > gladly give you access though. (E.g. by adding your public SSH key?) You could also try "git fast-export --anonymize" (read the doc first). I'm not sure if it keeps blob size though, and delta chain might also matter so an anonymized repo may or may not share the same problem, I'm not sure. -- Duy ``` ## René Scharfe, 2017-03-02 20:04 Subject: Re: git status --> Out of memory, realloc failed Message-ID: <6ba86c1a-87a9-f886-273b-7f94c30c4a95@web.de> URL: https://gitlist.dev/e/6ba86c1a-87a9-f886-273b-7f94c30c4a95%40web.de In-Reply-To: ``` Am 01.03.2017 um 21:12 schrieb Carsten Fuchs: > Hi René, > > Am 01.03.2017 um 11:02 schrieb René Scharfe: >>> I use Git at a web hosting service, where my user account has a memory >>> limit of 768 MB: >>> >>> (uiserver):p7715773:~$ uname -a >>> Linux infongp-de15 3.14.0-ui16322-uiabi1-infong-amd64 #1 SMP Debian >>> 3.14.79-2~ui80+4 (2016-11-17) x86_64 GNU/Linux >> >> What's the output of "ulimit -a"? > > (uiserver):p7715773:~$ ulimit -a > core file size (blocks, -c) 0 > data seg size (kbytes, -d) unlimited > scheduling priority (-e) 1 > file size (blocks, -f) unlimited > pending signals (-i) 16382 > max locked memory (kbytes, -l) 64 > max memory size (kbytes, -m) unlimited > open files (-n) 512 > pipe size (512 bytes, -p) 8 > POSIX message queues (bytes, -q) 819200 > real-time priority (-r) 0 > stack size (kbytes, -s) 8192 > cpu time (seconds, -t) 1800 > max user processes (-u) 42 > virtual memory (kbytes, -v) 786432 > file locks (-x) unlimited When I use ulimit -v with lower and lower numbers I can provoke mmap failures on bigger pack files, but not the realloc failures that you're seeing. And your packs should be only up to 20MB anyway (you can check that with "ls -l .git/objects/pack/*.pack"). >>> The repository is tracking about 19000 files which together take 260 MB. >>> The git server version is 2.7.4.1.g5468f9e (Bitbucket) >> >> Is your repository publicly accessible? > > Unfortunately, no. There are no big secrets in there, but just a couple > of database details so that I cannot make it universally available. I > can gladly give you access though. (E.g. by adding your public SSH key?) I'd rather not look at semi-confidential data, and you probably shouldn't hand it to a stranger on the internet anyway. ;) So a shot in the dark: Do you have a lot of untracked files? You could check by cloning your repository locally (which copies only tracked contents). Does "git status" work on the clone? Another one, darker yet: Does "git config core.preloadIndex 0" help? René ``` ## Carsten Fuchs, 2017-03-03 21:10 Subject: Re: git status --> Out of memory, realloc failed Message-ID: <6cdb5c44-76ad-b38a-c9f2-46f8916cbe5a@cafu.de> URL: https://gitlist.dev/e/6cdb5c44-76ad-b38a-c9f2-46f8916cbe5a%40cafu.de In-Reply-To: ``` Hi Duy, Am 2017-03-02 um 10:31 schrieb Duy Nguyen: > You could also try "git fast-export --anonymize" (read the doc first). Eventually this was not needed, but thanks for letting me know about it! I've not been aware of this feature beforehand, learned something. Best regards, Carsten ``` ## Carsten Fuchs, 2017-03-03 21:03 Subject: Re: git status --> Out of memory, realloc failed Message-ID: URL: https://gitlist.dev/e/f0787f93-5bba-f79c-2211-1558b3886c09%40cafu.de In-Reply-To: <6ba86c1a-87a9-f886-273b-7f94c30c4a95@web.de> ``` Hi René, Am 02.03.2017 um 21:04 schrieb René Scharfe: > When I use ulimit -v with lower and lower numbers I can provoke mmap failures on > bigger pack files, but not the realloc failures that you're seeing. And your > packs should be only up to 20MB anyway (you can check that with "ls -l > .git/objects/pack/*.pack"). Yes, there are 9 of them, all about 20 MB in size. > So a shot in the dark: Do you have a lot of untracked files? You could check by > cloning your repository locally (which copies only tracked contents). Does "git > status" work on the clone? I had about 40 modified or untracked files with a combined file size of about 5 MB. Before I got your latest mail, I tried something else: Instead of connecting to the system with the repository via SSH as usual, I accessed the repository by mounting its file system via gvfs-mount onto my local desktop machine. Then I used my local, normally working Git to commit all modified and some untracked files (and deleted those that were left). Running `git status` and other commands took a very long time over a network connection of only 6 MBit/s, but eventually I got everything committed and pushed. Quite in the spirit of your above words, getting rid of untracked files helped: Logging normally via SSH into the remote machine again, `git status` ran normally. Only then did I see your mail and got another clone, where `git status` worked equally well – but then `git diff` failed in a similar manner, continuing the problems of `git status`: (uiserver):p7715773:~/__TEST__$ git clone https://CarstenF@bitbucket.org/CarstenF/website-cafu.git Cloning into 'website-cafu'... Password for 'https://CarstenF@bitbucket.org': remote: Counting objects: 44359, done. remote: Compressing objects: 100% (28777/28777), done. remote: Total 44359 (delta 19201), reused 38661 (delta 14143) Receiving objects: 100% (44359/44359), 168.28 MiB | 20.95 MiB/s, done. Resolving deltas: 100% (19201/19201), done. Checking connectivity... done. Checking out files: 100% (18524/18524), done. (uiserver):p7715773:~/__TEST__$ cd website-cafu/ (uiserver):p7715773:~/__TEST__/website-cafu$ git status On branch master Your branch is up-to-date with 'origin/master'. nothing to commit, working directory clean (uiserver):p7715773:~/__TEST__/website-cafu$ git diff fatal: unable to create threaded lstat (uiserver):p7715773:~/__TEST__/website-cafu$ git diff fatal: Out of memory, realloc failed fatal: Out of memory, realloc failed fatal: Out of memory, realloc failed fatal: recursion detected in die handler > Another one, darker yet: Does "git config core.preloadIndex 0" help? That's it! :-) Once that was set, both `git status` and `git diff` worked normally again! I cannot test `git status` right now in the same situation as before, as I no longer have a working directory that is "as dirty" as before, but the above setting seems to fix the problem promptly and thoroughly! Many thanks for your help! :-) Best regards, Carsten ```