{"thread":{"id":"44321","subject":"git checkout crashes after server being updated to Debian X86_64","startedAt":"2016-10-18T15:17:28Z","lastAt":"2016-10-19T21:15:16Z","messageCount":7,"participants":["Raffael Reichelt","René Scharfe","Duy Nguyen","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"304339","messageId":"6B2BE996-F696-4EB4-91CA-849D40B8802D@gmail.com","threadId":"44321","inReplyTo":null,"subject":"git checkout crashes after server being updated to Debian X86_64","fromName":"Raffael Reichelt","fromEmail":"raffael.reichelt@gmail.com","sentAt":"2016-10-18T15:17:18Z","receivedAt":"2016-10-18T15:17:28Z","isPatch":false,"sender":{"key":"raffael.reichelt@gmail.com","avatar":"https://gravatar.com/avatar/0d28b854006ce157672805c7ce059a682c53d81f29b53890099a90dc3336a6ec?d=mp&s=160"},"body":"Hello! \n\nI have a serious problem with git, After my provider had updated to a X86_64 architecture git crashes with various memory-related errors. This is happening remote when pushing to the repository from my local machine as well as trying it on a shell on the server itself.\n\nThis are the error-messages:\n\nfatal: Out of memory, realloc failed\nfatal: recursion detected in die handler\nfatal: recursion detected in die handler\n\nor\nfatal: unable to create threaded lstat\nfatal: recursion detected in die handler\nor\nfatal: unable to create threaded lstat\n*** Error in `git': double free or corruption (fasttop): 0x0000000000a8ade0 ***\nfatal: recursion detected in die handler\nAborted\n\nIt’s obviously not a problem of the repository - happens with all of them. I think it is also not a question of size - happens with a 80M Repository as well as with a 500M one.\n\nAny way: did a \n\ngit fsck\nPrüfe Objekt-Verzeichnisse: 100% (256/256), Fertig.\nPrüfe Objekte: 100% (56305/56305), Fertig.\n\ngit gc --auto --prune=today —aggressive\ngit repack\n\nAdditionally I played around some config parameters so my config now looks like:\n[http]\n postbuffer = 524288000\n[pack]\n threads = 1\n deltaCacheSize = 128m\n packSizeLimit = 128m\n windowMemory = 128m\n[core]\n packedGitLimit = 128m\n packedGitWindowSize = 128m\n repositoryformatversion = 0\n filemode = true\n bare = true\n\nI am running \ngit version 2.1.4\n \non\nLinux infongp-de65 3.14.0-ui16196-uiabi1-infong-amd64 #1 SMP Debian 3.14.73-2~ui80+4 (2016-07-13) x86_64 GNU/Linux\n\nAnyone out there to help me getting out of this trouble?\n\nRegards,\nRaffael\n\n"},{"id":"304346","messageId":"1c70da04-d26b-f0af-8939-edc96de586d9@web.de","threadId":"44321","inReplyTo":"6B2BE996-F696-4EB4-91CA-849D40B8802D@gmail.com","subject":"Re: git checkout crashes after server being updated to Debian X86_64","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2016-10-18T16:42:07Z","receivedAt":"2016-10-18T16:42:18Z","isPatch":false,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 18.10.2016 um 17:17 schrieb Raffael Reichelt:\n> Hello!\n>\n> I have a serious problem with git, After my provider had updated to a\n> X86_64 architecture git crashes with various memory-related errors.\n> This is happening remote when pushing to the repository from my local\n> machine as well as trying it on a shell on the server itself.\n>\n> This are the error-messages:\n>\n> fatal: Out of memory, realloc failed\n> fatal: recursion detected in die handler\n> fatal: recursion detected in die handler\n>\n> or\n> fatal: unable to create threaded lstat\n> fatal: recursion detected in die handler\n> or\n> fatal: unable to create threaded lstat\n> *** Error in `git': double free or corruption (fasttop): 0x0000000000a8ade0 ***\n> fatal: recursion detected in die handler\n> Aborted\n>\n> It’s obviously not a problem of the repository - happens with all of\n> them. I think it is also not a question of size - happens with a 80M\n> Repository as well as with a 500M one.\n>\n> Any way: did a\n>\n> git fsck\n> Prüfe Objekt-Verzeichnisse: 100% (256/256), Fertig.\n> Prüfe Objekte: 100% (56305/56305), Fertig.\n>\n> git gc --auto --prune=today —aggressive\n> git repack\n>\n> Additionally I played around some config parameters so my config now looks like:\n> [http]\n> postbuffer = 524288000\n> [pack]\n> threads = 1\n> deltaCacheSize = 128m\n> packSizeLimit = 128m\n> windowMemory = 128m\n> [core]\n> packedGitLimit = 128m\n> packedGitWindowSize = 128m\n> repositoryformatversion = 0\n> filemode = true\n> bare = true\n>\n> I am running\n> git version 2.1.4\n>\n> on\n> Linux infongp-de65 3.14.0-ui16196-uiabi1-infong-amd64 #1 SMP Debian 3.14.73-2~ui80+4 (2016-07-13) x86_64 GNU/Linux\n>\n> Anyone out there to help me getting out of this trouble?\n\nGit 2.1.4 is the version that comes with Debian stable according to \nhttps://packages.debian.org/jessie/git, so I guess using a more recent \nversion is not a reasonable option.\n\nWhat do \"file $(which git)\" and \"ulimit -a\" return? Do you have an \nx86-64 binary and no unnecessarily low limits set?\n\nRené\n\n"},{"id":"304347","messageId":"CE31577A-2172-402B-90AF-683283D26B2D@gmail.com","threadId":"44321","inReplyTo":"1c70da04-d26b-f0af-8939-edc96de586d9@web.de","subject":"Re: git checkout crashes after server being updated to Debian X86_64","fromName":"Raffael Reichelt","fromEmail":"raffael.reichelt@gmail.com","sentAt":"2016-10-18T17:00:36Z","receivedAt":"2016-10-18T17:00:46Z","isPatch":false,"sender":{"key":"raffael.reichelt@gmail.com","avatar":"https://gravatar.com/avatar/0d28b854006ce157672805c7ce059a682c53d81f29b53890099a90dc3336a6ec?d=mp&s=160"},"body":"Hello Renè!\nfile is returning \n\n/usr/bin/git: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=ee62e538d6fe6673d3ba49f0e66bfec784cc32bc, stripped\n\nand ulimit is:\ncore file size (blocks, -c) 0\ndata seg size (kbytes, -d) unlimited\nscheduling priority (-e) 1\nfile size (blocks, -f) unlimited\npending signals (-i) 16382\nmax locked memory (kbytes, -l) 64\nmax memory size (kbytes, -m) unlimited\nopen files (-n) 512\npipe size (512 bytes, -p) 8\nPOSIX message queues (bytes, -q) 819200\nreal-time priority (-r) 0\nstack size (kbytes, -s) 8192\ncpu time (seconds, -t) 1800\nmax user processes (-u) 42\nvirtual memory (kbytes, -v) 786432\nfile locks (-x) unlimited\n\nSupport told me git is limited to 600M\n\nRegrads,\nRafael\n\n> Am 18.10.2016 um 18:42 schrieb René Scharfe :\n> \n> Am 18.10.2016 um 17:17 schrieb Raffael Reichelt:\n>> Hello!\n>> \n>> I have a serious problem with git, After my provider had updated to a\n>> X86_64 architecture git crashes with various memory-related errors.\n>> This is happening remote when pushing to the repository from my local\n>> machine as well as trying it on a shell on the server itself.\n>> \n>> This are the error-messages:\n>> \n>> fatal: Out of memory, realloc failed\n>> fatal: recursion detected in die handler\n>> fatal: recursion detected in die handler\n>> \n>> or\n>> fatal: unable to create threaded lstat\n>> fatal: recursion detected in die handler\n>> or\n>> fatal: unable to create threaded lstat\n>> *** Error in `git': double free or corruption (fasttop): 0x0000000000a8ade0 ***\n>> fatal: recursion detected in die handler\n>> Aborted\n>> \n>> It’s obviously not a problem of the repository - happens with all of\n>> them. I think it is also not a question of size - happens with a 80M\n>> Repository as well as with a 500M one.\n>> \n>> Any way: did a\n>> \n>> git fsck\n>> Prüfe Objekt-Verzeichnisse: 100% (256/256), Fertig.\n>> Prüfe Objekte: 100% (56305/56305), Fertig.\n>> \n>> git gc --auto --prune=today —aggressive\n>> git repack\n>> \n>> Additionally I played around some config parameters so my config now looks like:\n>> [http]\n>> postbuffer = 524288000\n>> [pack]\n>> threads = 1\n>> deltaCacheSize = 128m\n>> packSizeLimit = 128m\n>> windowMemory = 128m\n>> [core]\n>> packedGitLimit = 128m\n>> packedGitWindowSize = 128m\n>> repositoryformatversion = 0\n>> filemode = true\n>> bare = true\n>> \n>> I am running\n>> git version 2.1.4\n>> \n>> on\n>> Linux infongp-de65 3.14.0-ui16196-uiabi1-infong-amd64 #1 SMP Debian 3.14.73-2~ui80+4 (2016-07-13) x86_64 GNU/Linux\n>> \n>> Anyone out there to help me getting out of this trouble?\n> \n> Git 2.1.4 is the version that comes with Debian stable according to https://packages.debian.org/jessie/git, so I guess using a more recent version is not a reasonable option.\n> \n> What do \"file $(which git)\" and \"ulimit -a\" return? Do you have an x86-64 binary and no unnecessarily low limits set?\n> \n> René\n\n"},{"id":"304384","messageId":"95F58766-9A46-49E2-8E8F-7972D936F5F2@gmail.com","threadId":"44321","inReplyTo":"CACsJy8B50daiHWfu7zfVQnn_i_=HbUK3gBPmv1U=EUw7ZyuGPw@mail.gmail.com","subject":"Re: git checkout crashes after server being updated to Debian X86_64","fromName":"Raffael Reichelt","fromEmail":"raffael.reichelt@gmail.com","sentAt":"2016-10-19T14:05:49Z","receivedAt":"2016-10-19T14:13:34Z","isPatch":false,"sender":{"key":"raffael.reichelt@gmail.com","avatar":"https://gravatar.com/avatar/0d28b854006ce157672805c7ce059a682c53d81f29b53890099a90dc3336a6ec?d=mp&s=160"},"body":"\n> Am 19.10.2016 um 15:27 schrieb Duy Nguyen :\n> \n> On Tue, Oct 18, 2016 at 10:17 PM, Raffael Reichelt\n> wrote:\n>> Hello!\n>> \n>> I have a serious problem with git, After my provider had updated to a X86_64 architecture git crashes with various memory-related errors. This is happening remote when pushing to the repository from my local machine as well as trying it on a shell on the server itself.\n>> \n>> This are the error-messages:\n>> \n>> fatal: Out of memory, realloc failed\n>> fatal: recursion detected in die handler\n>> fatal: recursion detected in die handler\n> \n> You other mail said memory is capped at 600MB, which should be a lot\n> for normal repositories. If you set the environment variable\n> GIT_ALLOC_LIMIT to maybe 500MB or lower (convert it to kilobytes\n> first) and git attempts to allocate more than that (just that one\n> time, not total mem) then it's caught and we get a glimpse of how much\n> memory git may need. Unfortunately we can't get a stack trace or\n> anything like that unless you rebuild Git from source.\n\nThis was no change: crashed with the same errors …\n\n> \n>> or\n>> fatal: unable to create threaded lstat\n>> fatal: recursion detected in die handler\n> \n> Hmm.. with \"max user processes (-u) 42\" we should be fine because we\n> only create 20 threads max. What happens if you set core.preloadindex\n> to false? Can it run until the end or hit some other fatal errors?\n> \n\nThis did the trick :) I just repeatedly did a forced checkout and it went until the end without errors\nTHX a lot!\n\nRaffael\n\n"},{"id":"304388","messageId":"CACsJy8B50daiHWfu7zfVQnn_i_=HbUK3gBPmv1U=EUw7ZyuGPw@mail.gmail.com","threadId":"44321","inReplyTo":"6B2BE996-F696-4EB4-91CA-849D40B8802D@gmail.com","subject":"Re: git checkout crashes after server being updated to Debian X86_64","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2016-10-19T13:27:43Z","receivedAt":"2016-10-19T14:28:05Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Tue, Oct 18, 2016 at 10:17 PM, Raffael Reichelt\n wrote:\n> Hello!\n>\n> I have a serious problem with git, After my provider had updated to a X86_64 architecture git crashes with various memory-related errors. This is happening remote when pushing to the repository from my local machine as well as trying it on a shell on the server itself.\n>\n> This are the error-messages:\n>\n> fatal: Out of memory, realloc failed\n> fatal: recursion detected in die handler\n> fatal: recursion detected in die handler\n\nYou other mail said memory is capped at 600MB, which should be a lot\nfor normal repositories. If you set the environment variable\nGIT_ALLOC_LIMIT to maybe 500MB or lower (convert it to kilobytes\nfirst) and git attempts to allocate more than that (just that one\ntime, not total mem) then it's caught and we get a glimpse of how much\nmemory git may need. Unfortunately we can't get a stack trace or\nanything like that unless you rebuild Git from source.\n\n> or\n> fatal: unable to create threaded lstat\n> fatal: recursion detected in die handler\n\nHmm.. with \"max user processes (-u) 42\" we should be fine because we\nonly create 20 threads max. What happens if you set core.preloadindex\nto false? Can it run until the end or hit some other fatal errors?\n\nThere's room for improvement in preload_index(). If we hit resource\nlimit like this, it's not the end of the world and we should be able\nto keep going. But threaded lstat has been available for a long time\nand this is the first time I see a report like this, not sure if it's\nworth fixing.\n-- \nDuy\n"},{"id":"304399","messageId":"20161019134750.GA7256@ash","threadId":"44321","inReplyTo":"CACsJy8B50daiHWfu7zfVQnn_i_=HbUK3gBPmv1U=EUw7ZyuGPw@mail.gmail.com","subject":"[PATCH] Add a knob to abort on die() (was Re: git checkout crashes after ...)","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2016-10-19T13:47:50Z","receivedAt":"2016-10-19T16:21:40Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Wed, Oct 19, 2016 at 08:27:43PM +0700, Duy Nguyen wrote:\n> If you set the environment variable GIT_ALLOC_LIMIT ... git\n> attempts to allocate more than that ... then it's caught and we get\n> a glimpse of how much memory git may need. Unfortunately we can't\n> get a stack trace or anything like that unless you rebuild Git from\n> source.\n\nIt's moments like this that I wish we had a knob to force core\ndumps. And I often modify die_builtin() to add '*(char*)0 = 1;' to\nforce a core dump when I can't figure out some problem based on the\nerror message alone.\n\nSo.. how about we do something like this? We could extend it to abort\non error() as well as die(). Aborting on warning() may be a bit too\nmuch though. On glibc systems we could even print the back trace\nwithout aborting, which helps in some cases.\n\nThe long variable name and value are on purpose to hopefully not\ntrigger this by mistake.\n\ndiff --git a/git.c b/git.c\nindex ab5c99c..5fea224 100644\n--- a/git.c\n+++ b/git.c\n@@ -622,15 +622,34 @@ static int run_argv(int *argcp, const char ***argv)\n \treturn done_alias;\n }\n \n+static NORETURN void die_by_aborting(const char *err, va_list params)\n+{\n+\tvreportf(\"fatal: \", err, params);\n+\tabort();\n+}\n+\n+static NORETURN void die_silently_by_aborting(const char *err, va_list params)\n+{\n+\tabort();\n+}\n+\n int cmd_main(int argc, const char **argv)\n {\n \tconst char *cmd;\n \tint done_help = 0;\n+\tconst char *die_abort_env = getenv(\"GIT_ABORT_ON_FATAL_ERRORS\");\n \n \tcmd = argv[0];\n \tif (!cmd)\n \t\tcmd = \"git-help\";\n \n+\tif (die_abort_env) {\n+\t\tif (!strcmp(die_abort_env, \"yes please\"))\n+\t\t\tset_die_routine(die_by_aborting);\n+\t\telse if (!strcmp(die_abort_env, \"just die\"))\n+\t\t\tset_die_routine(die_silently_by_aborting);\n+\t}\n+\n \ttrace_command_performance(argv);\n \n \t/*\n--\nDuy\n"},{"id":"304423","messageId":"20161019211509.mqt5cqy24eu6wptk@sigill.intra.peff.net","threadId":"44321","inReplyTo":"20161019134750.GA7256@ash","subject":"Re: [PATCH] Add a knob to abort on die() (was Re: git checkout crashes after ...)","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-10-19T21:15:10Z","receivedAt":"2016-10-19T21:15:16Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Oct 19, 2016 at 08:47:50PM +0700, Duy Nguyen wrote:\n\n> On Wed, Oct 19, 2016 at 08:27:43PM +0700, Duy Nguyen wrote:\n> > If you set the environment variable GIT_ALLOC_LIMIT ... git\n> > attempts to allocate more than that ... then it's caught and we get\n> > a glimpse of how much memory git may need. Unfortunately we can't\n> > get a stack trace or anything like that unless you rebuild Git from\n> > source.\n> \n> It's moments like this that I wish we had a knob to force core\n> dumps. And I often modify die_builtin() to add '*(char*)0 = 1;' to\n> force a core dump when I can't figure out some problem based on the\n> error message alone.\n> \n> So.. how about we do something like this? We could extend it to abort\n> on error() as well as die(). Aborting on warning() may be a bit too\n> much though. On glibc systems we could even print the back trace\n> without aborting, which helps in some cases.\n\nI have been tempted by something like this, too, and have occasionally\nresorted to patching in an abort() call to get cores from sub-processes.\n\nI'm not sure how useful it would be in practice in deployed versions of\ngit, though. You'd have to coach the user into finding the core file and\ngenerating the backtrace from it. Something that called backtrace(3)\nautomatically would be more seamless (but provides worse information\nthan gdb), and otherwise it is probably just as easy to ship the user\ncut-and-paste instructions to use gdb.\n\nSee the previous discussion in this subthread:\n\n http://public-inbox.org/git/20150424201734.GA4747@peff.net/T/#u\n\n-Peff\n"}]}