{"thread":{"id":"46214","subject":"[PATCH] die routine: change recursion limit from 1 to 1024","startedAt":"2017-06-19T22:00:58Z","lastAt":"2017-06-24T18:32:45Z","messageCount":17,"participants":["Ævar Arnfjörð Bjarmason","Stefan Beller","Jeff King","Simon Ruderich","Morten Welinder","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"322622","messageId":"20170619220036.22656-1-avarab@gmail.com","threadId":"46214","inReplyTo":null,"subject":"[PATCH] die routine: change recursion limit from 1 to 1024","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2017-06-19T22:00:36Z","receivedAt":"2017-06-19T22:00:58Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Change the recursion limit for the default die routine from a *very*\nlow 1 to 1024. This ensures that infinite recursions are broken, but\ndoesn't lose error messages.\n\nThe intent of the existing code, as explained in commit\ncd163d4b4e (\"usage.c: detect recursion in die routines and bail out\nimmediately\", 2012-11-14), is to break infinite recursion in cases\nwhere the die routine itself dies.\n\nHowever, doing that very aggressively by immediately printing out\n\"recursion detected in die handler\" if we've already called die() once\nmeans that threaded invocations of git can go through the following\nflow:\n\n 1. Start a bunch of threads\n\n 2. The threads start invoking die(), pretty much at the same time.\n\n 3. The first die() invocation will be in the middle of trying to\n    print out its message by the time another thread dies, that other\n    thread then runs into the recursion limit and dies with \"recursion\n    detected in die handler\".\n\n 4. Due to a race condition the initial error may never get printed\n    before the \"recursion detected\" thread calls exit() and aborts the\n    program.\n\nAn example of this is running a threaded grep against e.g. linux.git:\n\n    git grep -P --threads=4 '(*LIMIT_RECURSION=1)(*LIMIT_MATCH=1)-?-?-?---$'\n\nWith the current version of git this will print some combination of\nmultiple PCRE failures that caused the abort and multiple \"recursion\ndetected\", some invocations will print out multiple \"recursion\ndetected\" errors with no PCRE error at all!\n\nNow, git-grep could make use of the pluggable error facility added in\ncommit c19a490e37 (\"usage: allow pluggable die-recursion checks\",\n2013-04-16).\n\nThat should be done for git-grep in particular because before this\nchange (and after) it'll potentially print out the exact same error\nfrom the N threads it starts, that should be de-duplicated.\n\nBut let's start by improving the default behavior shared across all of\ngit. Right now the common case is not an infinite recursion in the\nhandler, but us losing error messages by default because we're overly\nparanoid about our recursion check.\n\nSo let's just set the recursion limit to a number higher than the\nnumber of threads we're ever likely to spawn. Now we won't lose\nerrors, and if we have a recursing die handler we'll still die within\nmicroseconds.\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n usage.c | 4 +++-\n 1 file changed, 3 insertions(+), 1 deletion(-)\n\ndiff --git a/usage.c b/usage.c\nindex 2f87ca69a8..1c198d4882 100644\n--- a/usage.c\n+++ b/usage.c\n@@ -44,7 +44,9 @@ static void warn_builtin(const char *warn, va_list params)\n static int die_is_recursing_builtin(void)\n {\n \tstatic int dying;\n-\treturn dying++;\n+\tstatic int recursion_limit = 1024;\n+\n+\treturn dying++ > recursion_limit;\n }\n \n /* If we are in a dlopen()ed .so write to a global variable would segfault\n-- \n2.13.1.518.g3df882009\n\n"},{"id":"322624","messageId":"CAGZ79kaHXX4ne_hfjVs7PaeLzHc+CbVSZ1hCNz0ebG5F3uJEzg@mail.gmail.com","threadId":"46214","inReplyTo":"20170619220036.22656-1-avarab@gmail.com","subject":"Re: [PATCH] die routine: change recursion limit from 1 to 1024","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-06-19T22:08:20Z","receivedAt":"2017-06-19T22:08:55Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"> Now, git-grep could make use of the pluggable error facility added in\n> commit c19a490e37 (\"usage: allow pluggable die-recursion checks\",\n> 2013-04-16).\n\nI think we should do that instead (though I have not looked at the downsides\nof this), because...\n>\n> So let's just set the recursion limit to a number higher than the\n> number of threads we're ever likely to spawn. Now we won't lose\n> errors, and if we have a recursing die handler we'll still die within\n> microseconds.\n\nhow are we handling access to that global variable?\nDo we need to hold a mutex to be correct? or rather hope that\nit works across threads, not counting on it, because each thread\nindividually would count up to 1024?\n\nI would prefer if we kept the number as low as \"at most\none screen of lines\".\n\n> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n> ---\n>  usage.c | 4 +++-\n>  1 file changed, 3 insertions(+), 1 deletion(-)\n>\n> diff --git a/usage.c b/usage.c\n> index 2f87ca69a8..1c198d4882 100644\n> --- a/usage.c\n> +++ b/usage.c\n> @@ -44,7 +44,9 @@ static void warn_builtin(const char *warn, va_list params)\n>  static int die_is_recursing_builtin(void)\n>  {\n>         static int dying;\n> -       return dying++;\n> +       static int recursion_limit = 1024;\n> +\n> +       return dying++ > recursion_limit;\n>  }\n>\n>  /* If we are in a dlopen()ed .so write to a global variable would segfault\n> --\n> 2.13.1.518.g3df882009\n>\n"},{"id":"322627","messageId":"87podz8v6v.fsf@gmail.com","threadId":"46214","inReplyTo":"CAGZ79kaHXX4ne_hfjVs7PaeLzHc+CbVSZ1hCNz0ebG5F3uJEzg@mail.gmail.com","subject":"Re: [PATCH] die routine: change recursion limit from 1 to 1024","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2017-06-19T22:32:56Z","receivedAt":"2017-06-19T22:33:05Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Jun 19 2017, Stefan Beller jotted:\n\n>> Now, git-grep could make use of the pluggable error facility added in\n>> commit c19a490e37 (\"usage: allow pluggable die-recursion checks\",\n>> 2013-04-16).\n>\n> I think we should do that instead (though I have not looked at the downsides\n> of this), because...\n\nIt makes sense to do that in addition to what I'm doing here (or if the\napproach I'm taking doesn't make sense, some other patch to fix the\ngeneral issue in the default handler).\n\nI'm going to try to get around to fixing the grep behavior in a\nfollow-up patch, this is a fix for the overzealous recursion detection\nin the default handler needlessly causing other issues.\n\n>>\n>> So let's just set the recursion limit to a number higher than the\n>> number of threads we're ever likely to spawn. Now we won't lose\n>> errors, and if we have a recursing die handler we'll still die within\n>> microseconds.\n>\n> how are we handling access to that global variable?\n> Do we need to hold a mutex to be correct? or rather hope that\n> it works across threads, not counting on it, because each thread\n> individually would count up to 1024?\n\nIt's not guarded by a mutex and the ++ here and the reads from it are\nboth racy.\n\nHowever, for its stated purpose that's fine, even if we're racily\nincrementing it and losing some updates some will get through, which is\ngood enough for an infinite recursion detection. We don't really care if\nwe die at exactly 1 or exactly 1024.\n\n> I would prefer if we kept the number as low as \"at most\n> one screen of lines\".\n\nIn practice this is the case in git, because the programs that would\nencounter this are going to be spawning less than screenfull of threads,\nassuming (as is the case) that each thread might print out one error.\n\nThe semantics of that aren't changed with this patch, the difference is\nthat you're going to get e.g. N repeats of a meaningful error instead of\nN repeats of either the meaningful error OR \"recursion detected in die\nhandler\", depending on your luck.\n\nI.e. in current git (after a few runs to get an unlucky one):\n\n    $ git grep -P --threads=10 '(*LIMIT_RECURSION=1)(*LIMIT_MATCH=1)-?-?-?---$'\n    fatal: recursion detected in die handler\n    fatal: recursion detected in die handler\n    fatal: recursion detected in die handler\n\nOr if you're lucky at least one of these will be the actual error:\n\n    $ git grep -P --threads=10 '(*LIMIT_RECURSION=1)(*LIMIT_MATCH=1)-?-?-?---$'\n    fatal: recursion detected in die handler\n    fatal: pcre_exec failed with error code -8\n    fatal: recursion detected in die handler\n    fatal: recursion detected in die handler\n    fatal: recursion detected in die handler\n\nBut with this change:\n\n    $ ~/g/git/git-grep -P --threads=10 '(*LIMIT_RECURSION=1)(*LIMIT_MATCH=1)-?-?-?---$'\n    fatal: pcre2_jit_match failed with error code -47: match limit exceeded\n    fatal: pcre2_jit_match failed with error code -47: match limit exceeded\n    fatal: pcre2_jit_match failed with error code -47: match limit exceeded\n\n(The error message is different because I compiled with PCRE v2 locally,\ninstead of the system PCRE v1, but that doesn't matter for this example)\n\nOver 1000 runs thish is how that breaks down on my machine, without this\npatch. I've replaced the recursion error with just \"R\" and the PCRE\nerror with \"P\", and shown them in descending order by occurrence, lines\nwithout a \"P\" only printed out the recursion error:\n\n    $ (for i in {1..1000}; do git grep -P --threads=10 '(*LIMIT_RECURSION=1)(*LIMIT_MATCH=1)-?-?-?---$' 2>&1|perl -pe 's/^fatal: r.*/R/; s/^fatal: p.*/P/' | tr '\\n' ' '; echo; done)|sort|uniq -c|sort -nr|head -n 10\n        247 R R\n        136 P R R\n        122 P R\n        112 R R R\n        108 R\n         59 R P R\n         54 R P\n         54 P\n         31 P R R R\n         21 R R P\n\nThere's a long tail I've omitted there of alterations to that. As this\nshows in >10% of cases we don't print out any meaningful error at\nall. But with this change:\n\n    $ (for i in {1..1000}; do ~/g/git/git-grep -P --threads=10 '(*LIMIT_RECURSION=1)(*LIMIT_MATCH=1)-?-?-?---$' 2>&1|perl -pe 's/^fatal: r.*/R/; s/^fatal: p.*/P/' | tr '\\n' ' '; echo; done)|sort|uniq -c|sort -nr|head -n 10\n        377 P P\n        358 P P P\n        192 P\n         63 P P P P\n          8 P P P P P\n          2 P P P P P P\n\nWe will always show a meaningful error, but may of course do so multiple\ntimes, which is a subject for a fix in git-grep in particular, but the\npoint is again, to fix the general case for the default handler.\n\nSomething something sorry about the long mail didn't have time to write\na shorter one :)\n\n>> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n>> ---\n>>  usage.c | 4 +++-\n>>  1 file changed, 3 insertions(+), 1 deletion(-)\n>>\n>> diff --git a/usage.c b/usage.c\n>> index 2f87ca69a8..1c198d4882 100644\n>> --- a/usage.c\n>> +++ b/usage.c\n>> @@ -44,7 +44,9 @@ static void warn_builtin(const char *warn, va_list params)\n>>  static int die_is_recursing_builtin(void)\n>>  {\n>>         static int dying;\n>> -       return dying++;\n>> +       static int recursion_limit = 1024;\n>> +\n>> +       return dying++ > recursion_limit;\n>>  }\n>>\n>>  /* If we are in a dlopen()ed .so write to a global variable would segfault\n>> --\n>> 2.13.1.518.g3df882009\n>>\n"},{"id":"322628","messageId":"CAGZ79kbfQfFkQKANrs5TEKQGF5_kiEf5WR9wyZP6jcixbaqwZQ@mail.gmail.com","threadId":"46214","inReplyTo":"87podz8v6v.fsf@gmail.com","subject":"Re: [PATCH] die routine: change recursion limit from 1 to 1024","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-06-19T22:38:04Z","receivedAt":"2017-06-19T22:38:10Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Mon, Jun 19, 2017 at 3:32 PM, Ævar Arnfjörð Bjarmason\n<avarab@gmail.com> wrote:\n>\n> On Mon, Jun 19 2017, Stefan Beller jotted:\n>\n>>> Now, git-grep could make use of the pluggable error facility added in\n>>> commit c19a490e37 (\"usage: allow pluggable die-recursion checks\",\n>>> 2013-04-16).\n>>\n>> I think we should do that instead (though I have not looked at the downsides\n>> of this), because...\n>\n> It makes sense to do that in addition to what I'm doing here (or if the\n> approach I'm taking doesn't make sense, some other patch to fix the\n> general issue in the default handler).\n>\n> I'm going to try to get around to fixing the grep behavior in a\n> follow-up patch, this is a fix for the overzealous recursion detection\n> in the default handler needlessly causing other issues.\n>\n>>>\n>>> So let's just set the recursion limit to a number higher than the\n>>> number of threads we're ever likely to spawn. Now we won't lose\n>>> errors, and if we have a recursing die handler we'll still die within\n>>> microseconds.\n>>\n>> how are we handling access to that global variable?\n>> Do we need to hold a mutex to be correct? or rather hope that\n>> it works across threads, not counting on it, because each thread\n>> individually would count up to 1024?\n>\n> It's not guarded by a mutex and the ++ here and the reads from it are\n> both racy.\n>\n> However, for its stated purpose that's fine, even if we're racily\n> incrementing it and losing some updates some will get through, which is\n> good enough for an infinite recursion detection. We don't really care if\n> we die at exactly 1 or exactly 1024.\n>\n>> I would prefer if we kept the number as low as \"at most\n>> one screen of lines\".\n>\n> In practice this is the case in git, because the programs that would\n> encounter this are going to be spawning less than screenfull of threads,\n> assuming (as is the case) that each thread might print out one error.\n>\n> The semantics of that aren't changed with this patch, the difference is\n> that you're going to get e.g. N repeats of a meaningful error instead of\n> N repeats of either the meaningful error OR \"recursion detected in die\n> handler\", depending on your luck.\n>\n> I.e. in current git (after a few runs to get an unlucky one):\n>\n>     $ git grep -P --threads=10 '(*LIMIT_RECURSION=1)(*LIMIT_MATCH=1)-?-?-?---$'\n>     fatal: recursion detected in die handler\n>     fatal: recursion detected in die handler\n>     fatal: recursion detected in die handler\n>\n> Or if you're lucky at least one of these will be the actual error:\n>\n>     $ git grep -P --threads=10 '(*LIMIT_RECURSION=1)(*LIMIT_MATCH=1)-?-?-?---$'\n>     fatal: recursion detected in die handler\n>     fatal: pcre_exec failed with error code -8\n>     fatal: recursion detected in die handler\n>     fatal: recursion detected in die handler\n>     fatal: recursion detected in die handler\n>\n> But with this change:\n>\n>     $ ~/g/git/git-grep -P --threads=10 '(*LIMIT_RECURSION=1)(*LIMIT_MATCH=1)-?-?-?---$'\n>     fatal: pcre2_jit_match failed with error code -47: match limit exceeded\n>     fatal: pcre2_jit_match failed with error code -47: match limit exceeded\n>     fatal: pcre2_jit_match failed with error code -47: match limit exceeded\n>\n> (The error message is different because I compiled with PCRE v2 locally,\n> instead of the system PCRE v1, but that doesn't matter for this example)\n>\n> Over 1000 runs thish is how that breaks down on my machine, without this\n> patch. I've replaced the recursion error with just \"R\" and the PCRE\n> error with \"P\", and shown them in descending order by occurrence, lines\n> without a \"P\" only printed out the recursion error:\n>\n>     $ (for i in {1..1000}; do git grep -P --threads=10 '(*LIMIT_RECURSION=1)(*LIMIT_MATCH=1)-?-?-?---$' 2>&1|perl -pe 's/^fatal: r.*/R/; s/^fatal: p.*/P/' | tr '\\n' ' '; echo; done)|sort|uniq -c|sort -nr|head -n 10\n>         247 R R\n>         136 P R R\n>         122 P R\n>         112 R R R\n>         108 R\n>          59 R P R\n>          54 R P\n>          54 P\n>          31 P R R R\n>          21 R R P\n>\n> There's a long tail I've omitted there of alterations to that. As this\n> shows in >10% of cases we don't print out any meaningful error at\n> all. But with this change:\n>\n>     $ (for i in {1..1000}; do ~/g/git/git-grep -P --threads=10 '(*LIMIT_RECURSION=1)(*LIMIT_MATCH=1)-?-?-?---$' 2>&1|perl -pe 's/^fatal: r.*/R/; s/^fatal: p.*/P/' | tr '\\n' ' '; echo; done)|sort|uniq -c|sort -nr|head -n 10\n>         377 P P\n>         358 P P P\n>         192 P\n>          63 P P P P\n>           8 P P P P P\n>           2 P P P P P P\n>\n> We will always show a meaningful error, but may of course do so multiple\n> times, which is a subject for a fix in git-grep in particular, but the\n> point is again, to fix the general case for the default handler.\n>\n> Something something sorry about the long mail didn't have time to write\n> a shorter one :)\n>\n\nActually this convinced me (and it would be lovely to have such observations\nin the commit message).\n\nThanks,\nStefan\n"},{"id":"322739","messageId":"20170620155459.a6e7pypxx6vpdbvs@sigill.intra.peff.net","threadId":"46214","inReplyTo":"20170619220036.22656-1-avarab@gmail.com","subject":"Re: [PATCH] die routine: change recursion limit from 1 to 1024","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-06-20T15:54:59Z","receivedAt":"2017-06-20T15:55:05Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Jun 19, 2017 at 10:00:36PM +0000, Ævar Arnfjörð Bjarmason wrote:\n\n> Change the recursion limit for the default die routine from a *very*\n> low 1 to 1024. This ensures that infinite recursions are broken, but\n> doesn't lose error messages.\n> \n> The intent of the existing code, as explained in commit\n> cd163d4b4e (\"usage.c: detect recursion in die routines and bail out\n> immediately\", 2012-11-14), is to break infinite recursion in cases\n> where the die routine itself dies.\n\nI agree that was the original intent, but I think it also does something\nelse. Anytime die() recurses, even a single level, we're going to cover\nup the original failure with the one that happened inside die(), which\nis almost certainly the less interesting of the two.\n\nE.g., if I\n\n  die_errno(\"unable to open %s\", filename);\n\nand then the die handler calls malloc() and fails, you'd much rather see\nthat first message than \"out of memory\".\n\nTo be fair, \"die handler is recursing\" is _also_ not helpful, but at\nleast it's clear that this is a bug (and IMHO it should be marked with\nBUG()). Saying \"out of memory\" tells you about the second error, but it\ndoesn't tell you that we've masked the first error. So it may lead to\nmore confusion in the long run.\n\nI wonder if we can get the best of both, though. Can we make the logic\nmore like:\n\n  if (!dying) {\n\t/* ok, normal */\n\treturn 0;\n  } else if (dying < 1024) {\n\t/* only show the warning once */\n\tif (dying == 1)\n\t\twarning(\"I heard you liked errors, so I put a die() in your die()\");\n\treturn 0; /* don't bail yet */\n  } else {\n\tBUG(\"recursion detected in die handler\");\n  }\n\n> Now, git-grep could make use of the pluggable error facility added in\n> commit c19a490e37 (\"usage: allow pluggable die-recursion checks\",\n> 2013-04-16).\n\nYeah, I think this is a bug in git-grep and should be fixed, independent\nof this commit. You should be able to use as a template the callbacks\nadded by the child of c19a490e37:\n\n  1ece66bc9 (run-command: use thread-aware die_is_recursing routine,\n  2013-04-16)\n\n-Peff\n"},{"id":"322740","messageId":"20170620161514.ygbflanx4pldc7n7@sigill.intra.peff.net","threadId":"46214","inReplyTo":"20170620155459.a6e7pypxx6vpdbvs@sigill.intra.peff.net","subject":"Re: [PATCH] die routine: change recursion limit from 1 to 1024","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-06-20T16:15:14Z","receivedAt":"2017-06-20T16:15:20Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Jun 20, 2017 at 11:54:59AM -0400, Jeff King wrote:\n\n> > Now, git-grep could make use of the pluggable error facility added in\n> > commit c19a490e37 (\"usage: allow pluggable die-recursion checks\",\n> > 2013-04-16).\n> \n> Yeah, I think this is a bug in git-grep and should be fixed, independent\n> of this commit. You should be able to use as a template the callbacks\n> added by the child of c19a490e37:\n> \n>   1ece66bc9 (run-command: use thread-aware die_is_recursing routine,\n>   2013-04-16)\n\nTo clarify, I think anytime we spawn worker threads that might run die()\nwe probably need to be installing not just a custom recursion handler\nbut a custom die function.\n\nIt's weird to see:\n\n  $ git grep ...\n  fatal: some error\n  fatal: some error\n  fatal: some error\n\nOr even:\n\n  $ git grep ...\n  fatal: some error\n  some actual results\n  more actual results\n\nI'm not sure what the _right_ thing is there, but it probably involves\nrecording the per-thread errors in individual buffers, waiting for the\nmaster to pthread_join() them all, and then doing something like:\n\n  /* this also covers the case of only having 1 error */\n  int all_errors_identical = 1;\n  for (i = 1; i < nr_errors; i++) {\n\tif (strcmp(errors[i], errors[i-1])) {\n\t\tall_errors_identical = 0;\n\t\tbreak;\n\t}\n  }\n\n  if (all_errors_identical) {\n\t/* just show it */\n\tdie(\"%s\", errors[0]);\n  } else {\n\tfor (i = 0; i < nr_errors; i++)\n\t\terror(\"%s\", errors[i]);\n\tdie(\"multiple errors encountered\");\n  }\n\nI don't know if we'd want to actually get into the details of what\n\"multiple errors\" means. It isn't _all_ of the possible errors, because\neach thread stopped running when it hit the die(). But it's also not\njust one error.\n\nActually, I guess we could just pick one error and show only that one.\nThat would most closely match the non-threaded case. And it's way\nsimpler than what I wrote above.\n\nHrm. I guess you could even do that without buffering if you allow the\nthread to take down the whole process. The only thing you'd need to do\nis teach die() to take a mutex so that we don't racily show multiple\nerrors.\n\nThat seems like the best option (I almost just deleted my entire email,\nbut maybe the thought process in leading there is useful).\n\n-Peff\n"},{"id":"322753","messageId":"87lgom8pew.fsf@gmail.com","threadId":"46214","inReplyTo":"20170620155459.a6e7pypxx6vpdbvs@sigill.intra.peff.net","subject":"Re: [PATCH] die routine: change recursion limit from 1 to 1024","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2017-06-20T18:49:59Z","receivedAt":"2017-06-20T18:50:08Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Jun 20 2017, Jeff King jotted:\n\n> On Mon, Jun 19, 2017 at 10:00:36PM +0000, Ævar Arnfjörð Bjarmason wrote:\n>\n>> Change the recursion limit for the default die routine from a *very*\n>> low 1 to 1024. This ensures that infinite recursions are broken, but\n>> doesn't lose error messages.\n>>\n>> The intent of the existing code, as explained in commit\n>> cd163d4b4e (\"usage.c: detect recursion in die routines and bail out\n>> immediately\", 2012-11-14), is to break infinite recursion in cases\n>> where the die routine itself dies.\n>\n> I agree that was the original intent, but I think it also does something\n> else. Anytime die() recurses, even a single level, we're going to cover\n> up the original failure with the one that happened inside die(), which\n> is almost certainly the less interesting of the two.\n>\n> E.g., if I\n>\n>   die_errno(\"unable to open %s\", filename);\n>\n> and then the die handler calls malloc() and fails, you'd much rather see\n> that first message than \"out of memory\".\n>\n> To be fair, \"die handler is recursing\" is _also_ not helpful, but at\n> least it's clear that this is a bug (and IMHO it should be marked with\n> BUG()). Saying \"out of memory\" tells you about the second error, but it\n> doesn't tell you that we've masked the first error. So it may lead to\n> more confusion in the long run.\n>\n> I wonder if we can get the best of both, though. Can we make the logic\n> more like:\n>\n>   if (!dying) {\n> \t/* ok, normal */\n> \treturn 0;\n>   } else if (dying < 1024) {\n> \t/* only show the warning once */\n> \tif (dying == 1)\n> \t\twarning(\"I heard you liked errors, so I put a die() in your die()\");\n> \treturn 0; /* don't bail yet */\n>   } else {\n> \tBUG(\"recursion detected in die handler\");\n>   }\n\nIf I understand you correctly this on top:\n\n    diff --git a/usage.c b/usage.c\n    index 1c198d4882..f6d5af2bb4 100644\n    --- a/usage.c\n    +++ b/usage.c\n    @@ -46,7 +46,19 @@ static int die_is_recursing_builtin(void)\n     \tstatic int dying;\n     \tstatic int recursion_limit = 1024;\n\n    -\treturn dying++ > recursion_limit;\n    +\tdying++;\n    +\n    +\tif (!dying) {\n    +\t\t/* ok, normal */\n    +\t\treturn 0;\n    +\t} else if (dying < recursion_limit) {\n    +\t\t/* only show the warning once */\n    +\t\tif (dying == 1)\n    +\t\t\twarning(\"die() called many times. Recursion error or racy threaded death!\");\n    +\t\treturn 0; /* don't bail yet */\n    +\t} else {\n    +\t\treturn 1;\n    +\t}\n     }\n\n     /* If we are in a dlopen()ed .so write to a global variable would segfault\n\nWill yield this over 1000 runs, i.e. mostly this works and we emit the\nwarning (although sometimes we miss it, and we might even emit it twice\nor more due to an extra race condition we have now):\n\n    $ (for i in {1..1000}; do ~/g/git/git-grep -P --threads=10 '(*LIMIT_RECURSION=1)(*LIMIT_MATCH=1)-?-?-?---$' 2>&1|perl -pe 's/^fatal: r.*/R/; s/^fatal: p.*/P/; s/^warning.*/W/' | tr '\\n' ' '; echo; done)|sort|uniq -c|sort -nr|head -n 20\n        245 W P P\n        222 W P P P\n        212 W P\n         47 P W P\n         36 W P P P P\n         35 P P P\n         35 P P\n         30 P W P P\n         16 P P W\n         14 W W P P\n         12 P P W P\n         11 W W P P P\n         11 P\n          8 W P W P P\n          8 P P P P\n          7 W P P P P P\n          7 P W P P P\n          6 W P W P\n          4 P W P W\n          3 W W P P W\n\nI think it makes sense to apply that on top, even though we could print\nmore than 1 warning here it makes sense to alert the user that we're in\nthe middle of some racy death, it explains the multiple lines of output\nthey'll probably (but not always!) get.\n\nAs you can see the third most common case is that we needlessly print\nout the warning, i.e. we have only one error anyway, but we can't\nguarantee that, so it probably makes sense to emit it.\n\nTo reply to your 20170620161514.ygbflanx4pldc7n7@sigill.intra.peff.net\ndownthread here (where you talk about setting up a custom die handler\nfor grep) yeah that would make sense, but as long as we're supplying\nthis default behavior (and not outlawing using it with pthreads) it\nmakes sense to get out of our own way with this recursion detection.\n\nI think my patch (possibly with the fixup above, depending on what we\nthink about dupe warnings) is just fine to fix this. This is already a\nsuper-rare edge case in grep, and to the extent that it would be a\nproblem for anyone it's because our paranoid recursion detector totally\nhides the error, I don't think it's worth worrying about us printing a\nfew dupe error messages under threading for something that almost never\nhappens.\n\nAt least, that's starting to go beyond my bothering to hack on it :)\n\n>> Now, git-grep could make use of the pluggable error facility added in\n>> commit c19a490e37 (\"usage: allow pluggable die-recursion checks\",\n>> 2013-04-16).\n>\n> Yeah, I think this is a bug in git-grep and should be fixed, independent\n> of this commit. You should be able to use as a template the callbacks\n> added by the child of c19a490e37:\n>\n>   1ece66bc9 (run-command: use thread-aware die_is_recursing routine,\n>   2013-04-16)\n>\n> -Peff\n"},{"id":"322755","messageId":"20170620190500.ioqbo72usj6dmdqy@sigill.intra.peff.net","threadId":"46214","inReplyTo":"87lgom8pew.fsf@gmail.com","subject":"Re: [PATCH] die routine: change recursion limit from 1 to 1024","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-06-20T19:05:00Z","receivedAt":"2017-06-20T19:05:07Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Jun 20, 2017 at 08:49:59PM +0200, Ævar Arnfjörð Bjarmason wrote:\n\n> As you can see the third most common case is that we needlessly print\n> out the warning, i.e. we have only one error anyway, but we can't\n> guarantee that, so it probably makes sense to emit it.\n\nRight, my suggestion actually doesn't help much with the multi-threaded\ncase. The warning is pointless but it will still be racily shown,\nbecause we can't tell the difference between races and recursion. So\nit's better, but fundamentally still pretty ugly. And we'd want to\nactually fix the real problem.\n\n> To reply to your 20170620161514.ygbflanx4pldc7n7@sigill.intra.peff.net\n> downthread here (where you talk about setting up a custom die handler\n> for grep) yeah that would make sense, but as long as we're supplying\n> this default behavior (and not outlawing using it with pthreads) it\n> makes sense to get out of our own way with this recursion detection.\n\nI actually was more or less proposing that we consider stock die (no\ncustom handler) with pthreads to be outlawed. It sometimes kind of does\nthe right thing, but in a racy way. Probably threads calling die()\nshould consider their error-handling a bit more carefully.\n\nThat said, in my other mail I arrived at \"just put a mutex into die()\"\nwhich I think would remove the raciness, and give the default behavior\nof \"the first thread to call die will take down the whole process and\nget its single error printed\". That actually seems pretty reasonable.\nAnd it makes the \"recursion or race?\" question go away.\n\nI'm not quite sure how to implement it, though. I think you'd want to\ntake the lock in die() itself, because it would make the is-recursing\ncheck all the way to the exit an atomic unit. But who is responsible for\nunlocking then?  The point of die_routine() is that it never returns.\nThat's fine if it truly calls exit(), but the threaded die handler in\nrun-command.c does a pthread_exit(). So _somebody_ has to unlock that so\nother threads don't block if they try to die().\n\nWhat a mess.\n\n> I think my patch (possibly with the fixup above, depending on what we\n> think about dupe warnings) is just fine to fix this. This is already a\n> super-rare edge case in grep, and to the extent that it would be a\n> problem for anyone it's because our paranoid recursion detector totally\n> hides the error, I don't think it's worth worrying about us printing a\n> few dupe error messages under threading for something that almost never\n> happens.\n> \n> At least, that's starting to go beyond my bothering to hack on it :)\n\nYes, I think your patch with my suggestion above is a strict improvement\nover what's there. You'd see the warning any time you might have seen\n\"die is recursing\", but at least you _also_ get to see the real error.\n\n-Peff\n"},{"id":"322802","messageId":"20170621081209.gfqqkselek4bqdjl@ruderich.org","threadId":"46214","inReplyTo":"87lgom8pew.fsf@gmail.com","subject":"Re: [PATCH] die routine: change recursion limit from 1 to 1024","fromName":"Simon Ruderich","fromEmail":"simon@ruderich.org","sentAt":"2017-06-21T08:12:09Z","receivedAt":"2017-06-21T08:12:26Z","isPatch":true,"sender":{"key":"simon@ruderich.org","avatar":"https://avatars.githubusercontent.com/u/390994?v=4"},"body":"On Tue, Jun 20, 2017 at 08:49:59PM +0200, Ævar Arnfjörð Bjarmason wrote:\n> If I understand you correctly this on top:\n>\n>     diff --git a/usage.c b/usage.c\n>     index 1c198d4882..f6d5af2bb4 100644\n>     --- a/usage.c\n>     +++ b/usage.c\n>     @@ -46,7 +46,19 @@ static int die_is_recursing_builtin(void)\n>      \tstatic int dying;\n>      \tstatic int recursion_limit = 1024;\n>\n>     -\treturn dying++ > recursion_limit;\n>     +\tdying++;\n>     +\n>     +\tif (!dying) {\n\nThis will never trigger as dying was incremented two lines\nbefore. But I think it's already handled by the dying <\nrecursion_limit case so we can just omit it.\n\n>     +\t\t/* ok, normal */\n>     +\t\treturn 0;\n>     +\t} else if (dying < recursion_limit) {\n>     +\t\t/* only show the warning once */\n>     +\t\tif (dying == 1)\n>     +\t\t\twarning(\"die() called many times. Recursion error or racy threaded death!\");\n>     +\t\treturn 0; /* don't bail yet */\n>     +\t} else {\n>     +\t\treturn 1;\n>     +\t}\n>      }\n\nMaybe restructure it like this:\n\n    dying++\n    if (dying > recursion_limit)\n        return 1;\n    if (dying == 1)\n        warning();\n    return 0;\n\nBtw. is there a reason why recursion_limit is a static variable\nand not a constant/define?\n\nRegards\nSimon\n-- \n+ privacy is necessary\n+ using gnupg http://gnupg.org\n+ public key id: 0x92FEFDB7E44C32F9\n"},{"id":"322803","messageId":"87h8z98xcl.fsf@gmail.com","threadId":"46214","inReplyTo":"20170621081209.gfqqkselek4bqdjl@ruderich.org","subject":"Re: [PATCH] die routine: change recursion limit from 1 to 1024","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2017-06-21T10:10:50Z","receivedAt":"2017-06-21T10:10:58Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Jun 21 2017, Simon Ruderich jotted:\n\n> On Tue, Jun 20, 2017 at 08:49:59PM +0200, Ævar Arnfjörð Bjarmason wrote:\n>> If I understand you correctly this on top:\n>>\n>>     diff --git a/usage.c b/usage.c\n>>     index 1c198d4882..f6d5af2bb4 100644\n>>     --- a/usage.c\n>>     +++ b/usage.c\n>>     @@ -46,7 +46,19 @@ static int die_is_recursing_builtin(void)\n>>      \tstatic int dying;\n>>      \tstatic int recursion_limit = 1024;\n>>\n>>     -\treturn dying++ > recursion_limit;\n>>     +\tdying++;\n>>     +\n>>     +\tif (!dying) {\n>\n> This will never trigger as dying was incremented two lines\n> before. But I think it's already handled by the dying <\n> recursion_limit case so we can just omit it.\n>\n>>     +\t\t/* ok, normal */\n>>     +\t\treturn 0;\n>>     +\t} else if (dying < recursion_limit) {\n>>     +\t\t/* only show the warning once */\n>>     +\t\tif (dying == 1)\n>>     +\t\t\twarning(\"die() called many times. Recursion error or racy threaded death!\");\n>>     +\t\treturn 0; /* don't bail yet */\n>>     +\t} else {\n>>     +\t\treturn 1;\n>>     +\t}\n>>      }\n>\n> Maybe restructure it like this:\n>\n>     dying++\n>     if (dying > recursion_limit)\n>         return 1;\n>     if (dying == 1)\n>         warning();\n>     return 0;\n\nThanks, silly mistake. Will fix.\n\n> Btw. is there a reason why recursion_limit is a static variable\n> and not a constant/define?\n\nNope, will make it a const. Thanks.\n"},{"id":"322863","messageId":"20170621204742.15722-1-avarab@gmail.com","threadId":"46214","inReplyTo":"87podz8v6v.fsf@gmail.com","subject":"[PATCH v2] die(): stop hiding errors due to overzealous recursion guard","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2017-06-21T20:47:42Z","receivedAt":"2017-06-21T20:47:59Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"Change the recursion limit for the default die routine from a *very*\nlow 1 to 1024. This ensures that infinite recursions are broken, but\ndoesn't lose the meaningful error messages under threaded execution\nwhere threads concurrently start to die.\n\nThe intent of the existing code, as explained in commit\ncd163d4b4e (\"usage.c: detect recursion in die routines and bail out\nimmediately\", 2012-11-14), is to break infinite recursion in cases\nwhere the die routine itself calls die(), and would thus infinitely\nrecurse.\n\nHowever, doing that very aggressively by immediately printing out\n\"recursion detected in die handler\" if we've already called die() once\nmeans that threaded invocations of git can end up only printing out\nthe \"recursion detected\" error, while hiding the meaningful error.\n\nAn example of this is running a threaded grep which dies on execution\nagainst pretty much any repo, git.git will do:\n\n    git grep -P --threads=8 '(*LIMIT_MATCH=1)-?-?-?---$'\n\nWith the current version of git this will print some combination of\nmultiple PCRE failures that caused the abort and multiple \"recursion\ndetected\", some invocations will print out multiple \"recursion\ndetected\" errors with no PCRE error at all!\n\nBefore this change, running the above grep command 1000 times against\ngit.git[1] and taking the top 20 results will on my system yield the\nfollowing distribution of actual errors (\"E\") and recursion\nerrors (\"R\"):\n\n    322 E R\n    306 E\n    116 E R R\n     65 R R\n     54 R E\n     49 E E\n     44 R\n     15 E R R R\n      9 R R R\n      7 R E R\n      5 R R E\n      3 E R R R R\n      2 E E R\n      1 R R R R\n      1 R R R E\n      1 R E R R\n\nThe exact results are obviously random and system-dependent, but this\nshows the race condition in this code. Some small part of the time\nwe're about to print out the actual error (\"E\") but another thread's\nrecursion error beats us to it, and sometimes we print out nothing but\nthe recursion error.\n\nWith this change we get, now with \"W\" to mean the new warning being\nemitted indicating that we've called die() many times:\n\n    502 E\n    160 E W E\n    120 E E\n     53 E W\n     35 E W E E\n     34 W E E\n     29 W E E E\n     16 E E W\n     16 E E E\n     11 W E E E E\n      7 E E W E\n      4 W E\n      3 W W E E\n      2 E W E E E\n      1 W W E\n      1 W E W E\n      1 E W W E E E\n      1 E W W E E\n      1 E W W E\n      1 E W E E W\n\nWhich still sucks a bit, due to a still present race-condition in this\ncode we're sometimes going to print out several errors still, or\nseveral warnings, or two duplicate errors without the warning.\n\nBut we will never have a case where we completely hide the actual\nerror as we do now.\n\nNow, git-grep could make use of the pluggable error facility added in\ncommit c19a490e37 (\"usage: allow pluggable die-recursion checks\",\n2013-04-16). There's other threaded code that calls set_die_routine()\nor set_die_is_recursing_routine().\n\nBut this is about fixing the general die() behavior with threading\nwhen we don't have such a custom routine yet. Right now the common\ncase is not an infinite recursion in the handler, but us losing error\nmessages by default because we're overly paranoid about our recursion\ncheck.\n\nSo let's just set the recursion limit to a number higher than the\nnumber of threads we're ever likely to spawn. Now we won't lose\nerrors, and if we have a recursing die handler we'll still die within\nmicroseconds.\n\nThere are race conditions in this code itself, in particular the\n\"dying\" variable is not thread mutexed, so we e.g. won't be dying at\nexactly 1024, or for that matter even be able to accurately test\n\"dying == 2\", see the cases where we print out more than one \"W\"\nabove.\n\nBut that doesn't really matter, for the recursion guard we just need\nto die \"soon\", not at exactly 1024 calls, and for printing the correct\nerror and only one warning most of the time in the face of threaded\ndeath this is good enough and a net improvement on the current code.\n\n1. for i in {1..1000}; do git grep -P --threads=8 '(*LIMIT_MATCH=1)-?-?-?---$' 2>&1|perl -pe 's/^fatal: r.*/R/; s/^fatal: p.*/E/; s/^warning.*/W/' | tr '\\n' ' '; echo; done | sort | uniq -c | sort -nr | head -n 20\n\nSigned-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n---\n\nThis replaces v1 and takes into account the feedback in this thread\n(thanks everyone!).\n\nThe commit message is also much improved and includes more rationale\noriginally in my reply to Stefan in 87podz8v6v.fsf@gmail.com\n\n usage.c | 18 +++++++++++++++++-\n 1 file changed, 17 insertions(+), 1 deletion(-)\n\ndiff --git a/usage.c b/usage.c\nindex 2f87ca69a8..1ea7df9a20 100644\n--- a/usage.c\n+++ b/usage.c\n@@ -44,7 +44,23 @@ static void warn_builtin(const char *warn, va_list params)\n static int die_is_recursing_builtin(void)\n {\n \tstatic int dying;\n-\treturn dying++;\n+\t/*\n+\t * Just an arbitrary number X where \"a < x < b\" where \"a\" is\n+\t * \"maximum number of pthreads we'll ever plausibly spawn\" and\n+\t * \"b\" is \"something less than Inf\", since the point is to\n+\t * prevent infinite recursion.\n+\t */\n+\tstatic const int recursion_limit = 1024;\n+\n+\tdying++;\n+\tif (dying > recursion_limit) {\n+\t\treturn 1;\n+\t} else if (dying == 2) {\n+\t\twarning(\"die() called many times. Recursion error or racy threaded death!\");\n+\t\treturn 0;\n+\t} else {\n+\t\treturn 0;\n+\t}\n }\n \n /* If we are in a dlopen()ed .so write to a global variable would segfault\n-- \n2.13.1.611.g7e3b11ae1\n\n"},{"id":"322864","messageId":"CAGZ79kZdksGoUyn2F+YFZKZMj8SPN3k=3EzVD6nTSmWkBq4qDw@mail.gmail.com","threadId":"46214","inReplyTo":"20170621204742.15722-1-avarab@gmail.com","subject":"Re: [PATCH v2] die(): stop hiding errors due to overzealous recursion guard","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-06-21T21:12:06Z","receivedAt":"2017-06-21T21:12:15Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Wed, Jun 21, 2017 at 1:47 PM, Ævar Arnfjörð Bjarmason\n<avarab@gmail.com> wrote:\n> Change the recursion limit for the default die routine from a *very*\n> low 1 to 1024. This ensures that infinite recursions are broken, but\n> doesn't lose the meaningful error messages under threaded execution\n> where threads concurrently start to die.\n>\n> The intent of the existing code, as explained in commit\n> cd163d4b4e (\"usage.c: detect recursion in die routines and bail out\n> immediately\", 2012-11-14), is to break infinite recursion in cases\n> where the die routine itself calls die(), and would thus infinitely\n> recurse.\n>\n> However, doing that very aggressively by immediately printing out\n> \"recursion detected in die handler\" if we've already called die() once\n> means that threaded invocations of git can end up only printing out\n> the \"recursion detected\" error, while hiding the meaningful error.\n>\n> An example of this is running a threaded grep which dies on execution\n> against pretty much any repo, git.git will do:\n>\n>     git grep -P --threads=8 '(*LIMIT_MATCH=1)-?-?-?---$'\n>\n> With the current version of git this will print some combination of\n> multiple PCRE failures that caused the abort and multiple \"recursion\n> detected\", some invocations will print out multiple \"recursion\n> detected\" errors with no PCRE error at all!\n>\n> Before this change, running the above grep command 1000 times against\n> git.git[1] and taking the top 20 results will on my system yield the\n> following distribution of actual errors (\"E\") and recursion\n> errors (\"R\"):\n>\n>     322 E R\n>     306 E\n>     116 E R R\n>      65 R R\n>      54 R E\n>      49 E E\n>      44 R\n>      15 E R R R\n>       9 R R R\n>       7 R E R\n>       5 R R E\n>       3 E R R R R\n>       2 E E R\n>       1 R R R R\n>       1 R R R E\n>       1 R E R R\n>\n> The exact results are obviously random and system-dependent, but this\n> shows the race condition in this code. Some small part of the time\n> we're about to print out the actual error (\"E\") but another thread's\n> recursion error beats us to it, and sometimes we print out nothing but\n> the recursion error.\n>\n> With this change we get, now with \"W\" to mean the new warning being\n> emitted indicating that we've called die() many times:\n>\n>     502 E\n>     160 E W E\n>     120 E E\n>      53 E W\n>      35 E W E E\n>      34 W E E\n>      29 W E E E\n>      16 E E W\n>      16 E E E\n>      11 W E E E E\n>       7 E E W E\n>       4 W E\n>       3 W W E E\n>       2 E W E E E\n>       1 W W E\n>       1 W E W E\n>       1 E W W E E E\n>       1 E W W E E\n>       1 E W W E\n>       1 E W E E W\n>\n> Which still sucks a bit, due to a still present race-condition in this\n> code we're sometimes going to print out several errors still, or\n> several warnings, or two duplicate errors without the warning.\n>\n> But we will never have a case where we completely hide the actual\n> error as we do now.\n>\n> Now, git-grep could make use of the pluggable error facility added in\n> commit c19a490e37 (\"usage: allow pluggable die-recursion checks\",\n> 2013-04-16). There's other threaded code that calls set_die_routine()\n> or set_die_is_recursing_routine().\n>\n> But this is about fixing the general die() behavior with threading\n> when we don't have such a custom routine yet. Right now the common\n> case is not an infinite recursion in the handler, but us losing error\n> messages by default because we're overly paranoid about our recursion\n> check.\n>\n> So let's just set the recursion limit to a number higher than the\n> number of threads we're ever likely to spawn. Now we won't lose\n> errors, and if we have a recursing die handler we'll still die within\n> microseconds.\n>\n> There are race conditions in this code itself, in particular the\n> \"dying\" variable is not thread mutexed, so we e.g. won't be dying at\n> exactly 1024, or for that matter even be able to accurately test\n> \"dying == 2\", see the cases where we print out more than one \"W\"\n> above.\n>\n> But that doesn't really matter, for the recursion guard we just need\n> to die \"soon\", not at exactly 1024 calls, and for printing the correct\n> error and only one warning most of the time in the face of threaded\n> death this is good enough and a net improvement on the current code.\n>\n> 1. for i in {1..1000}; do git grep -P --threads=8 '(*LIMIT_MATCH=1)-?-?-?---$' 2>&1|perl -pe 's/^fatal: r.*/R/; s/^fatal: p.*/E/; s/^warning.*/W/' | tr '\\n' ' '; echo; done | sort | uniq -c | sort -nr | head -n 20\n>\n> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n> ---\n\nReviewed-by-and-found-no-nits: Stefan Beller <sbeller@google.com>\n;)\n\n>\n> This replaces v1 and takes into account the feedback in this thread\n> (thanks everyone!).\n>\n> The commit message is also much improved and includes more rationale\n> originally in my reply to Stefan in 87podz8v6v.fsf@gmail.com\n\nThanks!\nStefan\n\n>\n>  usage.c | 18 +++++++++++++++++-\n>  1 file changed, 17 insertions(+), 1 deletion(-)\n>\n> diff --git a/usage.c b/usage.c\n> index 2f87ca69a8..1ea7df9a20 100644\n> --- a/usage.c\n> +++ b/usage.c\n> @@ -44,7 +44,23 @@ static void warn_builtin(const char *warn, va_list params)\n>  static int die_is_recursing_builtin(void)\n>  {\n>         static int dying;\n> -       return dying++;\n> +       /*\n> +        * Just an arbitrary number X where \"a < x < b\" where \"a\" is\n> +        * \"maximum number of pthreads we'll ever plausibly spawn\" and\n> +        * \"b\" is \"something less than Inf\", since the point is to\n> +        * prevent infinite recursion.\n> +        */\n> +       static const int recursion_limit = 1024;\n> +\n> +       dying++;\n> +       if (dying > recursion_limit) {\n> +               return 1;\n> +       } else if (dying == 2) {\n> +               warning(\"die() called many times. Recursion error or racy threaded death!\");\n> +               return 0;\n> +       } else {\n> +               return 0;\n> +       }\n>  }\n>\n>  /* If we are in a dlopen()ed .so write to a global variable would segfault\n> --\n> 2.13.1.611.g7e3b11ae1\n>\n"},{"id":"322869","messageId":"CANv4PNmprzn+SkWERbYYzyB_qX62ohrhoOkxMxztzxnTq2+saQ@mail.gmail.com","threadId":"46214","inReplyTo":"20170621204742.15722-1-avarab@gmail.com","subject":"Re: [PATCH v2] die(): stop hiding errors due to overzealous recursion guard","fromName":"Morten Welinder","fromEmail":"mwelinder@gmail.com","sentAt":"2017-06-21T21:21:44Z","receivedAt":"2017-06-21T21:21:51Z","isPatch":true,"sender":{"key":"mwelinder@gmail.com","avatar":null},"body":"If threading is the issue, how do you get meaningful results from\nreading and updating\n\"dying\" with no use of atomic types or locks?  Other than winning the\nimplied race, of\ncourse.\n\nM.\n\n\nOn Wed, Jun 21, 2017 at 4:47 PM, Ævar Arnfjörð Bjarmason\n<avarab@gmail.com> wrote:\n> Change the recursion limit for the default die routine from a *very*\n> low 1 to 1024. This ensures that infinite recursions are broken, but\n> doesn't lose the meaningful error messages under threaded execution\n> where threads concurrently start to die.\n>\n> The intent of the existing code, as explained in commit\n> cd163d4b4e (\"usage.c: detect recursion in die routines and bail out\n> immediately\", 2012-11-14), is to break infinite recursion in cases\n> where the die routine itself calls die(), and would thus infinitely\n> recurse.\n>\n> However, doing that very aggressively by immediately printing out\n> \"recursion detected in die handler\" if we've already called die() once\n> means that threaded invocations of git can end up only printing out\n> the \"recursion detected\" error, while hiding the meaningful error.\n>\n> An example of this is running a threaded grep which dies on execution\n> against pretty much any repo, git.git will do:\n>\n>     git grep -P --threads=8 '(*LIMIT_MATCH=1)-?-?-?---$'\n>\n> With the current version of git this will print some combination of\n> multiple PCRE failures that caused the abort and multiple \"recursion\n> detected\", some invocations will print out multiple \"recursion\n> detected\" errors with no PCRE error at all!\n>\n> Before this change, running the above grep command 1000 times against\n> git.git[1] and taking the top 20 results will on my system yield the\n> following distribution of actual errors (\"E\") and recursion\n> errors (\"R\"):\n>\n>     322 E R\n>     306 E\n>     116 E R R\n>      65 R R\n>      54 R E\n>      49 E E\n>      44 R\n>      15 E R R R\n>       9 R R R\n>       7 R E R\n>       5 R R E\n>       3 E R R R R\n>       2 E E R\n>       1 R R R R\n>       1 R R R E\n>       1 R E R R\n>\n> The exact results are obviously random and system-dependent, but this\n> shows the race condition in this code. Some small part of the time\n> we're about to print out the actual error (\"E\") but another thread's\n> recursion error beats us to it, and sometimes we print out nothing but\n> the recursion error.\n>\n> With this change we get, now with \"W\" to mean the new warning being\n> emitted indicating that we've called die() many times:\n>\n>     502 E\n>     160 E W E\n>     120 E E\n>      53 E W\n>      35 E W E E\n>      34 W E E\n>      29 W E E E\n>      16 E E W\n>      16 E E E\n>      11 W E E E E\n>       7 E E W E\n>       4 W E\n>       3 W W E E\n>       2 E W E E E\n>       1 W W E\n>       1 W E W E\n>       1 E W W E E E\n>       1 E W W E E\n>       1 E W W E\n>       1 E W E E W\n>\n> Which still sucks a bit, due to a still present race-condition in this\n> code we're sometimes going to print out several errors still, or\n> several warnings, or two duplicate errors without the warning.\n>\n> But we will never have a case where we completely hide the actual\n> error as we do now.\n>\n> Now, git-grep could make use of the pluggable error facility added in\n> commit c19a490e37 (\"usage: allow pluggable die-recursion checks\",\n> 2013-04-16). There's other threaded code that calls set_die_routine()\n> or set_die_is_recursing_routine().\n>\n> But this is about fixing the general die() behavior with threading\n> when we don't have such a custom routine yet. Right now the common\n> case is not an infinite recursion in the handler, but us losing error\n> messages by default because we're overly paranoid about our recursion\n> check.\n>\n> So let's just set the recursion limit to a number higher than the\n> number of threads we're ever likely to spawn. Now we won't lose\n> errors, and if we have a recursing die handler we'll still die within\n> microseconds.\n>\n> There are race conditions in this code itself, in particular the\n> \"dying\" variable is not thread mutexed, so we e.g. won't be dying at\n> exactly 1024, or for that matter even be able to accurately test\n> \"dying == 2\", see the cases where we print out more than one \"W\"\n> above.\n>\n> But that doesn't really matter, for the recursion guard we just need\n> to die \"soon\", not at exactly 1024 calls, and for printing the correct\n> error and only one warning most of the time in the face of threaded\n> death this is good enough and a net improvement on the current code.\n>\n> 1. for i in {1..1000}; do git grep -P --threads=8 '(*LIMIT_MATCH=1)-?-?-?---$' 2>&1|perl -pe 's/^fatal: r.*/R/; s/^fatal: p.*/E/; s/^warning.*/W/' | tr '\\n' ' '; echo; done | sort | uniq -c | sort -nr | head -n 20\n>\n> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n> ---\n>\n> This replaces v1 and takes into account the feedback in this thread\n> (thanks everyone!).\n>\n> The commit message is also much improved and includes more rationale\n> originally in my reply to Stefan in 87podz8v6v.fsf@gmail.com\n>\n>  usage.c | 18 +++++++++++++++++-\n>  1 file changed, 17 insertions(+), 1 deletion(-)\n>\n> diff --git a/usage.c b/usage.c\n> index 2f87ca69a8..1ea7df9a20 100644\n> --- a/usage.c\n> +++ b/usage.c\n> @@ -44,7 +44,23 @@ static void warn_builtin(const char *warn, va_list params)\n>  static int die_is_recursing_builtin(void)\n>  {\n>         static int dying;\n> -       return dying++;\n> +       /*\n> +        * Just an arbitrary number X where \"a < x < b\" where \"a\" is\n> +        * \"maximum number of pthreads we'll ever plausibly spawn\" and\n> +        * \"b\" is \"something less than Inf\", since the point is to\n> +        * prevent infinite recursion.\n> +        */\n> +       static const int recursion_limit = 1024;\n> +\n> +       dying++;\n> +       if (dying > recursion_limit) {\n> +               return 1;\n> +       } else if (dying == 2) {\n> +               warning(\"die() called many times. Recursion error or racy threaded death!\");\n> +               return 0;\n> +       } else {\n> +               return 0;\n> +       }\n>  }\n>\n>  /* If we are in a dlopen()ed .so write to a global variable would segfault\n> --\n> 2.13.1.611.g7e3b11ae1\n>\n"},{"id":"322873","messageId":"xmqq1sqdni1r.fsf@gitster.mtv.corp.google.com","threadId":"46214","inReplyTo":"20170621204742.15722-1-avarab@gmail.com","subject":"Re: [PATCH v2] die(): stop hiding errors due to overzealous recursion guard","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-06-21T21:32:16Z","receivedAt":"2017-06-21T21:32:23Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason  <avarab@gmail.com> writes:\n\n> So let's just set the recursion limit to a number higher than the\n> number of threads we're ever likely to spawn. Now we won't lose\n> errors, and if we have a recursing die handler we'll still die within\n> microseconds.\n>\n> There are race conditions in this code itself, in particular the\n> \"dying\" variable is not thread mutexed, so we e.g. won't be dying at\n> exactly 1024, or for that matter even be able to accurately test\n> \"dying == 2\", see the cases where we print out more than one \"W\"\n> above.\n\nOne case I'd be worried about would be that the race is so bad that\ndie-is-recursing-builtin never returns 0 even once.  Everybody will\njust say \"recursing\" and die, without giving any useful information.\n\nWill queue, as it is nevertheless an improvement over the current\ncode.\n\nThanks.\n\n>  usage.c | 18 +++++++++++++++++-\n>  1 file changed, 17 insertions(+), 1 deletion(-)\n>\n> diff --git a/usage.c b/usage.c\n> index 2f87ca69a8..1ea7df9a20 100644\n> --- a/usage.c\n> +++ b/usage.c\n> @@ -44,7 +44,23 @@ static void warn_builtin(const char *warn, va_list params)\n>  static int die_is_recursing_builtin(void)\n>  {\n>  \tstatic int dying;\n> -\treturn dying++;\n> +\t/*\n> +\t * Just an arbitrary number X where \"a < x < b\" where \"a\" is\n> +\t * \"maximum number of pthreads we'll ever plausibly spawn\" and\n> +\t * \"b\" is \"something less than Inf\", since the point is to\n> +\t * prevent infinite recursion.\n> +\t */\n> +\tstatic const int recursion_limit = 1024;\n> +\n> +\tdying++;\n> +\tif (dying > recursion_limit) {\n> +\t\treturn 1;\n> +\t} else if (dying == 2) {\n> +\t\twarning(\"die() called many times. Recursion error or racy threaded death!\");\n> +\t\treturn 0;\n> +\t} else {\n> +\t\treturn 0;\n> +\t}\n>  }\n>  \n>  /* If we are in a dlopen()ed .so write to a global variable would segfault\n"},{"id":"322875","messageId":"87bmph81fa.fsf@gmail.com","threadId":"46214","inReplyTo":"CANv4PNmprzn+SkWERbYYzyB_qX62ohrhoOkxMxztzxnTq2+saQ@mail.gmail.com","subject":"Re: [PATCH v2] die(): stop hiding errors due to overzealous recursion guard","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2017-06-21T21:40:25Z","receivedAt":"2017-06-21T21:40:33Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Jun 21 2017, Morten Welinder jotted:\n\n> If threading is the issue, how do you get meaningful results from\n> reading and updating\n> \"dying\" with no use of atomic types or locks?  Other than winning the\n> implied race, of\n> course.\n\nThreading isn't the issue. The issue is that we have an overzelous\nrecursion guard that will demonstrably cause us to lose errors in the\nface of threading.\n\nBy amending the guard so that we won't run into it in practice so soon\nthat we'll hide errors (see the empirical results in the commit message)\nwe solve *that* issue in practice.\n\nThe current code & the code I'm adding here suffers from race conditions\n& non-atomic updates, but for the reasons explained at the bottom of the\nthe commit message that's OK.\n\nWe're not relying on being able to do x++ and have x be 1, 2, 3 etc. in\nthe face of threading, we're just currently relying on it being larger\nthan 1, or with my patch eventually larger than 1024.\n\nIt is possible with my patch that we'll never take the \"dying == 2\" ->\n\"warning(..)\" branch (and empirical results show that happens), but it's\nenough for the purposes of the default die handler (which we really\nshould be overriding if we're doing threading, but sometimes we're lazy)\nthat it works most of the time, and that we at least don't hide real\nerrors, which is the issue with it right now.\n\n> On Wed, Jun 21, 2017 at 4:47 PM, Ævar Arnfjörð Bjarmason\n> <avarab@gmail.com> wrote:\n>> Change the recursion limit for the default die routine from a *very*\n>> low 1 to 1024. This ensures that infinite recursions are broken, but\n>> doesn't lose the meaningful error messages under threaded execution\n>> where threads concurrently start to die.\n>>\n>> The intent of the existing code, as explained in commit\n>> cd163d4b4e (\"usage.c: detect recursion in die routines and bail out\n>> immediately\", 2012-11-14), is to break infinite recursion in cases\n>> where the die routine itself calls die(), and would thus infinitely\n>> recurse.\n>>\n>> However, doing that very aggressively by immediately printing out\n>> \"recursion detected in die handler\" if we've already called die() once\n>> means that threaded invocations of git can end up only printing out\n>> the \"recursion detected\" error, while hiding the meaningful error.\n>>\n>> An example of this is running a threaded grep which dies on execution\n>> against pretty much any repo, git.git will do:\n>>\n>>     git grep -P --threads=8 '(*LIMIT_MATCH=1)-?-?-?---$'\n>>\n>> With the current version of git this will print some combination of\n>> multiple PCRE failures that caused the abort and multiple \"recursion\n>> detected\", some invocations will print out multiple \"recursion\n>> detected\" errors with no PCRE error at all!\n>>\n>> Before this change, running the above grep command 1000 times against\n>> git.git[1] and taking the top 20 results will on my system yield the\n>> following distribution of actual errors (\"E\") and recursion\n>> errors (\"R\"):\n>>\n>>     322 E R\n>>     306 E\n>>     116 E R R\n>>      65 R R\n>>      54 R E\n>>      49 E E\n>>      44 R\n>>      15 E R R R\n>>       9 R R R\n>>       7 R E R\n>>       5 R R E\n>>       3 E R R R R\n>>       2 E E R\n>>       1 R R R R\n>>       1 R R R E\n>>       1 R E R R\n>>\n>> The exact results are obviously random and system-dependent, but this\n>> shows the race condition in this code. Some small part of the time\n>> we're about to print out the actual error (\"E\") but another thread's\n>> recursion error beats us to it, and sometimes we print out nothing but\n>> the recursion error.\n>>\n>> With this change we get, now with \"W\" to mean the new warning being\n>> emitted indicating that we've called die() many times:\n>>\n>>     502 E\n>>     160 E W E\n>>     120 E E\n>>      53 E W\n>>      35 E W E E\n>>      34 W E E\n>>      29 W E E E\n>>      16 E E W\n>>      16 E E E\n>>      11 W E E E E\n>>       7 E E W E\n>>       4 W E\n>>       3 W W E E\n>>       2 E W E E E\n>>       1 W W E\n>>       1 W E W E\n>>       1 E W W E E E\n>>       1 E W W E E\n>>       1 E W W E\n>>       1 E W E E W\n>>\n>> Which still sucks a bit, due to a still present race-condition in this\n>> code we're sometimes going to print out several errors still, or\n>> several warnings, or two duplicate errors without the warning.\n>>\n>> But we will never have a case where we completely hide the actual\n>> error as we do now.\n>>\n>> Now, git-grep could make use of the pluggable error facility added in\n>> commit c19a490e37 (\"usage: allow pluggable die-recursion checks\",\n>> 2013-04-16). There's other threaded code that calls set_die_routine()\n>> or set_die_is_recursing_routine().\n>>\n>> But this is about fixing the general die() behavior with threading\n>> when we don't have such a custom routine yet. Right now the common\n>> case is not an infinite recursion in the handler, but us losing error\n>> messages by default because we're overly paranoid about our recursion\n>> check.\n>>\n>> So let's just set the recursion limit to a number higher than the\n>> number of threads we're ever likely to spawn. Now we won't lose\n>> errors, and if we have a recursing die handler we'll still die within\n>> microseconds.\n>>\n>> There are race conditions in this code itself, in particular the\n>> \"dying\" variable is not thread mutexed, so we e.g. won't be dying at\n>> exactly 1024, or for that matter even be able to accurately test\n>> \"dying == 2\", see the cases where we print out more than one \"W\"\n>> above.\n>>\n>> But that doesn't really matter, for the recursion guard we just need\n>> to die \"soon\", not at exactly 1024 calls, and for printing the correct\n>> error and only one warning most of the time in the face of threaded\n>> death this is good enough and a net improvement on the current code.\n>>\n>> 1. for i in {1..1000}; do git grep -P --threads=8 '(*LIMIT_MATCH=1)-?-?-?---$' 2>&1|perl -pe 's/^fatal: r.*/R/; s/^fatal: p.*/E/; s/^warning.*/W/' | tr '\\n' ' '; echo; done | sort | uniq -c | sort -nr | head -n 20\n>>\n>> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n>> ---\n>>\n>> This replaces v1 and takes into account the feedback in this thread\n>> (thanks everyone!).\n>>\n>> The commit message is also much improved and includes more rationale\n>> originally in my reply to Stefan in 87podz8v6v.fsf@gmail.com\n>>\n>>  usage.c | 18 +++++++++++++++++-\n>>  1 file changed, 17 insertions(+), 1 deletion(-)\n>>\n>> diff --git a/usage.c b/usage.c\n>> index 2f87ca69a8..1ea7df9a20 100644\n>> --- a/usage.c\n>> +++ b/usage.c\n>> @@ -44,7 +44,23 @@ static void warn_builtin(const char *warn, va_list params)\n>>  static int die_is_recursing_builtin(void)\n>>  {\n>>         static int dying;\n>> -       return dying++;\n>> +       /*\n>> +        * Just an arbitrary number X where \"a < x < b\" where \"a\" is\n>> +        * \"maximum number of pthreads we'll ever plausibly spawn\" and\n>> +        * \"b\" is \"something less than Inf\", since the point is to\n>> +        * prevent infinite recursion.\n>> +        */\n>> +       static const int recursion_limit = 1024;\n>> +\n>> +       dying++;\n>> +       if (dying > recursion_limit) {\n>> +               return 1;\n>> +       } else if (dying == 2) {\n>> +               warning(\"die() called many times. Recursion error or racy threaded death!\");\n>> +               return 0;\n>> +       } else {\n>> +               return 0;\n>> +       }\n>>  }\n>>\n>>  /* If we are in a dlopen()ed .so write to a global variable would segfault\n>> --\n>> 2.13.1.611.g7e3b11ae1\n>>\n"},{"id":"323200","messageId":"20170624123639.scagiuzqalfr72fz@sigill.intra.peff.net","threadId":"46214","inReplyTo":"xmqq1sqdni1r.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v2] die(): stop hiding errors due to overzealous recursion guard","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-06-24T12:36:39Z","receivedAt":"2017-06-24T12:36:46Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jun 21, 2017 at 02:32:16PM -0700, Junio C Hamano wrote:\n\n> Ævar Arnfjörð Bjarmason  <avarab@gmail.com> writes:\n> \n> > So let's just set the recursion limit to a number higher than the\n> > number of threads we're ever likely to spawn. Now we won't lose\n> > errors, and if we have a recursing die handler we'll still die within\n> > microseconds.\n> >\n> > There are race conditions in this code itself, in particular the\n> > \"dying\" variable is not thread mutexed, so we e.g. won't be dying at\n> > exactly 1024, or for that matter even be able to accurately test\n> > \"dying == 2\", see the cases where we print out more than one \"W\"\n> > above.\n> \n> One case I'd be worried about would be that the race is so bad that\n> die-is-recursing-builtin never returns 0 even once.  Everybody will\n> just say \"recursing\" and die, without giving any useful information.\n\nI was trying to think how that would happen. If nobody's actually\nrecursing indefinitely, then the value in theory peaks at the number of\nthreads (modulo the fact that we're modifying a variable from multiple\nthreads without any locking; I'm not sure how reasonable it is to assume\nin practice that sheared writes may cause us to lose an increment but\nnot to put nonsense in to the variable). If they are, then one thread\nmay increment it to 1024 before another thread gets a chance to say\nanything. But in that case, the recursion-die is our expected outcome.\n\nAnyway, it might be reasonable to protect the counter with a mutex.\nLike:\n\ndiff --git a/usage.c b/usage.c\nindex fc2b31c54b..34fef0f9fa 100644\n--- a/usage.c\n+++ b/usage.c\n@@ -44,9 +44,19 @@ static void warn_builtin(const char *warn, va_list params)\n \tvreportf(\"warning: \", warn, params);\n }\n \n+#ifndef NO_PTHREADS\n+static pthread_mutex_t recursion_mutex = PTHREAD_MUTEX_INITIALIZER;\n+#define recursion_lock() pthread_mutex_lock(&recursion_mutex)\n+#define recursion_unlock() pthread_mutex_unlock(&recursion_mutex)\n+#else\n+#define recursion_lock()\n+#define recursion_unlock()\n+#endif\n+static int recursion_counter;\n+\n static int die_is_recursing_builtin(void)\n {\n-\tstatic int dying;\n+\tint dying;\n \t/*\n \t * Just an arbitrary number X where \"a < x < b\" where \"a\" is\n \t * \"maximum number of pthreads we'll ever plausibly spawn\" and\n@@ -55,7 +65,10 @@ static int die_is_recursing_builtin(void)\n \t */\n \tstatic const int recursion_limit = 1024;\n \n-\tdying++;\n+\trecursion_lock();\n+\tdying = ++recursion_counter;\n+\trecursion_unlock();\n+\n \tif (dying > recursion_limit) {\n \t\treturn 1;\n \t} else if (dying == 2) {\n\nI can't remember if there are problems on Windows with using constant\nmutex initializers, though. If so, I guess common-main would have to\ninitialize it.\n\nI left the rest of the logic as-is, but if we switched to post-increment:\n\n  dying = recursion_counter++;\n\nthen I think the numbers around \"dying\" would make more sense (e.g.,\n\"dying == 2\" would make more sense to me as \"dying == 1\" to check that\nwe were already dying).\n\nTo be honest, I'm not sure if it's worth giving it much more time,\nthough. I'd be fine with Ævar's patch as-is.\n\n-Peff\n"},{"id":"323217","messageId":"xmqqbmpd9qyg.fsf@gitster.mtv.corp.google.com","threadId":"46214","inReplyTo":"20170624123639.scagiuzqalfr72fz@sigill.intra.peff.net","subject":"Re: [PATCH v2] die(): stop hiding errors due to overzealous recursion guard","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-06-24T18:32:39Z","receivedAt":"2017-06-24T18:32:45Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King <peff@peff.net> writes:\n\n>> One case I'd be worried about would be that the race is so bad that\n>> die-is-recursing-builtin never returns 0 even once.  Everybody will\n>> just say \"recursing\" and die, without giving any useful information.\n>\n> I was trying to think how that would happen. If nobody's actually\n> recursing indefinitely, then the value in theory peaks at the number of\n> threads (modulo the fact that we're modifying a variable from multiple\n> threads without any locking; I'm not sure how reasonable it is to assume\n> in practice that sheared writes may cause us to lose an increment but\n> not to put nonsense in to the variable). If they are, then one thread\n> may increment it to 1024 before another thread gets a chance to say\n> anything. But in that case, the recursion-die is our expected outcome.\n>\n> Anyway, it might be reasonable to protect the counter with a mutex.\n> Like:\n> ...\n> To be honest, I'm not sure if it's worth giving it much more time,\n> though. I'd be fine with Ævar's patch as-is.\n\nThe scenario I had in mind was three or more threads simultaneously\ndying, each incrementing dying counter by one and before any of them\nhave a chance to say \"called many times, error or racy threaded\ndeath!\", because they all observe three (or more).  \n\nBut I was incorrectly reading the code---in that case, as long as\ndying is small enough, we'll return 0 and let at least one of the\ncaller give a chance to give a message that came in \"err\" from their\ninvocations of die()'s.\n\nSo I do not think it is worth worrying about too deeply.\n\nThanks.\n"}]}