{"thread":{"id":"34235","subject":"Re: HTTP tests fail on OS X","startedAt":"2013-06-21T04:27:02Z","lastAt":"2013-06-21T20:59:56Z","messageCount":12,"participants":["Jeff King","Brian Gernhardt","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"221550","messageId":"C44E969B-A5AE-44EF-BFCA-CAEF69516BEB@gernhardtsoftware.com","threadId":"34235","inReplyTo":null,"subject":"HTTP tests fail on OS X","fromName":"Brian Gernhardt","fromEmail":"brian@gernhardtsoftware.com","sentAt":"2013-06-21T04:27:02Z","receivedAt":"2013-06-21T04:27:02Z","isPatch":false,"sender":{"key":"brian@gernhardtsoftware.com","avatar":"https://avatars.githubusercontent.com/u/133455?v=4"},"body":"I've bisected it to this commit:\n\n> commit 0442743810c6f6c14386a5a9d6bf8e4d69adbc51\n> Author: Jeff King \n> Date: Sun Jun 9 04:07:59 2013 -0400\n> \n> t/lib-httpd/apache.conf: do not use LockFile in apache >= 2.4\n\nOS X 10.8.4, apache 2.2.22\n\nI'll also note that the tests fail the first time they attempt to access the server and the cleanup function notes that httpd is not running under the expected PID. Looking at the httpd setup code, I would have expected to see errors (with -v) and have the setup fail but neither happens.\n\nIt's way too late for me, so I can't look at it again for at least several hours. I figured I'd ping the list in case the problem/solution is obvious to someone else.\n\n~~ Brian Gernhardt\n"},{"id":"221549","messageId":"20130621043052.GA5318@sigill.intra.peff.net","threadId":"34235","inReplyTo":"C44E969B-A5AE-44EF-BFCA-CAEF69516BEB@gernhardtsoftware.com","subject":"Re: HTTP tests fail on OS X","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2013-06-21T04:30:52Z","receivedAt":"2013-06-21T04:30:52Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Jun 21, 2013 at 12:27:02AM -0400, Brian Gernhardt wrote:\n\n> I've bisected it to this commit:\n> \n> > commit 0442743810c6f6c14386a5a9d6bf8e4d69adbc51\n> > Author: Jeff King \n> > Date: Sun Jun 9 04:07:59 2013 -0400\n> > \n> > t/lib-httpd/apache.conf: do not use LockFile in apache >= 2.4\n> \n> OS X 10.8.4, apache 2.2.22\n\nHrm. That's quite odd. The patch in its entirety looks like this:\n\ndiff --git a/t/lib-httpd/apache.conf b/t/lib-httpd/apache.conf\nindex 542241b..891edd7 100644\n--- a/t/lib-httpd/apache.conf\n+++ b/t/lib-httpd/apache.conf\n@@ -1,5 +1,7 @@ LockFile accept.lock\n ServerName dummy\n+\n LockFile accept.lock\n+\n PidFile httpd.pid\n DocumentRoot www\n\nSo it should not be having any effect at all on apache 2.2, unless it\nsomehow does not like the tags.\n\nPuzzled...\n\n-Peff\n"},{"id":"221551","messageId":"20130621044236.GA5798@sigill.intra.peff.net","threadId":"34235","inReplyTo":"20130621043052.GA5318@sigill.intra.peff.net","subject":"Re: HTTP tests fail on OS X","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2013-06-21T04:42:37Z","receivedAt":"2013-06-21T04:42:37Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Jun 21, 2013 at 12:30:52AM -0400, Jeff King wrote:\n\n> > OS X 10.8.4, apache 2.2.22\n> \n> Hrm. That's quite odd. The patch in its entirety looks like this:\n> [...]\n> So it should not be having any effect at all on apache 2.2, unless it\n> somehow does not like the tags.\n\nJust to double-check, I ran all of the http tests in \"next\" (with my\nseries) successfully on apache 2.2.16. So the good news is I haven't\nbroken apache 2.2.x entirely. :)\n\nI'm not sure if there is something different between 2.2.16 and 2.2.22,\nor something with the particular build. Here's my -V output, in case it\nhelps:\n\n $ apache2 -V\n Server version: Apache/2.2.16 (Debian)\n Server built: Mar 3 2013 12:12:28\n Server's Module Magic Number: 20051115:24\n Server loaded: APR 1.4.2, APR-Util 1.3.9\n Compiled using: APR 1.4.2, APR-Util 1.3.9\n Architecture: 64-bit\n Server MPM: Worker\n threaded: yes (fixed thread count)\n forked: yes (variable process count)\n Server compiled with....\n -D APACHE_MPM_DIR=\"server/mpm/worker\"\n -D APR_HAS_SENDFILE\n -D APR_HAS_MMAP\n -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)\n -D APR_USE_SYSVSEM_SERIALIZE\n -D APR_USE_PTHREAD_SERIALIZE\n -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT\n -D APR_HAS_OTHER_CHILD\n -D AP_HAVE_RELIABLE_PIPED_LOGS\n -D DYNAMIC_MODULE_LIMIT=128\n -D HTTPD_ROOT=\"/etc/apache2\"\n -D SUEXEC_BIN=\"/usr/lib/apache2/suexec\"\n -D DEFAULT_PIDLOG=\"/var/run/apache2.pid\"\n -D DEFAULT_SCOREBOARD=\"logs/apache_runtime_status\"\n -D DEFAULT_ERRORLOG=\"logs/error_log\"\n -D AP_TYPES_CONFIG_FILE=\"mime.types\"\n -D SERVER_CONFIG_FILE=\"apache2.conf\"\n\n-Peff\n"},{"id":"221552","messageId":"26902D6D-D105-4943-BC67-461CEF82D888@gernhardtsoftware.com","threadId":"34235","inReplyTo":"20130621044236.GA5798@sigill.intra.peff.net","subject":"Re: HTTP tests fail on OS X","fromName":"Brian Gernhardt","fromEmail":"brian@gernhardtsoftware.com","sentAt":"2013-06-21T04:45:25Z","receivedAt":"2013-06-21T04:45:25Z","isPatch":false,"sender":{"key":"brian@gernhardtsoftware.com","avatar":"https://avatars.githubusercontent.com/u/133455?v=4"},"body":"\nOn Jun 21, 2013, at 12:42 AM, Jeff King wrote:\n\n> I'm not sure if there is something different between 2.2.16 and 2.2.22,\n> or something with the particular build. Here's my -V output, in case it\n> helps:\n> \n> $ apache2 -V\n> Server version: Apache/2.2.16 (Debian)\n> Server built: Mar 3 2013 12:12:28\n> Server's Module Magic Number: 20051115:24\n> Server loaded: APR 1.4.2, APR-Util 1.3.9\n> Compiled using: APR 1.4.2, APR-Util 1.3.9\n> Architecture: 64-bit\n> Server MPM: Worker\n> threaded: yes (fixed thread count)\n> forked: yes (variable process count)\n> Server compiled with....\n> -D APACHE_MPM_DIR=\"server/mpm/worker\"\n> -D APR_HAS_SENDFILE\n> -D APR_HAS_MMAP\n> -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)\n> -D APR_USE_SYSVSEM_SERIALIZE\n> -D APR_USE_PTHREAD_SERIALIZE\n> -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT\n> -D APR_HAS_OTHER_CHILD\n> -D AP_HAVE_RELIABLE_PIPED_LOGS\n> -D DYNAMIC_MODULE_LIMIT=128\n> -D HTTPD_ROOT=\"/etc/apache2\"\n> -D SUEXEC_BIN=\"/usr/lib/apache2/suexec\"\n> -D DEFAULT_PIDLOG=\"/var/run/apache2.pid\"\n> -D DEFAULT_SCOREBOARD=\"logs/apache_runtime_status\"\n> -D DEFAULT_ERRORLOG=\"logs/error_log\"\n> -D AP_TYPES_CONFIG_FILE=\"mime.types\"\n> -D SERVER_CONFIG_FILE=\"apache2.conf\"\n\nDoesn't look terribly different from mine.\n\n$ apachectl -V\nServer version: Apache/2.2.22 (Unix)\nServer built: Dec 9 2012 18:57:18\nServer's Module Magic Number: 20051115:30\nServer loaded: APR 1.4.5, APR-Util 1.3.12\nCompiled using: APR 1.4.5, APR-Util 1.3.12\nArchitecture: 64-bit\nServer MPM: Prefork\n threaded: no\n forked: yes (variable process count)\nServer compiled with....\n -D APACHE_MPM_DIR=\"server/mpm/prefork\"\n -D APR_HAS_SENDFILE\n -D APR_HAS_MMAP\n -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)\n -D APR_USE_FLOCK_SERIALIZE\n -D APR_USE_PTHREAD_SERIALIZE\n -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT\n -D APR_HAS_OTHER_CHILD\n -D AP_HAVE_RELIABLE_PIPED_LOGS\n -D DYNAMIC_MODULE_LIMIT=128\n -D HTTPD_ROOT=\"/usr\"\n -D SUEXEC_BIN=\"/usr/bin/suexec\"\n -D DEFAULT_PIDLOG=\"/private/var/run/httpd.pid\"\n -D DEFAULT_SCOREBOARD=\"logs/apache_runtime_status\"\n -D DEFAULT_LOCKFILE=\"/private/var/run/accept.lock\"\n -D DEFAULT_ERRORLOG=\"logs/error_log\"\n -D AP_TYPES_CONFIG_FILE=\"/private/etc/apache2/mime.types\"\n -D SERVER_CONFIG_FILE=\"/private/etc/apache2/httpd.conf\"\n"},{"id":"221553","messageId":"20130621044953.GA5962@sigill.intra.peff.net","threadId":"34235","inReplyTo":"26902D6D-D105-4943-BC67-461CEF82D888@gernhardtsoftware.com","subject":"Re: HTTP tests fail on OS X","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2013-06-21T04:49:54Z","receivedAt":"2013-06-21T04:49:54Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Jun 21, 2013 at 12:45:25AM -0400, Brian Gernhardt wrote:\n\n> > I'm not sure if there is something different between 2.2.16 and 2.2.22,\n> > or something with the particular build. Here's my -V output, in case it\n> > helps:\n> [...]\n> Doesn't look terribly different from mine.\n\nIt turns out that 2.2.22 is in Debian testing, so it was easy for me to\ndowngrade from 2.4 and try it. Seems to also work fine for me. So I\ndon't think it's an apache version issue.\n\nI also installed the prefork mpm in case that was part of the problem.\nBut it all seems fine here, and my build parameters look quite similar\nto yours.\n\nI'm not sure what else to look at...I guess try ratcheting up the\ndebugging/log level on your failing copy and see if it prints anything\nuseful.\n\n-Peff\n"},{"id":"221612","messageId":"D0852D0F-3282-41DF-8A5A-4BEF1B39ED66@gernhardtsoftware.com","threadId":"34235","inReplyTo":"20130621044953.GA5962@sigill.intra.peff.net","subject":"Re: HTTP tests fail on OS X","fromName":"Brian Gernhardt","fromEmail":"brian@gernhardtsoftware.com","sentAt":"2013-06-21T17:03:40Z","receivedAt":"2013-06-21T17:03:40Z","isPatch":false,"sender":{"key":"brian@gernhardtsoftware.com","avatar":"https://avatars.githubusercontent.com/u/133455?v=4"},"body":"\nOn Jun 21, 2013, at 12:49 AM, Jeff King wrote:\n\n> I'm not sure what else to look at...I guess try ratcheting up the\n> debugging/log level on your failing copy and see if it prints anything\n> useful.\n\nI found this error in the error.log:\n\n[Fri Jun 21 12:59:59 2013] [emerg] (2)No such file or directory: Couldn't create accept lock (/private/var/run/accept.lock.64288) (5)\n\nAnnoying that httpd returns before it tries to create the lock.\n\nWithout the IfVersion directive, it creates the lock in the ./httpd directory instead. Not sure why apache is doing that, it's very irritating.\n\n~~ Brian\n"},{"id":"221619","messageId":"20130621180334.GA4499@sigill.intra.peff.net","threadId":"34235","inReplyTo":"D0852D0F-3282-41DF-8A5A-4BEF1B39ED66@gernhardtsoftware.com","subject":"Re: HTTP tests fail on OS X","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2013-06-21T18:03:34Z","receivedAt":"2013-06-21T18:03:34Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Jun 21, 2013 at 01:03:40PM -0400, Brian Gernhardt wrote:\n\n> On Jun 21, 2013, at 12:49 AM, Jeff King wrote:\n> \n> > I'm not sure what else to look at...I guess try ratcheting up the\n> > debugging/log level on your failing copy and see if it prints anything\n> > useful.\n> \n> I found this error in the error.log:\n> \n> [Fri Jun 21 12:59:59 2013] [emerg] (2)No such file or directory: Couldn't create accept lock (/private/var/run/accept.lock.64288) (5)\n\nHmm. I am far from an apache expert, but I believe that is what would\nhappen if the LockFile directive was not there at all. IOW, it seems\nlike your apache is treating \"\" as false. Which seems\nweird to me.\n\nIt's possible I'm using IfVersion wrong, though it does seem to work for\nme elsewhere.\n\nIfVersion comes from mod_version. I assume that if it were not loaded,\napache would complain about the directive entirely. But it's true that\nwe don't load it until later. Maybe try moving the IfVersion/Lockfile\nstanza down below the mod_version LoadModule line?\n\n-Peff\n"},{"id":"221620","messageId":"1B9251F9-C1AF-41F7-8BF8-D36DDB27EF09@gernhardtsoftware.com","threadId":"34235","inReplyTo":"20130621180334.GA4499@sigill.intra.peff.net","subject":"Re: HTTP tests fail on OS X","fromName":"Brian Gernhardt","fromEmail":"brian@gernhardtsoftware.com","sentAt":"2013-06-21T18:08:49Z","receivedAt":"2013-06-21T18:08:49Z","isPatch":false,"sender":{"key":"brian@gernhardtsoftware.com","avatar":"https://avatars.githubusercontent.com/u/133455?v=4"},"body":"\nOn Jun 21, 2013, at 2:03 PM, Jeff King wrote:\n\n> IfVersion comes from mod_version. I assume that if it were not loaded,\n> apache would complain about the directive entirely. But it's true that\n> we don't load it until later. Maybe try moving the IfVersion/Lockfile\n> stanza down below the mod_version LoadModule line?\n\nApache is apparently overly accepting. Moving the IfVersion below all the IfModules fixes it.\n\n~~ Brian\n"},{"id":"221621","messageId":"20130621181250.GA5290@sigill.intra.peff.net","threadId":"34235","inReplyTo":"1B9251F9-C1AF-41F7-8BF8-D36DDB27EF09@gernhardtsoftware.com","subject":"[PATCH] lib-httpd/apache.conf: check version only after mod_version loads","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2013-06-21T18:12:51Z","receivedAt":"2013-06-21T18:12:51Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Jun 21, 2013 at 02:08:49PM -0400, Brian Gernhardt wrote:\n\n> On Jun 21, 2013, at 2:03 PM, Jeff King wrote:\n> \n> > IfVersion comes from mod_version. I assume that if it were not\n> > loaded, apache would complain about the directive entirely. But it's\n> > true that we don't load it until later. Maybe try moving the\n> > IfVersion/Lockfile stanza down below the mod_version LoadModule\n> > line?\n> \n> Apache is apparently overly accepting. Moving the IfVersion below all\n> the IfModules fixes it.\n\nCool. I think the patch should look like the one below, then.\n\nJust to double-check that I have explained the issue correctly, can you\nshare the output of \"apache2 -l\"? Mine has:\n\n $ apache2 -l\n Compiled in modules:\n core.c\n mod_log_config.c\n mod_logio.c\n mod_version.c\n prefork.c\n http_core.c\n mod_so.c\n\nwhich explains why it works here. I'm assuming you will not have\nmod_version.c compiled in.\n\n-- >8 --\nSubject: lib-httpd/apache.conf: check version only after mod_version loads\n\nCommit 0442743 introduced an directive near the\ntop of the apache config file. However, at that point we\nhave not yet checked for and loaded the mod_version module.\nThis means that the directive will behave oddly if\nmod_version is dynamically loaded, failing to match when it\nshould.\n\nWe can fix this by moving the whole block below the\nLoadModule directive for mod_version.\n\nReported-by: Brian Gernhardt \nSigned-off-by: Jeff King \n---\n t/lib-httpd/apache.conf | 7 ++++---\n 1 file changed, 4 insertions(+), 3 deletions(-)\n\ndiff --git a/t/lib-httpd/apache.conf b/t/lib-httpd/apache.conf\nindex 56ae548..dd17e3a 100644\n--- a/t/lib-httpd/apache.conf\n+++ b/t/lib-httpd/apache.conf\n@@ -1,7 +1,4 @@ ServerName dummy\n ServerName dummy\n-\n-LockFile accept.lock\n-\n PidFile httpd.pid\n DocumentRoot www\n LogFormat \"%h %l %u %t \\\"%r\\\" %>s %b\" common\n@@ -26,6 +23,10 @@ ErrorLog error.log\n \tLoadModule version_module modules/mod_version.so\n \n \n+\n+LockFile accept.lock\n+\n+\n \n \n \tLoadModule auth_module modules/mod_auth.so\n-- \n1.8.3.rc2.14.g7eee6b3\n"},{"id":"221622","messageId":"603055CE-E8DE-4C53-9A7B-AE49CE515CCA@gernhardtsoftware.com","threadId":"34235","inReplyTo":"20130621181250.GA5290@sigill.intra.peff.net","subject":"Re: [PATCH] lib-httpd/apache.conf: check version only after mod_version loads","fromName":"Brian Gernhardt","fromEmail":"brian@gernhardtsoftware.com","sentAt":"2013-06-21T18:15:39Z","receivedAt":"2013-06-21T18:15:39Z","isPatch":true,"sender":{"key":"brian@gernhardtsoftware.com","avatar":"https://avatars.githubusercontent.com/u/133455?v=4"},"body":"\nOn Jun 21, 2013, at 2:12 PM, Jeff King wrote:\n\n> On Fri, Jun 21, 2013 at 02:08:49PM -0400, Brian Gernhardt wrote:\n> \n>> On Jun 21, 2013, at 2:03 PM, Jeff King wrote:\n>> \n>>> IfVersion comes from mod_version. I assume that if it were not\n>>> loaded, apache would complain about the directive entirely. But it's\n>>> true that we don't load it until later. Maybe try moving the\n>>> IfVersion/Lockfile stanza down below the mod_version LoadModule\n>>> line?\n>> \n>> Apache is apparently overly accepting. Moving the IfVersion below all\n>> the IfModules fixes it.\n> \n> Cool. I think the patch should look like the one below, then.\n\nBasically identical to what I've done, you're just faster to the actual patch. :-D\n\n+1\n\n> Just to double-check that I have explained the issue correctly, can you\n> share the output of \"apache2 -l\"? Mine has:\n> \n> $ apache2 -l\n> Compiled in modules:\n> core.c\n> mod_log_config.c\n> mod_logio.c\n> mod_version.c\n> prefork.c\n> http_core.c\n> mod_so.c\n> \n> which explains why it works here. I'm assuming you will not have\n> mod_version.c compiled in.\n\nIndeed I do not.\n\n $ httpd -l\n Compiled in modules:\n core.c\n prefork.c\n http_core.c\n mod_so.c\n"},{"id":"221623","messageId":"20130621181813.GB5290@sigill.intra.peff.net","threadId":"34235","inReplyTo":"603055CE-E8DE-4C53-9A7B-AE49CE515CCA@gernhardtsoftware.com","subject":"Re: [PATCH] lib-httpd/apache.conf: check version only after mod_version loads","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2013-06-21T18:18:13Z","receivedAt":"2013-06-21T18:18:13Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Jun 21, 2013 at 02:15:39PM -0400, Brian Gernhardt wrote:\n\n> > Cool. I think the patch should look like the one below, then.\n> \n> Basically identical to what I've done, you're just faster to the actual patch. :-D\n\nI started writing the rationale immediately after making the suggestion,\nbefore you responded. Lucky for me my guess turned out to be right. :)\n\n> > which explains why it works here. I'm assuming you will not have\n> > mod_version.c compiled in.\n> \n> Indeed I do not.\n> \n> $ httpd -l\n> Compiled in modules:\n> core.c\n> prefork.c\n> http_core.c\n> mod_so.c\n\nMakes sense. Thanks for the bug report.\n\n-Peff\n"},{"id":"221639","messageId":"7vmwqjkwmr.fsf@alter.siamese.dyndns.org","threadId":"34235","inReplyTo":"20130621181250.GA5290@sigill.intra.peff.net","subject":"Re: [PATCH] lib-httpd/apache.conf: check version only after mod_version loads","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-06-21T20:59:56Z","receivedAt":"2013-06-21T20:59:56Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> Cool. I think the patch should look like the one below, then.\n>\n> Just to double-check that I have explained the issue correctly, can you\n> share the output of \"apache2 -l\"? Mine has:\n>\n> $ apache2 -l\n> Compiled in modules:\n> core.c\n> mod_log_config.c\n> mod_logio.c\n> mod_version.c\n> prefork.c\n> http_core.c\n> mod_so.c\n>\n> which explains why it works here. I'm assuming you will not have\n> mod_version.c compiled in.\n>\n> -- >8 --\n> Subject: lib-httpd/apache.conf: check version only after mod_version loads\n>\n> Commit 0442743 introduced an directive near the\n> top of the apache config file. However, at that point we\n> have not yet checked for and loaded the mod_version module.\n> This means that the directive will behave oddly if\n> mod_version is dynamically loaded, failing to match when it\n> should.\n>\n> We can fix this by moving the whole block below the\n> LoadModule directive for mod_version.\n>\n> Reported-by: Brian Gernhardt \n> Signed-off-by: Jeff King \n> ---\n> t/lib-httpd/apache.conf | 7 ++++---\n> 1 file changed, 4 insertions(+), 3 deletions(-)\n>\n> diff --git a/t/lib-httpd/apache.conf b/t/lib-httpd/apache.conf\n> index 56ae548..dd17e3a 100644\n> --- a/t/lib-httpd/apache.conf\n> +++ b/t/lib-httpd/apache.conf\n> @@ -1,7 +1,4 @@ ServerName dummy\n> ServerName dummy\n> -\n> -LockFile accept.lock\n> -\n> PidFile httpd.pid\n> DocumentRoot www\n> LogFormat \"%h %l %u %t \\\"%r\\\" %>s %b\" common\n> @@ -26,6 +23,10 @@ ErrorLog error.log\n> \tLoadModule version_module modules/mod_version.so\n> \n> \n> +\n> +LockFile accept.lock\n> +\n> +\n\nOnce you see it in the patch form, it is very clear what this change\ndoes and why it is necessary in the context ;-)\n\nThanks, both of you, for digging this down to the root cause; you\nguys have done before it graduates to 'master', which I especially\nappreciate.\n\nWill queue.\n"}]}