{"thread":{"id":"45862","subject":"[PATCH v2] travis-ci: retry if Git for Windows CI returns HTTP error 502 or 503","startedAt":"2017-05-03T21:50:26Z","lastAt":"2017-05-09T23:51:03Z","messageCount":5,"participants":["Lars Schneider","Johannes Schindelin","Junio C Hamano"],"isPatch":true,"patchVersion":2,"patchTotal":null},"messages":[{"id":"318705","messageId":"20170503215015.17949-1-larsxschneider@gmail.com","threadId":"45862","inReplyTo":null,"subject":"[PATCH v2] travis-ci: retry if Git for Windows CI returns HTTP error 502 or 503","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2017-05-03T21:50:15Z","receivedAt":"2017-05-03T21:50:26Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"The Git for Windows CI web app sometimes returns HTTP errors of\n\"502 bad gateway\" or \"503 service unavailable\" [1]. We also need to\ncheck the HTTP content because the GfW web app seems to pass through\n(error) results from other Azure calls with HTTP code 200.\nWait a little and retry the request if this happens.\n\n[1] https://docs.microsoft.com/en-in/azure/app-service-web/app-service-web-troubleshoot-http-502-http-503\n\nSigned-off-by: Lars Schneider <larsxschneider@gmail.com>\n---\n\nHi Junio,\n\nI can't really test this as my TravisCI account does not have the\nextended timeout and I am unable to reproduce the error.\n\nIt would be great if we could test this is a little bit in pu.\n\nThanks,\nLars\n\nNotes:\n    Base Ref: next\n    Web-Diff: https://github.com/larsxschneider/git/commit/af0f0f0eb8\n    Checkout: git fetch https://github.com/larsxschneider/git travisci/win-retry-v2 && git checkout af0f0f0eb8\n\n    Interdiff (v1..v2):\n\n    diff --git a/ci/run-windows-build.sh b/ci/run-windows-build.sh\n    index 7a9aa9c6a7..3e5a0abee0 100755\n    --- a/ci/run-windows-build.sh\n    +++ b/ci/run-windows-build.sh\n    @@ -14,26 +14,33 @@ COMMIT=$2\n\n     gfwci () {\n     \tlocal CURL_ERROR_CODE HTTP_CODE\n    -\texec 3>&1\n    +\tCONTENT_FILE=$(mktemp -t \"git-windows-ci-XXXXXX\")\n     \twhile test -z $HTTP_CODE\n     \tdo\n     \tHTTP_CODE=$(curl \\\n     \t\t-H \"Authentication: Bearer $GFW_CI_TOKEN\" \\\n     \t\t--silent --retry 5 --write-out '%{HTTP_CODE}' \\\n    -\t\t--output >(sed \"$(printf '1s/^\\xef\\xbb\\xbf//')\" >cat >&3) \\\n    +\t\t--output >(sed \"$(printf '1s/^\\xef\\xbb\\xbf//')\" >$CONTENT_FILE) \\\n     \t\t\"https://git-for-windows-ci.azurewebsites.net/api/TestNow?$1\" \\\n     \t)\n     \tCURL_ERROR_CODE=$?\n     \t\t# The GfW CI web app sometimes returns HTTP errors of\n     \t\t# \"502 bad gateway\" or \"503 service unavailable\".\n    -\t\t# Wait a little and retry if it happens. More info:\n    +\t\t# We also need to check the HTTP content because the GfW web\n    +\t\t# app seems to pass through (error) results from other Azure\n    +\t\t# calls with HTTP code 200.\n    +\t\t# Wait a little and retry if we detect this error. More info:\n     \t\t# https://docs.microsoft.com/en-in/azure/app-service-web/app-service-web-troubleshoot-http-502-http-503\n    -\t\tif test $HTTP_CODE -eq 502 || test $HTTP_CODE -eq 503\n    +\t\tif test $HTTP_CODE -eq 502 ||\n    +\t\t   test $HTTP_CODE -eq 503 ||\n    +\t\t   grep \"502 - Web server received an invalid response\" $CONTENT_FILE >/dev/null\n     \t\tthen\n     \t\t\tsleep 10\n     \t\t\tHTTP_CODE=\n     \t\tfi\n     \tdone\n    +\tcat $CONTENT_FILE\n    +\trm $CONTENT_FILE\n     \tif test $CURL_ERROR_CODE -ne 0\n     \tthen\n     \t\treturn $CURL_ERROR_CODE\n\n    \\0\n\n ci/run-windows-build.sh | 23 +++++++++++++++++++++--\n 1 file changed, 21 insertions(+), 2 deletions(-)\n\ndiff --git a/ci/run-windows-build.sh b/ci/run-windows-build.sh\nindex e043440799..3e5a0abee0 100755\n--- a/ci/run-windows-build.sh\n+++ b/ci/run-windows-build.sh\n@@ -14,14 +14,33 @@ COMMIT=$2\n\n gfwci () {\n \tlocal CURL_ERROR_CODE HTTP_CODE\n-\texec 3>&1\n+\tCONTENT_FILE=$(mktemp -t \"git-windows-ci-XXXXXX\")\n+\twhile test -z $HTTP_CODE\n+\tdo\n \tHTTP_CODE=$(curl \\\n \t\t-H \"Authentication: Bearer $GFW_CI_TOKEN\" \\\n \t\t--silent --retry 5 --write-out '%{HTTP_CODE}' \\\n-\t\t--output >(sed \"$(printf '1s/^\\xef\\xbb\\xbf//')\" >cat >&3) \\\n+\t\t--output >(sed \"$(printf '1s/^\\xef\\xbb\\xbf//')\" >$CONTENT_FILE) \\\n \t\t\"https://git-for-windows-ci.azurewebsites.net/api/TestNow?$1\" \\\n \t)\n \tCURL_ERROR_CODE=$?\n+\t\t# The GfW CI web app sometimes returns HTTP errors of\n+\t\t# \"502 bad gateway\" or \"503 service unavailable\".\n+\t\t# We also need to check the HTTP content because the GfW web\n+\t\t# app seems to pass through (error) results from other Azure\n+\t\t# calls with HTTP code 200.\n+\t\t# Wait a little and retry if we detect this error. More info:\n+\t\t# https://docs.microsoft.com/en-in/azure/app-service-web/app-service-web-troubleshoot-http-502-http-503\n+\t\tif test $HTTP_CODE -eq 502 ||\n+\t\t   test $HTTP_CODE -eq 503 ||\n+\t\t   grep \"502 - Web server received an invalid response\" $CONTENT_FILE >/dev/null\n+\t\tthen\n+\t\t\tsleep 10\n+\t\t\tHTTP_CODE=\n+\t\tfi\n+\tdone\n+\tcat $CONTENT_FILE\n+\trm $CONTENT_FILE\n \tif test $CURL_ERROR_CODE -ne 0\n \tthen\n \t\treturn $CURL_ERROR_CODE\n\nbase-commit: 1ea7e62026c5dde4d8be80b2544696fc6aa70121\n--\n2.12.2\n\n"},{"id":"318736","messageId":"alpine.DEB.2.21.1.1705041118190.4905@virtualbox","threadId":"45862","inReplyTo":"20170503215015.17949-1-larsxschneider@gmail.com","subject":"Re: [PATCH v2] travis-ci: retry if Git for Windows CI returns HTTP error 502 or 503","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2017-05-04T09:19:21Z","receivedAt":"2017-05-04T09:19:43Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Lars,\n\n\nOn Wed, 3 May 2017, Lars Schneider wrote:\n\n> The Git for Windows CI web app sometimes returns HTTP errors of\n> \"502 bad gateway\" or \"503 service unavailable\" [1]. We also need to\n> check the HTTP content because the GfW web app seems to pass through\n> (error) results from other Azure calls with HTTP code 200.\n> Wait a little and retry the request if this happens.\n\nThanks. In theory, it would be better to fix the web app to pass through\nalso the 502 error code, in practice I have a hard time finding the time\nto make it so ;-)\n\nTherefore, I would be very much in favor of the current version of the\npatch.\n\nCiao,\nDscho\n"},{"id":"319169","messageId":"xmqqpofi35ik.fsf@gitster.mtv.corp.google.com","threadId":"45862","inReplyTo":"20170503215015.17949-1-larsxschneider@gmail.com","subject":"Re: [PATCH v2] travis-ci: retry if Git for Windows CI returns HTTP error 502 or 503","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-05-09T06:31:47Z","receivedAt":"2017-05-09T06:31:57Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Lars Schneider <larsxschneider@gmail.com> writes:\n\n> The Git for Windows CI web app sometimes returns HTTP errors of\n> \"502 bad gateway\" or \"503 service unavailable\" [1]. We also need to\n> check the HTTP content because the GfW web app seems to pass through\n> (error) results from other Azure calls with HTTP code 200.\n> Wait a little and retry the request if this happens.\n>\n> [1] https://docs.microsoft.com/en-in/azure/app-service-web/app-service-web-troubleshoot-http-502-http-503\n>\n> Signed-off-by: Lars Schneider <larsxschneider@gmail.com>\n> ---\n>\n> Hi Junio,\n>\n> I can't really test this as my TravisCI account does not have the\n> extended timeout and I am unable to reproduce the error.\n>\n> It would be great if we could test this is a little bit in pu.\n\nThis has been in 'pu' for a while.  \n\nAs the patch simply discards 502 (and others), it is unclear if the\nfailing test on 'next' is now gone, or the attempt to run 'pu'\nhappened to be lucky not to get one, from the output we can see in\nhttps://travis-ci.org/git/git/jobs/229867212\n\nAre you comfortable enough to move this forward?  It's not like a\npossible breakage in this patch will harm anything (the relaying to\nthe Windows CI is flaky if the build server cannot deal with the\nload anyway), so I would rather have this early in 'next', while we\ndeal with a few other topics that Windows build is not happy with\nthat are on 'pu'.\n\n\n"},{"id":"319208","messageId":"1ADD08FD-1DC0-4AC5-9E69-AC1A5DCBED83@gmail.com","threadId":"45862","inReplyTo":"xmqqpofi35ik.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v2] travis-ci: retry if Git for Windows CI returns HTTP error 502 or 503","fromName":"Lars Schneider","fromEmail":"larsxschneider@gmail.com","sentAt":"2017-05-09T17:40:01Z","receivedAt":"2017-05-09T17:40:09Z","isPatch":true,"sender":{"key":"larsxschneider@gmail.com","avatar":"https://avatars.githubusercontent.com/u/477434?v=4"},"body":"\n> On 09 May 2017, at 07:31, Junio C Hamano <gitster@pobox.com> wrote:\n> \n> Lars Schneider <larsxschneider@gmail.com> writes:\n> \n>> The Git for Windows CI web app sometimes returns HTTP errors of\n>> \"502 bad gateway\" or \"503 service unavailable\" [1]. We also need to\n>> check the HTTP content because the GfW web app seems to pass through\n>> (error) results from other Azure calls with HTTP code 200.\n>> Wait a little and retry the request if this happens.\n>> \n>> [1] https://docs.microsoft.com/en-in/azure/app-service-web/app-service-web-troubleshoot-http-502-http-503\n>> \n>> Signed-off-by: Lars Schneider <larsxschneider@gmail.com>\n>> ---\n>> \n>> Hi Junio,\n>> \n>> I can't really test this as my TravisCI account does not have the\n>> extended timeout and I am unable to reproduce the error.\n>> \n>> It would be great if we could test this is a little bit in pu.\n> \n> This has been in 'pu' for a while.  \n> \n> As the patch simply discards 502 (and others), it is unclear if the\n> failing test on 'next' is now gone, or the attempt to run 'pu'\n> happened to be lucky not to get one, from the output we can see in\n> https://travis-ci.org/git/git/jobs/229867212\n> \n> Are you comfortable enough to move this forward?\n\nYes, please move it forward. I haven't seen a \"502 - Web server \nreceived an invalid response\" on pu for a while. That means the\npatch should work as expected.\n\n\nUnrelated to this patch I have, however, seen two kinds of timeouts:\n\n(1) Timeout in the \"notStarted\" state. This job eventually finished\nwith a failure but it did start only *after* 3h:\nhttps://travis-ci.org/git/git/jobs/230225611\n\n(2) Timeout in the \"in progress\" state. This job eventually finished\nsuccessfully but it took longer than 3h:\nhttps://travis-ci.org/git/git/jobs/229867248\n\nRight now the timeout generates potential false negative results. \nI would like to change that and respond with a successful build \n*before* we approach the 3h timeout. This means we could generate\nfalse positives. Although this is not ideal, I think that is the better \ncompromise as a failing Windows build would usually fail quickly \n(e.g. in the compile step).\n\nWhat do you guys think? Would you be OK with that reasoning?\nIf the Git for Windows builds get more stable over time then\nwe could reevaluate this compromise.\n\n\n- Lars"},{"id":"319241","messageId":"xmqqmvalzj1b.fsf@gitster.mtv.corp.google.com","threadId":"45862","inReplyTo":"1ADD08FD-1DC0-4AC5-9E69-AC1A5DCBED83@gmail.com","subject":"Re: [PATCH v2] travis-ci: retry if Git for Windows CI returns HTTP error 502 or 503","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-05-09T23:50:56Z","receivedAt":"2017-05-09T23:51:03Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Lars Schneider <larsxschneider@gmail.com> writes:\n\n>>> It would be great if we could test this is a little bit in pu.\n>> \n>> This has been in 'pu' for a while.  \n>> \n>> As the patch simply discards 502 (and others), it is unclear if the\n>> failing test on 'next' is now gone, or the attempt to run 'pu'\n>> happened to be lucky not to get one, from the output we can see in\n>> https://travis-ci.org/git/git/jobs/229867212\n>> \n>> Are you comfortable enough to move this forward?\n>\n> Yes, please move it forward. I haven't seen a \"502 - Web server \n> received an invalid response\" on pu for a while. That means the\n> patch should work as expected.\n\nWill do, thanks.\n\n> Unrelated to this patch I have, however, seen two kinds of timeouts:\n>\n> (1) Timeout in the \"notStarted\" state. This job eventually finished\n> with a failure but it did start only *after* 3h:\n> https://travis-ci.org/git/git/jobs/230225611\n>\n> (2) Timeout in the \"in progress\" state. This job eventually finished\n> successfully but it took longer than 3h:\n> https://travis-ci.org/git/git/jobs/229867248\n>\n> Right now the timeout generates potential false negative results. \n> I would like to change that and respond with a successful build \n> *before* we approach the 3h timeout. This means we could generate\n> false positives. Although this is not ideal, I think that is the better \n> compromise as a failing Windows build would usually fail quickly \n> (e.g. in the compile step).\n>\n> What do you guys think? Would you be OK with that reasoning?\n> If the Git for Windows builds get more stable over time then\n> we could reevaluate this compromise.\n\nI'd rather see a false breakage on Windows build (i.e. \"this might\nhave succeeded given enough time, but it didn't finish within the\nalloted time\") than a false sucess (i.e. \"we successfully launched\nand the build is still running, so let's assume the test succeeds\").\nBecause I do not pay attention to what the overall build page [*1*]\nsays about a particular branch tip, and I instead look at the\nsummary list of the indiviaul \"Build Jobs\", e.g. [*2*]), seeing\nerrored/failed on [*1*] does not bother me personally, if that is\nwhat you are getting at.\n\n\n[References]\n\n*1* https://travis-ci.org/git/git/builds/\n*2* https://travis-ci.org/git/git/builds/230235081\n"}]}