From: Ravi Mistry via GitGitGadget Date: Sat, 31 Oct 2026 15:57:02 GMT Subject: [PATCH 2/4] http: add exponential backoff without Retry-After Message-ID: <36cef34668b6e2ceb867ca99029e8bc11d9cd63f.1793462224.git.gitgitgadget@gmail.com> In-Reply-To: From: Ravi Mistry When a server returns HTTP 429 without a Retry-After header, Git waits the same fixed http.retryAfter delay on every retry. If that delay is short (e.g., 1s), Git burns through all of its retries in just a few seconds before a busy server can recover. If that delay is long, even a brief glitch has to wait a long time on the first retry. Pass the 1-based retry attempt count to handle_rate_limit_retry() and double http.retryAfter on each subsequent retry (1s, 2s, 4s, ..., via http.retryAfter * 2^(attempt - 1)), clamping the delay at http.maxRetryTime. This keeps the first retry fast while giving the server more time to recover on later attempts, matching curl's own --retry behavior. When the server provides an explicit Retry-After header, continue to honor the server-provided delay (or fail fast if it exceeds http.maxRetryTime). Signed-off-by: Ravi Mistry --- Documentation/config/http.adoc | 11 ++++++++--- http.c | 16 +++++++++++++--- t/t5584-http-429-retry.sh | 33 +++++++++++++++++++++++++++++++++ 3 files changed, 54 insertions(+), 6 deletions(-) diff --git a/Documentation/config/http.adoc b/Documentation/config/http.adoc index 792a71b413..b387b0ad6a 100644 --- a/Documentation/config/http.adoc +++ b/Documentation/config/http.adoc @@ -327,9 +327,12 @@ http.keepAliveCount:: `GIT_HTTP_KEEPALIVE_COUNT` environment variable. http.retryAfter:: - Default wait time in seconds before retrying when a server returns + Initial wait time in seconds before retrying when a server returns HTTP 429 (Too Many Requests) without a Retry-After header. - Defaults to 0 (retry immediately). When a Retry-After header is + Defaults to 0 (retry immediately). When set to a positive value and + no Retry-After header is present, the delay doubles on each + subsequent retry attempt (`http.retryAfter * 2^(attempt - 1)`), + capped at `http.maxRetryTime`. When a Retry-After header is present, its value takes precedence over this setting; however, automatic use of the server-provided `Retry-After` header requires libcurl 7.66.0 or later. On older versions, configure this setting @@ -348,7 +351,9 @@ http.maxRetryTime:: handling HTTP 429 (Too Many Requests) responses. If the server requests a delay (via Retry-After header) or if `http.retryAfter` is configured with a value that exceeds this maximum, Git will fail - immediately rather than waiting. Default is 300 seconds (5 minutes). + immediately rather than waiting. Exponentially backed-off delays + computed from `http.retryAfter` are capped at this maximum. Default + is 300 seconds (5 minutes). Can be overridden by the `GIT_HTTP_MAX_RETRY_TIME` environment variable. See also `http.retryAfter` and `http.maxRetries`. diff --git a/http.c b/http.c index 0308f973a6..0dec6377dd 100644 --- a/http.c +++ b/http.c @@ -2411,7 +2411,7 @@ static int update_url_from_redirect(struct strbuf *base, * Returns a negative value if configuration is invalid (delay exceeds * http.maxRetryTime), otherwise returns the delay in seconds (>= 0). */ -static long handle_rate_limit_retry(long slot_retry_after) +static long handle_rate_limit_retry(long slot_retry_after, int attempt) { /* Use the slot-specific retry_after value or configured default */ if (slot_retry_after >= 0) { @@ -2427,6 +2427,8 @@ static long handle_rate_limit_retry(long slot_retry_after) } return slot_retry_after; } else { + long delay = http_retry_after; + /* No Retry-After header provided, use configured default */ if (http_retry_after > http_max_retry_time) { error(_("configured http.retryAfter exceeds http.maxRetryTime (%ld > %ld seconds)"), @@ -2437,7 +2439,14 @@ static long handle_rate_limit_retry(long slot_retry_after) } trace2_data_string("http", the_repository, "http/429-retry-source", "config-default"); - return http_retry_after; + for (int a = 1; a < attempt && delay > 0; a++) { + if (delay >= http_max_retry_time - delay) { + delay = http_max_retry_time; + break; + } + delay *= 2; + } + return delay; } } @@ -2449,6 +2458,7 @@ static int http_request_recoverable(const char *url, int i = 3; int ret; int rate_limit_retries = http_max_retries; + int attempt = 0; if (!options) options = &empty_opts; @@ -2498,7 +2508,7 @@ static int http_request_recoverable(const char *url, BUG("Unknown http_request target"); } if (ret == HTTP_RATE_LIMITED) { - retry_delay = handle_rate_limit_retry(options->retry_after); + retry_delay = handle_rate_limit_retry(options->retry_after, ++attempt); if (retry_delay < 0) return HTTP_ERROR; diff --git a/t/t5584-http-429-retry.sh b/t/t5584-http-429-retry.sh index d92a26893d..f0c3cc3b61 100755 --- a/t/t5584-http-429-retry.sh +++ b/t/t5584-http-429-retry.sh @@ -82,6 +82,39 @@ test_expect_success 'HTTP 429 without Retry-After uses configured default' ' test_grep "refs/heads/" output ' +test_expect_success 'exponential backoff doubles http.retryAfter without Retry-After header' ' + test_config http.maxRetries 3 && + test_config http.retryAfter 1 && + git ls-remote "$HTTPD_URL/http_429/exp-backoff-fail-3/none/repo.git" >output 2>err && + test_grep "refs/heads/" output && + test_grep "waiting 1 seconds before retry" err && + test_grep "waiting 2 seconds before retry" err && + test_grep "waiting 4 seconds before retry" err +' + +test_expect_success 'exponential backoff clamps delay at http.maxRetryTime' ' + test_config http.maxRetries 3 && + test_config http.retryAfter 1 && + test_config http.maxRetryTime 2 && + GIT_TRACE2_EVENT="$(pwd)/trace-clamp.event" \ + git ls-remote "$HTTPD_URL/http_429/exp-clamp-fail-3/none/repo.git" >output 2>err && + test_grep "refs/heads/" output && + test_grep "waiting 1 seconds before retry" err && + test_grep "waiting 2 seconds before retry" err && + test_grep ! "waiting 4 seconds before retry" err && + grep "\"key\":\"http/retry-sleep-seconds\",\"value\":\"2\"" trace-clamp.event >clamped && + test_line_count = 2 clamped +' + +test_expect_success 'server Retry-After header overrides exponential backoff' ' + test_config http.maxRetries 2 && + test_config http.retryAfter 1 && + git ls-remote "$HTTPD_URL/http_429/header-override-fail-2/1/repo.git" >output 2>err && + test_grep "refs/heads/" output && + test_grep "waiting 1 seconds before retry" err && + test_grep ! "waiting 2 seconds before retry" err +' + test_expect_success 'HTTP 429 retry delays are respected' ' # Enable retries test_config http.maxRetries 3 && -- gitgitgadget