How to handle HTTP 429 with Retry-After

A 429 Too Many Requests response means the responding service is rate limiting the client; Retry-After gives a minimum delay in seconds or an HTTP date. Cap attempts and elapsed time, and stop if that delay exceeds the budget. Never rotate IPs to bypass the limit; the loopback fixture tests client behavior only.

Set the retry boundary

RFC 6585 §4 defines status 429 and allows Retry-After; RFC 9110 §10.2.3 defines a delay in seconds or an HTTP date. A response may come from an origin or an intermediary, so record the status and responding hop when that information is available.

The sample uses Python 3.12.0 and only the standard library. The example allows at most 4 requests within a 15-second retry-scheduling budget. Each request has a socket timeout of at most 2 seconds. When the header is absent or invalid, it uses capped exponential backoff. When the server’s requested delay exceeds the remaining deadline, it stops instead of retrying early.

Prerequisites

Use Python 3.12.0. Install it from the official Python 3.12.0 release page; the example uses only the standard library.

Important: The example contacts only a loopback HTTP fixture. It does not require proxy credentials, public targets, or paid services, and it proves nothing about an external proxy exit. Apply the same client policy only to requests the target permits; a rate limit is a request to reduce pressure, not a cue to change IP addresses.

Run the bounded example

Save this as retry-429.py, then run python3 retry-429.py. The fixture returns one 429 with a zero-second delay and then 200 on /retry. Its /too-long path always returns 429 with a 30-second delay, which exceeds the short test deadline and must not be retried.

from datetime import datetime, timezone
from email.utils import parsedate_to_datetime
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from threading import Thread
from time import monotonic, sleep
from urllib.error import HTTPError
from urllib.request import urlopen

MAX_ATTEMPTS = 4
JOB_DEADLINE_SECONDS = 15.0
REQUEST_TIMEOUT_SECONDS = 2.0
BACKOFF_BASE_SECONDS = 0.5
BACKOFF_CAP_SECONDS = 4.0


class RetryDeadlineExceeded(Exception):
    pass


class RetryLimitExceeded(Exception):
    pass


def parse_retry_after(value: str | None, now: datetime) -> float | None:
    if value is None:
        return None

    value = value.strip()
    if value.isascii() and value.isdecimal():
        try:
            return float(int(value))
        except (OverflowError, ValueError):
            return float("inf")

    try:
        retry_at = parsedate_to_datetime(value)
    except (TypeError, ValueError, OverflowError):
        return None

    if retry_at is None or retry_at.tzinfo is None:
        return None

    return max(0.0, (retry_at - now).total_seconds())


def fetch_with_retry(
    url: str,
    *,
    max_attempts: int = MAX_ATTEMPTS,
    deadline_seconds: float = JOB_DEADLINE_SECONDS,
    request_timeout_seconds: float = REQUEST_TIMEOUT_SECONDS,
) -> tuple[int, bytes, int]:
    if max_attempts < 1 or deadline_seconds <= 0 or request_timeout_seconds <= 0:
        raise ValueError("attempts and timeouts must be positive")

    deadline = monotonic() + deadline_seconds
    last_429: HTTPError | None = None

    for attempt in range(max_attempts):
        remaining = deadline - monotonic()
        if remaining <= 0:
            raise RetryDeadlineExceeded("job deadline expired before request") from last_429

        try:
            with urlopen(url, timeout=min(request_timeout_seconds, remaining)) as response:
                return response.status, response.read(), attempt + 1
        except HTTPError as error:
            if error.code != 429:
                raise
            last_429 = error
            retry_after = parse_retry_after(
                error.headers.get("Retry-After"), datetime.now(timezone.utc)
            )
            error.close()

        if attempt + 1 == max_attempts:
            raise RetryLimitExceeded("maximum total attempts reached") from last_429

        if retry_after is None:
            delay = min(BACKOFF_BASE_SECONDS * (2 ** attempt), BACKOFF_CAP_SECONDS)
        else:
            delay = retry_after

        remaining = deadline - monotonic()
        if delay > remaining:
            raise RetryDeadlineExceeded(
                "Retry-After or backoff exceeds remaining job deadline"
            ) from last_429
        sleep(delay)

    raise AssertionError("unreachable")


class FixtureHandler(BaseHTTPRequestHandler):
    retry_attempts = 0
    long_wait_attempts = 0

    def do_GET(self) -> None:
        if self.path == "/retry":
            type(self).retry_attempts += 1
            if type(self).retry_attempts == 1:
                self.send_response(429)
                self.send_header("Retry-After", "0")
                self.end_headers()
                return
            self.send_response(200)
            self.end_headers()
            self.wfile.write(b"fixture success")
            return

        if self.path == "/too-long":
            type(self).long_wait_attempts += 1
            self.send_response(429)
            self.send_header("Retry-After", "30")
            self.end_headers()
            return

        self.send_response(404)
        self.end_headers()

    def log_message(self, format: str, *args: object) -> None:
        pass


def run_checks() -> None:
    fixed_now = datetime(2026, 9, 26, 12, 0, tzinfo=timezone.utc)
    assert parse_retry_after("12", fixed_now) == 12.0
    assert parse_retry_after("invalid", fixed_now) is None
    assert parse_retry_after("9" * 5000, fixed_now) == float("inf")
    assert parse_retry_after("Sat, 26 Sep 2026 12:00:12 GMT", fixed_now) == 12.0

    server = ThreadingHTTPServer(("127.0.0.1", 0), FixtureHandler)
    thread = Thread(target=server.serve_forever, daemon=True)
    thread.start()
    base_url = f"http://127.0.0.1:{server.server_port}"

    try:
        status, body, attempts = fetch_with_retry(f"{base_url}/retry")
        assert (status, body, attempts) == (200, b"fixture success", 2)
        assert FixtureHandler.retry_attempts == 2
        print("PASS: 429 with delta-seconds was followed by fixture 200 in 2 attempts")

        try:
            fetch_with_retry(f"{base_url}/too-long", deadline_seconds=0.2)
        except RetryDeadlineExceeded:
            assert FixtureHandler.long_wait_attempts == 1
            print("PASS: over-deadline Retry-After stopped after 1 request")
        else:
            raise AssertionError("expected RetryDeadlineExceeded")

        print("PASS: delta-seconds, HTTP-date, and invalid-header parsing checks")
        print("Scope: loopback fixture only; no external proxy route was tested")
    finally:
        server.shutdown()
        server.server_close()
        thread.join(timeout=2)


if __name__ == "__main__":
    run_checks()

Expected output:

PASS: 429 with delta-seconds was followed by fixture 200 in 2 attempts
PASS: over-deadline Retry-After stopped after 1 request
PASS: delta-seconds, HTTP-date, and invalid-header parsing checks
Scope: loopback fixture only; no external proxy route was tested

The test uses a zero-second header only to avoid delaying a local check. A real positive Retry-After is not shortened to fit the deadline: the job stops and can be rescheduled after the required time. The HTTP-date parser uses the local UTC clock, so clock synchronization matters when a server sends an absolute time.

The retry-scheduling budget is checked between blocking requests, and the socket timeout is capped by its remaining time. Python’s urllib call cannot cancel an in-progress DNS lookup or slow response read at an exact wall-clock deadline. Use a process supervisor when hard termination at a fixed time is required.

Choose retry behavior by outcome

A retry is appropriate only when the operation is safe to repeat and the deadline allows the requested wait. The sample retries a GET; a timed-out write may already have reached the service. Check the operation’s idempotency or use the target’s supported idempotency key before replaying a state-changing request.

When the attempt limit is reached, surface a rate-limited result with the last response context and stop the job. Do not turn 401, 403, 407, or other errors into this retry path. 407 is a proxy authentication challenge described by RFC 9110; troubleshoot it separately with the proxy 407 guide.

Record the URL category, responding hop if known, status, Retry-After value, attempts, and elapsed time. Redact authorization fields and credential-bearing URLs. If the deadline stops the job, return it to a scheduler for a later permitted run or mark it for review; do not route around the limit by rotating proxy IPs.

ProxyLaneFrom $2/GB at 5 TB+

Your next connection
Starts here

Non-expiring traffic, location targeting and rotating or sticky sessions for your existing tools.

Create an account View plans