curl-cffi Proxy Setup for Browser-like HTTP Requests

curl_cffi is useful when the buyer's job is a raw HTTP request that needs a chosen egress and a browser-like TLS or HTTP/2 fingerprint. It follows a Requests-style API, supports HTTP and SOCKS proxies, and can keep cookies and connections in a Session. It is still an HTTP client. It does not render a DOM, run page JavaScript, or create a browser profile.

The current release page lists v0.16.3; the documentation requires Python 3.10 or newer. Install the package in the same environment that will run the job:

python -m pip install 'curl_cffi==0.16.3'

Choose the route and client behavior

The official quick start shows the proxies mapping and impersonate parameter. Its key is the destination scheme, so an HTTPS request normally uses the https entry even when the proxy URL begins with http://. Do not silently turn a SOCKS endpoint into an HTTP URL.

Configure authentication from the environment

Keep the proxy secret out of source, shell history, process arguments and logs. This example builds the authority from environment values, uses one client for both checks, refuses redirects, and validates response bodies:

import json
import os
from urllib.parse import quote

from curl_cffi import Session

proxy_scheme = os.environ.get("PROXY_SCHEME", "http")
proxy_host = os.environ["PROXY_HOST"]
proxy_port = os.environ["PROXY_PORT"]
proxy_user = quote(os.environ["PROXY_USER"], safe="")
proxy_password = quote(os.environ["PROXY_PASSWORD"], safe="")
proxy_url = f"{proxy_scheme}://{proxy_user}:{proxy_password}@{proxy_host}:{proxy_port}"

def require_ip_body(body: str) -> None:
    value = json.loads(body)
    if not isinstance(value.get("ip"), str) or not value["ip"]:
        raise RuntimeError("exit check returned no IP")

with Session() as client:
    exit_response = client.get(
        "https://api.ipify.org?format=json",
        impersonate="chrome",
        proxies={"https": proxy_url},
        timeout=20,
        allow_redirects=False,
    )
    exit_response.raise_for_status()
    require_ip_body(exit_response.text)

    target_response = client.get(
        "https://example.com/",
        impersonate="chrome",
        proxies={"https": proxy_url},
        timeout=20,
        allow_redirects=False,
    )
    target_response.raise_for_status()
    if "Example Domain" not in target_response.text:
        raise RuntimeError("target body did not contain the expected marker")
    print({"exit_body_valid": True, "target_status": target_response.status_code})

The same Session proves that both requests used the same configured client and route policy. It does not prove that two requests have the same provider exit if the provider rotates per request. Record the IP body privately if you need to compare it with provider-side evidence; never print the proxy URL or the full response when it may contain sensitive content.

timeout=20 bounds each request call. A timeout is not a business-level deadline for a whole batch, so add that deadline in the worker. The context manager closes the session. If the production job streams a response, consume it or close it before the session exits.

Sticky sessions versus rotation

Use a stable proxy and one session when cookies, login state, locale or pagination belong to one logical sequence. The session preserves cookies and reuses connections; it does not ask a provider to make the IP sticky. That behavior comes from the proxy service's session contract.

Rotate between independent records only when the target's rules allow it and your parser can treat each request as a new identity. Rebuild the proxies mapping or the session according to the provider's documented rotation mechanism. Do not rotate halfway through a flow that depends on cookies and then call the result one session.

Read the failure at the right layer

The exit check is transport evidence. The Example Domain marker is useful-output evidence for this bounded sample. Carry the same idea into the real job: require the field, JSON key or canonical URL that makes a record usable. A 200 with a challenge page is a response, not a completed extraction.

For browser pages, use the Playwright proxy guide. For proxy authentication failures, see HTTP 407 troubleshooting. The curl proxy guide covers CONNECT phases when you need to inspect the lower-level route.

curl_cffi's HTTP/3 support is a protocol feature with its own proxy requirements. Treat http_version="v3" as an explicit compatibility decision, and verify the proxy's UDP path before enabling it. Do not infer HTTP/3 or browser execution from a successful HTTP/2 request.