How to configure a proxy in Crawl4AI

Set CrawlerRunConfig.proxy_config for each Crawl4AI request, then validate the IP-check response, target status, and a required field in the extracted Markdown. The IP and target checks are separate requests; a rotating proxy can change exits between them, and result.success alone does not validate the document.

Prepare a small test run

Use Python 3.12, an authorized target, and a marker expected in its main content. Install the tested Crawl4AI release and its Chromium browser using the official installation guide. The example makes one GET crawl and disables cache use.

Warning: Proxy credentials can leak through configuration dumps, logs, or an unencrypted proxy hop. Load them from a runtime secret store, prefer an HTTPS proxy endpoint when available, and test against a permitted URL. Keep the sample to one URL; set a worker deadline when this runs inside a larger job, then remove any temporary secret binding.

python3.12 -m venv .venv
. .venv/bin/activate
python -m pip install 'crawl4ai==0.9.4'
python -m playwright install chromium

Set PROXY_SERVER, optional PROXY_USERNAME and PROXY_PASSWORD, IP_CHECK_URL, TARGET_URL, and EXPECTED_MARKER in the environment. IP_CHECK_URL must return JSON with an origin value. TARGET_URL should point to a page permitted for this workload.

Crawl and check the extracted result

Save this complete example as crawl-check.py. page_timeout is in milliseconds. The script caps marker inspection to 12,000 Markdown characters and prints no proxy details or response body.

import asyncio
import html
import ipaddress
import json
import os
import re
from urllib.parse import urlsplit

from crawl4ai import (
    AsyncWebCrawler,
    BrowserConfig,
    CacheMode,
    CrawlerRunConfig,
    ProxyConfig,
)


PROXY_SERVER = os.environ["PROXY_SERVER"]
PROXY_USERNAME = os.environ.get("PROXY_USERNAME", "")
PROXY_PASSWORD = os.environ.get("PROXY_PASSWORD", "")
IP_CHECK_URL = os.environ["IP_CHECK_URL"]
TARGET_URL = os.environ["TARGET_URL"]
EXPECTED_MARKER = os.environ["EXPECTED_MARKER"].strip()
MAX_MARKDOWN_CHARS = 12_000

proxy_parts = urlsplit(PROXY_SERVER)
try:
    proxy_port = proxy_parts.port
except ValueError:
    proxy_port = None
if (
    proxy_parts.scheme not in {"http", "https", "socks5"}
    or not proxy_parts.hostname
    or proxy_port is None
    or not 1 <= proxy_port <= 65_535
    or proxy_parts.username
    or proxy_parts.password
    or proxy_parts.path not in {"", "/"}
    or proxy_parts.query
    or proxy_parts.fragment
):
    raise SystemExit("Set PROXY_SERVER to a supported proxy scheme, host, and port without credentials")
if not EXPECTED_MARKER:
    raise SystemExit("EXPECTED_MARKER must not be empty")


async def crawl_one(crawler: AsyncWebCrawler, url: str, proxy: ProxyConfig):
    config = CrawlerRunConfig(
        proxy_config=proxy,
        cache_mode=CacheMode.BYPASS,
        page_timeout=20_000,
        wait_until="domcontentloaded",
        word_count_threshold=1,
        max_retries=0,
    )
    return await crawler.arun(url=url, config=config)


async def main() -> None:
    if bool(PROXY_USERNAME) != bool(PROXY_PASSWORD):
        raise SystemExit("Set both proxy credential variables or leave both empty")
    proxy = ProxyConfig(
        server=PROXY_SERVER,
        username=PROXY_USERNAME or None,
        password=PROXY_PASSWORD or None,
    )

    async with AsyncWebCrawler(
        config=BrowserConfig(headless=True, verbose=False)
    ) as crawler:
        ip_result = await crawl_one(crawler, IP_CHECK_URL, proxy)
        if not ip_result.success or ip_result.status_code != 200:
            raise SystemExit("IP check failed before target crawl")
        json_block = re.search(r"<pre[^>]*>(.*?)</pre>", ip_result.cleaned_html, re.DOTALL)
        if json_block is None:
            raise SystemExit("IP check did not return a JSON body")
        ip_payload = json.loads(html.unescape(json_block.group(1)))
        observed_ip = ip_payload.get("origin")
        if not isinstance(observed_ip, str):
            raise SystemExit("IP check returned no origin value")
        ipaddress.ip_address(observed_ip.strip())

        target_result = await crawl_one(crawler, TARGET_URL, proxy)
        if not target_result.success or target_result.status_code != 200:
            raise SystemExit("Target did not return a successful HTTP response")
        markdown = target_result.markdown.raw_markdown[:MAX_MARKDOWN_CHARS]
        if EXPECTED_MARKER not in markdown:
            raise SystemExit("Required marker was not found in extracted Markdown")

        print({"exit_ip_present": True, "target_status": 200, "content_valid": True})


try:
    asyncio.run(main())
except Exception as error:
    raise SystemExit(f"Crawl check failed ({type(error).__name__})") from None

Run the saved script with the environment variables set: python crawl-check.py.

Expected output

The script prints this only after the IP echo returned HTTP 200 with a valid IPv4 or IPv6 address and the target returned HTTP 200 with the required marker. exit_ip_present means the echo value passed IP parsing; the address is not printed. target_status is the target document’s HTTP status, and content_valid means the marker was found in the inspected content. This line matches the successful task-owned local-fixture run; it is an example result, not a claim about a proxy provider or an external target.

{'exit_ip_present': True, 'target_status': 200, 'content_valid': True}

The example makes 2 explicit crawls: one for the IP check and one for the target. Crawl4AI is configured with max_retries=0 for those crawls; this is not a global request budget and does not bound browser startup, subresources, or the whole job. Each page navigation has a 20-second timeout. The character cap applies after the browser returns page content; it does not cap network bytes or subresource downloads.

Read failures by layer

An http:// proxy URL does not encrypt the client-to-proxy hop. For an HTTPS target, target TLS protects the separate browser-to-target connection through a CONNECT tunnel when the proxy supports it. An https:// proxy endpoint adds TLS on the client-to-proxy hop. Keep credentials out of printed ProxyConfig values and logs.

The IP echo reports the address seen for that request. It does not prove that the next request uses the same address, that the address maps to a particular jurisdiction, or that a target accepts the crawl. Check target rules and validate the fields the parser needs. For browser traffic without Crawl4AI extraction, use the Playwright proxy guide.

ProxyLaneTraffic never expires

Premium residential proxies
From $2/GB

Scale your workflows with the tools you already use. Rotating or sticky sessions up to 72 hours.

Try 350 MB for $1.95 View plans