Amazon Scraper API vs Proxy: Choose a Product Data Source
Compare seller APIs, licensed Amazon product data and proxy page checks by record, access scope and reuse rights.
Read guideRoute curl_cffi requests through an authenticated proxy, check one observed exit and target response, and separate TLS impersonation from browser automation.

Use curl_cffi.Session with a proxy URL on each request, then validate an IP echo and an authorized target with that client. impersonate="chrome" selects a TLS and HTTP profile for the request; it does not run JavaScript or render a browser page. The echo records one observed exit, not an IP guarantee for later requests.
curl_cffi 0.16.3, the stable PyPI release tested on 2026-09-29Install the tested package in a virtual environment:
python -m pip install 'curl_cffi==0.16.3'
Load PROXY_URL from a runtime secret binding. Its form is http://<PROXY_USER>:<PROXY_PASSWORD>@<PROXY_HOST>:<PROXY_PORT>; percent-encode reserved characters. Do not store it in source control or pass it in a command-line argument. Set EXIT_CHECK_URL and TARGET_URL to override the defaults for a permitted test. If you change TARGET_URL, also replace the hard-coded "Example Domain" marker in the target validation with text expected from that page.
Warning: Proxy credentials can expose account access if they reach logs or source control, and requests can consume metered traffic. Use a secret binding and an authorized test target. Rotate credentials if exposed, then remove the logged or committed copy.
An HTTP proxy's Basic credentials are not encrypted between the client and proxy. HTTPS on the target protects the target connection after the proxy tunnel; it does not protect the proxy-authentication hop. Send credentials only over a trusted network path, or configure a TLS-protected proxy endpoint and verify its certificate.
Save this complete program as curl-cffi-probe.py:
import ipaddress
import json
import os
from urllib.parse import urlsplit
from curl_cffi import Session
PROXY_URL = os.environ["PROXY_URL"]
EXIT_CHECK_URL = os.environ.get("EXIT_CHECK_URL", "https://api.ipify.org?format=json")
TARGET_URL = os.environ.get("TARGET_URL", "https://example.com/")
TIMEOUT_SECONDS = 10
MAX_RESPONSE_BYTES = 65_536
def validate_proxy_url(proxy_url):
parsed = urlsplit(proxy_url)
if parsed.scheme != "http" or not parsed.hostname or not parsed.port:
raise ValueError("PROXY_URL must be an HTTP proxy URL with a host and port")
class ProbeFailure(Exception):
pass
def request_body(client, url, phase):
response = client.get(
url,
proxy=PROXY_URL,
impersonate="chrome",
timeout=TIMEOUT_SECONDS,
allow_redirects=False,
stream=True,
)
try:
if not 200 <= response.status_code < 300:
raise ProbeFailure(f"{phase} returned HTTP {response.status_code}")
chunks = []
size = 0
for chunk in response.iter_content(chunk_size=8192):
if not chunk:
continue
size += len(chunk)
if size > MAX_RESPONSE_BYTES:
raise ProbeFailure(f"{phase} response exceeded the byte limit")
chunks.append(chunk)
return b"".join(chunks)
finally:
response.close()
def main():
validate_proxy_url(PROXY_URL)
with Session() as client:
exit_body = request_body(client, EXIT_CHECK_URL, "exit check")
exit_ip = ipaddress.ip_address(json.loads(exit_body)["ip"])
target_body = request_body(client, TARGET_URL, "target").decode("utf-8", errors="replace")
if "Example Domain" not in target_body:
raise ProbeFailure("target content did not contain the expected marker")
print(json.dumps({"exit_ip": str(exit_ip), "target_check": "passed"}))
if __name__ == "__main__":
try:
main()
except ProbeFailure as error:
print(f"Probe failed: {error}")
raise SystemExit(1) from None
except Exception as error:
print(f"Probe failed ({type(error).__name__}); check proxy, status, and content separately.")
raise SystemExit(1) from None
Run from a directory where curl-cffi-probe.py already exists. Secrets must be injected through a secret manager or an untracked runtime binding. The program prints the observed IP and validation result but never prints the proxy URL or raw exception text.
python curl-cffi-probe.py
Expected output uses the IP returned by the configured echo service:
{"exit_ip":"<OBSERVED_EXIT_IP>","target_check":"passed"}
<OBSERVED_EXIT_IP> is output data, not an input placeholder. Each response is streamed and capped at 65,536 bytes. timeout=10 applies to a request call; it does not cap a whole worker's elapsed time.
The proxy URL in this example begins with http://, even though the target defaults use HTTPS. The proxy carries the HTTPS connection using its supported tunnel behavior, while target certificate validation remains enabled. impersonate="chrome" changes the HTTP client handshake profile supported by curl_cffi; it does not create a browser process, DOM, or JavaScript runtime.
| Observation | Boundary | Next check |
|---|---|---|
| Proxy URL validation fails | Local input | Check http scheme, host, port, and encoded credentials |
Connection error or 407 |
Proxy route or authentication | Confirm endpoint and account; 407 means proxy authentication is required under RFC 9110 |
| TLS error | Proxy handshake or target TLS | Separate the proxy transport from target TLS and keep certificate verification enabled |
| Non-2xx status | Proxy or target response | Record the status and request phase; inspect target policy before retrying |
| 2xx response without the marker | Content validation | Reject the record and inspect a redacted response sample |
Keep one session for a logical visit that depends on cookies. The session preserves client cookies and reuses connections, while IP persistence belongs to the proxy service's session contract. Two requests through one configured session can still leave through different exits if the service rotates per request.
For browser pages that require JavaScript, use the Playwright proxy guide. The HTTP 407 guide covers proxy authentication diagnostics.
Scale your workflows with the tools you already use. Rotating or sticky sessions up to 72 hours.
Compare seller APIs, licensed Amazon product data and proxy page checks by record, access scope and reuse rights.
Read guideChoose the right anti-detect browser proxy guide, map the shared verification steps, and keep account access separate from the network route.
Read guideOur picks for market research: ProxyLane for small, intermittent projects, plus options for regional tracking, monthly collection and managed extraction.
Read guide