Amazon Scraper API vs Proxy: Choose a Product Data Source
Compare seller APIs, licensed Amazon product data and proxy page checks by record, access scope and reuse rights.
Read guideCheck proxy reachability, the exit address used by one request, target acceptance, DNS routing, and browser WebRTC separately.

To diagnose a proxy accurately, test the intended request path with the same client and configuration: ask whether the proxy is reachable, observe the exit address through a trusted echo service, request the actual permitted target, then validate its content. Test browser WebRTC and DNS separately inside the browser. These checks answer different questions; none alone proves that a scraping job succeeded.
Audience: Python developers who can run a script and supply environment variables. Use Python 3.12.0; the local fixture was checked on macOS 26.5.2 arm64. The shell commands below cover a POSIX environment.
This example uses HTTPX 0.28.1 and a proxy URL provided through the PROXY_URL environment variable. It sends both requests through the same client object. Replace the example target with an authorized site and the echo URL with a trusted service. The echo service sees the request and its public source address, so do not send credentials, cookies, or sensitive query parameters to it.
Install the pinned client:
python3 -m venv .venv
. .venv/bin/activate
python -m pip install 'httpx==0.28.1'
Caution: These two checks can consume paid proxy traffic and disclose the request source to the echo service. Use a disposable, low-volume test route, keep credentials in a private runtime environment, and stop on authentication or content-validation failure. The client closes automatically; this script creates no persistent browser state.
Set PROXY_URL in a runtime secret manager or process environment; do not put it in a command argument, script, screenshot, or exported log. For an authenticated HTTP proxy, the value has the form http://<USERNAME>:<PASSWORD>@<HOST>:<PORT>. Here, <USERNAME> and <PASSWORD> are provider-issued credentials, <HOST> is the proxy hostname, and <PORT> is its port. URL-encode reserved characters in username and password. Keep this process environment private.
import json
import os
import ipaddress
import httpx
PROXY = os.environ["PROXY_URL"]
ECHO_URL = os.environ.get("DIAGNOSTIC_ECHO_URL", "https://api.ipify.org?format=json")
TARGET_URL = os.environ.get("DIAGNOSTIC_TARGET_URL", "https://books.toscrape.com/")
REQUEST_TIMEOUT = httpx.Timeout(20.0, connect=8.0)
MAX_BYTES = 256 * 1024
def read_bounded(response):
response.raise_for_status()
body = bytearray()
for chunk in response.iter_bytes():
if len(body) + len(chunk) > MAX_BYTES:
raise ValueError("Response exceeds the 256 KiB diagnostic limit")
body.extend(chunk)
return bytes(body)
try:
with httpx.Client(
proxy=PROXY,
trust_env=False,
follow_redirects=False,
timeout=REQUEST_TIMEOUT,
) as client:
with client.stream("GET", ECHO_URL) as response:
echo_body = read_bounded(response)
echo_value = json.loads(echo_body)["ip"]
exit_ip = str(ipaddress.ip_address(echo_value))
print("Exit address:", exit_ip)
with client.stream("GET", TARGET_URL) as response:
status = response.status_code
print("Target status:", status)
page_body = read_bounded(response)
if b"Books to Scrape" not in page_body:
raise ValueError("Expected page marker was absent")
print("Expected page marker found")
except (httpx.HTTPError, KeyError, ValueError, json.JSONDecodeError) as error:
# Print only the exception class; HTTPX messages can include proxy URLs.
print("Diagnostic failed:", type(error).__name__)
raise SystemExit(1) from None
Run the script from a file, for example python diagnose.py.
The connect timeout is 8 seconds; HTTPX's 20-second timeout applies to network-operation inactivity phases, not a strict end-to-end wall-clock deadline. Each response is capped at 256 KiB.
Set DIAGNOSTIC_ECHO_URL and DIAGNOSTIC_TARGET_URL to substitute services or a local test fixture. trust_env=False prevents inherited proxy environment variables from silently changing this HTTPX client's route.
Errors print only a class name because exception messages can contain proxy URLs; the target HTTP status is printed before HTTP error handling.
The exit observation must parse as an IP address and applies only to that echo request at that time.
A rotating proxy may select a different exit for the target request, even though both calls use the same client configuration. Record the address and timestamp; do not infer that every later request shares it.
For reference, this is the sanitized output from the local fixture used to exercise the example. 198.51.100.8 is a documentation-only test address, not a real proxy exit:
Exit address: 198.51.100.8
Target status: 200
Expected page marker found
| Observation | What it establishes | What it does not establish |
|---|---|---|
| Connection or proxy-authentication error | The client could not complete the requested path; the exception and proxy response narrow the cause | Whether the target would accept a successful request |
| Echo request returns an address | The echo service observed an address for that request | Target acceptance, future sticky behavior, or provider billing |
| Target returns an HTTP status | The target returned that status for this request | That the page contains usable or complete records |
| Expected content validation passes | The requested marker or fields were present | That every page, field, or later request succeeds |
| Reputation service reports a listing | That service's list contained the address at check time | A universal risk or fraud verdict |
Do not treat 200 OK as extraction success. Parse the fields the job needs and fail the run when they are absent or changed. A challenge page can be a successful HTTP response while still being unusable output.
A terminal request cannot establish how a browser profile routes traffic. Run a benign IP echo in the same browser context, then load the permitted target and verify its expected page state. Check the browser's configured proxy, extensions, service workers, DNS mode, and profile when terminal and browser results differ.
Use a WebRTC leak test inside that browser and configuration when the application uses WebRTC. A different media-network address is a reason to investigate that path; by itself it does not show that HTTP requests to the target bypassed the proxy. SOCKS remote-DNS behavior is also specific to the client and protocol configuration; it does not govern every operating-system lookup. See the HTTP versus SOCKS guide for those distinctions.
For a repeatable batch probe, retain proxy ID, timestamp, protocol, endpoint label, exit observation, proxy/tunnel result, target status, and content-validation result. Store only non-secret endpoint labels in reports. The Proxy Checker checks proxy reachability and target response but does not report an observed exit address; the CLI walkthrough explains its exact output. Use a separate trusted echo source, reputation list, or in-browser WebRTC test for those specialized questions.
| Symptom | Next check |
|---|---|
| Proxy authentication error | Confirm scheme, host, port, and encoded credentials in the secret store; do not paste the full URL into logs |
| Connect timeout | Check the endpoint, local network policy, and proxy availability; compare with the documented timeout |
| Echo works, target fails | Inspect target status and body; check authorization, rate limits, and target-specific access rules |
| Target status is success but records are empty | Validate required selectors/fields and stop instead of saving incomplete data |
| Terminal works, browser fails | Repeat the echo in that profile and inspect browser proxy, DNS, extension, and service-worker settings |
| IP reputation result looks alarming | Identify the exact list, timestamp, and listing reason; do not generalize to target acceptance |
Scope: the example demonstrates client-side routing and response checks. It does not measure provider-billed traffic, certify anonymity, predict anti-bot outcomes, or establish behavior for a different application or browser profile.
Scale your workflows with the tools you already use. Rotating or sticky sessions up to 72 hours.
Compare seller APIs, licensed Amazon product data and proxy page checks by record, access scope and reuse rights.
Read guideChoose the right anti-detect browser proxy guide, map the shared verification steps, and keep account access separate from the network route.
Read guideOur picks for market research: ProxyLane for small, intermittent projects, plus options for regional tracking, monthly collection and managed extraction.
Read guide