Amazon Scraper API vs Proxy: Choose a Product Data Source
Distinguish authorized Amazon APIs, licensed product data and proxy-based page checks by record quality and access rights.
Read guideA proxy timeout is a phase question. The client may be waiting for DNS, the proxy socket, a TLS handshake, the destination's first byte, or the rest of the response. Changing the proxy after every timeout hides that distinction and makes the next run harder to compare. The probe is evidence about one request, not a provider SLA.
Keep one total deadline, record the phase that consumed it, and test the same permitted URL with the same client settings.
Map the timeout to a phase
For an HTTP proxy reaching an HTTPS URL, the route usually has these boundaries:
| Phase | What the client is waiting for | Useful evidence | First check |
|---|---|---|---|
| Name lookup | An address for the proxy or destination | time_namelookup |
Hostname, resolver, and whether DNS is local or delegated |
| Proxy connection | TCP connection to the proxy | time_connect |
Host, port, firewall, and proxy scheme |
| TLS handshake | TLS to the proxy or destination | time_appconnect |
Certificate trust, hostname, and TLS endpoint |
| Proxy authentication | A proxy response to credentials | HTTP 407 |
Credential method and endpoint; see the 407 guide |
| Target wait | Time after client and proxy negotiation before the first byte | time_starttransfer - time_pretransfer when both are available |
Target policy, rate limit, application queue, or slow origin |
| Body read | Remaining response bytes for one transfer | time_total minus time_starttransfer |
Response size, stalled stream, and client read deadline |
The exact phase depends on the scheme. With an HTTPS proxy, the first TLS handshake is to the proxy. With an HTTP proxy and an HTTPS target, the proxy CONNECT happens before the destination's TLS handshake. A SOCKS route does not produce an HTTP CONNECT status. curl also supports SSL and QUIC handshakes, as well as SOCKS4 and SOCKS5 schemes. RFC 9110 and RFC 6585 describe HTTP status boundaries; 200 and 429 still need the correct responding layer. Assign each field to the endpoint that actually answered it.
Run one bounded probe
Use a harmless, permitted URL and keep credentials out of the command. --connect-timeout limits the connection phase; --max-time bounds the whole transfer. They solve different problems.
This example assumes an IP-allowlisted or no-auth route. If the endpoint requires username and password, provide them through the client’s private credential store or secret binding supported by your runtime; do not put them in PROXY_URL, shell history, or logs.
set -eu
: "${PROXY_URL:?Set PROXY_URL to scheme://host:port without credentials}"
: "${TARGET_URL:?Set TARGET_URL to one permitted HTTPS URL}"
curl --disable --proxy "$PROXY_URL" --noproxy '' \
--connect-timeout 10 --max-time 30 \
--silent --show-error --output /dev/null \
--write-out 'status=%{response_code} proxy_status=%{http_connect} dns=%{time_namelookup}s connect=%{time_connect}s appconnect=%{time_appconnect}s pretransfer=%{time_pretransfer}s starttransfer=%{time_starttransfer}s total=%{time_total}s\n' \
"$TARGET_URL"
The values are cumulative timings from curl's start. When both fields are available, time_starttransfer - time_pretransfer is the post-negotiation wait; a large difference can point to target work or policy. If a phase value is unavailable or zero for the client and route, report it as unavailable instead of inferring the broken hop. A proxy_status=407 identifies proxy authentication. A failure before a status can still be a DNS, socket, TLS, or protocol problem; keep the client error text with the timing line.
Do not increase --max-time until you know which phase is slow. A longer deadline can turn a clear connection failure into a queue of workers waiting on the same broken route. Keep the deadline in the job's own budget and count each retry against that budget.
Read browser timeouts separately
Playwright has operation and navigation timeouts. The timeout belongs to the operation being awaited, so a page.goto() timeout does not by itself identify whether the proxy, target, page script, or readiness condition consumed the time. Record the URL, the operation, and the last request or response event alongside the error.
| Browser observation | What it proves | What it does not prove |
|---|---|---|
| Navigation never commits | The navigation did not reach the chosen readiness point | Which network hop failed |
| Request failed before a response | The client saw a request failure | That the proxy was the only failing component |
| Response arrived but the assertion timed out | The page did not reach the required state | That transport was slow |
| One asset is pending | That resource has not completed | That the main document or proxy is unavailable |
Use the smallest reproduction that still shows the failure: one context, one target, one route, and one awaited operation. A clean browser profile can remove extension and cache noise, but it cannot turn a browser timeout into a provider measurement.
Choose the next check from the evidence
| Result | Next action |
|---|---|
| Name lookup is the last completed phase | Check the proxy hostname, resolver, and local network policy |
| Connect time approaches the connection deadline | Check endpoint reachability, port, scheme, and firewall |
| TLS fails before an HTTP status | Follow the certificate trust guide |
407 appears quickly |
Check proxy authentication; do not retry a bad credential blindly |
| CONNECT succeeds, first byte is late | Test the permitted target and its rate-limit or queue response |
| First byte arrives, body stalls | Bound the body read, capture response size, and test whether the target stream is incomplete |
| Curl succeeds but the browser fails | Compare browser proxy rules, extensions, DNS mode, and the operation timeout |
The tunnel failure guide is useful when the browser reports ERR_TUNNEL_CONNECTION_FAILED. The monitoring playbook is for repeated workload evidence; this guide is for isolating one timeout before you automate recovery.
When you need a route for a defined client, target geography, and traffic window, Sign up to choose your workflow and traffic package during onboarding.
The CTA is a request for access, not a claim about immediate credentials or a response SLA.