Amazon Scraper API vs Proxy: Choose a Product Data Source
Compare seller APIs, licensed Amazon product data and proxy page checks by record, access scope and reuse rights.
Read guideRoute curl through an HTTP or SOCKS proxy, authenticate safely, and separate CONNECT results from target responses.

Quick answer: route a request through an HTTP proxy with curl's --proxy, keep proxy authentication separate from target credentials, and validate the status and body. For HTTPS, connect=200 confirms CONNECT only; accept the result after the target returns the expected content.
The commands default to https://example.com/ and the marker Example Domain. Set TARGET_URL and EXPECTED_MARKER to test another permitted resource. PROXY_URL contains only the proxy scheme, host, and port. Use the same configured curl command for an egress-IP check, then inspect the application target body.
Warning: Keep the probe to one permitted proxy endpoint and target. Send one bounded request to a permitted test target, then confirm its status and body before enabling production jobs. With proxy Basic authentication, a plain HTTP proxy sends the login over the client-to-proxy hop without TLS. CONNECT only asks the proxy to open a tunnel; TLS to the target encrypts application traffic carried inside that tunnel, not the earlier proxy login. Use an https:// proxy endpoint when both client and provider support TLS to the proxy, and verify its certificate. Keep credentials in a secret manager or runtime binding; rotate them if exposed and check account usage after unexpected traffic.
| Supplied endpoint | curl setting | Domain Name System (DNS) location | What it tests |
|---|---|---|---|
| HTTP proxy | --proxy http://proxy.example:3128 |
Proxy for HTTPS CONNECT | HTTP proxy handshake followed by target TLS |
| HTTPS proxy | --proxy https://proxy.example:3128 |
Proxy | TLS to the proxy, then the request path |
| SOCKS5, client DNS | --proxy socks5://proxy.example:1080 |
Client | SOCKS transport with local name resolution |
| SOCKS5, proxy DNS | --proxy socks5h://proxy.example:1080 |
Proxy | SOCKS transport with proxy-side name resolution |
proxy.example is a documentation-only host name; replace it with the assigned proxy host and port. The curl manual defines these proxy URL prefixes, including http:// and https://. An HTTP proxy can carry an HTTPS destination through a CONNECT tunnel, so an HTTP proxy scheme for the proxy and an HTTPS scheme for the target is a normal combination. Do not change a SOCKS endpoint to an HTTP scheme to make a client accept it; the handshake and failure signals change with the protocol.
Set PROXY_URL to the HTTP proxy endpoint, such as http://proxy.example:3128. This unauthenticated example accepts target 200 OK only after checking the response marker. The command uses a 10-second connection limit and a 25-second total request limit; see the proxy connection timeout guide for how to choose those bounds:
: "${PROXY_URL:?Set PROXY_URL to your trusted proxy host and port}"
: "${EXPECTED_MARKER:=Example Domain}"
body_file=$(mktemp)
trap 'rm -f "$body_file"' EXIT
result=$(curl --disable \
--proxy "$PROXY_URL" --noproxy '' \
--connect-timeout 10 --max-time 25 \
--silent --show-error --output "$body_file" \
--write-out 'target=%{response_code} connect=%{http_connect}\n' \
"${TARGET_URL:-https://example.com/}") || exit $?
printf '%s\n' "$result"
target_status=${result#target=}
target_status=${target_status%% *}
test "$target_status" = 200 || { printf '%s\n' 'target status was not accepted' >&2; exit 1; }
grep -Fq -- "$EXPECTED_MARKER" "$body_file" || { printf '%s\n' 'expected target content was absent' >&2; exit 1; }
printf '%s\n' 'target marker present'
Expected output for an HTTPS request through an HTTP proxy:
target=200 connect=200
target marker present
--proxy (or -x) selects the route. --disable must be curl's first argument; it prevents a local .curlrc from changing this diagnostic request. --noproxy '' overrides a NO_PROXY exclusion for this fixed test. That is useful for a controlled probe; remove it when local or sensitive destinations are supposed to bypass the proxy.
curl's proxy environment documentation describes http_proxy, HTTPS_PROXY, ALL_PROXY, and NO_PROXY. An explicit proxy option takes precedence over proxy-selection variables, while a matching NO_PROXY entry can still bypass the named host; set --noproxy '' only for this controlled probe.
The output reports the final target status and the last proxy CONNECT response code. With an HTTPS target through an HTTP proxy, connect=200 means the tunnel was accepted. target=200 means the target returned that status. An HTTP request that does not use CONNECT can show connect=000; SOCKS is a different protocol path. These values are observations, not proof of egress identity or useful target content.
For a proxy that uses username and password authentication, let curl prompt for the password. Set only the username in the shell environment. The status from curl and the saved body must both pass acceptance checks:
: "${PROXY_URL:?Set PROXY_URL to your trusted proxy host and port}"
: "${PROXY_USER:?Set PROXY_USER for this proxy}"
body_file=$(mktemp)
trap 'rm -f "$body_file"' EXIT
case "$PROXY_USER" in
*:*) printf '%s\n' 'This prompt example requires a username without a colon' >&2; exit 1 ;;
esac
result=$(curl --disable \
--proxy "$PROXY_URL" --proxy-user "$PROXY_USER" --noproxy '' \
--connect-timeout 10 --max-time 25 \
--silent --show-error --output "$body_file" \
--write-out 'target=%{response_code} connect=%{http_connect}\n' \
"${TARGET_URL:-https://example.com/}") || exit $?
printf '%s\n' "$result"
target_status=${result#target=}
target_status=${target_status%% *}
test "$target_status" = 200 || { printf '%s\n' 'target status was not accepted' >&2; exit 1; }
grep -Fq -- "${EXPECTED_MARKER:-Example Domain}" "$body_file" || { printf '%s\n' 'expected target content was absent' >&2; exit 1; }
printf '%s\n' 'target marker present'
Expected output for an HTTPS request through an HTTP proxy:
target=200 connect=200
target marker present
The curl HTTP scripting documentation says that a user/password option with the password omitted prompts interactively. curl splits --proxy-user at a colon, so the guard prevents a colon in PROXY_USER from being mistaken for an inline password. Use --proxy-user for the intermediary; --user configures credentials for the origin server.
Avoid a credential-bearing proxy URL or --proxy-user user:password in a shared shell. The curl manual notes that command-line passwords can briefly appear in process listings. For unattended jobs, use the deployment's secret injection or a reviewed curl config with strict file permissions and suppressed command echo. Do not print the proxy URL, --verbose transcript, or trace into CI logs. After an interactive probe, remove only task-scoped shell values with unset PROXY_URL PROXY_USER.
curl uses Basic proxy authentication by default. --proxy-anyauth can negotiate a supported method but may add a challenge round trip. Basic credentials sent to a plain HTTP proxy are visible on the client-to-proxy connection, so use a trusted HTTPS proxy when the provider supports it and verify that proxy's certificate. The HTTPS target's TLS does not encrypt a separate plain HTTP proxy login.
| Phase | Signal | Next check |
|---|---|---|
| Local configuration | Unexpected direct result, or an option changed by a machine profile | Keep --disable, set --proxy explicitly, inspect NO_PROXY, and verify the target is not a local bypass |
| Proxy connection | Could not resolve host, connection refused, or connect timeout | Check endpoint spelling, port, firewall policy, and reachability from the machine running curl |
| Proxy authentication | 407 Proxy Authentication Required or a proxy auth failure before target headers |
Confirm account, scheme, username, and the proxy's Proxy-Authenticate challenge |
| Tunnel or TLS | connect=200 followed by certificate or handshake failure |
Separate TLS to a TLS-protected proxy from target TLS through an HTTP CONNECT tunnel; keep certificate verification enabled |
| Target response | 401, 403, a rate-limit response, or a redirect |
Diagnose target credentials, target policy, rate limits, or the allowed redirect boundary; changing the proxy is not the first fix |
| Useful output | 200 with the wrong page or no expected field |
Validate the body or parsed record and record the failure as content, not transport success |
For HTTPS through an HTTP proxy, curl first asks the proxy to establish a tunnel to the target's port 443. If the proxy returns 407, the tunnel was rejected before curl received the target page. RFC 9110 requires a proxy-generated 407 to include a Proxy-Authenticate challenge. A target 401 concerns target credentials; a target 403 may reflect target policy or a denied egress IP. A 429 rate-limit response means the target is asking the client to slow down; honor Retry-After when supplied, and do not rotate exits to evade the limit. Repeating a 407 with an origin Authorization header does not address the intermediary.
The examples deliberately do not follow redirects: inspect the Location value and decide whether the destination is permitted before making a second request. Do not use --location-trusted; it broadens credential forwarding across redirects. If a trace is necessary, use a controlled environment, redact it before storage, and keep it out of shared transcripts.
A matched connection in proxy logs and a response from the fixed target establish that the proxy handled a request. To check the exit Internet Protocol (IP) address, send a fixed request to an IP-check endpoint operated by the team or explicitly trusted. Do not include target credentials or query secrets. Confirm that the endpoint does not redirect to another host. Compare the returned address with the provider-assigned exit, then test the intended target. A route can pass an IP check while the target returns a policy page or a locale different from the one the parser expects.
For a repeated application call, move from the shell probe to a client with an explicit request policy. The Python Requests guide explains environment precedence and timeout handling. The Playwright guide covers browser contexts and page subresources. Configure and check browser or crawler clients separately; their proxy settings and authentication boundaries differ from curl's.
Scale your workflows with the tools you already use. Rotating or sticky sessions up to 72 hours.
Compare seller APIs, licensed Amazon product data and proxy page checks by record, access scope and reuse rights.
Read guideChoose the right anti-detect browser proxy guide, map the shared verification steps, and keep account access separate from the network route.
Read guideOur picks for market research: ProxyLane for small, intermittent projects, plus options for regional tracking, monthly collection and managed extraction.
Read guide