Amazon Scraper API vs Proxy: Choose a Product Data Source
Distinguish authorized Amazon APIs, licensed product data and proxy-based page checks by record quality and access rights.
Read guide“Certificate error” does not identify the certificate that failed. A proxied HTTPS request can have one TLS leg to an HTTPS proxy and another TLS leg through an HTTP CONNECT tunnel to the destination. TLS trust is part of the route. The safe fix is to repair trust or the endpoint name, then keep verification enabled.
Do not use --insecure, -k, --proxy-insecure, or an equivalent setting as the fix. Those options can make a broken route appear healthy while removing the identity check that protects the connection.
Identify the TLS leg first
| Route | Certificate being checked | Trust control to inspect |
|---|---|---|
| HTTP proxy → HTTPS destination | Destination certificate after CONNECT | Destination CA store and target hostname |
| HTTPS proxy → HTTPS destination | HTTPS proxy certificate first, destination certificate after the tunnel | Proxy CA store, then destination CA store |
| SOCKS5 proxy → HTTPS destination | Destination certificate after the SOCKS exchange | Destination CA store and target hostname |
| Direct HTTPS control | Destination certificate | The same destination CA store without the proxy |
An HTTP 407 is proxy authentication, not a certificate-chain diagnosis. A TLS failure before any proxy status can be the HTTPS proxy leg. A tunnel that opens before a TLS error can be failing at the destination leg. Capture the endpoint scheme and the phase before changing a CA file.
Check the endpoint name and chain
Certificate verification has two separate questions:
- Does a trusted CA path validate the presented chain?
- Does the certificate identify the hostname the client requested?
TLS 1.3 and RFC 5280 provide the standards context for certificate messages and X.509 path validation. The X.509 format name is not a claim that a chain is trusted.
The hostname matters even when the chain is signed by a trusted CA. A certificate for the proxy's internal name does not automatically identify the public destination, and a destination certificate does not identify an HTTPS proxy endpoint.
Use curl's verbose output for one permitted host and redact it before sharing. It can show which connection is being negotiated, while --cacert supplies a destination CA file and --proxy-cacert supplies a separate CA file for an HTTPS proxy.
This example assumes an IP-allowlisted or no-auth route. If the endpoint requires username and password, provide them through the client’s private credential store or secret binding supported by your runtime; do not put them in PROXY_URL, shell history, or logs.
The command below is for an HTTPS proxy with two existing PEM trust files: one for the destination and one for the proxy. If both legs use the normal system store, omit the two CA options and their variables instead of inventing paths.
set -eu
: "${PROXY_URL:?Set PROXY_URL without credentials}"
: "${TARGET_URL:?Set TARGET_URL to one permitted HTTPS URL}"
: "${TARGET_CA_FILE:?Set TARGET_CA_FILE to an existing PEM CA file}"
: "${PROXY_CA_FILE:?Set PROXY_CA_FILE to an existing PEM CA file}"
curl --disable --proxy "$PROXY_URL" --noproxy '' \
--connect-timeout 10 --max-time 30 \
--cacert "$TARGET_CA_FILE" \
--proxy-cacert "$PROXY_CA_FILE" \
--silent --show-error --output /dev/null \
--write-out 'target=%{response_code} proxy=%{http_connect}\n' \
"$TARGET_URL"
Use --proxy-cacert only when the proxy URL is HTTPS and the proxy's CA is different from the destination trust. If both legs use the normal system store, omit the custom options and inspect the store selected by the client. Never copy a real credential-bearing URL into a ticket or shell history.
Python's ssl.create_default_context() creates a client context with certificate validation and hostname checking enabled. PROTOCOL_TLS_CLIENT also sets CERT_REQUIRED and check_hostname=True; inspect ssl.get_default_verify_paths() when the expected CA file is not being used. A custom internal CA should be added to the intended client trust store, with its ownership and rotation documented.
import ssl
context = ssl.create_default_context()
print({
"verify_mode": context.verify_mode.name,
"check_hostname": context.check_hostname,
"default_verify_paths": ssl.get_default_verify_paths(),
})
This snippet inspects the context; it does not make a proxy request. Keep the application’s actual proxy adapter and trust configuration in the reproduction so a successful direct connection is not mistaken for a successful proxied route.
Match the error to the repair
| Evidence | Likely boundary | Repair to test with verification on |
|---|---|---|
| Unknown issuer or unable to get local issuer | CA store lacks a required trust anchor or intermediate | Use the correct managed CA store or a reviewed --cacert/context bundle |
| Hostname mismatch | URL name does not match the certificate identity | Use the intended endpoint name and check the proxy URL separately |
| Expired or not-yet-valid certificate | Clock or certificate validity window | Check the system clock and endpoint certificate lifecycle |
| HTTPS proxy fails before target negotiation | Proxy TLS trust or proxy name | Check --proxy-cacert, proxy hostname, and HTTPS-proxy support |
| Direct control works, proxied route fails | The proxy leg changes the endpoint or trust path | Compare the proxy scheme, certificate presented, and CA store |
Proxy returns 407 or origin returns 403 after trust passes |
HTTP authentication or target policy | Follow the responding layer; do not replace it with a CA change |
TLS certificate authentication is part of the handshake, not decoration. Keep the failing phase, endpoint name, client version, CA-source choice, and redacted error for a reproducible handoff. The support ticket template turns those fields into a compact packet.
If you need a route for an existing client and a defined permitted target, Sign up and include the TLS leg you need to validate.