Amazon Scraper API vs Proxy: Choose a Product Data Source
Compare seller APIs, licensed Amazon product data and proxy page checks by record, access scope and reuse rights.
Read guideMonitor a permitted proxy workflow at four points: route reachability, target response, validated output, and queue completion. Assign each check an owner and a bounded recovery rule. A successful connection alone does not show that the crawler produced a valid record.
Separate the observation layers
Use a harmless route probe for transport, a permitted target request for site behavior, and the production parser's validation rule for useful output. Keep queue freshness separate so completed work cannot hide items that remain due or failed.
The route check can cover Domain Name System (DNS) resolution and Transport Layer Security (TLS) negotiation. Record Hypertext Transfer Protocol (HTTP) status separately from the output validator.
| Layer | Record | First owner |
|---|---|---|
| Route | DNS, connect, TLS, response timing, and route label | Network or proxy account owner |
| Proxy authentication | Whether the proxy challenged the client with HTTP 407 | Proxy account owner |
| Target | Status, final host, response category, and rate-limit headers | Target-policy owner |
| Session | Client session or profile label and continuity result | Crawler or browser operator |
| Output | Required identifier, fields, and content identity | Parser or data owner |
| Queue | Due, completed, pending, and terminally failed logical jobs | Scheduler or operations owner |
Request for Comments (RFC) 9110 defines HTTP 407 as a proxy-authentication challenge and HTTP 403 as a refusal by the responding server. RFC 6585 defines HTTP 429 for too many requests; the response may include Retry-After. Preserve response source and headers before assigning the incident to a layer.
A 200 response still needs the application's content check. A fast response from an unrelated endpoint shows only that probe's result, not that the business target works. Keep expected fields and validation outcomes in the same record as the target request so the operator can distinguish transport from useful output.
Record one redacted observation per attempt
Caution: Recovery rules can interrupt active jobs, repeat writes, or expose retained request data. Test them on a small non-production queue with redacted records first. In production, keep an owner, a total retry and spend cap, and a saved configuration to restore; pause automation if validation or limits fail.
Give every attempt a non-secret identifier and a Coordinated Universal Time (UTC) timestamp. Keep the target label and route label stable across observations, while storing full URLs, cookies, credentials, and page bodies only in access-controlled systems when retention is necessary.
The following schema is an illustrative event shape, not captured telemetry. Replace its placeholders with actual observations; do not treat the sample as a passing or failing run. Record Autonomous System Number (ASN), SOCKS version 5 behavior, session time to live (TTL), and the job to be done (JTBD) only when they help diagnose the incident.
{
"task_id": "catalog-refresh-example",
"attempt_id": "attempt-example",
"observed_at": "<UTC_TIMESTAMP>",
"route_id": "<NON_SECRET_ROUTE_LABEL>",
"target_id": "<APPROVED_TARGET_LABEL>",
"requested_country": "<COUNTRY>",
"observed_country": "<COUNTRY_OR_UNKNOWN>",
"stage": "<CONNECT|AUTH|TARGET|VALIDATE>",
"http_status": null,
"latency_ms": null,
"bytes": null,
"valid_output": null,
"reason": "<BOUNDED_FAILURE_CATEGORY>"
}
Keep one record per attempt and a separate key for each accepted logical record. That makes retries visible without counting the same accepted item twice. Measure accepted jobs against jobs due in the reporting window; show attempts, timeouts, transferred bytes, and invalid-output reasons beside that measure. Define what bytes counts; request or response bytes are not a proxy-billing measure unless provider terms say so.
Set recovery limits before an incident
Define the observation window, minimum sample, failure threshold, cooldown, retry cap, and spend or byte cap before enabling automatic recovery. Set these values from the target's allowed schedule, job deadline, and traffic budget; this playbook supplies no universal threshold.
| Observation | Bounded action | Resume condition | Owner |
|---|---|---|---|
| Route probe fails repeatedly | Stop new assignments to that route and cap diagnostic probes | Route check passes, then an approved target and output check pass | Network owner |
| Proxy authentication fails after a secret change | Pause affected workers; do not cycle routes to hide the error | Same client authenticates with the authorized credential source | Proxy account owner |
Target returns 429 |
Pause requests to that target for the permitted wait or configured cooldown | Target policy allows the next request | Target-policy owner |
| Target responds but required fields disappear | Stop saving affected output | Parser and page-state validation pass on an approved sample | Data owner |
| Requested and observed exit differ | Hold the assignment and investigate route configuration | Route requirement and target check agree | Network owner |
| Session changes mid-task | Hold or restart the logical job within its existing retry and deadline budget | Client state is restored or the job restarts safely | Crawler operator |
An open circuit stops ordinary assignments; a half-open check admits only the bounded trial traffic defined by the operator. If that trial fails, reopen the circuit. A fallback must keep the job's total retry, byte, and deadline limits. Changing routes does not establish that a parser, account, or target-policy problem is fixed.
Client session lifecycle is a separate control from route health. For example, Crawlee's versioned SessionPool reference documents retiring sessions; that action addresses a crawler session, not the state of an upstream proxy. Check Crawlee's scaling guidance for client concurrency and queue settings before attributing low throughput to the route.
Alert on missing valid output
Alert on the age of the last valid result and the share of due logical jobs that remain incomplete, with timeout count beside latency. Excluding timed-out jobs makes a failing path appear faster; counting attempts as completed jobs can hide duplicates and missing work.
Include the target and route labels, last valid timestamp, failing stage, bounded sample window, and requested operator action. Prometheus Alertmanager can group related notifications so one shared incident does not page once for every affected record; group on labels that identify a common route, target, or phase.
After recovery, reconcile the queue into completed, pending, and terminally failed work. Re-run the original validator on the recovered sample and check for duplicate accepted records. Record the recovery timestamp and the support response separately from measured job output.
The proxy checker can help with a manual route check, but it does not replace target validation. For incident escalation, use the proxy support ticket template; for cost analysis, use the cost-per-successful-request guide.
Sources and further reading
ProxyLaneTraffic never expiresPremium residential proxies
From $2/GB
Scale your workflows with the tools you already use. Rotating or sticky sessions up to 72 hours.
