Amazon Scraper API vs Proxy: Choose a Product Data Source
Distinguish authorized Amazon APIs, licensed product data and proxy-based page checks by record quality and access rights.
Read guideMatch an invoice to the same route and billing window, compare client byte observations with their limits, and explain variance without guessing the provider meter.
An invoice and a client byte counter answer different questions. The invoice follows a provider’s billing meter. The client sees the requests and responses that made it through that client path. Reconcile the invoice to the same window and route before you explain a difference. Keep client observation separate from the provider meter.
This is a post-run accounting procedure. It does not forecast traffic and it does not turn bandwidth into cost per accepted result.
Start with the invoice period, account or project, route identifier, package or rate, and the provider’s definition of the billed unit. Then join it to the client run by a stable job ID and UTC window.
| Record | Capture | Limit |
|---|---|---|
| Provider invoice | Billed unit, quantity, rate, period, adjustments, and route or product label | The provider’s meter is authoritative for the charge, but its boundary must be documented |
| HTTP client | Request and response headers, body bytes, redirects, retries, status, and route label | A library counter may omit connection overhead or traffic outside that client |
| Browser | transferSize, encodedBodySize, resource URL, resource type, cache state, and context |
Cross-origin timing can be zero; a field for one resource is not the whole page |
| Job log | Attempt ID, target ID, phase, status, bytes, and accepted or rejected output | Redaction and sampling can remove detail needed for a later join |
| Billing rule | Decimal or binary units, headers, retries, failed attempts, minimums, rounding, and exclusions | Never infer an absent rule from a close-looking byte total |
If you capture headers for reconciliation, measure the byte count of the complete observed request and response metadata before redaction. After that measurement, store only the count and the allowlisted non-secret fields or values needed for the join. Redact Authorization, Proxy-Authorization, Cookie, Set-Cookie, bearer tokens, API keys, and token-like query parameters. Do not export raw headers or full URLs just because the client can record them, and do not describe an allowlisted subset as the complete header byte count.
MDN documents transferSize as including response headers and body size for a resource, with cache and cross-origin limitations. encodedBodySize describes the payload before content decoding, so it can differ from decoded bytes. decodedBodySize describes the payload after decoding. Neither field is a provider invoice field.
For a single permitted HTTP request, capture the same route, target, client version, and retry policy that produced the invoice window. The control below assumes a no-auth route whose source IP is already allowlisted. It keeps credentials out of shell history and the log. Keep PROXY_URL credential-free. For an authenticated route, configure the client separately through its supported private credential configuration; do not put credentials in the proxy URL, command arguments, or output. Keep the output small and record status and timing without saving the page body.
set -eu
: "${PROXY_URL:?Set PROXY_URL without credentials}"
: "${TARGET_URL:?Set TARGET_URL to one permitted HTTPS URL}"
curl --disable --proxy "$PROXY_URL" --noproxy '' \
--connect-timeout 10 --max-time 30 \
--silent --show-error --output /dev/null \
--write-out 'job=example attempt=1 status=%{response_code} proxy_status=%{http_connect} download_bytes=%{size_download} header_bytes=%{size_header} total=%{time_total}s\n' \
"$TARGET_URL"
For browser work, collect resource timing from the same context and record whether the resource came from cache. Playwright exposes request and response events that can be joined to a run ID, but its events still describe browser traffic, not the provider’s billing definition. For Python or another HTTP library, record the library’s observed request and response sizes at the transport boundary you control.
Here is a synthetic reconciliation row, not a provider invoice: in one matching window, the initial attempts contributed 620,000,000 body bytes and 4,000,000 request and response header bytes. That header count covers the complete observed metadata before values were redacted; only the count and allowlisted non-secret fields were retained. All bytes from retry and redirect attempts contributed another 80,000,000 bytes. These categories are disjoint: keep retry and redirect bytes out of the initial-attempt figures before adding them. The instrumented total is 704,000,000 bytes, or 0.704 GB using decimal units. If the invoice says 0.740 GB, the residual is 36,000,000 bytes, or 0.036 GB. Ask which documented meter category explains that residual before changing the client calculation. Do not count the same retry bytes again under a second category.
| Observation | Plausible explanation to test | Evidence to request or collect |
|---|---|---|
| Invoice is larger than response-body sum | Headers, request bytes, retries, redirects, or other workers are included | Provider meter definition and attempt-level client log |
| Browser total is smaller than expected | Cache, cross-origin timing restrictions, or resources outside the captured context | Cache state, Timing-Allow-Origin, and browser request list |
| Client total is larger than invoice | The client counted decoded bytes or direct traffic, or the invoice excludes a class | Client counter definition and provider exclusions |
| Difference appears only at period edges | UTC boundary, rounding, or delayed invoice aggregation | Invoice timestamps and provider aggregation window |
| Same bytes, different invoice quantity | Unit conversion, minimums, or adjustments | Written unit and line-item breakdown |
Do not infer the provider's billing rule from a browser byte field. A close total is a useful lead, not a settled reconciliation. Preserve the original invoice, an immutable redacted run export, the calculation, and the provider’s answer as separate records.
The bandwidth estimation guide models future traffic; this guide starts with an actual invoice. The cost-per-successful-request guide uses a validated-result denominator, which is a different accounting question. For a disputed line, include the support packet.
When you are ready to discuss a defined traffic route and reconciliation window, Sign up to choose your workflow and traffic package during onboarding.
Distinguish authorized Amazon APIs, licensed product data and proxy-based page checks by record quality and access rights.
Read guideConnect a buyer-owned proxy to an Apify Actor, keep the session boundary clear, and validate records instead of counting requests.
Read guideSeparate Australian egress from en-AU content, AUD pricing, GST display, postcode validation and the state delivery context.
Read guide