Amazon Scraper API vs Proxy: Choose a Product Data Source
Compare seller APIs, licensed Amazon product data and proxy page checks by record, access scope and reuse rights.
Read guideSet HTTP or SOCKS proxies for Playwright browsers and contexts. Add authentication, check exit IP, and diagnose connection failures.

Set Playwright's proxy option at browser launch to share one route across contexts, or on browser.newContext() to scope it to one context. Keep proxy credentials in those fields, then verify target output separately from cookies and observed egress; a successful page response alone does not prove which exit carried it.
Warning: The route allows up to 40 target requests plus one IP-check request and can consume paid traffic. Use a test allocation and authorized target, check quota first, and stop if usage is unexpected. Configure production workers separately.
The example uses Playwright's documented proxy options with the fixed target https://example.com/. Its route allows only that origin, blocks image, font, and media requests, caps request count, and checks reported response sizes after navigation. Adapt the policy to the permitted page; declared bytes are not a universal transfer quota.
Playwright documents two useful scopes: a proxy at browser launch for every context in that browser, or a proxy on an individual browser context. A context is usually the clearer boundary when tests need different routes or isolated cookies. Set the proxy at launch when the entire browser run should share one route.
The server value names the proxy, not the destination page. http://proxy.example:3128 can carry an HTTPS destination through an HTTP CONNECT tunnel. An https:// proxy URL means the client uses TLS to the proxy when the browser and endpoint support it. The HTTPS target URL alone does not encrypt a separate plain HTTP connection to the proxy.
Playwright also documents socks5: servers. Its username and password examples apply to HTTP(S) proxy authentication. Check the provider's SOCKS authentication documentation before assuming those fields work for a SOCKS endpoint. The fixture below rejects that assumption rather than sending credentials through an unverified path.
browser.newContext() creates an isolated, non-persistent context. This example discards its state when it closes; it does not create a saved browser profile. For authorized jobs that must reuse login state across runs, follow Playwright’s authentication-state workflow and protect saved state as a credential.
Install the package and Chromium in the Node.js project:
npm install --save-dev 'playwright@1.63.0'
npx playwright install chromium
Set PROXY_SERVER to an endpoint you trust, such as http://proxy.example:3128. Set PROXY_USER and PROXY_PASS only when that HTTP(S) endpoint requires proxy credentials. Supply both values through the runtime secret manager; do not put them in source control, a browser URL, a page header, a screenshot, or a shared trace.
Save this as proxy-check.mjs:
import { chromium } from 'playwright';
import { isIP } from 'node:net';
const TARGET_URL = 'https://example.com/';
const IP_CHECK_URL = 'https://api.ipify.org/?format=json';
const TARGET_ORIGIN = new URL(TARGET_URL).origin;
const MAX_REQUESTS = 40;
const DECLARED_BYTES_REVIEW_THRESHOLD = 2_000_000;
const BLOCKED_RESOURCE_TYPES = new Set(['image', 'font', 'media']);
const server = process.env.PROXY_SERVER;
let parsed;
try {
parsed = new URL(server);
} catch {
throw new Error('Set PROXY_SERVER to a trusted proxy host and port');
}
if (
!['http:', 'https:', 'socks5:'].includes(parsed.protocol) ||
!parsed.hostname ||
!parsed.port ||
parsed.username ||
parsed.password ||
parsed.pathname !== '/' ||
parsed.search ||
parsed.hash
) {
throw new Error('Use a supported proxy host and port without URL credentials or a path');
}
const proxy = { server };
const hasUser = Boolean(process.env.PROXY_USER);
const hasPass = Boolean(process.env.PROXY_PASS);
if (parsed.protocol === 'socks5:' && (hasUser || hasPass)) {
throw new Error('Confirm SOCKS5 authentication support before supplying credentials');
}
if (parsed.protocol !== 'socks5:' && (hasUser || hasPass)) {
if (!hasUser || !hasPass) {
throw new Error('Set both PROXY_USER and PROXY_PASS');
}
proxy.username = process.env.PROXY_USER;
proxy.password = process.env.PROXY_PASS;
}
const browser = await chromium.launch();
let context;
let exitContext;
let phase = 'target context setup';
try {
context = await browser.newContext({ proxy, serviceWorkers: 'block' });
let requestCount = 0;
let declaredBytes = 0;
const failedRequests = [];
await context.route('**/*', async route => {
const request = route.request();
const requested = new URL(request.url());
if (
requested.origin !== TARGET_ORIGIN ||
BLOCKED_RESOURCE_TYPES.has(request.resourceType())
) {
await route.abort();
return;
}
requestCount += 1;
if (requestCount > MAX_REQUESTS) {
await route.abort();
return;
}
await route.continue();
});
const page = await context.newPage();
page.on('response', response => {
const contentLength = Number(response.headers()['content-length']);
if (Number.isFinite(contentLength)) {
declaredBytes += contentLength;
}
});
page.on('requestfailed', request => {
failedRequests.push({
origin: new URL(request.url()).origin,
error: request.failure()?.errorText ?? 'unknown',
});
});
phase = 'target navigation';
const response = await page.goto(TARGET_URL, {
waitUntil: 'domcontentloaded',
timeout: 15000,
});
phase = 'target validation';
if (!response) throw new Error('Navigation returned no response');
if (new URL(page.url()).origin !== TARGET_ORIGIN) {
throw new Error('The page redirected outside the test target');
}
if (!response.ok() || (await page.title()) !== 'Example Domain') {
throw new Error('Target returned an unexpected response');
}
if (declaredBytes > DECLARED_BYTES_REVIEW_THRESHOLD) {
throw new Error('Declared response sizes exceeded the post-run review threshold');
}
phase = 'isolated exit check';
exitContext = await browser.newContext({ proxy, serviceWorkers: 'block' });
let exitRequestSent = false;
await exitContext.route('**/*', async route => {
if (exitRequestSent || route.request().url() !== IP_CHECK_URL) {
await route.abort();
return;
}
exitRequestSent = true;
await route.continue();
});
const exitPage = await exitContext.newPage();
const exitResponse = await exitPage.goto(IP_CHECK_URL, { timeout: 15000 });
if (!exitResponse?.ok() || exitResponse.request().redirectedFrom() ||
exitPage.url() !== IP_CHECK_URL) {
throw new Error('Exit check failed or redirected');
}
const { ip } = await exitResponse.json();
if (typeof ip !== 'string' || !isIP(ip)) {
throw new Error('Exit check returned an invalid IP address');
}
console.log(JSON.stringify({
observedExitIp: ip,
status: response.status(),
requests: requestCount,
declaredBytes,
failedRequests: failedRequests.length,
cookies: (await context.cookies(TARGET_URL)).length,
}));
} catch {
console.error(`Proxy check failed during ${phase}; inspect that stage`);
process.exitCode = 1;
} finally {
if (exitContext) await exitContext.close();
if (context) await context.close();
await browser.close();
}
Run the saved program from the project directory after supplying the proxy environment values:
node proxy-check.mjs
The unchanged program ran through a task-owned HTTPS CONNECT proxy with Playwright 1.63.0, Chromium 153.0.8010.12, and Node.js 22.23.1. It returned status: 200, one target request, zero failed target requests, zero cookies, and a valid observedExitIp from the separate check.
The declared-byte total was 0: the target did not advertise a counted size. That does not mean zero transferred traffic. Keep the observed IP private when sharing logs, and compare it with the proxy-side connection record when available.
This run checked the complete program through a local forwarding proxy with normal TLS verification. It did not establish residential-provider behavior, SOCKS support, geography, or acceptance by your production target.
The parser rejects embedded URL credentials, unsupported schemes, and paths. HTTP(S) credentials go to Playwright's proxy option, so page JavaScript never sees a Proxy-Authorization header. For a SOCKS5 endpoint, leave both credential variables unset unless the provider documents its authentication path. This example does not infer SOCKS authentication from the HTTP(S) fields.
The route policy aborts third-party origins and image, font, and media requests, then caps allowed requests. It counts content-length when a response advertises it, but that header does not measure all transferred bytes.
The declared-byte threshold is checked after navigation; it cannot stop an in-flight download or enforce a spend limit. Configure any required hard transfer budget at the proxy or network layer before the run.
If the page needs blocked resources or third-party APIs, change the allowlist and budget deliberately. A passing fixture with those resources blocked does not prove that the production page works unchanged.
The finally block closes the context before the browser. Playwright recommends this order for directly created contexts so cookies, traces, HAR files, and other artifacts finish flushing. The context is also where the cookie count is read; it is not used as evidence about the proxy's egress address.
For an existing Playwright Test suite, put shared proxy settings under use in playwright.config.ts, as shown in its network guide. This configuration fragment supplements existing tests; it does not create a test:
npm install --save-dev '@playwright/test@1.63.0'
// Partial snippet: merge into an existing Playwright Test config.
import { defineConfig } from '@playwright/test';
const server = process.env.PROXY_SERVER;
if (!server) throw new Error('Set PROXY_SERVER');
export default defineConfig({
use: {
proxy: {
server,
username: process.env.PROXY_USER,
password: process.env.PROXY_PASS,
},
},
});
This sends credentials through the configured HTTP(S) proxy setting rather than page headers. Use browser.newContext({ proxy }) when one test needs a different route. Keep parallelism within the provider's documented or observed concurrent-session allowance, and keep the resource policy in the test that owns the workload.
Cookies answer a browser-state question: which cookies does this context hold for the target? They do not answer an egress question. The fixture records context.cookies(TARGET_URL) separately from the target response. Do not send those cookies to an IP-check endpoint.
The example visits the fixed ipify endpoint in a fresh context using the same browser and proxy configuration. It shares no target cookies, rejects redirects, and validates the returned address. Approve that external endpoint before running; substitute your own trusted endpoint and response parser if required.
observedExitIp belongs to the IP-check request. A rotating proxy may assign another exit to the target request, even with identical configuration. To compare them, use the provider’s documented session binding and connection records; the separate check alone cannot prove they shared an exit.
A browser network event confirms that a request occurred, not which hop carried it. A page can load through a proxy while the target blocks its exit address or returns a challenge.
page.goto() can fail before an HTTP response exists when proxy setup, DNS, or TLS negotiation fails. A 407 during HTTP proxy CONNECT can therefore appear as a navigation failure instead of a normal target response. RFC 9110 defines 407 as a proxy authentication challenge, distinct from an origin 401.
| Observation | Likely boundary | Next check |
|---|---|---|
| Browser cannot launch or binary is missing | Local Playwright installation | Run the browser install command; download proxy settings are separate from runtime proxy settings |
| Navigation has no response | Proxy, DNS, connection, or TLS setup | Check proxy host, port, scheme, trust store, and credential scope |
| Proxy challenge or 407 | Proxy authentication | Check HTTP(S) auth support and the username and password attached to the proxy object |
Target returns 401 or 403 |
Target policy or target credentials | Diagnose the destination account, egress policy, or required fields |
requestfailed event |
Network-level request failure | Record the sanitized origin and failure text; an HTTP 404 or 503 is a response, not a request failure |
| Page title or required field is wrong | Target content or challenge | Validate the business output and keep it separate from transport status |
Playwright's browser guide uses HTTPS_PROXY for downloading browser binaries. That setting does not replace the runtime proxy option used by the browser or context. If TLS verification fails, inspect which peer's certificate failed and install an approved CA where required. Do not disable TLS checks to make the fixture pass.
| Requirement | Playwright setting | What to verify |
|---|---|---|
| Every context in a browser shares one route | Launch-level proxy | All contexts use the intended endpoint |
| One test needs its own route and cookies | browser.newContext({ proxy }) |
The context owns both route and browser state |
| HTTP proxy credentials are required | username and password on the proxy object |
Credentials never enter page headers or URLs |
| SOCKS5 endpoint is supplied | Provider-supported socks5: server |
Authentication behavior is documented for that endpoint |
| Resource transfer must stay bounded | Context route plus proxy-side budget | Allowed origins, resource types, request count, and actual provider usage |
The ProxyLane pricing page, checked 2026-09-25, lists the 1 TB package at USD 2.50/GB. The offer includes HTTP and SOCKS5, country, region, city, ZIP, ISP and ASN targeting, rotating or sticky sessions, and non-expiring traffic. Registration is free; payment and workflow setup follow during onboarding.
Sign up to choose a workflow and traffic package during onboarding. The target still controls authentication and acceptance, so keep the validator and failure categories in the job.
For a Python HTTP client, use the Requests proxy guide; for a terminal probe, use the curl proxy guide; for proxy challenges, use the 407 guide.
Connect Playwright to your proxy, read a JSON fixture and save the records that pass your acceptance rule.
Open source · MIT license · Runs locally
Clone the repository and follow the README to connect your proxy. Then run:
npm install
npx playwright install chromium
npm run collectRead setup instructions
Demonstration data, not a provider performance benchmark.
Scale your workflows with the tools you already use. Rotating or sticky sessions up to 72 hours.
Compare seller APIs, licensed Amazon product data and proxy page checks by record, access scope and reuse rights.
Read guideChoose the right anti-detect browser proxy guide, map the shared verification steps, and keep account access separate from the network route.
Read guideOur picks for market research: ProxyLane for small, intermittent projects, plus options for regional tracking, monthly collection and managed extraction.
Read guide