Amazon Scraper API vs Proxy: Choose a Product Data Source
Distinguish authorized Amazon APIs, licensed product data and proxy-based page checks by record quality and access rights.
Read guide →A browser automation team scaling parallel jobs needs each job to finish with the right account, region and session state. Adding proxy addresses helps only when the route is the constraint. A worker can also run out of memory, lose its browser context or spend its request budget retrying a target that asked it to slow down.
Define ownership before increasing concurrency: one logical task owns its browser state and the route policy needed for that task. Keep that relationship visible in the queue record so a retry cannot silently borrow another job's cookies or geography.
Map the task to a session
A task might visit a product page, select a delivery region and read an offer returned by a later request. Treat those steps as one unit when their result depends on shared state. A new route halfway through the task can make the final record incomparable with its starting conditions.
Crawlee's SessionPool documentation describes session management and retirement of blocked identities. That is useful lifecycle support, but the application still needs to decide which saved state belongs to which job. For browser proxy configuration, Playwright documents browser-wide and per-context settings. Follow the configuration boundary your chosen browser actually supports; changing a value in a queue does not reconfigure a running browser.
Use this worksheet before launching another worker:
| Resource | Owner | Release condition | What to measure |
|---|---|---|---|
| Queue lease | Logical task | Durable result or recorded terminal failure | Age of oldest unfinished task |
| Browser context and cookies | Task or explicitly shared account session | Task completion or session retirement | Active contexts and memory |
| Proxy session affinity | Stateful sequence | Sequence completion or provider session expiry | Observed route changes |
| Target request allowance | Target and permitted schedule | Request completes or cooldown expires | Requests and rate-limit responses |
| Retry budget | Logical task across all routes | Valid result or exhausted budget | Attempts per accepted result |
A pooled browser process can host multiple contexts where supported. Separate processes may be necessary for another client's proxy configuration. That choice changes memory use, so test the actual client rather than assuming a context is as expensive as a whole browser.
Keep concurrency and connections separate
One browser job may request the document and several page resources. A limit expressed as concurrent proxy connections cannot be copied directly into a worker count. Ask the supplier what counts toward its limit, then measure the connections generated by a representative permitted task.
Record the intended ceiling and the observed bottleneck in a capacity sheet. Include one row per fixed workload run:
| Run | Workers | Active browsers | Resident memory (GB) | Transferred bytes | Billed traffic (GB) | Billed cost | Valid tasks | Cost per valid task | Queue age | Retries |
|---|---|---|---|---|---|---|---|---|---|---|
| direct baseline | record | record | record | record | record from the supplier meter | record | record | billed cost / valid tasks | record | record |
| candidate route | record | record | record | record | record from the supplier meter | record | record | billed cost / valid tasks | record | record |
Use the supplier's definition of a billed GB; transferred bytes are a separate observation. If valid tasks is zero, keep cost per valid task undefined and report the total billed cost. Increase one setting at a time with the same workload. Stop increasing it when the job's acceptance, resource or cost budget fails, even if some proxy capacity remains. This connects the control you changed, the traffic it consumed and the valid work it produced.
A Crawlee Python issue about large proxy and session pools reports high memory use in a particular setup. It is a reason to measure memory in your stack, not a universal estimate of how many workers it supports.
Make retry ownership explicit
The following is a design sketch for an authorized workload, not a framework API:
claim task lease
acquire target allowance and worker slot
open task-owned browser state with the assigned route
run the permitted sequence
validate final URL, region and required result fields
if valid:
save result under the logical task ID
elif target requests a pause:
save cooldown and release the task for later
elif route failure and task budget remains:
close browser state before assigning a replacement route
else:
save terminal failure with its stage
release browser resources, worker slot and lease
Keep the same logical task ID across retries and make result writes idempotent. Before replaying a task that changes remote state, determine whether the action already happened. A timeout after submission is an uncertain outcome; retrying it blindly can duplicate the action.
Choose the next capacity investment
If memory grows while the proxy is mostly idle, investigate browser lifetime and reuse. If valid tasks are delayed by a documented connection ceiling, evaluate additional route capacity. If the target is rate-limiting you, reduce the schedule; switching addresses does not resolve that instruction. If extraction fails on successfully rendered pages, inspect the parser.
Use the monitoring playbook to preserve these distinctions in alerts. The session guide covers continuity choices, while cost per successful request helps account for bytes spent on failed and retried tasks.