Browser agent concurrency: isolate proxy sessions

Scale browser-agent workers only after identifying the constrained resource. Give each logical task an explicit owner for browser state, route policy, and retry budget; then increase one concurrency setting at a time and compare accepted work, resource use, and billed traffic.

If the first task is connecting a browser to a proxy, start with the Playwright setup and verification guide. Return here when deciding how many tasks to run together and how to isolate their state.

Caution: Comparison runs consume traffic and may incur charges. Use a bounded, authorized test workload with request and spend caps; stop at either cap. Use a sandbox when available.

Map each task to its session

Treat steps that depend on shared cookies, account state, or a selected region as one logical task. Keep its identifier with the queue record so a retry cannot silently reuse another task's browser state. A queue lease is a temporary reservation on that task.

Crawlee's SessionPool reference describes creation, rotation, persistence, and retirement of crawler sessions. That pool does not decide which application task owns saved state.

Playwright's proxy documentation allows proxy configuration for the browser or an individual browser context. Select the scope supported by the client, and record it with the task.

A browser process may host multiple contexts where the client supports them. Separate processes can isolate browser-level proxy settings, but increase the resources the application must manage. Measure the actual client and workload instead of treating a context and a process as equivalent units.

Separate workers from connections

One browser job can issue a document request and additional requests for page resources. A provider's concurrent-connection limit therefore does not translate directly into a worker count. Check the provider's definition of a connection and measure a representative, permitted task.

For example, use a fixed ten-minute authorized sample and record active connections from the provider's meter alongside queue age and accepted tasks. If the meter reaches its documented ceiling while the queue grows, hold worker count steady and investigate that limit. This is an illustrative observation plan, not a measured capacity result.

Use these worksheets for each fixed workload run. Populate them with observed values; the labels are not measurements.

Transferred bytes and provider-billed traffic are separate observations. If no task passes validation, leave cost per valid task undefined and report total cost. Change one setting at a time with the same workload. Stop when the task's acceptance, resource, or cost budget fails, even if route capacity remains.

A Crawlee issue about large proxy and session pools describes memory use in one setup. Treat it as an anecdote that motivates measurement, not a capacity estimate for another stack.

Keep retries under task ownership

The following pseudocode describes a design, not a framework API. The same task ID follows the work through retries, while result writes remain idempotent.

claim task lease
acquire target allowance and worker slot
open task-owned browser state with the assigned route
run the permitted sequence
validate final URL, region and required result fields
if valid:
    save result under the logical task ID
elif target requests a pause:
    save cooldown and release the task for later
elif route failure and task budget remains:
    close browser state before assigning a replacement route
else:
    save terminal failure with its stage
release browser resources, worker slot and lease

Before replaying a task that changes remote state, check whether the action already completed. A timeout after submission leaves the outcome uncertain; blindly retrying can duplicate it. If the target asks for a pause, store the cooldown and stop the current attempt.

Choose the next capacity change

If memory grows while the route is mostly idle, investigate browser lifetime and context reuse. If tasks wait behind a documented connection ceiling, evaluate route capacity. If the target returns a rate-limit response, reduce the schedule and honor its delay. If a successfully rendered page fails extraction, inspect the parser.

The proxy monitoring playbook separates these failure signals. The sticky-session guide explains route continuity, and cost per successful request covers failed and retried work.

ProxyLaneTraffic never expires

Premium residential proxies
From $2/GB

Scale your workflows with the tools you already use. Rotating or sticky sessions up to 72 hours.

Try 350 MB for $1.95 View plans