# Crawlee Proxy Rotation and SessionPool in JavaScript 3.14

[← All guides](https://proxylane.dev/blog) 

DEVELOPER GUIDES · 6 MIN READ

Choose custom proxy rotation and session affinity in Crawlee JavaScript 3.14, validate useful content, and keep retries auditable.

**Founder, ProxyLane**Published September 16, 2026

On this page [Choose the session policy first](https://proxylane.dev/blog/crawlee-proxy#choose-the-session-policy-first)  [Connect SessionPool and proxy rotation](https://proxylane.dev/blog/crawlee-proxy#connect-sessionpool-and-proxy-rotation)  [A record that can be audited](https://proxylane.dev/blog/crawlee-proxy#a-record-that-can-be-audited)  [Diagnose the failing layer](https://proxylane.dev/blog/crawlee-proxy#diagnose-the-failing-layer)

**Get the data you need**

Power your research, price monitoring or AI agent with useful web data. Spend less time on proxy setup.

 [Get started ↗](https://proxylane.dev/register)

Crawlee has two controls that are easy to confuse. `SessionPool` manages browser or request identities over time. `ProxyConfiguration` chooses a proxy URL for a request. A fresh IP on every retry can look like a new visitor; a sticky session can preserve cookies and a route across one permitted task.

 

This guide uses Crawlee JavaScript 3.14. Crawlee Python has a separate API. Apify Proxy is a separate built-in service. Neither should be treated as the same configuration surface as JavaScript `ProxyConfiguration`.

 

The request path is: Crawlee opens a Playwright page, `ProxyConfiguration` supplies the browser route, the target returns a document, and your handler validates the fields. Your crawler still owns navigation, rendering, parsing, retries and cleanup.

 

## Choose the session policy first

 

| Workload | Session policy | Proxy behavior to test | Acceptance signal |
| --- | --- | --- | --- |
| Stateless public pages | Short-lived sessions | Rotate after a blocked response or a bounded usage count | Required fields pass validation |
| Multi-step flow | One session per task | Keep the same session and proxy route through the flow | Cookies and final record agree |
| Rate-limited origin | Small pool with backoff | Rotate only under the documented policy | Fewer retries without lower completeness |
| Localized QA | One session per region | Keep region fixed for the route check | Locale marker and target output agree |

 

The right setting is a hypothesis about one workload. Measure valid records, bytes, retries, blocked responses and session age on a fixed sample. A successful request does not prove that a target permits the next one.

 

## Connect SessionPool and proxy rotation

 

Use Node.js with npm. In your crawler project, install the pinned line and browser:

 

```bash
npm install crawlee@3.14 playwright
npx playwright install chromium
```

 

This fixture uses Crawlee’s documented `PlaywrightCrawler` integration. `useSessionPool` gives each request a managed session, `persistCookiesPerSession` keeps cookies with that session, and the session ID keeps custom proxy selection associated with that identity.

 

```ts
import { PlaywrightCrawler, ProxyConfiguration } from "crawlee";
import { isIP } from "node:net";

const required = ["PROXY_URL", "FALLBACK_PROXY_URL", "TARGET_URL", "EXPECTED_MARKER"] as const;
const missing = required.filter((name) => !process.env[name]?.trim());
if (missing.length) throw new Error(`Missing required environment variables: ${missing.join(", ")}`);

const proxyConfiguration = new ProxyConfiguration({
  tieredProxyUrls: [
    [process.env.PROXY_URL!],
    [process.env.FALLBACK_PROXY_URL!],
  ],
});

const crawler = new PlaywrightCrawler({
  proxyConfiguration,
  useSessionPool: true,
  sessionPoolOptions: { maxPoolSize: 10 },
  persistCookiesPerSession: true,
  requestHandlerTimeoutSecs: 75,
  navigationTimeoutSecs: 25,
  maxRequestRetries: 1,
  maxSessionRotations: 1,
  maxRequestsPerCrawl: 1,
  async requestHandler({ page, request, session, proxyInfo }) {
    page.setDefaultTimeout(10_000);
    const responses: Array<{ url: string; status: number }> = [];
    page.on("response", (response) => {
      responses.push({ url: response.url(), status: response.status() });
    });

    const exitResponse = await page.goto("https://api.ipify.org?format=json", {
      waitUntil: "domcontentloaded",
      timeout: 25_000,
    });
    if (exitResponse?.status() !== 200) {
      throw new Error("The same crawler page exit-IP check returned a non-200 status");
    }
    let exitIp: unknown;
    try {
      const result = JSON.parse(await page.locator("body").innerText());
      exitIp = result?.ip;
    } catch {
      throw new Error("Exit-IP response is not valid JSON");
    }
    if (typeof exitIp !== "string" || isIP(exitIp) === 0) {
      throw new Error("Exit-IP response has no valid IPv4 or IPv6 address");
    }

    const mainResponse = await page.goto(request.url, {
      waitUntil: "domcontentloaded",
      timeout: 25_000,
    });
    const status = mainResponse?.status() ?? null;
    if (status !== null) session.retireOnBlockedStatusCodes(status);
    if (status === null) session.markBad();

    const body = await page.locator("body").innerText();
    const validRecord = status === 200 && body.includes(process.env.EXPECTED_MARKER!);
    if (!validRecord) {
      session.markBad();
      throw new Error("record validation failed");
    }
    console.log(JSON.stringify({
      sessionId: session.id,
      proxyTier: proxyInfo?.proxyTier ?? null,
      target: request.url,
      status,
      responseCount: responses.length,
      validRecord,
    }));
  },
});

try {
  await crawler.run([{ url: process.env.TARGET_URL! }]);
} finally {
  await crawler.teardown();
}
```

 

The output is a collector fixture. It records the page response and session identity so a retry can be compared with the first attempt. A blocked status can retire the session under the configured policy; a network failure marks it bad. An invalid record still needs parser and target investigation. Save a record only after its required fields pass. That marker check is the content validation step.

 

The handler’s `proxyInfo` is the current `ProxyInfo` object describing the request’s proxy connection. Its `url` can contain credentials; never log the object or URL. The example selects only the numeric `proxyTier` for logging alongside the crawler-generated session ID.

 

The example passes its tiered `ProxyConfiguration` directly to `PlaywrightCrawler`. Set both proxy environment variables to provider-supplied URLs. Crawlee starts with the first tier, escalates when it recognizes blocking, and periodically probes lower tiers. This requires a crawler instance; standalone `newUrl()` calls do not provide the expected behavior. Tier order is your choice; this code does not optimize price or benchmark domain performance.

 

For a simple list, use `proxyUrls` instead. For custom selection, `newUrlFunction` receives a session ID and optional request information; provide a default when the request is absent.

 

Keep proxy credentials in a server-side secret store. For a comparison, use the same target slice and fixed session policy. Record valid records, transfer bytes, retries, blocked responses and provider cost separately. The  [Crawlee session-management guide](https://crawlee.dev/js/docs/3.14/guides/session-management)  and  [proxy-management guide](https://crawlee.dev/js/docs/3.14/guides/proxy-management)  define the API boundary.

 

## A record that can be audited

 

```json
{
  "target": "https://example.com/item/17",
  "session_id": "session-17",
  "attempt": 2,
  "status": 200,
  "response_count": 14,
  "valid_record": true,
  "fields": ["title", "price", "currency"],
  "transfer_bytes": 184320,
  "failure_phase": null
}
```

 

The identifiers and values above are schema examples. Retain timestamps, parser version, route or supplier label, retry reason and billing evidence. If the page loads but an XHR field is missing, inspect resource-blocking and request-interception rules before rotating sessions. Keep those rules and parser versions constant between direct and proxy runs.

 

## Diagnose the failing layer

 

| Symptom | Layer | Next check |
| --- | --- | --- |
| Missing environment variable | Job configuration | Check secret binding names and target inputs without printing values |
| Crawler starts but navigation times out | Proxy or runner network | Check host, port, protocol and reachability from the worker |
| 407 or authentication failure | Proxy authentication | Check the provider URL format and use the  [proxy 407 guide](https://proxylane.dev/blog/proxy-error-407) |
| Exit check passes but marker fails | Target or parser | Inspect the received document, redirects, selector and target permissions |
| Session state is lost between steps | Session policy | Keep the same session and review  [proxy rotation and sticky sessions](https://proxylane.dev/blog/proxy-rotation-and-sticky-sessions) |
| Tier escalation never occurs | API wiring | Pass the `ProxyConfiguration` containing `tieredProxyUrls` to the crawler; inspect failure classification |

 

For the cost side of a route comparison, see the  [proxy cost guide](https://proxylane.dev/blog/proxy-cost-per-successful-request) . HTTP 200 and a visible body are transport signals, not extraction proof. Count a run only after the required content fields pass, and use explicit `teardown()` so browser resources are released after success or failure.

 

## Sources and further reading

- [https://crawlee.dev/js/api/3.14/core/class/SessionPool](https://crawlee.dev/js/api/3.14/core/class/SessionPool)

- [https://crawlee.dev/js/docs/3.14/guides/session-management](https://crawlee.dev/js/docs/3.14/guides/session-management)

- [https://crawlee.dev/js/docs/3.14/guides/proxy-management](https://crawlee.dev/js/docs/3.14/guides/proxy-management)

- [https://crawlee.dev/js/api/3.14/core/interface/ProxyInfo](https://crawlee.dev/js/api/3.14/core/interface/ProxyInfo)

- [https://crawlee.dev/js/api/3.14/core/class/ProxyConfiguration](https://crawlee.dev/js/api/3.14/core/class/ProxyConfiguration)

- [https://crawlee.dev/js/api/3.14/core/interface/ProxyConfigurationOptions](https://crawlee.dev/js/api/3.14/core/interface/ProxyConfigurationOptions)

- [https://crawlee.dev/js/api/3.14/playwright-crawler/class/PlaywrightCrawler](https://crawlee.dev/js/api/3.14/playwright-crawler/class/PlaywrightCrawler)

- [https://github.com/apify/crawlee/discussions/2943](https://github.com/apify/crawlee/discussions/2943)

[Sign in ↗](https://proxylane.dev/login)

## Keep reading

[Use cases · 3 min read

### Amazon Scraper API vs Proxy: Choose a Product Data Source

Distinguish authorized Amazon APIs, licensed product data and proxy-based page checks by record quality and access rights.

 Read guide →](https://proxylane.dev/blog/amazon-scraper-api-vs-proxy)   [Integrations · 5 min read

### Apify Custom Proxy Setup: Use Your Own Residential Route

Connect a buyer-owned proxy to an Apify Actor, keep the session boundary clear, and validate records instead of counting requests.

 Read guide →](https://proxylane.dev/blog/apify-custom-proxy)   [Proxy fundamentals · 6 min read

### Australia Residential Proxies: Verify the AU Exit and State-Level Result

Separate Australian egress from en-AU content, AUD pricing, GST display, postcode validation and the state delivery context.

 Read guide →](https://proxylane.dev/blog/australia-residential-proxies)

Canonical source: https://proxylane.dev/blog/crawlee-proxy

Documentation index: https://proxylane.dev/llms.txt