Crawlee Proxy Rotation and SessionPool in JavaScript 3.14

Crawlee has two controls that are easy to confuse. SessionPool manages browser or request identities over time. ProxyConfiguration chooses a proxy URL for a request. A fresh IP on every retry can look like a new visitor; a sticky session can preserve cookies and a route across one permitted task.

This guide uses Crawlee JavaScript 3.14. Crawlee Python has a separate API. Apify Proxy is a separate built-in service. Neither should be treated as the same configuration surface as JavaScript ProxyConfiguration.

The request path is: Crawlee opens a Playwright page, ProxyConfiguration supplies the browser route, the target returns a document, and your handler validates the fields. Your crawler still owns navigation, rendering, parsing, retries and cleanup.

Choose the session policy first

The right setting is a hypothesis about one workload. Measure valid records, bytes, retries, blocked responses and session age on a fixed sample. A successful request does not prove that a target permits the next one.

Connect SessionPool and proxy rotation

Use Node.js with npm. In your crawler project, install the pinned line and browser:

npm install crawlee@3.14 playwright
npx playwright install chromium

This fixture uses Crawlee’s documented PlaywrightCrawler integration. useSessionPool gives each request a managed session, persistCookiesPerSession keeps cookies with that session, and the session ID keeps custom proxy selection associated with that identity.

import { PlaywrightCrawler, ProxyConfiguration } from "crawlee";
import { isIP } from "node:net";

const required = ["PROXY_URL", "FALLBACK_PROXY_URL", "TARGET_URL", "EXPECTED_MARKER"] as const;
const missing = required.filter((name) => !process.env[name]?.trim());
if (missing.length) throw new Error(`Missing required environment variables: ${missing.join(", ")}`);

const proxyConfiguration = new ProxyConfiguration({
  tieredProxyUrls: [
    [process.env.PROXY_URL!],
    [process.env.FALLBACK_PROXY_URL!],
  ],
});

const crawler = new PlaywrightCrawler({
  proxyConfiguration,
  useSessionPool: true,
  sessionPoolOptions: { maxPoolSize: 10 },
  persistCookiesPerSession: true,
  requestHandlerTimeoutSecs: 75,
  navigationTimeoutSecs: 25,
  maxRequestRetries: 1,
  maxSessionRotations: 1,
  maxRequestsPerCrawl: 1,
  async requestHandler({ page, request, session, proxyInfo }) {
    page.setDefaultTimeout(10_000);
    const responses: Array<{ url: string; status: number }> = [];
    page.on("response", (response) => {
      responses.push({ url: response.url(), status: response.status() });
    });

    const exitResponse = await page.goto("https://api.ipify.org?format=json", {
      waitUntil: "domcontentloaded",
      timeout: 25_000,
    });
    if (exitResponse?.status() !== 200) {
      throw new Error("The same crawler page exit-IP check returned a non-200 status");
    }
    let exitIp: unknown;
    try {
      const result = JSON.parse(await page.locator("body").innerText());
      exitIp = result?.ip;
    } catch {
      throw new Error("Exit-IP response is not valid JSON");
    }
    if (typeof exitIp !== "string" || isIP(exitIp) === 0) {
      throw new Error("Exit-IP response has no valid IPv4 or IPv6 address");
    }

    const mainResponse = await page.goto(request.url, {
      waitUntil: "domcontentloaded",
      timeout: 25_000,
    });
    const status = mainResponse?.status() ?? null;
    if (status !== null) session.retireOnBlockedStatusCodes(status);
    if (status === null) session.markBad();

    const body = await page.locator("body").innerText();
    const validRecord = status === 200 && body.includes(process.env.EXPECTED_MARKER!);
    if (!validRecord) {
      session.markBad();
      throw new Error("record validation failed");
    }
    console.log(JSON.stringify({
      sessionId: session.id,
      proxyTier: proxyInfo?.proxyTier ?? null,
      target: request.url,
      status,
      responseCount: responses.length,
      validRecord,
    }));
  },
});

try {
  await crawler.run([{ url: process.env.TARGET_URL! }]);
} finally {
  await crawler.teardown();
}

The output is a collector fixture. It records the page response and session identity so a retry can be compared with the first attempt. A blocked status can retire the session under the configured policy; a network failure marks it bad. An invalid record still needs parser and target investigation. Save a record only after its required fields pass. That marker check is the content validation step.

The handler’s proxyInfo is the current ProxyInfo object describing the request’s proxy connection. Its url can contain credentials; never log the object or URL. The example selects only the numeric proxyTier for logging alongside the crawler-generated session ID.

The example passes its tiered ProxyConfiguration directly to PlaywrightCrawler. Set both proxy environment variables to provider-supplied URLs. Crawlee starts with the first tier, escalates when it recognizes blocking, and periodically probes lower tiers. This requires a crawler instance; standalone newUrl() calls do not provide the expected behavior. Tier order is your choice; this code does not optimize price or benchmark domain performance.

For a simple list, use proxyUrls instead. For custom selection, newUrlFunction receives a session ID and optional request information; provide a default when the request is absent.

Keep proxy credentials in a server-side secret store. For a comparison, use the same target slice and fixed session policy. Record valid records, transfer bytes, retries, blocked responses and provider cost separately. The Crawlee session-management guide and proxy-management guide define the API boundary.

A record that can be audited

{
  "target": "https://example.com/item/17",
  "session_id": "session-17",
  "attempt": 2,
  "status": 200,
  "response_count": 14,
  "valid_record": true,
  "fields": ["title", "price", "currency"],
  "transfer_bytes": 184320,
  "failure_phase": null
}

The identifiers and values above are schema examples. Retain timestamps, parser version, route or supplier label, retry reason and billing evidence. If the page loads but an XHR field is missing, inspect resource-blocking and request-interception rules before rotating sessions. Keep those rules and parser versions constant between direct and proxy runs.

Diagnose the failing layer

For the cost side of a route comparison, see the proxy cost guide. HTTP 200 and a visible body are transport signals, not extraction proof. Count a run only after the required content fields pass, and use explicit teardown() so browser resources are released after success or failure.