How to scrape a permitted web page with n8n

Fetch an allowed HTML page with n8n's HTTP Request node, extract fields with HTML, and reject incomplete records in a Code node. Add the same proxy configuration to the request that checks exit IP; a successful response proves only what that request returned.

This walkthrough uses one page from Books to Scrape, a practice catalogue. Check the target's terms and robots.txt first: the Robots Exclusion Protocol describes crawler instructions, not permission to collect data.

The complete HTTP Request → HTML → Code path was executed in a local self-hosted n8n 2.41.3 instance on Node.js 24.21.0 against a small static HTML fixture served only on loopback. The fixture verified the selectors and output fields without contacting Books to Scrape. The public catalogue and hosted n8n UI were not exercised here.

Prepare n8n

Audience: workflow builders familiar with node connections and JSON. The tested self-hosted environment used macOS 26.5.2 arm64, n8n 2.41.3 and Node.js 24.21.0. Follow the official npm installation guide for a local n8n instance, then use the HTTP Request, HTML and Code nodes. This guide covers the JavaScript Code node; Python variants are outside its scope. Create a manual workflow for one request to https://books.toscrape.com/. The sample reads the first catalogue page only; it does not paginate or run on a schedule.

Caution: A proxy request can consume paid traffic, and a workflow can repeat it when retried or scheduled. Begin with the single manual request below. Stop the workflow and lower its item count if it sends more requests than expected.

Fetch one catalogue page

Add a Manual Trigger and connect an HTTP Request node. Configure one GET request to https://books.toscrape.com/.

In Add Option → Response, set Response Format to Text and Put Output in Field to data. Turn on Include Response Headers and Status while checking the first run. The HTTP Request node follows redirects by default; inspect the status and resulting page before treating the response as catalogue HTML. Its Timeout is in milliseconds and limits the wait for response headers and the start of the body, not the complete workflow duration.

Run the node once. Confirm the result contains an HTML string in data and a successful HTTP status. A 200 sign-in page, challenge page, or JavaScript shell is not a product record. Stop here if the response is not the page the workflow expects.

Extract the product fields

Connect an HTML node and choose Extract HTML Content. Set Source Data to JSON and JSON Property to data. Add these two extraction values:

Both selectors match the same anchor elements on this page. Confirm that the arrays have the same nonzero length before pairing by index. If a target has variable card markup, stop and adapt the extraction to preserve each card as a record; two independently shortened arrays can silently pair the wrong title and link.

The HTML node selects content from the response string. It does not run JavaScript from the page. Use an authorized API or a browser-capable tool when the required data appears only after client-side rendering.

Reject incomplete records

Connect a Code node, set Mode to Run Once for All Items, and paste this complete JavaScript example:

const sourceUrl = 'https://books.toscrape.com/';
const { titles, links } = $input.first().json;

if (!Array.isArray(titles) || !Array.isArray(links) || titles.length === 0 || titles.length !== links.length) {
  throw new Error('Title/link extraction mismatch');
}

const seen = new Set();
return Array.from({ length: titles.length }, (_, index) => {
  const value = titles[index];
  const rawHref = links[index];
  if (typeof value !== 'string' || typeof rawHref !== 'string') {
    throw new Error(`Expected title and link strings at row ${index + 1}`);
  }

  const title = value.trim();
  const href = rawHref.trim();
  if (!title || !href) {
    throw new Error(`Missing title or link at row ${index + 1}`);
  }

  const baseUrl = sourceUrl.replace(/\/+$/, '');
  const detailUrl = /^https?:\/\//i.test(href)
    ? href
    : `${baseUrl}/${href.replace(/^\/+/, '')}`;
  if (!detailUrl.startsWith(`${baseUrl}/`)) {
    throw new Error(`Unexpected detail host at row ${index + 1}`);
  }
  if (seen.has(detailUrl)) {
    throw new Error(`Duplicate detail URL at row ${index + 1}`);
  }
  seen.add(detailUrl);

  return {
    json: {
      title,
      detail_url: detailUrl,
      source_url: sourceUrl,
      fetched_at: new Date().toISOString(),
    },
  };
});

Run the workflow manually. Accept the output only when each item has a nonempty title and absolute detail URL on the expected host and the item count matches the HTML node's titles array. This sample joins the relative links against the known host root and checks the resulting prefix; adapt that rule if the permitted target uses another host or a non-root base path. The timestamp records when n8n built the record; it is not a source-provided publication date.

The following two records were captured from the local fixture run. The fixture reproduced two catalogue cards; its source_url values are configured labels, not proof of a public-site fetch. A public catalogue page can return a different count.

[
  {
    "title": "A Light in the Attic",
    "detail_url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
    "source_url": "https://books.toscrape.com/",
    "fetched_at": "2026-09-29T15:46:31.760Z"
  },
  {
    "title": "Tipping the Velvet",
    "detail_url": "https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html",
    "source_url": "https://books.toscrape.com/",
    "fetched_at": "2026-09-29T15:46:31.761Z"
  }
]

Add pacing and a proxy when needed

For a permitted list of input URLs, use the HTTP Request node's Batching option. Items per Batch controls the group size, and Batch Interval is the delay between groups in milliseconds. Start with a small list. If the target returns 429, pause or reduce the request rate and follow its Retry-After guidance. Do not rotate routes to evade a limit or retry an authorization failure.

Caution: An authenticated proxy URL contains credentials. Start in a disposable local workflow, inject the URL from the deployment secret store, and exclude it from exports and retained execution data. If credentials appear in an export or log, revoke them and remove the exposed copy before continuing.

For self-hosted n8n, have the deployment secret manager inject HTTP_PROXY for HTTP targets or HTTPS_PROXY for HTTPS targets into the n8n process before it starts. These variables take a proxy URL in the form http://<USERNAME>:<PASSWORD>@<HOST>:<PORT>. The scheme describes the proxy connection; <HOST> and <PORT> are the provider's endpoint, and <USERNAME> and <PASSWORD> are its credentials. Percent-encode reserved characters in the credentials. Omit the credential segment only for a route that permits unauthenticated access.

Leave the HTTP Request node's Proxy option empty to use those process settings. A per-node Proxy value overrides the process proxy, so remove stale overrides before testing. Do not put an authenticated URL into that field or exported workflow JSON. This process-level recipe applies to self-hosted n8n; hosted deployments need their supported secret configuration.

Run one HTTP Request node against a trusted plain-text IP-echo endpoint using Text response format. Check its status and confirm that data parses as an IP address. Then run the permitted target request with the same process proxy and validate the extracted fields. An echo service reports only the address observed for its own request; rotating routes can select another address for the target.

The local test injected an ephemeral authenticated HTTP_PROXY without storing it in workflow JSON. The echo and catalogue HTTP Request nodes both reached the fixture through that proxy; the echo returned HTTP 200 and 198.51.100.8, a documentation-only address. The HTML and Code nodes then produced the two expected records. This proves the local plain-HTTP path, not HTTPS CONNECT, a public-provider exit, or hosted n8n behavior.

Troubleshoot a failed run

For dynamic pages, choose a permitted API or browser-rendering path. For route-specific failures, use the ProxyLane checker and proxy IP and browser diagnostics. A proxy changes the network path; it does not change access permission.

ProxyLaneTraffic never expires

Premium residential proxies
From $2/GB

Scale your workflows with the tools you already use. Rotating or sticky sessions up to 72 hours.

Try 350 MB for $1.95 View plans