# How to scrape Hacker News with Scrapling

[All guides](https://proxylane.dev/blog) 

DEVELOPER GUIDES · 12 MIN READ

Extract story titles, links, and scores; follow pagination; then run the same collection through a residential proxy in HTTP and Chromium modes.

**ProxyLane** Published September 25, 2026

On this page [Prepare the environment](https://proxylane.dev/blog/scrapling-proxy#prepare-the-environment)  [Run the complete collector](https://proxylane.dev/blog/scrapling-proxy#run-the-complete-collector)  [Understand the selectors](https://proxylane.dev/blog/scrapling-proxy#understand-the-selectors)  [Follow the next page and verify the file](https://proxylane.dev/blog/scrapling-proxy#follow-the-next-page-and-verify-the-file)  [Connect ProxyLane to the same collection](https://proxylane.dev/blog/scrapling-proxy#connect-proxylane-to-the-same-collection)  [Run the Chromium variant](https://proxylane.dev/blog/scrapling-proxy#run-the-chromium-variant)  [Troubleshoot the failed stage](https://proxylane.dev/blog/scrapling-proxy#troubleshoot-the-failed-stage)  [Judge the connection on the intended workload](https://proxylane.dev/blog/scrapling-proxy#judge-the-connection-on-the-intended-workload)

**Enterprise from $2/GB at 5 TB+**

Non-expiring traffic, location targeting and rotating or sticky sessions for your existing tools.

 [Sign up](https://proxylane.dev/register?interest=proxies)

Scrapling can collect Hacker News stories with `FetcherSession` and Cascading Style Sheets (CSS) selectors, then follow the page’s `More` link. Save each story’s identifier (ID), title, link, and available score as JavaScript Object Notation (JSON). Check for empty results and duplicate IDs before accepting the file; a successful Hypertext Transfer Protocol (HTTP) response alone does not prove extraction worked.

 

Hacker News returns these fields in its Hypertext Markup Language (HTML), so a browser and residential proxy are optional for this example. Start with the direct HTTP run. Add ProxyLane when evaluating a residential connection for a permitted workload, then use the browser variant to learn how the same extraction works in Chromium.

 

Audience: Python developers who can run a script and recognize HTML elements. The example reads public listing pages; it does not log in, vote, follow article links, or collect comments. For an ongoing Hacker News data feed, the  [official application programming interface (API)](https://github.com/HackerNews/API#items)  avoids dependence on HTML selectors.

 

## Prepare the environment

 

Required before starting:  [Python 3.12.0](https://www.python.org/downloads/release/python-3120/) , the runtime tested here. Commands use macOS or Linux shell syntax; Windows setup is outside this walkthrough. The recorded environment was macOS 26.5.2 on Apple Silicon, Scrapling 0.4.15, Playwright 1.63.0, and Chromium 153.0.8010.12.

 

Create an isolated directory and install the pinned packages. The browser installation is needed only for the optional browser run, but installing it now makes both commands available.

 

```bash
mkdir scrapling-hn
cd scrapling-hn
python3 -m venv .venv
source .venv/bin/activate
python -m pip install 'scrapling[fetchers]==0.4.15' 'playwright==1.63.0'
python -m playwright install chromium
```

 

The  [Scrapling installation instructions](https://github.com/D4Vinci/Scrapling/blob/v0.4.15/README.md#installation)  describe other fetchers and their browser dependencies. Linux machines may also need the  [Playwright system dependencies](https://playwright.dev/python/docs/browsers#install-system-dependencies) .

 

**Caution:** This example contacts the live Hacker News site. Its  [crawl rules](https://news.ycombinator.com/robots.txt)  specify a 30-second delay. Keep one run active, wait at least 30 seconds between manual runs, and stop on access or rate-limit errors. Proxy rotation is not permission to bypass that limit.

 

## Run the complete collector

 

Save the following code as `scrapling-proxy.py`, or download the  [identical Python file](https://proxylane.dev/assets/scrapling/scrapling-proxy.py) . All adjustable endpoints and timing values are at the top. Keep the default target for the first run.

 

The code uses Scrapling’s  [HTTP session parameters](https://github.com/D4Vinci/Scrapling/blob/v0.4.15/docs/fetching/static.md#shared-arguments)  and  [browser session parameters](https://github.com/D4Vinci/Scrapling/blob/v0.4.15/docs/fetching/dynamic.md#full-list-of-arguments) . In this pinned version, `retries=1` makes one attempt per fetch. HTTP timeouts are in seconds; browser operation timeouts are in milliseconds. Neither setting is a total job deadline.

 

```python
import argparse
import getpass
import ipaddress
import json
import logging
import os
import time
from datetime import datetime, timezone
from pathlib import Path
from urllib.parse import unquote, urljoin, urlsplit

from scrapling.fetchers import DynamicSession, FetcherSession

TARGET_URL = "https://news.ycombinator.com/"
IP_URL = "https://api.ipify.org?format=json"
PAGE_DELAY_SECONDS = 30
HTTP_TIMEOUT_SECONDS = 30
BROWSER_TIMEOUT_MS = 30000


class GuideError(Exception):
    pass


def read_proxy():
    raw = os.environ.get("PROXY_URL") or getpass.getpass("Proxy URL (hidden): ")
    try:
        parts = urlsplit(raw)
        valid = (parts.scheme in ("http", "https") and parts.hostname
                 and parts.port and parts.username and parts.password
                 and parts.path in ("", "/") and not parts.query and not parts.fragment)
    except ValueError:
        valid = False
    if not valid:
        raise GuideError("Use the complete HTTP proxy URL from the generator")
    return {"server": f"{parts.scheme}://{parts.hostname}:{parts.port}",
            "username": unquote(parts.username), "password": unquote(parts.password)}


def check_status(response):
    if response.status != 200:
        raise GuideError(f"HTTP {response.status}; stop and check the diagnostic table")


def extract_stories(response):
    check_status(response)
    rows = []
    for item in response.css("tr.athing"):
        story_id = item.attrib.get("id", "")
        title = item.css(".titleline > a::text").get()
        href = item.css(".titleline > a::attr(href)").get()
        if not story_id.isdigit() or not title or not href:
            raise GuideError("A story is missing its ID, title, or link")
        score = response.css(f"#score_{story_id}::text").get()
        points = int(score.split()[0]) if score else None
        url = urljoin(TARGET_URL, href)
        if urlsplit(url).scheme not in ("http", "https"):
            raise GuideError("A story link is not an HTTP or HTTPS URL")
        rows.append({"id": int(story_id), "title": title,
                     "url": url, "points": points})
    if not rows:
        raise GuideError("No stories found; inspect the response and selectors")
    return rows


def collect(session, mode, pages, use_proxy):
    def fetch(url):
        if mode == "http":
            response = session.get(url)
        else:
            response = session.fetch(url, google_search=False)
        check_status(response)
        return response

    exit_ip = None
    if use_proxy:
        response = fetch(IP_URL)
        payload = (response.json() if mode == "http"
                   else json.loads(response.css("body")[0].get_all_text()))
        exit_ip = str(ipaddress.ip_address(payload["ip"]))
    rows, seen = [], set()
    url = TARGET_URL
    for page_number in range(pages):
        if page_number:
            time.sleep(PAGE_DELAY_SECONDS)
        response = fetch(url)
        for row in extract_stories(response):
            if row["id"] not in seen:
                rows.append(row)
                seen.add(row["id"])
        if page_number + 1 < pages:
            next_path = response.css("a.morelink::attr(href)").get()
            if not next_path:
                raise GuideError("Next-page link missing; no complete run saved")
            url = urljoin(url, next_path)
            if urlsplit(url).netloc != urlsplit(TARGET_URL).netloc:
                raise GuideError("Next-page link leaves Hacker News; stopped")
    return {"captured_at": datetime.now(timezone.utc).isoformat(),
            "mode": mode, "pages": pages, "exit_ip": exit_ip, "stories": rows}


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("mode", choices=("http", "browser"))
    parser.add_argument("--pages", type=int, choices=(1, 2), default=1)
    parser.add_argument("--proxy", action="store_true")
    args = parser.parse_args()
    route = "proxy" if args.proxy else "direct"
    output = Path(f"hn-{args.mode}-{route}-{args.pages}.json")
    if output.exists():
        raise GuideError(f"Move {output.name} before running again")
    proxy = read_proxy() if args.proxy else None
    if args.mode == "http":
        settings = {"timeout": HTTP_TIMEOUT_SECONDS, "retries": 1}
        if proxy:
            settings.update(proxy=proxy["server"],
                            proxy_auth=(proxy["username"], proxy["password"]))
        with FetcherSession(**settings) as session:
            result = collect(session, args.mode, args.pages, args.proxy)
    else:
        settings = {"headless": True, "retries": 1, "timeout": BROWSER_TIMEOUT_MS}
        if proxy:
            settings["proxy"] = proxy
        with DynamicSession(**settings) as session:
            result = collect(session, args.mode, args.pages, args.proxy)
    with output.open("x", encoding="utf-8") as stream:
        json.dump(result, stream, ensure_ascii=False, indent=2)
    print(f"Saved {len(result['stories'])} unique stories to {output.name}")
    print(f"Pages: {result['pages']}; route: {'proxy' if args.proxy else 'direct'}")
    if result["exit_ip"]:
        print(f"Exit IP: {result['exit_ip']}")


if __name__ == "__main__":
    # Library errors can contain connection details; never print raw exceptions.
    logging.disable(logging.CRITICAL)
    try:
        main()
    except GuideError as error:
        raise SystemExit(str(error)) from None
    except (Exception, KeyboardInterrupt) as error:
        print(f"Stopped ({type(error).__name__}); no successful result reported.")
        raise SystemExit("Check proxy credentials, timeouts, and the diagnostic table.") from None
```

 

Run one page without a proxy:

 

```bash
python scrapling-proxy.py http
```

 

Expected output shape, with the count determined by the live page:

 

```text
Saved 30 unique stories to hn-http-direct-1.json
Pages: 1; route: direct
```

 

The file contains a capture timestamp, mode, page count, optional exit Internet Protocol (IP) address, and a `stories` array. A story includes an integer `id`, non-empty `title`, absolute link in `url`, and `points`. Job listings can omit a visible score; the collector stores `null` rather than inventing zero.

 

## Understand the selectors

 

Each `tr.athing` row carries the story ID. Its `.titleline > a` link supplies the headline and destination. The score lives outside that row, in the following metadata area, so the collector looks it up by the matching `score_<STORY_ID>` element ID across the response. `<STORY_ID>` means that row’s numeric ID.

 

| Required value | Scrapling selection | Acceptance check |
| --- | --- | --- |
| Story ID | `item.attrib.get("id")` | Digits, converted to an integer |
| Headline | `.titleline > a::text` | Non-empty text |
| Destination | `.titleline > a::attr(href)` | Present; resolved against the site URL |
| Score | `#score_<STORY_ID>::text` | Integer when present; otherwise `null` |
| Next page | `a.morelink::attr(href)` | Present when another page is requested; same host |

 

These are  [Scrapling CSS selectors](https://github.com/D4Vinci/Scrapling/blob/v0.4.15/docs/parsing/selection.md) , not browser interactions. `::text` selects text, and `::attr(href)` selects an attribute. Relative story links, such as an Ask HN discussion, become absolute links through `urljoin`.

 

Extraction uses explicit selectors without adaptive recovery. If Hacker News changes its markup, an empty page fails before a result file is written. Inspect the changed HTML and update the selectors deliberately; an old count or a blank JSON array is not an acceptable result.

 

## Follow the next page and verify the file

 

After waiting at least 30 seconds since the previous run, collect 2 pages:

 

```bash
python scrapling-proxy.py http --pages 2
```

 

The collector follows the actual `More` link, waits 30 seconds before the second page, and deduplicates by story ID. The live ranking can move between requests, so the output is not an atomic snapshot. The final-code direct run on 2026-09-25 returned 60 unique stories across 2 pages; an earlier run returned 59. Counts can differ between runs.

 

Verify the saved data independently:

 

```bash
python - <<'PY'
import json
from urllib.parse import urlsplit

with open("hn-http-direct-2.json", encoding="utf-8") as stream:
    result = json.load(stream)
rows = result["stories"]
assert result["pages"] == 2
assert rows, "No stories saved"
assert len(rows) == len({row["id"] for row in rows}), "Duplicate story IDs"
for row in rows:
    assert isinstance(row["id"], int) and row["title"].strip()
    assert urlsplit(row["url"]).scheme in {"http", "https"}
    assert row["points"] is None or isinstance(row["points"], int)
print(f"Validated {len(rows)} unique stories")
PY
```

 

A pre-existing output file stops the collector before network requests. Move that file to a separate archive name before repeating the same command. This preserves the earlier run for comparison and prevents stale output from being mistaken for a new success.

 

## Connect ProxyLane to the same collection

 

A residential proxy changes the connection used to reach the target; Scrapling still requests pages and extracts fields. The Hacker News exercise works directly, so no purchase is required to learn the collector.  [ProxyLane pricing](https://proxylane.dev/pricing)  includes a paid 350 MB trial for 1.95 USD. Enter `SCRAPLING25` manually at checkout for 25% off one purchase per account; using it on the trial consumes that one use.

 

1. Sign in to the  [ProxyLane dashboard](https://proxylane.dev/dashboard/proxies) . For a paid connection, wait until purchased traffic is active.
 
1. Open `Advanced settings`, select `HTTP / HTTPS`, and choose the intended country. For this related sequence, select `Keep the same IP` with a distinct `Session name`. Use `Rotate every request` for independent checks.
 
1. Select `Generate connection`. Require `Ready to connect`; `Example only` values are not usable credentials.
 
1. Open `Connection details` and copy the connection privately. `Test connection` is a useful first check, but the script must still reach Hacker News.

 

The  [connection settings](https://docs.proxylane.dev/connection-settings)  explain the generated fields. The uniform resource locator (URL) follows `http://<PROXY_USERNAME>:<PROXY_PASSWORD>@<PROXY_HOST>:<PROXY_PORT>`: replace every angle-bracket value with the matching generated field. Preserve the generated scheme; an HTTPS destination does not mean the proxy gateway itself uses HTTPS. Use the proxy password, not the account password.

 

**Warning:** Proxy runs consume real traffic from the selected account, even though the target is a learning example. Start with 1 page and check dashboard usage afterward. Keep the full connection URL private; the script prompts without echoing it. If credentials leak, replace them through support or the account controls. Stopping a run does not reverse traffic already consumed.

 

Wait at least 30 seconds after the previous Hacker News run, then run:

 

```bash
python scrapling-proxy.py http --proxy
```

 

At the hidden prompt, paste the generated connection URL. Automated jobs can instead provide `PROXY_URL` through a secret manager; do not paste it into shell commands, commit it, or print the environment. URL parsing separates scheme/host/port from username/password and decodes percent-encoded credentials.

 

In HTTP mode, `proxy` receives the server address and `proxy_auth` receives the credential tuple. In browser mode, the same fields enter the `proxy` dictionary. A failed proxy run stops without falling back to a direct connection.

 

The same configured session requests  [ipify](https://www.ipify.org/)  before the target and validates the returned IP address. The echo service sees the exit IP. This proves that connection’s observed address; it does not independently prove country, residential classification, or access to another target. A rotating route can use a different IP for the next request.

 

## Run the Chromium variant

 

After another 30-second pause, run the same extraction through a browser:

 

```bash
python scrapling-proxy.py browser --proxy
```

 

Omit `--proxy` to test Chromium directly. `DynamicSession` opens a headless Chromium context, requests the IP endpoint and target through that configuration, and returns a Scrapling response for the same selectors. The output goes to `hn-browser-proxy-1.json`, separate from the HTTP result.

 

Hacker News does not need JavaScript for these fields. Prefer HTTP for this task. The browser variant demonstrates the connection setup to reuse when a permitted target requires JavaScript or page interaction; it does not establish access to protected sites. For those workloads, add a target-specific  [wait selector or page action](https://github.com/D4Vinci/Scrapling/blob/v0.4.15/docs/fetching/dynamic.md#wait-conditions)  and verify the required fields.

 

| Decision | Use | Boundary |
| --- | --- | --- |
| Public fields already in HTML | `FetcherSession` | No JavaScript execution |
| Fields appear after browser execution | `DynamicSession` | Browser assets add requests and traffic |
| Related steps need IP continuity | ProxyLane sticky mode | Exit availability can end a session |
| Independent regional checks | ProxyLane rotating mode | Rotation can repeat an IP |

 

A Scrapling session preserves client state such as cookies. A ProxyLane session asks the provider to retain an exit IP while available. They solve different problems: neither a session name nor a browser context reserves a permanent, exclusive IP. Reuse the provider session settings across related steps.

 

## Troubleshoot the failed stage

 

Network exceptions appear as a sanitized exception class, not a raw connection string. That deliberately limits what the terminal reveals. Combine the failed stage with the dashboard test and the  [local proxy checker](https://proxylane.dev/blog/proxy-checker-cli)  before changing settings.

 

| Symptom | Check next | Corrective action |
| --- | --- | --- |
| Authentication failure or  [407](https://proxylane.dev/blog/proxy-error-407) | `Ready to connect`, copied fields, active traffic | Recopy the generated proxy credentials; do not retry unchanged values |
| `ConnectionError` or timeout | Gateway scheme/port, dashboard test, target availability | Check one small IP request; stop repeated failures |
| Transport Layer Security (TLS) certificate failure | System clock, certificate store, actual gateway scheme | Repair trust configuration; keep certificate verification enabled |
| [403](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/403)  or  [429](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/429) | Target rules and any retry instructions | Stop; honor restrictions and any required delay |
| IP check succeeds, extraction is empty | Response content and selectors | Inspect the permitted response; a challenge/login page is not story data |
| Browser executable missing | Chromium installation in this environment | Run the browser installation command from setup |
| Existing output filename | Result from an earlier run | Archive it before rerunning; no request has been made |
| Score is `null` | Listing has no visible score | Keep the missing value; do not replace it with zero |

 

For help, send  [ProxyLane support](https://t.me/proxylane_support)  the Scrapling version, mode, timestamp, selected country/session mode, redacted error, and dashboard test result. Keep passwords, full proxy URLs, and unrelated customer data out of the message. Support availability does not imply a guaranteed response or resolution time.

 

## Judge the connection on the intended workload

 

The 2026-09-25 execution check covered live Hacker News extraction, bounded pagination, and authenticated residential routing in HTTP and Chromium modes. It used an existing test account, not a new paid checkout. It is a functional example, not a speed comparison, a geographic audit, or a guarantee against blocks.

 

For IP reputation context,  [ProxyLane’s dated network snapshot](https://proxylane.dev/#network)  reports 1000 unique US-targeted IPs across 4 gateways on 2026-09-24: mean  [proxycheck.io risk score](https://proxycheck.io/api/)  9.8/100, with 89.5% scoring zero. This is a ProxyLane-run sample using a third-party classifier, not an independent audit or an estimate of fraud or ban probability.

 

Next, choose a small set of permitted URLs from the actual workload. Fix the required fields, country, request budget, and acceptable missing-data rate before testing. Record valid results, failures, retries, and dashboard traffic usage. The  [cost per valid result guide](https://proxylane.dev/blog/proxy-cost-per-successful-request)  explains why a low price per GB alone cannot decide provider fit.

 

## Sources and further reading

- [Scrapling 0.4.15: HTTP requests](https://github.com/D4Vinci/Scrapling/blob/v0.4.15/docs/fetching/static.md)

- [Scrapling 0.4.15: browser sessions](https://github.com/D4Vinci/Scrapling/blob/v0.4.15/docs/fetching/dynamic.md)

- [Hacker News crawl rules](https://news.ycombinator.com/robots.txt)

- [Official Hacker News API](https://github.com/HackerNews/API)

ProxyLane  From $2/GB at 5 TB+

 

## Your next connection Starts here

 

Non-expiring traffic, location targeting and rotating or sticky sessions for your existing tools.

 

[Create an account](https://proxylane.dev/register?interest=proxies)   [View plans](https://proxylane.dev/pricing)

## Keep reading

[Use cases · 3 min read

### Amazon Scraper API vs Proxy: Choose a Product Data Source

Distinguish authorized Amazon APIs, licensed product data and proxy-based page checks by record quality and access rights.

 Read guide](https://proxylane.dev/blog/amazon-scraper-api-vs-proxy)   [Integrations · 5 min read

### Apify Custom Proxy Setup: Use Your Own Residential Route

Connect a buyer-owned proxy to an Apify Actor, keep the session boundary clear, and validate records instead of counting requests.

 Read guide](https://proxylane.dev/blog/apify-custom-proxy)   [Proxy fundamentals · 6 min read

### Australia Residential Proxies: Verify the AU Exit and State-Level Result

Separate Australian egress from en-AU content, AUD pricing, GST display, postcode validation and the state delivery context.

 Read guide](https://proxylane.dev/blog/australia-residential-proxies)

Canonical source: https://proxylane.dev/blog/scrapling-proxy

Documentation index: https://proxylane.dev/llms.txt