Python web scraping tools: ScraperAPI, Playwright and ProxyLane residential proxies (2026 review)

Modern Python web scraping stacks are strongest when they combine three layers: lightweight HTTP clients (Requests + BeautifulSoup), headless browsers (Playwright), and resilient access infrastructure (ScraperAPI or high‑quality residential proxies such as ProxyLane).

This article shows how those tools fit together, how to plug in proxies for geo‑specific and anti‑bot‑resistant targets, and how to verify that your pipeline really reaches the "local web" you care about.

ProxyLane proxy service review

As of November 2026, ProxyLane positions itself as a premium residential proxy provider built around verifiable quality and non‑expiring bandwidth.

Vendor‑reported highlights (from public pages as of 2026‑11):

• Residential IP pool: 28M+ real P2P IPs (provider claim).
• Coverage: 195 countries, with city/ZIP/ISP targeting (provider claim).
• Session control: sticky sessions up to 72 hours (provider claim).
• Protocols: HTTP(S) and SOCKS5 (provider claim).
• Economics: traffic sold per GB with bandwidth that never expires (provider claim).
• Risk profile: 98.7% of sampled IPs rated low risk in a Scamalytics snapshot (provider claim, as of the referenced benchmark).
• Performance: ~2.10s average response in an independent benchmark (provider claim, citing an external test).

For resilient scraping, the key strengths are:

• Granular geo targeting for Google Maps, local SERPs, ecommerce, and TikTok research.
• Long‑lived sticky IPs for logged‑in, multi‑step flows.
• Non‑expiring traffic that matches batch‑style data collection.

The rest of this article assumes a ProxyLane‑style configuration (city/ISP targeting, rotating vs sticky modes, HTTP/SOCKS5) as the proxy layer behind Python tools.

ProxyLane residential proxies pricing features review

ProxyLane pricing per GB

ProxyLane uses per‑GB pricing with non‑expiring traffic.

From public pricing pages as of 2026‑11 (vendor‑reported):

• Entry trial: 350 MB for $1.95.
• Base packages: starting around $2/GB at scale, with smaller packages advertised from around $6.50 for 1 GB.
• Non‑expiring bandwidth: unused traffic does not expire.

A simplified pricing snapshot (illustrative based on public ranges, not a binding quote):

• 1 GB — approx. $6.50 (effective $6.50/GB).
• 5 GB — approx. $22.50 (effective $4.50/GB).
• 20 GB — approx. $50 (effective $2.50/GB).

Because traffic never expires, teams can:

• Buy bandwidth aligned to a specific batch or experiment.
• Park unused GB for future runs without “use‑it‑or‑lose‑it” pressure.
• Model cost per validated result instead of worrying about monthly reset dates.

Feature‑wise, ProxyLane’s pricing emphasizes:

• Bandwidth rollover by default (no expiry).
• City/ISP targeting included, not upsold as a separate geo tier.
• Sticky sessions available without separate fees.

Best residential proxies 2026 — ProxyLane vs competitors

For AI‑driven and Python‑based scraping in 2026, engineers typically compare residential proxy providers on:

• Pool size.
• Geo granularity.
• Sticky session support.
• Price per GB (or per request).

Below is a high‑level comparison using vendor‑reported numbers as of November 2026. Always confirm on each provider’s site before buying; details change frequently.

Notes (as of 2026‑11):

• Bright Data numbers and pricing come from their residential proxies and pay‑as‑you‑go pages (vendor‑reported).
• ProxyEmpire’s pool sizes, country count, and $1.50/GB figure are from their pricing page (vendor‑reported).
• ScraperAPI’s $49/month Hobby plan and 100,000 credits are from their pricing page; they do not bill per GB.
• ProxyLane’s numbers are from their public site; the exact entry tiers should be rechecked before purchase.

For most Python teams, a practical pattern is:

• Use ProxyLane or similar per‑GB residential proxies when you need fine geo control, long sessions, and direct HTTP/SOCKS integration.
• Use ScraperAPI when you prefer a managed scraping API that handles rotation, CAPTCHAs, and retries for you.

Residential proxies for Google Maps scraping

Google Maps is a clear example of the "local web": results depend heavily on:

• City or ZIP.
• Device language/locale.
• ISP/ASN and perceived risk.

A Google Maps scraping pipeline often needs:

1. Precise geo targeting — e.g., results as seen from Berlin on a consumer ISP.
2. Browser automation — Playwright, because Maps is JS‑heavy.
3. Residential proxies — to mimic consumer internet access.

A generic ProxyLane‑style username pattern for geo and ISP targeting looks like:

username: country=DE;city=Berlin;zip=10115;isp=Telekom;session=my-maps-run
password: <your_proxy_password>

Or in query‑param style when supported:

http://username:password@proxyhost:port?country=DE&city=Berlin&zip=10115&isp=Telekom

Core steps for verification:

1. Start a Playwright browser through the proxy.
2. Visit a geo diagnostic endpoint like https://ipinfo.io/json.
3. Confirm that the reported city/region/ASN match your intended parameters.

We’ll see Playwright examples later in the article.

Residential proxies for TikTok scraping

TikTok’s feed, trending sounds, and search results are heavily localized.

Common needs:

• One sticky IP per profile (72‑hour sessions are useful).
• Country/city/ISP alignment to emulate real mobile or home connections.
• Timezone and language matching.

A ProxyLane‑style username for a TikTok profile might be:

username: country=US;city=Austin;isp=Comcast;session=tiktok-profile-42
password: <proxy_password>

Operational tips:

• Use sticky sessions for login and interactions.
• Avoid reusing sessions across unrelated accounts.
• Rotate sessions slowly and respect TikTok’s rate limits and terms.

Python foundations: Requests and BeautifulSoup

Requests

Requests is still the baseline HTTP client for Python scraping.

As of 2026‑11, PyPI reports Requests under requests with official docs confirming:

• Python 3.10+ support.
• Built‑in support for:
  • Sessions (requests.Session).
  • TLS verification.
  • Proxy configuration via proxies argument.

A basic proxy configuration:

import requests

proxies = {
    "http": "http://username:password@proxyhost:port",
    "https": "http://username:password@proxyhost:port",
}

resp = requests.get("https://example.com", proxies=proxies, timeout=30)
print(resp.status_code)

BeautifulSoup

BeautifulSoup (package beautifulsoup4) is an HTML/XML parser, not a browser.

PyPI stats as of 2026‑11 report recent versions like 4.15.x with heavy usage, and official docs emphasize:

• Parsing HTML into a navigable tree.
• CSS‑like selectors via .select().

Example with Requests + BeautifulSoup:

import requests
from bs4 import BeautifulSoup

resp = requests.get("https://example.com", timeout=30)
soup = BeautifulSoup(resp.text, "html.parser")

for item in soup.select(".product-card"):
    title = item.select_one(".title").get_text(strip=True)
    price = item.select_one(".price").get_text(strip=True)
    print(title, price)

This stack works best when:

• The page is mostly static.
• The data is already in the HTML DOM.
• You don’t need JS rendering.

Playwright: browser automation for modern targets

Playwright is the go‑to open‑source browser automation layer for JS‑heavy sites.

As of 2026‑11, playwright on PyPI lists support for:

• Chromium, Firefox, WebKit.
• Headless or headed execution.
• Mobile emulation.
• Network APIs to monitor and modify traffic.

Residential proxies for Playwright Python

You can configure HTTP or SOCKS5 proxies in Playwright at browser launch.

Basic HTTP proxy with authentication:

from playwright.sync_api import sync_playwright

proxy_server = "http://proxyhost:port"
proxy_username = "country=DE;city=Berlin;session=test-1"
proxy_password = "your_proxy_password"

with sync_playwright() as p:
    browser = p.chromium.launch(
        headless=True,
        proxy={
            "server": proxy_server,
            "username": proxy_username,
            "password": proxy_password,
        },
    )

    page = browser.new_page()
    page.goto("https://ipinfo.io/json", timeout=60000)
    print(page.text_content("pre"))
    browser.close()

SOCKS5 example:

from playwright.sync_api import sync_playwright

proxy_server = "socks5://proxyhost:port"
proxy_username = "country=US;city=Chicago;session=socks-demo"
proxy_password = "your_proxy_password"

with sync_playwright() as p:
    browser = p.firefox.launch(
        headless=True,
        proxy={
            "server": proxy_server,
            "username": proxy_username,
            "password": proxy_password,
        },
    )
    page = browser.new_page()
    page.goto("https://ipinfo.io/json", timeout=60000)
    print(page.text_content("pre"))
    browser.close()

How to use rotating residential proxies with Playwright (ProxyLane example)

Rotation is useful for independent requests (e.g., list pages, non‑logged‑in scrapes).

A common pattern:

• Use one browser context per proxy.
• Rotate proxy credentials for each context.
• Close contexts after a batch to avoid building suspicious history.

Example rotation across contexts:

from playwright.sync_api import sync_playwright

proxy_hosts = [
    "http://username1:password1@proxyhost:port",
    "http://username2:password2@proxyhost:port",
    "http://username3:password3@proxyhost:port",
]

urls = [
    "https://example.com/page1",
    "https://example.com/page2",
    "https://example.com/page3",
]

def parse_page(page):
    return page.title()

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)

    results = []
    for proxy_url, url in zip(proxy_hosts, urls):
        # Extract server/credentials from proxy URL if needed; simple example:
        server = proxy_url

        context = browser.new_context(
            proxy={
                "server": server,
                # For providers like ProxyLane you would pass username/password
                # via separate fields rather than embedding in the URL if required.
            }
        )
        page = context.new_page()

        # Backoff-friendly timeout
        try:
            page.goto(url, timeout=70000)  # ScraperAPI recommends ~70s for hard targets
            results.append(parse_page(page))
        except Exception as e:
            print("Error fetching", url, e)
        finally:
            context.close()

    browser.close()

print(results)

Operational guidance:

• Use timeouts of 45–70 seconds on hard targets, following ScraperAPI’s recommendation of ~70s for difficult sites (docs as of 2026‑11).
• Implement retry with backoff (e.g., 2–3 attempts with exponential waits) when you see transient 5xx or network errors.
• Avoid over‑parallelization that could overwhelm your proxy pool.

ScraperAPI vs residential proxies (Python perspective)

ScraperAPI is a managed scraping API that bundles:

• Premium proxies.
• JS rendering.
• Automatic retries.
• CAPTCHA handling.

As of 2026‑11, ScraperAPI’s docs and pricing report:

• 7‑day trial with 5,000 API credits.
• Hobby plan at $49/month for 100,000 credits and 20 concurrent threads.
• Larger tiers up to 3,000,000 credits at $299/month.
• Billing based on request credits, not GB.

In Python, ScraperAPI looks like:

import requests

API_KEY = "your_scraperapi_key"
url = "https://example.com/hard-target"

params = {
    "api_key": API_KEY,
    "url": url,
    "render": "true",  # JS rendering
}

resp = requests.get("https://api.scraperapi.com/", params=params, timeout=70)
print(resp.status_code)
print(resp.text)

When to prefer ScraperAPI:

• You want managed unblocking and don’t want to manage rotation, CAPTCHAs, or browser automation yourself.
• You prefer predictable cost per request.

When to prefer residential proxies like ProxyLane:

• You need direct control over your tooling (Playwright, Requests, custom frameworks).
• You want to integrate proxies into AI agents or existing infrastructure.
• You care about non‑expiring bandwidth and custom session policies.

Verifying location, ASN, and risk: a repeatable checklist

Any claim about geo targeting, residential IP quality, or low risk should be validated.

Here is a concise checklist you can run manually or in Python.

1. Fetch a sample of IPs

If your provider offers an API or "sample IP" endpoint, pull 1,000–2,000 IPs.

Example (generic curl):

curl -u "username:password" "https://proxy-provider.example.com/sample-ips?limit=2000" > ips.json

2. GeoIP and ASN checks

Use services like MaxMind, ipinfo, or ipapi.

Example with ipinfo.io in Python:

import requests

ips = ["1.2.3.4", "5.6.7.8"]  # from your sample

for ip in ips:
    data = requests.get(f"https://ipinfo.io/{ip}/json", timeout=10).json()
    print(ip, data.get("city"), data.get("region"), data.get("country"), data.get("org"))

Check that:

• Cities/countries match the provider’s targeting claims.
• org/ASN suggests consumer ISPs rather than data centers.

3. Risk and abuse signals

Use tools like Scamalytics or abuse databases to sample risk ratings:

• Confirm that most IPs are rated low risk.
• Flag any concentration of high‑risk or data‑center tags.

4. Median latency and reliability

Measure end‑to‑end latency to targets via the proxy.

Example using time and curl:

for i in {1..50}; do
  time curl -x "http://username:password@proxyhost:port" \
       -o /dev/null -s "https://example.com/health"
done

Compute:

• Median response time.
• Error rate (timeouts, 5xx).

5. Assert target‑visible geolocation via Playwright

Use Playwright to see what the target believes your location is.

from playwright.sync_api import sync_playwright
import json

proxy_server = "http://proxyhost:port"
proxy_username = "country=GB;city=London;session=verify-geo"
proxy_password = "your_proxy_password"

with sync_playwright() as p:
    browser = p.chromium.launch(
        headless=True,
        proxy={
            "server": proxy_server,
            "username": proxy_username,
            "password": proxy_password,
        },
    )
    page = browser.new_page()
    page.goto("https://ipinfo.io/json", timeout=60000)
    data = json.loads(page.text_content("pre"))

    assert data["country"] == "GB"
    assert data["city"] in {"London", "Greater London"}

    print("Verified location:", data)
    browser.close()

This assertion‑style check ensures that geo parameters in your proxy credentials match what downstream sites actually see.

Building a resilient Python scraping pipeline

To tie everything together for modern targets like Maps and TikTok:

1. Classification layer

   • Use simple heuristics to decide whether a URL needs:
     • Requests + BeautifulSoup.
     • Playwright + proxies.
     • ScraperAPI.

2. Proxy strategy

   • Rotating residential proxies:
     • Independent pages.
     • Non‑logged‑in crawls.
   • Sticky residential proxies:
     • Logged‑in flows.
     • TikTok profiles.
     • Google My Business management.

3. Tooling integration

   • HTTP flows:
     • Configure proxies in Requests.
   • JS‑heavy flows:
     • Use Playwright with per‑context proxy configuration.
   • Managed flows:
     • Route via ScraperAPI when you want its automatic unblocking.

4. Verification and monitoring

   • Regularly sample IPs and geo.
   • Track success rate per GB or per request.
   • Keep latency and error metrics.

5. Ethics and compliance

   • Confirm that your proxy provider uses opt‑in participation and respects withdrawal.
   • Respect target‑site terms of service and applicable law.

The Imperva/Thales Bad Bot report (2026 edition) notes that automated traffic reached about 53% of all web traffic in 2025, with 27% of bot attacks targeting APIs (as of their public report). That environment makes resilience and responsible behavior non‑optional.

When you’re ready to assemble a stack like this, a provider that offers HTTP/SOCKS5 proxies, city/ISP targeting, sticky sessions, and non‑expiring bandwidth — like ProxyLane at proxylane.dev — fits naturally into Python and Playwright workflows without needing proprietary SDKs.

FAQ

ProxyLane residential proxies pricing features review

ProxyLane uses per‑GB pricing with non‑expiring bandwidth, plus fine‑grained geo targeting and sticky sessions.

As of 2026‑11 (vendor‑reported):

• Entry trial: 350 MB for $1.95.
• Packages starting around $6.50 for 1 GB, with effective per‑GB rates dropping toward $2.50/GB at higher tiers.
• Traffic never expires.
• HTTP/SOCKS5 support and country/city/ZIP/ISP targeting are included rather than add‑ons.

Always check current pricing on the site before purchase.

Buy residential proxies 1 GB package unused traffic never expires

Yes, you can buy 1 GB packages of residential proxies from providers like ProxyLane, where unused traffic does not expire (as of their public pages in 2026‑11).

This model is well suited to:

• Occasional batches.
• Experimentation and prototyping.
• AI retrieval workflows that run irregularly.

Best residential proxies web scraping 2026 ethical

In 2026, the “best” residential proxies for ethical web scraping are those that can demonstrate:

• Opt‑in participation from device owners.
• Clear withdrawal controls and an abuse‑response process.
• Transparent geo behavior (verifiable via IPinfo/MaxMind).
• Reasonable risk scores from third‑party tools like Scamalytics.

ProxyLane’s trust‑center messaging (as of 2026‑11) explicitly highlights informed participation and user responsibility for legal compliance.

ScraperAPI vs residential proxies Python

ScraperAPI:

• Managed scraping API.
• Bills per request credit.
• Automatically handles rotation, CAPTCHAs, and retries.

Residential proxies (ProxyLane‑style):

• Raw HTTP/SOCKS access.
• Bill per GB (often with rollover or non‑expiring traffic).
• Require you to manage rotation, browsers, and error handling.

In Python, ScraperAPI is a single endpoint call, while residential proxies integrate directly into Requests, Playwright, and your own stack.

Go spider, scrape web data Python

To "go spider" and scrape web data in Python responsibly:

• Start with Requests + BeautifulSoup for static HTML.
• Use Playwright for JS‑heavy and interaction‑heavy pages.
• Add residential proxies for geo‑specific and anti‑bot‑resistant targets.
• Consider ScraperAPI when you want managed unblocking.
• Monitor success rates and respect site terms and legal requirements.

ProxyLaneTraffic never expires

Premium residential proxies
From $2/GB

Scale your workflows with the tools you already use. Rotating or sticky sessions up to 72 hours.

Try 350 MB for $1.95 View plans