Reduce Browser Proxy Bandwidth Without Losing the Record

Browser traffic becomes expensive when a data job loads every image, font, analytics call and video even though the record needs one HTML field. Blocking resources can reduce waste, but an aggressive rule can remove an API response or script that contains the value you are collecting. Measure bytes and accepted records together.

Define the record before blocking anything

Write the acceptance rule first: target URL, required fields, final URL, response status and a parser check. Keep a small allowed sample. A response that is smaller is useful only when the required record still passes validation.

Apply a narrow Playwright route rule

Playwright’s network guide documents request interception. This example blocks only obvious presentation assets and keeps API calls available for inspection:

from playwright.sync_api import sync_playwright

# Presentation assets only; inspect the target before adding more types.
BLOCKED = {"image", "font", "media", "stylesheet"}

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()

    def handle(route):
        if route.request.resource_type in BLOCKED:
            route.abort()
        else:
            route.continue_()

    page.route("**/*", handle)
    # Playwright timeout values are milliseconds: 25_000 = 25 seconds.
    response = page.goto("https://example.com/", wait_until="domcontentloaded", timeout=25_000)
    text = page.text_content("body") or ""
    result = {
        "status": response.status if response else None,
        "url": page.url,
        "valid": "Example Domain" in text,
    }
    print(result)
    browser.close()

Run the same URL once without interception and once with the rule. Save request count, transfer bytes if your runtime exposes them, duration, and the validation result. Do not print request URLs containing tokens. If the target’s record arrives through XHR, add a specific allow rule after inspecting the documented interface rather than allowing every request by default. stylesheet is included here because it is a presentation resource; remove it when the page's layout or selector behavior depends on CSS, then rerun the parity check.

Use this worksheet for each paired run:

Keep the URL cohort, browser version, session policy and pacing identical. A blank Bytes cell means the runtime did not expose a trustworthy transfer measurement; do not replace it with an estimate. Fill Failure category with a concrete value such as timeout, parser_miss, challenge or duplicate, then inspect the accepted-record count before changing the rule.

Calculate cost per accepted record

For each paired run, keep these columns:

cost per accepted record = billable transfer and run cost / distinct valid records

Count a record only after its required fields pass. A block page with fewer bytes is still a failed record. A second successful retry for the same product is not a second distinct record. The cost per successful request guide shows the full denominator and retry treatment.

When blocking makes the result worse

Remove a block rule when the page depends on an image URL for the identifier, when a script performs the permitted navigation, or when an API response contains the structured field. A proxy does not fix a parser that expects a resource you deliberately removed. A smaller transfer can also hide a challenge page, so inspect final URL, title and required content instead of optimizing bytes alone.

Use the same target cohort, browser image, proxy session policy and pacing for the before and after runs. If a route is required for geography or worker egress, keep that variable fixed while changing resource rules. Record the browser workflow, traffic unit, target category, current route and recent failure before requesting a matched route test.