Amazon Scraper API vs Proxy: Choose a Product Data Source
Compare seller APIs, licensed Amazon product data and proxy page checks by record, access scope and reuse rights.
Read guideSet an authenticated Chromium proxy in SeleniumBase, confirm the browser exit and target response, and clean up temporary credentials.

SeleniumBase configures a browser proxy with its Driver interface or test-run options. For an authenticated Chromium route, bind credentials at runtime, keep SeleniumBase’s generated auth extension inside a private temporary directory, and accept a result only after the same browser verifies its exit response and target content.
python -m pip install seleniumbase==4.54.12 selenium==4.49.0.ip field, and a permitted target page with a stable text marker.Warning: Browser pages may request subresources, and even GET routes can change target state. Keep production disabled; start with a staging or sandbox destination and set a request or spend cap. SeleniumBase may generate an extension containing proxy credentials. The example confines it to a temporary working directory and deletes that directory after closing the driver; keep CI artifacts and other users out of that directory while the run is active.
Save this complete example as probe.py. Bind PROXY_HOSTPORT, PROXY_USER, and PROXY_PASSWORD through a secrets manager or protected runtime settings; omit both credential variables for an unauthenticated route. SeleniumBase 4.54.12 inserts these values directly into double-quoted JavaScript strings in its generated extension. This format rejects :, @, double quotes, backslashes, and nonprintable control characters before browser startup. It does not percent-decode credentials, so percent-encoding does not make an otherwise unsupported value safe or equivalent. Bind EGRESS_URL, EXPECTED_EXIT, TARGET_URL, and EXPECTED_MARKER to approved values. The egress and target URLs must not contain credentials.
import ipaddress
import json
import os
import tempfile
import time
from pathlib import Path
from urllib.parse import urlsplit
from seleniumbase import Driver
PROXY_HOSTPORT = os.environ.get("PROXY_HOSTPORT", "")
PROXY_USER = os.environ.get("PROXY_USER", "")
PROXY_PASSWORD = os.environ.get("PROXY_PASSWORD", "")
EGRESS_URL = os.environ.get("EGRESS_URL", "")
EXPECTED_EXIT = os.environ.get("EXPECTED_EXIT", "")
TARGET_URL = os.environ.get("TARGET_URL", "")
EXPECTED_MARKER = os.environ.get("EXPECTED_MARKER", "")
if not all((PROXY_HOSTPORT, EGRESS_URL, EXPECTED_EXIT, TARGET_URL, EXPECTED_MARKER)):
raise SystemExit("Set proxy, egress, expected exit, target, and marker settings")
parsed_proxy = urlsplit("http://" + PROXY_HOSTPORT)
if (
not parsed_proxy.hostname
or parsed_proxy.username
or parsed_proxy.password
or parsed_proxy.path
or parsed_proxy.query
or parsed_proxy.fragment
or not parsed_proxy.port
or not 1 <= parsed_proxy.port <= 65535
):
raise SystemExit("PROXY_HOSTPORT must be host:port without a scheme or credentials")
if bool(PROXY_USER) != bool(PROXY_PASSWORD):
raise SystemExit("Set both proxy credentials or neither")
UNSUPPORTED_PROXY_CREDENTIAL_CHARS = {":", "@", '"', "\\"}
if PROXY_USER and any(
character in UNSUPPORTED_PROXY_CREDENTIAL_CHARS or not character.isprintable()
for character in PROXY_USER + PROXY_PASSWORD
):
raise SystemExit("Use printable credentials without colon, at-sign, double quote, or backslash")
proxy = f"{PROXY_USER}:{PROXY_PASSWORD}@{PROXY_HOSTPORT}" if PROXY_USER else PROXY_HOSTPORT
try:
for name, value in (("EGRESS_URL", EGRESS_URL), ("TARGET_URL", TARGET_URL)):
parsed = urlsplit(value)
if parsed.scheme not in {"http", "https"} or not parsed.hostname or parsed.username or parsed.password:
raise SystemExit(f"{name} must be an HTTP(S) URL without credentials")
parsed.port
ipaddress.ip_address(EXPECTED_EXIT)
except ValueError:
raise SystemExit("Use valid HTTP(S) URLs and a valid EXPECTED_EXIT IP address") from None
def current_document_status(driver):
frame = driver.execute_cdp_cmd("Page.getFrameTree", {})["frameTree"]["frame"]
deadline = time.monotonic() + 5
while time.monotonic() < deadline:
for entry in driver.get_log("performance"):
event = json.loads(entry["message"])["message"]
params = event.get("params", {})
if (
event.get("method") == "Network.responseReceived"
and params.get("type") == "Document"
and params.get("frameId") == frame["id"]
and params.get("loaderId") == frame.get("loaderId")
):
return int(params["response"]["status"])
time.sleep(0.05)
return None
original_directory = Path.cwd()
with tempfile.TemporaryDirectory(prefix="seleniumbase-proxy-") as temporary_directory:
os.chdir(temporary_directory)
driver = None
try:
driver = Driver(
browser="chrome",
headless2=True,
proxy=proxy,
log_cdp_events=True,
)
driver.set_page_load_timeout(15)
driver.get(EGRESS_URL)
egress_status = current_document_status(driver)
try:
observed_exit = json.loads(driver.find_element("tag name", "body").text)["ip"]
except (KeyError, json.JSONDecodeError):
raise RuntimeError("Egress endpoint did not return JSON with an ip field") from None
if egress_status != 200 or observed_exit != EXPECTED_EXIT:
raise RuntimeError("Proxy egress check did not match the expected value")
driver.get(TARGET_URL)
target_status = current_document_status(driver)
target_body = driver.find_element("tag name", "body").text
target_origin = urlsplit(TARGET_URL)
final_origin = urlsplit(driver.current_url)
if (final_origin.scheme, final_origin.netloc) != (target_origin.scheme, target_origin.netloc):
raise RuntimeError("Target navigation left the configured origin")
if target_status != 200 or EXPECTED_MARKER not in target_body:
raise RuntimeError("Target status or required page marker did not match")
print({"egress_match": True, "target_status": target_status, "marker_match": True})
except Exception as error:
raise SystemExit(f"Probe failed ({type(error).__name__}); inspect local logs without sharing raw events") from None
finally:
try:
if driver is not None:
driver.quit()
finally:
os.chdir(original_directory)
Run python probe.py. SeleniumBase 4.54.12 accepts proxy on Driver; the official proxy setup page also documents proxy_pac_url, bypass lists, and the --multi-proxy pytest option. This example uses one route and one browser, so its test result is attributable to that session.
A successful fixture run prints:
{'egress_match': True, 'target_status': 200, 'marker_match': True}
egress_match means the egress JSON IP exactly matched EXPECTED_EXIT; target_status is the top-level target response status; and marker_match means the body contained EXPECTED_MARKER. The fixture returned a reserved documentation IP, not a real proxy exit.
The helper reads Chrome performance events and selects the final top-level document response by frame and loader identifiers. WebDriver’s get() does not return an HTTP response. Do not print or publish raw events: they can contain URLs and request details. The exit check compares the exact IP string parsed from JSON; the target check compares origin, status, and a required text marker before reporting success.
SeleniumBase documents the username:password@host:port form for authenticated Chromium proxy requests. Its installed 4.54.12 implementation creates a Chrome extension to answer proxy authentication challenges, and the generated extension code can contain those credentials. The extension listens for auth challenges on all URLs, so avoid a target that requires a different HTTP-auth identity in this browser session. The example changes into an empty temporary directory before creating the driver, closes the driver, restores the working directory, and then removes the temporary files. Use a protected temporary filesystem and prevent log/artifact collection during the run. If that storage boundary cannot be enforced, use provider-side IP allowlisting or another integration with an explicit secret-storage contract.
Do not pass credentials in a pytest --proxy=... command, commit them, or add them to the target’s Authorization header. Authenticated proxy support is documented for Chromium; do not assume the same path works in Firefox. Chrome also does not support proxy authentication for SOCKS5. An HTTP proxy connection is not encrypted to the proxy; HTTPS target connections use CONNECT and keep target TLS intact, while an HTTPS proxy protects the browser-to-proxy hop. Confirm the protocol and auth method with the endpoint operator.
| Evidence | Likely layer | Next check |
|---|---|---|
| Missing environment value or rejected proxy format | Test configuration | Check secret binding names and the documented separators without printing values |
| Browser or driver fails to start | Local setup | Check Chrome, Selenium Manager, and a compatible driver |
| Exit navigation fails or returns a different IP | Proxy connection or route | Check host, port, protocol, provider exit, and runner reachability |
| Proxy returns 407 | Proxy authentication | Check credentials and Chromium support; avoid sharing raw network events |
| Target status is not 200 | Target response | Check target policy, redirects, and the staging response |
| Target returns 200 without the marker | Page content or parser | Check the expected text and the actual selector/render timing |
The 15-second page-load timeout bounds each navigation wait, not the total job or all background requests. This example has 2 explicit navigations; Chrome can still request page resources. Add a separate job deadline and provider usage cap when the surrounding test runner needs a total bound.
The proof used SeleniumBase 4.54.12, Selenium 4.49.0, Python 3.14.6, Chrome 154.0.8037.58, and a loopback-only authenticated proxy fixture. The fixture returned the reserved test address 198.51.100.7; it did not measure public or geographic egress. Its successful run confirms the auth extension, same-browser egress check, main-document status, and body-marker logic against that fixture.
Scale your workflows with the tools you already use. Rotating or sticky sessions up to 72 hours.
Compare seller APIs, licensed Amazon product data and proxy page checks by record, access scope and reuse rights.
Read guideChoose the right anti-detect browser proxy guide, map the shared verification steps, and keep account access separate from the network route.
Read guideOur picks for market research: ProxyLane for small, intermittent projects, plus options for regional tracking, monthly collection and managed extraction.
Read guide