Amazon Scraper API vs Proxy: Choose a Product Data Source
Distinguish authorized Amazon APIs, licensed product data and proxy-based page checks by record quality and access rights.
Read guide →Configure a DrissionPage Chromium browser with a proxy, keep a session coherent, and validate the page your parser actually receives.
DrissionPage combines Chromium control with a packet-oriented mode. For a browser workflow, the proxy belongs in ChromiumOptions before the browser starts. The useful proof is a little stricter than “the window opened”: read the exit IP from the same browser, load a permitted target, and confirm a marker your parser needs.
This recipe targets the documented DrissionPage 4.0.5.6 Chromium API, pinned below. Its credential limitation must not be generalized to newer development versions. It is a good fit for Python teams that want simple element operations, multiple tabs or a path that can move between browser control and requests. The browser still owns navigation and rendering; the proxy only supplies the network route.
python -m pip install "DrissionPage==4.0.5.6"
Use PROXY_SERVER as a secret environment value such as http://proxy.example:8080. The official set_proxy() contract is protocol://ip:port, is applied once at browser startup, and does not support a username or password in that setting. Do not put credentials into the example and expect DrissionPage to handle them.
For an authenticated HTTP/HTTPS upstream, Apify’s proxy-chain documents anonymizeProxy({url, port}): it starts an unauthenticated local proxy and returns its URL. Install it with npm install proxy-chain in a Node.js helper, pass the upstream URL from a secret environment variable, and supply the returned http://127.0.0.1:<port> as PROXY_SERVER to the Python child process. Keep the helper alive until Python exits; call closeAnonymizedProxy(localUrl, true) in its finally block. This is a separate bridge, not native DrissionPage credential support. Diagnose upstream rejection with the 407 authentication guide.
import json
import os
from DrissionPage import ChromiumOptions, ChromiumPage
REQUIRED = ("PROXY_SERVER", "TARGET_URL", "EXPECTED_MARKER")
missing = [name for name in REQUIRED if not os.environ.get(name)]
if missing:
raise SystemExit(f"Missing required environment variables: {', '.join(missing)}")
options = ChromiumOptions().auto_port()
options.set_proxy(os.environ["PROXY_SERVER"])
page = ChromiumPage(options)
try:
page.get("https://api.ipify.org?format=json", show_errmsg=True, retry=0, timeout=25)
ip_payload = json.loads(page.run_js("return document.body.innerText", timeout=5))
exit_ip = ip_payload.get("ip") or ip_payload.get("origin")
if not exit_ip:
raise RuntimeError("The same browser returned no exit-IP value")
page.get(os.environ["TARGET_URL"], show_errmsg=True, retry=0, timeout=25)
body = page.html
if not body or os.environ["EXPECTED_MARKER"] not in body:
raise RuntimeError("Target marker was not found in the received page")
print({"exit_ip_present": True, "content_valid": True})
finally:
page.quit(timeout=5, force=True)
get() accepts a timeout in seconds; both navigations use 25 seconds and retries are zero and leaves the decision visible to the caller. Give the job an outer worker deadline as well. page.quit() closes the browser; page.close() closes the current tab and can leave a headless process running.
| Workload | Proxy choice | Session policy | Acceptance check |
|---|---|---|---|
| Fast, repeatable QA against your own service | Datacenter | Fresh browser per case | Expected marker and known record |
| Region-sensitive content research | Residential | One browser through the flow | Exit region and target fields agree |
| Login, pagination or checkout simulation | Residential or static ISP where permitted | Keep one browser and sticky provider session | Cookies and final state remain usable |
| Independent URLs with no shared state | Datacenter or rotating pool | Separate browser lifecycle | Each result validates independently |
DrissionPage’s proxy is browser-wide and fixed at startup. If the provider rotates on every request, a multi-step flow may cross identities even though the browser object stays alive. Use the sticky-session selection guide when cookies, login state or a sequence of pages must remain together. For separate regions, launch separate browsers with separate options and data paths rather than changing the proxy after startup.
| Symptom | Layer | Next check |
|---|---|---|
set_proxy() rejects the value |
Configuration | Use a scheme, host and port only; move auth to a documented bridge |
| Browser starts but IP page fails | Proxy or runner network | Check endpoint protocol, port and worker egress |
| IP is present but marker is absent | Target or parser | Inspect redirects, permissions and the expected field |
| Later step sees a different route | Session policy | Use provider-supported stickiness and one browser lifecycle |
| Process remains after a run | Cleanup | Call page.quit() and distinguish it from page.close() |
The IP response proves the browser reached an echo service through some route. The marker proves only that this run received the requested content. Keep those signals separate in logs, and do not treat either as a guarantee that a target will accept automation. Use the target’s rules and your parser’s required fields as the real success condition.
Distinguish authorized Amazon APIs, licensed product data and proxy-based page checks by record quality and access rights.
Read guide →Connect a buyer-owned proxy to an Apify Actor, keep the session boundary clear, and validate records instead of counting requests.
Read guide →Separate Australian egress from en-AU content, AUD pricing, GST display, postcode validation and the state delivery context.
Read guide →