Amazon Scraper API vs Proxy: Choose a Product Data Source
Distinguish authorized Amazon APIs, licensed product data and proxy-based page checks by record quality and access rights.
Read guide →Configure Pydoll with HTTP, HTTPS or SOCKS5 routing, preserve browser sessions, and validate the same browser’s exit IP and target content.
Pydoll drives Chromium directly over the DevTools Protocol and exposes an async Python API. That makes its proxy recipe different from a WebDriver setup: add --proxy-server to ChromiumOptions, start Chrome, and use the tab’s request client when you need a response object through the same browser context.
Pydoll’s official proxy guide handles credentials embedded in HTTP or HTTPS proxy URLs through Chrome’s Fetch domain and a 407 Proxy Authentication Required challenge. Chrome does not support authenticated SOCKS5 credentials in the URL. For that case, Pydoll documents SOCKS5Forwarder, which gives Chrome a local unauthenticated endpoint while the forwarder authenticates to the remote proxy.
Use Python 3.10+ and an installed Chrome or Edge browser. This example launches Chrome; Edge requires Pydoll’s Edge class. If Chrome is outside the detected locations, set options.binary_location to its executable path before startup.
python -m pip install -U pydoll-python
Use PROXY_URL, TARGET_URL and EXPECTED_MARKER as environment inputs. A credentialed HTTP URL may look like http://user:pass@proxy.example:8080, but keep real values in a secret binding. The code below parses the IP response from tab.request.get(), then loads the target in that same tab.
import asyncio
import os
from pydoll.browser.chromium import Chrome
from pydoll.browser.options import ChromiumOptions
async def main() -> None:
required = ("PROXY_URL", "TARGET_URL", "EXPECTED_MARKER")
missing = [name for name in required if not os.environ.get(name)]
if missing:
raise SystemExit(f"Missing required environment variables: {', '.join(missing)}")
options = ChromiumOptions()
options.add_argument(f"--proxy-server={os.environ['PROXY_URL']}")
async with Chrome(options=options) as browser:
tab = await asyncio.wait_for(browser.start(), timeout=30)
ip_response = await asyncio.wait_for(tab.request.get("https://httpbin.org/ip"), timeout=25)
if ip_response.status_code < 200 or ip_response.status_code >= 300:
raise RuntimeError(f"Exit-IP request returned {ip_response.status_code}")
ip_payload = ip_response.json()
exit_ip = ip_payload.get("origin")
if not exit_ip:
raise RuntimeError("The same browser returned no exit-IP value")
await tab.go_to(os.environ["TARGET_URL"], timeout=25)
target_response = await asyncio.wait_for(
tab.request.get(os.environ["TARGET_URL"]), timeout=25
)
if target_response.status_code < 200 or target_response.status_code >= 300:
raise RuntimeError(f"Target request returned {target_response.status_code}")
evaluation = await asyncio.wait_for(
tab.execute_script(
"return document.body?.innerText || ''",
return_by_value=True, timeout=5_000,
), timeout=10,
)
result = evaluation.get("result", {})
if "error" in evaluation or "exceptionDetails" in result:
raise RuntimeError("Browser script evaluation failed")
body = result.get("result", {}).get("value")
if not isinstance(body, str) or not body.strip():
raise RuntimeError("Browser script returned no text")
if os.environ["EXPECTED_MARKER"] not in body:
raise RuntimeError("Target marker was not found in the rendered page")
print({"exit_ip_present": True, "followup_request_status": target_response.status_code, "content_valid": True})
asyncio.run(main())
go_to(timeout=25) bounds navigation in seconds. The subsequent browser fetch is a separate request, made after navigation to keep it same-origin; its status does not prove the earlier navigation’s status. CORS still applies if redirects cross origins. The marker comes from the rendered document. execute_script() returns an EvaluateResponse: text is under result.result.value, while JavaScript exceptions appear under result.exceptionDetails. Its timeout is milliseconds; asyncio.wait_for bounds request and script waits in seconds. The Chrome context manager attempts cleanup on exit; give the worker a process deadline for startup or teardown failures.
Start with the residential versus datacenter guide when choosing a route. Use the sticky-session guide before combining login cookies with a rotating endpoint.
| Workload | Proxy choice | Session policy | Acceptance check |
|---|---|---|---|
| Stable, repeatable service checks | Datacenter | Fresh context per case | Status and expected field |
| Region-specific UI research | Residential | Sticky route for one browser flow | Exit route and page region agree |
| Login followed by API calls | Residential or static ISP where permitted | Reuse the same tab or context | Cookies and API response remain valid |
| Parallel geography comparison | One proxy per browser context | create_browser_context(proxy_server=...) |
Each context reports separately |
Pydoll contexts can carry different proxies in one browser. That is useful for isolated region checks, but cookies and route policy still belong together. Do not rotate the route mid-login unless the provider and target workflow explicitly support it.
| Symptom | Layer | Next check |
|---|---|---|
| HTTP proxy returns 407 | Proxy auth | Confirm credentials are in the HTTP/HTTPS URL and inspect the provider scheme |
| SOCKS5 auth fails silently | Chrome limitation | Use the documented SOCKS5Forwarder local bridge or an unauthenticated endpoint |
| IP echo succeeds but target status fails | Target or route | Compare endpoint reachability, status and target permissions |
| Status is successful but marker is absent | Rendering or parser | Inspect the rendered body, redirects and marker choice |
| Different tabs share unexpected state | Context isolation | Create separate browser contexts with explicit proxy routes |
| Browser remains after an exception | Lifecycle | Keep the Chrome context manager around the complete job |
Pydoll includes network controls and humanized interactions, but a proxy does not guarantee acceptance by a target. The useful record is the one whose route, response status and required content fields all pass under the permissions for your workload.
Distinguish authorized Amazon APIs, licensed product data and proxy-based page checks by record quality and access rights.
Read guide →Connect a buyer-owned proxy to an Apify Actor, keep the session boundary clear, and validate records instead of counting requests.
Read guide →Separate Australian egress from en-AU content, AUD pricing, GST display, postcode validation and the state delivery context.
Read guide →