Amazon Scraper API vs Proxy: Choose a Product Data Source
Distinguish authorized Amazon APIs, licensed product data and proxy-based page checks by record quality and access rights.
Read guideUse Guzzle with an explicit HTTP proxy, secret-safe credentials, streamed response limits, and clear proxy-versus-target failure handling.
Guzzle puts proxy routing in the request options, which is convenient when a PHP worker has different routes for different jobs. A useful integration still needs a credential boundary, an explicit NO_PROXY policy, body and time limits, and an acceptance check that means more than “the request returned.”
This guide uses Guzzle 8.2 on PHP 8.4 or later. The example makes a bounded, idempotent GET to https://example.com/ through an HTTP proxy, checks for a marker, and reports only a status and body size. Replace the target and marker with an endpoint you are allowed to access.
Guzzle's proxy request option accepts one proxy string or an array keyed by URI scheme. Supplying both http and https makes the route explicit for this request. The optional no list holds hosts that must bypass the proxy; the example reads NO_PROXY, trims it, and passes it deliberately instead of relying on an ambient process setting.
Set these values through your secret binding:
PROXY_HOST=proxy.example
PROXY_PORT=3128
PROXY_USER=account-name # optional
PROXY_PASSWORD=bound-secret # optional
NO_PROXY=localhost,127.0.0.1 # optional
The password is URL-encoded into a proxy URI in memory because Guzzle's proxy option accepts a credential-bearing URI. It never appears in source, a command argument, or a log message. Save this as proxy_get.php:
<?php
declare(strict_types=1);
require __DIR__.'/vendor/autoload.php';
use GuzzleHttp\Client;
use GuzzleHttp\Exception\TransferException;
use Psr\Http\Message\ResponseInterface;
const MAX_BODY_BYTES = 1_048_576;
function requiredEnv(string $name): string
{
$value = getenv($name);
if ($value === false || trim($value) === '') {
throw new RuntimeException("Missing required environment binding: {$name}");
}
return trim($value);
}
function readBoundedBody(ResponseInterface $response): string
{
$stream = $response->getBody();
$body = '';
try {
while (! $stream->eof()) {
$remaining = MAX_BODY_BYTES - strlen($body);
$chunk = $stream->read(min(8192, $remaining + 1));
if ($chunk === false || ($chunk === '' && ! $stream->eof())) {
throw new RuntimeException('Response body read made no progress');
}
$body .= $chunk;
if (strlen($body) > MAX_BODY_BYTES) {
throw new RuntimeException('Response body exceeded the local cap');
}
}
} finally {
$stream->close();
}
return $body;
}
$proxyHost = requiredEnv('PROXY_HOST');
$proxyPort = filter_var(getenv('PROXY_PORT'), FILTER_VALIDATE_INT);
if ($proxyPort === false || $proxyPort < 1 || $proxyPort > 65535) {
throw new RuntimeException('PROXY_PORT must be a TCP port from 1 through 65535');
}
$proxyUser = getenv('PROXY_USER');
$proxyPassword = getenv('PROXY_PASSWORD');
if (($proxyUser === false) !== ($proxyPassword === false)) {
throw new RuntimeException('PROXY_USER and PROXY_PASSWORD must be supplied together');
}
$proxy = 'http://';
if ($proxyUser !== false) {
$proxy .= rawurlencode($proxyUser).':'.rawurlencode($proxyPassword).'@';
}
$proxy .= $proxyHost.':'.$proxyPort;
$noProxy = array_values(array_filter(
array_map('trim', explode(',', (string) (getenv('NO_PROXY') ?: ''))),
static fn (string $host): bool => $host !== ''
));
$client = new Client([
'allow_redirects' => false,
'connect_timeout' => 5.0,
'http_errors' => false,
'proxy' => [
'http' => $proxy,
'https' => $proxy,
'no' => $noProxy,
],
'timeout' => 15.0,
]);
try {
$response = $client->request('GET', 'https://example.com/', [
'read_timeout' => 10.0,
'stream' => true,
]);
$status = $response->getStatusCode();
if ($status === 407) {
$response->getBody()->close();
throw new RuntimeException('Proxy authentication or CONNECT negotiation returned HTTP 407');
}
$body = readBoundedBody($response);
} catch (TransferException $exception) {
throw new RuntimeException('Request failed in transport or proxy negotiation');
}
if ($status !== 200) {
throw new RuntimeException('Target returned a non-accepted HTTP status');
}
if (! str_contains($body, 'Example Domain')) {
throw new RuntimeException('Target marker was absent');
}
printf("status=%d body_bytes=%d\n", $status, strlen($body));
Run it from the project that already has Guzzle installed:
php proxy_get.php
http_errors => false keeps HTTP statuses in application control, so a 407 or a target 503 can be classified before an exception policy decides what to do. stream => true and the per-read read_timeout make body consumption explicit, while the read-progress guard prevents a non-EOF empty read from spinning forever. The 1 MiB limit belongs to this example's memory policy; it is not a proxy billing quota or a total body deadline. Keep one configured Client for a bounded unit of work.
This core example uses one attempt. If a worker adds retries for an idempotent GET, classify the failure first, honor a valid Retry-After value, keep a job-level deadline, and close each response before the next request. Transfer, TLS, configuration, and proxy-authentication failures should stop immediately. A proxy credential problem will not be repaired by repeating the same secret, and a target rate limit needs the target's pacing policy.
Use a small contract at the job boundary:
input: permitted target URL + proxy host/port + optional secret binding + marker
output: status=200, marker present, body <= 1 MiB, no redirect followed
The marker stands for the field or schema your application actually needs. A 200 response confirms an HTTP status; it does not confirm that the parser received the right document.
| Observation | Boundary | Next decision |
|---|---|---|
| Proxy option cannot be constructed | Local configuration | Check host, port, scheme, and paired credentials without logging them |
TransferException before a response |
Client, proxy, DNS, TLS, or CONNECT | Check the route and proxy auth; stop before retrying |
| HTTP 407 | Proxy authentication | Repair proxy credentials or entitlement; never treat it as target authorization |
| HTTP 401 or 403 after a successful tunnel | Target policy | Check target login, permission, or target-side rules |
| HTTP 502, 503, or 504 | Temporary gateway or target | Retry within the bounded policy, then defer the item |
| HTTP 200 with no marker | Application validation | Reject the result and inspect content or parser assumptions |
| Body exceeds 1 MiB | Local resource policy | Reject or adopt a different bounded streaming rule |
The Python Requests proxy guide shows the same explicit-route and validation pattern in Python. Use the curl proxy guide to inspect a CONNECT hop, and read HTTP 407 troubleshooting when Guzzle reports a transfer exception before creating a target response.
ProxyLane's public residential offer starts from $2.50/GB with HTTP and SOCKS5 access, country, city, ISP, rotating and sticky options, and traffic that does not expire. Create a free account, then choose a traffic package separately; no payment is required at signup. These are route and commercial inputs, not a Guzzle benchmark or a target acceptance promise. When your target and acceptance rule are authorized, Sign up.
Distinguish authorized Amazon APIs, licensed product data and proxy-based page checks by record quality and access rights.
Read guideConnect a buyer-owned proxy to an Apify Actor, keep the session boundary clear, and validate records instead of counting requests.
Read guideSeparate Australian egress from en-AU content, AUD pricing, GST display, postcode validation and the state delivery context.
Read guide