Amazon Scraper API vs Proxy: Choose a Product Data Source
Distinguish authorized Amazon APIs, licensed product data and proxy-based page checks by record quality and access rights.
Read guideUse Java SE HttpClient with an explicit HTTP proxy, a scoped Authenticator, bounded response reads, and clear proxy-versus-target errors.
Java's built-in HttpClient separates proxy selection from proxy authentication. That is useful when a worker must show exactly which route it selected and must not attach a proxy password to an origin request by mistake.
This guide uses Java SE 21 and an HTTP proxy endpoint to make one authorized GET to https://example.com/. The example does not follow redirects, reads at most 1 MiB, makes one bounded attempt, and prints no endpoint, username, password, or exception object. Replace the target and marker with the resource your application is allowed to access.
HttpClient.Builder.proxy(ProxySelector) accepts a selector. ProxySelector.of returns a selector for one proxy, so it is a straightforward choice when this client should not inherit the JVM's system proxy selection. connectTimeout covers establishment of a new connection, while a request timeout covers the individual request.
The JDK's built-in client invokes an Authenticator for an authentication challenge. The implementation supports HTTP Basic authentication, and the callback exposes RequestorType.PROXY, the requesting host, and the requesting port. Restricting the callback to that requestor type keeps proxy credentials at the proxy boundary.
On Java SE 21, Basic is disabled by default for HTTP CONNECT tunneling through the jdk.http.auth.tunneling.disabledSchemes networking property. Treat the no-credential route as the supported baseline. If the proxy requires Basic authentication, verify the JVM policy and the proxy's advertised scheme before binding credentials; this example does not weaken a global security setting to force a challenge through.
Set these bindings through your runtime secret store:
PROXY_HOST=proxy.example
PROXY_PORT=3128
PROXY_USER=account-name # optional
PROXY_PASSWORD=bound-secret # optional
Save the following as ProxyGet.java:
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.net.Authenticator;
import java.net.InetSocketAddress;
import java.net.PasswordAuthentication;
import java.net.ProxySelector;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.nio.charset.StandardCharsets;
import java.time.Duration;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
public final class ProxyGet {
private static final int MAX_BODY_BYTES = 1 << 20;
private static String required(String name) {
String value = System.getenv(name);
if (value == null || value.isBlank()) {
throw new IllegalArgumentException("missing required environment binding: " + name);
}
return value.trim();
}
private static String readBody(HttpResponse<InputStream> response) throws IOException {
try (InputStream input = response.body(); ByteArrayOutputStream output = new ByteArrayOutputStream()) {
byte[] buffer = new byte[8192];
int total = 0;
int read;
while ((read = input.read(buffer)) != -1) {
if (total + read > MAX_BODY_BYTES) {
throw new IOException("response body exceeded local cap");
}
output.write(buffer, 0, read);
total += read;
}
return output.toString(StandardCharsets.UTF_8);
}
}
private static String readBodyWithDeadline(HttpResponse<InputStream> response) throws IOException {
ExecutorService readerExecutor = Executors.newSingleThreadExecutor();
Future<String> body = readerExecutor.submit(() -> readBody(response));
try {
return body.get(10, TimeUnit.SECONDS);
} catch (TimeoutException exception) {
try {
response.body().close();
} catch (IOException ignored) {
// The request is already being rejected for exceeding its read deadline.
}
body.cancel(true);
throw new IOException("response body exceeded the read deadline");
} catch (InterruptedException exception) {
Thread.currentThread().interrupt();
try {
response.body().close();
} catch (IOException ignored) {
// Preserve the interruption category without logging the stream details.
}
throw new IOException("response body read was interrupted");
} catch (ExecutionException exception) {
Throwable cause = exception.getCause();
if (cause instanceof IOException ioException) {
throw ioException;
}
throw new IOException("response body read failed");
} finally {
readerExecutor.shutdownNow();
}
}
public static void main(String[] args) throws Exception {
String proxyHost = required("PROXY_HOST");
int proxyPort;
try {
proxyPort = Integer.parseInt(required("PROXY_PORT"));
} catch (NumberFormatException exception) {
throw new IllegalArgumentException("PROXY_PORT must be an integer");
}
if (proxyPort < 1 || proxyPort > 65535) {
throw new IllegalArgumentException("PROXY_PORT is outside the TCP port range");
}
String proxyUser = System.getenv("PROXY_USER");
String proxyPassword = System.getenv("PROXY_PASSWORD");
if ((proxyUser == null) != (proxyPassword == null)) {
throw new IllegalArgumentException("PROXY_USER and PROXY_PASSWORD must be supplied together");
}
HttpClient.Builder clientBuilder = HttpClient.newBuilder()
.proxy(ProxySelector.of(new InetSocketAddress(proxyHost, proxyPort)))
.connectTimeout(Duration.ofSeconds(5))
.followRedirects(HttpClient.Redirect.NEVER);
if (proxyUser != null) {
clientBuilder.authenticator(new Authenticator() {
@Override
protected PasswordAuthentication getPasswordAuthentication() {
if (getRequestorType() != RequestorType.PROXY
|| !proxyHost.equalsIgnoreCase(getRequestingHost())
|| proxyPort != getRequestingPort()) {
return null;
}
return new PasswordAuthentication(proxyUser, proxyPassword.toCharArray());
}
});
}
HttpClient client = clientBuilder.build();
HttpRequest request = HttpRequest.newBuilder()
.uri(java.net.URI.create("https://example.com/"))
.timeout(Duration.ofSeconds(15))
.GET()
.build();
HttpResponse<InputStream> response;
try {
response = client.send(request, HttpResponse.BodyHandlers.ofInputStream());
} catch (IOException exception) {
throw new IllegalStateException("request failed in transport or proxy negotiation");
} catch (InterruptedException exception) {
Thread.currentThread().interrupt();
throw new IllegalStateException("request interrupted");
}
int status = response.statusCode();
if (status == 407) {
response.body().close();
throw new IllegalStateException("proxy authentication or CONNECT negotiation returned HTTP 407");
}
String body = readBodyWithDeadline(response);
if (status != 200) {
throw new IllegalStateException("target returned a non-accepted HTTP status");
}
if (!body.contains("Example Domain")) {
throw new IllegalStateException("target marker was absent");
}
System.out.printf("status=%d body_bytes=%d%n", status, body.getBytes(StandardCharsets.UTF_8).length);
}
}
Compile and run it with the JDK 21 toolchain:
javac ProxyGet.java
java ProxyGet
The code uses BodyHandlers.ofInputStream() so the application, rather than a convenience string handler, owns the body limit. readBodyWithDeadline adds a 10-second application read deadline and closes the stream if it expires; the try-with-resources block closes it after a normal body read. A 407 is checked before body acceptance because a proxy challenge means the route did not authenticate successfully. A 401 or 403 after a successful tunnel belongs to the target's policy and needs a different investigation.
When PROXY_USER and PROXY_PASSWORD are present, the JDK's built-in client can use HTTP Basic authentication only when the JVM policy permits it for the proxy's CONNECT challenge. Check the JVM's policy and the proxy's advertised scheme before treating an Authenticator callback as proof that credentials can be used. Choose a supported authentication policy or client and verify the route separately.
Treat this as a small record contract:
input: permitted target URI + explicit proxy host/port + expected marker
output: status=200, marker present, body <= 1 MiB, no redirect followed
The marker is a placeholder for your own response schema. HTTP status alone does not tell the worker whether it received the right page, JSON object, or business field. Keep that validation beside the client call so a successful transport cannot be mistaken for an accepted record.
| Observation | Boundary | Next decision |
|---|---|---|
| Client cannot connect or times out | Client, proxy, or CONNECT setup | Check host, port, proxy protocol, TLS, and network policy before another authorized attempt |
| Authenticator receives a non-proxy requestor | Credential scope | Return no credentials and inspect which hop requested authentication |
| HTTP 407 | Proxy authentication | Repair proxy credentials or entitlement; do not send the secret as an origin header |
| HTTP 401 or 403 after a tunnel | Target policy | Check target login or target-side permission |
| HTTP 502, 503, or 504 | Temporary gateway/target response | Defer or retry only under an explicit policy that honors Retry-After |
| HTTP 200 without the marker | Application validation | Reject the result and inspect target content or parser assumptions |
| Body exceeds 1 MiB | Local resource policy | Reject or define a different bounded streaming policy |
For a terminal probe that shows the CONNECT phase, see the curl proxy guide. The HTTP 407 guide explains why a challenge can occur before a target response, and proxy client settings covers environment and application routes. The Java client here uses one explicit selector; it does not prove what another JVM process or browser will do.
ProxyLane's public residential offer starts from $2.50/GB with HTTP and SOCKS5 access, country, city, ISP, rotating and sticky options, and traffic that does not expire. Create a free account, then choose a traffic package separately; no payment is required at signup. These are route and commercial inputs, not a Java benchmark or a target acceptance promise. When your target and acceptance rule are authorized, Sign up.
Distinguish authorized Amazon APIs, licensed product data and proxy-based page checks by record quality and access rights.
Read guideConnect a buyer-owned proxy to an Apify Actor, keep the session boundary clear, and validate records instead of counting requests.
Read guideSeparate Australian egress from en-AU content, AUD pricing, GST display, postcode validation and the state delivery context.
Read guide