Amazon Scraper API vs Proxy: Choose a Product Data Source
Distinguish authorized Amazon APIs, licensed product data and proxy-based page checks by record quality and access rights.
Read guideRoute Ruby Net::HTTP through an HTTP proxy, keep credentials separate, close sessions cleanly, and validate the target response.
Ruby's standard Net::HTTP can send an HTTPS request through an HTTP proxy, but the proxy belongs in the connection object, while TLS belongs on the connection to the target. Keeping those two hops visible makes a 407, a target 403, and a connection timeout easier to act on.
The example below uses Net::HTTP.new with an HTTP proxy endpoint, makes an authorized GET to https://example.com/, bounds connection and read work, caps the body at 1 MiB, and closes the session with a block. It assumes Ruby 2.6 or later and uses only the standard library. Replace the target and marker with the resource your application is allowed to access.
Net::HTTP.new accepts the target address and port, followed by the proxy address, proxy port, username, and password. The example keeps PROXY_URL free of userinfo and passes the credentials as separate environment bindings:
PROXY_URL=http://proxy.example:3128
PROXY_USER=account-name # optional
PROXY_PASSWORD=bound-secret # optional
This baseline intentionally supports an HTTP proxy endpoint. use_ssl = true enables TLS for the HTTPS target after the proxy connection is established; it does not turn the proxy itself into an HTTPS proxy. If your provider gives a different proxy protocol, select a Ruby client whose current API documents that protocol.
Save this as proxy_get.rb:
require 'net/http'
require 'openssl'
require 'uri'
MAX_BODY_BYTES = 1 << 20
def required_env(name)
value = ENV[name].to_s.strip
raise "missing required environment binding: #{name}" if value.empty?
value
end
def read_response(http, target)
request = Net::HTTP::Get.new(target)
status = nil
body = +''
http.request(request) do |response|
status = response.code.to_i
if status == 407
raise 'proxy authentication or CONNECT negotiation returned HTTP 407'
end
response.read_body do |chunk|
if body.bytesize + chunk.bytesize > MAX_BODY_BYTES
raise 'response body exceeded local cap'
end
body << chunk
end
end
[status, body]
end
target = URI('https://example.com/')
begin
proxy = URI(required_env('PROXY_URL'))
rescue URI::InvalidURIError
warn 'PROXY_URL is not a valid URI'
exit 1
end
unless proxy.scheme == 'http' && !proxy.host.nil? && proxy.user.nil? && proxy.password.nil?
raise 'PROXY_URL must be an HTTP proxy URL without credentials'
end
proxy_user = ENV['PROXY_USER']
proxy_password = ENV['PROXY_PASSWORD']
if proxy_user.nil? != proxy_password.nil?
raise 'PROXY_USER and PROXY_PASSWORD must be supplied together'
end
http = Net::HTTP.new(
target.host,
target.port,
proxy.host,
proxy.port || 80,
proxy_user,
proxy_password
)
http.use_ssl = true
http.open_timeout = 5
http.read_timeout = 10
http.write_timeout = 10
http.max_retries = 0
begin
http.start do |session|
status, body = read_response(session, target)
raise 'target returned a non-accepted HTTP status' unless status == 200
raise 'target marker was absent' unless body.include?('Example Domain')
puts "status=#{status} body_bytes=#{body.bytesize}"
end
rescue Net::OpenTimeout, Net::ReadTimeout, Net::WriteTimeout, SocketError, EOFError,
Errno::ECONNRESET, Errno::ECONNREFUSED, Errno::ETIMEDOUT,
OpenSSL::SSL::SSLError, Net::HTTPBadResponse, Net::ProtocolError
warn 'request failed in transport or proxy negotiation'
exit 1
end
Run it with the standard Ruby interpreter:
ruby proxy_get.rb
The Net::HTTP.start block closes the connection when it exits. open_timeout covers opening the connection to the proxy, while read_timeout and write_timeout apply to socket operations. The response block rejects a 407 before reading its body, then bounds accepted body data at 1 MiB. Malformed URI, TLS, and CONNECT failures exit with a generic message; the proxy URI and exception details never reach output.
This core example uses one attempt. If a worker adds retries for an idempotent GET, classify the failure first, honor a valid Retry-After value, and keep a job-level deadline. DNS, TLS, connection, configuration, and proxy-authentication failures should stop immediately because repeating an invalid password or an unreachable endpoint does not repair the route.
Use an application-level contract rather than counting a successful request call:
input: permitted HTTPS target + HTTP proxy host/port + expected marker
output: status=200, marker present, body <= 1 MiB, session closed
The marker is a placeholder for your own required field or schema. The target can return a valid HTTP status with an error page, login form, or challenge that your parser must reject.
| Observation | Boundary | Next decision |
|---|---|---|
| Proxy URL has credentials or an unsupported scheme | Local configuration | Remove userinfo and use a documented HTTP proxy endpoint |
| Open/read/TLS exception before a response | Client, proxy, or CONNECT | Check proxy host, port, certificate, and reachability; do not retry blindly |
| HTTP 407 | Proxy authentication | Repair proxy credentials or entitlement; keep them away from target headers |
| HTTP 401 or 403 after CONNECT | Target policy | Check target login, permission, or target-side rules |
| HTTP 502, 503, or 504 | Temporary gateway or target response | Defer or retry only under an explicit policy that honors Retry-After |
| HTTP 200 with no marker | Application validation | Reject the result and inspect the content or parser |
| Body exceeds 1 MiB | Local resource policy | Reject or define another bounded streaming rule |
For the same phase-by-phase view in a terminal, read the curl proxy guide. HTTP 407 troubleshooting explains why a CONNECT challenge can appear as an exception, and proxy client settings covers cases where an application-specific route differs from the operating system.
ProxyLane's public residential offer starts from $2.50/GB with HTTP and SOCKS5 access, country, city, ISP, rotating and sticky options, and traffic that does not expire. Create a free account, then choose a traffic package separately; no payment is required at signup. Those are route and commercial inputs, not a Ruby benchmark or a target acceptance promise. When your target and acceptance rule are authorized, Sign up.
Distinguish authorized Amazon APIs, licensed product data and proxy-based page checks by record quality and access rights.
Read guideConnect a buyer-owned proxy to an Apify Actor, keep the session boundary clear, and validate records instead of counting requests.
Read guideSeparate Australian egress from en-AU content, AUD pricing, GST display, postcode validation and the state delivery context.
Read guide