Amazon Scraper API vs Proxy: Choose a Product Data Source
Compare seller APIs, licensed Amazon product data and proxy page checks by record, access scope and reuse rights.
Read guideConfigure bogdanfinn/tls-client with a proxy, validate one observed exit and an authorized target response, and keep timeout and HTTP/3 behavior explicit.

Configure github.com/bogdanfinn/tls-client with WithProxyUrl, then use the same client to check an exit endpoint and an authorized target. The client provides HTTP requests with selectable TLS profiles; it does not render a DOM or execute page JavaScript. Validate both the returned IP and target content before accepting a result.
The example was executed with Go 1.27.1 and tls-client v1.16.0 on 2026-09-29. Create a module and pin the tested dependency:
go mod init example.com/tls-client-probe
go get github.com/bogdanfinn/tls-client@v1.16.0
Load the authenticated proxy URL through the runtime secret binding PROXY_URL. Its form is http://<PROXY_USER>:<PROXY_PASSWORD>@<PROXY_HOST>:<PROXY_PORT>; percent-encode reserved characters. Do not put the URL in source control, shell history, process arguments, or logs.
Warning: A proxy URL contains credentials and each request can consume metered traffic. Use a secret binding and a permitted test target, cap each response, and run one probe at a time. If the credential reaches a log or source file, rotate it and remove the exposed copy.
With an http:// proxy URL, Basic credentials travel unencrypted from this client to the proxy, even when the target uses HTTPS. Target TLS begins after the CONNECT tunnel is established; it cannot encrypt the earlier authentication exchange. Use a trusted network path or a TLS-protected proxy endpoint after confirming both sides support it.
Save the program as main.go:
package main
import (
"encoding/json"
"errors"
"fmt"
"io"
"net"
"net/url"
"os"
"strings"
tls_client "github.com/bogdanfinn/tls-client"
"github.com/bogdanfinn/tls-client/profiles"
)
const maxResponseBytes = 65_536
var errResponseTooLarge = errors.New("response exceeded the byte limit")
var (
proxyURL = os.Getenv("PROXY_URL")
exitCheckURL = envOr("EXIT_CHECK_URL", "https://api.ipify.org?format=json")
targetURL = envOr("TARGET_URL", "https://example.com/")
)
func envOr(name string, fallback string) string {
if value := os.Getenv(name); value != "" {
return value
}
return fallback
}
func validateProxyURL(value string) error {
parsed, err := url.Parse(value)
if err != nil || parsed.Scheme != "http" || parsed.Hostname() == "" || parsed.Port() == "" {
return errors.New("PROXY_URL must be an HTTP proxy URL with a host and port")
}
return nil
}
func getBody(client tls_client.HttpClient, requestURL string) (int, []byte, error) {
response, err := client.Get(requestURL)
if err != nil {
return 0, nil, err
}
defer response.Body.Close()
body, err := io.ReadAll(io.LimitReader(response.Body, maxResponseBytes+1))
if err != nil {
return response.StatusCode, nil, err
}
if len(body) > maxResponseBytes {
return response.StatusCode, nil, errResponseTooLarge
}
return response.StatusCode, body, nil
}
func run() error {
if err := validateProxyURL(proxyURL); err != nil {
return err
}
client, err := tls_client.NewHttpClient(
tls_client.NewNoopLogger(),
tls_client.WithProxyUrl(proxyURL),
tls_client.WithTimeoutSeconds(10),
tls_client.WithClientProfile(profiles.Chrome_150),
tls_client.WithNotFollowRedirects(),
)
if err != nil {
return errors.New("client setup failed")
}
defer client.CloseIdleConnections()
exitStatus, exitBody, err := getBody(client, exitCheckURL)
if err != nil {
if errors.Is(err, errResponseTooLarge) {
return errors.New("exit response exceeded the byte limit")
}
return errors.New("exit request failed")
}
if exitStatus < 200 || exitStatus >= 300 {
return fmt.Errorf("exit check returned HTTP %d", exitStatus)
}
var exitResult struct {
IP string `json:"ip"`
}
if err := json.Unmarshal(exitBody, &exitResult); err != nil || net.ParseIP(exitResult.IP) == nil {
return errors.New("exit check returned an invalid IP response")
}
targetStatus, targetBody, err := getBody(client, targetURL)
if err != nil {
if errors.Is(err, errResponseTooLarge) {
return errors.New("target response exceeded the byte limit")
}
return errors.New("target request failed")
}
if targetStatus < 200 || targetStatus >= 300 {
return fmt.Errorf("target returned HTTP %d", targetStatus)
}
if !strings.Contains(string(targetBody), "Example Domain") {
return errors.New("target content did not match the expected marker")
}
result, err := json.Marshal(map[string]string{
"exit_ip": exitResult.IP,
"target_check": "passed",
})
if err != nil {
return errors.New("result encoding failed")
}
fmt.Println(string(result))
return nil
}
func main() {
if err := run(); err != nil {
fmt.Fprintf(os.Stderr, "Probe failed: %s\n", err.Error())
os.Exit(1)
}
}
Run the probe from the module directory after loading secrets through the deployment platform or a local secret manager. If you change TARGET_URL, also replace the hard-coded "Example Domain" marker in the strings.Contains target check with text expected from that page. The program prints a validated exit address but does not include the proxy URL or response bodies in its output.
go run .
Expected output includes the address returned by the configured echo endpoint:
{"exit_ip":"<OBSERVED_EXIT_IP>","target_check":"passed"}
<OBSERVED_EXIT_IP> is an output value. WithTimeoutSeconds(10) applies a hard deadline to each request lifecycle, including body reads. The 65,536-byte reader cap bounds memory used for each response. Two sequential requests can each use their own timeout, so set a separate job deadline for the whole worker.
WithProxyUrl receives the complete proxy URL. For the default https:// target, tls-client routes a CONNECT tunnel through the HTTP proxy and negotiates target TLS inside it. Chrome_150 selects the client TLS and HTTP profile; it neither authorizes the target nor runs a browser. Certificate verification stays enabled.
The v1.16.0 release notes say protocol racing with a proxy scheme other than SOCKS5 is rejected during client setup or SetProxy. The release describes the earlier HTTP/3 leg as capable of going direct because only SOCKS5 can tunnel UDP. Keep protocol racing disabled for an HTTP proxy, or configure a supported SOCKS5 route before enabling HTTP/3. Recheck egress after changing transport options.
| Observation | Boundary | Next check |
|---|---|---|
| Proxy URL or client setup error | Configuration | Check the http scheme, host, port, and protocol options |
exit request failed or target request failed |
Proxy reachability or authentication | The generic message can include a CONNECT 407 and does not distinguish it from a dead endpoint. Check the secret binding and proxy account/status separately; do not print the raw library error. |
| TLS handshake error | Proxy TLS or target TLS | Identify the failing hop and preserve certificate verification |
| Non-2xx status | Proxy or target response | Record which request returned the status and apply that endpoint's policy |
| 2xx response without the marker | Target content or parser | Reject the record and inspect a redacted response sample |
For an HTTP proxy, this sample maps a CONNECT authentication rejection to the same generic request-failed message it uses for connection errors. Confirm the secret binding and proxy endpoint through your secret manager and provider-side account/status information; the program intentionally avoids raw library errors because they can contain credential-bearing URLs.
For an independent command-line route check, use the curl proxy guide. For authenticated proxy failures, use HTTP 407 troubleshooting.
Scale your workflows with the tools you already use. Rotating or sticky sessions up to 72 hours.
Compare seller APIs, licensed Amazon product data and proxy page checks by record, access scope and reuse rights.
Read guideChoose the right anti-detect browser proxy guide, map the shared verification steps, and keep account access separate from the network route.
Read guideOur picks for market research: ProxyLane for small, intermittent projects, plus options for regional tracking, monthly collection and managed extraction.
Read guide