Wget Proxy: Use WGETRC Safely and Verify the Download

Wget can use a proxy with one environment variable, but a production check needs to make more visible decisions: which proxy settings win, where authentication is stored, how many attempts are allowed, how redirects are bounded, and whether the downloaded file is the expected resource.

The script below uses a temporary WGETRC file with mode 0600, makes one authorized HTTPS fetch, captures server headers for classification, checks the downloaded file against an application size rule, and deletes its temporary files on exit. It assumes GNU Wget 1.21 or later and uses settings documented in the current GNU Wget 1.25.0 manual. Replace the target and marker with a resource you are allowed to access.

Put proxy settings in a protected Wgetrc

GNU Wget documents http_proxy, https_proxy, and no_proxy for environment-driven routing. A Wgetrc file can override those values and can hold proxy_user, proxy_password, timeout, tries, and waitretry. WGETRC selects a specific startup file, which lets a job create a short-lived file without changing a developer's home configuration.

Keep the endpoint and credentials in separate environment bindings:

PROXY_URL=http://proxy.example:3128
PROXY_USER=account-name             # optional
PROXY_PASSWORD=bound-secret         # optional
NO_PROXY=localhost,127.0.0.1         # optional
TARGET_URL=https://example.com/      # optional

Save this as fetch-through-proxy.sh:

#!/bin/sh
set -eu

: "${PROXY_URL:?Set PROXY_URL to an HTTP proxy URL without credentials}"
line_feed=$(printf '\nX')
line_feed=${line_feed%X}
carriage_return=$(printf '\rX')
carriage_return=${carriage_return%X}
check_single_line() {
  case "$1" in
    *"$line_feed"*|*"$carriage_return"*) printf '%s\n' 'proxy settings must not contain newlines' >&2; exit 2 ;;
  esac
}
check_single_line "$PROXY_URL"
check_single_line "${PROXY_USER:-}"
check_single_line "${PROXY_PASSWORD:-}"
check_single_line "${NO_PROXY:-}"
case "$PROXY_URL" in
  http://*@*) printf '%s\n' 'PROXY_URL must not contain userinfo' >&2; exit 2 ;;
  http://*) ;;
  *) printf '%s\n' 'PROXY_URL must be an HTTP proxy URL' >&2; exit 2 ;;
esac

target_url=${TARGET_URL:-https://example.com/}
output_file=${OUTPUT_FILE:-proxy-check.out}
check_single_line "$target_url"
check_single_line "$output_file"
umask 077
if [ -n "${PROXY_USER:-}" ] || [ -n "${PROXY_PASSWORD:-}" ]; then
  if [ -z "${PROXY_USER:-}" ] || [ -z "${PROXY_PASSWORD:-}" ]; then
    printf '%s\n' 'PROXY_USER and PROXY_PASSWORD must be supplied together' >&2
    exit 2
  fi
fi
config_file=$(mktemp "${TMPDIR:-/tmp}/wgetrc.XXXXXX")
trace_file=$(mktemp "${TMPDIR:-/tmp}/wget-trace.XXXXXX")
cleanup() {
  rm -f "$config_file" "$trace_file"
}
trap cleanup EXIT HUP INT TERM

{
  printf 'http_proxy = %s\n' "$PROXY_URL"
  printf 'https_proxy = %s\n' "$PROXY_URL"
  printf 'no_proxy = %s\n' "${NO_PROXY:-}"
  if [ -n "${PROXY_USER:-}" ]; then
    printf 'proxy_user = %s\n' "$PROXY_USER"
  fi
  if [ -n "${PROXY_PASSWORD:-}" ]; then
    printf 'proxy_password = %s\n' "$PROXY_PASSWORD"
  fi
  printf '%s\n' 'use_proxy = on' 'timeout = 15' 'tries = 1' 'waitretry = 0'
} > "$config_file"

if ! WGETRC="$config_file" wget \
  --server-response \
  --max-redirect=5 \
  --output-document="$output_file" \
  -- "$target_url" 2>"$trace_file"; then
  if grep -Eq 'HTTP/[0-9.]+ 407' "$trace_file"; then
    printf '%s\n' 'request failed at proxy authentication or CONNECT negotiation' >&2
  else
    printf '%s\n' 'request failed before an accepted target response' >&2
  fi
  exit 1
fi

if grep -Eq 'HTTP/[0-9.]+ 407' "$trace_file"; then
  printf '%s\n' 'proxy authentication or CONNECT negotiation returned HTTP 407' >&2
  exit 1
fi
if grep -Eq 'HTTP/[0-9.]+ (401|403)' "$trace_file"; then
  printf '%s\n' 'target returned an authentication or permission response' >&2
  exit 1
fi
if ! grep -q 'Example Domain' "$output_file"; then
  printf '%s\n' 'target marker was absent' >&2
  exit 1
fi

body_bytes=$(wc -c < "$output_file" | tr -d '[:space:]')
if [ "$body_bytes" -gt 1048576 ]; then
  printf '%s\n' 'download exceeded the local body cap' >&2
  exit 1
fi
printf 'status=accepted body_bytes=%s\n' "$body_bytes"

Run it after making the script executable:

chmod 700 fetch-through-proxy.sh
./fetch-through-proxy.sh

The temporary config is created with umask 077 and removed by the trap. The single-line checks prevent a secret or endpoint value from injecting another Wgetrc directive. The password is never placed in a command argument or printed. tries = 1 is intentional: Wget cannot know from a generic exit whether the failure was a proxy-auth challenge, a CONNECT failure, or an origin error, so the script classifies the trace before anyone decides whether a repeat is safe. If a job adds retries, make them an explicit policy for an idempotent operation, honor Retry-After where the target provides it, and never repeat a rejected proxy secret.

The final byte check is an acceptance rule applied after Wget writes the file; GNU Wget's quota does not cap a single-file download, so this script does not present it as a hard transfer limit. If a hard wall-clock deadline is required, wrap the command in a supervisor supplied by your runtime. timeout = 15 bounds applicable network operations, not the full job lifetime. --max-redirect=5 prevents an unbounded redirect chain. --server-response captures headers for a short-lived local trace; do not publish that trace if the target URL or surrounding environment is sensitive.

Use --spider as a separate header check

When you need to test route and response headers without saving a body, use the same protected config with Wget's spider mode:

WGETRC="$config_file" wget --spider --server-response --max-redirect=5 "$target_url"

Run that command inside the script before the download if your workflow needs a header-only preflight, then keep the download's marker and byte checks. --spider is not proof that a later fetch will return the same content; it is a separate request with its own target and policy.

Read the failure at the right hop

The curl proxy guide shows a compact CONNECT probe, while HTTP 407 troubleshooting explains why a proxy challenge can appear before an origin response. For application-owned bypass rules, compare proxy client settings.

ProxyLane's public residential offer starts from $2.50/GB with HTTP and SOCKS5 access, country, city, ISP, rotating and sticky options, and traffic that does not expire. Create a free account, then choose a traffic package separately; no payment is required at signup. These are route and commercial inputs, not a Wget benchmark or a target acceptance promise. When your target and acceptance rule are authorized, Sign up.