Proxies for job market data: measure vacancies, not URLs

Define the question and permitted source frame

Decide what the output must answer: for example, changes in postings on a named set of employer career sites, or roles observed in a defined region during a fixed period. This is a sample of selected sources, not a census of vacancies or labor demand.

Before fetching, list the source owner, permitted access path, target pages or application programming interface (API), relevant terms, request pace, role and location filters, observation window, and retention period. Prefer an official feed, export, or API when available. Read the site's terms and applicable access instructions; pause when a source denies access or asks clients to slow down. The Robots Exclusion Protocol is a crawler instruction mechanism, not a general access grant.

Keep a posting observation separate from a vacancy

One job can appear on an employer site, an applicant tracking system (ATS), and several boards. One posting can also describe multiple hires. Keep a row for each observed page, then link rows into a reviewed vacancy group only when evidence supports the match.

Google's JobPosting guidance says JobPosting structured data belongs on a page for one job, not a search-results page. It describes datePosted, validThrough, canonical URLs, and how publishers should update expired postings. These fields help describe a publisher's page; they are not a universal vacancy identifier, and absent or stale markup does not settle whether a role remains open. Schema.org's JobPosting lists the vocabulary fields.

Prerequisites

Run the schema example with Python 3.12.0, tested locally; it uses only the standard library.

Validate a small local batch

Start with a compact record contract before building a collector. The example below uses synthetic observations and validates fields; it does not fetch websites, confirm permission, or prove that two posts represent one vacancy.

from datetime import date
from urllib.parse import urlparse

REQUIRED = {"posting_id", "source_url", "employer", "title", "location", "retrieved_on"}
OBSERVATIONS = [
    {
        "posting_id": "obs-001",
        "source_url": "https://careers.example.org/jobs/eng-1",
        "employer": "Example Works",
        "title": "Platform Engineer",
        "location": "Remote, US",
        "retrieved_on": "2026-09-24",
    },
    {
        "posting_id": "obs-001",
        "source_url": "https://jobs.example.net/posting/44",
        "employer": "Example Works",
        "title": "Platform Engineer",
        "location": "Remote, US",
        "retrieved_on": "2026-09-24",
    },
    {
        "posting_id": "obs-003",
        "source_url": "file:///tmp/job.html",
        "employer": "Example Works",
        "title": "Analyst",
        "location": "New York",
        "retrieved_on": "not-a-date",
    },
]

def errors_for(row):
    errors = [f"missing {key}" for key in sorted(REQUIRED - row.keys())]
    parsed_url = urlparse(row.get("source_url", ""))
    if parsed_url.scheme != "https" or not parsed_url.netloc:
        errors.append("source_url must be an absolute HTTPS URL")
    try:
        retrieved_on = row.get("retrieved_on", "")
        if date.fromisoformat(retrieved_on).isoformat() != retrieved_on:
            raise ValueError
    except (TypeError, ValueError):
        errors.append("retrieved_on must be YYYY-MM-DD")
    if not isinstance(row.get("posting_id"), str) or not row["posting_id"].strip():
        errors.append("posting_id must be nonempty text")
    return errors

seen = set()
for row in OBSERVATIONS:
    errors = errors_for(row)
    posting_id = row.get("posting_id")
    if isinstance(posting_id, str) and posting_id.strip():
        if posting_id in seen:
            errors.append("posting_id is not unique")
        seen.add(posting_id)
    label = posting_id if isinstance(posting_id, str) and posting_id.strip() else "<missing ID>"
    print(f'{label}: {"accepted" if not errors else "; ".join(errors)}')

Expected result for the synthetic rows:

obs-001: accepted
obs-001: posting_id is not unique
obs-003: source_url must be an absolute HTTPS URL; retrieved_on must be YYYY-MM-DD

A production validator should also record source-specific identifiers, application URLs where present, structured-data version, response outcome, and missing fields. Keep the raw observation or a controlled reference to it so later corrections do not overwrite what was seen.

Review duplicate candidates instead of collapsing them

Use a requisition ID or canonical application URL when a publisher exposes a stable identifier. Otherwise, compare normalized employer, title, location, employment type, team, and dates as candidate signals. A matching title alone is weak evidence. Preserve both posting rows and route uncertain pairs to review; never turn fuzzy similarity into an unqualified vacancy count.

After review, store a group decision, reviewer, timestamp, matching evidence, and rule version. Re-run a fixed sample after changing normalization rules. A new URL with old text can indicate a repost, but it can also describe a separate opening. Keep the status uncertain until the source record or reviewer resolves it.

Recheck closure and describe the sample

Store first_seen, last_seen, checked_at, and status_evidence. Record explicit closure text, a closed application form, a redirect, or an unavailable page separately. A Hypertext Transfer Protocol (HTTP) 200 response can contain a login wall, an empty shell, or a careers index; HTTP success alone is not an open posting. Do not discard a last valid observation after a temporary fetch failure. Mark it stale and report its age.

A board-only sample reflects the sources and search visibility represented by that board. A career-site sample favors employers with discoverable pages. Report the employer frame, board/source mix, geography, remote policy, language, salary disclosure, collection period, field missingness, duplicate rule, closure cadence, and all exclusions before comparing counts. Do not infer the full labor market from that frame.

A proxy can supply a network route for a permitted fetch or a regional presentation check. It does not provide the source list, validate a posting, decide reuse rights, or remove sampling bias. For a route needed by an approved collection, review ProxyLane plans and test the client against one permitted source before scheduling a larger run.

ProxyLaneTraffic never expires

Premium residential proxies
From $2/GB

Scale your workflows with the tools you already use. Rotating or sticky sessions up to 72 hours.

Try 350 MB for $1.95 View plans