Amazon Scraper API vs Proxy: Choose a Product Data Source
Distinguish authorized Amazon APIs, licensed product data and proxy-based page checks by record quality and access rights.
Read guideJob pages are easy to count and easy to misread. One vacancy can appear on an employer site, an applicant-tracking system and several boards. A stale page can still return 200, while a new URL can be a repost of the same requisition. A proxy may help a permitted fetch reflect a target region. The output still needs unique-vacancy logic and a sample-bias note because posting rows do not describe the whole labor market. Treat sample bias as part of the output, not a footnote.
Define the unit before collecting
Start with the decision: compare hiring activity for named employers, map role demand in a region or monitor a set of public career pages. Freeze the employers, role vocabulary, locations, source types and observation dates. Check official feeds or APIs and the source terms first. Read the robots rules, keep request pacing bounded and pause when a target asks you to slow down.
| Input | Action | Accepted output |
|---|---|---|
| Employer, role and location frame | Collect a permitted single-job page and its visible fields | A dated posting record with source URL |
| Multiple URLs for one apparent role | Normalize title, employer, location and source signals | A provisional vacancy group with merge evidence |
| Recheck date and page status | Revisit the same URL or canonical page | open, closed, uncertain or not_checked with evidence |
Keep posting_id and vacancy_id separate. The first identifies a page observation. The second is a hypothesis that several observations refer to one hiring request. If the evidence is weak, leave the group provisional instead of presenting a guessed unique vacancy count.
Use publisher fields as signals
Google's JobPosting documentation says structured data belongs on a page dedicated to a single job posting, not a search or list page. It documents fields such as datePosted and validThrough, and recommends canonical job URLs and accurate change timestamps. Schema.org's JobPosting type provides a vocabulary for job-related fields.
These fields are useful observations, not a unique-vacancy key. Store the visible title, employer name, location or remote requirement, employment type, datePosted, validThrough, application URL, canonical URL, retrieval time and the structured-data version when present. Compare those fields with the page text. A page's markup can be missing, stale or copied across a board.
Deduplicate with evidence, then review
Create a candidate key from normalized employer, title, location, employment type and a time window. Add a requisition ID, canonical URL or application URL when the publisher exposes one. A title alone is too weak: “Data Analyst” can describe several teams, locations and levels. A URL alone is also weak when a board regenerates links during a repost.
Use the merge result as a review queue:
| Observation | Suggested state | Why |
|---|---|---|
| Same requisition ID and application URL | same_vacancy |
Strong publisher identifier |
| Same employer, title and location; close dates | likely_duplicate |
Useful hypothesis, still reviewable |
| Same title but different location or team | separate_or_unknown |
Do not collapse by title |
| New URL with older text and no new identifier | repost_candidate |
Preserve both posting observations |
Here is a hypothetical batch, included to show the denominator rather than report a market result: 120 posting URLs produce 86 candidate groups after normalization. Review accepts 73 groups for the stated frame and leaves 13 unresolved. A row-level audit also flags 19 duplicate links, eight repost candidates and seven ambiguous groups; those flags can overlap, so they are diagnostics rather than extra rows to subtract. The 73 groups are operational vacancy groups, not a headcount: one requisition can fill several seats, and one posting can describe a role family. The example is synthetic and says nothing about the size of a real market.
Track closed status as a time series
An absent validThrough date does not prove that a role is open. Record explicit “no longer accepting applications” text, a closed application form, a redirect to a careers index, 404 or 410, and the last date on which a valid job description was observed. Google's guidance says expired job postings should not remain presented as open; its documented handling includes current validThrough data or removing the expired posting. Your collector still needs its own status field and recheck schedule.
Store first_seen, last_seen, status_checked_at and status_evidence. Keep an older valid observation when a temporary fetch fails, but mark its age. A 200 page containing a login wall or a list of other jobs is not an open vacancy record. The cost-per-successful-request guide shows why a successful transport response is different from an accepted business record.
Describe the sample before the conclusion
A board-only sample favors employers that publish there. A career-site sample favors companies with discoverable pages. Search results favor indexed and crawlable postings. Geography, language, remote eligibility, salary disclosure, duplicate policy and observation time create more selection effects. Report the employer list, source mix, role and location filters, collection window, exclusions, missing fields and unique-vacancy rule before saying that demand rose or fell.
If a response is 429, RFC 6585 defines it as “Too Many Requests” and allows a Retry-After field. Honor that signal across the target schedule. Switching exits to defeat a target limit corrupts the observation plan and can create a more biased sample. A proxy belongs only where the permitted fetch needs a different route or regional presentation check.
Use the market research guide for sampling frames and named exclusions. The RAG crawling guide shows how to retain URL and freshness evidence when the job inventory is refreshed. The permission checklist covers source terms, robots and personal-data questions; legal conclusions remain source- and jurisdiction-specific.
ProxyLane offers HTTP and SOCKS5 residential access with rotating or sticky sessions, location targeting and non-expiring traffic. Plans start from $2.50/GB. Request access after defining the source set, pacing rule and accepted vacancy record.