Automated Ingestion Transparency

Active Vacancies Crawler Bot Verification

Active Vacancies operates an ethical, serverless web crawler and verification engine that aggregates publicly available career listings directly from official ATS APIs (Greenhouse, Lever, and Ashby).

Official User-Agent Identifiers

1. ATS Ingestion CrawlerPrimary Scraper

ActiveVacanciesBot/1.0 (+https://activevacancies.com/bot; bot@activevacancies.com)

Dispatched on scheduled cron intervals to retrieve public JSON feeds and job boards from ATS partner platforms.

2. Dead-Link & Expiry Verification ProberHealth Prober

ActiveVacanciesVerificationBot/1.0 (+https://activevacancies.com/bot; bot@activevacancies.com)

Executes lightweight HEAD and byte-limited GET probes to verify if previously indexed openings are still active.

Network Origin & Dedicated Static Egress

All outbound crawler traffic originates from our secured Google Cloud VPC in the us-central1 region, routed exclusively through dedicated Google Cloud NAT gateways with reserved static public IP addresses.

Cloud Architecture

Google Cloud VPC + Cloud NAT

Region: us-central1 (Iowa)

Reverse DNS (rDNS)

activevacancies.com

Valid PTR pointer verified

Crawl Governance & Politeness Policy

Strict Token-Bucket Rate Limiting: Crawler requests are throttled through Cloud Tasks queues to a maximum of 1–2 requests per second per ATS host.
HTTP 429 & Backoff Compliance: Our workers immediately halt and back off exponentially when an origin returns HTTP 429 (Too Many Requests) or HTTP 503.
Bandwidth Conservation: Verification workers request HTTP HEAD first. If GET is needed, requests specify Range: bytes=0-4096 to inspect only header closures without downloading asset payloads.
Read-Only Public Data: The crawler never executes POST/PUT mutations, never submits application forms, and never bypasses authentication walls.

Webmaster Inquiries & Crawl Rate Adjustments

If you represent an employer, career portal, or ATS vendor and would like to adjust the crawl frequency, verify crawler legitimacy, or request exclusion from our index, our operations team will accommodate your request promptly.

Contact bot@activevacancies.com Back to Active Vacancies