The Crawler You Didn't Have to Build: Acquiring an Entire Public-Records Site Without Writing One

Say you're building an agent that has to answer questions like "has this state agency posted any new enforcement actions this quarter?" The answer isn't on one page. It's scattered across a docket index, a dozen paginated result pages, and a pile of linked filings in HyperText Markup Language (HTML), Portable Document Format (PDF), and the occasional Comma-Separated Values (CSV) export. Before your agent can reason about any of that, something has to go acquire it — and that something is usually where the project quietly turns into an infrastructure project.

The instinct is to write a small crawler. It never stays small. You need to discover the universe of URLs on the site (parse a sitemap if one exists, fall back to crawling links if it doesn't). You need to decide how deep to go and when to stop, or you'll burn your time budget on a pagination archive instead of the docket entries you actually wanted. You need to be polite about it — hammering a government server with fifty concurrent requests is how you get your IP block-listed, which then breaks the feature for good. And you need to know, page by page, whether you're looking at static HyperText Markup Language or a JavaScript-rendered shell that requires an actual browser to produce content — get that wrong and you either waste money running a browser for a page that didn't need one, or you scrape an empty div and never notice.

None of that is the interesting part of the agent you're building. It's also not optional — skip it and your agent's answers are only as good as whatever the last person happened to link to.

Gyrence separates URL discovery from the walk itself, and bounds both by design rather than by convention. The Map primitive enumerates the URLs on a site from its sitemap or its anchor links for a single credit, so you get the shape of the site — every docket-year index, every filing page — before you spend anything trying to read it. From there, Gyre walks the site within a budget you set: up to maxPages: 100 per call, with an 8-second-per-page timeout, a 12-second allowance for the first page, and an 18-second worker ceiling, so a single request can't run away on you. Every concurrent Gyre walk respects a per-host cap of three simultaneous connections — PER_HOST_CAP = 3 — enforced across all walks hitting that host at once, which is the politeness budget the crawler-you-didn't-write would otherwise have had to invent from scratch.

The static-versus-JavaScript question is handled underneath both primitives. Gyrence's fetch pipeline tries a direct HyperText Transfer Protocol (HTTP) request first and escalates automatically to a real, self-hosted headless browser only when it hits a 403 or 429 response, a JavaScript-shell marker, a network error, or content that clearly didn't survive the conversion to markdown. A domain health cache remembers the outcome: a host that's answered cleanly over plain HTTP three times in a row graduates to HTTP-only, so you stop paying browser cost on pages that never needed it, while known browser-required hosts skip the HTTP attempt entirely and go straight to the tier that will actually work.

What comes back from each page isn't a markdown guess. Gyrence's Fetch layer detects what it actually retrieved — HyperText Markup Language, PDF, CSV, JavaScript Object Notation (JSON), a really-large-document XBRL filing — and returns a typed structure for that specific kind, with a via field telling you whether HyperText Transfer Protocol or the browser served it and a kind field telling you what it parsed. If a page is genuinely off-limits — blocked, gated, unsupported — you get a structured refusal back, not a markdown field quietly populated with the wrong thing. For an acquisition pipeline feeding an agent, that distinction matters more than almost anything else in the system: an agent that can tell "I got nothing" from "I got the filing" behaves very differently than one that can't.

Put together, discovering and acquiring a mid-sized public-records site — index pages, docket entries, linked filings, whatever formats they happen to be in — becomes one Map call to find the URLs and one budgeted Gyre call to walk them, instead of a bespoke crawler your team now has to maintain, rate-limit, and debug every time the target site changes its markup. It's worth being direct about the current edges: Gyre's single-call ceiling is 100 pages, bounded by a 25-second per-request deadline — sites bigger than that need multiple chained calls today, since asynchronous batch jobs aren't available yet. And a small set of heavily-defended origins still get past the stealth configuration and have to fall back elsewhere. Both primitives, along with Fetch, Search, Extract, and Resolve, are also exposed as Model Context Protocol (MCP) tools at /api/mcp/:key, so an agent runtime that already speaks MCP can call Map and Gyre directly instead of going through the REST contract by hand.

The point isn't that acquisition disappears — it's that it stops being a project of its own. Discovery, bounded traversal, politeness, and format detection are handled once, underneath the primitive, so the thing you're actually building can start at "here's the data" instead of "here's a scraper I now own."