Blog
Notes on source intelligence, web data infrastructure, and what we learn building Gyrence. RSS
2026
The Crawler You Didn't Have to Build: Acquiring an Entire Public-Records Site Without Writing One
Agent builders who need every filing off a public-records site end up writing throwaway crawlers to handle URL discovery, rate limiting, and JavaScript rendering — Gyrence's Map and Gyre primitives handle all three inside one credit-bounded call.
Your AI Agent Doesn't Have a Web-Scraping Problem — It Has a Web-Formatting Problem
Agent builders lose more time cleaning navigation chrome, broken tables, and bloated images out of scraped pages than they spend building the agent itself — Gyrence's fetch pipeline detects what it grabbed and reshapes it into markdown or structured data before your agent ever sees it.
Markdown Isn't a Format: Structuring Scraped Web Data So Agents Don't Have to Guess
Agents built to consume scraped web data usually inherit a different parser for every file type they hit; Gyrence's Fetch endpoint instead returns one typed contract across formats, so an agent can consume a press release, a spreadsheet, or a regulatory filing without knowing in advance what it's going to get.
The Same Number, Three Formats: Auditing Data Across Sources Without Losing Precision
Cross-checking a figure across an HTML press release, a CSV dataset, and a structured filing usually means writing a different scraper for each format — Gyrence's Fetch endpoint detects and parses all of them the same way, so the diff compares real values instead of scraper noise.
The Change That Doesn't Show Up in a Diff: Monitoring Investor Relations Pages for New Tax Filings
Investor-relations and tax pages barely change visually from quarter to quarter, so WebDoppler tracks the set of documents it has already seen and treats a newly appeared PDF as the real signal, independent of how little the surrounding DOM moved.
How We Stopped Our AI Agent From Looping on Wikipedia Pagination
Naive graph crawlers burn their entire page budget on pagination archives and mirrored press releases before reaching a single content page. Here's the deterministic fix we shipped — and the numbers.
