Your AI Agent Doesn't Have a Web-Scraping Problem — It Has a Web-Formatting Problem

If you've built an agent that reads the open web, you already know the real cost isn't fetching the page — any script can send a request to a web address, or URL, and get bytes back. The cost is everything that happens after: figuring out what you actually got, stripping out the navigation bar and cookie banner and footer links that have nothing to do with the content, fixing a table where merged cells turned into garbled rows, and hoping the page didn't just hand you a multi-megabyte image buried in the markup, quietly eating your model's context budget.

Most scraping tools treat this as the agent builder's problem to solve downstream. Gyrence, a retrieval application programming interface (API) built for AI agents, treats it as part of the fetch itself.

Fetch is one of six core Gyrence primitives — alongside search, structured extraction, site crawling, sitemap discovery, and query resolution — that share the same request and response conventions. But the formatting problem shows up most sharply in the raw fetch itself, which is where most of this work happens.

Know what you got before you decide what to do with it

Every Fetch request runs through a detection step before any parsing happens. The response gets classified into one of eighteen content types — ordinary web pages, JSON (a common structured data format), comma-separated data, spreadsheets, PDFs, Word and PowerPoint files, sitemaps, RSS-style feeds, financial filings marked up in a structured format regulators require, and more — checking file signatures first, then response headers, then the web address itself, then the actual content as a last resort. That classification comes back as a kind field, so an agent doesn't have to guess whether a link pointed at a webpage or a spreadsheet before deciding how to read it.

Markdown that isn't full of noise

For ordinary pages, Gyrence doesn't just strip HTML tags and call it markdown. It identifies the actual main content of the page — preferring a marked main region, then an article that clearly dominates the page's text, and only falling back to the full page body as a last resort — and discards the navigation bars, forms, and structural chrome around it. Tables with merged or spanning cells get rebuilt so data lines up correctly instead of scrambling into the wrong rows, and on financial or data-heavy tables, currency figures that got split across cells are re-merged. Embedded vector graphics are dropped entirely rather than dumped as unreadable code, and embedded frames become plain links instead of disappearing silently.

A browser only when the page actually needs one

Some pages return real content to a simple, fast request. Others are effectively empty shells until a browser runs their code first. Gyrence tries the fast path first and only escalates to a real headless browser — one it runs and maintains itself — when the fast path is blocked or the page shows signs of needing a browser to render anything. Every response carries a via field noting which path it took: a plain fetch, a browser render, or a document-parsing pass. That matters when you're debugging a bad extraction — you can tell immediately whether the page came back as raw markup, a rendered page, or a parsed file, instead of guessing.

Turning a page into structured data, not just text

For cases where an agent needs specific facts rather than a wall of markdown, Gyrence's Extract primitive takes a plain-language instruction — for example, "give me the product name and price" — and returns structured JSON pulled from the page's content. Long pages get truncated before that step runs, and the response explicitly reports whether truncation happened and how long the original content was, so a builder knows when they're looking at a partial answer instead of silently trusting an incomplete one.

Speaking the format agents already expect

Some sites have started publishing a simple text file, commonly called llms.txt, that describes their content for AI crawlers. Gyrence publishes its own documentation this way, and when Fetch encounters an llms.txt file on someone else's site, it parses it into structured sections rather than returning it as one undifferentiated block of text. And because agents increasingly connect to tools through the Model Context Protocol (MCP) — a standard for exposing external tools directly to AI models — Gyrence's core primitives are available as native MCP tools, using the exact same parameter definitions as the regular API. An agent can call fetch or extract directly as a tool, without a separate integration layer translating between a scraping API and whatever the agent framework expects.

None of this replaces good agent design, and it won't get past every anti-bot defense on the internet. But it moves a real, recurring cost — turning whatever the web hands back into something a model can actually use without choking on it — out of the agent builder's code and into the fetch itself.