Stop Feeding Your Agent HTML Soup

If you have built an agent that reads the web, you already know the annoying part isn't the model. It's everything upstream of the model. You fetch a page, you get back a wall of HyperText Markup Language (HTML) — navigation bars, cookie banners, script tags, and, somewhere in the middle, the three paragraphs you actually wanted. Now you're writing a cleanup pass. Then the site changes its layout and your cleanup pass breaks. Then someone points your agent at a CSV file, or a PDF (Portable Document Format), or a company's financial filing, and the whole pipeline needs a special case.

This is the part of "web data" work that never shows up in demos: the format problem. A large language model doesn't care about your data source's history, but it cares enormously about the shape of what you hand it. Loose HTML burns tokens on markup the model has to mentally discard. A flattened text dump loses the structure — tables become word salad, links disappear, headings blur into paragraphs. And when a fetch quietly fails and returns an empty or garbled string instead of an error, the agent doesn't know to ask for help. It just reasons confidently about nothing.

Gyrence, the retrieval application programming interface (API) we built at Crescentic, treats this as the actual problem to solve, not an afterthought bolted onto a scraper. The core idea is that Fetch — one of Gyrence's six primitives, alongside Search, Gyre, Extract, Map, and Resolve — is content-aware. It looks at a uniform resource locator (URL) and figures out what's actually behind it before deciding how to parse it. A news article and a spreadsheet and a sitemap all come back through the same endpoint, but each comes back shaped for what it is, not force-fit into one generic template.

Concretely: point Fetch at an HTML page and you get cleaned markdown, a link graph, and a title — not raw markup. Point it at a JavaScript Object Notation (JSON) endpoint and you get a typed object, never re-stringified into text you'd have to re-parse. Point it at a comma-separated values (CSV) file and the header row is inferred and rows come back as typed records. Point it at an RSS or Atom feed and you get individual items with titles, links, and publish timestamps instead of one long blob. Sixteen content types are handled this way in total, from spreadsheets and slide decks to financial filings in eXtensible Business Reporting Language (XBRL) format, and every single response carries a kind field telling you exactly what Gyrence decided the content was, plus a via field telling you whether it was served by a fast HyperText Transfer Protocol (HTTP) request or a full headless browser render.

That second part matters more than it sounds like it should. An agent pipeline that can't tell the difference between "this is a clean extraction" and "this is a best-effort guess" will eventually make a decision based on garbage with total confidence. Gyrence is built to be honest about failure instead: a blocked page returns a structured error, not a markdown field that looks populated but is actually wrong. A financial filing with decimal precision beyond what a JavaScript number can represent exactly gets flagged with an approximationRisk note rather than silently rounded. If your agent is going to act on a number, it should know when that number might not be exact.

For teams building with agent frameworks directly, Gyrence also runs a native Model Context Protocol (MCP) server, so the same six primitives show up as tools an agent can call directly — same handlers, same billing as the regular API, no separate integration to maintain. And for sites that publish an llms.txt or llms-full.txt file — a newer convention some sites use specifically to describe themselves to language models — Gyrence parses it into structured sections rather than treating it as one more text file to guess at.

None of this replaces good judgment about what to fetch and when. But it removes a whole category of work that has nothing to do with your actual product: writing brittle HTML cleanup, guessing at file types, and re-implementing parsers for CSV, YAML (Yet Another Markup Language), Extensible Markup Language (XML), and whatever else a target site throws at you. The web wasn't built for agents to read. Gyrence's job is to sit between the two and make sure what reaches your model looks like data, not like a page.

If you're building an agent that touches the open web, the format of what you feed it is worth as much attention as the model you feed it to.