Markdown Isn't a Format: Structuring Scraped Web Data So Agents Don't Have to Guess

Every agent builder who has wired a scraper into a tool call eventually hits the same wall: the open web doesn't hand you one kind of document, it hands you a dozen. A press release is HTML. A quarterly filing might be an Extensible Markup Language (XML) file with exact decimal values that matter. A dataset is a Comma-Separated Values (CSV) file, or increasingly a Parquet file. A filing amendment is a scanned Portable Document Format (PDF) with no selectable text at all. If your pipeline was built for one of these, it silently breaks — or worse, returns something plausible-looking and wrong — the moment it meets one of the others.

Most scraping tools solve this by converting everything to Markdown and calling it done. That works for a blog post. It does not work for a table of tax withholding rates, and it does not work for a regulatory filing where a rounded number is a different number.

Gyrence's Fetch endpoint takes a different approach: instead of forcing every response into one shape, it detects what the content actually is and returns a typed answer that says so. Every response carries a kind field — the detected content type — and a via field, noting whether the page was served by a plain HTTP fetch or needed a real browser to render. That's not cosmetic. An agent consuming the response can branch on kind instead of trying to sniff the payload itself, and the contract holds even for the failure cases: a blocked page comes back as a structured error, not a wall of challenge-page HTML dressed up as content.

One request, more than a dozen recognized shapes

Point Fetch at a URL and it will tell you, honestly, what it found — HyperText Markup Language (HTML) pages get cleaned and converted to Markdown; JavaScript Object Notation (JSON), CSV, and XML come back structured rather than flattened into prose; sitemaps and RSS/Atom feeds are parsed into their entries; PDFs, Word documents, Excel spreadsheets, and PowerPoint files are handed off to Gyrence's sibling document-parsing engine and come back as clean Markdown with tables and structure intact — scanned pages included, via an Optical Character Recognition (OCR) fallback that renders the page as an image and reads it. Even llms.txt files, the emerging convention sites use to describe themselves to AI agents, get parsed into their own structured sections instead of being treated as generic text.

The Markdown conversion itself is more deliberate than it looks. Before anything gets converted, Gyrence tries to isolate the actual article content from navigation, footers, and sidebars — the same heuristic reader-view tools use, falling back gracefully if no clear main content block exists. Tables with merged cells get expanded so a Markdown table renderer doesn't silently drop data that a rowspan was carrying. It's the unglamorous 20% of scraping that eats most of an agent-building afternoon, done once so an agent doesn't have to redo it per page.

Precision is part of the format

For financial and regulatory data specifically, "close enough" formatting is a real risk. Gyrence treats numeric fidelity as part of the contract, not an afterthought: values from formats like Parquet or Arrow that would lose precision if forced into a standard JavaScript number are flagged with an approximationRisk marker instead of being silently rounded, and values from tagged financial filing formats are returned as the exact strings the filing used — no float round-trip, no quiet rounding. A value you can see beats a wrong value you can't.

What agents can reach today — honestly

Gyrence also runs a Model Context Protocol (MCP) server, so an agent can call these primitives as tools directly instead of going through a Representational State Transfer (REST) client. As of this writing, that MCP surface wraps five of Gyrence's core primitives — Search, Fetch, Gyre (site traversal), Map, and Extract. It doesn't yet expose Resolve (Gyrence's entity-verification lookup) or WebDoppler (its change-monitoring feature) as MCP tools, so an agent that needs those still talks to the REST API directly. Worth knowing before you assume "MCP-native" means every feature is one tool call away — right now, most of them are.

One more thing worth surfacing rather than burying: Gyrence reads and reports the Content-Signal header sites can set to declare how they want their content used — for search, for AI input, for AI training. Gyrence passes that signal through on the response rather than enforcing it silently or inventing an answer when a site hasn't declared one. If your agent's pipeline needs to respect those signals, the information is there to act on; if a site hasn't said anything, Gyrence won't pretend it did.

That's the underlying bet: an agent doesn't need every source normalized into the same shallow format, it needs a format it can trust — one that tells the truth about what kind of thing it just fetched, how confident to be in the numbers, and what it couldn't do.