Coverageclear

Open source · MIT · For agent builders

Web data is a storm.Gyrence is built for it.

Spending caps you set, enforced before the call ships. Alerts the moment usage spikes. Weekly summaries every Sunday. Failures that announce themselves with structured errors — not empty arrays with 200 OK. Extract bundled with scrape. No separate AI subscription.

Traffic spikes. Your bill doesn't.

If your scraper is a line item on your runway, Gyrence is the rewrite.

Start free See the calculator

Production Stack Live. Servicing the full Suite of Crescentic LLC.

Switching from another provider? Bring your bill →

Gyrence gyre — slowly rotating spiral graphic representing the Gyrence retrieval engine

Live/api/v1/gyrecalm
Listening…

17formats live

verified in production

5primitives

search · gyre · fetch · extract · map

1envelope

{ok,data,code} — HTTP + MCP

25shard timeout

per call — fail fast, predictable cost

// capabilities

Five primitives. One pipeline. Plus MCP.

./ search

Search

Resolve a natural-language query into the pages that actually matter — ranked, deduped, with optional inline fetch as cleaned markdown in one call.

./ fetch

Fetch

One URL in, typed structured data out. A content-type-aware router emits markdown for HTML, parsed JSON/CSV/XML/YAML/feeds/sitemaps/llms.txt for everything else, and a structured refusal for binaries — never a confident-looking wrong answer.

./ gyre

Gyre

Walk a site breadth-first from a seed URL within a page budget. Same-domain by default, partial-success semantics, markdown per page.

./ extract

Extract

Schema-guided structured extraction powered by an LLM. Describe what you want, receive strict JSON. Prompt + optional schema hint.

./ map

Map

Enumerate every URL on a site via its sitemaps, with a depth-1 homepage fallback. One credit flat — no per-page billing, no browser rendering.

./ mcp

MCP

All five primitives exposed as Model Context Protocol tools at /api/mcp. Drop Gyrence into any MCP-compatible agent — Claude Desktop, Cursor, your own client.

The Doppler

Navigate the storm, don't just watch it

Every primitive above pulls data on demand. WebDoppler keeps the gyre turning after you've looked away — watching a page, waiting for the signal that matters, and handing you the structured data the moment it changes.

Point it at anything. A filing index, a regulator's notice board, a competitor's pricing page — any URL Gyrence can reach.

Describe what to capture, once. Plain English. Approve the extraction preview. No schema to maintain.

The storm reports back. A signed webhook fires the moment something changes — structured data already attached.

The gyre finds it. The Doppler tells you when it's worth finding again.

See how WebDoppler works

watching · no change detected

// coverage

Every format the web speaks — returned structured, or refused honestly.

Mint is live and verified in production. Dimmed is on the roadmap. Nothing reaches this grid until it actually returns clean data.

HTML

Live

kind:"html"

Any page on the open web. Returned as cleaned markdown plus the link graph, title, and lightly-normalized HTML.

JSON

Live

kind:"json"

Public JSON endpoints — GitHub, npm, status APIs, anything Content-Type: application/json. Returned typed, never re-stringified.

NDJSON

Live

kind:"ndjson"

Line-delimited JSON streams. EDGAR submission feeds, log exports. Parsed row-by-row, errors isolated per line.

CSV

Live

kind:"csv"

Comma/tab-separated tables — Census, FRED, BLS bulk exports. Header inferred, rows returned as typed records.

YAML

Live

kind:"yaml"

Config and metadata files (OpenAPI specs, GitHub workflows). Parsed to JSON-shaped objects with comments stripped.

XML

Live

kind:"xml"

Generic XML documents — arXiv OAI-PMH, scientific datasets. Returned as a structured tree, not a flattened string.

Sitemaps

Live

kind:"sitemap"

sitemap.xml and sitemap-index recursion. Powers Map and seeds Gyre. Discovered, deduped, normalized to absolute URLs.

RSS / Atom / JSON Feed

Live

kind:"feed"

Every common feed flavor. Items returned with title, link, published timestamp, and a clean body — not raw entity-encoded XML.

Plain text

Live

kind:"text"

robots.txt, ads.txt, LICENSE, README — anything Content-Type: text/plain. Returned verbatim, byte-faithful.

llms.txt / llms-full.txt

Live

kind:"llmstxt"

The site's own curated link map for LLMs. Detected by path + sniff, parsed into structured sections with relative URLs resolved.

XBRL

Live

kind:"xbrl"

Raw XBRL instance documents (.xml) — SEC financial filings at the source. Returned with precision preserved as exact source strings.

iXBRL

Live

kind:"ixbrl"

Inline XBRL — facts embedded in rendered HTML filings, up to ~8 MB. Different parser than raw XBRL; surfaces the same precision contract.

Parquet

Live

kind:"parquet"

Columnar data files → typed rows + schema. Large decimals (precision ≥16) are refused with an approximationRisk flag rather than silently approximated — a value you can see beats a wrong value you can't.

Arrow

Live

kind:"arrow"

Arrow IPC streams → typed rows + schema. Same precision contract as Parquet: decimals that would exceed JS Number's exact range are flagged, not coerced.

PDF

Live

kind:"pdf"

Server-side PDF → markdown. Prose, tables, and document structure preserved. Page-navigable output with metadata. Outperforms other providers' PDF standards on bullet normalization, complex table structure, and page metadata across both presentation decks and dense financial filings.

Office documents

Live

kind:"office" | "office_legacy"

Native structural parse — .docx/.xlsx/.pptx and macro .docm/.xlsm/.pptm. Paragraphs, tables, formulas, slides, speaker notes read directly. No OCR; macros never executed. Legacy .xls OK; old .doc/.ppt unsupported.

Images & OCR

Live

kind:"scanned_pdf" | "pdf_hybrid" | "image_ocr"

Scanned PDFs, hybrid PDFs (mixed text + image pages), and standalone images (PNG / JPEG / GIF / WEBP) run through Anthropic vision OCR. Text-layer pages are still read directly — OCR only touches the pages that need it. Every page returns a confidence signal (high / medium / low), and the model refuses honestly rather than guessing when a page is unreadable — refused and degraded pages are listed by index in the response. Up to 24 pages per document.

Avro — C4.1

Coming

roadmap

Row-oriented container format. Coming as a one-line flip alongside Parquet and Arrow.

Audio / Video transcripts

Coming

roadmap

Podcast and video transcript extraction. Returned as time-anchored text with speaker turns when available.

docABL™ is Crescentic LLC's document intelligence engine — extraction, reconciliation, and audit in a single stack. No third-party document processors. Ever.

Gyre (verb). To walk a starting URL outward — following links, respecting boundaries, surfacing the pages that matter.

Named for the slow rotation of an ocean gyre, which is what the traversal looks like at scale.

// honest by design

The trust is the product.

A scraper that returns a confident-looking wrong answer is worse than one that fails.

./ 01

No silent garbage

Content-type aware. A binary URL returns kind:"unsupported" with a reason — never a populated-but-wrong markdown field. Failure modes are part of the contract.

./ 02

Typed, never guessed

One discriminated union across every primitive. Switch on kind, get a contract — including the failure cases. No "sometimes it's a string, sometimes it's an array."

./ 03

Akamai-grade walls: detected, not faked

Bot-managed hosts (Akamai, PerimeterX, similar) are detected and reported with code:"unavailable" — never saved as fake content. We don't ship a residential-proxy rescue today, and we won't pretend we did. Bypass work is on the roadmap; honesty isn't a feature flag.

./ 04

Precision preserved

Numeric and financial values are returned as exact source strings. No float round-trip, no silent rounding. What the filing said is what you get.

./ 05

Content signals, surfaced — not buried

Every fetch reports the publisher's declared Content-Signal preferences — search, ai-input, ai-train — read from robots.txt and returned verbatim. You see the rules of the road and decide. No signal declared means none invented. Enforcement is on the roadmap; transparency ships today.

// pipeline

Discover. Follow. Extract. Deliver.

Each primitive composes. Search seeds the gyre, the gyre hands pages to fetch, fetch hands content to extract.

Discover

Search resolves a query → URLs. Or seed your own. Map enumerates an entire site's URL surface.

Acquire

Gyre walks the graph. Fetch normalizes each page, auto-escalating to the browser tier when the origin needs JS.

Structure

Extract converts content into the JSON shape your agent expects, guided by your prompt and optional schema.

Deliver

Same {ok, data, code} envelope over HTTP or MCP. Plug into RAG, agents, indexers — billing parity by construction.

// production-proven

Gyrence powers retrieval for Crescentic's product portfolio.

Lacuna Index

Financial-intelligence platform

~12k retrieval calls/day·Search · Fetch · Gyre · Extract

CorpAction Chain

Corporate-actions monitoring

~7.5k retrieval calls/day·Search · Fetch · Gyre · Extract

"Cleaner output than what we replaced. Infrastructure we can actually debug. A cost line that doesn't move when usage does. That's what self-hosted retrieval should mean."
— Glen Bover, Crescentic LLC

// api

An honest, agent-shaped surface.

Five calls. Predictable shapes. No hidden state. Built for the way LLM agents actually reason — call something, get JSON, decide what to do next. Or skip the HTTP entirely: every primitive is also a tool on our hosted MCP endpoint.

Try it live

gyre.search()

gyre.fetch()

gyre.gyre()

gyre.extract()

gyre.map()

/api/v1/fetch