Gyrence — Intelligence in Motion
Sign inSign up
Coverageclear
Open source · MIT · For agent builders

Web data is a storm.Gyrence is built for it.

Spending caps you set, enforced before the call ships. Alerts the moment usage spikes. Weekly summaries every Sunday. Failures that announce themselves with structured errors — not empty arrays with 200 OK. Extract bundled with scrape. No separate AI subscription.

Traffic spikes. Your bill doesn't.

If your scraper is a line item on your runway, Gyrence is the rewrite.

Start free See the calculator
Production Stack Live. Servicing the full Suite of Crescentic LLC.
Switching from another provider? Bring your bill →
Gyrence gyre — slowly rotating spiral graphic representing the Gyrence retrieval engine
Live/api/v1/gyrecalm
Listening…
18formats live
verified in production
5primitives
search · gyre · fetch · extract · map
1envelope
{ok,data,code} — HTTP + MCP
25shard timeout
per call — fail fast, predictable cost
// capabilities

Five primitives. One pipeline. Plus MCP.

./ search

Search

Resolve a natural-language query into the pages that actually matter — ranked, deduped, with optional inline fetch as cleaned markdown in one call.

./ fetch

Fetch

One URL in, typed structured data out. A content-type-aware router emits markdown for HTML, parsed JSON/CSV/XML/YAML/feeds/sitemaps/llms.txt for everything else, and a structured refusal for binaries — never a confident-looking wrong answer.

./ gyre

Gyre

Walk a site breadth-first from a seed URL within a page budget. Same-domain by default, partial-success semantics, markdown per page.

./ extract

Extract

Schema-guided structured extraction powered by an LLM. Describe what you want, receive strict JSON. Prompt + optional schema hint.

./ map

Map

Enumerate every URL on a site via its sitemaps, with a depth-1 homepage fallback. One credit flat — no per-page billing, no browser rendering.

./ mcp

MCP

All five primitives exposed as Model Context Protocol tools at /api/mcp. Drop Gyrence into any MCP-compatible agent — Claude Desktop, Cursor, your own client.

The Doppler

Navigate the storm, don't just watch it

Every primitive above pulls data on demand. WebDoppler keeps the gyre turning after you've looked away — watching a page, waiting for the signal that matters, and handing you the structured data the moment it changes.

01

Point it at anything. A filing index, a regulator's notice board, a competitor's pricing page — any URL Gyrence can reach.

02

Describe what to capture, once. Plain English. Approve the extraction preview. No schema to maintain.

03

The storm reports back. A signed webhook fires the moment something changes — structured data already attached.

The gyre finds it. The Doppler tells you when it's worth finding again.

See how WebDoppler works
watching · no change detected
// coverage

Every format the web speaks — returned structured, or refused honestly.

Mint is live and verified in production. Dimmed is on the roadmap. Nothing reaches this grid until it actually returns clean data.

Arrow

Live
kind:"arrow"

Arrow IPC streams → typed rows + schema. Same precision contract as Parquet: decimals that would exceed JS Number's exact range are flagged, not coerced.

Avro

Live
kind:"avro"

Avro object container files → typed rows + schema. null, deflate, and snappy codecs (CRC-verified); zstandard and bzip2 fail loudly rather than half-reading. Records, unions, enums, maps, arrays, fixed, and recursive schemas resolve. Same precision contract as Parquet: decimals and out-of-range longs return as exact strings, never rounded. Timestamps keep micro/nano precision as exact strings — decoded from raw bytes, so nothing passes through a float.

CSV

Live
kind:"csv"

Comma/tab-separated tables — Census, FRED, BLS bulk exports. Header inferred, rows returned as typed records.

HTML

Live
kind:"html"

Any page on the open web. Returned as cleaned markdown plus the link graph, title, and lightly-normalized HTML.

iXBRL

Live
kind:"ixbrl"

Inline XBRL — facts embedded in rendered HTML filings, up to ~8 MB. Different parser than raw XBRL; surfaces the same precision contract.

JSON

Live
kind:"json"

Public JSON endpoints — GitHub, npm, status APIs, anything Content-Type: application/json. Returned typed, never re-stringified.

llms.txt / llms-full.txt

Live
kind:"llmstxt"

The site's own curated link map for LLMs. Detected by path + sniff, parsed into structured sections with relative URLs resolved.

NDJSON

Live
kind:"ndjson"

Line-delimited JSON streams. EDGAR submission feeds, log exports. Parsed row-by-row, errors isolated per line.

Parquet

Live
kind:"parquet"

Columnar data files → typed rows + schema. Large decimals (precision ≥16) are refused with an approximationRisk flag rather than silently approximated — a value you can see beats a wrong value you can't.

Plain text

Live
kind:"text"

robots.txt, ads.txt, LICENSE, README — anything Content-Type: text/plain. Returned verbatim, byte-faithful.

RSS / Atom / JSON Feed

Live
kind:"feed"

Every common feed flavor. Items returned with title, link, published timestamp, and a clean body — not raw entity-encoded XML.

Sitemaps

Live
kind:"sitemap"

sitemap.xml and sitemap-index recursion. Powers Map and seeds Gyre. Discovered, deduped, normalized to absolute URLs.

XBRL

Live
kind:"xbrl"

Raw XBRL instance documents (.xml) — SEC financial filings at the source. Returned with precision preserved as exact source strings.

XML

Live
kind:"xml"

Generic XML documents — arXiv OAI-PMH, scientific datasets. Returned as a structured tree, not a flattened string.

YAML

Live
kind:"yaml"

Config and metadata files (OpenAPI specs, GitHub workflows). Parsed to JSON-shaped objects with comments stripped.

Images & OCR

Live
kind:"scanned_pdf" | "pdf_hybrid" | "image_ocr"

Scanned PDFs, hybrid PDFs (mixed text + image pages), and standalone images (PNG / JPEG / GIF / WEBP) run through Anthropic vision OCR. Text-layer pages are still read directly — OCR only touches the pages that need it. Every page returns a confidence signal (high / medium / low), and the model refuses honestly rather than guessing when a page is unreadable — refused and degraded pages are listed by index in the response. Up to 24 pages per document.

Powered bydocABL

Office documents

Live
kind:"office" | "office_legacy"

Native structural parse — .docx/.xlsx/.pptx and macro .docm/.xlsm/.pptm. Paragraphs, tables, formulas, slides, speaker notes read directly. No OCR; macros never executed. Legacy .xls OK; old .doc/.ppt unsupported.

Powered bydocABL

PDF

Live
kind:"pdf"

Server-side PDF → markdown. Prose, tables, and document structure preserved. Page-navigable output with metadata. Outperforms other providers' PDF standards on bullet normalization, complex table structure, and page metadata across both presentation decks and dense financial filings.

Powered bydocABL

Audio / Video transcripts

Coming
roadmap

Podcast and video transcript extraction. Returned as time-anchored text with speaker turns when available.

docABL

docABL™ is Crescentic LLC's document intelligence engine — extraction, reconciliation, and audit in a single stack. No third-party document processors. Ever.

Gyre (verb). To walk a starting URL outward — following links, respecting boundaries, surfacing the pages that matter.

Named for the slow rotation of an ocean gyre, which is what a Gyre run looks like at scale.

Why teams pick Gyrence

Source-first retrieval

The answer is only as good as the source behind it.

Gyrence scores every source on trust, machine readability, and freshness before it enters your pipeline. Trace every answer back to publisher, jurisdiction, and timestamp.

Trust first. Answer second. · Built for RAG, regulatory monitoring, and research pipelines.

Explore sourcesRead about content signals →
MCP-ready

Drop Gyrence into any agent workflow.

All five primitives — Search, Gyre, Fetch, Extract, Map — exposed as Model Context Protocol tools. No new SDK to learn. One envelope, one auth model, instant integration.

Agents ask. Gyrence answers. · Works with Claude, Cursor, and any MCP client.

Try the MCP endpointView API docs →
Verifiable coverage

Every format we claim is one we run in production.

HTML, JSON, CSV, XML, YAML, feeds, sitemaps, PDFs, Office documents, Avro, and scanned images with vision OCR — each one live, each one returning the same envelope with a structured refusal instead of a confident guess.

No demos. No asterisks. · One envelope, a 25s hard timeout, predictable cost per call.

See format coverageRead the fetch reference →
// honest by design

The trust is the product.

A scraper that returns a confident-looking wrong answer is worse than one that fails.

./ 01

No silent garbage

Content-type aware. A binary URL returns kind:"unsupported" with a reason — never a populated-but-wrong markdown field. Failure modes are part of the contract.

./ 02

Typed, never guessed

One discriminated union across every primitive. Switch on kind, get a contract — including the failure cases. No "sometimes it's a string, sometimes it's an array."

./ 03

Akamai-grade walls: detected, not faked

Bot-managed hosts (Akamai, PerimeterX, similar) are detected and reported with code:"unavailable" — never saved as fake content. We don't ship a residential-proxy rescue today, and we won't pretend we did. Bypass work is on the roadmap; honesty isn't a feature flag.

./ 04

Precision preserved

Numeric and financial values are returned as exact source strings. No float round-trip, no silent rounding. What the filing said is what you get.

./ 05

Content signals, surfaced — not buried

Every fetch reports the publisher's declared Content-Signal preferences — search, ai-input, ai-train — read from robots.txt and returned verbatim. You see the rules of the road and decide. No signal declared means none invented. Enforcement is on the roadmap; transparency ships today.

// pipeline

Discover. Follow. Extract. Deliver.

Each primitive composes. Search seeds the gyre, the gyre hands pages to fetch, fetch hands content to extract.

01
Discover

Search resolves a query → URLs. Or seed your own. Map enumerates an entire site's URL surface.

02
Acquire

Gyre walks the graph. Fetch normalizes each page, auto-escalating to the browser tier when the origin needs JS.

03
Structure

Extract converts content into the JSON shape your agent expects, guided by your prompt and optional schema.

04
Deliver

Same {ok, data, code} envelope over HTTP or MCP. Plug into RAG, agents, indexers — billing parity by construction.

// production-proven

Gyrence powers retrieval for Crescentic's product portfolio.

Lacuna Index logo
Lacuna Index
Financial-intelligence platform
~12k retrieval calls/day·Search · Fetch · Gyre · Extract
CorpAction Chain logo
CorpAction Chain
Corporate-actions monitoring
~7.5k retrieval calls/day·Search · Fetch · Gyre · Extract
the305Home logo
the305Home
Furniture & home-goods discovery
~3.2k retrieval calls/day·Search · Fetch · Extract
Leviathan Data Intelligence logo
Leviathan DI
Data-intelligence monitoring
~1.8k retrieval calls/day·WebDoppler · Fetch · Extract

"Cleaner output than what we replaced. Infrastructure we can actually debug. A cost line that doesn't move when usage does. That's what self-hosted retrieval should mean."

— Glen Bover, Crescentic LLC
// api

An honest, agent-shaped surface.

Five calls. Predictable shapes. No hidden state. Built for the way LLM agents actually reason — call something, get JSON, decide what to do next. Or skip the HTTP entirely: every primitive is also a tool on our hosted MCP endpoint.

Try it live
gyre.search()
gyre.fetch()
gyre.gyre()
gyre.extract()
gyre.map()
/api/v1/fetch