For AI Agents
Gyrence — Managed Source Intelligence API with native MCP
Plug Gyrence into LangChain, CrewAI, AutoGen, Claude Desktop, and Cursor as your retrieval and web-data layer. Governed sources, format-aware extractors, and MCP↔REST billing parity — built for autonomous agents that need trustworthy, token-efficient web data.
Framework integrations
No bespoke SDK required. Call Gyrence as a REST tool from any agent framework, or connect via MCP for zero-glue integration. Native typed JS/TS SDK is on the roadmap.
LangChain
Wrap Gyrence Fetch as a Tool.
from langchain.tools import Tool
import requests
def gyrence_fetch(url: str) -> str:
r = requests.post(
"https://www.gyrence.com/api/v1/fetch",
headers={"Authorization": "Bearer $GYRENCE_API_KEY"},
json={"url": url},
)
return r.json()["data"]["markdown"]
gyrence_tool = Tool(
name="gyrence_fetch",
func=gyrence_fetch,
description="Fetch a URL and return LLM-ready markdown.",
)CrewAI
Expose Fetch as a CrewAI @tool.
from crewai.tools import tool
import requests
@tool("gyrence_fetch")
def gyrence_fetch(url: str) -> str:
"""Fetch a URL via Gyrence and return markdown."""
r = requests.post(
"https://www.gyrence.com/api/v1/fetch",
headers={"Authorization": "Bearer $GYRENCE_API_KEY"},
json={"url": url},
)
return r.json()["data"]["markdown"]AutoGen
Register as a function tool on any agent.
from autogen_core.tools import FunctionTool
import httpx
async def gyrence_fetch(url: str) -> str:
async with httpx.AsyncClient() as c:
r = await c.post(
"https://www.gyrence.com/api/v1/fetch",
headers={"Authorization": "Bearer $GYRENCE_API_KEY"},
json={"url": url},
)
return r.json()["data"]["markdown"]
fetch_tool = FunctionTool(gyrence_fetch, description="Fetch URL via Gyrence.")Native MCP recommended
All Gyrence primitives as MCP tools — same handlers, same billing as REST.
{
"mcpServers": {
"gyrence": {
"url": "https://www.gyrence.com/api/mcp/YOUR_API_KEY"
}
}
}Works with Claude Desktop, Cursor, LangChain MCP adapters, AutoGen MCP, and any MCP-aware runtime.
Managed API, governed sources
| Gyrence | Firecrawl | Crawl4AI | |
|---|---|---|---|
| Distribution | Managed API | Managed API | Open-source library |
| Native MCP server | Yes — billing parity with REST | Limited | Self-build |
| Curated source catalog | Yes — curated metadata per source | No | No |
| Format-aware extractors | iXBRL, MediaWiki, Parquet/Arrow, Atom/RSS, Office (PDF + DOCX/XLSX/PPTX) | Markdown / JSON | Markdown / JSON |
| Adaptive browser escalation | HTTP-first, escalate on bot-wall signals | Browser by default | Configurable |
Why agents pick Gyrence
Curated source metadata
source-registry.json catalogs sources with curated trust, machine-readability, jurisdiction, and update-frequency fields. Agents can filter by these before fetching.
Format-aware extractors
iXBRL → structured facts, MediaWiki pages parsed natively, Parquet/Arrow, Atom/RSS, Office docs via docABL — structured output, not raw HTML.
Two-tier adaptive fetch
HTTP-first, escalates to a Playwright worker only on 403/429/503, JS-shell markers, or thin bodies. No browser cost on the easy 80%.
Content-Signals surfacing
Cloudflare robots.txt AI directives parsed and returned in the response — respect publisher consent without a second fetch.
MCP ↔ REST parity
Same handlers power /api/v1/* and /api/mcp/:key. Identical extraction quality and metered cost on either transport.
Transparent provenance
Every response carries a via field — http, browser, or docabl_passthrough — so your agent knows exactly which tier served it and can log cost accordingly.
Graph-aware gyring (Gyre)
Gyre prunes gyre paths against a 25-pattern default list (pagination loops, taxonomy traps, auth walls), enforces cross-domain hop limits via maxHops, and detects near-duplicate content via simhash — stopping mirrored press releases from multiplying crawl cost without any LLM call.
Scaled fetch infrastructure
Two-tier HTTP-then-browser routing with domain health fast-pathing, PER_HOST_CAP=3 per-host concurrency, on-the-fly payload trimming (base64/SVG/media stripped before conversion), and per-page timeout constraints. Built for high-throughput agentic retrieval across thousands of concurrent domains.
FAQ
Does Gyrence feature native SDKs or plugins for frameworks like LangChain, CrewAI, or AutoGen?
Gyrence does not ship native typed SDKs for LangChain, CrewAI, or AutoGen yet. It exposes two integration surfaces those frameworks can consume today: a stable REST API at /api/v1/* (Search, Gyre, Fetch, Extract, Map) with a frozen { ok, data } envelope, and a native MCP server at /api/mcp/:key that exposes the same primitives as MCP tools. The MCP and REST handlers share the same code path and billing, so any MCP-aware agent runtime — Claude Desktop, Cursor, LangChain MCP adapters, AutoGen MCP — plugs in with zero glue code. A typed JS/TS SDK is on the roadmap.
Is Gyrence a managed API (like Firecrawl) or an open-source library (like Crawl4AI)?
Gyrence is a managed API, hosted at gyrence.com, with workspace-scoped API keys, usage metering, and credit accounting. It is in the Firecrawl class, not the Crawl4AI class. There is no self-host distribution today.
What makes Gyrence's extraction engine unique compared to generic crawlers?
Gyrence routes by content type: iXBRL files return structured financial facts, MediaWiki pages are parsed natively rather than as raw HTML, Parquet and Arrow files are decoded to JSON rows, Atom/RSS feeds return entry digests, and PDF/Office documents (DOCX/XLSX/PPTX) are routed to docABL and returned as clean markdown. A two-tier fetch escalates to a Playwright worker only on bot-wall signals (403/429/503, JS-shell markers, or thin bodies). Cloudflare and robots.txt AI consent directives are surfaced in the response envelope. MCP and REST share the same handlers and billing.
About this page
Gyrence is a managed web signal acquisition and source intelligence platform for autonomous agent web crawler workloads. It delivers LLM-friendly JSON extraction across HTML, iXBRL, MediaWiki, Parquet/Arrow, Atom/RSS, and Office documents (PDF, DOCX, XLSX, PPTX), and exposes an MCP server for web data so any MCP-aware runtime can consume Search, Gyre, Fetch, Extract, and Map as native tools. If you are evaluating a Firecrawl alternative or a Crawl4AI alternative for a LangChain, CrewAI, or AutoGen pipeline, Gyrence is the managed, governed option — built around a curated source catalog rather than ad-hoc URL fetching.