For AI Agents
Governed sources, format-aware extraction, change detection, and per-response provenance — for agents working with financial and regulatory data.
No bespoke SDK required. Call Gyrence as a REST tool from any agent framework, or connect via MCP for zero-glue integration. Native typed JS/TS SDK is on the roadmap.
Wrap Gyrence Fetch as a Tool.
import os
from langchain.tools import Tool
import requests
def gyrence_fetch(url: str) -> str:
r = requests.post(
"https://www.gyrence.com/api/v1/fetch",
headers={"Authorization": f"Bearer {os.environ['GYRENCE_API_KEY']}"},
json={"url": url},
)
return r.json()["data"]["markdown"]
gyrence_tool = Tool(
name="gyrence_fetch",
func=gyrence_fetch,
description="Fetch a URL and return LLM-ready markdown.",
)Expose Fetch as a CrewAI @tool.
import os
from crewai.tools import tool
import requests
@tool("gyrence_fetch")
def gyrence_fetch(url: str) -> str:
"""Fetch a URL via Gyrence and return markdown."""
r = requests.post(
"https://www.gyrence.com/api/v1/fetch",
headers={"Authorization": f"Bearer {os.environ['GYRENCE_API_KEY']}"},
json={"url": url},
)
return r.json()["data"]["markdown"]Register as a function tool on any agent.
import os
from autogen_core.tools import FunctionTool
import httpx
async def gyrence_fetch(url: str) -> str:
async with httpx.AsyncClient() as c:
r = await c.post(
"https://www.gyrence.com/api/v1/fetch",
headers={"Authorization": f"Bearer {os.environ['GYRENCE_API_KEY']}"},
json={"url": url},
)
return r.json()["data"]["markdown"]
fetch_tool = FunctionTool(gyrence_fetch, description="Fetch URL via Gyrence.")Search, Fetch, Gyre, Extract, and Map as MCP tools — same handlers, same billing as REST.
{
"mcpServers": {
"gyrence": {
"url": "https://www.gyrence.com/api/mcp",
"headers": {
"Authorization": "Bearer mc_your_key_here"
}
}
}
}Works with Claude Desktop, Cursor, LangChain MCP adapters, AutoGen MCP, and any MCP-aware runtime.
| Gyrence | Firecrawl | Crawl4AI | |
|---|---|---|---|
| Distribution | Managed API | Managed API | Open-source library |
| Curated source catalog | Yes — curated metadata per source | No | No |
| Format-aware extractors | iXBRL, MediaWiki, Parquet/Arrow, Atom/RSS, Office (PDF + DOCX/XLSX/PPTX) | Markdown / JSON | Markdown / JSON |
The source catalog includes curated trust, machine-readability, jurisdiction, and update-frequency fields. Agents can filter by these before fetching.
iXBRL → structured facts, MediaWiki pages parsed natively, Parquet/Arrow, Atom/RSS, Office docs via docABL — structured output, not raw HTML.
HTTP-first, escalates to a Playwright worker only on 403/429/503, JS-shell markers, or thin bodies. No browser cost on the easy 80%.
Cloudflare robots.txt AI directives parsed and returned in the response — respect publisher consent without a second fetch.
Same handlers power /api/v1/* and /api/mcp. Identical extraction quality and metered cost on either transport.
Every response carries a via field — http, browser, or docabl_passthrough — so your agent knows exactly which tier served it and can log cost accordingly.
Gyre prunes crawl paths against a 25-pattern default list (pagination loops, taxonomy traps, auth walls), enforces cross-domain hop limits via maxHops, and detects near-duplicate content via simhash — stopping mirrored press releases from multiplying crawl cost without any LLM call.
Two-tier HTTP-then-browser routing with domain health fast-pathing, per-host concurrency caps, on-the-fly payload trimming (base64/SVG/media stripped before conversion), and per-page timeout constraints.
WebDoppler monitors a URL, detects content changes, returns the diff, and sends configured notifications.
Gyrence does not ship native typed SDKs for LangChain, CrewAI, or AutoGen yet. It exposes two integration surfaces those frameworks can consume today: a stable REST API at /api/v1/* (Search, Fetch, Gyre, Extract, Map) with a frozen { ok, data } envelope, and a native MCP server at /api/mcp that exposes the same primitives in the same order. The MCP and REST handlers share the same code path and billing, so any MCP-aware agent runtime — Claude Desktop, Cursor, LangChain MCP adapters, AutoGen MCP — plugs in with zero glue code. A typed JS/TS SDK is on the roadmap.
Gyrence is a managed API, hosted at gyrence.com, with workspace-scoped API keys, usage metering, and credit accounting. It is in the Firecrawl class, not the Crawl4AI class. There is no self-host distribution today.
Gyrence routes by content type: iXBRL files return structured financial facts, MediaWiki pages are parsed natively rather than as raw HTML, Parquet and Arrow files are decoded to JSON rows, Atom/RSS feeds return entry digests, and PDF/Office documents (DOCX/XLSX/PPTX) are routed to docABL and returned as clean markdown. A two-tier fetch escalates to a Playwright worker only on bot-wall signals (403/429/503, JS-shell markers, or thin bodies). Cloudflare and robots.txt AI consent directives are surfaced in the response envelope. MCP and REST share the same handlers and billing.
Gyrence is a managed web signal acquisition and source intelligence platform for autonomous agent web crawler workloads. It delivers LLM-friendly JSON extraction across HTML, iXBRL, MediaWiki, Parquet/Arrow, Atom/RSS, and Office documents (PDF, DOCX, XLSX, PPTX), and exposes an MCP server for web data so any MCP-aware runtime can consume Search, Fetch, Gyre, Extract, and Map as native tools. If you are evaluating a Firecrawl alternative or a Crawl4AI alternative for a LangChain, CrewAI, or AutoGen pipeline, Gyrence is the managed, governed option — built around a curated source catalog rather than ad-hoc URL fetching.