Gyre

Walk a site from a seed URL within a page budget, following outbound links breadth-first. Returns parsed markdown for every page successfully visited, plus per-URL errors. Same-domain by default. The link-traversal primitive in the Gyrence pipeline — Search finds, Fetch reads one, Gyre walks many. Gyre includes built-in pagination loop pruning, simhash near-duplicate content deduplication, cross-domain hop control, and per-host concurrency limiting — purpose-built for AI agent graph traversal workloads.

Graph traversal capabilities

Gyre executes targeted network graph discovery across publicly indexable web nodes, isolating graph navigation from data extraction.

• Deterministic graph pruning — traversal paths are evaluated against a 25-pattern default prune list (pagination traps, taxonomy loops, auth walls). Override via prunePaths or disable with allowPrune: false.
• Cross-domain hop control — maxHops enforces strict domain-boundary limits for multi-hop research workflows. Combine with sameDomain: false for controlled cross-domain traversal.
• Near-duplicate suppression — simhash fingerprinting prunes redundant branches spawned by mirrored content (≥85% overlap threshold) without LLM cost. The page itself is still returned in pages[].
• Two-tier dynamic routing — automatic escalation from lightweight HTTP to headless browser rendering on JS-shell detection. No browser cost on the easy 80%.

Use this when

• You need a bounded slice of a site (1–50 pages) without standing up a full crawler.
• You're feeding RAG and want a seed page plus everything it links to, in one call.
• You want partial-success semantics — per-URL failures land in errors[] and the walk continues.
• You're prototyping coverage before committing to a scheduled crawl.

Request

Example body

{
  "url": "https://example.com",
  "maxPages": 10,
  "sameDomain": true
}

Response

Example response

{
  "ok": true,
  "data": {
    "startUrl": "https://example.com",
    "totalPages": 3,
    "pages": [
      {
        "url": "https://example.com",
        "title": "Example",
        "description": "Home page",
        "markdown": "# Example\n\n...",
        "statusCode": 200,
        "via": "http"
      }
    ],
    "errors": []
  }
}

Example

curl -X POST https://www.gyrence.com/api/v1/gyre \
  -H "Authorization: Bearer $GYRENCE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com","maxPages":10}'

Errors

Notes

• Walk semantics. Breadth-first from the seed. Each page's outbound links[] (as extracted by the underlying fetcher) is enqueued. Visited URLs are tracked verbatim.
• Page budget vs queue. maxPages caps successful fetches, not URLs considered. The walk stops as soon as the budget is reached, even if the queue still has unvisited URLs.
• Per-page fetcher. Each page goes through the same two-tier pipeline as /fetch (HTTP → browser escalation, SEC.gov fast path, block-page detection). See the Fetch docs for tier mechanics.
• Failure tolerance. Per-URL failures are pushed to errors[] and the walk continues. The request only fails when nothing was fetched.