Gyre

Walk a site from a seed URL within a page budget, following outbound links breadth-first. Returns parsed markdown for every page successfully visited, plus per-URL errors. Same-domain by default. The link-traversal primitive in the Gyrence pipeline — Search finds, Fetch reads one, Gyre walks many.

Use this when

• You need a bounded slice of a site (1–50 pages) without standing up a full crawler.
• You're feeding RAG and want a seed page plus everything it links to, in one call.
• You want partial-success semantics — per-URL failures land in errors[] and the walk continues.
• You're prototyping coverage before committing to a scheduled crawl.

Request

Example body

{
  "url": "https://example.com",
  "maxPages": 10,
  "sameDomain": true
}

Response

Example response

{
  "ok": true,
  "data": {
    "startUrl": "https://example.com",
    "totalPages": 3,
    "pages": [
      {
        "url": "https://example.com",
        "title": "Example",
        "description": "Home page",
        "markdown": "# Example\n\n...",
        "statusCode": 200,
        "via": "http"
      }
    ],
    "errors": []
  }
}

Example

curl -X POST https://www.gyrence.com/api/v1/gyre \
  -H "Authorization: Bearer $GYRENCE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com","maxPages":10}'

Errors

Notes

• Walk semantics. Breadth-first from the seed. Each page's outbound links[] (as extracted by the underlying fetcher) is enqueued. Visited URLs are tracked verbatim.
• Page budget vs queue. maxPages caps successful fetches, not URLs considered. The walk stops as soon as the budget is reached, even if the queue still has unvisited URLs.
• Per-page fetcher. Each page goes through the same two-tier pipeline as /fetch (HTTP → browser escalation, SEC.gov fast path, block-page detection). See the Fetch docs for tier mechanics.
• Failure tolerance. Per-URL failures are pushed to errors[] and the walk continues. The request only fails when nothing was fetched.