Crawl API

Overview

Use Crawl when you need content from multiple pages under one site. The endpoint streams Server-Sent Events as pages finish, so your application can process pages without waiting for the full crawl to complete. The public crawl endpoint currently returns markdown page content only.

Endpoint

POST /api/v2/crawl_stream

Quickstart

import { LLMLayerClient } from 'llmlayer';

const client = new LLMLayerClient({
  apiKey: process.env.LLMLAYER_API_KEY,
});

for await (const event of client.crawlStream({
  url: 'https://www.ycombinator.com',
  maxPages: 10,
  maxDepth: 2,
  mainContentOnly: true,
})) {
  if (event.type === 'page') {
    console.log(event.page.final_url, event.page.markdown?.slice(0, 120));
  }

  if (event.type === 'usage') {
    console.log('Cost:', event.cost);
  }
}

from llmlayer import LLMLayerClient

client = LLMLayerClient(api_key="YOUR_LLMLAYER_API_KEY")

for event in client.crawl_stream(
    "https://www.ycombinator.com",
    max_pages=10,
    max_depth=2,
    main_content_only=True,
):
    if event.get("type") == "page":
        page = event["page"]
        print(page.get("final_url"), (page.get("markdown") or "")[:120])

    if event.get("type") == "usage":
        print("Cost:", event.get("cost"))

curl -N -X POST https://api.llmlayer.dev/api/v2/crawl_stream \
  -H "Authorization: Bearer YOUR_LLMLAYER_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Accept: text/event-stream" \
  -d '{
    "url": "https://www.ycombinator.com",
    "max_pages": 10,
    "max_depth": 2,
    "main_content_only": true
  }'

Request Parameters

Parameter	Type	Required	Default	Description
`url`	`string`	Yes	-	Seed URL
`max_pages`	`integer`	No	`25`	Maximum pages to return, hard limit `100`
`max_depth`	`integer`	No	`2`	Link depth from the seed URL
`timeout`	`number \| null`	No	`60`	Total crawl time budget in seconds
`include_subdomains`	`boolean`	No	`false`	Include subdomains
`include_links`	`boolean`	No	`true`	Keep links in markdown content
`include_images`	`boolean`	No	`true`	Keep image references in markdown content
`advanced_proxy`	`boolean`	No	`false`	Use for protected sites
`main_content_only`	`boolean`	No	`false`	Reduce navigation and boilerplate
`formats`	`["markdown"]`	No	`["markdown"]`	Accepted for compatibility; only markdown is honored

HTTP requests use snake_case. The TypeScript SDK uses camelCase, for example maxPages, maxDepth, and mainContentOnly.

Stream Events

Each SSE frame contains a JSON object under data:.

Page

{
  "type": "page",
  "page": {
    "requested_url": "https://www.ycombinator.com",
    "final_url": "https://www.ycombinator.com",
    "title": "Y Combinator",
    "hash_sha256": "abc123...",
    "markdown": "# Docs\n\n...",
    "success": true,
    "error": null
  }
}

Usage

{
  "type": "usage",
  "billed_count": 10,
  "unit_cost": 0.001,
  "cost": 0.01
}

Done

{
  "type": "done",
  "response_time": "21.44"
}

Error

{
  "type": "error",
  "error": "Upstream crawl failed"
}

Pricing

Crawl reports usage as $0.001 per successfully scraped page in the usage event. Advanced proxy can improve success rates on protected sites. Use the emitted usage.cost and your dashboard ledger as the billing source of truth.

When to Use Map First

Use Map before Crawl when you want to inspect or filter URLs before fetching page content.

site = client.map("https://www.ycombinator.com", limit=100)
print([str(link.url) for link in site.links[:10]])

Errors

Status / event	Meaning
`400`	Invalid URL, invalid `max_pages`, or PDF URL
`401`	Missing or invalid LLMLayer API key
`500` event	Upstream crawl failure after streaming starts

See Errors & Refunds for the shared error format.

More Examples

Streaming Crawl Recipes

Persist pages, handle usage events, and retry failures.

Map API

Discover URLs before crawling content.

Get Started

Resources

Examples

Overview

Endpoint

Quickstart

Request Parameters

Stream Events

Page

Usage

Done

Error

Pricing

When to Use Map First

Errors

More Examples

Streaming Crawl Recipes

Map API

​Overview

​Endpoint

​Quickstart

​Request Parameters

​Stream Events

​Page

​Usage

​Done

​Error

​Pricing

​When to Use Map First

​Errors

​More Examples

Streaming Crawl Recipes

Map API

Overview

Endpoint

Quickstart

Request Parameters

Stream Events

Page

Usage

Done

Error

Pricing

When to Use Map First

Errors

More Examples