> ## Documentation Index
> Fetch the complete documentation index at: https://docs.llmlayer.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Crawl API

> Stream markdown content from multiple pages on a website.

## Overview

Use Crawl when you need content from multiple pages under one site. The endpoint streams Server-Sent Events as pages finish, so your application can process pages without waiting for the full crawl to complete.

The public crawl endpoint currently returns markdown page content only.

## Endpoint

`POST /api/v2/crawl_stream`

## Quickstart

<CodeGroup>
  ```typescript TypeScript theme={null}
  import { LLMLayerClient } from 'llmlayer';

  const client = new LLMLayerClient({
    apiKey: process.env.LLMLAYER_API_KEY,
  });

  for await (const event of client.crawlStream({
    url: 'https://www.ycombinator.com',
    maxPages: 10,
    maxDepth: 2,
    mainContentOnly: true,
  })) {
    if (event.type === 'page') {
      console.log(event.page.final_url, event.page.markdown?.slice(0, 120));
    }

    if (event.type === 'usage') {
      console.log('Cost:', event.cost);
    }
  }
  ```

  ```python Python theme={null}
  from llmlayer import LLMLayerClient

  client = LLMLayerClient(api_key="YOUR_LLMLAYER_API_KEY")

  for event in client.crawl_stream(
      "https://www.ycombinator.com",
      max_pages=10,
      max_depth=2,
      main_content_only=True,
  ):
      if event.get("type") == "page":
          page = event["page"]
          print(page.get("final_url"), (page.get("markdown") or "")[:120])

      if event.get("type") == "usage":
          print("Cost:", event.get("cost"))
  ```

  ```bash cURL theme={null}
  curl -N -X POST https://api.llmlayer.dev/api/v2/crawl_stream \
    -H "Authorization: Bearer YOUR_LLMLAYER_API_KEY" \
    -H "Content-Type: application/json" \
    -H "Accept: text/event-stream" \
    -d '{
      "url": "https://www.ycombinator.com",
      "max_pages": 10,
      "max_depth": 2,
      "main_content_only": true
    }'
  ```
</CodeGroup>

## Request Parameters

| Parameter            | Type             | Required | Default        | Description                                          |
| -------------------- | ---------------- | -------- | -------------- | ---------------------------------------------------- |
| `url`                | `string`         | Yes      | -              | Seed URL                                             |
| `max_pages`          | `integer`        | No       | `25`           | Maximum pages to return, hard limit `100`            |
| `max_depth`          | `integer`        | No       | `2`            | Link depth from the seed URL                         |
| `timeout`            | `number \| null` | No       | `60`           | Total crawl time budget in seconds                   |
| `include_subdomains` | `boolean`        | No       | `false`        | Include subdomains                                   |
| `include_links`      | `boolean`        | No       | `true`         | Keep links in markdown content                       |
| `include_images`     | `boolean`        | No       | `true`         | Keep image references in markdown content            |
| `advanced_proxy`     | `boolean`        | No       | `false`        | Use for protected sites                              |
| `main_content_only`  | `boolean`        | No       | `false`        | Reduce navigation and boilerplate                    |
| `formats`            | `["markdown"]`   | No       | `["markdown"]` | Accepted for compatibility; only markdown is honored |

<Note>
  HTTP requests use `snake_case`. The TypeScript SDK uses `camelCase`, for example `maxPages`, `maxDepth`, and `mainContentOnly`.
</Note>

## Stream Events

Each SSE frame contains a JSON object under `data:`.

### Page

```json theme={null}
{
  "type": "page",
  "page": {
    "requested_url": "https://www.ycombinator.com",
    "final_url": "https://www.ycombinator.com",
    "title": "Y Combinator",
    "hash_sha256": "abc123...",
    "markdown": "# Docs\n\n...",
    "success": true,
    "error": null
  }
}
```

### Usage

```json theme={null}
{
  "type": "usage",
  "billed_count": 10,
  "unit_cost": 0.001,
  "cost": 0.01
}
```

### Done

```json theme={null}
{
  "type": "done",
  "response_time": "21.44"
}
```

### Error

```json theme={null}
{
  "type": "error",
  "error": "Upstream crawl failed"
}
```

## Pricing

Crawl reports usage as `$0.001` per successfully scraped page in the `usage` event. Advanced proxy can improve success rates on protected sites.

Use the emitted `usage.cost` and your dashboard ledger as the billing source of truth.

## When to Use Map First

Use [Map](/map) before Crawl when you want to inspect or filter URLs before fetching page content.

```python theme={null}
site = client.map("https://www.ycombinator.com", limit=100)
print([str(link.url) for link in site.links[:10]])
```

## Errors

| Status / event | Meaning                                       |
| -------------- | --------------------------------------------- |
| `400`          | Invalid URL, invalid `max_pages`, or PDF URL  |
| `401`          | Missing or invalid LLMLayer API key           |
| `500` event    | Upstream crawl failure after streaming starts |

See [Errors & Refunds](/errors) for the shared error format.

## More Examples

<CardGroup cols={2}>
  <Card title="Streaming Crawl Recipes" icon="code" href="/examples/crawl">
    Persist pages, handle usage events, and retry failures.
  </Card>

  <Card title="Map API" icon="sitemap" href="/map">
    Discover URLs before crawling content.
  </Card>
</CardGroup>