> ## Documentation Index > Fetch the complete documentation index at: https://docs.llmlayer.ai/llms.txt > Use this file to discover all available pages before exploring further. # Crawl API - Stream markdown pages from a website > Crawl multiple pages from a website with depth control. Streams Server-Sent Events as pages are scraped. The public endpoint currently returns markdown content per page. Billed per successfully scraped page ($0.001/page). ## OpenAPI ````yaml post /api/v2/crawl_stream openapi: 3.0.0 info: title: LLMLayer API version: 2.0.0 description: >- The Web API for AI Agents - Search, scrape, extract content, crawl websites, and generate AI-powered answers with multiple LLMs servers: - url: https://api.llmlayer.dev security: [] tags: - name: Answer API description: >- Web-enhanced AI responses combining search with multiple LLM models. Supports streaming, custom prompts, and structured JSON output. - name: Web Search description: >- Direct web search across multiple content types (general, news, shopping, videos, images, scholar) without AI processing. - name: Content Extraction description: >- Extract content from websites, PDFs, and YouTube videos in multiple formats. - name: Web Crawling description: >- Map websites and crawl multiple pages with depth control. Supports streaming with markdown output. paths: /api/v2/crawl_stream: post: tags: - Web Crawling summary: Crawl API - Stream markdown pages from a website description: >- Crawl multiple pages from a website with depth control. Streams Server-Sent Events as pages are scraped. The public endpoint currently returns markdown content per page. Billed per successfully scraped page ($0.001/page). operationId: crawlWebsite requestBody: required: true content: application/json: schema: $ref: '#/components/schemas/CrawlRequest' responses: '200': description: Server-Sent Events stream of crawled pages content: text/event-stream: schema: type: string description: 'SSE stream with event types: page, usage, done, error' examples: page_event: value: >+ data: {"type":"page","page":{"requested_url":"https://www.ycombinator.com","final_url":"https://www.ycombinator.com","title":"Y Combinator","hash_sha256":"abc123...","markdown":"# Content...","success":true}} usage_event: value: >+ data: {"type":"usage","billed_count":25,"unit_cost":0.001,"cost":0.025} done_event: value: |+ data: {"type":"done","response_time":"45.67"} '400': $ref: '#/components/responses/BadRequest' '401': $ref: '#/components/responses/Unauthorized' '500': $ref: '#/components/responses/InternalServerError' security: - BearerAuth: [] components: schemas: CrawlRequest: type: object required: - url properties: url: type: string format: uri description: Seed URL to start crawling from example: https://www.ycombinator.com max_pages: type: integer default: 25 minimum: 1 maximum: 100 description: 'Maximum number of pages to crawl (hard limit: 100)' max_depth: type: integer default: 2 minimum: 1 description: Maximum depth to crawl from seed URL timeout: type: number nullable: true default: 60 description: Total timeout in seconds for the entire crawl operation include_subdomains: type: boolean default: false description: If true, includes pages from subdomains include_links: type: boolean default: true description: Include hyperlinks in extracted content include_images: type: boolean default: true description: Include images in extracted content advanced_proxy: type: boolean default: false nullable: true description: Enable advanced proxy for protected sites. main_content_only: type: boolean default: false nullable: true description: Extract only main page content. formats: type: array items: type: string enum: - markdown default: - markdown description: >- Accepted for compatibility; only markdown is currently honored by /api/v2/crawl_stream. example: - markdown ErrorResponse: type: object required: - detail properties: detail: type: object required: - error_type - error_code - message properties: error_type: type: string enum: - validation_error - authentication_error - provider_error - rate_limit - internal_error - scraping_error - search_error - map_error description: Category of error error_code: type: string description: Specific error code for programmatic handling example: missing_query message: type: string description: Human-readable error message details: type: object additionalProperties: true nullable: true description: Additional error context and debugging information responses: BadRequest: description: Invalid request parameters content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' examples: missing_query: value: detail: error_type: validation_error error_code: missing_query message: Query parameter cannot be empty missing_json_schema: value: detail: error_type: validation_error error_code: missing_json_schema message: JSON schema is required for JSON response type invalid_max_pages: value: detail: error_type: validation_error error_code: invalid_max_pages message: max_pages must be between 1 and 100 Unauthorized: description: Authentication error - missing or invalid API key content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' example: detail: error_type: authentication_error error_code: missing_llmlayer_api_key message: 'Provide LLMLayer API key via ''Authorization: Bearer ''' InternalServerError: description: Internal server error content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' examples: unexpected_error: value: detail: error_type: internal_error error_code: unexpected_error message: An unexpected error occurred. Please try again later scraping_error: value: detail: error_type: scraping_error error_code: url_scrape_failed message: Failed to scrape content from the provided URL details: url: https://www.ycombinator.com error: Connection timeout securitySchemes: BearerAuth: type: http scheme: bearer description: >- Bearer token authentication using your LLMLayer API key. Include in Authorization header as: Bearer YOUR_LLMLAYER_API_KEY ````