1 line
57 KiB
JSON
1 line
57 KiB
JSON
{"repo": "3lokai/icb-backend-v2", "files": {"docs/crawl4ai-docs/core/docker-deployment.md": {"f8d7b62d08225d6f8a61987dd9e13be30dbb1972": {"t": 1785747174.474086, "n": 27915, "h": "c5900a0bc553dec4", "c": "# Crawl4AI Docker Guide 🐳\n\n## Table of Contents\n- [Prerequisites](#prerequisites)\n- [Installation](#installation)\n - [Option 1: Using Pre-built Docker Hub Images (Recommended)](#option-1-using-pre-built-docker-hub-images-recommended)\n - [Option 2: Using Docker Compose](#option-2-using-docker-compose)\n - [Option 3: Manual Local Build & Run](#option-3-manual-local-build--run)\n- [Dockerfile Parameters](#dockerfile-parameters)\n- [Using the API](#using-the-api)\n - [Playground Interface](#playground-interface)\n - [Python SDK](#python-sdk)\n - [Understanding Request Schema](#understanding-request-schema)\n - [REST API Examples](#rest-api-examples)\n- [Additional API Endpoints](#additional-api-endpoints)\n - [HTML Extraction Endpoint](#html-extraction-endpoint)\n - [Screenshot Endpoint](#screenshot-endpoint)\n - [PDF Export Endpoint](#pdf-export-endpoint)\n - [JavaScript Execution Endpoint](#javascript-execution-endpoint)\n - [Library Context Endpoint](#library-context-endpoint)\n- [MCP (Model Context Protocol) Support](#mcp-model-context-protocol-support)\n - [What is MCP?](#what-is-mcp)\n - [Connecting via MCP](#connecting-via-mcp)\n - [Using with Claude Code](#using-with-claude-code)\n - [Available MCP Tools](#available-mcp-tools)\n - [Testing MCP Connections](#testing-mcp-connections)\n - [MCP Schemas](#mcp-schemas)\n- [Metrics & Monitoring](#metrics--monitoring)\n- [Deployment Scenarios](#deployment-scenarios)\n- [Complete Examples](#complete-examples)\n- [Server Configuration](#server-configuration)\n - [Understanding config.yml](#understanding-configyml)\n - [JWT Authentication](#jwt-authentication)\n - [Configuration Tips and Best Practices](#configuration-tips-and-best-practices)\n - [Customizing Your Configuration](#customizing-your-configuration)\n - [Configuration Recommendations](#configuration-recommendations)\n- [Getting Help](#getting-help)\n- [Summary](#summary)\n\n## Prerequisites\n\nBefore we dive in, make sure you have:\n- Docker installed and running (version 20.10.0 or higher), including `docker compose` (usually bundled with Docker Desktop).\n- `git` for cloning the repository.\n- At least 4GB of RAM available for the container (more recommended for heavy use).\n- Python 3.10+ (if using the Python SDK).\n- Node.js 16+ (if using the Node.js examples).\n\n> 💡 **Pro tip**: Run `docker info` to check your Docker installation and available resources.\n\n## Installation\n\nWe offer several ways to get the Crawl4AI server running. The quickest way is to use our pre-built Docker Hub images.\n\n### Option 1: Using Pre-built Docker Hub Images (Recommended)\n\nPull and run images directly from Docker Hub without building locally.\n\n#### 1. Pull the Image\n\nOur latest release candidate is `0.6.0-r2`. Images are built with multi-arch manifests, so Docker automatically pulls the correct version for your system.\n\n```bash\n# Pull the release candidate (recommended for latest features)\ndocker pull unclecode/crawl4ai:0.6.0-r1\n\n# Or pull the latest stable version\ndocker pull unclecode/crawl4ai:latest\n```\n\n#### 2. Setup Environment (API Keys)\n\nIf you plan to use LLMs, create a `.llm.env` file in your working directory:\n\n```bash\n# Create a .llm.env file with your API keys\ncat > .llm.env << EOL\n# OpenAI\nOPENAI_API_KEY=sk-your-key\n\n# Anthropic\nANTHROPIC_API_KEY=your-anthropic-key\n\n# Other providers as needed\n# DEEPSEEK_API_KEY=your-deepseek-key\n# GROQ_API_KEY=your-groq-key\n# TOGETHER_API_KEY=your-together-key\n# MISTRAL_API_KEY=your-mistral-key\n# GEMINI_API_TOKEN=your-gemini-token\nEOL\n```\n> 🔑 **Note**: Keep your API keys secure! Never commit `.llm.env` to version control.\n\n#### 3. Run the Container\n\n* **Basic run:**\n ```bash\n docker run -d \\\n -p 11235:11235 \\\n --name crawl4ai \\\n --shm-size=1g \\\n unclecode/crawl4ai:latest\n ```\n\n* **With LLM support:**\n ```bash\n # Make sure .llm.env is in the current directory\n docker run -d \\\n -p 11235:11235 \\\n --name crawl4ai \\\n --env-file .llm.env \\\n --shm-size=1g \\\n unclecode/crawl4ai:latest\n ```\n\n> The server will be available at `http://localhost:11235`. Visit `/playground` to access the interactive testing interface.\n\n#### 4. Stopping the Container\n\n```bash\ndocker stop crawl4ai && docker rm crawl4ai\n```\n\n#### Docker Hub Versioning Explained\n\n* **Image Name:** `unclecode/crawl4ai`\n* **Tag Format:** `LIBRARY_VERSION[-SUFFIX]` (e.g., `0.6.0-r2`)\n * `LIBRARY_VERSION`: The semantic version of the core `crawl4ai` Python library\n * `SUFFIX`: Optional tag for release candidates (``) and revisions (`r1`)\n* **`latest` Tag:** Points to the most recent stable version\n* **Multi-Architecture Support:** All images support both `linux/amd64` and `linux/arm64` architectures through a single tag\n\n### Option 2: Using Docker Compose\n\nDocker Compose simplifies building and running the service, especially for local development and testing.\n\n#### 1. Clone Repository\n\n```bash\ngit clone https://github.com/unclecode/crawl4ai.git\ncd crawl4ai\n```\n\n#### 2. Environment Setup (API Keys)\n\nIf you plan to use LLMs, copy the example environment file and add your API keys. This file should be in the **project root directory**.\n\n```bash\n# Make sure you are in the 'crawl4ai' root directory\ncp deploy/docker/.llm.env.example .llm.env\n\n# Now edit .llm.env and add your API keys\n```\n\n#### 3. Build and Run with Compose\n\nThe `docker-compose.yml` file in the project root provides a simplified approach that automatically handles architecture detection using buildx.\n\n* **Run Pre-built Image from Docker Hub:**\n ```bash\n # Pulls and runs the release candidate from Docker Hub\n # Automatically selects the correct architecture\n IMAGE=unclecode/crawl4ai:latest docker compose up -d\n ```\n\n* **Build and Run Locally:**\n ```bash\n # Builds the image locally using Dockerfile and runs it\n # Automatically uses the correct architecture for your machine\n docker compose up --build -d\n ```\n\n* **Customize the Build:**\n ```bash\n # Build with all features (includes torch and transformers)\n INSTALL_TYPE=all docker compose up --build -d\n \n # Build with GPU support (for AMD64 platforms)\n ENABLE_GPU=true docker compose up --build -d\n ```\n\n> The server will be available at `http://localhost:11235`.\n\n#### 4. Stopping the Service\n\n```bash\n# Stop the service\ndocker compose down\n```\n\n### Option 3: Manual Local Build & Run\n\nIf you prefer not to use Docker Compose for direct control over the build and run process.\n\n#### 1. Clone Repository & Setup Environment\n\nFollow steps 1 and 2 from the Docker Compose section above (clone repo, `cd crawl4ai`, create `.llm.env` in the root).\n\n#### 2. Build the Image (Multi-Arch)\n\nUse `docker buildx` to build the image. Crawl4AI now uses buildx to handle multi-architecture builds automatically.\n\n```bash\n# Make sure you are in the 'crawl4ai' root directory\n# Build for the current architecture and load it into Docker\ndocker buildx build -t crawl4ai-local:latest --load .\n\n# Or build for multiple architectures (useful for publishing)\ndocker buildx build --platform linux/amd64,linux/arm64 -t crawl4ai-local:latest --load .\n\n# Build with additional options\ndocker buildx build \\\n --build-arg INSTALL_TYPE=all \\\n --build-arg ENABLE_GPU=false \\\n -t crawl4ai-local:latest --load .\n```\n\n#### 3. Run the Container\n\n* **Basic run (no LLM support):**\n ```bash\n docker run -d \\\n -p 11235:11235 \\\n --name crawl4ai-standalone \\\n --shm-size=1g \\\n crawl4ai-local:latest\n ```\n\n* **With LLM support:**\n ```bash\n # Make sure .llm.env is in the current directory (project root)\n docker run -d \\\n -p 11235:11235 \\\n --name crawl4ai-standalone \\\n --env-file .llm.env \\\n --shm-size=1g \\\n crawl4ai-local:latest\n ```\n\n> The server will be available at `http://localhost:11235`.\n\n#### 4. Stopping the Manual Container\n\n```bash\ndocker stop crawl4ai-standalone && docker rm crawl4ai-standalone\n```\n\n---\n\n## MCP (Model Context Protocol) Support\n\nCrawl4AI server includes support for the Model Context Protocol (MCP), allowing you to connect the server's capabilities directly to MCP-compatible clients like Claude Code.\n\n### What is MCP?\n\nMCP is an open protocol that standardizes how applications provide context to LLMs. It allows AI models to access external tools, data sources, and services through a standardized interface.\n\n### Connecting via MCP\n\nThe Crawl4AI server exposes two MCP endpoints:\n\n- **Server-Sent Events (SSE)**: `http://localhost:11235/mcp/sse`\n- **WebSocket**: `ws://localhost:11235/mcp/ws`\n\n### Using with Claude Code\n\nYou can add Crawl4AI as an MCP tool provider in Claude Code with a simple command:\n\n```bash\n# Add the Crawl4AI server as an MCP provider\nclaude mcp add --transport sse c4ai-sse http://localhost:11235/mcp/sse\n\n# List all MCP providers to verify it was added\nclaude mcp list\n```\n\nOnce connected, Claude Code can directly use Crawl4AI's capabilities like screenshot capture, PDF generation, and HTML processing without having to make separate API calls.\n\n### Available MCP Tools\n\nWhen connected via MCP, the following tools are available:\n\n- `md` - Generate markdown from web content\n- `html` - Extract preprocessed HTML\n- `screenshot` - Capture webpage screenshots\n- `pdf` - Generate PDF documents\n- `execute_js` - Run JavaScript on web pages\n- `crawl` - Perform multi-URL crawling\n- `ask` - Query the Crawl4AI library context\n\n### Testing MCP Connections\n\nYou can test the MCP WebSocket connection using the test file included in the repository:\n\n```bash\n# From the repository root\npython tests/mcp/test_mcp_socket.py\n```\n\n### MCP Schemas\n\nAccess the MCP tool schemas at `http://localhost:11235/mcp/schema` for detailed information on each tool's parameters and capabilities.\n\n---\n\n## Additional API Endpoints\n\nIn addition to the core `/crawl` and `/crawl/stream` endpoints, the server provides several specialized endpoints:\n\n### HTML Extraction Endpoint\n\n```\nPOST /html\n```\n\nCrawls the URL and returns preprocessed HTML optimized for schema extraction.\n\n```json\n{\n \"url\": \"https://example.com\"\n}\n```\n\n### Screenshot Endpoint\n\n```\nPOST /screenshot\n```\n\nCaptures a full-page PNG screenshot of the specified URL.\n\n```json\n{\n \"url\": \"https://example.com\",\n \"screenshot_wait_for\": 2,\n \"output_path\": \"/path/to/save/screenshot.png\"\n}\n```\n\n- `screenshot_wait_for`: Optional delay in seconds before capture (default: 2)\n- `output_path`: Optional path to save the screenshot (recommended)\n\n### PDF Export Endpoint\n\n```\nPOST /pdf\n```\n\nGenerates a PDF document of the specified URL.\n\n```json\n{\n \"url\": \"https://example.com\",\n \"output_path\": \"/path/to/save/document.pdf\"\n}\n```\n\n- `output_path`: Optional path to save the PDF (recommended)\n\n### JavaScript Execution Endpoint\n\n```\nPOST /execute_js\n```\n\nExecutes JavaScript snippets on the specified URL and returns the full crawl result.\n\n```json\n{\n \"url\": \"https://example.com\",\n \"scripts\": [\n \"return document.title\",\n \"return Array.from(document.querySelectorAll('a')).map(a => a.href)\"\n ]\n}\n```\n\n- `scripts`: List of JavaScript snippets to execute sequentially\n\n---\n\n## Dockerfile Parameters\n\nYou can customize the image build process using build arguments (`--build-arg`). These are typically used via `docker buildx build` or within the `docker-compose.yml` file.\n\n```bash\n# Example: Build with 'all' features using buildx\ndocker buildx build \\\n --platform linux/amd64,linux/arm64 \\\n --build-arg INSTALL_TYPE=all \\\n -t yourname/crawl4ai-all:latest \\\n --load \\\n . # Build from root context\n```\n\n### Build Arguments Explained\n\n| Argument | Description | Default | Options |\n| :----------- | :--------------------------------------- | :-------- | :--------------------------------- |\n| INSTALL_TYPE | Feature set | `default` | `default`, `all`, `torch`, `transformer` |\n| ENABLE_GPU | GPU support (CUDA for AMD64) | `false` | `true`, `false` |\n| APP_HOME | Install path inside container (advanced) | `/app` | any valid path |\n| USE_LOCAL | Install library from local source | `true` | `true`, `false` |\n| GITHUB_REPO | Git repo to clone if USE_LOCAL=false | *(see Dockerfile)* | any git URL |\n| GITHUB_BRANCH| Git branch to clone if USE_LOCAL=false | `main` | any branch name |\n\n*(Note: PYTHON_VERSION is fixed by the `FROM` instruction in the Dockerfile)*\n\n### Build Best Practices\n\n1. **Choose the Right Install Type**\n * `default`: Basic installation, smallest image size. Suitable for most standard web scraping and markdown generation.\n * `all`: Full features including `torch` and `transformers` for advanced extraction strategies (e.g., CosineStrategy, certain LLM filters). Significantly larger image. Ensure you need these extras.\n2. **Platform Considerations**\n * Use `buildx` for building multi-architecture images, especially for pushing to registries.\n * Use `docker compose` profiles (`local-amd64`, `local-arm64`) for easy platform-specific local builds.\n3. **Performance Optimization**\n * The image automatically includes platform-specific optimizations (OpenMP for AMD64, OpenBLAS for ARM64).\n\n---\n\n## Using the API\n\nCommunicate with the running Docker server via its REST API (defaulting to `http://localhost:11235`). You can use the Python SDK or make direct HTTP requests.\n\n### Playground Interface\n\nA built-in web playground is available at `http://localhost:11235/playground` for testing and generating API requests. The playground allows you to:\n\n1. Configure `CrawlerRunConfig` and `BrowserConfig` using the main library's Python syntax\n2. Test crawling operations directly from the interface\n3. Generate corresponding JSON for REST API requests based on your configuration\n\nThis is the easiest way to translate Python configuration to JSON requests when building integrations.\n\n### Python SDK\n\nInstall the SDK: `pip install crawl4ai`\n\n```python\nimport asyncio\nfrom crawl4ai.docker_client import Crawl4aiDockerClient\nfrom crawl4ai import BrowserConfig, CrawlerRunConfig, CacheMode # Assuming you have crawl4ai installed\n\nasync def main():\n # Point to the correct server port\n async with Crawl4aiDockerClient(base_url=\"http://localhost:11235\", verbose=True) as client:\n # If JWT is enabled on the server, authenticate first:\n # await client.authenticate(\"user@example.com\") # See Server Configuration section\n\n # Example Non-streaming crawl\n print(\"--- Running Non-Streaming Crawl ---\")\n results = await client.crawl(\n [\"https://httpbin.org/html\"],\n browser_config=BrowserConfig(headless=True), # Use library classes for config aid\n crawler_config=CrawlerRunConfig(cache_mode=CacheMode.BYPASS)\n )\n if results: # client.crawl returns None on failure\n print(f\"Non-streaming results success: {results.success}\")\n if results.success:\n for result in results: # Iterate through the CrawlResultContainer\n print(f\"URL: {result.url}, Success: {result.success}\")\n else:\n print(\"Non-streaming crawl failed.\")\n\n\n # Example Streaming crawl\n print(\"\\n--- Running Streaming Crawl ---\")\n stream_config = CrawlerRunConfig(stream=True, cache_mode=CacheMode.BYPASS)\n try:\n async for result in await client.crawl( # client.crawl returns an async generator for streaming\n [\"https://httpbin.org/html\", \"https://httpbin.org/links/5/0\"],\n browser_config=BrowserConfig(headless=True),\n crawler_config=stream_config\n ):\n print(f\"Streamed result: URL: {result.url}, Success: {result.success}\")\n except Exception as e:\n print(f\"Streaming crawl failed: {e}\")\n\n\n # Example Get schema\n print(\"\\n--- Getting Schema ---\")\n schema = await client.get_schema()\n print(f\"Schema received: {bool(schema)}\") # Print whether schema was received\n\nif __name__ == \"__main__\":\n asyncio.run(main())\n```\n\n*(SDK parameters like timeout, verify_ssl etc. remain the same)*\n\n### Second Approach: Direct API Calls\n\nCrucially, when sending configurations directly via JSON, they **must** follow the `{\"type\": \"ClassName\", \"params\": {...}}` structure for any non-primitive value (like config objects or strategies). Dictionaries must be wrapped as `{\"type\": \"dict\", \"value\": {...}}`.\n\n*(Keep the detailed explanation of Configuration Structure, Basic Pattern, Simple vs Complex, Strategy Pattern, Complex Nested Example, Quick Grammar Overview, Important Rules, Pro Tip)*\n\n#### More Examples *(Ensure Schema example uses type/value wrapper)*\n\n**Advanced Crawler Configuration**\n*(Keep example, ensure cache_mode uses valid enum value like \"bypass\")*\n\n**Extraction Strategy**\n```json\n{\n \"crawler_config\": {\n \"type\": \"CrawlerRunConfig\",\n \"params\": {\n \"extraction_strategy\": {\n \"type\": \"JsonCssExtractionStrategy\",\n \"params\": {\n \"schema\": {\n \"type\": \"dict\",\n \"value\": {\n \"baseSelector\": \"article.post\",\n \"fields\": [\n {\"name\": \"title\", \"selector\": \"h1\", \"type\": \"text\"},\n {\"name\": \"content\", \"selector\": \".content\", \"type\": \"html\"}\n ]\n }\n }\n }\n }\n }\n }\n}\n```\n\n**LLM Extraction Strategy** *(Keep example, ensure schema uses type/value wrapper)*\n*(Keep Deep Crawler Example)*\n\n### REST API Examples\n\nUpdate URLs to use port `11235`.\n\n#### Simple Crawl\n\n```python\nimport requests\n\n# Configuration objects converted to the required JSON structure\nbrowser_config_payload = {\n \"type\": \"BrowserConfig\",\n \"params\": {\"headless\": True}\n}\ncrawler_config_payload = {\n \"type\": \"CrawlerRunConfig\",\n \"params\": {\"stream\": False, \"cache_mode\": \"bypass\"} # Use string value of enum\n}\n\ncrawl_payload = {\n \"urls\": [\"https://httpbin.org/html\"],\n \"browser_config\": browser_config_payload,\n \"crawler_config\": crawler_config_payload\n}\nresponse = requests.post(\n \"http://localhost:11235/crawl\", # Updated port\n # headers={\"Authorization\": f\"Bearer {token}\"}, # If JWT is enabled\n json=crawl_payload\n)\nprint(f\"Status Code: {response.status_code}\")\nif response.ok:\n print(response.json())\nelse:\n print(f\"Error: {response.text}\")\n\n```\n\n#### Streaming Results\n\n```python\nimport json\nimport httpx # Use httpx for async streaming example\n\nasync def test_stream_crawl(token: str = None): # Made token optional\n \"\"\"Test the /crawl/stream endpoint with multiple URLs.\"\"\"\n url = \"http://localhost:11235/crawl/stream\" # Updated port\n payload = {\n \"urls\": [\n \"https://httpbin.org/html\",\n \"https://httpbin.org/links/5/0\",\n ],\n \"browser_config\": {\n \"type\": \"BrowserConfig\",\n \"params\": {\"headless\": True, \"viewport\": {\"type\": \"dict\", \"value\": {\"width\": 1200, \"height\": 800}}} # Viewport needs type:dict\n },\n \"crawler_config\": {\n \"type\": \"CrawlerRunConfig\",\n \"params\": {\"stream\": True, \"cache_mode\": \"bypass\"}\n }\n }\n\n headers = {}\n # if token:\n # headers = {\"Authorization\": f\"Bearer {token}\"} # If JWT is enabled\n\n try:\n async with httpx.AsyncClient() as client:\n async with client.stream(\"POST\", url, json=payload, headers=headers, timeout=120.0) as response:\n print(f\"Status: {response.status_code} (Expected: 200)\")\n response.raise_for_status() # Raise exception for bad status codes\n\n # Read streaming response line-by-line (NDJSON)\n async for line in response.aiter_lines():\n if line:\n try:\n data = json.loads(line)\n # Check for completion marker\n if data.get(\"status\") == \"completed\":\n print(\"Stream completed.\")\n break\n print(f\"Streamed Result: {json.dumps(data, indent=2)}\")\n except json.JSONDecodeError:\n print(f\"Warning: Could not decode JSON line: {line}\")\n\n except httpx.HTTPStatusError as e:\n print(f\"HTTP error occurred: {e.response.status_code} - {e.response.text}\")\n except Exception as e:\n print(f\"Error in streaming crawl test: {str(e)}\")\n\n# To run this example:\n# import asyncio\n# asyncio.run(test_stream_crawl())\n```\n\n---\n\n## Metrics & Monitoring\n\nKeep an eye on your crawler with these endpoints:\n\n- `/health` - Quick health check\n- `/metrics` - Detailed Prometheus metrics\n- `/schema` - Full API schema\n\nExample health check:\n```bash\ncurl http://localhost:11235/health\n```\n\n---\n\n*(Deployment Scenarios and Complete Examples sections remain the same, maybe update links if examples moved)*\n\n---\n\n## Server Configuration\n\nThe server's behavior can be customized through the `config.yml` file.\n\n### Understanding config.yml\n\nThe configuration file is loaded from `/app/config.yml` inside the container. By default, the file from `deploy/docker/config.yml` in the repository is copied there during the build.\n\nHere's a detailed breakdown of the configuration options (using defaults from `deploy/docker/config.yml`):\n\n```yaml\n# Application Configuration\napp:\n title: \"Crawl4AI API\"\n version: \"1.0.0\" # Consider setting this to match library version, e.g., \"0.5.1\"\n host: \"0.0.0.0\"\n port: 8020 # NOTE: This port is used ONLY when running server.py directly. Gunicorn overrides this (see supervisord.conf).\n reload: False # Default set to False - suitable for production\n timeout_keep_alive: 300\n\n# Default LLM Configuration\nllm:\n provider: \"openai/gpt-4o-mini\"\n api_key_env: \"OPENAI_API_KEY\"\n # api_key: sk-... # If you pass the API key directly then api_key_env will be ignored\n\n# Redis Configuration (Used by internal Redis server managed by supervisord)\nredis:\n host: \"localhost\"\n port: 6379\n db: 0\n password: \"\"\n # ... other redis options ...\n\n# Rate Limiting Configuration\nrate_limiting:\n enabled: True\n default_limit: \"1000/minute\"\n trusted_proxies: []\n storage_uri: \"memory://\" # Use \"redis://localhost:6379\" if you need persistent/shared limits\n\n# Security Configuration\nsecurity:\n enabled: false # Master toggle for security features\n jwt_enabled: false # Enable JWT authentication (requires security.enabled=true)\n https_redirect: false # Force HTTPS (requires security.enabled=true)\n trusted_hosts: [\"*\"] # Allowed hosts (use specific domains in production)\n headers: # Security headers (applied if security.enabled=true)\n x_content_type_options: \"nosniff\"\n x_frame_options: \"DENY\"\n content_security_policy: \"default-src 'self'\"\n strict_transport_security: \"max-age=63072000; includeSubDomains\"\n\n# Crawler Configuration\ncrawler:\n memory_threshold_percent: 95.0\n rate_limiter:\n base_delay: [1.0, 2.0] # Min/max delay between requests in seconds for dispatcher\n timeouts:\n stream_init: 30.0 # Timeout for stream initialization\n batch_process: 300.0 # Timeout for non-streaming /crawl processing\n\n# Logging Configuration\nlogging:\n level: \"INFO\"\n format: \"%(asctime)s - %(name)s - %(levelname)s - %(message)s\"\n\n# Observability Configuration\nobservability:\n prometheus:\n enabled: True\n endpoint: \"/metrics\"\n health_check:\n endpoint: \"/health\"\n```\n\n*(JWT Authentication section remains the same, just note the default port is now 11235 for requests)*\n\n*(Configuration Tips and Best Practices remain the same)*\n\n### Customizing Your Configuration\n\nYou can override the default `config.yml`.\n\n#### Method 1: Modify Before Build\n\n1. Edit the `deploy/docker/config.yml` file in your local repository clone.\n2. Build the image using `docker buildx` or `docker compose --profile local-... up --build`. The modified file will be copied into the image.\n\n#### Method 2: Runtime Mount (Recommended for Custom Deploys)\n\n1. Create your custom configuration file, e.g., `my-custom-config.yml` locally. Ensure it contains all necessary sections.\n2. Mount it when running the container:\n\n * **Using `docker run`:**\n ```bash\n # Assumes my-custom-config.yml is in the current directory\n docker run -d -p 11235:11235 \\\n --name crawl4ai-custom-config \\\n --env-file .llm.env \\\n --shm-size=1g \\\n -v $(pwd)/my-custom-config.yml:/app/config.yml \\\n unclecode/crawl4ai:latest # Or your specific tag\n ```\n\n * **Using `docker-compose.yml`:** Add a `volumes` section to the service definition:\n ```yaml\n services:\n crawl4ai-hub-amd64: # Or your chosen service\n image: unclecode/crawl4ai:latest\n profiles: [\"hub-amd64\"]\n <<: *base-config\n volumes:\n # Mount local custom config over the default one in the container\n - ./my-custom-config.yml:/app/config.yml\n # Keep the shared memory volume from base-config\n - /dev/shm:/dev/shm\n ```\n *(Note: Ensure `my-custom-config.yml` is in the same directory as `docker-compose.yml`)*\n\n> 💡 When mounting, your custom file *completely replaces* the default one. Ensure it's a valid and complete configuration.\n\n### Configuration Recommendations\n\n1. **Security First** 🔒\n - Always enable security in production\n - Use specific trusted_hosts instead of wildcards\n - Set up proper rate limiting to protect your server\n - Consider your environment before enabling HTTPS redirect\n\n2. **Resource Management** 💻\n - Adjust memory_threshold_percent based on available RAM\n - Set timeouts according to your content size and network conditions\n - Use Redis for rate limiting in multi-container setups\n\n3. **Monitoring** 📊\n - Enable Prometheus if you need metrics\n - Set DEBUG logging in development, INFO in production\n - Regular health check monitoring is crucial\n\n4. **Performance Tuning** ⚡\n - Start with conservative rate limiter delays\n - Increase batch_process timeout for large content\n - Adjust stream_init timeout based on initial response times\n\n## Getting Help\n\nWe're here to help you succeed with Crawl4AI! Here's how to get support:\n\n- 📖 Check our [full documentation](https://docs.crawl4ai.com)\n- 🐛 Found a bug? [Open an issue](https://github.com/unclecode/crawl4ai/issues)\n- 💬 Join our [Discord community](https://discord.gg/crawl4ai)\n- ⭐ Star us on GitHub to show support!\n\n## Summary\n\nIn this guide, we've covered everything you need to get started with Crawl4AI's Docker deployment:\n- Building and running the Docker container\n- Configuring the environment \n- Using the interactive playground for testing\n- Making API requests with proper typing\n- Using the Python SDK\n- Leveraging specialized endpoints for screenshots, PDFs, and JavaScript execution\n- Connecting via the Model Context Protocol (MCP)\n- Monitoring your deployment\n\nThe new playground interface at `http://localhost:11235/playground` makes it much easier to test configurations and generate the corresponding JSON for API requests.\n\nFor AI application developers, the MCP integration allows tools like Claude Code to directly access Crawl4AI's capabilities without complex API handling.\n\nRemember, the examples in the `examples` folder are your friends - they show real-world usage patterns that you can adapt for your needs.\n\nKeep exploring, and don't hesitate to reach out if you need help! We're building something amazing together. 🚀\n\nHappy crawling! 🕷️\n"}, "59b5f1cee3a1be9463f77f76ec265f2ce70ebe6c": {"t": 1785770982.0975142, "n": 27915, "h": "c5900a0bc553dec4", "c": "# Crawl4AI Docker Guide 🐳\n\n## Table of Contents\n- [Prerequisites](#prerequisites)\n- [Installation](#installation)\n - [Option 1: Using Pre-built Docker Hub Images (Recommended)](#option-1-using-pre-built-docker-hub-images-recommended)\n - [Option 2: Using Docker Compose](#option-2-using-docker-compose)\n - [Option 3: Manual Local Build & Run](#option-3-manual-local-build--run)\n- [Dockerfile Parameters](#dockerfile-parameters)\n- [Using the API](#using-the-api)\n - [Playground Interface](#playground-interface)\n - [Python SDK](#python-sdk)\n - [Understanding Request Schema](#understanding-request-schema)\n - [REST API Examples](#rest-api-examples)\n- [Additional API Endpoints](#additional-api-endpoints)\n - [HTML Extraction Endpoint](#html-extraction-endpoint)\n - [Screenshot Endpoint](#screenshot-endpoint)\n - [PDF Export Endpoint](#pdf-export-endpoint)\n - [JavaScript Execution Endpoint](#javascript-execution-endpoint)\n - [Library Context Endpoint](#library-context-endpoint)\n- [MCP (Model Context Protocol) Support](#mcp-model-context-protocol-support)\n - [What is MCP?](#what-is-mcp)\n - [Connecting via MCP](#connecting-via-mcp)\n - [Using with Claude Code](#using-with-claude-code)\n - [Available MCP Tools](#available-mcp-tools)\n - [Testing MCP Connections](#testing-mcp-connections)\n - [MCP Schemas](#mcp-schemas)\n- [Metrics & Monitoring](#metrics--monitoring)\n- [Deployment Scenarios](#deployment-scenarios)\n- [Complete Examples](#complete-examples)\n- [Server Configuration](#server-configuration)\n - [Understanding config.yml](#understanding-configyml)\n - [JWT Authentication](#jwt-authentication)\n - [Configuration Tips and Best Practices](#configuration-tips-and-best-practices)\n - [Customizing Your Configuration](#customizing-your-configuration)\n - [Configuration Recommendations](#configuration-recommendations)\n- [Getting Help](#getting-help)\n- [Summary](#summary)\n\n## Prerequisites\n\nBefore we dive in, make sure you have:\n- Docker installed and running (version 20.10.0 or higher), including `docker compose` (usually bundled with Docker Desktop).\n- `git` for cloning the repository.\n- At least 4GB of RAM available for the container (more recommended for heavy use).\n- Python 3.10+ (if using the Python SDK).\n- Node.js 16+ (if using the Node.js examples).\n\n> 💡 **Pro tip**: Run `docker info` to check your Docker installation and available resources.\n\n## Installation\n\nWe offer several ways to get the Crawl4AI server running. The quickest way is to use our pre-built Docker Hub images.\n\n### Option 1: Using Pre-built Docker Hub Images (Recommended)\n\nPull and run images directly from Docker Hub without building locally.\n\n#### 1. Pull the Image\n\nOur latest release candidate is `0.6.0-r2`. Images are built with multi-arch manifests, so Docker automatically pulls the correct version for your system.\n\n```bash\n# Pull the release candidate (recommended for latest features)\ndocker pull unclecode/crawl4ai:0.6.0-r1\n\n# Or pull the latest stable version\ndocker pull unclecode/crawl4ai:latest\n```\n\n#### 2. Setup Environment (API Keys)\n\nIf you plan to use LLMs, create a `.llm.env` file in your working directory:\n\n```bash\n# Create a .llm.env file with your API keys\ncat > .llm.env << EOL\n# OpenAI\nOPENAI_API_KEY=sk-your-key\n\n# Anthropic\nANTHROPIC_API_KEY=your-anthropic-key\n\n# Other providers as needed\n# DEEPSEEK_API_KEY=your-deepseek-key\n# GROQ_API_KEY=your-groq-key\n# TOGETHER_API_KEY=your-together-key\n# MISTRAL_API_KEY=your-mistral-key\n# GEMINI_API_TOKEN=your-gemini-token\nEOL\n```\n> 🔑 **Note**: Keep your API keys secure! Never commit `.llm.env` to version control.\n\n#### 3. Run the Container\n\n* **Basic run:**\n ```bash\n docker run -d \\\n -p 11235:11235 \\\n --name crawl4ai \\\n --shm-size=1g \\\n unclecode/crawl4ai:latest\n ```\n\n* **With LLM support:**\n ```bash\n # Make sure .llm.env is in the current directory\n docker run -d \\\n -p 11235:11235 \\\n --name crawl4ai \\\n --env-file .llm.env \\\n --shm-size=1g \\\n unclecode/crawl4ai:latest\n ```\n\n> The server will be available at `http://localhost:11235`. Visit `/playground` to access the interactive testing interface.\n\n#### 4. Stopping the Container\n\n```bash\ndocker stop crawl4ai && docker rm crawl4ai\n```\n\n#### Docker Hub Versioning Explained\n\n* **Image Name:** `unclecode/crawl4ai`\n* **Tag Format:** `LIBRARY_VERSION[-SUFFIX]` (e.g., `0.6.0-r2`)\n * `LIBRARY_VERSION`: The semantic version of the core `crawl4ai` Python library\n * `SUFFIX`: Optional tag for release candidates (``) and revisions (`r1`)\n* **`latest` Tag:** Points to the most recent stable version\n* **Multi-Architecture Support:** All images support both `linux/amd64` and `linux/arm64` architectures through a single tag\n\n### Option 2: Using Docker Compose\n\nDocker Compose simplifies building and running the service, especially for local development and testing.\n\n#### 1. Clone Repository\n\n```bash\ngit clone https://github.com/unclecode/crawl4ai.git\ncd crawl4ai\n```\n\n#### 2. Environment Setup (API Keys)\n\nIf you plan to use LLMs, copy the example environment file and add your API keys. This file should be in the **project root directory**.\n\n```bash\n# Make sure you are in the 'crawl4ai' root directory\ncp deploy/docker/.llm.env.example .llm.env\n\n# Now edit .llm.env and add your API keys\n```\n\n#### 3. Build and Run with Compose\n\nThe `docker-compose.yml` file in the project root provides a simplified approach that automatically handles architecture detection using buildx.\n\n* **Run Pre-built Image from Docker Hub:**\n ```bash\n # Pulls and runs the release candidate from Docker Hub\n # Automatically selects the correct architecture\n IMAGE=unclecode/crawl4ai:latest docker compose up -d\n ```\n\n* **Build and Run Locally:**\n ```bash\n # Builds the image locally using Dockerfile and runs it\n # Automatically uses the correct architecture for your machine\n docker compose up --build -d\n ```\n\n* **Customize the Build:**\n ```bash\n # Build with all features (includes torch and transformers)\n INSTALL_TYPE=all docker compose up --build -d\n \n # Build with GPU support (for AMD64 platforms)\n ENABLE_GPU=true docker compose up --build -d\n ```\n\n> The server will be available at `http://localhost:11235`.\n\n#### 4. Stopping the Service\n\n```bash\n# Stop the service\ndocker compose down\n```\n\n### Option 3: Manual Local Build & Run\n\nIf you prefer not to use Docker Compose for direct control over the build and run process.\n\n#### 1. Clone Repository & Setup Environment\n\nFollow steps 1 and 2 from the Docker Compose section above (clone repo, `cd crawl4ai`, create `.llm.env` in the root).\n\n#### 2. Build the Image (Multi-Arch)\n\nUse `docker buildx` to build the image. Crawl4AI now uses buildx to handle multi-architecture builds automatically.\n\n```bash\n# Make sure you are in the 'crawl4ai' root directory\n# Build for the current architecture and load it into Docker\ndocker buildx build -t crawl4ai-local:latest --load .\n\n# Or build for multiple architectures (useful for publishing)\ndocker buildx build --platform linux/amd64,linux/arm64 -t crawl4ai-local:latest --load .\n\n# Build with additional options\ndocker buildx build \\\n --build-arg INSTALL_TYPE=all \\\n --build-arg ENABLE_GPU=false \\\n -t crawl4ai-local:latest --load .\n```\n\n#### 3. Run the Container\n\n* **Basic run (no LLM support):**\n ```bash\n docker run -d \\\n -p 11235:11235 \\\n --name crawl4ai-standalone \\\n --shm-size=1g \\\n crawl4ai-local:latest\n ```\n\n* **With LLM support:**\n ```bash\n # Make sure .llm.env is in the current directory (project root)\n docker run -d \\\n -p 11235:11235 \\\n --name crawl4ai-standalone \\\n --env-file .llm.env \\\n --shm-size=1g \\\n crawl4ai-local:latest\n ```\n\n> The server will be available at `http://localhost:11235`.\n\n#### 4. Stopping the Manual Container\n\n```bash\ndocker stop crawl4ai-standalone && docker rm crawl4ai-standalone\n```\n\n---\n\n## MCP (Model Context Protocol) Support\n\nCrawl4AI server includes support for the Model Context Protocol (MCP), allowing you to connect the server's capabilities directly to MCP-compatible clients like Claude Code.\n\n### What is MCP?\n\nMCP is an open protocol that standardizes how applications provide context to LLMs. It allows AI models to access external tools, data sources, and services through a standardized interface.\n\n### Connecting via MCP\n\nThe Crawl4AI server exposes two MCP endpoints:\n\n- **Server-Sent Events (SSE)**: `http://localhost:11235/mcp/sse`\n- **WebSocket**: `ws://localhost:11235/mcp/ws`\n\n### Using with Claude Code\n\nYou can add Crawl4AI as an MCP tool provider in Claude Code with a simple command:\n\n```bash\n# Add the Crawl4AI server as an MCP provider\nclaude mcp add --transport sse c4ai-sse http://localhost:11235/mcp/sse\n\n# List all MCP providers to verify it was added\nclaude mcp list\n```\n\nOnce connected, Claude Code can directly use Crawl4AI's capabilities like screenshot capture, PDF generation, and HTML processing without having to make separate API calls.\n\n### Available MCP Tools\n\nWhen connected via MCP, the following tools are available:\n\n- `md` - Generate markdown from web content\n- `html` - Extract preprocessed HTML\n- `screenshot` - Capture webpage screenshots\n- `pdf` - Generate PDF documents\n- `execute_js` - Run JavaScript on web pages\n- `crawl` - Perform multi-URL crawling\n- `ask` - Query the Crawl4AI library context\n\n### Testing MCP Connections\n\nYou can test the MCP WebSocket connection using the test file included in the repository:\n\n```bash\n# From the repository root\npython tests/mcp/test_mcp_socket.py\n```\n\n### MCP Schemas\n\nAccess the MCP tool schemas at `http://localhost:11235/mcp/schema` for detailed information on each tool's parameters and capabilities.\n\n---\n\n## Additional API Endpoints\n\nIn addition to the core `/crawl` and `/crawl/stream` endpoints, the server provides several specialized endpoints:\n\n### HTML Extraction Endpoint\n\n```\nPOST /html\n```\n\nCrawls the URL and returns preprocessed HTML optimized for schema extraction.\n\n```json\n{\n \"url\": \"https://example.com\"\n}\n```\n\n### Screenshot Endpoint\n\n```\nPOST /screenshot\n```\n\nCaptures a full-page PNG screenshot of the specified URL.\n\n```json\n{\n \"url\": \"https://example.com\",\n \"screenshot_wait_for\": 2,\n \"output_path\": \"/path/to/save/screenshot.png\"\n}\n```\n\n- `screenshot_wait_for`: Optional delay in seconds before capture (default: 2)\n- `output_path`: Optional path to save the screenshot (recommended)\n\n### PDF Export Endpoint\n\n```\nPOST /pdf\n```\n\nGenerates a PDF document of the specified URL.\n\n```json\n{\n \"url\": \"https://example.com\",\n \"output_path\": \"/path/to/save/document.pdf\"\n}\n```\n\n- `output_path`: Optional path to save the PDF (recommended)\n\n### JavaScript Execution Endpoint\n\n```\nPOST /execute_js\n```\n\nExecutes JavaScript snippets on the specified URL and returns the full crawl result.\n\n```json\n{\n \"url\": \"https://example.com\",\n \"scripts\": [\n \"return document.title\",\n \"return Array.from(document.querySelectorAll('a')).map(a => a.href)\"\n ]\n}\n```\n\n- `scripts`: List of JavaScript snippets to execute sequentially\n\n---\n\n## Dockerfile Parameters\n\nYou can customize the image build process using build arguments (`--build-arg`). These are typically used via `docker buildx build` or within the `docker-compose.yml` file.\n\n```bash\n# Example: Build with 'all' features using buildx\ndocker buildx build \\\n --platform linux/amd64,linux/arm64 \\\n --build-arg INSTALL_TYPE=all \\\n -t yourname/crawl4ai-all:latest \\\n --load \\\n . # Build from root context\n```\n\n### Build Arguments Explained\n\n| Argument | Description | Default | Options |\n| :----------- | :--------------------------------------- | :-------- | :--------------------------------- |\n| INSTALL_TYPE | Feature set | `default` | `default`, `all`, `torch`, `transformer` |\n| ENABLE_GPU | GPU support (CUDA for AMD64) | `false` | `true`, `false` |\n| APP_HOME | Install path inside container (advanced) | `/app` | any valid path |\n| USE_LOCAL | Install library from local source | `true` | `true`, `false` |\n| GITHUB_REPO | Git repo to clone if USE_LOCAL=false | *(see Dockerfile)* | any git URL |\n| GITHUB_BRANCH| Git branch to clone if USE_LOCAL=false | `main` | any branch name |\n\n*(Note: PYTHON_VERSION is fixed by the `FROM` instruction in the Dockerfile)*\n\n### Build Best Practices\n\n1. **Choose the Right Install Type**\n * `default`: Basic installation, smallest image size. Suitable for most standard web scraping and markdown generation.\n * `all`: Full features including `torch` and `transformers` for advanced extraction strategies (e.g., CosineStrategy, certain LLM filters). Significantly larger image. Ensure you need these extras.\n2. **Platform Considerations**\n * Use `buildx` for building multi-architecture images, especially for pushing to registries.\n * Use `docker compose` profiles (`local-amd64`, `local-arm64`) for easy platform-specific local builds.\n3. **Performance Optimization**\n * The image automatically includes platform-specific optimizations (OpenMP for AMD64, OpenBLAS for ARM64).\n\n---\n\n## Using the API\n\nCommunicate with the running Docker server via its REST API (defaulting to `http://localhost:11235`). You can use the Python SDK or make direct HTTP requests.\n\n### Playground Interface\n\nA built-in web playground is available at `http://localhost:11235/playground` for testing and generating API requests. The playground allows you to:\n\n1. Configure `CrawlerRunConfig` and `BrowserConfig` using the main library's Python syntax\n2. Test crawling operations directly from the interface\n3. Generate corresponding JSON for REST API requests based on your configuration\n\nThis is the easiest way to translate Python configuration to JSON requests when building integrations.\n\n### Python SDK\n\nInstall the SDK: `pip install crawl4ai`\n\n```python\nimport asyncio\nfrom crawl4ai.docker_client import Crawl4aiDockerClient\nfrom crawl4ai import BrowserConfig, CrawlerRunConfig, CacheMode # Assuming you have crawl4ai installed\n\nasync def main():\n # Point to the correct server port\n async with Crawl4aiDockerClient(base_url=\"http://localhost:11235\", verbose=True) as client:\n # If JWT is enabled on the server, authenticate first:\n # await client.authenticate(\"user@example.com\") # See Server Configuration section\n\n # Example Non-streaming crawl\n print(\"--- Running Non-Streaming Crawl ---\")\n results = await client.crawl(\n [\"https://httpbin.org/html\"],\n browser_config=BrowserConfig(headless=True), # Use library classes for config aid\n crawler_config=CrawlerRunConfig(cache_mode=CacheMode.BYPASS)\n )\n if results: # client.crawl returns None on failure\n print(f\"Non-streaming results success: {results.success}\")\n if results.success:\n for result in results: # Iterate through the CrawlResultContainer\n print(f\"URL: {result.url}, Success: {result.success}\")\n else:\n print(\"Non-streaming crawl failed.\")\n\n\n # Example Streaming crawl\n print(\"\\n--- Running Streaming Crawl ---\")\n stream_config = CrawlerRunConfig(stream=True, cache_mode=CacheMode.BYPASS)\n try:\n async for result in await client.crawl( # client.crawl returns an async generator for streaming\n [\"https://httpbin.org/html\", \"https://httpbin.org/links/5/0\"],\n browser_config=BrowserConfig(headless=True),\n crawler_config=stream_config\n ):\n print(f\"Streamed result: URL: {result.url}, Success: {result.success}\")\n except Exception as e:\n print(f\"Streaming crawl failed: {e}\")\n\n\n # Example Get schema\n print(\"\\n--- Getting Schema ---\")\n schema = await client.get_schema()\n print(f\"Schema received: {bool(schema)}\") # Print whether schema was received\n\nif __name__ == \"__main__\":\n asyncio.run(main())\n```\n\n*(SDK parameters like timeout, verify_ssl etc. remain the same)*\n\n### Second Approach: Direct API Calls\n\nCrucially, when sending configurations directly via JSON, they **must** follow the `{\"type\": \"ClassName\", \"params\": {...}}` structure for any non-primitive value (like config objects or strategies). Dictionaries must be wrapped as `{\"type\": \"dict\", \"value\": {...}}`.\n\n*(Keep the detailed explanation of Configuration Structure, Basic Pattern, Simple vs Complex, Strategy Pattern, Complex Nested Example, Quick Grammar Overview, Important Rules, Pro Tip)*\n\n#### More Examples *(Ensure Schema example uses type/value wrapper)*\n\n**Advanced Crawler Configuration**\n*(Keep example, ensure cache_mode uses valid enum value like \"bypass\")*\n\n**Extraction Strategy**\n```json\n{\n \"crawler_config\": {\n \"type\": \"CrawlerRunConfig\",\n \"params\": {\n \"extraction_strategy\": {\n \"type\": \"JsonCssExtractionStrategy\",\n \"params\": {\n \"schema\": {\n \"type\": \"dict\",\n \"value\": {\n \"baseSelector\": \"article.post\",\n \"fields\": [\n {\"name\": \"title\", \"selector\": \"h1\", \"type\": \"text\"},\n {\"name\": \"content\", \"selector\": \".content\", \"type\": \"html\"}\n ]\n }\n }\n }\n }\n }\n }\n}\n```\n\n**LLM Extraction Strategy** *(Keep example, ensure schema uses type/value wrapper)*\n*(Keep Deep Crawler Example)*\n\n### REST API Examples\n\nUpdate URLs to use port `11235`.\n\n#### Simple Crawl\n\n```python\nimport requests\n\n# Configuration objects converted to the required JSON structure\nbrowser_config_payload = {\n \"type\": \"BrowserConfig\",\n \"params\": {\"headless\": True}\n}\ncrawler_config_payload = {\n \"type\": \"CrawlerRunConfig\",\n \"params\": {\"stream\": False, \"cache_mode\": \"bypass\"} # Use string value of enum\n}\n\ncrawl_payload = {\n \"urls\": [\"https://httpbin.org/html\"],\n \"browser_config\": browser_config_payload,\n \"crawler_config\": crawler_config_payload\n}\nresponse = requests.post(\n \"http://localhost:11235/crawl\", # Updated port\n # headers={\"Authorization\": f\"Bearer {token}\"}, # If JWT is enabled\n json=crawl_payload\n)\nprint(f\"Status Code: {response.status_code}\")\nif response.ok:\n print(response.json())\nelse:\n print(f\"Error: {response.text}\")\n\n```\n\n#### Streaming Results\n\n```python\nimport json\nimport httpx # Use httpx for async streaming example\n\nasync def test_stream_crawl(token: str = None): # Made token optional\n \"\"\"Test the /crawl/stream endpoint with multiple URLs.\"\"\"\n url = \"http://localhost:11235/crawl/stream\" # Updated port\n payload = {\n \"urls\": [\n \"https://httpbin.org/html\",\n \"https://httpbin.org/links/5/0\",\n ],\n \"browser_config\": {\n \"type\": \"BrowserConfig\",\n \"params\": {\"headless\": True, \"viewport\": {\"type\": \"dict\", \"value\": {\"width\": 1200, \"height\": 800}}} # Viewport needs type:dict\n },\n \"crawler_config\": {\n \"type\": \"CrawlerRunConfig\",\n \"params\": {\"stream\": True, \"cache_mode\": \"bypass\"}\n }\n }\n\n headers = {}\n # if token:\n # headers = {\"Authorization\": f\"Bearer {token}\"} # If JWT is enabled\n\n try:\n async with httpx.AsyncClient() as client:\n async with client.stream(\"POST\", url, json=payload, headers=headers, timeout=120.0) as response:\n print(f\"Status: {response.status_code} (Expected: 200)\")\n response.raise_for_status() # Raise exception for bad status codes\n\n # Read streaming response line-by-line (NDJSON)\n async for line in response.aiter_lines():\n if line:\n try:\n data = json.loads(line)\n # Check for completion marker\n if data.get(\"status\") == \"completed\":\n print(\"Stream completed.\")\n break\n print(f\"Streamed Result: {json.dumps(data, indent=2)}\")\n except json.JSONDecodeError:\n print(f\"Warning: Could not decode JSON line: {line}\")\n\n except httpx.HTTPStatusError as e:\n print(f\"HTTP error occurred: {e.response.status_code} - {e.response.text}\")\n except Exception as e:\n print(f\"Error in streaming crawl test: {str(e)}\")\n\n# To run this example:\n# import asyncio\n# asyncio.run(test_stream_crawl())\n```\n\n---\n\n## Metrics & Monitoring\n\nKeep an eye on your crawler with these endpoints:\n\n- `/health` - Quick health check\n- `/metrics` - Detailed Prometheus metrics\n- `/schema` - Full API schema\n\nExample health check:\n```bash\ncurl http://localhost:11235/health\n```\n\n---\n\n*(Deployment Scenarios and Complete Examples sections remain the same, maybe update links if examples moved)*\n\n---\n\n## Server Configuration\n\nThe server's behavior can be customized through the `config.yml` file.\n\n### Understanding config.yml\n\nThe configuration file is loaded from `/app/config.yml` inside the container. By default, the file from `deploy/docker/config.yml` in the repository is copied there during the build.\n\nHere's a detailed breakdown of the configuration options (using defaults from `deploy/docker/config.yml`):\n\n```yaml\n# Application Configuration\napp:\n title: \"Crawl4AI API\"\n version: \"1.0.0\" # Consider setting this to match library version, e.g., \"0.5.1\"\n host: \"0.0.0.0\"\n port: 8020 # NOTE: This port is used ONLY when running server.py directly. Gunicorn overrides this (see supervisord.conf).\n reload: False # Default set to False - suitable for production\n timeout_keep_alive: 300\n\n# Default LLM Configuration\nllm:\n provider: \"openai/gpt-4o-mini\"\n api_key_env: \"OPENAI_API_KEY\"\n # api_key: sk-... # If you pass the API key directly then api_key_env will be ignored\n\n# Redis Configuration (Used by internal Redis server managed by supervisord)\nredis:\n host: \"localhost\"\n port: 6379\n db: 0\n password: \"\"\n # ... other redis options ...\n\n# Rate Limiting Configuration\nrate_limiting:\n enabled: True\n default_limit: \"1000/minute\"\n trusted_proxies: []\n storage_uri: \"memory://\" # Use \"redis://localhost:6379\" if you need persistent/shared limits\n\n# Security Configuration\nsecurity:\n enabled: false # Master toggle for security features\n jwt_enabled: false # Enable JWT authentication (requires security.enabled=true)\n https_redirect: false # Force HTTPS (requires security.enabled=true)\n trusted_hosts: [\"*\"] # Allowed hosts (use specific domains in production)\n headers: # Security headers (applied if security.enabled=true)\n x_content_type_options: \"nosniff\"\n x_frame_options: \"DENY\"\n content_security_policy: \"default-src 'self'\"\n strict_transport_security: \"max-age=63072000; includeSubDomains\"\n\n# Crawler Configuration\ncrawler:\n memory_threshold_percent: 95.0\n rate_limiter:\n base_delay: [1.0, 2.0] # Min/max delay between requests in seconds for dispatcher\n timeouts:\n stream_init: 30.0 # Timeout for stream initialization\n batch_process: 300.0 # Timeout for non-streaming /crawl processing\n\n# Logging Configuration\nlogging:\n level: \"INFO\"\n format: \"%(asctime)s - %(name)s - %(levelname)s - %(message)s\"\n\n# Observability Configuration\nobservability:\n prometheus:\n enabled: True\n endpoint: \"/metrics\"\n health_check:\n endpoint: \"/health\"\n```\n\n*(JWT Authentication section remains the same, just note the default port is now 11235 for requests)*\n\n*(Configuration Tips and Best Practices remain the same)*\n\n### Customizing Your Configuration\n\nYou can override the default `config.yml`.\n\n#### Method 1: Modify Before Build\n\n1. Edit the `deploy/docker/config.yml` file in your local repository clone.\n2. Build the image using `docker buildx` or `docker compose --profile local-... up --build`. The modified file will be copied into the image.\n\n#### Method 2: Runtime Mount (Recommended for Custom Deploys)\n\n1. Create your custom configuration file, e.g., `my-custom-config.yml` locally. Ensure it contains all necessary sections.\n2. Mount it when running the container:\n\n * **Using `docker run`:**\n ```bash\n # Assumes my-custom-config.yml is in the current directory\n docker run -d -p 11235:11235 \\\n --name crawl4ai-custom-config \\\n --env-file .llm.env \\\n --shm-size=1g \\\n -v $(pwd)/my-custom-config.yml:/app/config.yml \\\n unclecode/crawl4ai:latest # Or your specific tag\n ```\n\n * **Using `docker-compose.yml`:** Add a `volumes` section to the service definition:\n ```yaml\n services:\n crawl4ai-hub-amd64: # Or your chosen service\n image: unclecode/crawl4ai:latest\n profiles: [\"hub-amd64\"]\n <<: *base-config\n volumes:\n # Mount local custom config over the default one in the container\n - ./my-custom-config.yml:/app/config.yml\n # Keep the shared memory volume from base-config\n - /dev/shm:/dev/shm\n ```\n *(Note: Ensure `my-custom-config.yml` is in the same directory as `docker-compose.yml`)*\n\n> 💡 When mounting, your custom file *completely replaces* the default one. Ensure it's a valid and complete configuration.\n\n### Configuration Recommendations\n\n1. **Security First** 🔒\n - Always enable security in production\n - Use specific trusted_hosts instead of wildcards\n - Set up proper rate limiting to protect your server\n - Consider your environment before enabling HTTPS redirect\n\n2. **Resource Management** 💻\n - Adjust memory_threshold_percent based on available RAM\n - Set timeouts according to your content size and network conditions\n - Use Redis for rate limiting in multi-container setups\n\n3. **Monitoring** 📊\n - Enable Prometheus if you need metrics\n - Set DEBUG logging in development, INFO in production\n - Regular health check monitoring is crucial\n\n4. **Performance Tuning** ⚡\n - Start with conservative rate limiter delays\n - Increase batch_process timeout for large content\n - Adjust stream_init timeout based on initial response times\n\n## Getting Help\n\nWe're here to help you succeed with Crawl4AI! Here's how to get support:\n\n- 📖 Check our [full documentation](https://docs.crawl4ai.com)\n- 🐛 Found a bug? [Open an issue](https://github.com/unclecode/crawl4ai/issues)\n- 💬 Join our [Discord community](https://discord.gg/crawl4ai)\n- ⭐ Star us on GitHub to show support!\n\n## Summary\n\nIn this guide, we've covered everything you need to get started with Crawl4AI's Docker deployment:\n- Building and running the Docker container\n- Configuring the environment \n- Using the interactive playground for testing\n- Making API requests with proper typing\n- Using the Python SDK\n- Leveraging specialized endpoints for screenshots, PDFs, and JavaScript execution\n- Connecting via the Model Context Protocol (MCP)\n- Monitoring your deployment\n\nThe new playground interface at `http://localhost:11235/playground` makes it much easier to test configurations and generate the corresponding JSON for API requests.\n\nFor AI application developers, the MCP integration allows tools like Claude Code to directly access Crawl4AI's capabilities without complex API handling.\n\nRemember, the examples in the `examples` folder are your friends - they show real-world usage patterns that you can adapt for your needs.\n\nKeep exploring, and don't hesitate to reach out if you need help! We're building something amazing together. 🚀\n\nHappy crawling! 🕷️\n"}}}} |