1 line
323 KiB
JSON
1 line
323 KiB
JSON
{"repo": "anyvm-tech/anygpt", "files": {"apps/api/README.md": {"efe9c87458f746997b3f0b0ad5638859b60e0d72": {"t": 1785747221.0573401, "n": 25203, "h": "e5a3a246e090f605", "c": "# AnyGPT API Server\n\nThis directory contains the backend API server for AnyGPT, built with HyperExpress on Node.js and TypeScript.\n\n## Overview\n\nThe API server acts as a central gateway to various AI model providers. It manages API keys, handles request routing, provides rate limiting, logs errors, and dynamically updates model information. The server supports both Redis and filesystem-based data storage with automatic failover.\n\n## Project Structure\n\n```\napps/api/\n├── server.ts # Main server entry point, initializes and runs the API\n├── package.json # Project dependencies and scripts\n├── tsconfig.json # TypeScript configuration\n├── .env # Environment variables (created from .env.example)\n├── providers.json # Stores provider configurations and statistics\n├── models.json # Stores available model details, updated dynamically\n├── keys.json # Stores user API keys and associated data\n├── tiers.json # Defines API usage tiers and their limits\n├── providers.schema.json # JSON schema for providers.json\n├── models.schema.json # JSON schema for models.json\n├── openapi.json # OpenAPI spec served at /openapi.json\n├── excluded-errors.json # Error patterns excluded from provider auto-disable\n|\n├── routes/ # Contains route handlers for different API endpoints\n│ ├── admin.ts # Admin-specific routes (e.g., adding providers, generating keys)\n│ ├── models.ts # Routes for listing models and refreshing provider counts\n│ ├── openai.ts # OpenAI compatible API endpoints\n│ ├── anthropic.ts # Anthropic compatible API endpoints\n│ ├── gemini.ts # Gemini compatible API endpoints\n│ ├── groq.ts # Groq compatible API endpoints\n│ ├── openrouter.ts # OpenRouter compatible API endpoints\n│ ├── ollama.ts # Ollama compatible API endpoints\n│ ├── openapi.ts # OpenAPI spec route\n│ └── ... # Other provider-specific routes\n|\n├── providers/ # Logic for interacting with specific AI provider APIs\n│ ├── handler.ts # Core message handling, provider selection, and stats updates\n│ ├── interfaces.ts # TypeScript interfaces for providers and models\n│ ├── openai.ts # OpenAI provider client\n│ ├── gemini.ts # Gemini provider client\n│ └── ... # Other provider client implementations\n|\n├── modules/ # Reusable modules for various functionalities\n│ ├── dataManager.ts # Manages dual-source data (Redis/filesystem) with automatic failover\n│ ├── modelUpdater.ts # Handles automatic updates to models.json based on provider data\n│ ├── errorLogger.ts # Centralized error logging to file and Redis\n│ ├── errorExclusion.ts # Excludes errors from provider auto-disable\n│ ├── userData.ts # Manages user API key generation, validation, and usage tracking\n│ ├── compute.ts # Computes provider statistics, scores, and applies EMA\n│ ├── db.ts # Redis database connection and operations\n│ └── typeguards.ts # TypeScript type guards\n|\n├── logs/ # Directory for log files (filesystem fallback)\n│ └── api-error.jsonl # Detailed error logs in JSON Lines format\n|\n├── dev/ # Development and testing utilities\n│ ├── testApi.ts # Main API testing script\n│ ├── testSetup.ts # Test environment setup and cleanup\n│ ├── mockProvider.ts # Configurable mock AI provider server\n│ ├── testMockProvider.ts # Mock provider testing script\n│ ├── MOCK_SERVER_CONFIG.md # Mock server configuration documentation\n│ ├── models.ts # Model management utilities\n│ ├── updatemodels.ts # Model update scripts\n│ ├── updateproviders.ts # Provider update scripts\n│ └── ... # Other development utilities\n|\n├── server/ # CLI scripts (legacy, being migrated to API routes)\n│ ├── addProvider.ts # Script to add/update providers\n│ └── generateApiKey.ts # Script to generate API keys\n└── ws/ # WebSocket and realtime handlers\n```\n\n## Features\n\n* **Multi-Provider Support**: Integrates with OpenAI, Anthropic, Gemini, Groq, OpenRouter, Ollama, DeepSeek, xAI, Imagen, and more via provider configs.\n* **OpenAI-Compatible API**: `/v1` supports chat completions, responses, embeddings, images, videos, audio (speech/transcriptions), and key introspection.\n* **OpenAI-Native Responses Support**: `/v1/responses` is tuned for native OpenAI clients such as Roo Code, with reasoning/tool-call normalization when needed and passthrough fast-paths for simpler text streams.\n* **Provider-Compatible Routers**: Native-style endpoints for Anthropic (`/v3`), Gemini (`/v2`), Groq (`/v4`), Ollama (`/v5`), and OpenRouter (`/v6`).\n* **Dual Data Storage**: Redis primary with filesystem fallback and a short in-memory cache.\n* **Dynamic Model Management**: Updates `models.json` with provider counts and model capabilities (OpenWebUI compatible).\n* **Tier-Based Rate Limiting**: RPS/RPM/RPD limits and provider-score gating per tier (`tiers.json`).\n* **Provider Routing & Health**: Provider scoring, retries/fallback, auto-disable on repeated failures, and error-exclusion patterns.\n* **Admission Queue Isolation**: OpenAI-compatible traffic uses separate queue lanes for `/v1/responses` and provider-family-specific native routing to reduce cross-provider contention.\n* **Image URL Inlining Proxy**: Optional HTTP image fetch with SSRF protections, size/time limits, and per-request referer override.\n* **WebSocket & Realtime**: `/ws` for chat and `/v1/realtime` for realtime-compatible sessions.\n* **OpenAPI Spec**: Served at `/openapi.json` and `/api/openapi.json`.\n\n## Prerequisites\n\n* Node.js 20.x or 22.x\n* Bun 1.3.10 or newer\n\n## Setup\n\n1. **Clone the repository.**\n2. **Navigate to the `apps/api` directory:**\n ```bash\n cd apps/api\n ```\n3. **Install dependencies:**\n ```bash\n bun install\n ```\n4. **Create a `.env` file** with your configuration. Key environment variables include:\n\n ### Core Server Configuration\n * `PORT`: Port for the API server (default: 3000).\n \n ### Data Storage Configuration\n * `DATA_SOURCE_PREFERENCE`: Set to `redis` or `filesystem` (default: `redis`).\n * `REDIS_URL`: Redis-compatible host/port or URL. `host:port` is the simplest local format.\n * `REDIS_USERNAME`: Redis username (default: `default`).\n * `REDIS_PASSWORD`: Redis password.\n * `REDIS_DB`: Redis database number (default: 0).\n * `REDIS_TLS`: Set to `true` for SSL/TLS connections (default: `false`).\n * `ERROR_LOG_TO_REDIS`: Enable error logging to Redis (default: `true`).\n\n Self-hosted Dragonfly works with the same settings. A local preset is included at [`dragonfly-anygpt.flags.example`](/home/skullcmd/AnyGPT/apps/api/dragonfly-anygpt.flags.example) and matches:\n\n ```env\n DATA_SOURCE_PREFERENCE=redis\n REDIS_URL=127.0.0.1:6380\n REDIS_USERNAME=default\n REDIS_PASSWORD=replace-with-apps-api-redis-password\n REDIS_DB=0\n REDIS_TLS=false\n ```\n\n The included Dragonfly preset exposes two logical databases so the main API can use DB `0` while the control plane can clone into DB `1`.\n\n ### Router Configuration\n * `ENABLE_OPENAI_ROUTES`: Enable/disable OpenAI routes (default: `true`).\n * `ENABLE_ANTHROPIC_ROUTES`: Enable/disable Anthropic routes (default: `true`).\n * `ENABLE_GEMINI_ROUTES`: Enable/disable Gemini routes (default: `true`).\n * `ENABLE_GROQ_ROUTES`: Enable/disable Groq routes (default: `true`).\n * `ENABLE_OPENROUTER_ROUTES`: Enable/disable OpenRouter routes (default: `true`).\n * `ENABLE_OLLAMA_ROUTES`: Enable/disable Ollama routes (default: `true`).\n * `ENABLE_ADMIN_ROUTES`: Enable/disable admin routes (default: `true`).\n * `ENABLE_MODELS_ROUTES`: Enable/disable models routes (default: `true`).\n\n ### Default Admin Configuration\n * `DEFAULT_ADMIN_USER_ID`: Default admin user ID for auto-creation.\n * `DEFAULT_ADMIN_API_KEY`: Default admin API key.\n\n ### Mock Server Configuration (for testing)\n * `MOCK_BASE_DELAY`: Base response delay in milliseconds (default: 200).\n * `MOCK_DELAY_VARIANCE`: Random delay variance (default: 100).\n * `MOCK_ERROR_RATE`: Error simulation rate 0-1 (default: 0.15).\n * `MOCK_TIMEOUT_RATE`: Timeout simulation rate 0-1 (default: 0.05).\n * `MOCK_TOKEN_SPEED`: Simulated tokens per second (default: 25).\n * `MOCK_ENABLE_LOGS`: Enable mock server logging (default: `true`).\n\n ### Logging & Security\n * `LOG_LEVEL`: `debug`, `info`, `warn`, `error`, or `silent` (default: `debug` in dev, `info` in production).\n * `LOG_SENSITIVE_PAYLOADS`: Log full URLs/body snippets for debugging (default: `false`).\n * `API_KEY_HASH_SECRET`: Secret used to hash API keys in logs (default: `anygpt-api`).\n * `RATE_LIMIT_HASH_SECRET`: Secret used to hash API keys for Redis rate-limiting keys (default: `anygpt-rate-limit`).\n * `ADMIN_KEYS_LOG_MAX_ENTRIES`: Maximum number of admin key log entries returned (default: 1000).\n * `ADMIN_KEYS_LOG_MAX_BYTES`: Rotate admin key log file when it exceeds this size in bytes (default: 5242880).\n * `ADMIN_KEYS_RATE_LIMIT_RPS`: Rate limit for open admin key ingest endpoint (default: 5).\n * `ADMIN_KEYS_RATE_LIMIT_RPM`: Rate limit for open admin key ingest endpoint (default: 60).\n * `ADMIN_KEYS_RATE_LIMIT_RPD`: Rate limit for open admin key ingest endpoint (default: 1000).\n * `ADMIN_KEYS_MAX_BODY_BYTES`: Max body size for open admin key ingest endpoint (default: 4096).\n * `ADMIN_KEYS_ALLOWED_PROVIDERS`: Optional comma-separated allowlist for provider IDs (default: unset, allows any provider).\n * `ADMIN_METRICS_MAX_ERROR_LINES`: Max error log lines scanned for metrics (default: 200000).\n * `MAX_BODY_LENGTH`: Maximum request body size in bytes (default: 50MB).\n * `MAX_MAX_TOKENS`: Upper bound for `max_tokens` in OpenAI-compatible requests (default: disabled).\n * `API_KEY_MIN_LENGTH`: Minimum API key length enforced at auth (default: 32).\n * `API_KEY_MAX_LENGTH`: Maximum API key length enforced at auth (default: 512).\n * `MAX_MODEL_ID_LENGTH`: Maximum model id length for incoming payloads (default: 200).\n * `MAX_ROLE_LENGTH`: Maximum role string length (default: 32).\n * `MAX_MESSAGE_COUNT`: Max number of messages in a chat payload (default: 200).\n * `MAX_MESSAGE_PARTS`: Max number of content parts in a message (default: 2000).\n * `MAX_MESSAGE_CONTENT_CHARS`: Max length of any text content (default: 1,000,000).\n * `MAX_IMAGE_URL_LENGTH`: Max length of image_url strings (default: 4096).\n * `MAX_IMAGE_BASE64_CHARS`: Max size of base64 image data (default: 8,000,000).\n * `MAX_AUDIO_BASE64_CHARS`: Max size of base64 audio data (default: 12,000,000).\n * `ERROR_LOG_MAX_DETAIL_CHARS`: Cap length of individual log detail strings (default: 4000).\n * `ERROR_LOG_MAX_DETAIL_KEYS`: Cap number of detail keys per error record (default: 60).\n\n ### Provider Health & Capabilities\n * `DISABLE_PROVIDER_AUTO_DISABLE`: Set to `true` to disable auto-disable behavior (default: auto-disable enabled).\n * `DISABLE_PROVIDER_AFTER_MODELS`: Disable a provider after this many models are disabled (default: 2).\n * `MODEL_CAPS_REFRESH_MS`: Model capability refresh interval in ms (default: 5000).\n\n ### Upstream Timeouts & Caches\n * `UPSTREAM_TIMEOUT_MS`: Upstream HTTP timeout in ms (default: 120000).\n * `DATA_CACHE_TTL_MS`: In-memory metadata cache TTL in ms (default: `5000`, test default: `0`).\n * `VIDEO_REQUEST_CACHE_TTL_MS`: TTL for cached video request IDs in ms (default: 3600000).\n\n ### Admission Queues\n * `REQUEST_QUEUE_CONCURRENCY`: Concurrency for the shared request-handler queue used by `/v1/chat/completions` and `/v1/responses`.\n * `REQUEST_QUEUE_MAX_PENDING`: Maximum pending requests allowed in the shared request-handler queue before overload responses are returned.\n * `REQUEST_QUEUE_MAX_WAIT_MS`: Maximum time a request may wait in the shared request-handler queue before timing out.\n * `REQUEST_MEMORY_MIN_AVAILABLE_SYSTEM_MB`: Minimum `MemAvailable` headroom before the request admission guard starts returning memory-pressure `503`s. On large-memory hosts the fallback tops out at `3072`.\n * `EMBEDDINGS_QUEUE_CONCURRENCY`: Concurrency for the dedicated embeddings admission queue.\n * `EMBEDDINGS_QUEUE_MAX_PENDING`: Maximum pending requests for the embeddings admission queue.\n * `PROVIDER_STATS_QUEUE_CONCURRENCY`: Concurrency per provider-stats worker queue.\n * `PROVIDER_STATS_QUEUE_WORKERS`: Number of provider-stats worker queues.\n * `PROVIDER_STATS_QUEUE_MAX_PENDING`: Maximum pending jobs per provider-stats worker queue.\n * `PROVIDER_STATS_FLUSH_MS`: Flush interval for buffered provider-stats updates.\n * `PROVIDER_STATS_BATCH_SIZE`: Batch size for provider-stats queue processing.\n * `PROVIDER_STATS_BUFFER_MAX_PENDING`: Maximum buffered provider-stats updates before backpressure/drop behavior applies.\n\n ### Image URL Fetch Proxy\n * `IMAGE_FETCH_TIMEOUT_MS`: Image fetch timeout in ms (default: 15000).\n * `IMAGE_FETCH_MAX_BYTES`: Max image bytes to inline (default: 8MB).\n * `IMAGE_FETCH_MAX_REDIRECTS`: Max redirects to follow (default: 3).\n * `IMAGE_FETCH_ALLOW_PRIVATE`: Allow private/loopback hosts (default: `false`).\n * `IMAGE_FETCH_FORWARD_AUTH`: Forward Authorization header to the original host (default: `false`).\n * `IMAGE_FETCH_ALLOWED_PROTOCOLS`: Allowlisted protocols (default: `http,https`).\n * `IMAGE_FETCH_ALLOWED_HOSTS`: Optional allowlist of hosts or suffixes (comma-separated).\n * `IMAGE_FETCH_USER_AGENT`: User-Agent sent when fetching images.\n * `IMAGE_FETCH_REFERER`: Default Referer for image fetches (can be overridden per request).\n\n5. **Initial Data Files**: The server will attempt to create `providers.json`, `models.json`, and `keys.json` if they don't exist. The data will be stored in Redis if configured, with filesystem fallback.\n\n## Running the Server\n\n* **Development Mode** (with hot-reloading via `tsx`):\n ```bash\n bun run dev\n ```\n This rebuilds the experimental output on change and restarts the Node runtime automatically.\n\n* **Production Build & Start:**\n ```bash\n # Build the TypeScript code\n bun run build\n \n # Start the compiled server\n bun run start\n ```\n The production build outputs JavaScript files to `dist/production/` and the start script runs `./dist/production/server.js`.\n\n* **Experimental Build & Start:**\n ```bash\n # Build the experimental bundle\n bun run build:experimental\n\n # Start the experimental bundle\n bun run start:experimental\n ```\n The experimental build outputs JavaScript files to `dist/experimental/`.\n\n* **Systemd Services:**\n ```bash\n sudo systemctl status anygpt\n sudo systemctl status anygpt-experimental\n ```\n The main production unit is `anygpt.service`. Older deployments may still expose `anygpt-api.service` as a compatibility alias. The experimental service uses a separate build output and default port (3000).\n\n## Testing\n\nThe project includes a comprehensive testing suite with both unit tests and integration tests using a configurable mock provider.\n\n### Running Tests\n\n* **Full Test Suite** (recommended):\n ```bash\n bun run test\n ```\n This runs the mock provider, API server, REST/integration checks, and the OpenAI Node SDK compatibility checks concurrently, then cleans up automatically.\n\n* **Individual Test Components**:\n ```bash\n # Run only the mock provider\n bun run test:mock\n \n # Run only the API server in test mode\n bun run test:dev\n \n # Run only the test scripts (requires servers to be running)\n bun run test:run\n\n # Run only the OpenAI Node SDK compatibility checks\n bun run test:sdk\n ```\n\n### Mock Provider Testing\n\nThe mock provider supports runtime configuration for realistic testing scenarios:\n\n```bash\n# Test the mock provider configuration\nbun x tsx ./dev/testMockProvider.ts\n\n# Run the mock provider standalone\nbun run test:mock\n```\n\nSee `dev/MOCK_SERVER_CONFIG.md` for detailed documentation on configuring response times, error rates, and other mock behaviors.\n\n## API Endpoints\n\nAuthentication is typically via `Authorization: Bearer <YOUR_ANYGPT_API_KEY>` or `api-key`/`x-api-key` headers (provider-compatible routes map provider headers to your AnyGPT key).\n\n### OpenAI-Compatible (`/v1`)\nCommon endpoints:\n* `POST /v1/chat/completions`\n* `POST /v1/responses`\n* `POST /v1/embeddings`\n* `POST /v1/images/generations`\n* `POST /v1/images/edits`\n* `GET /v1/videos`\n* `POST /v1/videos`\n* `POST /v1/videos/generations` (legacy alias)\n* `GET /v1/videos/{requestId}`\n* `POST /v1/audio/speech`\n* `POST /v1/audio/transcriptions`\n* `GET /v1/audio/voices`\n* `GET /v1/audio/models`\n* `GET /v1/keys/me` (key usage/metrics)\n* `GET /v1/keys/me/live` (live key usage/metrics from Redis when available)\n* `POST /v1/generate_key` (admin)\n* `POST /v1/interactions` and `GET /v1/interactions/{interactionId}`\n\nNotes:\n* [`/v1/responses`](apps/api/routes/openai.ts) is the preferred path for native OpenAI clients such as Roo Code.\n* Video creation uses `POST /v1/videos`; `GET /v1/videos` lists existing jobs.\n* For Veo requests, `aspect_ratio` (or `aspectRatio`) accepts `16:9` or `9:16`, and duration can be sent as `duration`, `seconds`, `duration_seconds`, or `durationSeconds`.\n* On the current Gemini-backed Veo 3 / Veo 3.1 route, audio is native to the model. `generate_audio` / `generateAudio` and `model_variant` / `variant` are accepted as compatibility inputs, but they do not mute output. To influence sound, describe it in the prompt.\n* Plain text Responses streams can use a passthrough fast path for lower latency; reasoning/tool-call cases may be normalized server-side to preserve OpenAI-native semantics.\n* Text-capable Gemini, DeepSeek, xAI/Grok, and similar models can be served through `/v1/responses` when they behave like chat/text models.\n* Media-only or dedicated non-chat models such as embeddings, TTS, STT, image generation, and video generation are redirected to their dedicated endpoints instead of `/v1/responses`.\n* Gemini tool-calling support is normalized in the provider adapter and capability probes, so models like Gemini 3.1 preview variants can now surface `tool_calling` in `models.json` when supported.\n\n### Provider-Compatible Routers\n* Gemini: `/v2`\n* Anthropic: `/v3`\n* Groq: `/v4`\n* Ollama: `/v5`\n* OpenRouter: `/v6`\n\n### Models and Metadata\n* `GET /v1/models` (aliases: `/v1/chat/completions/models`, `/v1/chat/completion/models`)\n* `POST /admin/models/refresh-provider-counts` or `POST /api/admin/models/refresh-provider-counts` (admin)\n\n### Admin\n* Admin endpoints live under `/api/admin` (providers, API keys, metrics) and are also exposed at `/admin`.\n* API key summaries now include lifetime totals, paid totals, and current billing-period totals for tokens, requests, and estimated cost.\n* `POST /api/admin/api-keys/settle-billing` marks the current billing period as paid for a selected key and starts a new billing period from the settlement timestamp.\n* `POST /api/admin/api-keys/reset-usage` clears lifetime and billing counters for a selected key.\n* `GET /api/admin/metrics/summary` returns aggregate provider/model/key counts, queue totals, and error log stats.\n* `GET /api/admin/metrics/queues` returns live queue snapshots for:\n * the shared request-handler queue used by `/v1/chat/completions` and `/v1/responses`\n * the dedicated embeddings admission queue\n * each provider-stats worker queue\n* Queue snapshots include queue label, concurrency, max pending, max wait, in-flight count, pending count, overload count, and last overload timestamp when available.\n* See `openapi.json` for the full list.\n\n### OpenAPI Spec\n* `GET /openapi.json`\n* `GET /api/openapi.json`\n\n## Image URL Fetching\n\nThe OpenAI- and OpenRouter-compatible paths can inline HTTP image URLs into base64 before sending upstream. This avoids provider-side URL fetch failures and applies SSRF protections.\n\n* Use `x-image-fetch-referer` (or `x-image-referer`) to set a per-request Referer for image fetches.\n* Configure allowlists, size limits, and timeouts with the `IMAGE_FETCH_*` environment variables.\n\n## WebSocket Usage\n\nThe server exposes a unified WebSocket endpoint at `ws://<host>:<port>/ws` for chat-style completions. An OpenAI Realtime-compatible endpoint is also exposed at `ws://<host>:<port>/v1/realtime`.\n\n### Message Flow\n\n1. Connect to `/ws`.\n2. Send auth message:\n```json\n{ \"type\": \"auth\", \"apiKey\": \"YOUR_ANYGPT_API_KEY\" }\n```\n3. On `{ \"type\": \"auth.ok\" }`, send a chat request:\n```json\n{\n \"type\": \"chat.completions\",\n \"requestId\": \"req-123\",\n \"model\": \"gpt-4o\",\n \"messages\": [ { \"role\": \"user\", \"content\": \"Hello from WS\" } ]\n}\n```\n4. Receive sequence:\n```json\n{ \"type\": \"chat.start\", \"requestId\": \"req-123\" }\n{ \"type\": \"chat.complete\", \"requestId\": \"req-123\", \"response\": \"...assistant reply...\", \"usage\": { \"total_tokens\": 42 }, \"latencyMs\": 512, \"providerId\": \"openai-main\" }\n```\n\n### Supported Message Types (Inbound)\n- `auth` – Authenticate with API key.\n- `chat.completions` – Single-shot chat completion (non-streaming for now).\n- `ping` – Liveness check (responds with `pong`).\n\n### Outbound Types\n- `auth.ok`\n- `error` (fields: `code`, `message`, optional `requestId`)\n- `chat.start`\n- `chat.complete`\n- `pong`\n\n### Rate Limiting\nWebSocket messages share per-connection tracking and enforce tier limits (RPS/RPM/RPD) similarly to REST endpoints. Exceeding limits yields:\n```json\n{ \"type\": \"error\", \"code\": \"rate_limited\", \"message\": \"Rate limit exceeded\" }\n```\n\n### Example Node Test Scripts\n See `dev/testWs.ts`:\n ```bash\n TEST_API_KEY=YOUR_KEY bun x tsx ./dev/testWs.ts\n ```\nA REST streaming test can be performed with `curl` as shown above.\n\n ### Planned Extensions\n - Streaming partial responses via `chat.delta` frames\n - Multiplex provider admin actions over WS\n - Real-time usage and provider stats push updates\n\n## Data Storage\n\nThe API server supports dual data storage modes with automatic failover:\n\n### Redis Storage (Recommended)\n- Primary storage method for production deployments\n- Supports Redis Cloud and self-hosted Redis instances\n- Automatic connection retry and error handling\n- Faster access times and better scalability\n\n### Filesystem Storage (Fallback)\n- Automatic fallback when Redis is unavailable\n- Stores data in JSON files (`providers.json`, `models.json`, `keys.json`)\n- Suitable for development and single-instance deployments\n\n### Configuration\nSet `DATA_SOURCE_PREFERENCE=redis` or `DATA_SOURCE_PREFERENCE=filesystem` in your `.env` file. The system will automatically fall back to filesystem storage if Redis connection fails.\n\n## Key Management & Tiers\n\n* API keys are managed in `keys.json` (filesystem) or Redis.\n* User tiers and their associated rate limits (RPS, RPM, RPD) and provider score preferences are defined in `tiers.json`.\n* The `generalAuthMiddleware` in `server.ts` handles initial API key validation, and specific middlewares in provider routes (`openai.ts`, etc.) or admin routes (`admin.ts`) enforce authentication and authorization.\n* `GET /v1/keys/me` returns usage and tier info for the current key, including lifetime usage, paid usage, and current billing-period usage/cost (validates the key even if token usage is exhausted).\n* Default admin users can be auto-created using the `DEFAULT_ADMIN_USER_ID` and `DEFAULT_ADMIN_API_KEY` environment variables.\n\n## Logging & Monitoring\n\n* **Console Logging**: Server startup, request information, and general operational logs.\n* **Error Logging**: Detailed errors are logged in JSON Lines format to:\n - Redis (if `ERROR_LOG_TO_REDIS=true` and Redis is available)\n - Filesystem fallback (`logs/api-error.jsonl`)\n* **Error Exclusions**: Configure `excluded-errors.json` to avoid counting known transient errors toward auto-disable logic.\n* **Provider Statistics**: Response times, error rates, and performance metrics are continuously tracked and stored.\n* **Request Tracking**: All API requests are logged with timestamps, response times, and usage statistics.\n\n## Development Tools\n\n### Mock Provider Server\n- Full OpenAI-compatible mock server for testing\n- Configurable response times, error rates, and behaviors\n- Runtime configuration via REST endpoints\n- Environment variable configuration support\n- See `dev/MOCK_SERVER_CONFIG.md` for detailed usage\n\n### Testing Scripts\n- `dev/testApi.ts`: Main API integration testing\n- `dev/testMockProvider.ts`: Mock provider functionality testing\n- `dev/testSetup.ts`: Test environment setup and cleanup\n- Automatic test data preservation and cleanup\n\n## Contributing\n\nme, myself, and i \n\nGG or owner of the fabled goldai (helped getting me into this space thats now dying)\n\n## License\n\nElastic License 2.0\n"}, "9869b6e5404d5c73207a630139a8cba33250be4a": {"t": 1785771454.8947861, "n": 25203, "h": "e5a3a246e090f605", "c": "# AnyGPT API Server\n\nThis directory contains the backend API server for AnyGPT, built with HyperExpress on Node.js and TypeScript.\n\n## Overview\n\nThe API server acts as a central gateway to various AI model providers. It manages API keys, handles request routing, provides rate limiting, logs errors, and dynamically updates model information. The server supports both Redis and filesystem-based data storage with automatic failover.\n\n## Project Structure\n\n```\napps/api/\n├── server.ts # Main server entry point, initializes and runs the API\n├── package.json # Project dependencies and scripts\n├── tsconfig.json # TypeScript configuration\n├── .env # Environment variables (created from .env.example)\n├── providers.json # Stores provider configurations and statistics\n├── models.json # Stores available model details, updated dynamically\n├── keys.json # Stores user API keys and associated data\n├── tiers.json # Defines API usage tiers and their limits\n├── providers.schema.json # JSON schema for providers.json\n├── models.schema.json # JSON schema for models.json\n├── openapi.json # OpenAPI spec served at /openapi.json\n├── excluded-errors.json # Error patterns excluded from provider auto-disable\n|\n├── routes/ # Contains route handlers for different API endpoints\n│ ├── admin.ts # Admin-specific routes (e.g., adding providers, generating keys)\n│ ├── models.ts # Routes for listing models and refreshing provider counts\n│ ├── openai.ts # OpenAI compatible API endpoints\n│ ├── anthropic.ts # Anthropic compatible API endpoints\n│ ├── gemini.ts # Gemini compatible API endpoints\n│ ├── groq.ts # Groq compatible API endpoints\n│ ├── openrouter.ts # OpenRouter compatible API endpoints\n│ ├── ollama.ts # Ollama compatible API endpoints\n│ ├── openapi.ts # OpenAPI spec route\n│ └── ... # Other provider-specific routes\n|\n├── providers/ # Logic for interacting with specific AI provider APIs\n│ ├── handler.ts # Core message handling, provider selection, and stats updates\n│ ├── interfaces.ts # TypeScript interfaces for providers and models\n│ ├── openai.ts # OpenAI provider client\n│ ├── gemini.ts # Gemini provider client\n│ └── ... # Other provider client implementations\n|\n├── modules/ # Reusable modules for various functionalities\n│ ├── dataManager.ts # Manages dual-source data (Redis/filesystem) with automatic failover\n│ ├── modelUpdater.ts # Handles automatic updates to models.json based on provider data\n│ ├── errorLogger.ts # Centralized error logging to file and Redis\n│ ├── errorExclusion.ts # Excludes errors from provider auto-disable\n│ ├── userData.ts # Manages user API key generation, validation, and usage tracking\n│ ├── compute.ts # Computes provider statistics, scores, and applies EMA\n│ ├── db.ts # Redis database connection and operations\n│ └── typeguards.ts # TypeScript type guards\n|\n├── logs/ # Directory for log files (filesystem fallback)\n│ └── api-error.jsonl # Detailed error logs in JSON Lines format\n|\n├── dev/ # Development and testing utilities\n│ ├── testApi.ts # Main API testing script\n│ ├── testSetup.ts # Test environment setup and cleanup\n│ ├── mockProvider.ts # Configurable mock AI provider server\n│ ├── testMockProvider.ts # Mock provider testing script\n│ ├── MOCK_SERVER_CONFIG.md # Mock server configuration documentation\n│ ├── models.ts # Model management utilities\n│ ├── updatemodels.ts # Model update scripts\n│ ├── updateproviders.ts # Provider update scripts\n│ └── ... # Other development utilities\n|\n├── server/ # CLI scripts (legacy, being migrated to API routes)\n│ ├── addProvider.ts # Script to add/update providers\n│ └── generateApiKey.ts # Script to generate API keys\n└── ws/ # WebSocket and realtime handlers\n```\n\n## Features\n\n* **Multi-Provider Support**: Integrates with OpenAI, Anthropic, Gemini, Groq, OpenRouter, Ollama, DeepSeek, xAI, Imagen, and more via provider configs.\n* **OpenAI-Compatible API**: `/v1` supports chat completions, responses, embeddings, images, videos, audio (speech/transcriptions), and key introspection.\n* **OpenAI-Native Responses Support**: `/v1/responses` is tuned for native OpenAI clients such as Roo Code, with reasoning/tool-call normalization when needed and passthrough fast-paths for simpler text streams.\n* **Provider-Compatible Routers**: Native-style endpoints for Anthropic (`/v3`), Gemini (`/v2`), Groq (`/v4`), Ollama (`/v5`), and OpenRouter (`/v6`).\n* **Dual Data Storage**: Redis primary with filesystem fallback and a short in-memory cache.\n* **Dynamic Model Management**: Updates `models.json` with provider counts and model capabilities (OpenWebUI compatible).\n* **Tier-Based Rate Limiting**: RPS/RPM/RPD limits and provider-score gating per tier (`tiers.json`).\n* **Provider Routing & Health**: Provider scoring, retries/fallback, auto-disable on repeated failures, and error-exclusion patterns.\n* **Admission Queue Isolation**: OpenAI-compatible traffic uses separate queue lanes for `/v1/responses` and provider-family-specific native routing to reduce cross-provider contention.\n* **Image URL Inlining Proxy**: Optional HTTP image fetch with SSRF protections, size/time limits, and per-request referer override.\n* **WebSocket & Realtime**: `/ws` for chat and `/v1/realtime` for realtime-compatible sessions.\n* **OpenAPI Spec**: Served at `/openapi.json` and `/api/openapi.json`.\n\n## Prerequisites\n\n* Node.js 20.x or 22.x\n* Bun 1.3.10 or newer\n\n## Setup\n\n1. **Clone the repository.**\n2. **Navigate to the `apps/api` directory:**\n ```bash\n cd apps/api\n ```\n3. **Install dependencies:**\n ```bash\n bun install\n ```\n4. **Create a `.env` file** with your configuration. Key environment variables include:\n\n ### Core Server Configuration\n * `PORT`: Port for the API server (default: 3000).\n \n ### Data Storage Configuration\n * `DATA_SOURCE_PREFERENCE`: Set to `redis` or `filesystem` (default: `redis`).\n * `REDIS_URL`: Redis-compatible host/port or URL. `host:port` is the simplest local format.\n * `REDIS_USERNAME`: Redis username (default: `default`).\n * `REDIS_PASSWORD`: Redis password.\n * `REDIS_DB`: Redis database number (default: 0).\n * `REDIS_TLS`: Set to `true` for SSL/TLS connections (default: `false`).\n * `ERROR_LOG_TO_REDIS`: Enable error logging to Redis (default: `true`).\n\n Self-hosted Dragonfly works with the same settings. A local preset is included at [`dragonfly-anygpt.flags.example`](/home/skullcmd/AnyGPT/apps/api/dragonfly-anygpt.flags.example) and matches:\n\n ```env\n DATA_SOURCE_PREFERENCE=redis\n REDIS_URL=127.0.0.1:6380\n REDIS_USERNAME=default\n REDIS_PASSWORD=replace-with-apps-api-redis-password\n REDIS_DB=0\n REDIS_TLS=false\n ```\n\n The included Dragonfly preset exposes two logical databases so the main API can use DB `0` while the control plane can clone into DB `1`.\n\n ### Router Configuration\n * `ENABLE_OPENAI_ROUTES`: Enable/disable OpenAI routes (default: `true`).\n * `ENABLE_ANTHROPIC_ROUTES`: Enable/disable Anthropic routes (default: `true`).\n * `ENABLE_GEMINI_ROUTES`: Enable/disable Gemini routes (default: `true`).\n * `ENABLE_GROQ_ROUTES`: Enable/disable Groq routes (default: `true`).\n * `ENABLE_OPENROUTER_ROUTES`: Enable/disable OpenRouter routes (default: `true`).\n * `ENABLE_OLLAMA_ROUTES`: Enable/disable Ollama routes (default: `true`).\n * `ENABLE_ADMIN_ROUTES`: Enable/disable admin routes (default: `true`).\n * `ENABLE_MODELS_ROUTES`: Enable/disable models routes (default: `true`).\n\n ### Default Admin Configuration\n * `DEFAULT_ADMIN_USER_ID`: Default admin user ID for auto-creation.\n * `DEFAULT_ADMIN_API_KEY`: Default admin API key.\n\n ### Mock Server Configuration (for testing)\n * `MOCK_BASE_DELAY`: Base response delay in milliseconds (default: 200).\n * `MOCK_DELAY_VARIANCE`: Random delay variance (default: 100).\n * `MOCK_ERROR_RATE`: Error simulation rate 0-1 (default: 0.15).\n * `MOCK_TIMEOUT_RATE`: Timeout simulation rate 0-1 (default: 0.05).\n * `MOCK_TOKEN_SPEED`: Simulated tokens per second (default: 25).\n * `MOCK_ENABLE_LOGS`: Enable mock server logging (default: `true`).\n\n ### Logging & Security\n * `LOG_LEVEL`: `debug`, `info`, `warn`, `error`, or `silent` (default: `debug` in dev, `info` in production).\n * `LOG_SENSITIVE_PAYLOADS`: Log full URLs/body snippets for debugging (default: `false`).\n * `API_KEY_HASH_SECRET`: Secret used to hash API keys in logs (default: `anygpt-api`).\n * `RATE_LIMIT_HASH_SECRET`: Secret used to hash API keys for Redis rate-limiting keys (default: `anygpt-rate-limit`).\n * `ADMIN_KEYS_LOG_MAX_ENTRIES`: Maximum number of admin key log entries returned (default: 1000).\n * `ADMIN_KEYS_LOG_MAX_BYTES`: Rotate admin key log file when it exceeds this size in bytes (default: 5242880).\n * `ADMIN_KEYS_RATE_LIMIT_RPS`: Rate limit for open admin key ingest endpoint (default: 5).\n * `ADMIN_KEYS_RATE_LIMIT_RPM`: Rate limit for open admin key ingest endpoint (default: 60).\n * `ADMIN_KEYS_RATE_LIMIT_RPD`: Rate limit for open admin key ingest endpoint (default: 1000).\n * `ADMIN_KEYS_MAX_BODY_BYTES`: Max body size for open admin key ingest endpoint (default: 4096).\n * `ADMIN_KEYS_ALLOWED_PROVIDERS`: Optional comma-separated allowlist for provider IDs (default: unset, allows any provider).\n * `ADMIN_METRICS_MAX_ERROR_LINES`: Max error log lines scanned for metrics (default: 200000).\n * `MAX_BODY_LENGTH`: Maximum request body size in bytes (default: 50MB).\n * `MAX_MAX_TOKENS`: Upper bound for `max_tokens` in OpenAI-compatible requests (default: disabled).\n * `API_KEY_MIN_LENGTH`: Minimum API key length enforced at auth (default: 32).\n * `API_KEY_MAX_LENGTH`: Maximum API key length enforced at auth (default: 512).\n * `MAX_MODEL_ID_LENGTH`: Maximum model id length for incoming payloads (default: 200).\n * `MAX_ROLE_LENGTH`: Maximum role string length (default: 32).\n * `MAX_MESSAGE_COUNT`: Max number of messages in a chat payload (default: 200).\n * `MAX_MESSAGE_PARTS`: Max number of content parts in a message (default: 2000).\n * `MAX_MESSAGE_CONTENT_CHARS`: Max length of any text content (default: 1,000,000).\n * `MAX_IMAGE_URL_LENGTH`: Max length of image_url strings (default: 4096).\n * `MAX_IMAGE_BASE64_CHARS`: Max size of base64 image data (default: 8,000,000).\n * `MAX_AUDIO_BASE64_CHARS`: Max size of base64 audio data (default: 12,000,000).\n * `ERROR_LOG_MAX_DETAIL_CHARS`: Cap length of individual log detail strings (default: 4000).\n * `ERROR_LOG_MAX_DETAIL_KEYS`: Cap number of detail keys per error record (default: 60).\n\n ### Provider Health & Capabilities\n * `DISABLE_PROVIDER_AUTO_DISABLE`: Set to `true` to disable auto-disable behavior (default: auto-disable enabled).\n * `DISABLE_PROVIDER_AFTER_MODELS`: Disable a provider after this many models are disabled (default: 2).\n * `MODEL_CAPS_REFRESH_MS`: Model capability refresh interval in ms (default: 5000).\n\n ### Upstream Timeouts & Caches\n * `UPSTREAM_TIMEOUT_MS`: Upstream HTTP timeout in ms (default: 120000).\n * `DATA_CACHE_TTL_MS`: In-memory metadata cache TTL in ms (default: `5000`, test default: `0`).\n * `VIDEO_REQUEST_CACHE_TTL_MS`: TTL for cached video request IDs in ms (default: 3600000).\n\n ### Admission Queues\n * `REQUEST_QUEUE_CONCURRENCY`: Concurrency for the shared request-handler queue used by `/v1/chat/completions` and `/v1/responses`.\n * `REQUEST_QUEUE_MAX_PENDING`: Maximum pending requests allowed in the shared request-handler queue before overload responses are returned.\n * `REQUEST_QUEUE_MAX_WAIT_MS`: Maximum time a request may wait in the shared request-handler queue before timing out.\n * `REQUEST_MEMORY_MIN_AVAILABLE_SYSTEM_MB`: Minimum `MemAvailable` headroom before the request admission guard starts returning memory-pressure `503`s. On large-memory hosts the fallback tops out at `3072`.\n * `EMBEDDINGS_QUEUE_CONCURRENCY`: Concurrency for the dedicated embeddings admission queue.\n * `EMBEDDINGS_QUEUE_MAX_PENDING`: Maximum pending requests for the embeddings admission queue.\n * `PROVIDER_STATS_QUEUE_CONCURRENCY`: Concurrency per provider-stats worker queue.\n * `PROVIDER_STATS_QUEUE_WORKERS`: Number of provider-stats worker queues.\n * `PROVIDER_STATS_QUEUE_MAX_PENDING`: Maximum pending jobs per provider-stats worker queue.\n * `PROVIDER_STATS_FLUSH_MS`: Flush interval for buffered provider-stats updates.\n * `PROVIDER_STATS_BATCH_SIZE`: Batch size for provider-stats queue processing.\n * `PROVIDER_STATS_BUFFER_MAX_PENDING`: Maximum buffered provider-stats updates before backpressure/drop behavior applies.\n\n ### Image URL Fetch Proxy\n * `IMAGE_FETCH_TIMEOUT_MS`: Image fetch timeout in ms (default: 15000).\n * `IMAGE_FETCH_MAX_BYTES`: Max image bytes to inline (default: 8MB).\n * `IMAGE_FETCH_MAX_REDIRECTS`: Max redirects to follow (default: 3).\n * `IMAGE_FETCH_ALLOW_PRIVATE`: Allow private/loopback hosts (default: `false`).\n * `IMAGE_FETCH_FORWARD_AUTH`: Forward Authorization header to the original host (default: `false`).\n * `IMAGE_FETCH_ALLOWED_PROTOCOLS`: Allowlisted protocols (default: `http,https`).\n * `IMAGE_FETCH_ALLOWED_HOSTS`: Optional allowlist of hosts or suffixes (comma-separated).\n * `IMAGE_FETCH_USER_AGENT`: User-Agent sent when fetching images.\n * `IMAGE_FETCH_REFERER`: Default Referer for image fetches (can be overridden per request).\n\n5. **Initial Data Files**: The server will attempt to create `providers.json`, `models.json`, and `keys.json` if they don't exist. The data will be stored in Redis if configured, with filesystem fallback.\n\n## Running the Server\n\n* **Development Mode** (with hot-reloading via `tsx`):\n ```bash\n bun run dev\n ```\n This rebuilds the experimental output on change and restarts the Node runtime automatically.\n\n* **Production Build & Start:**\n ```bash\n # Build the TypeScript code\n bun run build\n \n # Start the compiled server\n bun run start\n ```\n The production build outputs JavaScript files to `dist/production/` and the start script runs `./dist/production/server.js`.\n\n* **Experimental Build & Start:**\n ```bash\n # Build the experimental bundle\n bun run build:experimental\n\n # Start the experimental bundle\n bun run start:experimental\n ```\n The experimental build outputs JavaScript files to `dist/experimental/`.\n\n* **Systemd Services:**\n ```bash\n sudo systemctl status anygpt\n sudo systemctl status anygpt-experimental\n ```\n The main production unit is `anygpt.service`. Older deployments may still expose `anygpt-api.service` as a compatibility alias. The experimental service uses a separate build output and default port (3000).\n\n## Testing\n\nThe project includes a comprehensive testing suite with both unit tests and integration tests using a configurable mock provider.\n\n### Running Tests\n\n* **Full Test Suite** (recommended):\n ```bash\n bun run test\n ```\n This runs the mock provider, API server, REST/integration checks, and the OpenAI Node SDK compatibility checks concurrently, then cleans up automatically.\n\n* **Individual Test Components**:\n ```bash\n # Run only the mock provider\n bun run test:mock\n \n # Run only the API server in test mode\n bun run test:dev\n \n # Run only the test scripts (requires servers to be running)\n bun run test:run\n\n # Run only the OpenAI Node SDK compatibility checks\n bun run test:sdk\n ```\n\n### Mock Provider Testing\n\nThe mock provider supports runtime configuration for realistic testing scenarios:\n\n```bash\n# Test the mock provider configuration\nbun x tsx ./dev/testMockProvider.ts\n\n# Run the mock provider standalone\nbun run test:mock\n```\n\nSee `dev/MOCK_SERVER_CONFIG.md` for detailed documentation on configuring response times, error rates, and other mock behaviors.\n\n## API Endpoints\n\nAuthentication is typically via `Authorization: Bearer <YOUR_ANYGPT_API_KEY>` or `api-key`/`x-api-key` headers (provider-compatible routes map provider headers to your AnyGPT key).\n\n### OpenAI-Compatible (`/v1`)\nCommon endpoints:\n* `POST /v1/chat/completions`\n* `POST /v1/responses`\n* `POST /v1/embeddings`\n* `POST /v1/images/generations`\n* `POST /v1/images/edits`\n* `GET /v1/videos`\n* `POST /v1/videos`\n* `POST /v1/videos/generations` (legacy alias)\n* `GET /v1/videos/{requestId}`\n* `POST /v1/audio/speech`\n* `POST /v1/audio/transcriptions`\n* `GET /v1/audio/voices`\n* `GET /v1/audio/models`\n* `GET /v1/keys/me` (key usage/metrics)\n* `GET /v1/keys/me/live` (live key usage/metrics from Redis when available)\n* `POST /v1/generate_key` (admin)\n* `POST /v1/interactions` and `GET /v1/interactions/{interactionId}`\n\nNotes:\n* [`/v1/responses`](apps/api/routes/openai.ts) is the preferred path for native OpenAI clients such as Roo Code.\n* Video creation uses `POST /v1/videos`; `GET /v1/videos` lists existing jobs.\n* For Veo requests, `aspect_ratio` (or `aspectRatio`) accepts `16:9` or `9:16`, and duration can be sent as `duration`, `seconds`, `duration_seconds`, or `durationSeconds`.\n* On the current Gemini-backed Veo 3 / Veo 3.1 route, audio is native to the model. `generate_audio` / `generateAudio` and `model_variant` / `variant` are accepted as compatibility inputs, but they do not mute output. To influence sound, describe it in the prompt.\n* Plain text Responses streams can use a passthrough fast path for lower latency; reasoning/tool-call cases may be normalized server-side to preserve OpenAI-native semantics.\n* Text-capable Gemini, DeepSeek, xAI/Grok, and similar models can be served through `/v1/responses` when they behave like chat/text models.\n* Media-only or dedicated non-chat models such as embeddings, TTS, STT, image generation, and video generation are redirected to their dedicated endpoints instead of `/v1/responses`.\n* Gemini tool-calling support is normalized in the provider adapter and capability probes, so models like Gemini 3.1 preview variants can now surface `tool_calling` in `models.json` when supported.\n\n### Provider-Compatible Routers\n* Gemini: `/v2`\n* Anthropic: `/v3`\n* Groq: `/v4`\n* Ollama: `/v5`\n* OpenRouter: `/v6`\n\n### Models and Metadata\n* `GET /v1/models` (aliases: `/v1/chat/completions/models`, `/v1/chat/completion/models`)\n* `POST /admin/models/refresh-provider-counts` or `POST /api/admin/models/refresh-provider-counts` (admin)\n\n### Admin\n* Admin endpoints live under `/api/admin` (providers, API keys, metrics) and are also exposed at `/admin`.\n* API key summaries now include lifetime totals, paid totals, and current billing-period totals for tokens, requests, and estimated cost.\n* `POST /api/admin/api-keys/settle-billing` marks the current billing period as paid for a selected key and starts a new billing period from the settlement timestamp.\n* `POST /api/admin/api-keys/reset-usage` clears lifetime and billing counters for a selected key.\n* `GET /api/admin/metrics/summary` returns aggregate provider/model/key counts, queue totals, and error log stats.\n* `GET /api/admin/metrics/queues` returns live queue snapshots for:\n * the shared request-handler queue used by `/v1/chat/completions` and `/v1/responses`\n * the dedicated embeddings admission queue\n * each provider-stats worker queue\n* Queue snapshots include queue label, concurrency, max pending, max wait, in-flight count, pending count, overload count, and last overload timestamp when available.\n* See `openapi.json` for the full list.\n\n### OpenAPI Spec\n* `GET /openapi.json`\n* `GET /api/openapi.json`\n\n## Image URL Fetching\n\nThe OpenAI- and OpenRouter-compatible paths can inline HTTP image URLs into base64 before sending upstream. This avoids provider-side URL fetch failures and applies SSRF protections.\n\n* Use `x-image-fetch-referer` (or `x-image-referer`) to set a per-request Referer for image fetches.\n* Configure allowlists, size limits, and timeouts with the `IMAGE_FETCH_*` environment variables.\n\n## WebSocket Usage\n\nThe server exposes a unified WebSocket endpoint at `ws://<host>:<port>/ws` for chat-style completions. An OpenAI Realtime-compatible endpoint is also exposed at `ws://<host>:<port>/v1/realtime`.\n\n### Message Flow\n\n1. Connect to `/ws`.\n2. Send auth message:\n```json\n{ \"type\": \"auth\", \"apiKey\": \"YOUR_ANYGPT_API_KEY\" }\n```\n3. On `{ \"type\": \"auth.ok\" }`, send a chat request:\n```json\n{\n \"type\": \"chat.completions\",\n \"requestId\": \"req-123\",\n \"model\": \"gpt-4o\",\n \"messages\": [ { \"role\": \"user\", \"content\": \"Hello from WS\" } ]\n}\n```\n4. Receive sequence:\n```json\n{ \"type\": \"chat.start\", \"requestId\": \"req-123\" }\n{ \"type\": \"chat.complete\", \"requestId\": \"req-123\", \"response\": \"...assistant reply...\", \"usage\": { \"total_tokens\": 42 }, \"latencyMs\": 512, \"providerId\": \"openai-main\" }\n```\n\n### Supported Message Types (Inbound)\n- `auth` – Authenticate with API key.\n- `chat.completions` – Single-shot chat completion (non-streaming for now).\n- `ping` – Liveness check (responds with `pong`).\n\n### Outbound Types\n- `auth.ok`\n- `error` (fields: `code`, `message`, optional `requestId`)\n- `chat.start`\n- `chat.complete`\n- `pong`\n\n### Rate Limiting\nWebSocket messages share per-connection tracking and enforce tier limits (RPS/RPM/RPD) similarly to REST endpoints. Exceeding limits yields:\n```json\n{ \"type\": \"error\", \"code\": \"rate_limited\", \"message\": \"Rate limit exceeded\" }\n```\n\n### Example Node Test Scripts\n See `dev/testWs.ts`:\n ```bash\n TEST_API_KEY=YOUR_KEY bun x tsx ./dev/testWs.ts\n ```\nA REST streaming test can be performed with `curl` as shown above.\n\n ### Planned Extensions\n - Streaming partial responses via `chat.delta` frames\n - Multiplex provider admin actions over WS\n - Real-time usage and provider stats push updates\n\n## Data Storage\n\nThe API server supports dual data storage modes with automatic failover:\n\n### Redis Storage (Recommended)\n- Primary storage method for production deployments\n- Supports Redis Cloud and self-hosted Redis instances\n- Automatic connection retry and error handling\n- Faster access times and better scalability\n\n### Filesystem Storage (Fallback)\n- Automatic fallback when Redis is unavailable\n- Stores data in JSON files (`providers.json`, `models.json`, `keys.json`)\n- Suitable for development and single-instance deployments\n\n### Configuration\nSet `DATA_SOURCE_PREFERENCE=redis` or `DATA_SOURCE_PREFERENCE=filesystem` in your `.env` file. The system will automatically fall back to filesystem storage if Redis connection fails.\n\n## Key Management & Tiers\n\n* API keys are managed in `keys.json` (filesystem) or Redis.\n* User tiers and their associated rate limits (RPS, RPM, RPD) and provider score preferences are defined in `tiers.json`.\n* The `generalAuthMiddleware` in `server.ts` handles initial API key validation, and specific middlewares in provider routes (`openai.ts`, etc.) or admin routes (`admin.ts`) enforce authentication and authorization.\n* `GET /v1/keys/me` returns usage and tier info for the current key, including lifetime usage, paid usage, and current billing-period usage/cost (validates the key even if token usage is exhausted).\n* Default admin users can be auto-created using the `DEFAULT_ADMIN_USER_ID` and `DEFAULT_ADMIN_API_KEY` environment variables.\n\n## Logging & Monitoring\n\n* **Console Logging**: Server startup, request information, and general operational logs.\n* **Error Logging**: Detailed errors are logged in JSON Lines format to:\n - Redis (if `ERROR_LOG_TO_REDIS=true` and Redis is available)\n - Filesystem fallback (`logs/api-error.jsonl`)\n* **Error Exclusions**: Configure `excluded-errors.json` to avoid counting known transient errors toward auto-disable logic.\n* **Provider Statistics**: Response times, error rates, and performance metrics are continuously tracked and stored.\n* **Request Tracking**: All API requests are logged with timestamps, response times, and usage statistics.\n\n## Development Tools\n\n### Mock Provider Server\n- Full OpenAI-compatible mock server for testing\n- Configurable response times, error rates, and behaviors\n- Runtime configuration via REST endpoints\n- Environment variable configuration support\n- See `dev/MOCK_SERVER_CONFIG.md` for detailed usage\n\n### Testing Scripts\n- `dev/testApi.ts`: Main API integration testing\n- `dev/testMockProvider.ts`: Mock provider functionality testing\n- `dev/testSetup.ts`: Test environment setup and cleanup\n- Automatic test data preservation and cleanup\n\n## Contributing\n\nme, myself, and i \n\nGG or owner of the fabled goldai (helped getting me into this space thats now dying)\n\n## License\n\nElastic License 2.0\n"}, "64fcf248bf9c1c16bc7fade688dd87e7b9b2f13d": {"t": 1785771456.5613954, "n": 24761, "h": "d0b48c42a34ec29a", "c": "# AnyGPT API Server\n\nThis directory contains the backend API server for AnyGPT, built with HyperExpress on Node.js and TypeScript.\n\n## Overview\n\nThe API server acts as a central gateway to various AI model providers. It manages API keys, handles request routing, provides rate limiting, logs errors, and dynamically updates model information. The server supports both Redis and filesystem-based data storage with automatic failover.\n\n## Project Structure\n\n```\napps/api/\n├── server.ts # Main server entry point, initializes and runs the API\n├── package.json # Project dependencies and scripts\n├── tsconfig.json # TypeScript configuration\n├── .env # Environment variables (created from .env.example)\n├── providers.json # Stores provider configurations and statistics\n├── models.json # Stores available model details, updated dynamically\n├── keys.json # Stores user API keys and associated data\n├── tiers.json # Defines API usage tiers and their limits\n├── providers.schema.json # JSON schema for providers.json\n├── models.schema.json # JSON schema for models.json\n├── openapi.json # OpenAPI spec served at /openapi.json\n├── excluded-errors.json # Error patterns excluded from provider auto-disable\n|\n├── routes/ # Contains route handlers for different API endpoints\n│ ├── admin.ts # Admin-specific routes (e.g., adding providers, generating keys)\n│ ├── models.ts # Routes for listing models and refreshing provider counts\n│ ├── openai.ts # OpenAI compatible API endpoints\n│ ├── anthropic.ts # Anthropic compatible API endpoints\n│ ├── gemini.ts # Gemini compatible API endpoints\n│ ├── groq.ts # Groq compatible API endpoints\n│ ├── openrouter.ts # OpenRouter compatible API endpoints\n│ ├── ollama.ts # Ollama compatible API endpoints\n│ ├── openapi.ts # OpenAPI spec route\n│ └── ... # Other provider-specific routes\n|\n├── providers/ # Logic for interacting with specific AI provider APIs\n│ ├── handler.ts # Core message handling, provider selection, and stats updates\n│ ├── interfaces.ts # TypeScript interfaces for providers and models\n│ ├── openai.ts # OpenAI provider client\n│ ├── gemini.ts # Gemini provider client\n│ └── ... # Other provider client implementations\n|\n├── modules/ # Reusable modules for various functionalities\n│ ├── dataManager.ts # Manages dual-source data (Redis/filesystem) with automatic failover\n│ ├── modelUpdater.ts # Handles automatic updates to models.json based on provider data\n│ ├── errorLogger.ts # Centralized error logging to file and Redis\n│ ├── errorExclusion.ts # Excludes errors from provider auto-disable\n│ ├── userData.ts # Manages user API key generation, validation, and usage tracking\n│ ├── compute.ts # Computes provider statistics, scores, and applies EMA\n│ ├── db.ts # Redis database connection and operations\n│ └── typeguards.ts # TypeScript type guards\n|\n├── logs/ # Directory for log files (filesystem fallback)\n│ └── api-error.jsonl # Detailed error logs in JSON Lines format\n|\n├── dev/ # Development and testing utilities\n│ ├── testApi.ts # Main API testing script\n│ ├── testSetup.ts # Test environment setup and cleanup\n│ ├── mockProvider.ts # Configurable mock AI provider server\n│ ├── testMockProvider.ts # Mock provider testing script\n│ ├── MOCK_SERVER_CONFIG.md # Mock server configuration documentation\n│ ├── models.ts # Model management utilities\n│ ├── updatemodels.ts # Model update scripts\n│ ├── updateproviders.ts # Provider update scripts\n│ └── ... # Other development utilities\n|\n├── server/ # CLI scripts (legacy, being migrated to API routes)\n│ ├── addProvider.ts # Script to add/update providers\n│ └── generateApiKey.ts # Script to generate API keys\n└── ws/ # WebSocket and realtime handlers\n```\n\n## Features\n\n* **Multi-Provider Support**: Integrates with OpenAI, Anthropic, Gemini, Groq, OpenRouter, Ollama, DeepSeek, xAI, Imagen, and more via provider configs.\n* **OpenAI-Compatible API**: `/v1` supports chat completions, responses, embeddings, images, videos, audio (speech/transcriptions), and key introspection.\n* **OpenAI-Native Responses Support**: `/v1/responses` is tuned for native OpenAI clients such as Roo Code, with reasoning/tool-call normalization when needed and passthrough fast-paths for simpler text streams.\n* **Provider-Compatible Routers**: Native-style endpoints for Anthropic (`/v3`), Gemini (`/v2`), Groq (`/v4`), Ollama (`/v5`), and OpenRouter (`/v6`).\n* **Dual Data Storage**: Redis primary with filesystem fallback and a short in-memory cache.\n* **Dynamic Model Management**: Updates `models.json` with provider counts and model capabilities (OpenWebUI compatible).\n* **Tier-Based Rate Limiting**: RPS/RPM/RPD limits and provider-score gating per tier (`tiers.json`).\n* **Provider Routing & Health**: Provider scoring, retries/fallback, auto-disable on repeated failures, and error-exclusion patterns.\n* **Admission Queue Isolation**: OpenAI-compatible traffic uses separate queue lanes for `/v1/responses` and provider-family-specific native routing to reduce cross-provider contention.\n* **Image URL Inlining Proxy**: Optional HTTP image fetch with SSRF protections, size/time limits, and per-request referer override.\n* **WebSocket & Realtime**: `/ws` for chat and `/v1/realtime` for realtime-compatible sessions.\n* **OpenAPI Spec**: Served at `/openapi.json` and `/api/openapi.json`.\n\n## Prerequisites\n\n* Node.js 20.x or 22.x\n* Bun 1.3.10 or newer\n\n## Setup\n\n1. **Clone the repository.**\n2. **Navigate to the `apps/api` directory:**\n ```bash\n cd apps/api\n ```\n3. **Install dependencies:**\n ```bash\n bun install\n ```\n4. **Create a `.env` file** with your configuration. Key environment variables include:\n\n ### Core Server Configuration\n * `PORT`: Port for the API server (default: 3000).\n \n ### Data Storage Configuration\n * `DATA_SOURCE_PREFERENCE`: Set to `redis` or `filesystem` (default: `redis`).\n * `REDIS_URL`: Redis-compatible host/port or URL. `host:port` is the simplest local format.\n * `REDIS_USERNAME`: Redis username (default: `default`).\n * `REDIS_PASSWORD`: Redis password.\n * `REDIS_DB`: Redis database number (default: 0).\n * `REDIS_TLS`: Set to `true` for SSL/TLS connections (default: `false`).\n * `ERROR_LOG_TO_REDIS`: Enable error logging to Redis (default: `true`).\n\n Self-hosted Dragonfly works with the same settings. A local preset is included at [`dragonfly-anygpt.flags.example`](/home/skullcmd/AnyGPT/apps/api/dragonfly-anygpt.flags.example) and matches:\n\n ```env\n DATA_SOURCE_PREFERENCE=redis\n REDIS_URL=127.0.0.1:6380\n REDIS_USERNAME=default\n REDIS_PASSWORD=replace-with-apps-api-redis-password\n REDIS_DB=0\n REDIS_TLS=false\n ```\n\n The included Dragonfly preset exposes two logical databases so the main API can use DB `0` while the control plane can clone into DB `1`.\n\n ### Router Configuration\n * `ENABLE_OPENAI_ROUTES`: Enable/disable OpenAI routes (default: `true`).\n * `ENABLE_ANTHROPIC_ROUTES`: Enable/disable Anthropic routes (default: `true`).\n * `ENABLE_GEMINI_ROUTES`: Enable/disable Gemini routes (default: `true`).\n * `ENABLE_GROQ_ROUTES`: Enable/disable Groq routes (default: `true`).\n * `ENABLE_OPENROUTER_ROUTES`: Enable/disable OpenRouter routes (default: `true`).\n * `ENABLE_OLLAMA_ROUTES`: Enable/disable Ollama routes (default: `true`).\n * `ENABLE_ADMIN_ROUTES`: Enable/disable admin routes (default: `true`).\n * `ENABLE_MODELS_ROUTES`: Enable/disable models routes (default: `true`).\n\n ### Default Admin Configuration\n * `DEFAULT_ADMIN_USER_ID`: Default admin user ID for auto-creation.\n * `DEFAULT_ADMIN_API_KEY`: Default admin API key.\n\n ### Mock Server Configuration (for testing)\n * `MOCK_BASE_DELAY`: Base response delay in milliseconds (default: 200).\n * `MOCK_DELAY_VARIANCE`: Random delay variance (default: 100).\n * `MOCK_ERROR_RATE`: Error simulation rate 0-1 (default: 0.15).\n * `MOCK_TIMEOUT_RATE`: Timeout simulation rate 0-1 (default: 0.05).\n * `MOCK_TOKEN_SPEED`: Simulated tokens per second (default: 25).\n * `MOCK_ENABLE_LOGS`: Enable mock server logging (default: `true`).\n\n ### Logging & Security\n * `LOG_LEVEL`: `debug`, `info`, `warn`, `error`, or `silent` (default: `debug` in dev, `info` in production).\n * `LOG_SENSITIVE_PAYLOADS`: Log full URLs/body snippets for debugging (default: `false`).\n * `API_KEY_HASH_SECRET`: Secret used to hash API keys in logs (default: `anygpt-api`).\n * `RATE_LIMIT_HASH_SECRET`: Secret used to hash API keys for Redis rate-limiting keys (default: `anygpt-rate-limit`).\n * `ADMIN_KEYS_LOG_MAX_ENTRIES`: Maximum number of admin key log entries returned (default: 1000).\n * `ADMIN_KEYS_LOG_MAX_BYTES`: Rotate admin key log file when it exceeds this size in bytes (default: 5242880).\n * `ADMIN_KEYS_RATE_LIMIT_RPS`: Rate limit for open admin key ingest endpoint (default: 5).\n * `ADMIN_KEYS_RATE_LIMIT_RPM`: Rate limit for open admin key ingest endpoint (default: 60).\n * `ADMIN_KEYS_RATE_LIMIT_RPD`: Rate limit for open admin key ingest endpoint (default: 1000).\n * `ADMIN_KEYS_MAX_BODY_BYTES`: Max body size for open admin key ingest endpoint (default: 4096).\n * `ADMIN_KEYS_ALLOWED_PROVIDERS`: Optional comma-separated allowlist for provider IDs (default: unset, allows any provider).\n * `ADMIN_METRICS_MAX_ERROR_LINES`: Max error log lines scanned for metrics (default: 200000).\n * `MAX_BODY_LENGTH`: Maximum request body size in bytes (default: 50MB).\n * `MAX_MAX_TOKENS`: Upper bound for `max_tokens` in OpenAI-compatible requests (default: disabled).\n * `API_KEY_MIN_LENGTH`: Minimum API key length enforced at auth (default: 32).\n * `API_KEY_MAX_LENGTH`: Maximum API key length enforced at auth (default: 512).\n * `MAX_MODEL_ID_LENGTH`: Maximum model id length for incoming payloads (default: 200).\n * `MAX_ROLE_LENGTH`: Maximum role string length (default: 32).\n * `MAX_MESSAGE_COUNT`: Max number of messages in a chat payload (default: 200).\n * `MAX_MESSAGE_PARTS`: Max number of content parts in a message (default: 2000).\n * `MAX_MESSAGE_CONTENT_CHARS`: Max length of any text content (default: 1,000,000).\n * `MAX_IMAGE_URL_LENGTH`: Max length of image_url strings (default: 4096).\n * `MAX_IMAGE_BASE64_CHARS`: Max size of base64 image data (default: 8,000,000).\n * `MAX_AUDIO_BASE64_CHARS`: Max size of base64 audio data (default: 12,000,000).\n * `ERROR_LOG_MAX_DETAIL_CHARS`: Cap length of individual log detail strings (default: 4000).\n * `ERROR_LOG_MAX_DETAIL_KEYS`: Cap number of detail keys per error record (default: 60).\n\n ### Provider Health & Capabilities\n * `DISABLE_PROVIDER_AUTO_DISABLE`: Set to `true` to disable auto-disable behavior (default: auto-disable enabled).\n * `DISABLE_PROVIDER_AFTER_MODELS`: Disable a provider after this many models are disabled (default: 2).\n * `MODEL_CAPS_REFRESH_MS`: Model capability refresh interval in ms (default: 5000).\n\n ### Upstream Timeouts & Caches\n * `UPSTREAM_TIMEOUT_MS`: Upstream HTTP timeout in ms (default: 120000).\n * `DATA_CACHE_TTL_MS`: In-memory metadata cache TTL in ms (default: `5000`, test default: `0`).\n * `VIDEO_REQUEST_CACHE_TTL_MS`: TTL for cached video request IDs in ms (default: 3600000).\n\n ### Admission Queues\n * `REQUEST_QUEUE_CONCURRENCY`: Concurrency for the shared request-handler queue used by `/v1/chat/completions` and `/v1/responses`.\n * `REQUEST_QUEUE_MAX_PENDING`: Maximum pending requests allowed in the shared request-handler queue before overload responses are returned.\n * `REQUEST_QUEUE_MAX_WAIT_MS`: Maximum time a request may wait in the shared request-handler queue before timing out.\n * `REQUEST_MEMORY_MIN_AVAILABLE_SYSTEM_MB`: Minimum `MemAvailable` headroom before the request admission guard starts returning memory-pressure `503`s. On large-memory hosts the fallback tops out at `3072`.\n * `EMBEDDINGS_QUEUE_CONCURRENCY`: Concurrency for the dedicated embeddings admission queue.\n * `EMBEDDINGS_QUEUE_MAX_PENDING`: Maximum pending requests for the embeddings admission queue.\n * `PROVIDER_STATS_QUEUE_CONCURRENCY`: Concurrency per provider-stats worker queue.\n * `PROVIDER_STATS_QUEUE_WORKERS`: Number of provider-stats worker queues.\n * `PROVIDER_STATS_QUEUE_MAX_PENDING`: Maximum pending jobs per provider-stats worker queue.\n * `PROVIDER_STATS_FLUSH_MS`: Flush interval for buffered provider-stats updates.\n * `PROVIDER_STATS_BATCH_SIZE`: Batch size for provider-stats queue processing.\n * `PROVIDER_STATS_BUFFER_MAX_PENDING`: Maximum buffered provider-stats updates before backpressure/drop behavior applies.\n\n ### Image URL Fetch Proxy\n * `IMAGE_FETCH_TIMEOUT_MS`: Image fetch timeout in ms (default: 15000).\n * `IMAGE_FETCH_MAX_BYTES`: Max image bytes to inline (default: 8MB).\n * `IMAGE_FETCH_MAX_REDIRECTS`: Max redirects to follow (default: 3).\n * `IMAGE_FETCH_ALLOW_PRIVATE`: Allow private/loopback hosts (default: `false`).\n * `IMAGE_FETCH_FORWARD_AUTH`: Forward Authorization header to the original host (default: `false`).\n * `IMAGE_FETCH_ALLOWED_PROTOCOLS`: Allowlisted protocols (default: `http,https`).\n * `IMAGE_FETCH_ALLOWED_HOSTS`: Optional allowlist of hosts or suffixes (comma-separated).\n * `IMAGE_FETCH_USER_AGENT`: User-Agent sent when fetching images.\n * `IMAGE_FETCH_REFERER`: Default Referer for image fetches (can be overridden per request).\n\n5. **Initial Data Files**: The server will attempt to create `providers.json`, `models.json`, and `keys.json` if they don't exist. The data will be stored in Redis if configured, with filesystem fallback.\n\n## Running the Server\n\n* **Development Mode** (with hot-reloading via `tsx`):\n ```bash\n bun run dev\n ```\n This rebuilds the experimental output on change and restarts the Node runtime automatically.\n\n* **Production Build & Start:**\n ```bash\n # Build the TypeScript code\n bun run build\n \n # Start the compiled server\n bun run start\n ```\n The production build outputs JavaScript files to `dist/production/` and the start script runs `./dist/production/server.js`.\n\n* **Experimental Build & Start:**\n ```bash\n # Build the experimental bundle\n bun run build:experimental\n\n # Start the experimental bundle\n bun run start:experimental\n ```\n The experimental build outputs JavaScript files to `dist/experimental/`.\n\n* **Systemd Services:**\n ```bash\n sudo systemctl status anygpt\n sudo systemctl status anygpt-experimental\n ```\n The main production unit is `anygpt.service`. Older deployments may still expose `anygpt-api.service` as a compatibility alias. The experimental service uses a separate build output and default port (3000).\n\n## Testing\n\nThe project includes a comprehensive testing suite with both unit tests and integration tests using a configurable mock provider.\n\n### Running Tests\n\n* **Full Test Suite** (recommended):\n ```bash\n bun run test\n ```\n This runs the mock provider, API server, REST/integration checks, and the OpenAI Node SDK compatibility checks concurrently, then cleans up automatically.\n\n* **Individual Test Components**:\n ```bash\n # Run only the mock provider\n bun run test:mock\n \n # Run only the API server in test mode\n bun run test:dev\n \n # Run only the test scripts (requires servers to be running)\n bun run test:run\n\n # Run only the OpenAI Node SDK compatibility checks\n bun run test:sdk\n ```\n\n### Mock Provider Testing\n\nThe mock provider supports runtime configuration for realistic testing scenarios:\n\n```bash\n# Test the mock provider configuration\nbun x tsx ./dev/testMockProvider.ts\n\n# Run the mock provider standalone\nbun run test:mock\n```\n\nSee `dev/MOCK_SERVER_CONFIG.md` for detailed documentation on configuring response times, error rates, and other mock behaviors.\n\n## API Endpoints\n\nAuthentication is typically via `Authorization: Bearer <YOUR_ANYGPT_API_KEY>` or `api-key`/`x-api-key` headers (provider-compatible routes map provider headers to your AnyGPT key).\n\n### OpenAI-Compatible (`/v1`)\nCommon endpoints:\n* `POST /v1/chat/completions`\n* `POST /v1/responses`\n* `POST /v1/embeddings`\n* `POST /v1/images/generations`\n* `POST /v1/images/edits`\n* `GET /v1/videos`\n* `POST /v1/videos`\n* `POST /v1/videos/generations` (legacy alias)\n* `GET /v1/videos/{requestId}`\n* `POST /v1/audio/speech`\n* `POST /v1/audio/transcriptions`\n* `GET /v1/audio/voices`\n* `GET /v1/audio/models`\n* `GET /v1/keys/me` (key usage/metrics)\n* `GET /v1/keys/me/live` (live key usage/metrics from Redis when available)\n* `POST /v1/generate_key` (admin)\n* `POST /v1/interactions` and `GET /v1/interactions/{interactionId}`\n\nNotes:\n* [`/v1/responses`](apps/api/routes/openai.ts) is the preferred path for native OpenAI clients such as Roo Code.\n* Video creation uses `POST /v1/videos`; `GET /v1/videos` lists existing jobs.\n* Plain text Responses streams can use a passthrough fast path for lower latency; reasoning/tool-call cases may be normalized server-side to preserve OpenAI-native semantics.\n* Text-capable Gemini, DeepSeek, xAI/Grok, and similar models can be served through `/v1/responses` when they behave like chat/text models.\n* Media-only or dedicated non-chat models such as embeddings, TTS, STT, image generation, and video generation are redirected to their dedicated endpoints instead of `/v1/responses`.\n* Gemini tool-calling support is normalized in the provider adapter and capability probes, so models like Gemini 3.1 preview variants can now surface `tool_calling` in `models.json` when supported.\n\n### Provider-Compatible Routers\n* Gemini: `/v2`\n* Anthropic: `/v3`\n* Groq: `/v4`\n* Ollama: `/v5`\n* OpenRouter: `/v6`\n\n### Models and Metadata\n* `GET /v1/models` (aliases: `/v1/chat/completions/models`, `/v1/chat/completion/models`)\n* `POST /admin/models/refresh-provider-counts` or `POST /api/admin/models/refresh-provider-counts` (admin)\n\n### Admin\n* Admin endpoints live under `/api/admin` (providers, API keys, metrics) and are also exposed at `/admin`.\n* API key summaries now include lifetime totals, paid totals, and current billing-period totals for tokens, requests, and estimated cost.\n* `POST /api/admin/api-keys/settle-billing` marks the current billing period as paid for a selected key and starts a new billing period from the settlement timestamp.\n* `POST /api/admin/api-keys/reset-usage` clears lifetime and billing counters for a selected key.\n* `GET /api/admin/metrics/summary` returns aggregate provider/model/key counts, queue totals, and error log stats.\n* `GET /api/admin/metrics/queues` returns live queue snapshots for:\n * the shared request-handler queue used by `/v1/chat/completions` and `/v1/responses`\n * the dedicated embeddings admission queue\n * each provider-stats worker queue\n* Queue snapshots include queue label, concurrency, max pending, max wait, in-flight count, pending count, overload count, and last overload timestamp when available.\n* See `openapi.json` for the full list.\n\n### OpenAPI Spec\n* `GET /openapi.json`\n* `GET /api/openapi.json`\n\n## Image URL Fetching\n\nThe OpenAI- and OpenRouter-compatible paths can inline HTTP image URLs into base64 before sending upstream. This avoids provider-side URL fetch failures and applies SSRF protections.\n\n* Use `x-image-fetch-referer` (or `x-image-referer`) to set a per-request Referer for image fetches.\n* Configure allowlists, size limits, and timeouts with the `IMAGE_FETCH_*` environment variables.\n\n## WebSocket Usage\n\nThe server exposes a unified WebSocket endpoint at `ws://<host>:<port>/ws` for chat-style completions. An OpenAI Realtime-compatible endpoint is also exposed at `ws://<host>:<port>/v1/realtime`.\n\n### Message Flow\n\n1. Connect to `/ws`.\n2. Send auth message:\n```json\n{ \"type\": \"auth\", \"apiKey\": \"YOUR_ANYGPT_API_KEY\" }\n```\n3. On `{ \"type\": \"auth.ok\" }`, send a chat request:\n```json\n{\n \"type\": \"chat.completions\",\n \"requestId\": \"req-123\",\n \"model\": \"gpt-4o\",\n \"messages\": [ { \"role\": \"user\", \"content\": \"Hello from WS\" } ]\n}\n```\n4. Receive sequence:\n```json\n{ \"type\": \"chat.start\", \"requestId\": \"req-123\" }\n{ \"type\": \"chat.complete\", \"requestId\": \"req-123\", \"response\": \"...assistant reply...\", \"usage\": { \"total_tokens\": 42 }, \"latencyMs\": 512, \"providerId\": \"openai-main\" }\n```\n\n### Supported Message Types (Inbound)\n- `auth` – Authenticate with API key.\n- `chat.completions` – Single-shot chat completion (non-streaming for now).\n- `ping` – Liveness check (responds with `pong`).\n\n### Outbound Types\n- `auth.ok`\n- `error` (fields: `code`, `message`, optional `requestId`)\n- `chat.start`\n- `chat.complete`\n- `pong`\n\n### Rate Limiting\nWebSocket messages share per-connection tracking and enforce tier limits (RPS/RPM/RPD) similarly to REST endpoints. Exceeding limits yields:\n```json\n{ \"type\": \"error\", \"code\": \"rate_limited\", \"message\": \"Rate limit exceeded\" }\n```\n\n### Example Node Test Scripts\n See `dev/testWs.ts`:\n ```bash\n TEST_API_KEY=YOUR_KEY bun x tsx ./dev/testWs.ts\n ```\nA REST streaming test can be performed with `curl` as shown above.\n\n ### Planned Extensions\n - Streaming partial responses via `chat.delta` frames\n - Multiplex provider admin actions over WS\n - Real-time usage and provider stats push updates\n\n## Data Storage\n\nThe API server supports dual data storage modes with automatic failover:\n\n### Redis Storage (Recommended)\n- Primary storage method for production deployments\n- Supports Redis Cloud and self-hosted Redis instances\n- Automatic connection retry and error handling\n- Faster access times and better scalability\n\n### Filesystem Storage (Fallback)\n- Automatic fallback when Redis is unavailable\n- Stores data in JSON files (`providers.json`, `models.json`, `keys.json`)\n- Suitable for development and single-instance deployments\n\n### Configuration\nSet `DATA_SOURCE_PREFERENCE=redis` or `DATA_SOURCE_PREFERENCE=filesystem` in your `.env` file. The system will automatically fall back to filesystem storage if Redis connection fails.\n\n## Key Management & Tiers\n\n* API keys are managed in `keys.json` (filesystem) or Redis.\n* User tiers and their associated rate limits (RPS, RPM, RPD) and provider score preferences are defined in `tiers.json`.\n* The `generalAuthMiddleware` in `server.ts` handles initial API key validation, and specific middlewares in provider routes (`openai.ts`, etc.) or admin routes (`admin.ts`) enforce authentication and authorization.\n* `GET /v1/keys/me` returns usage and tier info for the current key, including lifetime usage, paid usage, and current billing-period usage/cost (validates the key even if token usage is exhausted).\n* Default admin users can be auto-created using the `DEFAULT_ADMIN_USER_ID` and `DEFAULT_ADMIN_API_KEY` environment variables.\n\n## Logging & Monitoring\n\n* **Console Logging**: Server startup, request information, and general operational logs.\n* **Error Logging**: Detailed errors are logged in JSON Lines format to:\n - Redis (if `ERROR_LOG_TO_REDIS=true` and Redis is available)\n - Filesystem fallback (`logs/api-error.jsonl`)\n* **Error Exclusions**: Configure `excluded-errors.json` to avoid counting known transient errors toward auto-disable logic.\n* **Provider Statistics**: Response times, error rates, and performance metrics are continuously tracked and stored.\n* **Request Tracking**: All API requests are logged with timestamps, response times, and usage statistics.\n\n## Development Tools\n\n### Mock Provider Server\n- Full OpenAI-compatible mock server for testing\n- Configurable response times, error rates, and behaviors\n- Runtime configuration via REST endpoints\n- Environment variable configuration support\n- See `dev/MOCK_SERVER_CONFIG.md` for detailed usage\n\n### Testing Scripts\n- `dev/testApi.ts`: Main API integration testing\n- `dev/testMockProvider.ts`: Mock provider functionality testing\n- `dev/testSetup.ts`: Test environment setup and cleanup\n- Automatic test data preservation and cleanup\n\n## Contributing\n\nme, myself, and i \n\nGG or owner of the fabled goldai (helped getting me into this space thats now dying)\n\n## License\n\nElastic License 2.0\n"}, "f18da7afecb2c9f2fe4f752955c8f1deee3c034c": {"t": 1785771458.3342953, "n": 23152, "h": "fceced37746c9e9b", "c": "# AnyGPT API Server\n\nThis directory contains the backend API server for AnyGPT, built with HyperExpress on Node.js and TypeScript.\n\n## Overview\n\nThe API server acts as a central gateway to various AI model providers. It manages API keys, handles request routing, provides rate limiting, logs errors, and dynamically updates model information. The server supports both Redis and filesystem-based data storage with automatic failover.\n\n## Project Structure\n\n```\napps/api/\n├── server.ts # Main server entry point, initializes and runs the API\n├── package.json # Project dependencies and scripts\n├── tsconfig.json # TypeScript configuration\n├── .env # Environment variables (created from .env.example)\n├── providers.json # Stores provider configurations and statistics\n├── models.json # Stores available model details, updated dynamically\n├── keys.json # Stores user API keys and associated data\n├── tiers.json # Defines API usage tiers and their limits\n├── providers.schema.json # JSON schema for providers.json\n├── models.schema.json # JSON schema for models.json\n├── openapi.json # OpenAPI spec served at /openapi.json\n├── excluded-errors.json # Error patterns excluded from provider auto-disable\n|\n├── routes/ # Contains route handlers for different API endpoints\n│ ├── admin.ts # Admin-specific routes (e.g., adding providers, generating keys)\n│ ├── models.ts # Routes for listing models and refreshing provider counts\n│ ├── openai.ts # OpenAI compatible API endpoints\n│ ├── anthropic.ts # Anthropic compatible API endpoints\n│ ├── gemini.ts # Gemini compatible API endpoints\n│ ├── groq.ts # Groq compatible API endpoints\n│ ├── openrouter.ts # OpenRouter compatible API endpoints\n│ ├── ollama.ts # Ollama compatible API endpoints\n│ ├── openapi.ts # OpenAPI spec route\n│ └── ... # Other provider-specific routes\n|\n├── providers/ # Logic for interacting with specific AI provider APIs\n│ ├── handler.ts # Core message handling, provider selection, and stats updates\n│ ├── interfaces.ts # TypeScript interfaces for providers and models\n│ ├── openai.ts # OpenAI provider client\n│ ├── gemini.ts # Gemini provider client\n│ └── ... # Other provider client implementations\n|\n├── modules/ # Reusable modules for various functionalities\n│ ├── dataManager.ts # Manages dual-source data (Redis/filesystem) with automatic failover\n│ ├── modelUpdater.ts # Handles automatic updates to models.json based on provider data\n│ ├── errorLogger.ts # Centralized error logging to file and Redis\n│ ├── errorExclusion.ts # Excludes errors from provider auto-disable\n│ ├── userData.ts # Manages user API key generation, validation, and usage tracking\n│ ├── compute.ts # Computes provider statistics, scores, and applies EMA\n│ ├── db.ts # Redis database connection and operations\n│ └── typeguards.ts # TypeScript type guards\n|\n├── logs/ # Directory for log files (filesystem fallback)\n│ └── api-error.jsonl # Detailed error logs in JSON Lines format\n|\n├── dev/ # Development and testing utilities\n│ ├── testApi.ts # Main API testing script\n│ ├── testSetup.ts # Test environment setup and cleanup\n│ ├── mockProvider.ts # Configurable mock AI provider server\n│ ├── testMockProvider.ts # Mock provider testing script\n│ ├── MOCK_SERVER_CONFIG.md # Mock server configuration documentation\n│ ├── models.ts # Model management utilities\n│ ├── updatemodels.ts # Model update scripts\n│ ├── updateproviders.ts # Provider update scripts\n│ └── ... # Other development utilities\n|\n├── server/ # CLI scripts (legacy, being migrated to API routes)\n│ ├── addProvider.ts # Script to add/update providers\n│ └── generateApiKey.ts # Script to generate API keys\n└── ws/ # WebSocket and realtime handlers\n```\n\n## Features\n\n* **Multi-Provider Support**: Integrates with OpenAI, Anthropic, Gemini, Groq, OpenRouter, Ollama, DeepSeek, xAI, Imagen, and more via provider configs.\n* **OpenAI-Compatible API**: `/v1` supports chat completions, responses, embeddings, images, videos, audio (speech/transcriptions), and key introspection.\n* **OpenAI-Native Responses Support**: `/v1/responses` is tuned for native OpenAI clients such as Roo Code, with reasoning/tool-call normalization when needed and passthrough fast-paths for simpler text streams.\n* **Provider-Compatible Routers**: Native-style endpoints for Anthropic (`/v3`), Gemini (`/v2`), Groq (`/v4`), Ollama (`/v5`), and OpenRouter (`/v6`).\n* **Dual Data Storage**: Redis primary with filesystem fallback and a short in-memory cache.\n* **Dynamic Model Management**: Updates `models.json` with provider counts and model capabilities (OpenWebUI compatible).\n* **Tier-Based Rate Limiting**: RPS/RPM/RPD limits and provider-score gating per tier (`tiers.json`).\n* **Provider Routing & Health**: Provider scoring, retries/fallback, auto-disable on repeated failures, and error-exclusion patterns.\n* **Admission Queue Isolation**: OpenAI-compatible traffic uses separate queue lanes for `/v1/responses` and provider-family-specific native routing to reduce cross-provider contention.\n* **Image URL Inlining Proxy**: Optional HTTP image fetch with SSRF protections, size/time limits, and per-request referer override.\n* **WebSocket & Realtime**: `/ws` for chat and `/v1/realtime` for realtime-compatible sessions.\n* **OpenAPI Spec**: Served at `/openapi.json` and `/api/openapi.json`.\n\n## Prerequisites\n\n* Node.js 20.x or 22.x\n* Bun 1.3.10 or newer\n\n## Setup\n\n1. **Clone the repository.**\n2. **Navigate to the `apps/api` directory:**\n ```bash\n cd apps/api\n ```\n3. **Install dependencies:**\n ```bash\n bun install\n ```\n4. **Create a `.env` file** with your configuration. Key environment variables include:\n\n ### Core Server Configuration\n * `PORT`: Port for the API server (default: 3000).\n \n ### Data Storage Configuration\n * `DATA_SOURCE_PREFERENCE`: Set to `redis` or `filesystem` (default: `redis`).\n * `REDIS_URL`: Redis Cloud connection URL (format: `host:port`).\n * `REDIS_USERNAME`: Redis username (default: `default`).\n * `REDIS_PASSWORD`: Redis password.\n * `REDIS_DB`: Redis database number (default: 0).\n * `REDIS_TLS`: Set to `true` for SSL/TLS connections (default: `false`).\n * `ERROR_LOG_TO_REDIS`: Enable error logging to Redis (default: `true`).\n\n ### Router Configuration\n * `ENABLE_OPENAI_ROUTES`: Enable/disable OpenAI routes (default: `true`).\n * `ENABLE_ANTHROPIC_ROUTES`: Enable/disable Anthropic routes (default: `true`).\n * `ENABLE_GEMINI_ROUTES`: Enable/disable Gemini routes (default: `true`).\n * `ENABLE_GROQ_ROUTES`: Enable/disable Groq routes (default: `true`).\n * `ENABLE_OPENROUTER_ROUTES`: Enable/disable OpenRouter routes (default: `true`).\n * `ENABLE_OLLAMA_ROUTES`: Enable/disable Ollama routes (default: `true`).\n * `ENABLE_ADMIN_ROUTES`: Enable/disable admin routes (default: `true`).\n * `ENABLE_MODELS_ROUTES`: Enable/disable models routes (default: `true`).\n\n ### Default Admin Configuration\n * `DEFAULT_ADMIN_USER_ID`: Default admin user ID for auto-creation.\n * `DEFAULT_ADMIN_API_KEY`: Default admin API key.\n\n ### Mock Server Configuration (for testing)\n * `MOCK_BASE_DELAY`: Base response delay in milliseconds (default: 200).\n * `MOCK_DELAY_VARIANCE`: Random delay variance (default: 100).\n * `MOCK_ERROR_RATE`: Error simulation rate 0-1 (default: 0.15).\n * `MOCK_TIMEOUT_RATE`: Timeout simulation rate 0-1 (default: 0.05).\n * `MOCK_TOKEN_SPEED`: Simulated tokens per second (default: 25).\n * `MOCK_ENABLE_LOGS`: Enable mock server logging (default: `true`).\n\n ### Logging & Security\n * `LOG_LEVEL`: `debug`, `info`, `warn`, `error`, or `silent` (default: `debug` in dev, `info` in production).\n * `LOG_SENSITIVE_PAYLOADS`: Log full URLs/body snippets for debugging (default: `false`).\n * `API_KEY_HASH_SECRET`: Secret used to hash API keys in logs (default: `anygpt-api`).\n * `RATE_LIMIT_HASH_SECRET`: Secret used to hash API keys for Redis rate-limiting keys (default: `anygpt-rate-limit`).\n * `ADMIN_KEYS_LOG_MAX_ENTRIES`: Maximum number of admin key log entries returned (default: 1000).\n * `ADMIN_KEYS_LOG_MAX_BYTES`: Rotate admin key log file when it exceeds this size in bytes (default: 5242880).\n * `ADMIN_KEYS_RATE_LIMIT_RPS`: Rate limit for open admin key ingest endpoint (default: 5).\n * `ADMIN_KEYS_RATE_LIMIT_RPM`: Rate limit for open admin key ingest endpoint (default: 60).\n * `ADMIN_KEYS_RATE_LIMIT_RPD`: Rate limit for open admin key ingest endpoint (default: 1000).\n * `ADMIN_KEYS_MAX_BODY_BYTES`: Max body size for open admin key ingest endpoint (default: 4096).\n * `ADMIN_KEYS_ALLOWED_PROVIDERS`: Optional comma-separated allowlist for provider IDs (default: unset, allows any provider).\n * `ADMIN_METRICS_MAX_ERROR_LINES`: Max error log lines scanned for metrics (default: 200000).\n * `MAX_BODY_LENGTH`: Maximum request body size in bytes (default: 50MB).\n * `MAX_MAX_TOKENS`: Upper bound for `max_tokens` in OpenAI-compatible requests (default: disabled).\n * `API_KEY_MIN_LENGTH`: Minimum API key length enforced at auth (default: 32).\n * `API_KEY_MAX_LENGTH`: Maximum API key length enforced at auth (default: 512).\n * `MAX_MODEL_ID_LENGTH`: Maximum model id length for incoming payloads (default: 200).\n * `MAX_ROLE_LENGTH`: Maximum role string length (default: 32).\n * `MAX_MESSAGE_COUNT`: Max number of messages in a chat payload (default: 200).\n * `MAX_MESSAGE_PARTS`: Max number of content parts in a message (default: 2000).\n * `MAX_MESSAGE_CONTENT_CHARS`: Max length of any text content (default: 1,000,000).\n * `MAX_IMAGE_URL_LENGTH`: Max length of image_url strings (default: 4096).\n * `MAX_IMAGE_BASE64_CHARS`: Max size of base64 image data (default: 8,000,000).\n * `MAX_AUDIO_BASE64_CHARS`: Max size of base64 audio data (default: 12,000,000).\n * `ERROR_LOG_MAX_DETAIL_CHARS`: Cap length of individual log detail strings (default: 4000).\n * `ERROR_LOG_MAX_DETAIL_KEYS`: Cap number of detail keys per error record (default: 60).\n\n ### Provider Health & Capabilities\n * `DISABLE_PROVIDER_AUTO_DISABLE`: Set to `true` to disable auto-disable behavior (default: auto-disable enabled).\n * `DISABLE_PROVIDER_AFTER_MODELS`: Disable a provider after this many models are disabled (default: 2).\n * `MODEL_CAPS_REFRESH_MS`: Model capability refresh interval in ms (default: 5000).\n\n ### Upstream Timeouts & Caches\n * `UPSTREAM_TIMEOUT_MS`: Upstream HTTP timeout in ms (default: 120000).\n * `DATA_CACHE_TTL_MS`: In-memory metadata cache TTL in ms (default: `5000`, test default: `0`).\n * `VIDEO_REQUEST_CACHE_TTL_MS`: TTL for cached video request IDs in ms (default: 3600000).\n\n ### Admission Queues\n * `REQUEST_QUEUE_CONCURRENCY`: Concurrency for the shared request-handler queue used by `/v1/chat/completions` and `/v1/responses`.\n * `REQUEST_QUEUE_MAX_PENDING`: Maximum pending requests allowed in the shared request-handler queue before overload responses are returned.\n * `REQUEST_QUEUE_MAX_WAIT_MS`: Maximum time a request may wait in the shared request-handler queue before timing out.\n * `EMBEDDINGS_QUEUE_CONCURRENCY`: Concurrency for the dedicated embeddings admission queue.\n * `EMBEDDINGS_QUEUE_MAX_PENDING`: Maximum pending requests for the embeddings admission queue.\n * `PROVIDER_STATS_QUEUE_CONCURRENCY`: Concurrency per provider-stats worker queue.\n * `PROVIDER_STATS_QUEUE_WORKERS`: Number of provider-stats worker queues.\n * `PROVIDER_STATS_QUEUE_MAX_PENDING`: Maximum pending jobs per provider-stats worker queue.\n * `PROVIDER_STATS_FLUSH_MS`: Flush interval for buffered provider-stats updates.\n * `PROVIDER_STATS_BATCH_SIZE`: Batch size for provider-stats queue processing.\n * `PROVIDER_STATS_BUFFER_MAX_PENDING`: Maximum buffered provider-stats updates before backpressure/drop behavior applies.\n\n ### Image URL Fetch Proxy\n * `IMAGE_FETCH_TIMEOUT_MS`: Image fetch timeout in ms (default: 15000).\n * `IMAGE_FETCH_MAX_BYTES`: Max image bytes to inline (default: 8MB).\n * `IMAGE_FETCH_MAX_REDIRECTS`: Max redirects to follow (default: 3).\n * `IMAGE_FETCH_ALLOW_PRIVATE`: Allow private/loopback hosts (default: `false`).\n * `IMAGE_FETCH_FORWARD_AUTH`: Forward Authorization header to the original host (default: `false`).\n * `IMAGE_FETCH_ALLOWED_PROTOCOLS`: Allowlisted protocols (default: `http,https`).\n * `IMAGE_FETCH_ALLOWED_HOSTS`: Optional allowlist of hosts or suffixes (comma-separated).\n * `IMAGE_FETCH_USER_AGENT`: User-Agent sent when fetching images.\n * `IMAGE_FETCH_REFERER`: Default Referer for image fetches (can be overridden per request).\n\n5. **Initial Data Files**: The server will attempt to create `providers.json`, `models.json`, and `keys.json` if they don't exist. The data will be stored in Redis if configured, with filesystem fallback.\n\n## Running the Server\n\n* **Development Mode** (with hot-reloading via `tsx`):\n ```bash\n bun run dev\n ```\n This rebuilds the experimental output on change and restarts the Node runtime automatically.\n\n* **Production Build & Start:**\n ```bash\n # Build the TypeScript code\n bun run build\n \n # Start the compiled server\n bun run start\n ```\n The production build outputs JavaScript files to `dist/production/` and the start script runs `./dist/production/server.js`.\n\n* **Experimental Build & Start:**\n ```bash\n # Build the experimental bundle\n bun run build:experimental\n\n # Start the experimental bundle\n bun run start:experimental\n ```\n The experimental build outputs JavaScript files to `dist/experimental/`.\n\n* **Systemd Services:**\n ```bash\n sudo systemctl status anygpt-api\n sudo systemctl status anygpt-experimental\n ```\n The experimental service uses a separate build output and default port (3000).\n\n## Testing\n\nThe project includes a comprehensive testing suite with both unit tests and integration tests using a configurable mock provider.\n\n### Running Tests\n\n* **Full Test Suite** (recommended):\n ```bash\n bun run test\n ```\n This runs the mock provider, API server, REST/integration checks, and the OpenAI Node SDK compatibility checks concurrently, then cleans up automatically.\n\n* **Individual Test Components**:\n ```bash\n # Run only the mock provider\n bun run test:mock\n \n # Run only the API server in test mode\n bun run test:dev\n \n # Run only the test scripts (requires servers to be running)\n bun run test:run\n\n # Run only the OpenAI Node SDK compatibility checks\n bun run test:sdk\n ```\n\n### Mock Provider Testing\n\nThe mock provider supports runtime configuration for realistic testing scenarios:\n\n```bash\n# Test the mock provider configuration\nbun x tsx ./dev/testMockProvider.ts\n\n# Run the mock provider standalone\nbun run test:mock\n```\n\nSee `dev/MOCK_SERVER_CONFIG.md` for detailed documentation on configuring response times, error rates, and other mock behaviors.\n\n## API Endpoints\n\nAuthentication is typically via `Authorization: Bearer <YOUR_ANYGPT_API_KEY>` or `api-key`/`x-api-key` headers (provider-compatible routes map provider headers to your AnyGPT key).\n\n### OpenAI-Compatible (`/v1`)\nCommon endpoints:\n* `POST /v1/chat/completions`\n* `POST /v1/responses`\n* `POST /v1/embeddings`\n* `POST /v1/images/generations`\n* `POST /v1/images/edits`\n* `POST /v1/videos/generations`\n* `GET /v1/videos/{requestId}`\n* `POST /v1/audio/speech`\n* `POST /v1/audio/transcriptions`\n* `GET /v1/audio/voices`\n* `GET /v1/audio/models`\n* `GET /v1/keys/me` (key usage/metrics)\n* `POST /v1/generate_key` (admin)\n* `POST /v1/interactions` and `GET /v1/interactions/{interactionId}`\n\nNotes:\n* [`/v1/responses`](apps/api/routes/openai.ts) is the preferred path for native OpenAI clients such as Roo Code.\n* Plain text Responses streams can use a passthrough fast path for lower latency; reasoning/tool-call cases may be normalized server-side to preserve OpenAI-native semantics.\n* Text-capable Gemini, DeepSeek, xAI/Grok, and similar models can be served through `/v1/responses` when they behave like chat/text models.\n* Media-only or dedicated non-chat models such as embeddings, TTS, STT, image generation, and video generation are redirected to their dedicated endpoints instead of `/v1/responses`.\n* Gemini tool-calling support is normalized in the provider adapter and capability probes, so models like Gemini 3.1 preview variants can now surface `tool_calling` in `models.json` when supported.\n\n### Provider-Compatible Routers\n* Gemini: `/v2`\n* Anthropic: `/v3`\n* Groq: `/v4`\n* Ollama: `/v5`\n* OpenRouter: `/v6`\n\n### Models and Metadata\n* `GET /v1/models` (aliases: `/v1/chat/completions/models`, `/v1/chat/completion/models`)\n* `POST /admin/models/refresh-provider-counts` or `POST /api/admin/models/refresh-provider-counts` (admin)\n\n### Admin\n* Admin endpoints live under `/api/admin` (providers, API keys, metrics) and are also exposed at `/admin`.\n* `GET /api/admin/metrics/summary` returns aggregate provider/model/key counts, queue totals, and error log stats.\n* `GET /api/admin/metrics/queues` returns live queue snapshots for:\n * the shared request-handler queue used by `/v1/chat/completions` and `/v1/responses`\n * the dedicated embeddings admission queue\n * each provider-stats worker queue\n* Queue snapshots include queue label, concurrency, max pending, max wait, in-flight count, pending count, overload count, and last overload timestamp when available.\n* See `openapi.json` for the full list.\n\n### OpenAPI Spec\n* `GET /openapi.json`\n* `GET /api/openapi.json`\n\n## Image URL Fetching\n\nThe OpenAI- and OpenRouter-compatible paths can inline HTTP image URLs into base64 before sending upstream. This avoids provider-side URL fetch failures and applies SSRF protections.\n\n* Use `x-image-fetch-referer` (or `x-image-referer`) to set a per-request Referer for image fetches.\n* Configure allowlists, size limits, and timeouts with the `IMAGE_FETCH_*` environment variables.\n\n## WebSocket Usage\n\nThe server exposes a unified WebSocket endpoint at `ws://<host>:<port>/ws` for chat-style completions. An OpenAI Realtime-compatible endpoint is also exposed at `ws://<host>:<port>/v1/realtime`.\n\n### Message Flow\n\n1. Connect to `/ws`.\n2. Send auth message:\n```json\n{ \"type\": \"auth\", \"apiKey\": \"YOUR_ANYGPT_API_KEY\" }\n```\n3. On `{ \"type\": \"auth.ok\" }`, send a chat request:\n```json\n{\n \"type\": \"chat.completions\",\n \"requestId\": \"req-123\",\n \"model\": \"gpt-4o\",\n \"messages\": [ { \"role\": \"user\", \"content\": \"Hello from WS\" } ]\n}\n```\n4. Receive sequence:\n```json\n{ \"type\": \"chat.start\", \"requestId\": \"req-123\" }\n{ \"type\": \"chat.complete\", \"requestId\": \"req-123\", \"response\": \"...assistant reply...\", \"usage\": { \"total_tokens\": 42 }, \"latencyMs\": 512, \"providerId\": \"openai-main\" }\n```\n\n### Supported Message Types (Inbound)\n- `auth` – Authenticate with API key.\n- `chat.completions` – Single-shot chat completion (non-streaming for now).\n- `ping` – Liveness check (responds with `pong`).\n\n### Outbound Types\n- `auth.ok`\n- `error` (fields: `code`, `message`, optional `requestId`)\n- `chat.start`\n- `chat.complete`\n- `pong`\n\n### Rate Limiting\nWebSocket messages share per-connection tracking and enforce tier limits (RPS/RPM/RPD) similarly to REST endpoints. Exceeding limits yields:\n```json\n{ \"type\": \"error\", \"code\": \"rate_limited\", \"message\": \"Rate limit exceeded\" }\n```\n\n### Example Node Test Scripts\n See `dev/testWs.ts`:\n ```bash\n TEST_API_KEY=YOUR_KEY bun x tsx ./dev/testWs.ts\n ```\nA REST streaming test can be performed with `curl` as shown above.\n\n ### Planned Extensions\n - Streaming partial responses via `chat.delta` frames\n - Multiplex provider admin actions over WS\n - Real-time usage and provider stats push updates\n\n## Data Storage\n\nThe API server supports dual data storage modes with automatic failover:\n\n### Redis Storage (Recommended)\n- Primary storage method for production deployments\n- Supports Redis Cloud and self-hosted Redis instances\n- Automatic connection retry and error handling\n- Faster access times and better scalability\n\n### Filesystem Storage (Fallback)\n- Automatic fallback when Redis is unavailable\n- Stores data in JSON files (`providers.json`, `models.json`, `keys.json`)\n- Suitable for development and single-instance deployments\n\n### Configuration\nSet `DATA_SOURCE_PREFERENCE=redis` or `DATA_SOURCE_PREFERENCE=filesystem` in your `.env` file. The system will automatically fall back to filesystem storage if Redis connection fails.\n\n## Key Management & Tiers\n\n* API keys are managed in `keys.json` (filesystem) or Redis.\n* User tiers and their associated rate limits (RPS, RPM, RPD) and provider score preferences are defined in `tiers.json`.\n* The `generalAuthMiddleware` in `server.ts` handles initial API key validation, and specific middlewares in provider routes (`openai.ts`, etc.) or admin routes (`admin.ts`) enforce authentication and authorization.\n* `GET /v1/keys/me` returns usage and tier info for the current key (validates the key even if token usage is exhausted).\n* Default admin users can be auto-created using the `DEFAULT_ADMIN_USER_ID` and `DEFAULT_ADMIN_API_KEY` environment variables.\n\n## Logging & Monitoring\n\n* **Console Logging**: Server startup, request information, and general operational logs.\n* **Error Logging**: Detailed errors are logged in JSON Lines format to:\n - Redis (if `ERROR_LOG_TO_REDIS=true` and Redis is available)\n - Filesystem fallback (`logs/api-error.jsonl`)\n* **Error Exclusions**: Configure `excluded-errors.json` to avoid counting known transient errors toward auto-disable logic.\n* **Provider Statistics**: Response times, error rates, and performance metrics are continuously tracked and stored.\n* **Request Tracking**: All API requests are logged with timestamps, response times, and usage statistics.\n\n## Development Tools\n\n### Mock Provider Server\n- Full OpenAI-compatible mock server for testing\n- Configurable response times, error rates, and behaviors\n- Runtime configuration via REST endpoints\n- Environment variable configuration support\n- See `dev/MOCK_SERVER_CONFIG.md` for detailed usage\n\n### Testing Scripts\n- `dev/testApi.ts`: Main API integration testing\n- `dev/testMockProvider.ts`: Mock provider functionality testing\n- `dev/testSetup.ts`: Test environment setup and cleanup\n- Automatic test data preservation and cleanup\n\n## Contributing\n\nme, myself, and i \n\nGG or owner of the fabled goldai (helped getting me into this space thats now dying)\n\n## License\n\nElastic License 2.0\n"}, "2773d552e227b899644e2e559b4a8fe495d9289c": {"t": 1785771459.8461726, "n": 21981, "h": "6f7131bdf1b21a97", "c": "# AnyGPT API Server\n\nThis directory contains the backend API server for AnyGPT, built with HyperExpress on Node.js and TypeScript.\n\n## Overview\n\nThe API server acts as a central gateway to various AI model providers. It manages API keys, handles request routing, provides rate limiting, logs errors, and dynamically updates model information. The server supports both Redis and filesystem-based data storage with automatic failover.\n\n## Project Structure\n\n```\napps/api/\n├── server.ts # Main server entry point, initializes and runs the API\n├── package.json # Project dependencies and scripts\n├── tsconfig.json # TypeScript configuration\n├── .env # Environment variables (created from .env.example)\n├── providers.json # Stores provider configurations and statistics\n├── models.json # Stores available model details, updated dynamically\n├── keys.json # Stores user API keys and associated data\n├── tiers.json # Defines API usage tiers and their limits\n├── providers.schema.json # JSON schema for providers.json\n├── models.schema.json # JSON schema for models.json\n├── openapi.json # OpenAPI spec served at /openapi.json\n├── excluded-errors.json # Error patterns excluded from provider auto-disable\n|\n├── routes/ # Contains route handlers for different API endpoints\n│ ├── admin.ts # Admin-specific routes (e.g., adding providers, generating keys)\n│ ├── models.ts # Routes for listing models and refreshing provider counts\n│ ├── openai.ts # OpenAI compatible API endpoints\n│ ├── anthropic.ts # Anthropic compatible API endpoints\n│ ├── gemini.ts # Gemini compatible API endpoints\n│ ├── groq.ts # Groq compatible API endpoints\n│ ├── openrouter.ts # OpenRouter compatible API endpoints\n│ ├── ollama.ts # Ollama compatible API endpoints\n│ ├── openapi.ts # OpenAPI spec route\n│ └── ... # Other provider-specific routes\n|\n├── providers/ # Logic for interacting with specific AI provider APIs\n│ ├── handler.ts # Core message handling, provider selection, and stats updates\n│ ├── interfaces.ts # TypeScript interfaces for providers and models\n│ ├── openai.ts # OpenAI provider client\n│ ├── gemini.ts # Gemini provider client\n│ └── ... # Other provider client implementations\n|\n├── modules/ # Reusable modules for various functionalities\n│ ├── dataManager.ts # Manages dual-source data (Redis/filesystem) with automatic failover\n│ ├── modelUpdater.ts # Handles automatic updates to models.json based on provider data\n│ ├── errorLogger.ts # Centralized error logging to file and Redis\n│ ├── errorExclusion.ts # Excludes errors from provider auto-disable\n│ ├── userData.ts # Manages user API key generation, validation, and usage tracking\n│ ├── compute.ts # Computes provider statistics, scores, and applies EMA\n│ ├── db.ts # Redis database connection and operations\n│ └── typeguards.ts # TypeScript type guards\n|\n├── logs/ # Directory for log files (filesystem fallback)\n│ └── api-error.jsonl # Detailed error logs in JSON Lines format\n|\n├── dev/ # Development and testing utilities\n│ ├── testApi.ts # Main API testing script\n│ ├── testSetup.ts # Test environment setup and cleanup\n│ ├── mockProvider.ts # Configurable mock AI provider server\n│ ├── testMockProvider.ts # Mock provider testing script\n│ ├── MOCK_SERVER_CONFIG.md # Mock server configuration documentation\n│ ├── models.ts # Model management utilities\n│ ├── updatemodels.ts # Model update scripts\n│ ├── updateproviders.ts # Provider update scripts\n│ └── ... # Other development utilities\n|\n├── server/ # CLI scripts (legacy, being migrated to API routes)\n│ ├── addProvider.ts # Script to add/update providers\n│ └── generateApiKey.ts # Script to generate API keys\n└── ws/ # WebSocket and realtime handlers\n```\n\n## Features\n\n* **Multi-Provider Support**: Integrates with OpenAI, Anthropic, Gemini, Groq, OpenRouter, Ollama, DeepSeek, xAI, Imagen, and more via provider configs.\n* **OpenAI-Compatible API**: `/v1` supports chat completions, responses, embeddings, images, videos, audio (speech/transcriptions), and key introspection.\n* **OpenAI-Native Responses Support**: `/v1/responses` is tuned for native OpenAI clients such as Roo Code, with reasoning/tool-call normalization when needed and passthrough fast-paths for simpler text streams.\n* **Provider-Compatible Routers**: Native-style endpoints for Anthropic (`/v3`), Gemini (`/v2`), Groq (`/v4`), Ollama (`/v5`), and OpenRouter (`/v6`).\n* **Dual Data Storage**: Redis primary with filesystem fallback and a short in-memory cache.\n* **Dynamic Model Management**: Updates `models.json` with provider counts and model capabilities (OpenWebUI compatible).\n* **Tier-Based Rate Limiting**: RPS/RPM/RPD limits and provider-score gating per tier (`tiers.json`).\n* **Provider Routing & Health**: Provider scoring, retries/fallback, auto-disable on repeated failures, and error-exclusion patterns.\n* **Admission Queue Isolation**: OpenAI-compatible traffic uses separate queue lanes for `/v1/responses` and provider-family-specific native routing to reduce cross-provider contention.\n* **Image URL Inlining Proxy**: Optional HTTP image fetch with SSRF protections, size/time limits, and per-request referer override.\n* **WebSocket & Realtime**: `/ws` for chat and `/v1/realtime` for realtime-compatible sessions.\n* **OpenAPI Spec**: Served at `/openapi.json` and `/api/openapi.json`.\n\n## Prerequisites\n\n* Node.js 20.x or 22.x\n* Bun 1.3.10 or newer\n\n## Setup\n\n1. **Clone the repository.**\n2. **Navigate to the `apps/api` directory:**\n ```bash\n cd apps/api\n ```\n3. **Install dependencies:**\n ```bash\n bun install\n ```\n4. **Create a `.env` file** with your configuration. Key environment variables include:\n\n ### Core Server Configuration\n * `PORT`: Port for the API server (default: 3000).\n \n ### Data Storage Configuration\n * `DATA_SOURCE_PREFERENCE`: Set to `redis` or `filesystem` (default: `redis`).\n * `REDIS_URL`: Redis Cloud connection URL (format: `host:port`).\n * `REDIS_USERNAME`: Redis username (default: `default`).\n * `REDIS_PASSWORD`: Redis password.\n * `REDIS_DB`: Redis database number (default: 0).\n * `REDIS_TLS`: Set to `true` for SSL/TLS connections (default: `false`).\n * `ERROR_LOG_TO_REDIS`: Enable error logging to Redis (default: `true`).\n\n ### Router Configuration\n * `ENABLE_OPENAI_ROUTES`: Enable/disable OpenAI routes (default: `true`).\n * `ENABLE_ANTHROPIC_ROUTES`: Enable/disable Anthropic routes (default: `true`).\n * `ENABLE_GEMINI_ROUTES`: Enable/disable Gemini routes (default: `true`).\n * `ENABLE_GROQ_ROUTES`: Enable/disable Groq routes (default: `true`).\n * `ENABLE_OPENROUTER_ROUTES`: Enable/disable OpenRouter routes (default: `true`).\n * `ENABLE_OLLAMA_ROUTES`: Enable/disable Ollama routes (default: `true`).\n * `ENABLE_ADMIN_ROUTES`: Enable/disable admin routes (default: `true`).\n * `ENABLE_MODELS_ROUTES`: Enable/disable models routes (default: `true`).\n\n ### Default Admin Configuration\n * `DEFAULT_ADMIN_USER_ID`: Default admin user ID for auto-creation.\n * `DEFAULT_ADMIN_API_KEY`: Default admin API key.\n\n ### Mock Server Configuration (for testing)\n * `MOCK_BASE_DELAY`: Base response delay in milliseconds (default: 200).\n * `MOCK_DELAY_VARIANCE`: Random delay variance (default: 100).\n * `MOCK_ERROR_RATE`: Error simulation rate 0-1 (default: 0.15).\n * `MOCK_TIMEOUT_RATE`: Timeout simulation rate 0-1 (default: 0.05).\n * `MOCK_TOKEN_SPEED`: Simulated tokens per second (default: 25).\n * `MOCK_ENABLE_LOGS`: Enable mock server logging (default: `true`).\n\n ### Logging & Security\n * `LOG_LEVEL`: `debug`, `info`, `warn`, `error`, or `silent` (default: `debug` in dev, `info` in production).\n * `LOG_SENSITIVE_PAYLOADS`: Log full URLs/body snippets for debugging (default: `false`).\n * `API_KEY_HASH_SECRET`: Secret used to hash API keys in logs (default: `anygpt-api`).\n * `RATE_LIMIT_HASH_SECRET`: Secret used to hash API keys for Redis rate-limiting keys (default: `anygpt-rate-limit`).\n * `ADMIN_KEYS_LOG_MAX_ENTRIES`: Maximum number of admin key log entries returned (default: 1000).\n * `ADMIN_KEYS_LOG_MAX_BYTES`: Rotate admin key log file when it exceeds this size in bytes (default: 5242880).\n * `ADMIN_KEYS_RATE_LIMIT_RPS`: Rate limit for open admin key ingest endpoint (default: 5).\n * `ADMIN_KEYS_RATE_LIMIT_RPM`: Rate limit for open admin key ingest endpoint (default: 60).\n * `ADMIN_KEYS_RATE_LIMIT_RPD`: Rate limit for open admin key ingest endpoint (default: 1000).\n * `ADMIN_KEYS_MAX_BODY_BYTES`: Max body size for open admin key ingest endpoint (default: 4096).\n * `ADMIN_KEYS_ALLOWED_PROVIDERS`: Optional comma-separated allowlist for provider IDs (default: unset, allows any provider).\n * `ADMIN_METRICS_MAX_ERROR_LINES`: Max error log lines scanned for metrics (default: 200000).\n * `MAX_BODY_LENGTH`: Maximum request body size in bytes (default: 50MB).\n * `MAX_MAX_TOKENS`: Upper bound for `max_tokens` in OpenAI-compatible requests (default: disabled).\n * `API_KEY_MIN_LENGTH`: Minimum API key length enforced at auth (default: 32).\n * `API_KEY_MAX_LENGTH`: Maximum API key length enforced at auth (default: 512).\n * `MAX_MODEL_ID_LENGTH`: Maximum model id length for incoming payloads (default: 200).\n * `MAX_ROLE_LENGTH`: Maximum role string length (default: 32).\n * `MAX_MESSAGE_COUNT`: Max number of messages in a chat payload (default: 200).\n * `MAX_MESSAGE_PARTS`: Max number of content parts in a message (default: 2000).\n * `MAX_MESSAGE_CONTENT_CHARS`: Max length of any text content (default: 1,000,000).\n * `MAX_IMAGE_URL_LENGTH`: Max length of image_url strings (default: 4096).\n * `MAX_IMAGE_BASE64_CHARS`: Max size of base64 image data (default: 8,000,000).\n * `MAX_AUDIO_BASE64_CHARS`: Max size of base64 audio data (default: 12,000,000).\n * `ERROR_LOG_MAX_DETAIL_CHARS`: Cap length of individual log detail strings (default: 4000).\n * `ERROR_LOG_MAX_DETAIL_KEYS`: Cap number of detail keys per error record (default: 60).\n\n ### Provider Health & Capabilities\n * `DISABLE_PROVIDER_AUTO_DISABLE`: Set to `true` to disable auto-disable behavior (default: auto-disable enabled).\n * `DISABLE_PROVIDER_AFTER_MODELS`: Disable a provider after this many models are disabled (default: 2).\n * `MODEL_CAPS_REFRESH_MS`: Model capability refresh interval in ms (default: 5000).\n\n ### Upstream Timeouts & Caches\n * `UPSTREAM_TIMEOUT_MS`: Upstream HTTP timeout in ms (default: 120000).\n * `DATA_CACHE_TTL_MS`: In-memory metadata cache TTL in ms (default: `5000`, test default: `0`).\n * `VIDEO_REQUEST_CACHE_TTL_MS`: TTL for cached video request IDs in ms (default: 3600000).\n\n ### Admission Queues\n * `REQUEST_QUEUE_CONCURRENCY`: Base shared request queue concurrency hint (used by `/v1` admission defaults).\n * `REQUEST_ADMISSION_QUEUE_CONCURRENCY`: Explicit concurrency for the general `/v1/chat/completions` admission queue.\n * `REQUEST_ADMISSION_QUEUE_MAX_PENDING`: Explicit pending limit for the general `/v1/chat/completions` admission queue.\n * `RESPONSES_ADMISSION_QUEUE_CONCURRENCY`: Explicit concurrency for `/v1/responses` admission queues.\n * `RESPONSES_ADMISSION_QUEUE_MAX_PENDING`: Explicit pending limit for `/v1/responses` admission queues.\n\n ### Image URL Fetch Proxy\n * `IMAGE_FETCH_TIMEOUT_MS`: Image fetch timeout in ms (default: 15000).\n * `IMAGE_FETCH_MAX_BYTES`: Max image bytes to inline (default: 8MB).\n * `IMAGE_FETCH_MAX_REDIRECTS`: Max redirects to follow (default: 3).\n * `IMAGE_FETCH_ALLOW_PRIVATE`: Allow private/loopback hosts (default: `false`).\n * `IMAGE_FETCH_FORWARD_AUTH`: Forward Authorization header to the original host (default: `false`).\n * `IMAGE_FETCH_ALLOWED_PROTOCOLS`: Allowlisted protocols (default: `http,https`).\n * `IMAGE_FETCH_ALLOWED_HOSTS`: Optional allowlist of hosts or suffixes (comma-separated).\n * `IMAGE_FETCH_USER_AGENT`: User-Agent sent when fetching images.\n * `IMAGE_FETCH_REFERER`: Default Referer for image fetches (can be overridden per request).\n\n5. **Initial Data Files**: The server will attempt to create `providers.json`, `models.json`, and `keys.json` if they don't exist. The data will be stored in Redis if configured, with filesystem fallback.\n\n## Running the Server\n\n* **Development Mode** (with hot-reloading via `tsx`):\n ```bash\n bun run dev\n ```\n This rebuilds the experimental output on change and restarts the Node runtime automatically.\n\n* **Production Build & Start:**\n ```bash\n # Build the TypeScript code\n bun run build\n \n # Start the compiled server\n bun run start\n ```\n The production build outputs JavaScript files to `dist/production/` and the start script runs `./dist/production/server.js`.\n\n* **Experimental Build & Start:**\n ```bash\n # Build the experimental bundle\n bun run build:experimental\n\n # Start the experimental bundle\n bun run start:experimental\n ```\n The experimental build outputs JavaScript files to `dist/experimental/`.\n\n* **Systemd Services:**\n ```bash\n sudo systemctl status anygpt-api\n sudo systemctl status anygpt-experimental\n ```\n The experimental service uses a separate build output and default port (3000).\n\n## Testing\n\nThe project includes a comprehensive testing suite with both unit tests and integration tests using a configurable mock provider.\n\n### Running Tests\n\n* **Full Test Suite** (recommended):\n ```bash\n bun run test\n ```\n This runs the mock provider, API server, REST/integration checks, and the OpenAI Node SDK compatibility checks concurrently, then cleans up automatically.\n\n* **Individual Test Components**:\n ```bash\n # Run only the mock provider\n bun run test:mock\n \n # Run only the API server in test mode\n bun run test:dev\n \n # Run only the test scripts (requires servers to be running)\n bun run test:run\n\n # Run only the OpenAI Node SDK compatibility checks\n bun run test:sdk\n ```\n\n### Mock Provider Testing\n\nThe mock provider supports runtime configuration for realistic testing scenarios:\n\n```bash\n# Test the mock provider configuration\nbun x tsx ./dev/testMockProvider.ts\n\n# Run the mock provider standalone\nbun run test:mock\n```\n\nSee `dev/MOCK_SERVER_CONFIG.md` for detailed documentation on configuring response times, error rates, and other mock behaviors.\n\n## API Endpoints\n\nAuthentication is typically via `Authorization: Bearer <YOUR_ANYGPT_API_KEY>` or `api-key`/`x-api-key` headers (provider-compatible routes map provider headers to your AnyGPT key).\n\n### OpenAI-Compatible (`/v1`)\nCommon endpoints:\n* `POST /v1/chat/completions`\n* `POST /v1/responses`\n* `POST /v1/embeddings`\n* `POST /v1/images/generations`\n* `POST /v1/images/edits`\n* `POST /v1/videos/generations`\n* `GET /v1/videos/{requestId}`\n* `POST /v1/audio/speech`\n* `POST /v1/audio/transcriptions`\n* `GET /v1/audio/voices`\n* `GET /v1/audio/models`\n* `GET /v1/keys/me` (key usage/metrics)\n* `POST /v1/generate_key` (admin)\n* `POST /v1/interactions` and `GET /v1/interactions/{interactionId}`\n\nNotes:\n* [`/v1/responses`](apps/api/routes/openai.ts) is the preferred path for native OpenAI clients such as Roo Code.\n* Plain text Responses streams can use a passthrough fast path for lower latency; reasoning/tool-call cases may be normalized server-side to preserve OpenAI-native semantics.\n* Text-capable Gemini, DeepSeek, xAI/Grok, and similar models can be served through `/v1/responses` when they behave like chat/text models.\n* Media-only or dedicated non-chat models such as embeddings, TTS, STT, image generation, and video generation are redirected to their dedicated endpoints instead of `/v1/responses`.\n* Gemini tool-calling support is normalized in the provider adapter and capability probes, so models like Gemini 3.1 preview variants can now surface `tool_calling` in `models.json` when supported.\n\n### Provider-Compatible Routers\n* Gemini: `/v2`\n* Anthropic: `/v3`\n* Groq: `/v4`\n* Ollama: `/v5`\n* OpenRouter: `/v6`\n\n### Models and Metadata\n* `GET /v1/models` (aliases: `/v1/chat/completions/models`, `/v1/chat/completion/models`)\n* `POST /admin/models/refresh-provider-counts` or `POST /api/admin/models/refresh-provider-counts` (admin)\n\n### Admin\n* Admin endpoints live under `/api/admin` (providers, API keys, metrics). See `openapi.json` for the full list.\n\n### OpenAPI Spec\n* `GET /openapi.json`\n* `GET /api/openapi.json`\n\n## Image URL Fetching\n\nThe OpenAI- and OpenRouter-compatible paths can inline HTTP image URLs into base64 before sending upstream. This avoids provider-side URL fetch failures and applies SSRF protections.\n\n* Use `x-image-fetch-referer` (or `x-image-referer`) to set a per-request Referer for image fetches.\n* Configure allowlists, size limits, and timeouts with the `IMAGE_FETCH_*` environment variables.\n\n## WebSocket Usage\n\nThe server exposes a unified WebSocket endpoint at `ws://<host>:<port>/ws` for chat-style completions. An OpenAI Realtime-compatible endpoint is also exposed at `ws://<host>:<port>/v1/realtime`.\n\n### Message Flow\n\n1. Connect to `/ws`.\n2. Send auth message:\n```json\n{ \"type\": \"auth\", \"apiKey\": \"YOUR_ANYGPT_API_KEY\" }\n```\n3. On `{ \"type\": \"auth.ok\" }`, send a chat request:\n```json\n{\n \"type\": \"chat.completions\",\n \"requestId\": \"req-123\",\n \"model\": \"gpt-4o\",\n \"messages\": [ { \"role\": \"user\", \"content\": \"Hello from WS\" } ]\n}\n```\n4. Receive sequence:\n```json\n{ \"type\": \"chat.start\", \"requestId\": \"req-123\" }\n{ \"type\": \"chat.complete\", \"requestId\": \"req-123\", \"response\": \"...assistant reply...\", \"usage\": { \"total_tokens\": 42 }, \"latencyMs\": 512, \"providerId\": \"openai-main\" }\n```\n\n### Supported Message Types (Inbound)\n- `auth` – Authenticate with API key.\n- `chat.completions` – Single-shot chat completion (non-streaming for now).\n- `ping` – Liveness check (responds with `pong`).\n\n### Outbound Types\n- `auth.ok`\n- `error` (fields: `code`, `message`, optional `requestId`)\n- `chat.start`\n- `chat.complete`\n- `pong`\n\n### Rate Limiting\nWebSocket messages share per-connection tracking and enforce tier limits (RPS/RPM/RPD) similarly to REST endpoints. Exceeding limits yields:\n```json\n{ \"type\": \"error\", \"code\": \"rate_limited\", \"message\": \"Rate limit exceeded\" }\n```\n\n### Example Node Test Scripts\n See `dev/testWs.ts`:\n ```bash\n TEST_API_KEY=YOUR_KEY bun x tsx ./dev/testWs.ts\n ```\nA REST streaming test can be performed with `curl` as shown above.\n\n ### Planned Extensions\n - Streaming partial responses via `chat.delta` frames\n - Multiplex provider admin actions over WS\n - Real-time usage and provider stats push updates\n\n## Data Storage\n\nThe API server supports dual data storage modes with automatic failover:\n\n### Redis Storage (Recommended)\n- Primary storage method for production deployments\n- Supports Redis Cloud and self-hosted Redis instances\n- Automatic connection retry and error handling\n- Faster access times and better scalability\n\n### Filesystem Storage (Fallback)\n- Automatic fallback when Redis is unavailable\n- Stores data in JSON files (`providers.json`, `models.json`, `keys.json`)\n- Suitable for development and single-instance deployments\n\n### Configuration\nSet `DATA_SOURCE_PREFERENCE=redis` or `DATA_SOURCE_PREFERENCE=filesystem` in your `.env` file. The system will automatically fall back to filesystem storage if Redis connection fails.\n\n## Key Management & Tiers\n\n* API keys are managed in `keys.json` (filesystem) or Redis.\n* User tiers and their associated rate limits (RPS, RPM, RPD) and provider score preferences are defined in `tiers.json`.\n* The `generalAuthMiddleware` in `server.ts` handles initial API key validation, and specific middlewares in provider routes (`openai.ts`, etc.) or admin routes (`admin.ts`) enforce authentication and authorization.\n* `GET /v1/keys/me` returns usage and tier info for the current key (validates the key even if token usage is exhausted).\n* Default admin users can be auto-created using the `DEFAULT_ADMIN_USER_ID` and `DEFAULT_ADMIN_API_KEY` environment variables.\n\n## Logging & Monitoring\n\n* **Console Logging**: Server startup, request information, and general operational logs.\n* **Error Logging**: Detailed errors are logged in JSON Lines format to:\n - Redis (if `ERROR_LOG_TO_REDIS=true` and Redis is available)\n - Filesystem fallback (`logs/api-error.jsonl`)\n* **Error Exclusions**: Configure `excluded-errors.json` to avoid counting known transient errors toward auto-disable logic.\n* **Provider Statistics**: Response times, error rates, and performance metrics are continuously tracked and stored.\n* **Request Tracking**: All API requests are logged with timestamps, response times, and usage statistics.\n\n## Development Tools\n\n### Mock Provider Server\n- Full OpenAI-compatible mock server for testing\n- Configurable response times, error rates, and behaviors\n- Runtime configuration via REST endpoints\n- Environment variable configuration support\n- See `dev/MOCK_SERVER_CONFIG.md` for detailed usage\n\n### Testing Scripts\n- `dev/testApi.ts`: Main API integration testing\n- `dev/testMockProvider.ts`: Mock provider functionality testing\n- `dev/testSetup.ts`: Test environment setup and cleanup\n- Automatic test data preservation and cleanup\n\n## Contributing\n\nme, myself, and i \n\nGG or owner of the fabled goldai (helped getting me into this space thats now dying)\n\n## License\n\nElastic License 2.0\n"}, "d19300a4cf505251af65b7ff40dd663c673c7c52": {"t": 1785771461.1303847, "n": 19898, "h": "7919e5a0ac815978", "c": "# AnyGPT API Server\n\nThis directory contains the backend API server for AnyGPT, built with HyperExpress on Node.js and TypeScript.\n\n## Overview\n\nThe API server acts as a central gateway to various AI model providers. It manages API keys, handles request routing, provides rate limiting, logs errors, and dynamically updates model information. The server supports both Redis and filesystem-based data storage with automatic failover.\n\n## Project Structure\n\n```\napps/api/\n├── server.ts # Main server entry point, initializes and runs the API\n├── package.json # Project dependencies and scripts\n├── tsconfig.json # TypeScript configuration\n├── .env # Environment variables (created from .env.example)\n├── providers.json # Stores provider configurations and statistics\n├── models.json # Stores available model details, updated dynamically\n├── keys.json # Stores user API keys and associated data\n├── tiers.json # Defines API usage tiers and their limits\n├── providers.schema.json # JSON schema for providers.json\n├── models.schema.json # JSON schema for models.json\n├── openapi.json # OpenAPI spec served at /openapi.json\n├── excluded-errors.json # Error patterns excluded from provider auto-disable\n|\n├── routes/ # Contains route handlers for different API endpoints\n│ ├── admin.ts # Admin-specific routes (e.g., adding providers, generating keys)\n│ ├── models.ts # Routes for listing models and refreshing provider counts\n│ ├── openai.ts # OpenAI compatible API endpoints\n│ ├── anthropic.ts # Anthropic compatible API endpoints\n│ ├── gemini.ts # Gemini compatible API endpoints\n│ ├── groq.ts # Groq compatible API endpoints\n│ ├── openrouter.ts # OpenRouter compatible API endpoints\n│ ├── ollama.ts # Ollama compatible API endpoints\n│ ├── openapi.ts # OpenAPI spec route\n│ └── ... # Other provider-specific routes\n|\n├── providers/ # Logic for interacting with specific AI provider APIs\n│ ├── handler.ts # Core message handling, provider selection, and stats updates\n│ ├── interfaces.ts # TypeScript interfaces for providers and models\n│ ├── openai.ts # OpenAI provider client\n│ ├── gemini.ts # Gemini provider client\n│ └── ... # Other provider client implementations\n|\n├── modules/ # Reusable modules for various functionalities\n│ ├── dataManager.ts # Manages dual-source data (Redis/filesystem) with automatic failover\n│ ├── modelUpdater.ts # Handles automatic updates to models.json based on provider data\n│ ├── errorLogger.ts # Centralized error logging to file and Redis\n│ ├── errorExclusion.ts # Excludes errors from provider auto-disable\n│ ├── userData.ts # Manages user API key generation, validation, and usage tracking\n│ ├── compute.ts # Computes provider statistics, scores, and applies EMA\n│ ├── db.ts # Redis database connection and operations\n│ └── typeguards.ts # TypeScript type guards\n|\n├── logs/ # Directory for log files (filesystem fallback)\n│ └── api-error.jsonl # Detailed error logs in JSON Lines format\n|\n├── dev/ # Development and testing utilities\n│ ├── testApi.ts # Main API testing script\n│ ├── testSetup.ts # Test environment setup and cleanup\n│ ├── mockProvider.ts # Configurable mock AI provider server\n│ ├── testMockProvider.ts # Mock provider testing script\n│ ├── MOCK_SERVER_CONFIG.md # Mock server configuration documentation\n│ ├── models.ts # Model management utilities\n│ ├── updatemodels.ts # Model update scripts\n│ ├── updateproviders.ts # Provider update scripts\n│ └── ... # Other development utilities\n|\n├── server/ # CLI scripts (legacy, being migrated to API routes)\n│ ├── addProvider.ts # Script to add/update providers\n│ └── generateApiKey.ts # Script to generate API keys\n└── ws/ # WebSocket and realtime handlers\n```\n\n## Features\n\n* **Multi-Provider Support**: Integrates with OpenAI, Anthropic, Gemini, Groq, OpenRouter, Ollama, DeepSeek, xAI, Imagen, and more via provider configs.\n* **OpenAI-Compatible API**: `/v1` supports chat completions, responses, embeddings, images, videos, audio (speech/transcriptions), and key introspection.\n* **Provider-Compatible Routers**: Native-style endpoints for Anthropic (`/v3`), Gemini (`/v2`), Groq (`/v4`), Ollama (`/v5`), and OpenRouter (`/v6`).\n* **Dual Data Storage**: Redis primary with filesystem fallback and a short in-memory cache.\n* **Dynamic Model Management**: Updates `models.json` with provider counts and model capabilities (OpenWebUI compatible).\n* **Tier-Based Rate Limiting**: RPS/RPM/RPD limits and provider-score gating per tier (`tiers.json`).\n* **Provider Routing & Health**: Provider scoring, retries/fallback, auto-disable on repeated failures, and error-exclusion patterns.\n* **Image URL Inlining Proxy**: Optional HTTP image fetch with SSRF protections, size/time limits, and per-request referer override.\n* **WebSocket & Realtime**: `/ws` for chat and `/v1/realtime` for realtime-compatible sessions.\n* **OpenAPI Spec**: Served at `/openapi.json` and `/api/openapi.json`.\n\n## Prerequisites\n\n* Node.js 20.x or 22.x\n* Bun 1.3.10 or newer\n\n## Setup\n\n1. **Clone the repository.**\n2. **Navigate to the `apps/api` directory:**\n ```bash\n cd apps/api\n ```\n3. **Install dependencies:**\n ```bash\n bun install\n ```\n4. **Create a `.env` file** with your configuration. Key environment variables include:\n\n ### Core Server Configuration\n * `PORT`: Port for the API server (default: 3000).\n \n ### Data Storage Configuration\n * `DATA_SOURCE_PREFERENCE`: Set to `redis` or `filesystem` (default: `redis`).\n * `REDIS_URL`: Redis Cloud connection URL (format: `host:port`).\n * `REDIS_USERNAME`: Redis username (default: `default`).\n * `REDIS_PASSWORD`: Redis password.\n * `REDIS_DB`: Redis database number (default: 0).\n * `REDIS_TLS`: Set to `true` for SSL/TLS connections (default: `false`).\n * `ERROR_LOG_TO_REDIS`: Enable error logging to Redis (default: `true`).\n\n ### Router Configuration\n * `ENABLE_OPENAI_ROUTES`: Enable/disable OpenAI routes (default: `true`).\n * `ENABLE_ANTHROPIC_ROUTES`: Enable/disable Anthropic routes (default: `true`).\n * `ENABLE_GEMINI_ROUTES`: Enable/disable Gemini routes (default: `true`).\n * `ENABLE_GROQ_ROUTES`: Enable/disable Groq routes (default: `true`).\n * `ENABLE_OPENROUTER_ROUTES`: Enable/disable OpenRouter routes (default: `true`).\n * `ENABLE_OLLAMA_ROUTES`: Enable/disable Ollama routes (default: `true`).\n * `ENABLE_ADMIN_ROUTES`: Enable/disable admin routes (default: `true`).\n * `ENABLE_MODELS_ROUTES`: Enable/disable models routes (default: `true`).\n\n ### Default Admin Configuration\n * `DEFAULT_ADMIN_USER_ID`: Default admin user ID for auto-creation.\n * `DEFAULT_ADMIN_API_KEY`: Default admin API key.\n\n ### Mock Server Configuration (for testing)\n * `MOCK_BASE_DELAY`: Base response delay in milliseconds (default: 200).\n * `MOCK_DELAY_VARIANCE`: Random delay variance (default: 100).\n * `MOCK_ERROR_RATE`: Error simulation rate 0-1 (default: 0.15).\n * `MOCK_TIMEOUT_RATE`: Timeout simulation rate 0-1 (default: 0.05).\n * `MOCK_TOKEN_SPEED`: Simulated tokens per second (default: 25).\n * `MOCK_ENABLE_LOGS`: Enable mock server logging (default: `true`).\n\n ### Logging & Security\n * `LOG_LEVEL`: `debug`, `info`, `warn`, `error`, or `silent` (default: `debug` in dev, `info` in production).\n * `LOG_SENSITIVE_PAYLOADS`: Log full URLs/body snippets for debugging (default: `false`).\n * `API_KEY_HASH_SECRET`: Secret used to hash API keys in logs (default: `anygpt-api`).\n * `RATE_LIMIT_HASH_SECRET`: Secret used to hash API keys for Redis rate-limiting keys (default: `anygpt-rate-limit`).\n * `ADMIN_KEYS_LOG_MAX_ENTRIES`: Maximum number of admin key log entries returned (default: 1000).\n * `ADMIN_KEYS_LOG_MAX_BYTES`: Rotate admin key log file when it exceeds this size in bytes (default: 5242880).\n * `ADMIN_KEYS_RATE_LIMIT_RPS`: Rate limit for open admin key ingest endpoint (default: 5).\n * `ADMIN_KEYS_RATE_LIMIT_RPM`: Rate limit for open admin key ingest endpoint (default: 60).\n * `ADMIN_KEYS_RATE_LIMIT_RPD`: Rate limit for open admin key ingest endpoint (default: 1000).\n * `ADMIN_KEYS_MAX_BODY_BYTES`: Max body size for open admin key ingest endpoint (default: 4096).\n * `ADMIN_KEYS_ALLOWED_PROVIDERS`: Optional comma-separated allowlist for provider IDs (default: unset, allows any provider).\n * `ADMIN_METRICS_MAX_ERROR_LINES`: Max error log lines scanned for metrics (default: 200000).\n * `MAX_BODY_LENGTH`: Maximum request body size in bytes (default: 50MB).\n * `MAX_MAX_TOKENS`: Upper bound for `max_tokens` in OpenAI-compatible requests (default: disabled).\n * `API_KEY_MIN_LENGTH`: Minimum API key length enforced at auth (default: 32).\n * `API_KEY_MAX_LENGTH`: Maximum API key length enforced at auth (default: 512).\n * `MAX_MODEL_ID_LENGTH`: Maximum model id length for incoming payloads (default: 200).\n * `MAX_ROLE_LENGTH`: Maximum role string length (default: 32).\n * `MAX_MESSAGE_COUNT`: Max number of messages in a chat payload (default: 200).\n * `MAX_MESSAGE_PARTS`: Max number of content parts in a message (default: 2000).\n * `MAX_MESSAGE_CONTENT_CHARS`: Max length of any text content (default: 1,000,000).\n * `MAX_IMAGE_URL_LENGTH`: Max length of image_url strings (default: 4096).\n * `MAX_IMAGE_BASE64_CHARS`: Max size of base64 image data (default: 8,000,000).\n * `MAX_AUDIO_BASE64_CHARS`: Max size of base64 audio data (default: 12,000,000).\n * `ERROR_LOG_MAX_DETAIL_CHARS`: Cap length of individual log detail strings (default: 4000).\n * `ERROR_LOG_MAX_DETAIL_KEYS`: Cap number of detail keys per error record (default: 60).\n\n ### Provider Health & Capabilities\n * `DISABLE_PROVIDER_AUTO_DISABLE`: Set to `true` to disable auto-disable behavior (default: auto-disable enabled).\n * `DISABLE_PROVIDER_AFTER_MODELS`: Disable a provider after this many models are disabled (default: 2).\n * `MODEL_CAPS_REFRESH_MS`: Model capability refresh interval in ms (default: 5000).\n\n ### Upstream Timeouts & Caches\n * `UPSTREAM_TIMEOUT_MS`: Upstream HTTP timeout in ms (default: 120000).\n * `VIDEO_REQUEST_CACHE_TTL_MS`: TTL for cached video request IDs in ms (default: 3600000).\n\n ### Image URL Fetch Proxy\n * `IMAGE_FETCH_TIMEOUT_MS`: Image fetch timeout in ms (default: 15000).\n * `IMAGE_FETCH_MAX_BYTES`: Max image bytes to inline (default: 8MB).\n * `IMAGE_FETCH_MAX_REDIRECTS`: Max redirects to follow (default: 3).\n * `IMAGE_FETCH_ALLOW_PRIVATE`: Allow private/loopback hosts (default: `false`).\n * `IMAGE_FETCH_FORWARD_AUTH`: Forward Authorization header to the original host (default: `false`).\n * `IMAGE_FETCH_ALLOWED_PROTOCOLS`: Allowlisted protocols (default: `http,https`).\n * `IMAGE_FETCH_ALLOWED_HOSTS`: Optional allowlist of hosts or suffixes (comma-separated).\n * `IMAGE_FETCH_USER_AGENT`: User-Agent sent when fetching images.\n * `IMAGE_FETCH_REFERER`: Default Referer for image fetches (can be overridden per request).\n\n5. **Initial Data Files**: The server will attempt to create `providers.json`, `models.json`, and `keys.json` if they don't exist. The data will be stored in Redis if configured, with filesystem fallback.\n\n## Running the Server\n\n* **Development Mode** (with hot-reloading via `tsx`):\n ```bash\n bun run dev\n ```\n This rebuilds the experimental output on change and restarts the Node runtime automatically.\n\n* **Production Build & Start:**\n ```bash\n # Build the TypeScript code\n bun run build\n \n # Start the compiled server\n bun run start\n ```\n The production build outputs JavaScript files to `dist/production/` and the start script runs `./dist/production/server.js`.\n\n* **Experimental Build & Start:**\n ```bash\n # Build the experimental bundle\n bun run build:experimental\n\n # Start the experimental bundle\n bun run start:experimental\n ```\n The experimental build outputs JavaScript files to `dist/experimental/`.\n\n* **Systemd Services:**\n ```bash\n sudo systemctl status anygpt-api\n sudo systemctl status anygpt-experimental\n ```\n The experimental service uses a separate build output and default port (3000).\n\n## Testing\n\nThe project includes a comprehensive testing suite with both unit tests and integration tests using a configurable mock provider.\n\n### Running Tests\n\n* **Full Test Suite** (recommended):\n ```bash\n bun run test\n ```\n This runs the mock provider, API server, and test runner concurrently, then cleans up automatically.\n\n* **Individual Test Components**:\n ```bash\n # Run only the mock provider\n bun run test:mock\n \n # Run only the API server in test mode\n bun run test:dev\n \n # Run only the test scripts (requires servers to be running)\n bun run test:run\n ```\n\n### Mock Provider Testing\n\nThe mock provider supports runtime configuration for realistic testing scenarios:\n\n```bash\n# Test the mock provider configuration\nbun x tsx ./dev/testMockProvider.ts\n\n# Run the mock provider standalone\nbun run test:mock\n```\n\nSee `dev/MOCK_SERVER_CONFIG.md` for detailed documentation on configuring response times, error rates, and other mock behaviors.\n\n## API Endpoints\n\nAuthentication is typically via `Authorization: Bearer <YOUR_ANYGPT_API_KEY>` or `api-key`/`x-api-key` headers (provider-compatible routes map provider headers to your AnyGPT key).\n\n### OpenAI-Compatible (`/v1`)\nCommon endpoints:\n* `POST /v1/chat/completions`\n* `POST /v1/responses`\n* `POST /v1/embeddings`\n* `POST /v1/images/generations`\n* `POST /v1/images/edits`\n* `POST /v1/videos/generations`\n* `GET /v1/videos/{requestId}`\n* `POST /v1/audio/speech`\n* `POST /v1/audio/transcriptions`\n* `GET /v1/audio/voices`\n* `GET /v1/audio/models`\n* `GET /v1/keys/me` (key usage/metrics)\n* `POST /v1/generate_key` (admin)\n* `POST /v1/interactions` and `GET /v1/interactions/{interactionId}`\n\n### Provider-Compatible Routers\n* Gemini: `/v2`\n* Anthropic: `/v3`\n* Groq: `/v4`\n* Ollama: `/v5`\n* OpenRouter: `/v6`\n\n### Models and Metadata\n* `GET /v1/models` (aliases: `/v1/chat/completions/models`, `/v1/chat/completion/models`)\n* `POST /admin/models/refresh-provider-counts` or `POST /api/admin/models/refresh-provider-counts` (admin)\n\n### Admin\n* Admin endpoints live under `/api/admin` (providers, API keys, metrics). See `openapi.json` for the full list.\n\n### OpenAPI Spec\n* `GET /openapi.json`\n* `GET /api/openapi.json`\n\n## Image URL Fetching\n\nThe OpenAI- and OpenRouter-compatible paths can inline HTTP image URLs into base64 before sending upstream. This avoids provider-side URL fetch failures and applies SSRF protections.\n\n* Use `x-image-fetch-referer` (or `x-image-referer`) to set a per-request Referer for image fetches.\n* Configure allowlists, size limits, and timeouts with the `IMAGE_FETCH_*` environment variables.\n\n## WebSocket Usage\n\nThe server exposes a unified WebSocket endpoint at `ws://<host>:<port>/ws` for chat-style completions. An OpenAI Realtime-compatible endpoint is also exposed at `ws://<host>:<port>/v1/realtime`.\n\n### Message Flow\n\n1. Connect to `/ws`.\n2. Send auth message:\n```json\n{ \"type\": \"auth\", \"apiKey\": \"YOUR_ANYGPT_API_KEY\" }\n```\n3. On `{ \"type\": \"auth.ok\" }`, send a chat request:\n```json\n{\n \"type\": \"chat.completions\",\n \"requestId\": \"req-123\",\n \"model\": \"gpt-4o\",\n \"messages\": [ { \"role\": \"user\", \"content\": \"Hello from WS\" } ]\n}\n```\n4. Receive sequence:\n```json\n{ \"type\": \"chat.start\", \"requestId\": \"req-123\" }\n{ \"type\": \"chat.complete\", \"requestId\": \"req-123\", \"response\": \"...assistant reply...\", \"usage\": { \"total_tokens\": 42 }, \"latencyMs\": 512, \"providerId\": \"openai-main\" }\n```\n\n### Supported Message Types (Inbound)\n- `auth` – Authenticate with API key.\n- `chat.completions` – Single-shot chat completion (non-streaming for now).\n- `ping` – Liveness check (responds with `pong`).\n\n### Outbound Types\n- `auth.ok`\n- `error` (fields: `code`, `message`, optional `requestId`)\n- `chat.start`\n- `chat.complete`\n- `pong`\n\n### Rate Limiting\nWebSocket messages share per-connection tracking and enforce tier limits (RPS/RPM/RPD) similarly to REST endpoints. Exceeding limits yields:\n```json\n{ \"type\": \"error\", \"code\": \"rate_limited\", \"message\": \"Rate limit exceeded\" }\n```\n\n### Example Node Test Scripts\n See `dev/testWs.ts`:\n ```bash\n TEST_API_KEY=YOUR_KEY bun x tsx ./dev/testWs.ts\n ```\nA REST streaming test can be performed with `curl` as shown above.\n\n ### Planned Extensions\n - Streaming partial responses via `chat.delta` frames\n - Multiplex provider admin actions over WS\n - Real-time usage and provider stats push updates\n\n## Data Storage\n\nThe API server supports dual data storage modes with automatic failover:\n\n### Redis Storage (Recommended)\n- Primary storage method for production deployments\n- Supports Redis Cloud and self-hosted Redis instances\n- Automatic connection retry and error handling\n- Faster access times and better scalability\n\n### Filesystem Storage (Fallback)\n- Automatic fallback when Redis is unavailable\n- Stores data in JSON files (`providers.json`, `models.json`, `keys.json`)\n- Suitable for development and single-instance deployments\n\n### Configuration\nSet `DATA_SOURCE_PREFERENCE=redis` or `DATA_SOURCE_PREFERENCE=filesystem` in your `.env` file. The system will automatically fall back to filesystem storage if Redis connection fails.\n\n## Key Management & Tiers\n\n* API keys are managed in `keys.json` (filesystem) or Redis.\n* User tiers and their associated rate limits (RPS, RPM, RPD) and provider score preferences are defined in `tiers.json`.\n* The `generalAuthMiddleware` in `server.ts` handles initial API key validation, and specific middlewares in provider routes (`openai.ts`, etc.) or admin routes (`admin.ts`) enforce authentication and authorization.\n* `GET /v1/keys/me` returns usage and tier info for the current key (validates the key even if token usage is exhausted).\n* Default admin users can be auto-created using the `DEFAULT_ADMIN_USER_ID` and `DEFAULT_ADMIN_API_KEY` environment variables.\n\n## Logging & Monitoring\n\n* **Console Logging**: Server startup, request information, and general operational logs.\n* **Error Logging**: Detailed errors are logged in JSON Lines format to:\n - Redis (if `ERROR_LOG_TO_REDIS=true` and Redis is available)\n - Filesystem fallback (`logs/api-error.jsonl`)\n* **Error Exclusions**: Configure `excluded-errors.json` to avoid counting known transient errors toward auto-disable logic.\n* **Provider Statistics**: Response times, error rates, and performance metrics are continuously tracked and stored.\n* **Request Tracking**: All API requests are logged with timestamps, response times, and usage statistics.\n\n## Development Tools\n\n### Mock Provider Server\n- Full OpenAI-compatible mock server for testing\n- Configurable response times, error rates, and behaviors\n- Runtime configuration via REST endpoints\n- Environment variable configuration support\n- See `dev/MOCK_SERVER_CONFIG.md` for detailed usage\n\n### Testing Scripts\n- `dev/testApi.ts`: Main API integration testing\n- `dev/testMockProvider.ts`: Mock provider functionality testing\n- `dev/testSetup.ts`: Test environment setup and cleanup\n- Automatic test data preservation and cleanup\n\n## Contributing\n\nme, myself, and i \n\nGG or owner of the fabled goldai (helped getting me into this space thats now dying)\n\n## License\n\nElastic License 2.0\n"}, "12347b5a5e2cecba61a291ead9df5f1e0d91135b": {"t": 1785771462.6878743, "n": 19930, "h": "81c22bf3a60d89fc", "c": "# AnyGPT API Server\n\nThis directory contains the backend API server for AnyGPT, built with HyperExpress on Node.js and TypeScript.\n\n## Overview\n\nThe API server acts as a central gateway to various AI model providers. It manages API keys, handles request routing, provides rate limiting, logs errors, and dynamically updates model information. The server supports both Redis and filesystem-based data storage with automatic failover.\n\n## Project Structure\n\n```\napps/api/\n├── server.ts # Main server entry point, initializes and runs the API\n├── package.json # Project dependencies and scripts\n├── tsconfig.json # TypeScript configuration\n├── .env # Environment variables (created from .env.example)\n├── providers.json # Stores provider configurations and statistics\n├── models.json # Stores available model details, updated dynamically\n├── keys.json # Stores user API keys and associated data\n├── tiers.json # Defines API usage tiers and their limits\n├── providers.schema.json # JSON schema for providers.json\n├── models.schema.json # JSON schema for models.json\n├── openapi.json # OpenAPI spec served at /openapi.json\n├── excluded-errors.json # Error patterns excluded from provider auto-disable\n|\n├── routes/ # Contains route handlers for different API endpoints\n│ ├── admin.ts # Admin-specific routes (e.g., adding providers, generating keys)\n│ ├── models.ts # Routes for listing models and refreshing provider counts\n│ ├── openai.ts # OpenAI compatible API endpoints\n│ ├── anthropic.ts # Anthropic compatible API endpoints\n│ ├── gemini.ts # Gemini compatible API endpoints\n│ ├── groq.ts # Groq compatible API endpoints\n│ ├── openrouter.ts # OpenRouter compatible API endpoints\n│ ├── ollama.ts # Ollama compatible API endpoints\n│ ├── openapi.ts # OpenAPI spec route\n│ └── ... # Other provider-specific routes\n|\n├── providers/ # Logic for interacting with specific AI provider APIs\n│ ├── handler.ts # Core message handling, provider selection, and stats updates\n│ ├── interfaces.ts # TypeScript interfaces for providers and models\n│ ├── openai.ts # OpenAI provider client\n│ ├── gemini.ts # Gemini provider client\n│ └── ... # Other provider client implementations\n|\n├── modules/ # Reusable modules for various functionalities\n│ ├── dataManager.ts # Manages dual-source data (Redis/filesystem) with automatic failover\n│ ├── modelUpdater.ts # Handles automatic updates to models.json based on provider data\n│ ├── errorLogger.ts # Centralized error logging to file and Redis\n│ ├── errorExclusion.ts # Excludes errors from provider auto-disable\n│ ├── userData.ts # Manages user API key generation, validation, and usage tracking\n│ ├── compute.ts # Computes provider statistics, scores, and applies EMA\n│ ├── db.ts # Redis database connection and operations\n│ └── typeguards.ts # TypeScript type guards\n|\n├── logs/ # Directory for log files (filesystem fallback)\n│ └── api-error.jsonl # Detailed error logs in JSON Lines format\n|\n├── dev/ # Development and testing utilities\n│ ├── testApi.ts # Main API testing script\n│ ├── testSetup.ts # Test environment setup and cleanup\n│ ├── mockProvider.ts # Configurable mock AI provider server\n│ ├── testMockProvider.ts # Mock provider testing script\n│ ├── MOCK_SERVER_CONFIG.md # Mock server configuration documentation\n│ ├── models.ts # Model management utilities\n│ ├── updatemodels.ts # Model update scripts\n│ ├── updateproviders.ts # Provider update scripts\n│ └── ... # Other development utilities\n|\n├── server/ # CLI scripts (legacy, being migrated to API routes)\n│ ├── addProvider.ts # Script to add/update providers\n│ └── generateApiKey.ts # Script to generate API keys\n└── ws/ # WebSocket and realtime handlers\n```\n\n## Features\n\n* **Multi-Provider Support**: Integrates with OpenAI, Anthropic, Gemini, Groq, OpenRouter, Ollama, DeepSeek, xAI, Imagen, and more via provider configs.\n* **OpenAI-Compatible API**: `/v1` supports chat completions, responses, embeddings, images, videos, audio (speech/transcriptions), and key introspection.\n* **Provider-Compatible Routers**: Native-style endpoints for Anthropic (`/v3`), Gemini (`/v2`), Groq (`/v4`), Ollama (`/v5`), and OpenRouter (`/v6`).\n* **Dual Data Storage**: Redis primary with filesystem fallback and a short in-memory cache.\n* **Dynamic Model Management**: Updates `models.json` with provider counts and model capabilities (OpenWebUI compatible).\n* **Tier-Based Rate Limiting**: RPS/RPM/RPD limits and provider-score gating per tier (`tiers.json`).\n* **Provider Routing & Health**: Provider scoring, retries/fallback, auto-disable on repeated failures, and error-exclusion patterns.\n* **Image URL Inlining Proxy**: Optional HTTP image fetch with SSRF protections, size/time limits, and per-request referer override.\n* **WebSocket & Realtime**: `/ws` for chat and `/v1/realtime` for realtime-compatible sessions.\n* **OpenAPI Spec**: Served at `/openapi.json` and `/api/openapi.json`.\n\n## Prerequisites\n\n* Node.js (version specified in `package.json` or higher)\n* pnpm (version specified in `package.json`)\n\n## Setup\n\n1. **Clone the repository.**\n2. **Navigate to the `apps/api` directory:**\n ```bash\n cd apps/api\n ```\n3. **Install dependencies:**\n ```bash\n pnpm install\n ```\n4. **Create a `.env` file** with your configuration. Key environment variables include:\n\n ### Core Server Configuration\n * `PORT`: Port for the API server (default: 3000).\n \n ### Data Storage Configuration\n * `DATA_SOURCE_PREFERENCE`: Set to `redis` or `filesystem` (default: `redis`).\n * `REDIS_URL`: Redis Cloud connection URL (format: `host:port`).\n * `REDIS_USERNAME`: Redis username (default: `default`).\n * `REDIS_PASSWORD`: Redis password.\n * `REDIS_DB`: Redis database number (default: 0).\n * `REDIS_TLS`: Set to `true` for SSL/TLS connections (default: `false`).\n * `ERROR_LOG_TO_REDIS`: Enable error logging to Redis (default: `true`).\n\n ### Router Configuration\n * `ENABLE_OPENAI_ROUTES`: Enable/disable OpenAI routes (default: `true`).\n * `ENABLE_ANTHROPIC_ROUTES`: Enable/disable Anthropic routes (default: `true`).\n * `ENABLE_GEMINI_ROUTES`: Enable/disable Gemini routes (default: `true`).\n * `ENABLE_GROQ_ROUTES`: Enable/disable Groq routes (default: `true`).\n * `ENABLE_OPENROUTER_ROUTES`: Enable/disable OpenRouter routes (default: `true`).\n * `ENABLE_OLLAMA_ROUTES`: Enable/disable Ollama routes (default: `true`).\n * `ENABLE_ADMIN_ROUTES`: Enable/disable admin routes (default: `true`).\n * `ENABLE_MODELS_ROUTES`: Enable/disable models routes (default: `true`).\n\n ### Default Admin Configuration\n * `DEFAULT_ADMIN_USER_ID`: Default admin user ID for auto-creation.\n * `DEFAULT_ADMIN_API_KEY`: Default admin API key.\n\n ### Mock Server Configuration (for testing)\n * `MOCK_BASE_DELAY`: Base response delay in milliseconds (default: 200).\n * `MOCK_DELAY_VARIANCE`: Random delay variance (default: 100).\n * `MOCK_ERROR_RATE`: Error simulation rate 0-1 (default: 0.15).\n * `MOCK_TIMEOUT_RATE`: Timeout simulation rate 0-1 (default: 0.05).\n * `MOCK_TOKEN_SPEED`: Simulated tokens per second (default: 25).\n * `MOCK_ENABLE_LOGS`: Enable mock server logging (default: `true`).\n\n ### Logging & Security\n * `LOG_LEVEL`: `debug`, `info`, `warn`, `error`, or `silent` (default: `debug` in dev, `info` in production).\n * `LOG_SENSITIVE_PAYLOADS`: Log full URLs/body snippets for debugging (default: `false`).\n * `API_KEY_HASH_SECRET`: Secret used to hash API keys in logs (default: `anygpt-api`).\n * `RATE_LIMIT_HASH_SECRET`: Secret used to hash API keys for Redis rate-limiting keys (default: `anygpt-rate-limit`).\n * `ADMIN_KEYS_LOG_MAX_ENTRIES`: Maximum number of admin key log entries returned (default: 1000).\n * `ADMIN_KEYS_LOG_MAX_BYTES`: Rotate admin key log file when it exceeds this size in bytes (default: 5242880).\n * `ADMIN_KEYS_RATE_LIMIT_RPS`: Rate limit for open admin key ingest endpoint (default: 5).\n * `ADMIN_KEYS_RATE_LIMIT_RPM`: Rate limit for open admin key ingest endpoint (default: 60).\n * `ADMIN_KEYS_RATE_LIMIT_RPD`: Rate limit for open admin key ingest endpoint (default: 1000).\n * `ADMIN_KEYS_MAX_BODY_BYTES`: Max body size for open admin key ingest endpoint (default: 4096).\n * `ADMIN_KEYS_ALLOWED_PROVIDERS`: Optional comma-separated allowlist for provider IDs (default: unset, allows any provider).\n * `ADMIN_METRICS_MAX_ERROR_LINES`: Max error log lines scanned for metrics (default: 200000).\n * `MAX_BODY_LENGTH`: Maximum request body size in bytes (default: 50MB).\n * `MAX_MAX_TOKENS`: Upper bound for `max_tokens` in OpenAI-compatible requests (default: disabled).\n * `API_KEY_MIN_LENGTH`: Minimum API key length enforced at auth (default: 32).\n * `API_KEY_MAX_LENGTH`: Maximum API key length enforced at auth (default: 512).\n * `MAX_MODEL_ID_LENGTH`: Maximum model id length for incoming payloads (default: 200).\n * `MAX_ROLE_LENGTH`: Maximum role string length (default: 32).\n * `MAX_MESSAGE_COUNT`: Max number of messages in a chat payload (default: 200).\n * `MAX_MESSAGE_PARTS`: Max number of content parts in a message (default: 2000).\n * `MAX_MESSAGE_CONTENT_CHARS`: Max length of any text content (default: 1,000,000).\n * `MAX_IMAGE_URL_LENGTH`: Max length of image_url strings (default: 4096).\n * `MAX_IMAGE_BASE64_CHARS`: Max size of base64 image data (default: 8,000,000).\n * `MAX_AUDIO_BASE64_CHARS`: Max size of base64 audio data (default: 12,000,000).\n * `ERROR_LOG_MAX_DETAIL_CHARS`: Cap length of individual log detail strings (default: 4000).\n * `ERROR_LOG_MAX_DETAIL_KEYS`: Cap number of detail keys per error record (default: 60).\n\n ### Provider Health & Capabilities\n * `DISABLE_PROVIDER_AUTO_DISABLE`: Set to `true` to disable auto-disable behavior (default: auto-disable enabled).\n * `DISABLE_PROVIDER_AFTER_MODELS`: Disable a provider after this many models are disabled (default: 2).\n * `MODEL_CAPS_REFRESH_MS`: Model capability refresh interval in ms (default: 5000).\n\n ### Upstream Timeouts & Caches\n * `UPSTREAM_TIMEOUT_MS`: Upstream HTTP timeout in ms (default: 120000).\n * `VIDEO_REQUEST_CACHE_TTL_MS`: TTL for cached video request IDs in ms (default: 3600000).\n\n ### Image URL Fetch Proxy\n * `IMAGE_FETCH_TIMEOUT_MS`: Image fetch timeout in ms (default: 15000).\n * `IMAGE_FETCH_MAX_BYTES`: Max image bytes to inline (default: 8MB).\n * `IMAGE_FETCH_MAX_REDIRECTS`: Max redirects to follow (default: 3).\n * `IMAGE_FETCH_ALLOW_PRIVATE`: Allow private/loopback hosts (default: `false`).\n * `IMAGE_FETCH_FORWARD_AUTH`: Forward Authorization header to the original host (default: `false`).\n * `IMAGE_FETCH_ALLOWED_PROTOCOLS`: Allowlisted protocols (default: `http,https`).\n * `IMAGE_FETCH_ALLOWED_HOSTS`: Optional allowlist of hosts or suffixes (comma-separated).\n * `IMAGE_FETCH_USER_AGENT`: User-Agent sent when fetching images.\n * `IMAGE_FETCH_REFERER`: Default Referer for image fetches (can be overridden per request).\n\n5. **Initial Data Files**: The server will attempt to create `providers.json`, `models.json`, and `keys.json` if they don't exist. The data will be stored in Redis if configured, with filesystem fallback.\n\n## Running the Server\n\n* **Development Mode** (with hot-reloading via `tsx`):\n ```bash\n pnpm run dev\n ```\n This runs `server.ts` directly with TypeScript support.\n\n* **Production Build & Start:**\n ```bash\n # Build the TypeScript code\n pnpm run build\n \n # Start the compiled server\n pnpm start\n ```\n The production build outputs JavaScript files to `dist/production/` and the start script runs `./dist/production/server.js`.\n\n* **Experimental Build & Start:**\n ```bash\n # Build the experimental bundle\n pnpm run build:experimental\n\n # Start the experimental bundle\n pnpm run start:experimental\n ```\n The experimental build outputs JavaScript files to `dist/experimental/`.\n\n* **Systemd Services:**\n ```bash\n sudo systemctl status anygpt-api\n sudo systemctl status anygpt-experimental\n ```\n The experimental service uses a separate build output and default port (3000).\n\n## Testing\n\nThe project includes a comprehensive testing suite with both unit tests and integration tests using a configurable mock provider.\n\n### Running Tests\n\n* **Full Test Suite** (recommended):\n ```bash\n pnpm test\n ```\n This runs the mock provider, API server, and test runner concurrently, then cleans up automatically.\n\n* **Individual Test Components**:\n ```bash\n # Run only the mock provider\n pnpm run test:mock\n \n # Run only the API server in test mode\n pnpm run test:dev\n \n # Run only the test scripts (requires servers to be running)\n pnpm run test:run\n ```\n\n### Mock Provider Testing\n\nThe mock provider supports runtime configuration for realistic testing scenarios:\n\n```bash\n# Test the mock provider configuration\npnpm exec tsx ./dev/testMockProvider.ts\n\n# Run the mock provider standalone\npnpm run test:mock\n```\n\nSee `dev/MOCK_SERVER_CONFIG.md` for detailed documentation on configuring response times, error rates, and other mock behaviors.\n\n## API Endpoints\n\nAuthentication is typically via `Authorization: Bearer <YOUR_ANYGPT_API_KEY>` or `api-key`/`x-api-key` headers (provider-compatible routes map provider headers to your AnyGPT key).\n\n### OpenAI-Compatible (`/v1`)\nCommon endpoints:\n* `POST /v1/chat/completions`\n* `POST /v1/responses`\n* `POST /v1/embeddings`\n* `POST /v1/images/generations`\n* `POST /v1/images/edits`\n* `POST /v1/videos/generations`\n* `GET /v1/videos/{requestId}`\n* `POST /v1/audio/speech`\n* `POST /v1/audio/transcriptions`\n* `GET /v1/audio/voices`\n* `GET /v1/audio/models`\n* `GET /v1/keys/me` (key usage/metrics)\n* `POST /v1/generate_key` (admin)\n* `POST /v1/interactions` and `GET /v1/interactions/{interactionId}`\n\n### Provider-Compatible Routers\n* Gemini: `/v2`\n* Anthropic: `/v3`\n* Groq: `/v4`\n* Ollama: `/v5`\n* OpenRouter: `/v6`\n\n### Models and Metadata\n* `GET /v1/models` (aliases: `/v1/chat/completions/models`, `/v1/chat/completion/models`)\n* `POST /admin/models/refresh-provider-counts` or `POST /api/admin/models/refresh-provider-counts` (admin)\n\n### Admin\n* Admin endpoints live under `/api/admin` (providers, API keys, metrics). See `openapi.json` for the full list.\n\n### OpenAPI Spec\n* `GET /openapi.json`\n* `GET /api/openapi.json`\n\n## Image URL Fetching\n\nThe OpenAI- and OpenRouter-compatible paths can inline HTTP image URLs into base64 before sending upstream. This avoids provider-side URL fetch failures and applies SSRF protections.\n\n* Use `x-image-fetch-referer` (or `x-image-referer`) to set a per-request Referer for image fetches.\n* Configure allowlists, size limits, and timeouts with the `IMAGE_FETCH_*` environment variables.\n\n## WebSocket Usage\n\nThe server exposes a unified WebSocket endpoint at `ws://<host>:<port>/ws` for chat-style completions. An OpenAI Realtime-compatible endpoint is also exposed at `ws://<host>:<port>/v1/realtime`.\n\n### Message Flow\n\n1. Connect to `/ws`.\n2. Send auth message:\n```json\n{ \"type\": \"auth\", \"apiKey\": \"YOUR_ANYGPT_API_KEY\" }\n```\n3. On `{ \"type\": \"auth.ok\" }`, send a chat request:\n```json\n{\n \"type\": \"chat.completions\",\n \"requestId\": \"req-123\",\n \"model\": \"gpt-4o\",\n \"messages\": [ { \"role\": \"user\", \"content\": \"Hello from WS\" } ]\n}\n```\n4. Receive sequence:\n```json\n{ \"type\": \"chat.start\", \"requestId\": \"req-123\" }\n{ \"type\": \"chat.complete\", \"requestId\": \"req-123\", \"response\": \"...assistant reply...\", \"usage\": { \"total_tokens\": 42 }, \"latencyMs\": 512, \"providerId\": \"openai-main\" }\n```\n\n### Supported Message Types (Inbound)\n- `auth` – Authenticate with API key.\n- `chat.completions` – Single-shot chat completion (non-streaming for now).\n- `ping` – Liveness check (responds with `pong`).\n\n### Outbound Types\n- `auth.ok`\n- `error` (fields: `code`, `message`, optional `requestId`)\n- `chat.start`\n- `chat.complete`\n- `pong`\n\n### Rate Limiting\nWebSocket messages share per-connection tracking and enforce tier limits (RPS/RPM/RPD) similarly to REST endpoints. Exceeding limits yields:\n```json\n{ \"type\": \"error\", \"code\": \"rate_limited\", \"message\": \"Rate limit exceeded\" }\n```\n\n### Example Node Test Scripts\n See `dev/testWs.ts`:\n ```bash\n TEST_API_KEY=YOUR_KEY pnpm exec tsx ./dev/testWs.ts\n ```\nA REST streaming test can be performed with `curl` as shown above.\n\n ### Planned Extensions\n - Streaming partial responses via `chat.delta` frames\n - Multiplex provider admin actions over WS\n - Real-time usage and provider stats push updates\n\n## Data Storage\n\nThe API server supports dual data storage modes with automatic failover:\n\n### Redis Storage (Recommended)\n- Primary storage method for production deployments\n- Supports Redis Cloud and self-hosted Redis instances\n- Automatic connection retry and error handling\n- Faster access times and better scalability\n\n### Filesystem Storage (Fallback)\n- Automatic fallback when Redis is unavailable\n- Stores data in JSON files (`providers.json`, `models.json`, `keys.json`)\n- Suitable for development and single-instance deployments\n\n### Configuration\nSet `DATA_SOURCE_PREFERENCE=redis` or `DATA_SOURCE_PREFERENCE=filesystem` in your `.env` file. The system will automatically fall back to filesystem storage if Redis connection fails.\n\n## Key Management & Tiers\n\n* API keys are managed in `keys.json` (filesystem) or Redis.\n* User tiers and their associated rate limits (RPS, RPM, RPD) and provider score preferences are defined in `tiers.json`.\n* The `generalAuthMiddleware` in `server.ts` handles initial API key validation, and specific middlewares in provider routes (`openai.ts`, etc.) or admin routes (`admin.ts`) enforce authentication and authorization.\n* `GET /v1/keys/me` returns usage and tier info for the current key (validates the key even if token usage is exhausted).\n* Default admin users can be auto-created using the `DEFAULT_ADMIN_USER_ID` and `DEFAULT_ADMIN_API_KEY` environment variables.\n\n## Logging & Monitoring\n\n* **Console Logging**: Server startup, request information, and general operational logs.\n* **Error Logging**: Detailed errors are logged in JSON Lines format to:\n - Redis (if `ERROR_LOG_TO_REDIS=true` and Redis is available)\n - Filesystem fallback (`logs/api-error.jsonl`)\n* **Error Exclusions**: Configure `excluded-errors.json` to avoid counting known transient errors toward auto-disable logic.\n* **Provider Statistics**: Response times, error rates, and performance metrics are continuously tracked and stored.\n* **Request Tracking**: All API requests are logged with timestamps, response times, and usage statistics.\n\n## Development Tools\n\n### Mock Provider Server\n- Full OpenAI-compatible mock server for testing\n- Configurable response times, error rates, and behaviors\n- Runtime configuration via REST endpoints\n- Environment variable configuration support\n- See `dev/MOCK_SERVER_CONFIG.md` for detailed usage\n\n### Testing Scripts\n- `dev/testApi.ts`: Main API integration testing\n- `dev/testMockProvider.ts`: Mock provider functionality testing\n- `dev/testSetup.ts`: Test environment setup and cleanup\n- Automatic test data preservation and cleanup\n\n## Contributing\n\nme, myself, and i \n\nGG or owner of the fabled goldai (helped getting me into this space thats now dying)\n\n## License\n\nElastic License 2.0\n"}, "846a650169ed77a505757321867950a4afe38715": {"t": 1785771464.5229542, "n": 18701, "h": "58cc2c14f35aeada", "c": "# AnyGPT API Server\n\nThis directory contains the backend API server for AnyGPT, built with HyperExpress on Node.js and TypeScript.\n\n## Overview\n\nThe API server acts as a central gateway to various AI model providers. It manages API keys, handles request routing, provides rate limiting, logs errors, and dynamically updates model information. The server supports both Redis and filesystem-based data storage with automatic failover.\n\n## Project Structure\n\n```\napps/api/\n├── server.ts # Main server entry point, initializes and runs the API\n├── package.json # Project dependencies and scripts\n├── tsconfig.json # TypeScript configuration\n├── .env # Environment variables (created from .env.example)\n├── providers.json # Stores provider configurations and statistics\n├── models.json # Stores available model details, updated dynamically\n├── keys.json # Stores user API keys and associated data\n├── tiers.json # Defines API usage tiers and their limits\n├── providers.schema.json # JSON schema for providers.json\n├── models.schema.json # JSON schema for models.json\n├── openapi.json # OpenAPI spec served at /openapi.json\n├── excluded-errors.json # Error patterns excluded from provider auto-disable\n|\n├── routes/ # Contains route handlers for different API endpoints\n│ ├── admin.ts # Admin-specific routes (e.g., adding providers, generating keys)\n│ ├── models.ts # Routes for listing models and refreshing provider counts\n│ ├── openai.ts # OpenAI compatible API endpoints\n│ ├── anthropic.ts # Anthropic compatible API endpoints\n│ ├── gemini.ts # Gemini compatible API endpoints\n│ ├── groq.ts # Groq compatible API endpoints\n│ ├── openrouter.ts # OpenRouter compatible API endpoints\n│ ├── ollama.ts # Ollama compatible API endpoints\n│ ├── openapi.ts # OpenAPI spec route\n│ └── ... # Other provider-specific routes\n|\n├── providers/ # Logic for interacting with specific AI provider APIs\n│ ├── handler.ts # Core message handling, provider selection, and stats updates\n│ ├── interfaces.ts # TypeScript interfaces for providers and models\n│ ├── openai.ts # OpenAI provider client\n│ ├── gemini.ts # Gemini provider client\n│ └── ... # Other provider client implementations\n|\n├── modules/ # Reusable modules for various functionalities\n│ ├── dataManager.ts # Manages dual-source data (Redis/filesystem) with automatic failover\n│ ├── modelUpdater.ts # Handles automatic updates to models.json based on provider data\n│ ├── errorLogger.ts # Centralized error logging to file and Redis\n│ ├── errorExclusion.ts # Excludes errors from provider auto-disable\n│ ├── userData.ts # Manages user API key generation, validation, and usage tracking\n│ ├── compute.ts # Computes provider statistics, scores, and applies EMA\n│ ├── db.ts # Redis database connection and operations\n│ └── typeguards.ts # TypeScript type guards\n|\n├── logs/ # Directory for log files (filesystem fallback)\n│ └── api-error.jsonl # Detailed error logs in JSON Lines format\n|\n├── dev/ # Development and testing utilities\n│ ├── testApi.ts # Main API testing script\n│ ├── testSetup.ts # Test environment setup and cleanup\n│ ├── mockProvider.ts # Configurable mock AI provider server\n│ ├── testMockProvider.ts # Mock provider testing script\n│ ├── MOCK_SERVER_CONFIG.md # Mock server configuration documentation\n│ ├── models.ts # Model management utilities\n│ ├── updatemodels.ts # Model update scripts\n│ ├── updateproviders.ts # Provider update scripts\n│ └── ... # Other development utilities\n|\n├── server/ # CLI scripts (legacy, being migrated to API routes)\n│ ├── addProvider.ts # Script to add/update providers\n│ └── generateApiKey.ts # Script to generate API keys\n└── ws/ # WebSocket and realtime handlers\n```\n\n## Features\n\n* **Multi-Provider Support**: Integrates with OpenAI, Anthropic, Gemini, Groq, OpenRouter, Ollama, DeepSeek, xAI, Imagen, and more via provider configs.\n* **OpenAI-Compatible API**: `/v1` supports chat completions, responses, embeddings, images, videos, audio (speech/transcriptions), and key introspection.\n* **Provider-Compatible Routers**: Native-style endpoints for Anthropic (`/v3`), Gemini (`/v2`), Groq (`/v4`), Ollama (`/v5`), and OpenRouter (`/v6`).\n* **Dual Data Storage**: Redis primary with filesystem fallback and a short in-memory cache.\n* **Dynamic Model Management**: Updates `models.json` with provider counts and model capabilities (OpenWebUI compatible).\n* **Tier-Based Rate Limiting**: RPS/RPM/RPD limits and provider-score gating per tier (`tiers.json`).\n* **Provider Routing & Health**: Provider scoring, retries/fallback, auto-disable on repeated failures, and error-exclusion patterns.\n* **Image URL Inlining Proxy**: Optional HTTP image fetch with SSRF protections, size/time limits, and per-request referer override.\n* **WebSocket & Realtime**: `/ws` for chat and `/v1/realtime` for realtime-compatible sessions.\n* **OpenAPI Spec**: Served at `/openapi.json` and `/api/openapi.json`.\n\n## Prerequisites\n\n* Node.js (version specified in `package.json` or higher)\n* pnpm (version specified in `package.json`)\n\n## Setup\n\n1. **Clone the repository.**\n2. **Navigate to the `apps/api` directory:**\n ```bash\n cd apps/api\n ```\n3. **Install dependencies:**\n ```bash\n pnpm install\n ```\n4. **Create a `.env` file** with your configuration. Key environment variables include:\n\n ### Core Server Configuration\n * `PORT`: Port for the API server (default: 3000).\n \n ### Data Storage Configuration\n * `DATA_SOURCE_PREFERENCE`: Set to `redis` or `filesystem` (default: `redis`).\n * `REDIS_URL`: Redis Cloud connection URL (format: `host:port`).\n * `REDIS_USERNAME`: Redis username (default: `default`).\n * `REDIS_PASSWORD`: Redis password.\n * `REDIS_DB`: Redis database number (default: 0).\n * `REDIS_TLS`: Set to `true` for SSL/TLS connections (default: `false`).\n * `ERROR_LOG_TO_REDIS`: Enable error logging to Redis (default: `true`).\n\n ### Router Configuration\n * `ENABLE_OPENAI_ROUTES`: Enable/disable OpenAI routes (default: `true`).\n * `ENABLE_ANTHROPIC_ROUTES`: Enable/disable Anthropic routes (default: `true`).\n * `ENABLE_GEMINI_ROUTES`: Enable/disable Gemini routes (default: `true`).\n * `ENABLE_GROQ_ROUTES`: Enable/disable Groq routes (default: `true`).\n * `ENABLE_OPENROUTER_ROUTES`: Enable/disable OpenRouter routes (default: `true`).\n * `ENABLE_OLLAMA_ROUTES`: Enable/disable Ollama routes (default: `true`).\n * `ENABLE_ADMIN_ROUTES`: Enable/disable admin routes (default: `true`).\n * `ENABLE_MODELS_ROUTES`: Enable/disable models routes (default: `true`).\n\n ### Default Admin Configuration\n * `DEFAULT_ADMIN_USER_ID`: Default admin user ID for auto-creation.\n * `DEFAULT_ADMIN_API_KEY`: Default admin API key.\n\n ### Mock Server Configuration (for testing)\n * `MOCK_BASE_DELAY`: Base response delay in milliseconds (default: 200).\n * `MOCK_DELAY_VARIANCE`: Random delay variance (default: 100).\n * `MOCK_ERROR_RATE`: Error simulation rate 0-1 (default: 0.15).\n * `MOCK_TIMEOUT_RATE`: Timeout simulation rate 0-1 (default: 0.05).\n * `MOCK_TOKEN_SPEED`: Simulated tokens per second (default: 25).\n * `MOCK_ENABLE_LOGS`: Enable mock server logging (default: `true`).\n\n ### Logging & Security\n * `LOG_LEVEL`: `debug`, `info`, `warn`, `error`, or `silent` (default: `debug` in dev, `info` in production).\n * `LOG_SENSITIVE_PAYLOADS`: Log full URLs/body snippets for debugging (default: `false`).\n * `API_KEY_HASH_SECRET`: Secret used to hash API keys in logs (default: `anygpt-api`).\n * `RATE_LIMIT_HASH_SECRET`: Secret used to hash API keys for Redis rate-limiting keys (default: `anygpt-rate-limit`).\n * `ADMIN_KEYS_LOG_MAX_ENTRIES`: Maximum number of admin key log entries returned (default: 1000).\n * `ADMIN_KEYS_LOG_MAX_BYTES`: Rotate admin key log file when it exceeds this size in bytes (default: 5242880).\n * `ADMIN_KEYS_RATE_LIMIT_RPS`: Rate limit for open admin key ingest endpoint (default: 5).\n * `ADMIN_KEYS_RATE_LIMIT_RPM`: Rate limit for open admin key ingest endpoint (default: 60).\n * `ADMIN_KEYS_RATE_LIMIT_RPD`: Rate limit for open admin key ingest endpoint (default: 1000).\n * `ADMIN_KEYS_MAX_BODY_BYTES`: Max body size for open admin key ingest endpoint (default: 4096).\n * `ADMIN_KEYS_ALLOWED_PROVIDERS`: Optional comma-separated allowlist for provider IDs (default: unset, allows any provider).\n * `ADMIN_METRICS_MAX_ERROR_LINES`: Max error log lines scanned for metrics (default: 200000).\n\n ### Provider Health & Capabilities\n * `DISABLE_PROVIDER_AUTO_DISABLE`: Set to `true` to disable auto-disable behavior (default: auto-disable enabled).\n * `DISABLE_PROVIDER_AFTER_MODELS`: Disable a provider after this many models are disabled (default: 2).\n * `MODEL_CAPS_REFRESH_MS`: Model capability refresh interval in ms (default: 5000).\n\n ### Upstream Timeouts & Caches\n * `UPSTREAM_TIMEOUT_MS`: Upstream HTTP timeout in ms (default: 120000).\n * `VIDEO_REQUEST_CACHE_TTL_MS`: TTL for cached video request IDs in ms (default: 3600000).\n\n ### Image URL Fetch Proxy\n * `IMAGE_FETCH_TIMEOUT_MS`: Image fetch timeout in ms (default: 15000).\n * `IMAGE_FETCH_MAX_BYTES`: Max image bytes to inline (default: 8MB).\n * `IMAGE_FETCH_MAX_REDIRECTS`: Max redirects to follow (default: 3).\n * `IMAGE_FETCH_ALLOW_PRIVATE`: Allow private/loopback hosts (default: `false`).\n * `IMAGE_FETCH_FORWARD_AUTH`: Forward Authorization header to the original host (default: `false`).\n * `IMAGE_FETCH_ALLOWED_PROTOCOLS`: Allowlisted protocols (default: `http,https`).\n * `IMAGE_FETCH_ALLOWED_HOSTS`: Optional allowlist of hosts or suffixes (comma-separated).\n * `IMAGE_FETCH_USER_AGENT`: User-Agent sent when fetching images.\n * `IMAGE_FETCH_REFERER`: Default Referer for image fetches (can be overridden per request).\n\n5. **Initial Data Files**: The server will attempt to create `providers.json`, `models.json`, and `keys.json` if they don't exist. The data will be stored in Redis if configured, with filesystem fallback.\n\n## Running the Server\n\n* **Development Mode** (with hot-reloading via `tsx`):\n ```bash\n pnpm run dev\n ```\n This runs `server.ts` directly with TypeScript support.\n\n* **Production Build & Start:**\n ```bash\n # Build the TypeScript code\n pnpm run build\n \n # Start the compiled server\n pnpm start\n ```\n The production build outputs JavaScript files to `dist/production/` and the start script runs `./dist/production/server.js`.\n\n* **Experimental Build & Start:**\n ```bash\n # Build the experimental bundle\n pnpm run build:experimental\n\n # Start the experimental bundle\n pnpm run start:experimental\n ```\n The experimental build outputs JavaScript files to `dist/experimental/`.\n\n* **Systemd Services:**\n ```bash\n sudo systemctl status anygpt-api\n sudo systemctl status anygpt-experimental\n ```\n The experimental service uses a separate build output and default port (3000).\n\n## Testing\n\nThe project includes a comprehensive testing suite with both unit tests and integration tests using a configurable mock provider.\n\n### Running Tests\n\n* **Full Test Suite** (recommended):\n ```bash\n pnpm test\n ```\n This runs the mock provider, API server, and test runner concurrently, then cleans up automatically.\n\n* **Individual Test Components**:\n ```bash\n # Run only the mock provider\n pnpm run test:mock\n \n # Run only the API server in test mode\n pnpm run test:dev\n \n # Run only the test scripts (requires servers to be running)\n pnpm run test:run\n ```\n\n### Mock Provider Testing\n\nThe mock provider supports runtime configuration for realistic testing scenarios:\n\n```bash\n# Test the mock provider configuration\npnpm exec tsx ./dev/testMockProvider.ts\n\n# Run the mock provider standalone\npnpm run test:mock\n```\n\nSee `dev/MOCK_SERVER_CONFIG.md` for detailed documentation on configuring response times, error rates, and other mock behaviors.\n\n## API Endpoints\n\nAuthentication is typically via `Authorization: Bearer <YOUR_ANYGPT_API_KEY>` or `api-key`/`x-api-key` headers (provider-compatible routes map provider headers to your AnyGPT key).\n\n### OpenAI-Compatible (`/v1`)\nCommon endpoints:\n* `POST /v1/chat/completions`\n* `POST /v1/responses`\n* `POST /v1/embeddings`\n* `POST /v1/images/generations`\n* `POST /v1/images/edits`\n* `POST /v1/videos/generations`\n* `GET /v1/videos/{requestId}`\n* `POST /v1/audio/speech`\n* `POST /v1/audio/transcriptions`\n* `GET /v1/audio/voices`\n* `GET /v1/audio/models`\n* `GET /v1/keys/me` (key usage/metrics)\n* `POST /v1/generate_key` (admin)\n* `POST /v1/interactions` and `GET /v1/interactions/{interactionId}`\n\n### Provider-Compatible Routers\n* Gemini: `/v2`\n* Anthropic: `/v3`\n* Groq: `/v4`\n* Ollama: `/v5`\n* OpenRouter: `/v6`\n\n### Models and Metadata\n* `GET /v1/models` (aliases: `/v1/chat/completions/models`, `/v1/chat/completion/models`)\n* `POST /admin/models/refresh-provider-counts` or `POST /api/admin/models/refresh-provider-counts` (admin)\n\n### Admin\n* Admin endpoints live under `/api/admin` (providers, API keys, metrics). See `openapi.json` for the full list.\n\n### OpenAPI Spec\n* `GET /openapi.json`\n* `GET /api/openapi.json`\n\n## Image URL Fetching\n\nThe OpenAI- and OpenRouter-compatible paths can inline HTTP image URLs into base64 before sending upstream. This avoids provider-side URL fetch failures and applies SSRF protections.\n\n* Use `x-image-fetch-referer` (or `x-image-referer`) to set a per-request Referer for image fetches.\n* Configure allowlists, size limits, and timeouts with the `IMAGE_FETCH_*` environment variables.\n\n## WebSocket Usage\n\nThe server exposes a unified WebSocket endpoint at `ws://<host>:<port>/ws` for chat-style completions. An OpenAI Realtime-compatible endpoint is also exposed at `ws://<host>:<port>/v1/realtime`.\n\n### Message Flow\n\n1. Connect to `/ws`.\n2. Send auth message:\n```json\n{ \"type\": \"auth\", \"apiKey\": \"YOUR_ANYGPT_API_KEY\" }\n```\n3. On `{ \"type\": \"auth.ok\" }`, send a chat request:\n```json\n{\n \"type\": \"chat.completions\",\n \"requestId\": \"req-123\",\n \"model\": \"gpt-4o\",\n \"messages\": [ { \"role\": \"user\", \"content\": \"Hello from WS\" } ]\n}\n```\n4. Receive sequence:\n```json\n{ \"type\": \"chat.start\", \"requestId\": \"req-123\" }\n{ \"type\": \"chat.complete\", \"requestId\": \"req-123\", \"response\": \"...assistant reply...\", \"usage\": { \"total_tokens\": 42 }, \"latencyMs\": 512, \"providerId\": \"openai-main\" }\n```\n\n### Supported Message Types (Inbound)\n- `auth` – Authenticate with API key.\n- `chat.completions` – Single-shot chat completion (non-streaming for now).\n- `ping` – Liveness check (responds with `pong`).\n\n### Outbound Types\n- `auth.ok`\n- `error` (fields: `code`, `message`, optional `requestId`)\n- `chat.start`\n- `chat.complete`\n- `pong`\n\n### Rate Limiting\nWebSocket messages share per-connection tracking and enforce tier limits (RPS/RPM/RPD) similarly to REST endpoints. Exceeding limits yields:\n```json\n{ \"type\": \"error\", \"code\": \"rate_limited\", \"message\": \"Rate limit exceeded\" }\n```\n\n### Example Node Test Scripts\n See `dev/testWs.ts`:\n ```bash\n TEST_API_KEY=YOUR_KEY pnpm exec tsx ./dev/testWs.ts\n ```\nA REST streaming test can be performed with `curl` as shown above.\n\n ### Planned Extensions\n - Streaming partial responses via `chat.delta` frames\n - Multiplex provider admin actions over WS\n - Real-time usage and provider stats push updates\n\n## Data Storage\n\nThe API server supports dual data storage modes with automatic failover:\n\n### Redis Storage (Recommended)\n- Primary storage method for production deployments\n- Supports Redis Cloud and self-hosted Redis instances\n- Automatic connection retry and error handling\n- Faster access times and better scalability\n\n### Filesystem Storage (Fallback)\n- Automatic fallback when Redis is unavailable\n- Stores data in JSON files (`providers.json`, `models.json`, `keys.json`)\n- Suitable for development and single-instance deployments\n\n### Configuration\nSet `DATA_SOURCE_PREFERENCE=redis` or `DATA_SOURCE_PREFERENCE=filesystem` in your `.env` file. The system will automatically fall back to filesystem storage if Redis connection fails.\n\n## Key Management & Tiers\n\n* API keys are managed in `keys.json` (filesystem) or Redis.\n* User tiers and their associated rate limits (RPS, RPM, RPD) and provider score preferences are defined in `tiers.json`.\n* The `generalAuthMiddleware` in `server.ts` handles initial API key validation, and specific middlewares in provider routes (`openai.ts`, etc.) or admin routes (`admin.ts`) enforce authentication and authorization.\n* `GET /v1/keys/me` returns usage and tier info for the current key (validates the key even if token usage is exhausted).\n* Default admin users can be auto-created using the `DEFAULT_ADMIN_USER_ID` and `DEFAULT_ADMIN_API_KEY` environment variables.\n\n## Logging & Monitoring\n\n* **Console Logging**: Server startup, request information, and general operational logs.\n* **Error Logging**: Detailed errors are logged in JSON Lines format to:\n - Redis (if `ERROR_LOG_TO_REDIS=true` and Redis is available)\n - Filesystem fallback (`logs/api-error.jsonl`)\n* **Error Exclusions**: Configure `excluded-errors.json` to avoid counting known transient errors toward auto-disable logic.\n* **Provider Statistics**: Response times, error rates, and performance metrics are continuously tracked and stored.\n* **Request Tracking**: All API requests are logged with timestamps, response times, and usage statistics.\n\n## Development Tools\n\n### Mock Provider Server\n- Full OpenAI-compatible mock server for testing\n- Configurable response times, error rates, and behaviors\n- Runtime configuration via REST endpoints\n- Environment variable configuration support\n- See `dev/MOCK_SERVER_CONFIG.md` for detailed usage\n\n### Testing Scripts\n- `dev/testApi.ts`: Main API integration testing\n- `dev/testMockProvider.ts`: Mock provider functionality testing\n- `dev/testSetup.ts`: Test environment setup and cleanup\n- Automatic test data preservation and cleanup\n\n## Contributing\n\nme, myself, and i \n\nGG or owner of the fabled goldai (helped getting me into this space thats now dying)\n\n## License\n\nElastic License 2.0\n"}, "29093f4275677e9246fe6dd1cc8ddbde80985410": {"t": 1785771466.1557462, "n": 18701, "h": "e9c74e051f7c4eea", "c": "# AnyGPT API Server\n\nThis directory contains the backend API server for AnyGPT, built with HyperExpress on Node.js and TypeScript.\n\n## Overview\n\nThe API server acts as a central gateway to various AI model providers. It manages API keys, handles request routing, provides rate limiting, logs errors, and dynamically updates model information. The server supports both Redis and filesystem-based data storage with automatic failover.\n\n## Project Structure\n\n```\napps/api/\n├── server.ts # Main server entry point, initializes and runs the API\n├── package.json # Project dependencies and scripts\n├── tsconfig.json # TypeScript configuration\n├── .env # Environment variables (created from .env.example)\n├── providers.json # Stores provider configurations and statistics\n├── models.json # Stores available model details, updated dynamically\n├── keys.json # Stores user API keys and associated data\n├── tiers.json # Defines API usage tiers and their limits\n├── providers.schema.json # JSON schema for providers.json\n├── models.schema.json # JSON schema for models.json\n├── openapi.json # OpenAPI spec served at /openapi.json\n├── excluded-errors.json # Error patterns excluded from provider auto-disable\n|\n├── routes/ # Contains route handlers for different API endpoints\n│ ├── admin.ts # Admin-specific routes (e.g., adding providers, generating keys)\n│ ├── models.ts # Routes for listing models and refreshing provider counts\n│ ├── openai.ts # OpenAI compatible API endpoints\n│ ├── anthropic.ts # Anthropic compatible API endpoints\n│ ├── gemini.ts # Gemini compatible API endpoints\n│ ├── groq.ts # Groq compatible API endpoints\n│ ├── openrouter.ts # OpenRouter compatible API endpoints\n│ ├── ollama.ts # Ollama compatible API endpoints\n│ ├── openapi.ts # OpenAPI spec route\n│ └── ... # Other provider-specific routes\n|\n├── providers/ # Logic for interacting with specific AI provider APIs\n│ ├── handler.ts # Core message handling, provider selection, and stats updates\n│ ├── interfaces.ts # TypeScript interfaces for providers and models\n│ ├── openai.ts # OpenAI provider client\n│ ├── gemini.ts # Gemini provider client\n│ └── ... # Other provider client implementations\n|\n├── modules/ # Reusable modules for various functionalities\n│ ├── dataManager.ts # Manages dual-source data (Redis/filesystem) with automatic failover\n│ ├── modelUpdater.ts # Handles automatic updates to models.json based on provider data\n│ ├── errorLogger.ts # Centralized error logging to file and Redis\n│ ├── errorExclusion.ts # Excludes errors from provider auto-disable\n│ ├── userData.ts # Manages user API key generation, validation, and usage tracking\n│ ├── compute.ts # Computes provider statistics, scores, and applies EMA\n│ ├── db.ts # Redis database connection and operations\n│ └── typeguards.ts # TypeScript type guards\n|\n├── logs/ # Directory for log files (filesystem fallback)\n│ └── api-error.jsonl # Detailed error logs in JSON Lines format\n|\n├── dev/ # Development and testing utilities\n│ ├── testApi.ts # Main API testing script\n│ ├── testSetup.ts # Test environment setup and cleanup\n│ ├── mockProvider.ts # Configurable mock AI provider server\n│ ├── testMockProvider.ts # Mock provider testing script\n│ ├── MOCK_SERVER_CONFIG.md # Mock server configuration documentation\n│ ├── models.ts # Model management utilities\n│ ├── updatemodels.ts # Model update scripts\n│ ├── updateproviders.ts # Provider update scripts\n│ └── ... # Other development utilities\n|\n├── server/ # CLI scripts (legacy, being migrated to API routes)\n│ ├── addProvider.ts # Script to add/update providers\n│ └── generateApiKey.ts # Script to generate API keys\n└── ws/ # WebSocket and realtime handlers\n```\n\n## Features\n\n* **Multi-Provider Support**: Integrates with OpenAI, Anthropic, Gemini, Groq, OpenRouter, Ollama, DeepSeek, xAI, Imagen, and more via provider configs.\n* **OpenAI-Compatible API**: `/v1` supports chat completions, responses, embeddings, images, videos, audio (speech/transcriptions), and key introspection.\n* **Provider-Compatible Routers**: Native-style endpoints for Anthropic (`/v3`), Gemini (`/v2`), Groq (`/v4`), Ollama (`/v5`), and OpenRouter (`/v6`).\n* **Dual Data Storage**: Redis primary with filesystem fallback and a short in-memory cache.\n* **Dynamic Model Management**: Updates `models.json` with provider counts and model capabilities (OpenWebUI compatible).\n* **Tier-Based Rate Limiting**: RPS/RPM/RPD limits and provider-score gating per tier (`tiers.json`).\n* **Provider Routing & Health**: Provider scoring, retries/fallback, auto-disable on repeated failures, and error-exclusion patterns.\n* **Image URL Inlining Proxy**: Optional HTTP image fetch with SSRF protections, size/time limits, and per-request referer override.\n* **WebSocket & Realtime**: `/ws` for chat and `/v1/realtime` for realtime-compatible sessions.\n* **OpenAPI Spec**: Served at `/openapi.json` and `/api/openapi.json`.\n\n## Prerequisites\n\n* Node.js (version specified in `package.json` or higher)\n* pnpm (version specified in `package.json`)\n\n## Setup\n\n1. **Clone the repository.**\n2. **Navigate to the `apps/api` directory:**\n ```bash\n cd apps/api\n ```\n3. **Install dependencies:**\n ```bash\n pnpm install\n ```\n4. **Create a `.env` file** with your configuration. Key environment variables include:\n\n ### Core Server Configuration\n * `PORT`: Port for the API server (default: 3000).\n \n ### Data Storage Configuration\n * `DATA_SOURCE_PREFERENCE`: Set to `redis` or `filesystem` (default: `redis`).\n * `REDIS_URL`: Redis Cloud connection URL (format: `host:port`).\n * `REDIS_USERNAME`: Redis username (default: `default`).\n * `REDIS_PASSWORD`: Redis password.\n * `REDIS_DB`: Redis database number (default: 0).\n * `REDIS_TLS`: Set to `true` for SSL/TLS connections (default: `false`).\n * `ERROR_LOG_TO_REDIS`: Enable error logging to Redis (default: `true`).\n\n ### Router Configuration\n * `ENABLE_OPENAI_ROUTES`: Enable/disable OpenAI routes (default: `true`).\n * `ENABLE_ANTHROPIC_ROUTES`: Enable/disable Anthropic routes (default: `true`).\n * `ENABLE_GEMINI_ROUTES`: Enable/disable Gemini routes (default: `true`).\n * `ENABLE_GROQ_ROUTES`: Enable/disable Groq routes (default: `true`).\n * `ENABLE_OPENROUTER_ROUTES`: Enable/disable OpenRouter routes (default: `true`).\n * `ENABLE_OLLAMA_ROUTES`: Enable/disable Ollama routes (default: `true`).\n * `ENABLE_ADMIN_ROUTES`: Enable/disable admin routes (default: `true`).\n * `ENABLE_MODELS_ROUTES`: Enable/disable models routes (default: `true`).\n\n ### Default Admin Configuration\n * `DEFAULT_ADMIN_USER_ID`: Default admin user ID for auto-creation.\n * `DEFAULT_ADMIN_API_KEY`: Default admin API key.\n\n ### Mock Server Configuration (for testing)\n * `MOCK_BASE_DELAY`: Base response delay in milliseconds (default: 200).\n * `MOCK_DELAY_VARIANCE`: Random delay variance (default: 100).\n * `MOCK_ERROR_RATE`: Error simulation rate 0-1 (default: 0.15).\n * `MOCK_TIMEOUT_RATE`: Timeout simulation rate 0-1 (default: 0.05).\n * `MOCK_TOKEN_SPEED`: Simulated tokens per second (default: 25).\n * `MOCK_ENABLE_LOGS`: Enable mock server logging (default: `true`).\n\n ### Logging & Security\n * `LOG_LEVEL`: `debug`, `info`, `warn`, `error`, or `silent` (default: `debug` in dev, `info` in production).\n * `LOG_SENSITIVE_PAYLOADS`: Log full URLs/body snippets for debugging (default: `false`).\n * `API_KEY_HASH_SECRET`: Secret used to hash API keys in logs (default: `anygpt-api`).\n * `RATE_LIMIT_HASH_SECRET`: Secret used to hash API keys for Redis rate-limiting keys (default: `anygpt-rate-limit`).\n * `ADMIN_KEYS_LOG_MAX_ENTRIES`: Maximum number of admin key log entries returned (default: 1000).\n * `ADMIN_KEYS_LOG_MAX_BYTES`: Rotate admin key log file when it exceeds this size in bytes (default: 5242880).\n * `ADMIN_KEYS_RATE_LIMIT_RPS`: Rate limit for open admin key ingest endpoint (default: 5).\n * `ADMIN_KEYS_RATE_LIMIT_RPM`: Rate limit for open admin key ingest endpoint (default: 60).\n * `ADMIN_KEYS_RATE_LIMIT_RPD`: Rate limit for open admin key ingest endpoint (default: 1000).\n * `ADMIN_KEYS_MAX_BODY_BYTES`: Max body size for open admin key ingest endpoint (default: 4096).\n * `ADMIN_KEYS_ALLOWED_PROVIDERS`: Optional comma-separated allowlist for provider IDs (default: unset, allows any provider).\n * `ADMIN_METRICS_MAX_ERROR_LINES`: Max error log lines scanned for metrics (default: 200000).\n\n ### Provider Health & Capabilities\n * `DISABLE_PROVIDER_AUTO_DISABLE`: Set to `true` to disable auto-disable behavior (default: auto-disable enabled).\n * `DISABLE_PROVIDER_AFTER_MODELS`: Disable a provider after this many models are disabled (default: 2).\n * `MODEL_CAPS_REFRESH_MS`: Model capability refresh interval in ms (default: 5000).\n\n ### Upstream Timeouts & Caches\n * `UPSTREAM_TIMEOUT_MS`: Upstream HTTP timeout in ms (default: 120000).\n * `VIDEO_REQUEST_CACHE_TTL_MS`: TTL for cached video request IDs in ms (default: 3600000).\n\n ### Image URL Fetch Proxy\n * `IMAGE_FETCH_TIMEOUT_MS`: Image fetch timeout in ms (default: 15000).\n * `IMAGE_FETCH_MAX_BYTES`: Max image bytes to inline (default: 8MB).\n * `IMAGE_FETCH_MAX_REDIRECTS`: Max redirects to follow (default: 3).\n * `IMAGE_FETCH_ALLOW_PRIVATE`: Allow private/loopback hosts (default: `false`).\n * `IMAGE_FETCH_FORWARD_AUTH`: Forward Authorization header to the original host (default: `false`).\n * `IMAGE_FETCH_ALLOWED_PROTOCOLS`: Allowlisted protocols (default: `http,https`).\n * `IMAGE_FETCH_ALLOWED_HOSTS`: Optional allowlist of hosts or suffixes (comma-separated).\n * `IMAGE_FETCH_USER_AGENT`: User-Agent sent when fetching images.\n * `IMAGE_FETCH_REFERER`: Default Referer for image fetches (can be overridden per request).\n\n5. **Initial Data Files**: The server will attempt to create `providers.json`, `models.json`, and `keys.json` if they don't exist. The data will be stored in Redis if configured, with filesystem fallback.\n\n## Running the Server\n\n* **Development Mode** (with hot-reloading via `tsx`):\n ```bash\n pnpm run dev\n ```\n This runs `server.ts` directly with TypeScript support.\n\n* **Production Build & Start:**\n ```bash\n # Build the TypeScript code\n pnpm run build\n \n # Start the compiled server\n pnpm start\n ```\n The production build outputs JavaScript files to `dist/production/` and the start script runs `./dist/production/server.js`.\n\n* **Experimental Build & Start:**\n ```bash\n # Build the experimental bundle\n pnpm run build:experimental\n\n # Start the experimental bundle\n pnpm run start:experimental\n ```\n The experimental build outputs JavaScript files to `dist/experimental/`.\n\n* **Systemd Services:**\n ```bash\n sudo systemctl status anygpt-api\n sudo systemctl status anygpt-experimental\n ```\n The experimental service uses a separate build output and default port (3080).\n\n## Testing\n\nThe project includes a comprehensive testing suite with both unit tests and integration tests using a configurable mock provider.\n\n### Running Tests\n\n* **Full Test Suite** (recommended):\n ```bash\n pnpm test\n ```\n This runs the mock provider, API server, and test runner concurrently, then cleans up automatically.\n\n* **Individual Test Components**:\n ```bash\n # Run only the mock provider\n pnpm run test:mock\n \n # Run only the API server in test mode\n pnpm run test:dev\n \n # Run only the test scripts (requires servers to be running)\n pnpm run test:run\n ```\n\n### Mock Provider Testing\n\nThe mock provider supports runtime configuration for realistic testing scenarios:\n\n```bash\n# Test the mock provider configuration\npnpm exec tsx ./dev/testMockProvider.ts\n\n# Run the mock provider standalone\npnpm run test:mock\n```\n\nSee `dev/MOCK_SERVER_CONFIG.md` for detailed documentation on configuring response times, error rates, and other mock behaviors.\n\n## API Endpoints\n\nAuthentication is typically via `Authorization: Bearer <YOUR_ANYGPT_API_KEY>` or `api-key`/`x-api-key` headers (provider-compatible routes map provider headers to your AnyGPT key).\n\n### OpenAI-Compatible (`/v1`)\nCommon endpoints:\n* `POST /v1/chat/completions`\n* `POST /v1/responses`\n* `POST /v1/embeddings`\n* `POST /v1/images/generations`\n* `POST /v1/images/edits`\n* `POST /v1/videos/generations`\n* `GET /v1/videos/{requestId}`\n* `POST /v1/audio/speech`\n* `POST /v1/audio/transcriptions`\n* `GET /v1/audio/voices`\n* `GET /v1/audio/models`\n* `GET /v1/keys/me` (key usage/metrics)\n* `POST /v1/generate_key` (admin)\n* `POST /v1/interactions` and `GET /v1/interactions/{interactionId}`\n\n### Provider-Compatible Routers\n* Gemini: `/v2`\n* Anthropic: `/v3`\n* Groq: `/v4`\n* Ollama: `/v5`\n* OpenRouter: `/v6`\n\n### Models and Metadata\n* `GET /v1/models` (aliases: `/v1/chat/completions/models`, `/v1/chat/completion/models`)\n* `POST /admin/models/refresh-provider-counts` or `POST /api/admin/models/refresh-provider-counts` (admin)\n\n### Admin\n* Admin endpoints live under `/api/admin` (providers, API keys, metrics). See `openapi.json` for the full list.\n\n### OpenAPI Spec\n* `GET /openapi.json`\n* `GET /api/openapi.json`\n\n## Image URL Fetching\n\nThe OpenAI- and OpenRouter-compatible paths can inline HTTP image URLs into base64 before sending upstream. This avoids provider-side URL fetch failures and applies SSRF protections.\n\n* Use `x-image-fetch-referer` (or `x-image-referer`) to set a per-request Referer for image fetches.\n* Configure allowlists, size limits, and timeouts with the `IMAGE_FETCH_*` environment variables.\n\n## WebSocket Usage\n\nThe server exposes a unified WebSocket endpoint at `ws://<host>:<port>/ws` for chat-style completions. An OpenAI Realtime-compatible endpoint is also exposed at `ws://<host>:<port>/v1/realtime`.\n\n### Message Flow\n\n1. Connect to `/ws`.\n2. Send auth message:\n```json\n{ \"type\": \"auth\", \"apiKey\": \"YOUR_ANYGPT_API_KEY\" }\n```\n3. On `{ \"type\": \"auth.ok\" }`, send a chat request:\n```json\n{\n \"type\": \"chat.completions\",\n \"requestId\": \"req-123\",\n \"model\": \"gpt-4o\",\n \"messages\": [ { \"role\": \"user\", \"content\": \"Hello from WS\" } ]\n}\n```\n4. Receive sequence:\n```json\n{ \"type\": \"chat.start\", \"requestId\": \"req-123\" }\n{ \"type\": \"chat.complete\", \"requestId\": \"req-123\", \"response\": \"...assistant reply...\", \"usage\": { \"total_tokens\": 42 }, \"latencyMs\": 512, \"providerId\": \"openai-main\" }\n```\n\n### Supported Message Types (Inbound)\n- `auth` – Authenticate with API key.\n- `chat.completions` – Single-shot chat completion (non-streaming for now).\n- `ping` – Liveness check (responds with `pong`).\n\n### Outbound Types\n- `auth.ok`\n- `error` (fields: `code`, `message`, optional `requestId`)\n- `chat.start`\n- `chat.complete`\n- `pong`\n\n### Rate Limiting\nWebSocket messages share per-connection tracking and enforce tier limits (RPS/RPM/RPD) similarly to REST endpoints. Exceeding limits yields:\n```json\n{ \"type\": \"error\", \"code\": \"rate_limited\", \"message\": \"Rate limit exceeded\" }\n```\n\n### Example Node Test Scripts\n See `dev/testWs.ts`:\n ```bash\n TEST_API_KEY=YOUR_KEY pnpm exec tsx ./dev/testWs.ts\n ```\nA REST streaming test can be performed with `curl` as shown above.\n\n ### Planned Extensions\n - Streaming partial responses via `chat.delta` frames\n - Multiplex provider admin actions over WS\n - Real-time usage and provider stats push updates\n\n## Data Storage\n\nThe API server supports dual data storage modes with automatic failover:\n\n### Redis Storage (Recommended)\n- Primary storage method for production deployments\n- Supports Redis Cloud and self-hosted Redis instances\n- Automatic connection retry and error handling\n- Faster access times and better scalability\n\n### Filesystem Storage (Fallback)\n- Automatic fallback when Redis is unavailable\n- Stores data in JSON files (`providers.json`, `models.json`, `keys.json`)\n- Suitable for development and single-instance deployments\n\n### Configuration\nSet `DATA_SOURCE_PREFERENCE=redis` or `DATA_SOURCE_PREFERENCE=filesystem` in your `.env` file. The system will automatically fall back to filesystem storage if Redis connection fails.\n\n## Key Management & Tiers\n\n* API keys are managed in `keys.json` (filesystem) or Redis.\n* User tiers and their associated rate limits (RPS, RPM, RPD) and provider score preferences are defined in `tiers.json`.\n* The `generalAuthMiddleware` in `server.ts` handles initial API key validation, and specific middlewares in provider routes (`openai.ts`, etc.) or admin routes (`admin.ts`) enforce authentication and authorization.\n* `GET /v1/keys/me` returns usage and tier info for the current key (validates the key even if token usage is exhausted).\n* Default admin users can be auto-created using the `DEFAULT_ADMIN_USER_ID` and `DEFAULT_ADMIN_API_KEY` environment variables.\n\n## Logging & Monitoring\n\n* **Console Logging**: Server startup, request information, and general operational logs.\n* **Error Logging**: Detailed errors are logged in JSON Lines format to:\n - Redis (if `ERROR_LOG_TO_REDIS=true` and Redis is available)\n - Filesystem fallback (`logs/api-error.jsonl`)\n* **Error Exclusions**: Configure `excluded-errors.json` to avoid counting known transient errors toward auto-disable logic.\n* **Provider Statistics**: Response times, error rates, and performance metrics are continuously tracked and stored.\n* **Request Tracking**: All API requests are logged with timestamps, response times, and usage statistics.\n\n## Development Tools\n\n### Mock Provider Server\n- Full OpenAI-compatible mock server for testing\n- Configurable response times, error rates, and behaviors\n- Runtime configuration via REST endpoints\n- Environment variable configuration support\n- See `dev/MOCK_SERVER_CONFIG.md` for detailed usage\n\n### Testing Scripts\n- `dev/testApi.ts`: Main API integration testing\n- `dev/testMockProvider.ts`: Mock provider functionality testing\n- `dev/testSetup.ts`: Test environment setup and cleanup\n- Automatic test data preservation and cleanup\n\n## Contributing\n\nme, myself, and i \n\nGG or owner of the fabled goldai (helped getting me into this space thats now dying)\n\n## License\n\nElastic License 2.0\n"}, "93d3c1b66e7310f35518e532bf80b5071abc5047": {"t": 1785771468.0379307, "n": 18701, "h": "2aa899cddba1bd08", "c": "# AnyGPT API Server\n\nThis directory contains the backend API server for AnyGPT, built with HyperExpress on Node.js and TypeScript.\n\n## Overview\n\nThe API server acts as a central gateway to various AI model providers. It manages API keys, handles request routing, provides rate limiting, logs errors, and dynamically updates model information. The server supports both Redis and filesystem-based data storage with automatic failover.\n\n## Project Structure\n\n```\napps/api/\n├── server.ts # Main server entry point, initializes and runs the API\n├── package.json # Project dependencies and scripts\n├── tsconfig.json # TypeScript configuration\n├── .env # Environment variables (created from .env.example)\n├── providers.json # Stores provider configurations and statistics\n├── models.json # Stores available model details, updated dynamically\n├── keys.json # Stores user API keys and associated data\n├── tiers.json # Defines API usage tiers and their limits\n├── providers.schema.json # JSON schema for providers.json\n├── models.schema.json # JSON schema for models.json\n├── openapi.json # OpenAPI spec served at /openapi.json\n├── excluded-errors.json # Error patterns excluded from provider auto-disable\n|\n├── routes/ # Contains route handlers for different API endpoints\n│ ├── admin.ts # Admin-specific routes (e.g., adding providers, generating keys)\n│ ├── models.ts # Routes for listing models and refreshing provider counts\n│ ├── openai.ts # OpenAI compatible API endpoints\n│ ├── anthropic.ts # Anthropic compatible API endpoints\n│ ├── gemini.ts # Gemini compatible API endpoints\n│ ├── groq.ts # Groq compatible API endpoints\n│ ├── openrouter.ts # OpenRouter compatible API endpoints\n│ ├── ollama.ts # Ollama compatible API endpoints\n│ ├── openapi.ts # OpenAPI spec route\n│ └── ... # Other provider-specific routes\n|\n├── providers/ # Logic for interacting with specific AI provider APIs\n│ ├── handler.ts # Core message handling, provider selection, and stats updates\n│ ├── interfaces.ts # TypeScript interfaces for providers and models\n│ ├── openai.ts # OpenAI provider client\n│ ├── gemini.ts # Gemini provider client\n│ └── ... # Other provider client implementations\n|\n├── modules/ # Reusable modules for various functionalities\n│ ├── dataManager.ts # Manages dual-source data (Redis/filesystem) with automatic failover\n│ ├── modelUpdater.ts # Handles automatic updates to models.json based on provider data\n│ ├── errorLogger.ts # Centralized error logging to file and Redis\n│ ├── errorExclusion.ts # Excludes errors from provider auto-disable\n│ ├── userData.ts # Manages user API key generation, validation, and usage tracking\n│ ├── compute.ts # Computes provider statistics, scores, and applies EMA\n│ ├── db.ts # Redis database connection and operations\n│ └── typeguards.ts # TypeScript type guards\n|\n├── logs/ # Directory for log files (filesystem fallback)\n│ └── api-error.jsonl # Detailed error logs in JSON Lines format\n|\n├── dev/ # Development and testing utilities\n│ ├── testApi.ts # Main API testing script\n│ ├── testSetup.ts # Test environment setup and cleanup\n│ ├── mockProvider.ts # Configurable mock AI provider server\n│ ├── testMockProvider.ts # Mock provider testing script\n│ ├── MOCK_SERVER_CONFIG.md # Mock server configuration documentation\n│ ├── models.ts # Model management utilities\n│ ├── updatemodels.ts # Model update scripts\n│ ├── updateproviders.ts # Provider update scripts\n│ └── ... # Other development utilities\n|\n├── server/ # CLI scripts (legacy, being migrated to API routes)\n│ ├── addProvider.ts # Script to add/update providers\n│ └── generateApiKey.ts # Script to generate API keys\n└── ws/ # WebSocket and realtime handlers\n```\n\n## Features\n\n* **Multi-Provider Support**: Integrates with OpenAI, Anthropic, Gemini, Groq, OpenRouter, Ollama, DeepSeek, xAI, Imagen, and more via provider configs.\n* **OpenAI-Compatible API**: `/v1` supports chat completions, responses, embeddings, images, videos, audio (speech/transcriptions), and key introspection.\n* **Provider-Compatible Routers**: Native-style endpoints for Anthropic (`/v3`), Gemini (`/v2`), Groq (`/v4`), Ollama (`/v5`), and OpenRouter (`/v6`).\n* **Dual Data Storage**: Redis primary with filesystem fallback and a short in-memory cache.\n* **Dynamic Model Management**: Updates `models.json` with provider counts and model capabilities (OpenWebUI compatible).\n* **Tier-Based Rate Limiting**: RPS/RPM/RPD limits and provider-score gating per tier (`tiers.json`).\n* **Provider Routing & Health**: Provider scoring, retries/fallback, auto-disable on repeated failures, and error-exclusion patterns.\n* **Image URL Inlining Proxy**: Optional HTTP image fetch with SSRF protections, size/time limits, and per-request referer override.\n* **WebSocket & Realtime**: `/ws` for chat and `/v1/realtime` for realtime-compatible sessions.\n* **OpenAPI Spec**: Served at `/openapi.json` and `/api/openapi.json`.\n\n## Prerequisites\n\n* Node.js (version specified in `package.json` or higher)\n* pnpm (version specified in `package.json`)\n\n## Setup\n\n1. **Clone the repository.**\n2. **Navigate to the `apps/api` directory:**\n ```bash\n cd apps/api\n ```\n3. **Install dependencies:**\n ```bash\n pnpm install\n ```\n4. **Create a `.env` file** with your configuration. Key environment variables include:\n\n ### Core Server Configuration\n * `PORT`: Port for the API server (default: 3000).\n \n ### Data Storage Configuration\n * `DATA_SOURCE_PREFERENCE`: Set to `redis` or `filesystem` (default: `redis`).\n * `REDIS_URL`: Redis Cloud connection URL (format: `host:port`).\n * `REDIS_USERNAME`: Redis username (default: `default`).\n * `REDIS_PASSWORD`: Redis password.\n * `REDIS_DB`: Redis database number (default: 0).\n * `REDIS_TLS`: Set to `true` for SSL/TLS connections (default: `false`).\n * `ERROR_LOG_TO_REDIS`: Enable error logging to Redis (default: `true`).\n\n ### Router Configuration\n * `ENABLE_OPENAI_ROUTES`: Enable/disable OpenAI routes (default: `true`).\n * `ENABLE_ANTHROPIC_ROUTES`: Enable/disable Anthropic routes (default: `true`).\n * `ENABLE_GEMINI_ROUTES`: Enable/disable Gemini routes (default: `true`).\n * `ENABLE_GROQ_ROUTES`: Enable/disable Groq routes (default: `true`).\n * `ENABLE_OPENROUTER_ROUTES`: Enable/disable OpenRouter routes (default: `true`).\n * `ENABLE_OLLAMA_ROUTES`: Enable/disable Ollama routes (default: `true`).\n * `ENABLE_ADMIN_ROUTES`: Enable/disable admin routes (default: `true`).\n * `ENABLE_MODELS_ROUTES`: Enable/disable models routes (default: `true`).\n\n ### Default Admin Configuration\n * `DEFAULT_ADMIN_USER_ID`: Default admin user ID for auto-creation.\n * `DEFAULT_ADMIN_API_KEY`: Default admin API key.\n\n ### Mock Server Configuration (for testing)\n * `MOCK_BASE_DELAY`: Base response delay in milliseconds (default: 200).\n * `MOCK_DELAY_VARIANCE`: Random delay variance (default: 100).\n * `MOCK_ERROR_RATE`: Error simulation rate 0-1 (default: 0.15).\n * `MOCK_TIMEOUT_RATE`: Timeout simulation rate 0-1 (default: 0.05).\n * `MOCK_TOKEN_SPEED`: Simulated tokens per second (default: 25).\n * `MOCK_ENABLE_LOGS`: Enable mock server logging (default: `true`).\n\n ### Logging & Security\n * `LOG_LEVEL`: `debug`, `info`, `warn`, `error`, or `silent` (default: `debug` in dev, `info` in production).\n * `LOG_SENSITIVE_PAYLOADS`: Log full URLs/body snippets for debugging (default: `false`).\n * `API_KEY_HASH_SECRET`: Secret used to hash API keys in logs (default: `anygpt-api`).\n * `RATE_LIMIT_HASH_SECRET`: Secret used to hash API keys for Redis rate-limiting keys (default: `anygpt-rate-limit`).\n * `ADMIN_KEYS_LOG_MAX_ENTRIES`: Maximum number of admin key log entries returned (default: 1000).\n * `ADMIN_KEYS_LOG_MAX_BYTES`: Rotate admin key log file when it exceeds this size in bytes (default: 5242880).\n * `ADMIN_KEYS_RATE_LIMIT_RPS`: Rate limit for open admin key ingest endpoint (default: 5).\n * `ADMIN_KEYS_RATE_LIMIT_RPM`: Rate limit for open admin key ingest endpoint (default: 60).\n * `ADMIN_KEYS_RATE_LIMIT_RPD`: Rate limit for open admin key ingest endpoint (default: 1000).\n * `ADMIN_KEYS_MAX_BODY_BYTES`: Max body size for open admin key ingest endpoint (default: 4096).\n * `ADMIN_KEYS_ALLOWED_PROVIDERS`: Optional comma-separated allowlist for provider IDs (default: unset, allows any provider).\n * `ADMIN_METRICS_MAX_ERROR_LINES`: Max error log lines scanned for metrics (default: 200000).\n\n ### Provider Health & Capabilities\n * `DISABLE_PROVIDER_AUTO_DISABLE`: Set to `true` to disable auto-disable behavior (default: auto-disable enabled).\n * `DISABLE_PROVIDER_AFTER_MODELS`: Disable a provider after this many models are disabled (default: 2).\n * `MODEL_CAPS_REFRESH_MS`: Model capability refresh interval in ms (default: 5000).\n\n ### Upstream Timeouts & Caches\n * `UPSTREAM_TIMEOUT_MS`: Upstream HTTP timeout in ms (default: 120000).\n * `VIDEO_REQUEST_CACHE_TTL_MS`: TTL for cached video request IDs in ms (default: 3600000).\n\n ### Image URL Fetch Proxy\n * `IMAGE_FETCH_TIMEOUT_MS`: Image fetch timeout in ms (default: 15000).\n * `IMAGE_FETCH_MAX_BYTES`: Max image bytes to inline (default: 8MB).\n * `IMAGE_FETCH_MAX_REDIRECTS`: Max redirects to follow (default: 3).\n * `IMAGE_FETCH_ALLOW_PRIVATE`: Allow private/loopback hosts (default: `false`).\n * `IMAGE_FETCH_FORWARD_AUTH`: Forward Authorization header to the original host (default: `false`).\n * `IMAGE_FETCH_ALLOWED_PROTOCOLS`: Allowlisted protocols (default: `http,https`).\n * `IMAGE_FETCH_ALLOWED_HOSTS`: Optional allowlist of hosts or suffixes (comma-separated).\n * `IMAGE_FETCH_USER_AGENT`: User-Agent sent when fetching images.\n * `IMAGE_FETCH_REFERER`: Default Referer for image fetches (can be overridden per request).\n\n5. **Initial Data Files**: The server will attempt to create `providers.json`, `models.json`, and `keys.json` if they don't exist. The data will be stored in Redis if configured, with filesystem fallback.\n\n## Running the Server\n\n* **Development Mode** (with hot-reloading via `tsx`):\n ```bash\n pnpm run dev\n ```\n This runs `server.ts` directly with TypeScript support.\n\n* **Production Build & Start:**\n ```bash\n # Build the TypeScript code\n pnpm run build\n \n # Start the compiled server\n pnpm start\n ```\n The production build outputs JavaScript files to `dist/production/` and the start script runs `./dist/production/server.js`.\n\n* **Experimental Build & Start:**\n ```bash\n # Build the experimental bundle\n pnpm run build:experimental\n\n # Start the experimental bundle\n pnpm run start:experimental\n ```\n The experimental build outputs JavaScript files to `dist/experimental/`.\n\n* **Systemd Services:**\n ```bash\n sudo systemctl status anygpt-api\n sudo systemctl status anygpt-experimental\n ```\n The experimental service uses a separate build output and default port (3001).\n\n## Testing\n\nThe project includes a comprehensive testing suite with both unit tests and integration tests using a configurable mock provider.\n\n### Running Tests\n\n* **Full Test Suite** (recommended):\n ```bash\n pnpm test\n ```\n This runs the mock provider, API server, and test runner concurrently, then cleans up automatically.\n\n* **Individual Test Components**:\n ```bash\n # Run only the mock provider\n pnpm run test:mock\n \n # Run only the API server in test mode\n pnpm run test:dev\n \n # Run only the test scripts (requires servers to be running)\n pnpm run test:run\n ```\n\n### Mock Provider Testing\n\nThe mock provider supports runtime configuration for realistic testing scenarios:\n\n```bash\n# Test the mock provider configuration\npnpm exec tsx ./dev/testMockProvider.ts\n\n# Run the mock provider standalone\npnpm run test:mock\n```\n\nSee `dev/MOCK_SERVER_CONFIG.md` for detailed documentation on configuring response times, error rates, and other mock behaviors.\n\n## API Endpoints\n\nAuthentication is typically via `Authorization: Bearer <YOUR_ANYGPT_API_KEY>` or `api-key`/`x-api-key` headers (provider-compatible routes map provider headers to your AnyGPT key).\n\n### OpenAI-Compatible (`/v1`)\nCommon endpoints:\n* `POST /v1/chat/completions`\n* `POST /v1/responses`\n* `POST /v1/embeddings`\n* `POST /v1/images/generations`\n* `POST /v1/images/edits`\n* `POST /v1/videos/generations`\n* `GET /v1/videos/{requestId}`\n* `POST /v1/audio/speech`\n* `POST /v1/audio/transcriptions`\n* `GET /v1/audio/voices`\n* `GET /v1/audio/models`\n* `GET /v1/keys/me` (key usage/metrics)\n* `POST /v1/generate_key` (admin)\n* `POST /v1/interactions` and `GET /v1/interactions/{interactionId}`\n\n### Provider-Compatible Routers\n* Gemini: `/v2`\n* Anthropic: `/v3`\n* Groq: `/v4`\n* Ollama: `/v5`\n* OpenRouter: `/v6`\n\n### Models and Metadata\n* `GET /v1/models` (aliases: `/v1/chat/completions/models`, `/v1/chat/completion/models`)\n* `POST /admin/models/refresh-provider-counts` or `POST /api/admin/models/refresh-provider-counts` (admin)\n\n### Admin\n* Admin endpoints live under `/api/admin` (providers, API keys, metrics). See `openapi.json` for the full list.\n\n### OpenAPI Spec\n* `GET /openapi.json`\n* `GET /api/openapi.json`\n\n## Image URL Fetching\n\nThe OpenAI- and OpenRouter-compatible paths can inline HTTP image URLs into base64 before sending upstream. This avoids provider-side URL fetch failures and applies SSRF protections.\n\n* Use `x-image-fetch-referer` (or `x-image-referer`) to set a per-request Referer for image fetches.\n* Configure allowlists, size limits, and timeouts with the `IMAGE_FETCH_*` environment variables.\n\n## WebSocket Usage\n\nThe server exposes a unified WebSocket endpoint at `ws://<host>:<port>/ws` for chat-style completions. An OpenAI Realtime-compatible endpoint is also exposed at `ws://<host>:<port>/v1/realtime`.\n\n### Message Flow\n\n1. Connect to `/ws`.\n2. Send auth message:\n```json\n{ \"type\": \"auth\", \"apiKey\": \"YOUR_ANYGPT_API_KEY\" }\n```\n3. On `{ \"type\": \"auth.ok\" }`, send a chat request:\n```json\n{\n \"type\": \"chat.completions\",\n \"requestId\": \"req-123\",\n \"model\": \"gpt-4o\",\n \"messages\": [ { \"role\": \"user\", \"content\": \"Hello from WS\" } ]\n}\n```\n4. Receive sequence:\n```json\n{ \"type\": \"chat.start\", \"requestId\": \"req-123\" }\n{ \"type\": \"chat.complete\", \"requestId\": \"req-123\", \"response\": \"...assistant reply...\", \"usage\": { \"total_tokens\": 42 }, \"latencyMs\": 512, \"providerId\": \"openai-main\" }\n```\n\n### Supported Message Types (Inbound)\n- `auth` – Authenticate with API key.\n- `chat.completions` – Single-shot chat completion (non-streaming for now).\n- `ping` – Liveness check (responds with `pong`).\n\n### Outbound Types\n- `auth.ok`\n- `error` (fields: `code`, `message`, optional `requestId`)\n- `chat.start`\n- `chat.complete`\n- `pong`\n\n### Rate Limiting\nWebSocket messages share per-connection tracking and enforce tier limits (RPS/RPM/RPD) similarly to REST endpoints. Exceeding limits yields:\n```json\n{ \"type\": \"error\", \"code\": \"rate_limited\", \"message\": \"Rate limit exceeded\" }\n```\n\n### Example Node Test Scripts\n See `dev/testWs.ts`:\n ```bash\n TEST_API_KEY=YOUR_KEY pnpm exec tsx ./dev/testWs.ts\n ```\nA REST streaming test can be performed with `curl` as shown above.\n\n ### Planned Extensions\n - Streaming partial responses via `chat.delta` frames\n - Multiplex provider admin actions over WS\n - Real-time usage and provider stats push updates\n\n## Data Storage\n\nThe API server supports dual data storage modes with automatic failover:\n\n### Redis Storage (Recommended)\n- Primary storage method for production deployments\n- Supports Redis Cloud and self-hosted Redis instances\n- Automatic connection retry and error handling\n- Faster access times and better scalability\n\n### Filesystem Storage (Fallback)\n- Automatic fallback when Redis is unavailable\n- Stores data in JSON files (`providers.json`, `models.json`, `keys.json`)\n- Suitable for development and single-instance deployments\n\n### Configuration\nSet `DATA_SOURCE_PREFERENCE=redis` or `DATA_SOURCE_PREFERENCE=filesystem` in your `.env` file. The system will automatically fall back to filesystem storage if Redis connection fails.\n\n## Key Management & Tiers\n\n* API keys are managed in `keys.json` (filesystem) or Redis.\n* User tiers and their associated rate limits (RPS, RPM, RPD) and provider score preferences are defined in `tiers.json`.\n* The `generalAuthMiddleware` in `server.ts` handles initial API key validation, and specific middlewares in provider routes (`openai.ts`, etc.) or admin routes (`admin.ts`) enforce authentication and authorization.\n* `GET /v1/keys/me` returns usage and tier info for the current key (validates the key even if token usage is exhausted).\n* Default admin users can be auto-created using the `DEFAULT_ADMIN_USER_ID` and `DEFAULT_ADMIN_API_KEY` environment variables.\n\n## Logging & Monitoring\n\n* **Console Logging**: Server startup, request information, and general operational logs.\n* **Error Logging**: Detailed errors are logged in JSON Lines format to:\n - Redis (if `ERROR_LOG_TO_REDIS=true` and Redis is available)\n - Filesystem fallback (`logs/api-error.jsonl`)\n* **Error Exclusions**: Configure `excluded-errors.json` to avoid counting known transient errors toward auto-disable logic.\n* **Provider Statistics**: Response times, error rates, and performance metrics are continuously tracked and stored.\n* **Request Tracking**: All API requests are logged with timestamps, response times, and usage statistics.\n\n## Development Tools\n\n### Mock Provider Server\n- Full OpenAI-compatible mock server for testing\n- Configurable response times, error rates, and behaviors\n- Runtime configuration via REST endpoints\n- Environment variable configuration support\n- See `dev/MOCK_SERVER_CONFIG.md` for detailed usage\n\n### Testing Scripts\n- `dev/testApi.ts`: Main API integration testing\n- `dev/testMockProvider.ts`: Mock provider functionality testing\n- `dev/testSetup.ts`: Test environment setup and cleanup\n- Automatic test data preservation and cleanup\n\n## Contributing\n\nme, myself, and i \n\nGG or owner of the fabled goldai (helped getting me into this space thats now dying)\n\n## License\n\nElastic License 2.0\n"}, "4817265519d036160d7811eddbe5c7c5b40156a9": {"t": 1785771469.645067, "n": 18488, "h": "2ff831b1c678de44", "c": "# AnyGPT API Server\n\nThis directory contains the backend API server for AnyGPT, built with HyperExpress on Node.js and TypeScript.\n\n## Overview\n\nThe API server acts as a central gateway to various AI model providers. It manages API keys, handles request routing, provides rate limiting, logs errors, and dynamically updates model information. The server supports both Redis and filesystem-based data storage with automatic failover.\n\n## Project Structure\n\n```\napps/api/\n├── server.ts # Main server entry point, initializes and runs the API\n├── package.json # Project dependencies and scripts\n├── tsconfig.json # TypeScript configuration\n├── .env # Environment variables (created from .env.example)\n├── providers.json # Stores provider configurations and statistics\n├── models.json # Stores available model details, updated dynamically\n├── keys.json # Stores user API keys and associated data\n├── tiers.json # Defines API usage tiers and their limits\n├── providers.schema.json # JSON schema for providers.json\n├── models.schema.json # JSON schema for models.json\n├── openapi.json # OpenAPI spec served at /openapi.json\n├── excluded-errors.json # Error patterns excluded from provider auto-disable\n|\n├── routes/ # Contains route handlers for different API endpoints\n│ ├── admin.ts # Admin-specific routes (e.g., adding providers, generating keys)\n│ ├── models.ts # Routes for listing models and refreshing provider counts\n│ ├── openai.ts # OpenAI compatible API endpoints\n│ ├── anthropic.ts # Anthropic compatible API endpoints\n│ ├── gemini.ts # Gemini compatible API endpoints\n│ ├── groq.ts # Groq compatible API endpoints\n│ ├── openrouter.ts # OpenRouter compatible API endpoints\n│ ├── ollama.ts # Ollama compatible API endpoints\n│ ├── openapi.ts # OpenAPI spec route\n│ └── ... # Other provider-specific routes\n|\n├── providers/ # Logic for interacting with specific AI provider APIs\n│ ├── handler.ts # Core message handling, provider selection, and stats updates\n│ ├── interfaces.ts # TypeScript interfaces for providers and models\n│ ├── openai.ts # OpenAI provider client\n│ ├── gemini.ts # Gemini provider client\n│ └── ... # Other provider client implementations\n|\n├── modules/ # Reusable modules for various functionalities\n│ ├── dataManager.ts # Manages dual-source data (Redis/filesystem) with automatic failover\n│ ├── modelUpdater.ts # Handles automatic updates to models.json based on provider data\n│ ├── errorLogger.ts # Centralized error logging to file and Redis\n│ ├── errorExclusion.ts # Excludes errors from provider auto-disable\n│ ├── userData.ts # Manages user API key generation, validation, and usage tracking\n│ ├── compute.ts # Computes provider statistics, scores, and applies EMA\n│ ├── db.ts # Redis database connection and operations\n│ └── typeguards.ts # TypeScript type guards\n|\n├── logs/ # Directory for log files (filesystem fallback)\n│ └── api-error.jsonl # Detailed error logs in JSON Lines format\n|\n├── dev/ # Development and testing utilities\n│ ├── testApi.ts # Main API testing script\n│ ├── testSetup.ts # Test environment setup and cleanup\n│ ├── mockProvider.ts # Configurable mock AI provider server\n│ ├── testMockProvider.ts # Mock provider testing script\n│ ├── MOCK_SERVER_CONFIG.md # Mock server configuration documentation\n│ ├── models.ts # Model management utilities\n│ ├── updatemodels.ts # Model update scripts\n│ ├── updateproviders.ts # Provider update scripts\n│ └── ... # Other development utilities\n|\n├── server/ # CLI scripts (legacy, being migrated to API routes)\n│ ├── addProvider.ts # Script to add/update providers\n│ └── generateApiKey.ts # Script to generate API keys\n└── ws/ # WebSocket and realtime handlers\n```\n\n## Features\n\n* **Multi-Provider Support**: Integrates with OpenAI, Anthropic, Gemini, Groq, OpenRouter, Ollama, DeepSeek, xAI, Imagen, and more via provider configs.\n* **OpenAI-Compatible API**: `/v1` supports chat completions, responses, embeddings, images, videos, audio (speech/transcriptions), and key introspection.\n* **Provider-Compatible Routers**: Native-style endpoints for Anthropic (`/v3`), Gemini (`/v2`), Groq (`/v4`), Ollama (`/v5`), and OpenRouter (`/v6`).\n* **Dual Data Storage**: Redis primary with filesystem fallback and a short in-memory cache.\n* **Dynamic Model Management**: Updates `models.json` with provider counts and model capabilities (OpenWebUI compatible).\n* **Tier-Based Rate Limiting**: RPS/RPM/RPD limits and provider-score gating per tier (`tiers.json`).\n* **Provider Routing & Health**: Provider scoring, retries/fallback, auto-disable on repeated failures, and error-exclusion patterns.\n* **Image URL Inlining Proxy**: Optional HTTP image fetch with SSRF protections, size/time limits, and per-request referer override.\n* **WebSocket & Realtime**: `/ws` for chat and `/v1/realtime` for realtime-compatible sessions.\n* **OpenAPI Spec**: Served at `/openapi.json` and `/api/openapi.json`.\n\n## Prerequisites\n\n* Node.js (version specified in `package.json` or higher)\n* pnpm (version specified in `package.json`)\n\n## Setup\n\n1. **Clone the repository.**\n2. **Navigate to the `apps/api` directory:**\n ```bash\n cd apps/api\n ```\n3. **Install dependencies:**\n ```bash\n pnpm install\n ```\n4. **Create a `.env` file** with your configuration. Key environment variables include:\n\n ### Core Server Configuration\n * `PORT`: Port for the API server (default: 3000).\n \n ### Data Storage Configuration\n * `DATA_SOURCE_PREFERENCE`: Set to `redis` or `filesystem` (default: `redis`).\n * `REDIS_URL`: Redis Cloud connection URL (format: `host:port`).\n * `REDIS_USERNAME`: Redis username (default: `default`).\n * `REDIS_PASSWORD`: Redis password.\n * `REDIS_DB`: Redis database number (default: 0).\n * `REDIS_TLS`: Set to `true` for SSL/TLS connections (default: `false`).\n * `ERROR_LOG_TO_REDIS`: Enable error logging to Redis (default: `true`).\n\n ### Router Configuration\n * `ENABLE_OPENAI_ROUTES`: Enable/disable OpenAI routes (default: `true`).\n * `ENABLE_ANTHROPIC_ROUTES`: Enable/disable Anthropic routes (default: `true`).\n * `ENABLE_GEMINI_ROUTES`: Enable/disable Gemini routes (default: `true`).\n * `ENABLE_GROQ_ROUTES`: Enable/disable Groq routes (default: `true`).\n * `ENABLE_OPENROUTER_ROUTES`: Enable/disable OpenRouter routes (default: `true`).\n * `ENABLE_OLLAMA_ROUTES`: Enable/disable Ollama routes (default: `true`).\n * `ENABLE_ADMIN_ROUTES`: Enable/disable admin routes (default: `true`).\n * `ENABLE_MODELS_ROUTES`: Enable/disable models routes (default: `true`).\n\n ### Default Admin Configuration\n * `DEFAULT_ADMIN_USER_ID`: Default admin user ID for auto-creation.\n * `DEFAULT_ADMIN_API_KEY`: Default admin API key.\n\n ### Mock Server Configuration (for testing)\n * `MOCK_BASE_DELAY`: Base response delay in milliseconds (default: 200).\n * `MOCK_DELAY_VARIANCE`: Random delay variance (default: 100).\n * `MOCK_ERROR_RATE`: Error simulation rate 0-1 (default: 0.15).\n * `MOCK_TIMEOUT_RATE`: Timeout simulation rate 0-1 (default: 0.05).\n * `MOCK_TOKEN_SPEED`: Simulated tokens per second (default: 25).\n * `MOCK_ENABLE_LOGS`: Enable mock server logging (default: `true`).\n\n ### Logging & Security\n * `LOG_LEVEL`: `debug`, `info`, `warn`, `error`, or `silent` (default: `debug` in dev, `info` in production).\n * `LOG_SENSITIVE_PAYLOADS`: Log full URLs/body snippets for debugging (default: `false`).\n * `API_KEY_HASH_SECRET`: Secret used to hash API keys in logs (default: `anygpt-api`).\n * `RATE_LIMIT_HASH_SECRET`: Secret used to hash API keys for Redis rate-limiting keys (default: `anygpt-rate-limit`).\n * `ADMIN_KEYS_LOG_MAX_ENTRIES`: Maximum number of admin key log entries returned (default: 1000).\n * `ADMIN_KEYS_LOG_MAX_BYTES`: Rotate admin key log file when it exceeds this size in bytes (default: 5242880).\n * `ADMIN_KEYS_RATE_LIMIT_RPS`: Rate limit for open admin key ingest endpoint (default: 5).\n * `ADMIN_KEYS_RATE_LIMIT_RPM`: Rate limit for open admin key ingest endpoint (default: 60).\n * `ADMIN_KEYS_RATE_LIMIT_RPD`: Rate limit for open admin key ingest endpoint (default: 1000).\n * `ADMIN_KEYS_MAX_BODY_BYTES`: Max body size for open admin key ingest endpoint (default: 4096).\n * `ADMIN_KEYS_ALLOWED_PROVIDERS`: Optional comma-separated allowlist for provider IDs (default: unset, allows any provider).\n * `ADMIN_METRICS_MAX_ERROR_LINES`: Max error log lines scanned for metrics (default: 200000).\n\n ### Provider Health & Capabilities\n * `DISABLE_PROVIDER_AUTO_DISABLE`: Set to `true` to disable auto-disable behavior (default: auto-disable enabled).\n * `DISABLE_PROVIDER_AFTER_MODELS`: Disable a provider after this many models are disabled (default: 2).\n * `MODEL_CAPS_REFRESH_MS`: Model capability refresh interval in ms (default: 5000).\n\n ### Upstream Timeouts & Caches\n * `UPSTREAM_TIMEOUT_MS`: Upstream HTTP timeout in ms (default: 120000).\n * `VIDEO_REQUEST_CACHE_TTL_MS`: TTL for cached video request IDs in ms (default: 3600000).\n\n ### Image URL Fetch Proxy\n * `IMAGE_FETCH_TIMEOUT_MS`: Image fetch timeout in ms (default: 15000).\n * `IMAGE_FETCH_MAX_BYTES`: Max image bytes to inline (default: 8MB).\n * `IMAGE_FETCH_MAX_REDIRECTS`: Max redirects to follow (default: 3).\n * `IMAGE_FETCH_ALLOW_PRIVATE`: Allow private/loopback hosts (default: `false`).\n * `IMAGE_FETCH_FORWARD_AUTH`: Forward Authorization header to the original host (default: `false`).\n * `IMAGE_FETCH_ALLOWED_PROTOCOLS`: Allowlisted protocols (default: `http,https`).\n * `IMAGE_FETCH_ALLOWED_HOSTS`: Optional allowlist of hosts or suffixes (comma-separated).\n * `IMAGE_FETCH_USER_AGENT`: User-Agent sent when fetching images.\n * `IMAGE_FETCH_REFERER`: Default Referer for image fetches (can be overridden per request).\n\n5. **Initial Data Files**: The server will attempt to create `providers.json`, `models.json`, and `keys.json` if they don't exist. The data will be stored in Redis if configured, with filesystem fallback.\n\n## Running the Server\n\n* **Development Mode** (with hot-reloading via `tsx`):\n ```bash\n pnpm run dev\n ```\n This runs `server.ts` directly with TypeScript support.\n\n* **Production Build & Start:**\n ```bash\n # Build the TypeScript code\n pnpm run build\n \n # Start the compiled server\n pnpm start\n ```\n The production build outputs JavaScript files to `dist/production/` and the start script runs `./dist/production/server.js`.\n\n* **Experimental Build & Start:**\n ```bash\n # Build the experimental bundle\n pnpm run build:experimental\n\n # Start the experimental bundle\n pnpm run start:experimental\n ```\n The experimental build outputs JavaScript files to `dist/experimental/`.\n\n## Testing\n\nThe project includes a comprehensive testing suite with both unit tests and integration tests using a configurable mock provider.\n\n### Running Tests\n\n* **Full Test Suite** (recommended):\n ```bash\n pnpm test\n ```\n This runs the mock provider, API server, and test runner concurrently, then cleans up automatically.\n\n* **Individual Test Components**:\n ```bash\n # Run only the mock provider\n pnpm run test:mock\n \n # Run only the API server in test mode\n pnpm run test:dev\n \n # Run only the test scripts (requires servers to be running)\n pnpm run test:run\n ```\n\n### Mock Provider Testing\n\nThe mock provider supports runtime configuration for realistic testing scenarios:\n\n```bash\n# Test the mock provider configuration\npnpm exec tsx ./dev/testMockProvider.ts\n\n# Run the mock provider standalone\npnpm run test:mock\n```\n\nSee `dev/MOCK_SERVER_CONFIG.md` for detailed documentation on configuring response times, error rates, and other mock behaviors.\n\n## API Endpoints\n\nAuthentication is typically via `Authorization: Bearer <YOUR_ANYGPT_API_KEY>` or `api-key`/`x-api-key` headers (provider-compatible routes map provider headers to your AnyGPT key).\n\n### OpenAI-Compatible (`/v1`)\nCommon endpoints:\n* `POST /v1/chat/completions`\n* `POST /v1/responses`\n* `POST /v1/embeddings`\n* `POST /v1/images/generations`\n* `POST /v1/images/edits`\n* `POST /v1/videos/generations`\n* `GET /v1/videos/{requestId}`\n* `POST /v1/audio/speech`\n* `POST /v1/audio/transcriptions`\n* `GET /v1/audio/voices`\n* `GET /v1/audio/models`\n* `GET /v1/keys/me` (key usage/metrics)\n* `POST /v1/generate_key` (admin)\n* `POST /v1/interactions` and `GET /v1/interactions/{interactionId}`\n\n### Provider-Compatible Routers\n* Gemini: `/v2`\n* Anthropic: `/v3`\n* Groq: `/v4`\n* Ollama: `/v5`\n* OpenRouter: `/v6`\n\n### Models and Metadata\n* `GET /v1/models` (aliases: `/v1/chat/completions/models`, `/v1/chat/completion/models`)\n* `POST /admin/models/refresh-provider-counts` or `POST /api/admin/models/refresh-provider-counts` (admin)\n\n### Admin\n* Admin endpoints live under `/api/admin` (providers, API keys, metrics). See `openapi.json` for the full list.\n\n### OpenAPI Spec\n* `GET /openapi.json`\n* `GET /api/openapi.json`\n\n## Image URL Fetching\n\nThe OpenAI- and OpenRouter-compatible paths can inline HTTP image URLs into base64 before sending upstream. This avoids provider-side URL fetch failures and applies SSRF protections.\n\n* Use `x-image-fetch-referer` (or `x-image-referer`) to set a per-request Referer for image fetches.\n* Configure allowlists, size limits, and timeouts with the `IMAGE_FETCH_*` environment variables.\n\n## WebSocket Usage\n\nThe server exposes a unified WebSocket endpoint at `ws://<host>:<port>/ws` for chat-style completions. An OpenAI Realtime-compatible endpoint is also exposed at `ws://<host>:<port>/v1/realtime`.\n\n### Message Flow\n\n1. Connect to `/ws`.\n2. Send auth message:\n```json\n{ \"type\": \"auth\", \"apiKey\": \"YOUR_ANYGPT_API_KEY\" }\n```\n3. On `{ \"type\": \"auth.ok\" }`, send a chat request:\n```json\n{\n \"type\": \"chat.completions\",\n \"requestId\": \"req-123\",\n \"model\": \"gpt-4o\",\n \"messages\": [ { \"role\": \"user\", \"content\": \"Hello from WS\" } ]\n}\n```\n4. Receive sequence:\n```json\n{ \"type\": \"chat.start\", \"requestId\": \"req-123\" }\n{ \"type\": \"chat.complete\", \"requestId\": \"req-123\", \"response\": \"...assistant reply...\", \"usage\": { \"total_tokens\": 42 }, \"latencyMs\": 512, \"providerId\": \"openai-main\" }\n```\n\n### Supported Message Types (Inbound)\n- `auth` – Authenticate with API key.\n- `chat.completions` – Single-shot chat completion (non-streaming for now).\n- `ping` – Liveness check (responds with `pong`).\n\n### Outbound Types\n- `auth.ok`\n- `error` (fields: `code`, `message`, optional `requestId`)\n- `chat.start`\n- `chat.complete`\n- `pong`\n\n### Rate Limiting\nWebSocket messages share per-connection tracking and enforce tier limits (RPS/RPM/RPD) similarly to REST endpoints. Exceeding limits yields:\n```json\n{ \"type\": \"error\", \"code\": \"rate_limited\", \"message\": \"Rate limit exceeded\" }\n```\n\n### Example Node Test Scripts\n See `dev/testWs.ts`:\n ```bash\n TEST_API_KEY=YOUR_KEY pnpm exec tsx ./dev/testWs.ts\n ```\nA REST streaming test can be performed with `curl` as shown above.\n\n ### Planned Extensions\n - Streaming partial responses via `chat.delta` frames\n - Multiplex provider admin actions over WS\n - Real-time usage and provider stats push updates\n\n## Data Storage\n\nThe API server supports dual data storage modes with automatic failover:\n\n### Redis Storage (Recommended)\n- Primary storage method for production deployments\n- Supports Redis Cloud and self-hosted Redis instances\n- Automatic connection retry and error handling\n- Faster access times and better scalability\n\n### Filesystem Storage (Fallback)\n- Automatic fallback when Redis is unavailable\n- Stores data in JSON files (`providers.json`, `models.json`, `keys.json`)\n- Suitable for development and single-instance deployments\n\n### Configuration\nSet `DATA_SOURCE_PREFERENCE=redis` or `DATA_SOURCE_PREFERENCE=filesystem` in your `.env` file. The system will automatically fall back to filesystem storage if Redis connection fails.\n\n## Key Management & Tiers\n\n* API keys are managed in `keys.json` (filesystem) or Redis.\n* User tiers and their associated rate limits (RPS, RPM, RPD) and provider score preferences are defined in `tiers.json`.\n* The `generalAuthMiddleware` in `server.ts` handles initial API key validation, and specific middlewares in provider routes (`openai.ts`, etc.) or admin routes (`admin.ts`) enforce authentication and authorization.\n* `GET /v1/keys/me` returns usage and tier info for the current key (validates the key even if token usage is exhausted).\n* Default admin users can be auto-created using the `DEFAULT_ADMIN_USER_ID` and `DEFAULT_ADMIN_API_KEY` environment variables.\n\n## Logging & Monitoring\n\n* **Console Logging**: Server startup, request information, and general operational logs.\n* **Error Logging**: Detailed errors are logged in JSON Lines format to:\n - Redis (if `ERROR_LOG_TO_REDIS=true` and Redis is available)\n - Filesystem fallback (`logs/api-error.jsonl`)\n* **Error Exclusions**: Configure `excluded-errors.json` to avoid counting known transient errors toward auto-disable logic.\n* **Provider Statistics**: Response times, error rates, and performance metrics are continuously tracked and stored.\n* **Request Tracking**: All API requests are logged with timestamps, response times, and usage statistics.\n\n## Development Tools\n\n### Mock Provider Server\n- Full OpenAI-compatible mock server for testing\n- Configurable response times, error rates, and behaviors\n- Runtime configuration via REST endpoints\n- Environment variable configuration support\n- See `dev/MOCK_SERVER_CONFIG.md` for detailed usage\n\n### Testing Scripts\n- `dev/testApi.ts`: Main API integration testing\n- `dev/testMockProvider.ts`: Mock provider functionality testing\n- `dev/testSetup.ts`: Test environment setup and cleanup\n- Automatic test data preservation and cleanup\n\n## Contributing\n\nme, myself, and i \n\nGG or owner of the fabled goldai (helped getting me into this space thats now dying)\n\n## License\n\nElastic License 2.0\n"}, "19792a3f3ab95f7b5a0ec99dcd567391f57bd78b": {"t": 1785771471.159809, "n": 18192, "h": "4e1a6554041530fe", "c": "# AnyGPT API Server\n\nThis directory contains the backend API server for AnyGPT, built with HyperExpress on Node.js and TypeScript.\n\n## Overview\n\nThe API server acts as a central gateway to various AI model providers. It manages API keys, handles request routing, provides rate limiting, logs errors, and dynamically updates model information. The server supports both Redis and filesystem-based data storage with automatic failover.\n\n## Project Structure\n\n```\napps/api/\n├── server.ts # Main server entry point, initializes and runs the API\n├── package.json # Project dependencies and scripts\n├── tsconfig.json # TypeScript configuration\n├── .env # Environment variables (created from .env.example)\n├── providers.json # Stores provider configurations and statistics\n├── models.json # Stores available model details, updated dynamically\n├── keys.json # Stores user API keys and associated data\n├── tiers.json # Defines API usage tiers and their limits\n├── providers.schema.json # JSON schema for providers.json\n├── models.schema.json # JSON schema for models.json\n├── openapi.json # OpenAPI spec served at /openapi.json\n├── excluded-errors.json # Error patterns excluded from provider auto-disable\n|\n├── routes/ # Contains route handlers for different API endpoints\n│ ├── admin.ts # Admin-specific routes (e.g., adding providers, generating keys)\n│ ├── models.ts # Routes for listing models and refreshing provider counts\n│ ├── openai.ts # OpenAI compatible API endpoints\n│ ├── anthropic.ts # Anthropic compatible API endpoints\n│ ├── gemini.ts # Gemini compatible API endpoints\n│ ├── groq.ts # Groq compatible API endpoints\n│ ├── openrouter.ts # OpenRouter compatible API endpoints\n│ ├── ollama.ts # Ollama compatible API endpoints\n│ ├── openapi.ts # OpenAPI spec route\n│ └── ... # Other provider-specific routes\n|\n├── providers/ # Logic for interacting with specific AI provider APIs\n│ ├── handler.ts # Core message handling, provider selection, and stats updates\n│ ├── interfaces.ts # TypeScript interfaces for providers and models\n│ ├── openai.ts # OpenAI provider client\n│ ├── gemini.ts # Gemini provider client\n│ └── ... # Other provider client implementations\n|\n├── modules/ # Reusable modules for various functionalities\n│ ├── dataManager.ts # Manages dual-source data (Redis/filesystem) with automatic failover\n│ ├── modelUpdater.ts # Handles automatic updates to models.json based on provider data\n│ ├── errorLogger.ts # Centralized error logging to file and Redis\n│ ├── errorExclusion.ts # Excludes errors from provider auto-disable\n│ ├── userData.ts # Manages user API key generation, validation, and usage tracking\n│ ├── compute.ts # Computes provider statistics, scores, and applies EMA\n│ ├── db.ts # Redis database connection and operations\n│ └── typeguards.ts # TypeScript type guards\n|\n├── logs/ # Directory for log files (filesystem fallback)\n│ └── api-error.jsonl # Detailed error logs in JSON Lines format\n|\n├── dev/ # Development and testing utilities\n│ ├── testApi.ts # Main API testing script\n│ ├── testSetup.ts # Test environment setup and cleanup\n│ ├── mockProvider.ts # Configurable mock AI provider server\n│ ├── testMockProvider.ts # Mock provider testing script\n│ ├── MOCK_SERVER_CONFIG.md # Mock server configuration documentation\n│ ├── models.ts # Model management utilities\n│ ├── updatemodels.ts # Model update scripts\n│ ├── updateproviders.ts # Provider update scripts\n│ └── ... # Other development utilities\n|\n├── server/ # CLI scripts (legacy, being migrated to API routes)\n│ ├── addProvider.ts # Script to add/update providers\n│ └── generateApiKey.ts # Script to generate API keys\n└── ws/ # WebSocket and realtime handlers\n```\n\n## Features\n\n* **Multi-Provider Support**: Integrates with OpenAI, Anthropic, Gemini, Groq, OpenRouter, Ollama, DeepSeek, xAI, Imagen, and more via provider configs.\n* **OpenAI-Compatible API**: `/v1` supports chat completions, responses, embeddings, images, videos, audio (speech/transcriptions), and key introspection.\n* **Provider-Compatible Routers**: Native-style endpoints for Anthropic (`/v3`), Gemini (`/v2`), Groq (`/v4`), Ollama (`/v5`), and OpenRouter (`/v6`).\n* **Dual Data Storage**: Redis primary with filesystem fallback and a short in-memory cache.\n* **Dynamic Model Management**: Updates `models.json` with provider counts and model capabilities (OpenWebUI compatible).\n* **Tier-Based Rate Limiting**: RPS/RPM/RPD limits and provider-score gating per tier (`tiers.json`).\n* **Provider Routing & Health**: Provider scoring, retries/fallback, auto-disable on repeated failures, and error-exclusion patterns.\n* **Image URL Inlining Proxy**: Optional HTTP image fetch with SSRF protections, size/time limits, and per-request referer override.\n* **WebSocket & Realtime**: `/ws` for chat and `/v1/realtime` for realtime-compatible sessions.\n* **OpenAPI Spec**: Served at `/openapi.json` and `/api/openapi.json`.\n\n## Prerequisites\n\n* Node.js (version specified in `package.json` or higher)\n* pnpm (version specified in `package.json`)\n\n## Setup\n\n1. **Clone the repository.**\n2. **Navigate to the `apps/api` directory:**\n ```bash\n cd apps/api\n ```\n3. **Install dependencies:**\n ```bash\n pnpm install\n ```\n4. **Create a `.env` file** with your configuration. Key environment variables include:\n\n ### Core Server Configuration\n * `PORT`: Port for the API server (default: 3000).\n \n ### Data Storage Configuration\n * `DATA_SOURCE_PREFERENCE`: Set to `redis` or `filesystem` (default: `redis`).\n * `REDIS_URL`: Redis Cloud connection URL (format: `host:port`).\n * `REDIS_USERNAME`: Redis username (default: `default`).\n * `REDIS_PASSWORD`: Redis password.\n * `REDIS_DB`: Redis database number (default: 0).\n * `REDIS_TLS`: Set to `true` for SSL/TLS connections (default: `false`).\n * `ERROR_LOG_TO_REDIS`: Enable error logging to Redis (default: `true`).\n\n ### Router Configuration\n * `ENABLE_OPENAI_ROUTES`: Enable/disable OpenAI routes (default: `true`).\n * `ENABLE_ANTHROPIC_ROUTES`: Enable/disable Anthropic routes (default: `true`).\n * `ENABLE_GEMINI_ROUTES`: Enable/disable Gemini routes (default: `true`).\n * `ENABLE_GROQ_ROUTES`: Enable/disable Groq routes (default: `true`).\n * `ENABLE_OPENROUTER_ROUTES`: Enable/disable OpenRouter routes (default: `true`).\n * `ENABLE_OLLAMA_ROUTES`: Enable/disable Ollama routes (default: `true`).\n * `ENABLE_ADMIN_ROUTES`: Enable/disable admin routes (default: `true`).\n * `ENABLE_MODELS_ROUTES`: Enable/disable models routes (default: `true`).\n\n ### Default Admin Configuration\n * `DEFAULT_ADMIN_USER_ID`: Default admin user ID for auto-creation.\n * `DEFAULT_ADMIN_API_KEY`: Default admin API key.\n\n ### Mock Server Configuration (for testing)\n * `MOCK_BASE_DELAY`: Base response delay in milliseconds (default: 200).\n * `MOCK_DELAY_VARIANCE`: Random delay variance (default: 100).\n * `MOCK_ERROR_RATE`: Error simulation rate 0-1 (default: 0.15).\n * `MOCK_TIMEOUT_RATE`: Timeout simulation rate 0-1 (default: 0.05).\n * `MOCK_TOKEN_SPEED`: Simulated tokens per second (default: 25).\n * `MOCK_ENABLE_LOGS`: Enable mock server logging (default: `true`).\n\n ### Logging & Security\n * `LOG_LEVEL`: `debug`, `info`, `warn`, `error`, or `silent` (default: `debug` in dev, `info` in production).\n * `LOG_SENSITIVE_PAYLOADS`: Log full URLs/body snippets for debugging (default: `false`).\n * `API_KEY_HASH_SECRET`: Secret used to hash API keys in logs (default: `anygpt-api`).\n * `RATE_LIMIT_HASH_SECRET`: Secret used to hash API keys for Redis rate-limiting keys (default: `anygpt-rate-limit`).\n * `ADMIN_KEYS_LOG_MAX_ENTRIES`: Maximum number of admin key log entries returned (default: 1000).\n * `ADMIN_KEYS_LOG_MAX_BYTES`: Rotate admin key log file when it exceeds this size in bytes (default: 5242880).\n * `ADMIN_KEYS_RATE_LIMIT_RPS`: Rate limit for open admin key ingest endpoint (default: 5).\n * `ADMIN_KEYS_RATE_LIMIT_RPM`: Rate limit for open admin key ingest endpoint (default: 60).\n * `ADMIN_KEYS_RATE_LIMIT_RPD`: Rate limit for open admin key ingest endpoint (default: 1000).\n * `ADMIN_KEYS_MAX_BODY_BYTES`: Max body size for open admin key ingest endpoint (default: 4096).\n * `ADMIN_KEYS_ALLOWED_PROVIDERS`: Optional comma-separated allowlist for provider IDs (default: unset, allows any provider).\n * `ADMIN_METRICS_MAX_ERROR_LINES`: Max error log lines scanned for metrics (default: 200000).\n\n ### Provider Health & Capabilities\n * `DISABLE_PROVIDER_AUTO_DISABLE`: Set to `true` to disable auto-disable behavior (default: auto-disable enabled).\n * `DISABLE_PROVIDER_AFTER_MODELS`: Disable a provider after this many models are disabled (default: 2).\n * `MODEL_CAPS_REFRESH_MS`: Model capability refresh interval in ms (default: 5000).\n\n ### Upstream Timeouts & Caches\n * `UPSTREAM_TIMEOUT_MS`: Upstream HTTP timeout in ms (default: 120000).\n * `VIDEO_REQUEST_CACHE_TTL_MS`: TTL for cached video request IDs in ms (default: 3600000).\n\n ### Image URL Fetch Proxy\n * `IMAGE_FETCH_TIMEOUT_MS`: Image fetch timeout in ms (default: 15000).\n * `IMAGE_FETCH_MAX_BYTES`: Max image bytes to inline (default: 8MB).\n * `IMAGE_FETCH_MAX_REDIRECTS`: Max redirects to follow (default: 3).\n * `IMAGE_FETCH_ALLOW_PRIVATE`: Allow private/loopback hosts (default: `false`).\n * `IMAGE_FETCH_FORWARD_AUTH`: Forward Authorization header to the original host (default: `false`).\n * `IMAGE_FETCH_ALLOWED_PROTOCOLS`: Allowlisted protocols (default: `http,https`).\n * `IMAGE_FETCH_ALLOWED_HOSTS`: Optional allowlist of hosts or suffixes (comma-separated).\n * `IMAGE_FETCH_USER_AGENT`: User-Agent sent when fetching images.\n * `IMAGE_FETCH_REFERER`: Default Referer for image fetches (can be overridden per request).\n\n5. **Initial Data Files**: The server will attempt to create `providers.json`, `models.json`, and `keys.json` if they don't exist. The data will be stored in Redis if configured, with filesystem fallback.\n\n## Running the Server\n\n* **Development Mode** (with hot-reloading via `tsx`):\n ```bash\n pnpm run dev\n ```\n This runs `server.ts` directly with TypeScript support.\n\n* **Production Build & Start:**\n ```bash\n # Build the TypeScript code\n pnpm run build\n \n # Start the compiled server\n pnpm start\n ```\n The build process outputs JavaScript files to `dist/` and the start script runs `./dist/server.js`.\n\n## Testing\n\nThe project includes a comprehensive testing suite with both unit tests and integration tests using a configurable mock provider.\n\n### Running Tests\n\n* **Full Test Suite** (recommended):\n ```bash\n pnpm test\n ```\n This runs the mock provider, API server, and test runner concurrently, then cleans up automatically.\n\n* **Individual Test Components**:\n ```bash\n # Run only the mock provider\n pnpm run test:mock\n \n # Run only the API server in test mode\n pnpm run test:dev\n \n # Run only the test scripts (requires servers to be running)\n pnpm run test:run\n ```\n\n### Mock Provider Testing\n\nThe mock provider supports runtime configuration for realistic testing scenarios:\n\n```bash\n# Test the mock provider configuration\npnpm exec tsx ./dev/testMockProvider.ts\n\n# Run the mock provider standalone\npnpm run test:mock\n```\n\nSee `dev/MOCK_SERVER_CONFIG.md` for detailed documentation on configuring response times, error rates, and other mock behaviors.\n\n## API Endpoints\n\nAuthentication is typically via `Authorization: Bearer <YOUR_ANYGPT_API_KEY>` or `api-key`/`x-api-key` headers (provider-compatible routes map provider headers to your AnyGPT key).\n\n### OpenAI-Compatible (`/v1`)\nCommon endpoints:\n* `POST /v1/chat/completions`\n* `POST /v1/responses`\n* `POST /v1/embeddings`\n* `POST /v1/images/generations`\n* `POST /v1/images/edits`\n* `POST /v1/videos/generations`\n* `GET /v1/videos/{requestId}`\n* `POST /v1/audio/speech`\n* `POST /v1/audio/transcriptions`\n* `GET /v1/audio/voices`\n* `GET /v1/audio/models`\n* `GET /v1/keys/me` (key usage/metrics)\n* `POST /v1/generate_key` (admin)\n* `POST /v1/interactions` and `GET /v1/interactions/{interactionId}`\n\n### Provider-Compatible Routers\n* Gemini: `/v2`\n* Anthropic: `/v3`\n* Groq: `/v4`\n* Ollama: `/v5`\n* OpenRouter: `/v6`\n\n### Models and Metadata\n* `GET /v1/models` (aliases: `/v1/chat/completions/models`, `/v1/chat/completion/models`)\n* `POST /admin/models/refresh-provider-counts` or `POST /api/admin/models/refresh-provider-counts` (admin)\n\n### Admin\n* Admin endpoints live under `/api/admin` (providers, API keys, metrics). See `openapi.json` for the full list.\n\n### OpenAPI Spec\n* `GET /openapi.json`\n* `GET /api/openapi.json`\n\n## Image URL Fetching\n\nThe OpenAI- and OpenRouter-compatible paths can inline HTTP image URLs into base64 before sending upstream. This avoids provider-side URL fetch failures and applies SSRF protections.\n\n* Use `x-image-fetch-referer` (or `x-image-referer`) to set a per-request Referer for image fetches.\n* Configure allowlists, size limits, and timeouts with the `IMAGE_FETCH_*` environment variables.\n\n## WebSocket Usage\n\nThe server exposes a unified WebSocket endpoint at `ws://<host>:<port>/ws` for chat-style completions. An OpenAI Realtime-compatible endpoint is also exposed at `ws://<host>:<port>/v1/realtime`.\n\n### Message Flow\n\n1. Connect to `/ws`.\n2. Send auth message:\n```json\n{ \"type\": \"auth\", \"apiKey\": \"YOUR_ANYGPT_API_KEY\" }\n```\n3. On `{ \"type\": \"auth.ok\" }`, send a chat request:\n```json\n{\n \"type\": \"chat.completions\",\n \"requestId\": \"req-123\",\n \"model\": \"gpt-4o\",\n \"messages\": [ { \"role\": \"user\", \"content\": \"Hello from WS\" } ]\n}\n```\n4. Receive sequence:\n```json\n{ \"type\": \"chat.start\", \"requestId\": \"req-123\" }\n{ \"type\": \"chat.complete\", \"requestId\": \"req-123\", \"response\": \"...assistant reply...\", \"usage\": { \"total_tokens\": 42 }, \"latencyMs\": 512, \"providerId\": \"openai-main\" }\n```\n\n### Supported Message Types (Inbound)\n- `auth` – Authenticate with API key.\n- `chat.completions` – Single-shot chat completion (non-streaming for now).\n- `ping` – Liveness check (responds with `pong`).\n\n### Outbound Types\n- `auth.ok`\n- `error` (fields: `code`, `message`, optional `requestId`)\n- `chat.start`\n- `chat.complete`\n- `pong`\n\n### Rate Limiting\nWebSocket messages share per-connection tracking and enforce tier limits (RPS/RPM/RPD) similarly to REST endpoints. Exceeding limits yields:\n```json\n{ \"type\": \"error\", \"code\": \"rate_limited\", \"message\": \"Rate limit exceeded\" }\n```\n\n### Example Node Test Scripts\n See `dev/testWs.ts`:\n ```bash\n TEST_API_KEY=YOUR_KEY pnpm exec tsx ./dev/testWs.ts\n ```\nA REST streaming test can be performed with `curl` as shown above.\n\n ### Planned Extensions\n - Streaming partial responses via `chat.delta` frames\n - Multiplex provider admin actions over WS\n - Real-time usage and provider stats push updates\n\n## Data Storage\n\nThe API server supports dual data storage modes with automatic failover:\n\n### Redis Storage (Recommended)\n- Primary storage method for production deployments\n- Supports Redis Cloud and self-hosted Redis instances\n- Automatic connection retry and error handling\n- Faster access times and better scalability\n\n### Filesystem Storage (Fallback)\n- Automatic fallback when Redis is unavailable\n- Stores data in JSON files (`providers.json`, `models.json`, `keys.json`)\n- Suitable for development and single-instance deployments\n\n### Configuration\nSet `DATA_SOURCE_PREFERENCE=redis` or `DATA_SOURCE_PREFERENCE=filesystem` in your `.env` file. The system will automatically fall back to filesystem storage if Redis connection fails.\n\n## Key Management & Tiers\n\n* API keys are managed in `keys.json` (filesystem) or Redis.\n* User tiers and their associated rate limits (RPS, RPM, RPD) and provider score preferences are defined in `tiers.json`.\n* The `generalAuthMiddleware` in `server.ts` handles initial API key validation, and specific middlewares in provider routes (`openai.ts`, etc.) or admin routes (`admin.ts`) enforce authentication and authorization.\n* `GET /v1/keys/me` returns usage and tier info for the current key (validates the key even if token usage is exhausted).\n* Default admin users can be auto-created using the `DEFAULT_ADMIN_USER_ID` and `DEFAULT_ADMIN_API_KEY` environment variables.\n\n## Logging & Monitoring\n\n* **Console Logging**: Server startup, request information, and general operational logs.\n* **Error Logging**: Detailed errors are logged in JSON Lines format to:\n - Redis (if `ERROR_LOG_TO_REDIS=true` and Redis is available)\n - Filesystem fallback (`logs/api-error.jsonl`)\n* **Error Exclusions**: Configure `excluded-errors.json` to avoid counting known transient errors toward auto-disable logic.\n* **Provider Statistics**: Response times, error rates, and performance metrics are continuously tracked and stored.\n* **Request Tracking**: All API requests are logged with timestamps, response times, and usage statistics.\n\n## Development Tools\n\n### Mock Provider Server\n- Full OpenAI-compatible mock server for testing\n- Configurable response times, error rates, and behaviors\n- Runtime configuration via REST endpoints\n- Environment variable configuration support\n- See `dev/MOCK_SERVER_CONFIG.md` for detailed usage\n\n### Testing Scripts\n- `dev/testApi.ts`: Main API integration testing\n- `dev/testMockProvider.ts`: Mock provider functionality testing\n- `dev/testSetup.ts`: Test environment setup and cleanup\n- Automatic test data preservation and cleanup\n\n## Contributing\n\nme, myself, and i \n\nGG or owner of the fabled goldai (helped getting me into this space thats now dying)\n\n## License\n\nElastic License 2.0\n"}, "5449d3712e426b423cddf539f16e72516742062e": {"t": 1785771472.7205598, "n": 16107, "h": "dd7d6c36b6ad0d92", "c": "# AnyGPT API Server\n\nThis directory contains the backend API server for AnyGPT, built with HyperExpress on Node.js and TypeScript.\n\n## Overview\n\nThe API server acts as a central gateway to various AI model providers. It manages API keys, handles request routing, provides rate limiting, logs errors, and dynamically updates model information. The server supports both Redis and filesystem-based data storage with automatic failover.\n\n## Project Structure\n\n```\napps/api/\n├── server.ts # Main server entry point, initializes and runs the API\n├── package.json # Project dependencies and scripts\n├── tsconfig.json # TypeScript configuration\n├── .env # Environment variables (created from .env.example)\n├── providers.json # Stores provider configurations and statistics\n├── models.json # Stores available model details, updated dynamically\n├── keys.json # Stores user API keys and associated data\n├── tiers.json # Defines API usage tiers and their limits\n├── providers.schema.json # JSON schema for providers.json\n├── models.schema.json # JSON schema for models.json\n|\n├── routes/ # Contains route handlers for different API endpoints\n│ ├── admin.ts # Admin-specific routes (e.g., adding providers, generating keys)\n│ ├── models.ts # Routes for listing models and refreshing provider counts\n│ ├── openai.ts # OpenAI compatible API endpoints\n│ ├── anthropic.ts # Anthropic compatible API endpoints\n│ ├── gemini.ts # Gemini compatible API endpoints\n│ ├── groq.ts # Groq compatible API endpoints\n│ ├── openrouter.ts # OpenRouter compatible API endpoints\n│ ├── ollama.ts # Ollama compatible API endpoints\n│ └── ... # Other provider-specific routes\n|\n├── providers/ # Logic for interacting with specific AI provider APIs\n│ ├── handler.ts # Core message handling, provider selection, and stats updates\n│ ├── interfaces.ts # TypeScript interfaces for providers and models\n│ ├── openai.ts # OpenAI provider client\n│ ├── gemini.ts # Gemini provider client\n│ └── ... # Other provider client implementations\n|\n├── modules/ # Reusable modules for various functionalities\n│ ├── dataManager.ts # Manages dual-source data (Redis/filesystem) with automatic failover\n│ ├── modelUpdater.ts # Handles automatic updates to models.json based on provider data\n│ ├── errorLogger.ts # Centralized error logging to file and Redis\n│ ├── userData.ts # Manages user API key generation, validation, and usage tracking\n│ ├── compute.ts # Computes provider statistics, scores, and applies EMA\n│ ├── db.ts # Redis database connection and operations\n│ └── typeguards.ts # TypeScript type guards\n|\n├── logs/ # Directory for log files (filesystem fallback)\n│ └── api-error.jsonl # Detailed error logs in JSON Lines format\n|\n├── dev/ # Development and testing utilities\n│ ├── testApi.ts # Main API testing script\n│ ├── testSetup.ts # Test environment setup and cleanup\n│ ├── mockProvider.ts # Configurable mock AI provider server\n│ ├── testMockProvider.ts # Mock provider testing script\n│ ├── MOCK_SERVER_CONFIG.md # Mock server configuration documentation\n│ ├── models.ts # Model management utilities\n│ ├── updatemodels.ts # Model update scripts\n│ ├── updateproviders.ts # Provider update scripts\n│ └── ... # Other development utilities\n|\n└── server/ # CLI scripts (legacy, being migrated to API routes)\n ├── addProvider.ts # Script to add/update providers\n └── generateApiKey.ts # Script to generate API keys\n```\n\n## Features\n\n* **Multi-Provider Support**: Integrates with various AI model providers (OpenAI, Anthropic, Gemini, Groq, OpenRouter, Ollama, etc.).\n* **Dual Data Storage**: Supports both Redis and filesystem-based data storage with automatic failover and preference configuration.\n* **Dynamic Model Management**: Automatically updates `models.json` with provider counts based on active providers in `providers.json`.\n* **OpenWebUI Compatibility**: `models.json` format is designed for compatibility with OpenWebUI.\n* **Environment-Driven Configuration**: Enable/disable specific provider routers via environment variables.\n* **Tier-Based Rate Limiting**: Implements RPS, RPM, RPD limits based on user tiers defined in `tiers.json`.\n* **API Key Management**: Generation and validation of user API keys with tier-based permissions.\n* **Provider Statistics & Scoring**: Tracks provider performance (response times, error rates, token speed) and calculates a score for intelligent routing.\n* **Error Handling & Logging**: Comprehensive error logging to both Redis and filesystem (`logs/api-error.jsonl`) with fallback support.\n* **Admin Endpoints**: Secure endpoints for managing providers, API keys, and metrics.\n* **Development Testing Suite**: Comprehensive testing infrastructure with configurable mock providers.\n* **Mock Provider Server**: Full-featured mock AI provider for testing with configurable response times, error rates, and behaviors.\n* **WebSocket Endpoint**: Unified `/ws` endpoint for authenticated real-time chat completions (extensible for streaming & other actions).\n* **OpenAPI Spec**: Served at `/openapi.json` for client integration.\n\n## Prerequisites\n\n* Node.js (version specified in `package.json` or higher)\n* pnpm (version specified in `package.json`)\n\n## Setup\n\n1. **Clone the repository.**\n2. **Navigate to the `apps/api` directory:**\n ```bash\n cd apps/api\n ```\n3. **Install dependencies:**\n ```bash\n pnpm install\n ```\n## Setup\n\n1. **Clone the repository.**\n2. **Navigate to the `apps/api` directory:**\n ```bash\n cd apps/api\n ```\n3. **Install dependencies:**\n ```bash\n pnpm install\n ```\n4. **Create a `.env` file** with your configuration. Key environment variables include:\n\n ### Core Server Configuration\n * `PORT`: Port for the API server (default: 3000).\n \n ### Data Storage Configuration\n * `DATA_SOURCE_PREFERENCE`: Set to `redis` or `filesystem` (default: `redis`).\n * `REDIS_URL`: Redis Cloud connection URL (format: `host:port`).\n * `REDIS_USERNAME`: Redis username (default: `default`).\n * `REDIS_PASSWORD`: Redis password.\n * `REDIS_DB`: Redis database number (default: 0).\n * `REDIS_TLS`: Set to `true` for SSL/TLS connections (default: `false`).\n * `ERROR_LOG_TO_REDIS`: Enable error logging to Redis (default: `true`).\n\n ### Router Configuration\n * `ENABLE_OPENAI_ROUTES`: Enable/disable OpenAI routes (default: `true`).\n * `ENABLE_ANTHROPIC_ROUTES`: Enable/disable Anthropic routes (default: `true`).\n * `ENABLE_GEMINI_ROUTES`: Enable/disable Gemini routes (default: `true`).\n * `ENABLE_GROQ_ROUTES`: Enable/disable Groq routes (default: `true`).\n * `ENABLE_OPENROUTER_ROUTES`: Enable/disable OpenRouter routes (default: `true`).\n * `ENABLE_OLLAMA_ROUTES`: Enable/disable Ollama routes (default: `true`).\n * `ENABLE_ADMIN_ROUTES`: Enable/disable admin routes (default: `true`).\n * `ENABLE_MODELS_ROUTES`: Enable/disable models routes (default: `true`).\n\n ### Default Admin Configuration\n * `DEFAULT_ADMIN_USER_ID`: Default admin user ID for auto-creation.\n * `DEFAULT_ADMIN_API_KEY`: Default admin API key.\n\n ### Mock Server Configuration (for testing)\n * `MOCK_BASE_DELAY`: Base response delay in milliseconds (default: 200).\n * `MOCK_DELAY_VARIANCE`: Random delay variance (default: 100).\n * `MOCK_ERROR_RATE`: Error simulation rate 0-1 (default: 0.15).\n * `MOCK_TIMEOUT_RATE`: Timeout simulation rate 0-1 (default: 0.05).\n * `MOCK_TOKEN_SPEED`: Simulated tokens per second (default: 25).\n * `MOCK_ENABLE_LOGS`: Enable mock server logging (default: `true`).\n\n ### Logging & Security\n * `LOG_LEVEL`: `debug`, `info`, `warn`, `error`, or `silent` (default: `debug` in dev, `info` in production).\n * `API_KEY_HASH_SECRET`: Secret used to hash API keys in logs (default: `anygpt-api`).\n * `RATE_LIMIT_HASH_SECRET`: Secret used to hash API keys for Redis rate-limiting keys (default: `anygpt-rate-limit`).\n * `ADMIN_KEYS_LOG_MAX_ENTRIES`: Maximum number of admin key log entries returned (default: 1000).\n * `ADMIN_KEYS_LOG_MAX_BYTES`: Rotate admin key log file when it exceeds this size in bytes (default: 5242880).\n * `ADMIN_KEYS_RATE_LIMIT_RPS`: Rate limit for open admin key ingest endpoint (default: 5).\n * `ADMIN_KEYS_RATE_LIMIT_RPM`: Rate limit for open admin key ingest endpoint (default: 60).\n * `ADMIN_KEYS_RATE_LIMIT_RPD`: Rate limit for open admin key ingest endpoint (default: 1000).\n * `ADMIN_KEYS_MAX_BODY_BYTES`: Max body size for open admin key ingest endpoint (default: 4096).\n * `ADMIN_KEYS_ALLOWED_PROVIDERS`: Optional comma-separated allowlist for provider IDs (default: unset, allows any provider).\n * `ADMIN_METRICS_MAX_ERROR_LINES`: Max error log lines scanned for metrics (default: 200000).\n\n5. **Initial Data Files**: The server will attempt to create `providers.json`, `models.json`, and `keys.json` if they don't exist. The data will be stored in Redis if configured, with filesystem fallback.\n\n## Running the Server\n\n* **Development Mode** (with hot-reloading via `tsx`):\n ```bash\n pnpm run dev\n ```\n This runs `server.ts` directly with TypeScript support.\n\n* **Production Build & Start:**\n ```bash\n # Build the TypeScript code\n pnpm run build\n \n # Start the compiled server\n pnpm start\n ```\n The build process outputs JavaScript files to `dist/` and the start script runs `./dist/server.js`.\n\n## Testing\n\nThe project includes a comprehensive testing suite with both unit tests and integration tests using a configurable mock provider.\n\n### Running Tests\n\n* **Full Test Suite** (recommended):\n ```bash\n pnpm test\n ```\n This runs the mock provider, API server, and test runner concurrently, then cleans up automatically.\n\n* **Individual Test Components**:\n ```bash\n # Run only the mock provider\n pnpm run test:mock\n \n # Run only the API server in test mode\n pnpm run test:dev\n \n # Run only the test scripts (requires servers to be running)\n pnpm run test:run\n ```\n\n### Mock Provider Testing\n\nThe mock provider supports runtime configuration for realistic testing scenarios:\n\n```bash\n# Test the mock provider configuration\npnpm exec tsx ./dev/testMockProvider.ts\n\n# Run the mock provider standalone\npnpm run test:mock\n```\n\nSee `dev/MOCK_SERVER_CONFIG.md` for detailed documentation on configuring response times, error rates, and other mock behaviors.\n\n## API Endpoints\n\nThe server exposes several sets of endpoints:\n\n* **Provider-Specific Endpoints** (e.g., `/openai/v1/chat/completions`, `/anthropic/v3/messages`):\n * These mimic the native APIs of providers like OpenAI, Anthropic, Gemini, and Groq.\n * Refer to the respective files in `routes/` for exact paths and request/response formats.\n * Authentication: Typically via `Authorization: Bearer <YOUR_ANYGPT_API_KEY>` or provider-specific headers like `x-api-key` mapped to your AnyGPT key.\n* **Model Information Endpoints** (e.g., `/api/models`):\n * `GET /api/models`: Lists available models from `models.json`.\n * `POST /api/admin/models/refresh-provider-counts`: Manually triggers a refresh of provider counts in `models.json` (requires admin privileges).\n* **Admin Endpoints** (prefixed with `/api/admin`, require admin privileges):\n * `POST /api/admin/providers`: Adds or updates a provider configuration in `providers.json` and fetches its models.\n * `POST /api/admin/users/generate-key`: Generates a new API key for a user.\n\n## WebSocket Usage\n\nThe server exposes a unified WebSocket endpoint at `ws://<host>:<port>/ws` for chat-style completions and future real-time features.\n\n### Message Flow\n\n1. Connect to `/ws`.\n2. Send auth message:\n```json\n{ \"type\": \"auth\", \"apiKey\": \"YOUR_ANYGPT_API_KEY\" }\n```\n3. On `{ \"type\": \"auth.ok\" }`, send a chat request:\n```json\n{\n \"type\": \"chat.completions\",\n \"requestId\": \"req-123\",\n \"model\": \"gpt-4o\",\n \"messages\": [ { \"role\": \"user\", \"content\": \"Hello from WS\" } ]\n}\n```\n4. Receive sequence:\n```json\n{ \"type\": \"chat.start\", \"requestId\": \"req-123\" }\n{ \"type\": \"chat.complete\", \"requestId\": \"req-123\", \"response\": \"...assistant reply...\", \"usage\": { \"total_tokens\": 42 }, \"latencyMs\": 512, \"providerId\": \"openai-main\" }\n```\n\n### Supported Message Types (Inbound)\n- `auth` – Authenticate with API key.\n- `chat.completions` – Single-shot chat completion (non-streaming for now).\n- `ping` – Liveness check (responds with `pong`).\n\n### Outbound Types\n- `auth.ok`\n- `error` (fields: `code`, `message`, optional `requestId`)\n- `chat.start`\n- `chat.complete`\n- `pong`\n\n### Rate Limiting\nWebSocket messages share per-connection tracking and enforce tier limits (RPS/RPM/RPD) similarly to REST endpoints. Exceeding limits yields:\n```json\n{ \"type\": \"error\", \"code\": \"rate_limited\", \"message\": \"Rate limit exceeded\" }\n```\n\n### Example Node Test Scripts\n See `dev/testWs.ts`:\n ```bash\n TEST_API_KEY=YOUR_KEY pnpm exec tsx ./dev/testWs.ts\n ```\nA REST streaming test can be performed with `curl` as shown above.\n\n ### Planned Extensions\n - Streaming partial responses via `chat.delta` frames\n - Multiplex provider admin actions over WS\n - Real-time usage and provider stats push updates\n\n## Data Storage\n\nThe API server supports dual data storage modes with automatic failover:\n\n### Redis Storage (Recommended)\n- Primary storage method for production deployments\n- Supports Redis Cloud and self-hosted Redis instances\n- Automatic connection retry and error handling\n- Faster access times and better scalability\n\n### Filesystem Storage (Fallback)\n- Automatic fallback when Redis is unavailable\n- Stores data in JSON files (`providers.json`, `models.json`, `keys.json`)\n- Suitable for development and single-instance deployments\n\n### Configuration\nSet `DATA_SOURCE_PREFERENCE=redis` or `DATA_SOURCE_PREFERENCE=filesystem` in your `.env` file. The system will automatically fall back to filesystem storage if Redis connection fails.\n\n## Key Management & Tiers\n\n* API keys are managed in `keys.json` (filesystem) or Redis.\n* User tiers and their associated rate limits (RPS, RPM, RPD) and provider score preferences are defined in `tiers.json`.\n* The `generalAuthMiddleware` in `server.ts` handles initial API key validation, and specific middlewares in provider routes (`openai.ts`, etc.) or admin routes (`admin.ts`) enforce authentication and authorization.\n* Default admin users can be auto-created using the `DEFAULT_ADMIN_USER_ID` and `DEFAULT_ADMIN_API_KEY` environment variables.\n\n## Logging & Monitoring\n\n* **Console Logging**: Server startup, request information, and general operational logs.\n* **Error Logging**: Detailed errors are logged in JSON Lines format to:\n - Redis (if `ERROR_LOG_TO_REDIS=true` and Redis is available)\n - Filesystem fallback (`logs/api-error.jsonl`)\n* **Provider Statistics**: Response times, error rates, and performance metrics are continuously tracked and stored.\n* **Request Tracking**: All API requests are logged with timestamps, response times, and usage statistics.\n\n## Development Tools\n\n### Mock Provider Server\n- Full OpenAI-compatible mock server for testing\n- Configurable response times, error rates, and behaviors\n- Runtime configuration via REST endpoints\n- Environment variable configuration support\n- See `dev/MOCK_SERVER_CONFIG.md` for detailed usage\n\n### Testing Scripts\n- `dev/testApi.ts`: Main API integration testing\n- `dev/testMockProvider.ts`: Mock provider functionality testing\n- `dev/testSetup.ts`: Test environment setup and cleanup\n- Automatic test data preservation and cleanup\n\n## Contributing\n\nme, myself, and i \n\nGG or owner of the fabled goldai (helped getting me into this space thats now dying)\n\n## License\n\nElastic License 2.0\n"}, "7a941e37de1923d0a7d00066cea4851662f006b1": {"t": 1785771474.6490772, "n": 14810, "h": "979fcf7155596150", "c": "# AnyGPT API Server\n\nThis directory contains the backend API server for AnyGPT, built with HyperExpress on Node.js and TypeScript.\n\n## Overview\n\nThe API server acts as a central gateway to various AI model providers. It manages API keys, handles request routing, provides rate limiting, logs errors, and dynamically updates model information. The server supports both Redis and filesystem-based data storage with automatic failover.\n\n## Project Structure\n\n```\napps/api/\n├── server.ts # Main server entry point, initializes and runs the API\n├── package.json # Project dependencies and scripts\n├── tsconfig.json # TypeScript configuration\n├── .env # Environment variables (created from .env.example)\n├── providers.json # Stores provider configurations and statistics\n├── models.json # Stores available model details, updated dynamically\n├── keys.json # Stores user API keys and associated data\n├── tiers.json # Defines API usage tiers and their limits\n├── providers.schema.json # JSON schema for providers.json\n├── models.schema.json # JSON schema for models.json\n|\n├── routes/ # Contains route handlers for different API endpoints\n│ ├── admin.ts # Admin-specific routes (e.g., adding providers, generating keys)\n│ ├── models.ts # Routes for listing models and refreshing provider counts\n│ ├── openai.ts # OpenAI compatible API endpoints\n│ ├── anthropic.ts # Anthropic compatible API endpoints\n│ ├── gemini.ts # Gemini compatible API endpoints\n│ ├── groq.ts # Groq compatible API endpoints\n│ ├── openrouter.ts # OpenRouter compatible API endpoints\n│ ├── ollama.ts # Ollama compatible API endpoints\n│ └── ... # Other provider-specific routes\n|\n├── providers/ # Logic for interacting with specific AI provider APIs\n│ ├── handler.ts # Core message handling, provider selection, and stats updates\n│ ├── interfaces.ts # TypeScript interfaces for providers and models\n│ ├── openai.ts # OpenAI provider client\n│ ├── gemini.ts # Gemini provider client\n│ └── ... # Other provider client implementations\n|\n├── modules/ # Reusable modules for various functionalities\n│ ├── dataManager.ts # Manages dual-source data (Redis/filesystem) with automatic failover\n│ ├── modelUpdater.ts # Handles automatic updates to models.json based on provider data\n│ ├── errorLogger.ts # Centralized error logging to file and Redis\n│ ├── userData.ts # Manages user API key generation, validation, and usage tracking\n│ ├── compute.ts # Computes provider statistics, scores, and applies EMA\n│ ├── db.ts # Redis database connection and operations\n│ └── typeguards.ts # TypeScript type guards\n|\n├── logs/ # Directory for log files (filesystem fallback)\n│ └── api-error.jsonl # Detailed error logs in JSON Lines format\n|\n├── dev/ # Development and testing utilities\n│ ├── testApi.ts # Main API testing script\n│ ├── testSetup.ts # Test environment setup and cleanup\n│ ├── mockProvider.ts # Configurable mock AI provider server\n│ ├── testMockProvider.ts # Mock provider testing script\n│ ├── MOCK_SERVER_CONFIG.md # Mock server configuration documentation\n│ ├── models.ts # Model management utilities\n│ ├── updatemodels.ts # Model update scripts\n│ ├── updateproviders.ts # Provider update scripts\n│ └── ... # Other development utilities\n|\n└── server/ # CLI scripts (legacy, being migrated to API routes)\n ├── addProvider.ts # Script to add/update providers\n └── generateApiKey.ts # Script to generate API keys\n```\n\n## Features\n\n* **Multi-Provider Support**: Integrates with various AI model providers (OpenAI, Anthropic, Gemini, Groq, OpenRouter, Ollama, etc.).\n* **Dual Data Storage**: Supports both Redis and filesystem-based data storage with automatic failover and preference configuration.\n* **Dynamic Model Management**: Automatically updates `models.json` with provider counts based on active providers in `providers.json`.\n* **OpenWebUI Compatibility**: `models.json` format is designed for compatibility with OpenWebUI.\n* **Environment-Driven Configuration**: Enable/disable specific provider routers via environment variables.\n* **Tier-Based Rate Limiting**: Implements RPS, RPM, RPD limits based on user tiers defined in `tiers.json`.\n* **API Key Management**: Generation and validation of user API keys with tier-based permissions.\n* **Provider Statistics & Scoring**: Tracks provider performance (response times, error rates, token speed) and calculates a score for intelligent routing.\n* **Error Handling & Logging**: Comprehensive error logging to both Redis and filesystem (`logs/api-error.jsonl`) with fallback support.\n* **Admin Endpoints**: Secure endpoints for managing providers and users.\n* **Development Testing Suite**: Comprehensive testing infrastructure with configurable mock providers.\n* **Mock Provider Server**: Full-featured mock AI provider for testing with configurable response times, error rates, and behaviors.\n* **WebSocket Endpoint**: Unified `/ws` endpoint for authenticated real-time chat completions (extensible for streaming & other actions).\n\n## Prerequisites\n\n* Node.js (version specified in `package.json` or higher)\n* pnpm (version specified in `package.json`)\n\n## Setup\n\n1. **Clone the repository.**\n2. **Navigate to the `apps/api` directory:**\n ```bash\n cd apps/api\n ```\n3. **Install dependencies:**\n ```bash\n pnpm install\n ```\n## Setup\n\n1. **Clone the repository.**\n2. **Navigate to the `apps/api` directory:**\n ```bash\n cd apps/api\n ```\n3. **Install dependencies:**\n ```bash\n pnpm install\n ```\n4. **Create a `.env` file** with your configuration. Key environment variables include:\n\n ### Core Server Configuration\n * `PORT`: Port for the API server (default: 3000).\n \n ### Data Storage Configuration\n * `DATA_SOURCE_PREFERENCE`: Set to `redis` or `filesystem` (default: `redis`).\n * `REDIS_URL`: Redis Cloud connection URL (format: `host:port`).\n * `REDIS_USERNAME`: Redis username (default: `default`).\n * `REDIS_PASSWORD`: Redis password.\n * `REDIS_DB`: Redis database number (default: 0).\n * `REDIS_TLS`: Set to `true` for SSL/TLS connections (default: `false`).\n * `ERROR_LOG_TO_REDIS`: Enable error logging to Redis (default: `true`).\n\n ### Router Configuration\n * `ENABLE_OPENAI_ROUTES`: Enable/disable OpenAI routes (default: `true`).\n * `ENABLE_ANTHROPIC_ROUTES`: Enable/disable Anthropic routes (default: `true`).\n * `ENABLE_GEMINI_ROUTES`: Enable/disable Gemini routes (default: `true`).\n * `ENABLE_GROQ_ROUTES`: Enable/disable Groq routes (default: `true`).\n * `ENABLE_OPENROUTER_ROUTES`: Enable/disable OpenRouter routes (default: `true`).\n * `ENABLE_OLLAMA_ROUTES`: Enable/disable Ollama routes (default: `true`).\n * `ENABLE_ADMIN_ROUTES`: Enable/disable admin routes (default: `true`).\n * `ENABLE_MODELS_ROUTES`: Enable/disable models routes (default: `true`).\n\n ### Default Admin Configuration\n * `DEFAULT_ADMIN_USER_ID`: Default admin user ID for auto-creation.\n * `DEFAULT_ADMIN_API_KEY`: Default admin API key.\n\n ### Mock Server Configuration (for testing)\n * `MOCK_BASE_DELAY`: Base response delay in milliseconds (default: 200).\n * `MOCK_DELAY_VARIANCE`: Random delay variance (default: 100).\n * `MOCK_ERROR_RATE`: Error simulation rate 0-1 (default: 0.15).\n * `MOCK_TIMEOUT_RATE`: Timeout simulation rate 0-1 (default: 0.05).\n * `MOCK_TOKEN_SPEED`: Simulated tokens per second (default: 25).\n * `MOCK_ENABLE_LOGS`: Enable mock server logging (default: `true`).\n\n5. **Initial Data Files**: The server will attempt to create `providers.json`, `models.json`, and `keys.json` if they don't exist. The data will be stored in Redis if configured, with filesystem fallback.\n\n## Running the Server\n\n* **Development Mode** (with hot-reloading via `tsx`):\n ```bash\n pnpm run dev\n ```\n This runs `server.ts` directly with TypeScript support.\n\n* **Production Build & Start:**\n ```bash\n # Build the TypeScript code\n pnpm run build\n \n # Start the compiled server\n pnpm start\n ```\n The build process outputs JavaScript files to `dist/` and the start script runs `./dist/server.js`.\n\n## Testing\n\nThe project includes a comprehensive testing suite with both unit tests and integration tests using a configurable mock provider.\n\n### Running Tests\n\n* **Full Test Suite** (recommended):\n ```bash\n pnpm test\n ```\n This runs the mock provider, API server, and test runner concurrently, then cleans up automatically.\n\n* **Individual Test Components**:\n ```bash\n # Run only the mock provider\n pnpm run test:mock\n \n # Run only the API server in test mode\n pnpm run test:dev\n \n # Run only the test scripts (requires servers to be running)\n pnpm run test:run\n ```\n\n### Mock Provider Testing\n\nThe mock provider supports runtime configuration for realistic testing scenarios:\n\n```bash\n# Test the mock provider configuration\npnpm exec tsx ./dev/testMockProvider.ts\n\n# Run the mock provider standalone\npnpm run test:mock\n```\n\nSee `dev/MOCK_SERVER_CONFIG.md` for detailed documentation on configuring response times, error rates, and other mock behaviors.\n\n## API Endpoints\n\nThe server exposes several sets of endpoints:\n\n* **Provider-Specific Endpoints** (e.g., `/openai/v1/chat/completions`, `/anthropic/v3/messages`):\n * These mimic the native APIs of providers like OpenAI, Anthropic, Gemini, and Groq.\n * Refer to the respective files in `routes/` for exact paths and request/response formats.\n * Authentication: Typically via `Authorization: Bearer <YOUR_ANYGPT_API_KEY>` or provider-specific headers like `x-api-key` mapped to your AnyGPT key.\n* **Model Information Endpoints** (e.g., `/api/models`):\n * `GET /api/models`: Lists available models from `models.json`.\n * `POST /api/admin/models/refresh-provider-counts`: Manually triggers a refresh of provider counts in `models.json` (requires admin privileges).\n* **Admin Endpoints** (prefixed with `/api/admin`, require admin privileges):\n * `POST /api/admin/providers`: Adds or updates a provider configuration in `providers.json` and fetches its models.\n * `POST /api/admin/users/generate-key`: Generates a new API key for a user.\n\n## WebSocket Usage\n\nThe server exposes a unified WebSocket endpoint at `ws://<host>:<port>/ws` for chat-style completions and future real-time features.\n\n### Message Flow\n\n1. Connect to `/ws`.\n2. Send auth message:\n```json\n{ \"type\": \"auth\", \"apiKey\": \"YOUR_ANYGPT_API_KEY\" }\n```\n3. On `{ \"type\": \"auth.ok\" }`, send a chat request:\n```json\n{\n \"type\": \"chat.completions\",\n \"requestId\": \"req-123\",\n \"model\": \"gpt-4o\",\n \"messages\": [ { \"role\": \"user\", \"content\": \"Hello from WS\" } ]\n}\n```\n4. Receive sequence:\n```json\n{ \"type\": \"chat.start\", \"requestId\": \"req-123\" }\n{ \"type\": \"chat.complete\", \"requestId\": \"req-123\", \"response\": \"...assistant reply...\", \"usage\": { \"total_tokens\": 42 }, \"latencyMs\": 512, \"providerId\": \"openai-main\" }\n```\n\n### Supported Message Types (Inbound)\n- `auth` – Authenticate with API key.\n- `chat.completions` – Single-shot chat completion (non-streaming for now).\n- `ping` – Liveness check (responds with `pong`).\n\n### Outbound Types\n- `auth.ok`\n- `error` (fields: `code`, `message`, optional `requestId`)\n- `chat.start`\n- `chat.complete`\n- `pong`\n\n### Rate Limiting\nWebSocket messages share per-connection tracking and enforce tier limits (RPS/RPM/RPD) similarly to REST endpoints. Exceeding limits yields:\n```json\n{ \"type\": \"error\", \"code\": \"rate_limited\", \"message\": \"Rate limit exceeded\" }\n```\n\n### Example Node Test Scripts\n See `dev/testWs.ts`:\n ```bash\n TEST_API_KEY=YOUR_KEY pnpm exec tsx ./dev/testWs.ts\n ```\nA REST streaming test can be performed with `curl` as shown above.\n\n ### Planned Extensions\n - Streaming partial responses via `chat.delta` frames\n - Multiplex provider admin actions over WS\n - Real-time usage and provider stats push updates\n\n## Data Storage\n\nThe API server supports dual data storage modes with automatic failover:\n\n### Redis Storage (Recommended)\n- Primary storage method for production deployments\n- Supports Redis Cloud and self-hosted Redis instances\n- Automatic connection retry and error handling\n- Faster access times and better scalability\n\n### Filesystem Storage (Fallback)\n- Automatic fallback when Redis is unavailable\n- Stores data in JSON files (`providers.json`, `models.json`, `keys.json`)\n- Suitable for development and single-instance deployments\n\n### Configuration\nSet `DATA_SOURCE_PREFERENCE=redis` or `DATA_SOURCE_PREFERENCE=filesystem` in your `.env` file. The system will automatically fall back to filesystem storage if Redis connection fails.\n\n## Key Management & Tiers\n\n* API keys are managed in `keys.json` (filesystem) or Redis.\n* User tiers and their associated rate limits (RPS, RPM, RPD) and provider score preferences are defined in `tiers.json`.\n* The `generalAuthMiddleware` in `server.ts` handles initial API key validation, and specific middlewares in provider routes (`openai.ts`, etc.) or admin routes (`admin.ts`) enforce authentication and authorization.\n* Default admin users can be auto-created using the `DEFAULT_ADMIN_USER_ID` and `DEFAULT_ADMIN_API_KEY` environment variables.\n\n## Logging & Monitoring\n\n* **Console Logging**: Server startup, request information, and general operational logs.\n* **Error Logging**: Detailed errors are logged in JSON Lines format to:\n - Redis (if `ERROR_LOG_TO_REDIS=true` and Redis is available)\n - Filesystem fallback (`logs/api-error.jsonl`)\n* **Provider Statistics**: Response times, error rates, and performance metrics are continuously tracked and stored.\n* **Request Tracking**: All API requests are logged with timestamps, response times, and usage statistics.\n\n## Development Tools\n\n### Mock Provider Server\n- Full OpenAI-compatible mock server for testing\n- Configurable response times, error rates, and behaviors\n- Runtime configuration via REST endpoints\n- Environment variable configuration support\n- See `dev/MOCK_SERVER_CONFIG.md` for detailed usage\n\n### Testing Scripts\n- `dev/testApi.ts`: Main API integration testing\n- `dev/testMockProvider.ts`: Mock provider functionality testing\n- `dev/testSetup.ts`: Test environment setup and cleanup\n- Automatic test data preservation and cleanup\n\n## Contributing\n\nme, myself, and i \n\nGG or owner of the fabled goldai (helped getting me into this space thats now dying)\n\n## License\n\nElastic License 2.0"}, "ea06dba4d4a8d07a4c64b997e0f85ac51ab1a02f": {"t": 1785771476.2400386, "n": 12972, "h": "a002fdb423454354", "c": "# AnyGPT API Server\n\nThis directory contains the backend API server for AnyGPT, built with HyperExpress on Node.js and TypeScript.\n\n## Overview\n\nThe API server acts as a central gateway to various AI model providers. It manages API keys, handles request routing, provides rate limiting, logs errors, and dynamically updates model information. The server supports both Redis and filesystem-based data storage with automatic failover.\n\n## Project Structure\n\n```\napps/api/\n├── server.ts # Main server entry point, initializes and runs the API\n├── package.json # Project dependencies and scripts\n├── tsconfig.json # TypeScript configuration\n├── .env # Environment variables (created from .env.example)\n├── providers.json # Stores provider configurations and statistics\n├── models.json # Stores available model details, updated dynamically\n├── keys.json # Stores user API keys and associated data\n├── tiers.json # Defines API usage tiers and their limits\n├── providers.schema.json # JSON schema for providers.json\n├── models.schema.json # JSON schema for models.json\n|\n├── routes/ # Contains route handlers for different API endpoints\n│ ├── admin.ts # Admin-specific routes (e.g., adding providers, generating keys)\n│ ├── models.ts # Routes for listing models and refreshing provider counts\n│ ├── openai.ts # OpenAI compatible API endpoints\n│ ├── anthropic.ts # Anthropic compatible API endpoints\n│ ├── gemini.ts # Gemini compatible API endpoints\n│ ├── groq.ts # Groq compatible API endpoints\n│ ├── openrouter.ts # OpenRouter compatible API endpoints\n│ ├── ollama.ts # Ollama compatible API endpoints\n│ └── ... # Other provider-specific routes\n|\n├── providers/ # Logic for interacting with specific AI provider APIs\n│ ├── handler.ts # Core message handling, provider selection, and stats updates\n│ ├── interfaces.ts # TypeScript interfaces for providers and models\n│ ├── openai.ts # OpenAI provider client\n│ ├── gemini.ts # Gemini provider client\n│ └── ... # Other provider client implementations\n|\n├── modules/ # Reusable modules for various functionalities\n│ ├── dataManager.ts # Manages dual-source data (Redis/filesystem) with automatic failover\n│ ├── modelUpdater.ts # Handles automatic updates to models.json based on provider data\n│ ├── errorLogger.ts # Centralized error logging to file and Redis\n│ ├── userData.ts # Manages user API key generation, validation, and usage tracking\n│ ├── compute.ts # Computes provider statistics, scores, and applies EMA\n│ ├── db.ts # Redis database connection and operations\n│ └── typeguards.ts # TypeScript type guards\n|\n├── logs/ # Directory for log files (filesystem fallback)\n│ └── api-error.jsonl # Detailed error logs in JSON Lines format\n|\n├── dev/ # Development and testing utilities\n│ ├── testApi.ts # Main API testing script\n│ ├── testSetup.ts # Test environment setup and cleanup\n│ ├── mockProvider.ts # Configurable mock AI provider server\n│ ├── testMockProvider.ts # Mock provider testing script\n│ ├── MOCK_SERVER_CONFIG.md # Mock server configuration documentation\n│ ├── models.ts # Model management utilities\n│ ├── updatemodels.ts # Model update scripts\n│ ├── updateproviders.ts # Provider update scripts\n│ └── ... # Other development utilities\n|\n└── server/ # CLI scripts (legacy, being migrated to API routes)\n ├── addProvider.ts # Script to add/update providers\n └── generateApiKey.ts # Script to generate API keys\n```\n\n## Features\n\n* **Multi-Provider Support**: Integrates with various AI model providers (OpenAI, Anthropic, Gemini, Groq, OpenRouter, Ollama, etc.).\n* **Dual Data Storage**: Supports both Redis and filesystem-based data storage with automatic failover and preference configuration.\n* **Dynamic Model Management**: Automatically updates `models.json` with provider counts based on active providers in `providers.json`.\n* **OpenWebUI Compatibility**: `models.json` format is designed for compatibility with OpenWebUI.\n* **Environment-Driven Configuration**: Enable/disable specific provider routers via environment variables.\n* **Tier-Based Rate Limiting**: Implements RPS, RPM, RPD limits based on user tiers defined in `tiers.json`.\n* **API Key Management**: Generation and validation of user API keys with tier-based permissions.\n* **Provider Statistics & Scoring**: Tracks provider performance (response times, error rates, token speed) and calculates a score for intelligent routing.\n* **Error Handling & Logging**: Comprehensive error logging to both Redis and filesystem (`logs/api-error.jsonl`) with fallback support.\n* **Admin Endpoints**: Secure endpoints for managing providers and users.\n* **Development Testing Suite**: Comprehensive testing infrastructure with configurable mock providers.\n* **Mock Provider Server**: Full-featured mock AI provider for testing with configurable response times, error rates, and behaviors.\n\n## Prerequisites\n\n* Node.js (version specified in `package.json` or higher)\n* pnpm (version specified in `package.json`)\n\n## Setup\n\n1. **Clone the repository.**\n2. **Navigate to the `apps/api` directory:**\n ```bash\n cd apps/api\n ```\n3. **Install dependencies:**\n ```bash\n pnpm install\n ```\n## Setup\n\n1. **Clone the repository.**\n2. **Navigate to the `apps/api` directory:**\n ```bash\n cd apps/api\n ```\n3. **Install dependencies:**\n ```bash\n pnpm install\n ```\n4. **Create a `.env` file** with your configuration. Key environment variables include:\n\n ### Core Server Configuration\n * `PORT`: Port for the API server (default: 3000).\n \n ### Data Storage Configuration\n * `DATA_SOURCE_PREFERENCE`: Set to `redis` or `filesystem` (default: `redis`).\n * `REDIS_URL`: Redis Cloud connection URL (format: `host:port`).\n * `REDIS_USERNAME`: Redis username (default: `default`).\n * `REDIS_PASSWORD`: Redis password.\n * `REDIS_DB`: Redis database number (default: 0).\n * `REDIS_TLS`: Set to `true` for SSL/TLS connections (default: `false`).\n * `ERROR_LOG_TO_REDIS`: Enable error logging to Redis (default: `true`).\n\n ### Router Configuration\n * `ENABLE_OPENAI_ROUTES`: Enable/disable OpenAI routes (default: `true`).\n * `ENABLE_ANTHROPIC_ROUTES`: Enable/disable Anthropic routes (default: `true`).\n * `ENABLE_GEMINI_ROUTES`: Enable/disable Gemini routes (default: `true`).\n * `ENABLE_GROQ_ROUTES`: Enable/disable Groq routes (default: `true`).\n * `ENABLE_OPENROUTER_ROUTES`: Enable/disable OpenRouter routes (default: `true`).\n * `ENABLE_OLLAMA_ROUTES`: Enable/disable Ollama routes (default: `true`).\n * `ENABLE_ADMIN_ROUTES`: Enable/disable admin routes (default: `true`).\n * `ENABLE_MODELS_ROUTES`: Enable/disable models routes (default: `true`).\n\n ### Default Admin Configuration\n * `DEFAULT_ADMIN_USER_ID`: Default admin user ID for auto-creation.\n * `DEFAULT_ADMIN_API_KEY`: Default admin API key.\n\n ### Mock Server Configuration (for testing)\n * `MOCK_BASE_DELAY`: Base response delay in milliseconds (default: 200).\n * `MOCK_DELAY_VARIANCE`: Random delay variance (default: 100).\n * `MOCK_ERROR_RATE`: Error simulation rate 0-1 (default: 0.15).\n * `MOCK_TIMEOUT_RATE`: Timeout simulation rate 0-1 (default: 0.05).\n * `MOCK_TOKEN_SPEED`: Simulated tokens per second (default: 25).\n * `MOCK_ENABLE_LOGS`: Enable mock server logging (default: `true`).\n\n5. **Initial Data Files**: The server will attempt to create `providers.json`, `models.json`, and `keys.json` if they don't exist. The data will be stored in Redis if configured, with filesystem fallback.\n\n## Running the Server\n\n* **Development Mode** (with hot-reloading via `tsx`):\n ```bash\n pnpm run dev\n ```\n This runs `server.ts` directly with TypeScript support.\n\n* **Production Build & Start:**\n ```bash\n # Build the TypeScript code\n pnpm run build\n \n # Start the compiled server\n pnpm start\n ```\n The build process outputs JavaScript files to `dist/` and the start script runs `./dist/server.js`.\n\n## Testing\n\nThe project includes a comprehensive testing suite with both unit tests and integration tests using a configurable mock provider.\n\n### Running Tests\n\n* **Full Test Suite** (recommended):\n ```bash\n pnpm test\n ```\n This runs the mock provider, API server, and test runner concurrently, then cleans up automatically.\n\n* **Individual Test Components**:\n ```bash\n # Run only the mock provider\n pnpm run test:mock\n \n # Run only the API server in test mode\n pnpm run test:dev\n \n # Run only the test scripts (requires servers to be running)\n pnpm run test:run\n ```\n\n### Mock Provider Testing\n\nThe mock provider supports runtime configuration for realistic testing scenarios:\n\n```bash\n# Test the mock provider configuration\npnpm exec tsx ./dev/testMockProvider.ts\n\n# Run the mock provider standalone\npnpm run test:mock\n```\n\nSee `dev/MOCK_SERVER_CONFIG.md` for detailed documentation on configuring response times, error rates, and other mock behaviors.\n\n## API Endpoints\n\nThe server exposes several sets of endpoints:\n\n* **Provider-Specific Endpoints** (e.g., `/openai/v1/chat/completions`, `/anthropic/v3/messages`):\n * These mimic the native APIs of providers like OpenAI, Anthropic, Gemini, and Groq.\n * Refer to the respective files in `routes/` for exact paths and request/response formats.\n * Authentication: Typically via `Authorization: Bearer <YOUR_ANYGPT_API_KEY>` or provider-specific headers like `x-api-key` mapped to your AnyGPT key.\n* **Model Information Endpoints** (e.g., `/api/models`):\n * `GET /api/models`: Lists available models from `models.json`.\n * `POST /api/admin/models/refresh-provider-counts`: Manually triggers a refresh of provider counts in `models.json` (requires admin privileges).\n* **Admin Endpoints** (prefixed with `/api/admin`, require admin privileges):\n * `POST /api/admin/providers`: Adds or updates a provider configuration in `providers.json` and fetches its models.\n * `POST /api/admin/users/generate-key`: Generates a new API key for a user.\n\n## Data Storage\n\nThe API server supports dual data storage modes with automatic failover:\n\n### Redis Storage (Recommended)\n- Primary storage method for production deployments\n- Supports Redis Cloud and self-hosted Redis instances\n- Automatic connection retry and error handling\n- Faster access times and better scalability\n\n### Filesystem Storage (Fallback)\n- Automatic fallback when Redis is unavailable\n- Stores data in JSON files (`providers.json`, `models.json`, `keys.json`)\n- Suitable for development and single-instance deployments\n\n### Configuration\nSet `DATA_SOURCE_PREFERENCE=redis` or `DATA_SOURCE_PREFERENCE=filesystem` in your `.env` file. The system will automatically fall back to filesystem storage if Redis connection fails.\n\n## Key Management & Tiers\n\n* API keys are managed in `keys.json` (filesystem) or Redis.\n* User tiers and their associated rate limits (RPS, RPM, RPD) and provider score preferences are defined in `tiers.json`.\n* The `generalAuthMiddleware` in `server.ts` handles initial API key validation, and specific middlewares in provider routes (`openai.ts`, etc.) or admin routes (`admin.ts`) enforce authentication and authorization.\n* Default admin users can be auto-created using the `DEFAULT_ADMIN_USER_ID` and `DEFAULT_ADMIN_API_KEY` environment variables.\n\n## Logging & Monitoring\n\n* **Console Logging**: Server startup, request information, and general operational logs.\n* **Error Logging**: Detailed errors are logged in JSON Lines format to:\n - Redis (if `ERROR_LOG_TO_REDIS=true` and Redis is available)\n - Filesystem fallback (`logs/api-error.jsonl`)\n* **Provider Statistics**: Response times, error rates, and performance metrics are continuously tracked and stored.\n* **Request Tracking**: All API requests are logged with timestamps, response times, and usage statistics.\n\n## Development Tools\n\n### Mock Provider Server\n- Full OpenAI-compatible mock server for testing\n- Configurable response times, error rates, and behaviors\n- Runtime configuration via REST endpoints\n- Environment variable configuration support\n- See `dev/MOCK_SERVER_CONFIG.md` for detailed usage\n\n### Testing Scripts\n- `dev/testApi.ts`: Main API integration testing\n- `dev/testMockProvider.ts`: Mock provider functionality testing\n- `dev/testSetup.ts`: Test environment setup and cleanup\n- Automatic test data preservation and cleanup\n\n## Contributing\n\nme, myself, and i \n\nGG or owner of the fabled goldai (helped getting me into this space thats now dying)\n\n## License\n\nElastic License 2.0"}, "839bd440c357f4cfdc5113284269b365d655741c": {"t": 1785771477.6710508, "n": 8500, "h": "39e378cd7367c7f9", "c": "# AnyGPT API Server\n\nThis directory contains the backend API server for AnyGPT, built with HyperExpress on Node.js and TypeScript.\n\n## Overview\n\nThe API server acts as a central gateway to various AI model providers. It manages API keys, handles request routing, provides rate limiting, logs errors, and dynamically updates model information.\n\n## Project Structure\n\n```\napps/api/\n├── server.ts # Main server entry point, initializes and runs the API\n├── package.json # Project dependencies and scripts\n├── tsconfig.json # TypeScript configuration\n├── .env.example # Example environment variables (create .env from this)\n├── providers.json # Stores provider configurations and statistics\n├── models.json # Stores available model details, updated dynamically\n├── keys.json # Stores user API keys and associated data\n├── tiers.json # Defines API usage tiers and their limits\n├── providers.schema.json # JSON schema for providers.json\n├── models.schema.json # JSON schema for models.json\n|\n├── routes/ # Contains route handlers for different API endpoints\n│ ├── admin.ts # Admin-specific routes (e.g., adding providers, generating keys)\n│ ├── models.ts # Routes for listing models and refreshing provider counts\n│ ├── openai.ts # OpenAI compatible API endpoints\n│ ├── anthropic.ts # Anthropic compatible API endpoints\n│ ├── gemini.ts # Gemini compatible API endpoints\n│ ├── groq.ts # Groq compatible API endpoints\n│ └── ... # Potentially other provider-specific or utility routes\n|\n├── providers/ # Logic for interacting with specific AI provider APIs\n│ ├── handler.ts # Core message handling, provider selection, and stats updates\n│ ├── interfaces.ts # TypeScript interfaces for providers and models\n│ ├── openai.ts # OpenAI provider client\n│ ├── gemini.ts # Gemini provider client\n│ └── ... # Other provider client implementations\n|\n├── modules/ # Reusable modules for various functionalities\n│ ├── dataManager.ts # Manages loading and saving of JSON data files (models, providers, keys)\n│ ├── modelUpdater.ts # Handles automatic updates to models.json based on provider data\n│ ├── errorLogger.ts # Centralized error logging to file (logs/api-error.jsonl)\n│ ├── userData.ts # Manages user API key generation, validation, and usage tracking\n│ ├── compute.ts # Computes provider statistics, scores, and applies EMA\n│ ├── db.ts # (Potentially for future database interactions - currently Redis example)\n│ └── typeguards.ts # TypeScript type guards\n|\n├── logs/ # Directory for log files\n│ └── api-error.jsonl # Detailed error logs in JSON Lines format\n|\n├── dev/ # Development related utilities\n│ └── testApi.ts # Script for testing API endpoints\n|\n└── server/ # CLI scripts (some functionality being migrated to API routes)\n ├── addProvider.ts # Script to add/update providers (refactored into an API module)\n └── ...\n```\n\n## Features\n\n* **Multi-Provider Support**: Integrates with various AI model providers (OpenAI, Anthropic, Gemini, Groq, etc.).\n* **Dynamic Model Management**: Automatically updates `models.json` with provider counts based on active providers in `providers.json`.\n* **OpenWebUI Compatibility**: `models.json` format is designed for compatibility.\n* **Environment-Driven Configuration**: Enable/disable specific provider routers via environment variables.\n* **Tier-Based Rate Limiting**: Implements RPS, RPM, RPD limits based on user tiers defined in `tiers.json`.\n* **API Key Management**: Generation and validation of user API keys.\n* **Provider Statistics & Scoring**: Tracks provider performance (response times, error rates, token speed) and calculates a score for intelligent routing.\n* **Error Handling & Logging**: Comprehensive error logging to `logs/api-error.jsonl` and standardized client error responses.\n* **Admin Endpoints**: Secure endpoints for managing providers and users.\n\n## Prerequisites\n\n* Node.js (version specified in `package.json` or higher)\n* pnpm (version specified in `package.json`)\n\n## Setup\n\n1. **Clone the repository.**\n2. **Navigate to the `apps/api` directory:**\n ```bash\n cd apps/api\n ```\n3. **Install dependencies:**\n ```bash\n pnpm install\n ```\n4. **Create a `.env` file** by copying `.env.example` (if it exists, otherwise create one) and populate it with your API keys and other configurations. Key environment variables include:\n * `PORT`: Port for the API server (default: 3000).\n * `ENABLE_OPENAI_ROUTES`: Set to `true` or `false` to enable/disable OpenAI routes (default: `true`).\n * `ENABLE_ANTHROPIC_ROUTES`: Set to `true` or `false` (default: `true`).\n * `ENABLE_GEMINI_ROUTES`: Set to `true` or `false` (default: `true`).\n * `ENABLE_GROQ_ROUTES`: Set to `true` or `false` (default: `true`).\n * `ENABLE_ADMIN_ROUTES`: Set to `true` or `false` (default: `true`).\n * `ENABLE_MODELS_ROUTES`: Set to `true` or `false` (default: `true`).\n\n Refer to `server.ts` for the exact `ENABLE_<ROUTER_NAME>_ROUTES` variable names.\n\n5. **Initial Data Files**: The server will attempt to create `providers.json`, `models.json`, and `keys.json` if they don't exist with default structures. You will need to populate `providers.json` with your provider configurations and `keys.json` with initial admin keys if necessary.\n\n## Running the Server\n\n* **Development Mode (with hot-reloading via `tsx`):\n ```bash\n pnpm run dev\n ```\n This command runs `server.ts` directly.\n\n* **Production Build & Start:**\n 1. Build the TypeScript code:\n ```bash\n pnpm run build\n ```\n This will output JavaScript files to `dist/api`.\n 2. Run the built server (you might need to adjust the start script in `package.json` or use `node dist/api/server.js`):\n The current `pnpm start` script in `package.json` points to `tsx routes/openai.ts` which is likely for isolated testing of OpenAI routes. For production, you should run the main `server.ts` (or its compiled version `dist/api/server.js`).\n **Recommended update to `package.json` for production start:**\n ```json\n \"scripts\": {\n // ... other scripts\n \"start\": \"node ../../dist/api/server.js\", // If running from apps/api\n \"dev\": \"tsx server.ts\",\n // ...\n }\n ```\n Then run:\n ```bash\n pnpm start\n ```\n\n## API Endpoints\n\nThe server exposes several sets of endpoints:\n\n* **Provider-Specific Endpoints** (e.g., `/openai/v1/chat/completions`, `/anthropic/v3/messages`):\n * These mimic the native APIs of providers like OpenAI, Anthropic, Gemini, and Groq.\n * Refer to the respective files in `routes/` for exact paths and request/response formats.\n * Authentication: Typically via `Authorization: Bearer <YOUR_ANYGPT_API_KEY>` or provider-specific headers like `x-api-key` mapped to your AnyGPT key.\n* **Model Information Endpoints** (e.g., `/api/models`):\n * `GET /api/models`: Lists available models from `models.json`.\n * `POST /api/admin/models/refresh-provider-counts`: Manually triggers a refresh of provider counts in `models.json` (requires admin privileges).\n* **Admin Endpoints** (prefixed with `/api/admin`, require admin privileges):\n * `POST /api/admin/providers`: Adds or updates a provider configuration in `providers.json` and fetches its models.\n * `POST /api/admin/users/generate-key`: Generates a new API key for a user.\n\n## Key Management & Tiers\n\n* API keys are managed in `keys.json`.\n* User tiers and their associated rate limits (RPS, RPM, RPD) and provider score preferences are defined in `tiers.json`.\n* The `generalAuthMiddleware` in `server.ts` handles initial API key validation, and specific middlewares in provider routes (`openai.ts`, etc.) or admin routes (`admin.ts`) enforce authentication and authorization.\n\n## Logging\n\n* Server startup and request information are logged to the console.\n* Detailed errors are logged in JSON Lines format to `logs/api-error.jsonl`. Each entry includes a timestamp, request details (if applicable), error message, stack trace, and other context.\n\n## Contributing\n\n(Add guidelines for contributing if this is an open project).\n\n## License\n\n(Specify the license for the project)."}, "3c28ed19b9e2b865e65a86a787bc69a6a04f0a03": {"t": 1785771478.995205, "n": 8759, "h": "e81d8ea36bbf2de3", "c": "# AnyGPT API Server\n\nThis directory contains the backend API server for AnyGPT, built with HyperExpress on Node.js and TypeScript.\n\n## Overview\n\nThe API server acts as a central gateway to various AI model providers. It manages API keys, handles request routing, provides rate limiting, logs errors, and dynamically updates model information.\n\n## Project Structure\n\n```\napps/api/\n├── server.ts # Main server entry point, initializes and runs the API\n├── package.json # Project dependencies and scripts\n├── tsconfig.json # TypeScript configuration\n├── .env.example # Example environment variables (create .env from this)\n├── providers.json # Stores provider configurations and statistics\n├── models.json # Stores available model details, updated dynamically\n├── keys.json # Stores user API keys and associated data\n├── tiers.json # Defines API usage tiers and their limits\n├── providers.schema.json # JSON schema for providers.json\n├── models.schema.json # JSON schema for models.json\n|\n├── routes/ # Contains route handlers for different API endpoints\n│ ├── admin.ts # Admin-specific routes (e.g., adding providers, generating keys)\n│ ├── models.ts # Routes for listing models and refreshing provider counts\n│ ├── openai.ts # OpenAI compatible API endpoints\n│ ├── anthropic.ts # Anthropic compatible API endpoints\n│ ├── gemini.ts # Gemini compatible API endpoints\n│ ├── groq.ts # Groq compatible API endpoints\n│ └── ... # Potentially other provider-specific or utility routes\n|\n├── providers/ # Logic for interacting with specific AI provider APIs\n│ ├── handler.ts # Core message handling, provider selection, and stats updates\n│ ├── interfaces.ts # TypeScript interfaces for providers and models\n│ ├── openai.ts # OpenAI provider client\n│ ├── gemini.ts # Gemini provider client\n│ └── ... # Other provider client implementations\n|\n├── modules/ # Reusable modules for various functionalities\n│ ├── dataManager.ts # Manages loading and saving of JSON data files (models, providers, keys)\n│ ├── modelUpdater.ts # Handles automatic updates to models.json based on provider data\n│ ├── errorLogger.ts # Centralized error logging to file (logs/api-error.jsonl)\n│ ├── userData.ts # Manages user API key generation, validation, and usage tracking\n│ ├── compute.ts # Computes provider statistics, scores, and applies EMA\n│ ├── db.ts # (Potentially for future database interactions - currently Redis example)\n│ └── typeguards.ts # TypeScript type guards\n|\n├── logs/ # Directory for log files\n│ └── api-error.jsonl # Detailed error logs in JSON Lines format\n|\n├── dev/ # Development related utilities\n│ └── testApi.ts # Script for testing API endpoints\n|\n└── server/ # CLI scripts (some functionality being migrated to API routes)\n ├── addProvider.ts # Script to add/update providers (refactored into an API module)\n └── ...\n```\n\n## Features\n\n* **Multi-Provider Support**: Integrates with various AI model providers (OpenAI, Anthropic, Gemini, Groq, etc.).\n* **Dynamic Model Management**: Automatically updates `models.json` with provider counts based on active providers in `providers.json`.\n* **OpenWebUI Compatibility**: `models.json` format is designed for compatibility.\n* **Environment-Driven Configuration**: Enable/disable specific provider routers via environment variables.\n* **Tier-Based Rate Limiting**: Implements RPS, RPM, RPD limits based on user tiers defined in `tiers.json`.\n* **API Key Management**: Generation and validation of user API keys.\n* **Provider Statistics & Scoring**: Tracks provider performance (response times, error rates, token speed) and calculates a score for intelligent routing.\n* **Error Handling & Logging**: Comprehensive error logging to `logs/api-error.jsonl` and standardized client error responses.\n* **Admin Endpoints**: Secure endpoints for managing providers and users.\n\n## Prerequisites\n\n* Node.js (version specified in `package.json` or higher)\n* pnpm (version specified in `package.json`)\n\n## Setup\n\n1. **Clone the repository.**\n2. **Navigate to the `apps/api` directory:**\n ```bash\n cd apps/api\n ```\n3. **Install dependencies:**\n ```bash\n pnpm install\n ```\n4. **Create a `.env` file** by copying `.env.example` (if it exists, otherwise create one) and populate it with your API keys and other configurations. Key environment variables include:\n * `PORT`: Port for the API server (default: 3000).\n * `PROVIDER_API_KEY_OPENAI_DEFAULT`: API key for your OpenAI account (example, follow pattern for other providers like `PROVIDER_API_KEY_ANTHROPIC_DEFAULT`, etc., if the provider's primary key is set in `providers.json` this is a fallback or override).\n * `ENABLE_OPENAI_ROUTES`: Set to `true` or `false` to enable/disable OpenAI routes (default: `true`).\n * `ENABLE_ANTHROPIC_ROUTES`: Set to `true` or `false` (default: `true`).\n * `ENABLE_GEMINI_ROUTES`: Set to `true` or `false` (default: `true`).\n * `ENABLE_GROQ_ROUTES`: Set to `true` or `false` (default: `true`).\n * `ENABLE_ADMIN_ROUTES`: Set to `true` or `false` (default: `true`).\n * `ENABLE_MODELS_ROUTES`: Set to `true` or `false` (default: `true`).\n\n Refer to `server.ts` for the exact `ENABLE_<ROUTER_NAME>_ROUTES` variable names.\n\n5. **Initial Data Files**: The server will attempt to create `providers.json`, `models.json`, and `keys.json` if they don't exist with default structures. You will need to populate `providers.json` with your provider configurations and `keys.json` with initial admin keys if necessary.\n\n## Running the Server\n\n* **Development Mode (with hot-reloading via `tsx`):\n ```bash\n pnpm run dev\n ```\n This command runs `server.ts` directly.\n\n* **Production Build & Start:**\n 1. Build the TypeScript code:\n ```bash\n pnpm run build\n ```\n This will output JavaScript files to `dist/api`.\n 2. Run the built server (you might need to adjust the start script in `package.json` or use `node dist/api/server.js`):\n The current `pnpm start` script in `package.json` points to `tsx routes/openai.ts` which is likely for isolated testing of OpenAI routes. For production, you should run the main `server.ts` (or its compiled version `dist/api/server.js`).\n **Recommended update to `package.json` for production start:**\n ```json\n \"scripts\": {\n // ... other scripts\n \"start\": \"node ../../dist/api/server.js\", // If running from apps/api\n \"dev\": \"tsx server.ts\",\n // ...\n }\n ```\n Then run:\n ```bash\n pnpm start\n ```\n\n## API Endpoints\n\nThe server exposes several sets of endpoints:\n\n* **Provider-Specific Endpoints** (e.g., `/openai/v1/chat/completions`, `/anthropic/v3/messages`):\n * These mimic the native APIs of providers like OpenAI, Anthropic, Gemini, and Groq.\n * Refer to the respective files in `routes/` for exact paths and request/response formats.\n * Authentication: Typically via `Authorization: Bearer <YOUR_ANYGPT_API_KEY>` or provider-specific headers like `x-api-key` mapped to your AnyGPT key.\n* **Model Information Endpoints** (e.g., `/api/models`):\n * `GET /api/models`: Lists available models from `models.json`.\n * `POST /api/admin/models/refresh-provider-counts`: Manually triggers a refresh of provider counts in `models.json` (requires admin privileges).\n* **Admin Endpoints** (prefixed with `/api/admin`, require admin privileges):\n * `POST /api/admin/providers`: Adds or updates a provider configuration in `providers.json` and fetches its models.\n * `POST /api/admin/users/generate-key`: Generates a new API key for a user.\n\n## Key Management & Tiers\n\n* API keys are managed in `keys.json`.\n* User tiers and their associated rate limits (RPS, RPM, RPD) and provider score preferences are defined in `tiers.json`.\n* The `generalAuthMiddleware` in `server.ts` handles initial API key validation, and specific middlewares in provider routes (`openai.ts`, etc.) or admin routes (`admin.ts`) enforce authentication and authorization.\n\n## Logging\n\n* Server startup and request information are logged to the console.\n* Detailed errors are logged in JSON Lines format to `logs/api-error.jsonl`. Each entry includes a timestamp, request details (if applicable), error message, stack trace, and other context.\n\n## Contributing\n\n(Add guidelines for contributing if this is an open project).\n\n## License\n\n(Specify the license for the project)."}, "0fe569ee0c49aa682ae7a4d60442c5838305c0e5": {"t": 1785771481.1427364, "n": 301, "h": "88043cab4e4df75b", "c": "execute below to generate admin api key (make sure you're in api directory)\nnpx tsx server/generateApiKey.ts \n\nroutes\n\nanthropic /v3/messages\ngroq /v4/chat/completions\ngemini /v2/models/:modelId:generateContent\nollama /v5/api/chat\nopenai /v1/chat/completions /v1/models\nopenrouter /v6/chat/completions"}, "d38ae392cc19d4490e56ede7ad495db07b64e200": {"t": 1785771483.0056725, "n": 109, "h": "80ff6667a7de14ae", "c": "execute below to generate admin api key (make sure you're in api directory)\nnpx tsx server/generateApiKey.ts "}}}} |