1 line
11 KiB
JSON
1 line
11 KiB
JSON
{"repo": "aryan-b-parikh/smartcontainerriskengine", "files": {"docs/11-environment-and-devops.md": {"5beeb7972afc80d073b2efe651e39378902c48ef": {"t": 1785596461.4490354, "n": 10253, "h": "e949b60a390f4165", "c": "# 11 — Environment & DevOps\n**SmartContainer Risk Engine**\n**Version:** 1.0 — MVP\n**Date:** 2026-03-05\n**Status:** Draft\n\n---\n\n## 1. Overview\n\nThis document specifies all environment configurations, local development setup, CI/CD pipeline design, containerization strategy, and deployment procedures for the SmartContainer Risk Engine.\n\n---\n\n## 2. Environment Architecture\n\nThree environments are defined:\n\n| Environment | Purpose | Trigger | Database |\n|---|---|---|---|\n| `local` | Individual developer machines | `make dev` / `docker-compose up` | Local PostgreSQL (Docker) |\n| `staging` | Integration testing + pre-demo validation | Merge to `develop` branch | Shared staging DB |\n| `production` | Stakeholder demos + live system | Merge to `main` + manual approval | Managed PostgreSQL |\n\n---\n\n## 3. Environment Variables\n\nAll secrets and configuration are managed via environment variables. **No secrets in code or version control.**\n\n### 3.1 Required Variables\n\n```env\n# Application\nAPP_ENV=local # local | staging | production\nAPP_HOST=0.0.0.0\nAPP_PORT=8000\nLOG_LEVEL=INFO # DEBUG | INFO | WARNING | ERROR\n\n# Database\nDATABASE_URL=postgresql+asyncpg://scre_user:password@db:5432/scre_db\nDB_POOL_SIZE=5\nDB_MAX_OVERFLOW=10\n\n# Authentication\nAPI_KEY_SALT=<random-32-char-string>\nADMIN_API_KEY=<hashed-admin-key>\n\n# ML Model\nACTIVE_MODEL_VERSION=v1.0\nMODEL_ARTIFACTS_DIR=/app/models\n\n# File Storage\nUPLOAD_DIR=/app/data/uploads\nOUTPUT_DIR=/app/data/outputs\nMAX_UPLOAD_SIZE_MB=500\n\n# Anomaly Detection\nANOMALY_THRESHOLD=0.65\n```\n\n### 3.2 `.env.example` Template\n\nThe repository ships an `.env.example` with all variable names and safe placeholder values. Copy to `.env` and populate before running:\n\n```bash\ncp .env.example .env\n```\n\n---\n\n## 4. Local Development Setup\n\n### 4.1 Prerequisites\n\n| Tool | Minimum Version | Purpose |\n|---|---|---|\n| Python | 3.11 | Application runtime |\n| Docker Desktop | 25.x | Container orchestration |\n| Docker Compose | 2.24 | Multi-service management |\n| Make | 4.x (GNU) | Developer commands |\n| Git | 2.40 | Version control |\n\n### 4.2 First-Time Setup\n\n```bash\n# 1. Clone repository\ngit clone https://github.com/<org>/smartcontainer-risk-engine.git\ncd smartcontainer-risk-engine\n\n# 2. Copy environment config\ncp .env.example .env\n# Edit .env — fill in API_KEY_SALT and ADMIN_API_KEY\n\n# 3. Install Python packages (editable mode)\nmake install\n\n# 4. Start all services via Docker Compose\nmake dev\n\n# 5. Run database migrations\nmake migrate\n\n# 6. Seed with sample data (optional)\nmake seed\n\n# 7. Verify health\ncurl http://localhost:8000/api/v1/health\n```\n\n### 4.3 Docker Compose Services\n\n```yaml\n# docker-compose.yml (summary)\nservices:\n api:\n build: ./services/api\n ports: [\"8000:8000\"]\n env_file: .env\n volumes:\n - ./data/uploads:/app/data/uploads\n - ./data/outputs:/app/data/outputs\n - ./models:/app/models\n depends_on: [db]\n healthcheck:\n test: [\"CMD\", \"curl\", \"-f\", \"http://localhost:8000/api/v1/health\"]\n interval: 30s\n retries: 3\n\n db:\n image: postgres:15\n environment:\n POSTGRES_USER: scre_user\n POSTGRES_PASSWORD: password\n POSTGRES_DB: scre_db\n ports: [\"5432:5432\"]\n volumes:\n - postgres_data:/var/lib/postgresql/data\n\n dashboard:\n build: ./services/dashboard\n ports: [\"8501:8501\"]\n profiles: [\"dashboard\"] # opt-in: docker-compose --profile dashboard up\n depends_on: [api]\n\nvolumes:\n postgres_data:\n```\n\n---\n\n## 5. Dockerfile Specifications\n\n### 5.1 API Service Dockerfile\n\n```dockerfile\n# services/api/Dockerfile\nFROM python:3.11-slim\n\nWORKDIR /app\n\n# Install system deps\nRUN apt-get update && apt-get install -y \\\n curl libpq-dev gcc && rm -rf /var/lib/apt/lists/*\n\n# Install Python deps\nCOPY requirements.txt .\nRUN pip install --no-cache-dir -r requirements.txt\n\n# Copy source\nCOPY . .\n\n# Non-root user\nRUN adduser --disabled-password --gecos '' appuser\nUSER appuser\n\nEXPOSE 8000\n\nCMD [\"uvicorn\", \"app.main:app\", \"--host\", \"0.0.0.0\", \"--port\", \"8000\"]\n```\n\n### 5.2 Build Targets\n\n```\ndevelopment: base image (includes dev deps)\nproduction: multi-stage build — strip dev deps, use compiled binaries\n```\n\n---\n\n## 6. CI/CD Pipeline\n\n### 6.1 Pipeline Overview\n\n```\nPull Request → CI Pipeline (GitHub Actions: ci.yml)\n ├── Lint (ruff)\n ├── Format Check (black)\n ├── Type Check (mypy)\n ├── Unit Tests (pytest - packages/engine, packages/db)\n ├── Integration Tests (pytest - services/api with test DB)\n └── Build Docker Image (validate Dockerfile)\n\nMerge to develop → Staging Deploy (deploy.yml - staging job)\n ├── Build & push Docker image to registry\n ├── Deploy to staging environment\n ├── Run Alembic migrations\n └── Smoke test: POST /health\n\nManual approval → Production Deploy (deploy.yml - production job)\n ├── Build & push Docker image (pinned tag)\n ├── Create DB backup\n ├── Deploy to production\n ├── Run Alembic migrations\n └── Smoke test\n```\n\n### 6.2 GitHub Actions: CI Pipeline (`ci.yml`)\n\n```yaml\nname: CI\non:\n pull_request:\n branches: [main, develop]\n\njobs:\n lint-and-test:\n runs-on: ubuntu-latest\n services:\n postgres:\n image: postgres:15\n env:\n POSTGRES_USER: scre_user\n POSTGRES_PASSWORD: testpassword\n POSTGRES_DB: scre_test\n ports: [\"5432:5432\"]\n steps:\n - uses: actions/checkout@v4\n - uses: actions/setup-python@v5\n with: { python-version: '3.11' }\n - name: Install deps\n run: make install\n - name: Lint\n run: ruff check .\n - name: Format check\n run: black --check .\n - name: Type check\n run: mypy packages/ services/api/\n - name: Unit tests\n run: pytest packages/ -v --tb=short\n - name: Integration tests\n env:\n DATABASE_URL: postgresql+asyncpg://scre_user:testpassword@localhost:5432/scre_test\n run: pytest services/api/tests/integration/ -v --tb=short\n - name: Build Docker image\n run: docker build ./services/api -t scre-api:pr-${{ github.run_number }}\n```\n\n### 6.3 Deploy Pipeline (`deploy.yml`)\n\n```yaml\nname: Deploy\non:\n push:\n branches: [develop, main]\n\njobs:\n deploy-staging:\n if: github.ref == 'refs/heads/develop'\n runs-on: ubuntu-latest\n steps:\n - name: Build and push image\n - name: Deploy to staging (docker-compose pull && up -d)\n - name: Run migrations (docker exec alembic upgrade head)\n - name: Smoke test (curl /health)\n\n deploy-production:\n if: github.ref == 'refs/heads/main'\n runs-on: ubuntu-latest\n environment: production # requires manual approval in GitHub\n steps:\n - name: Create DB backup\n - name: Build and push image (tagged with git SHA)\n - name: Deploy to production\n - name: Run migrations\n - name: Smoke test\n - name: Notify team (Slack/email)\n```\n\n---\n\n## 7. Container Registry\n\n| Environment | Registry | Image Tag Pattern |\n|---|---|---|\n| Development | Local Docker | `scre-api:dev` |\n| Staging | GitHub Container Registry | `scre-api:develop-{sha}` |\n| Production | GitHub Container Registry | `scre-api:v{major}.{minor}.{patch}` |\n\n---\n\n## 8. Database Management\n\n### 8.1 Migration Commands\n\n```bash\n# Run all pending migrations\nmake migrate\n\n# Generate a new migration\ncd packages/db && alembic revision --autogenerate -m \"description\"\n\n# Downgrade one version\ncd packages/db && alembic downgrade -1\n\n# Check current revision\ncd packages/db && alembic current\n```\n\n### 8.2 Backup Strategy\n\n| Environment | Backup Frequency | Retention | Method |\n|---|---|---|---|\n| Staging | Daily | 7 days | pg_dump to S3 / cloud storage |\n| Production | Hourly | 30 days | Managed DB automated backup |\n\n### 8.3 Migration Policy\n\n- **Always** run migrations before deploying new application version\n- **Never** modify existing migration files — create new ones\n- **Test** migrations on staging before production\n\n---\n\n## 9. Secrets Management\n\n| MVP (Local/Staging) | Production |\n|---|---|\n| `.env` files | Cloud secrets manager (AWS Secrets Manager / Azure Key Vault) |\n| Values set in GitHub Actions secrets | Injected at runtime, not stored in image |\n\n**Rules:**\n- API keys rotated every 90 days\n- Database passwords never shared via messaging apps\n- `.env` files excluded from version control via `.gitignore`\n- `grep` the repo for any hard-coded secrets as part of CI (secret scanning)\n\n---\n\n## 10. Observability Setup\n\n### 10.1 Structured Logging\n\nAll API request and pipeline stage logs emitted as JSON:\n\n```json\n{\n \"timestamp\": \"2026-03-05T21:51:00Z\",\n \"level\": \"INFO\",\n \"service\": \"scre-api\",\n \"event\": \"pipeline_stage_completed\",\n \"run_id\": \"r9f8e7d6-...\",\n \"stage\": \"FEATURE_ENGINEERING\",\n \"duration_ms\": 1240,\n \"record_count\": 10432\n}\n```\n\nLog output directed to `stdout` (captured by Docker / cloud log aggregator).\n\n### 10.2 Health Check Endpoint\n\n`GET /api/v1/health` — returns service component statuses:\n```json\n{\n \"status\": \"ok\",\n \"database\": \"connected\",\n \"model_loaded\": true,\n \"active_model_version\": \"v1.0\",\n \"uptime_seconds\": 38420\n}\n```\n\n### 10.3 Metrics (Phase 2)\n\nFuture metrics to integrate via Prometheus:\n- `scre_batch_duration_seconds` (histogram)\n- `scre_records_processed_total` (counter)\n- `scre_api_request_duration_seconds` (histogram)\n- `scre_anomaly_rate` (gauge per run)\n\n---\n\n## 11. Rollback Procedure\n\nIf a production deployment causes failures:\n\n```bash\n# Step 1: Identify last good image tag (from registry)\n# Step 2: Deploy previous image\ndocker pull ghcr.io/org/scre-api:v{previous-version}\ndocker tag ghcr.io/org/scre-api:v{previous-version} scre-api:current\ndocker-compose up -d\n\n# Step 3: If DB migration rolled forward, run downgrade\ncd packages/db && alembic downgrade -1\n\n# Step 4: Verify health\ncurl https://api.scre.io/api/v1/health\n\n# Step 5: Post-mortem within 48h\n```\n\n---\n\n## 12. Developer Runbook Summary\n\n| Command | Purpose |\n|---|---|\n| `make dev` | Start full stack |\n| `make migrate` | Apply DB migrations |\n| `make test` | Run all tests |\n| `make lint` | Lint all code |\n| `make format` | Auto-format code |\n| `make train` | Train new model version |\n| `make benchmark` | Run pipeline benchmark |\n| `make seed` | Seed test data |\n"}}}} |