1 line
17 KiB
JSON
1 line
17 KiB
JSON
{"repo": "askmy-stack/py-outlier", "files": {"src/anomaly_detection/api/auth.py": {"64e51c7efde5a6106519975ac359e442f51a0da2": {"t": 1785596328.451854, "n": 1430, "h": "919112059a53af9b", "c": "\"\"\"Optional API key authentication for the anomaly detection service.\"\"\"\n\nfrom __future__ import annotations\n\nimport os\nfrom collections.abc import Awaitable, Callable\n\nfrom starlette.middleware.base import BaseHTTPMiddleware\nfrom starlette.requests import Request\nfrom starlette.responses import JSONResponse, Response\n\nAPI_KEY_HEADER = \"X-API-Key\"\nAPI_KEY_ENV_VAR = \"ANOMALY_API_KEY\"\n\n# Paths that never require an API key, regardless of configuration.\nPUBLIC_PATHS = frozenset({\"/health\", \"/docs\", \"/redoc\", \"/openapi.json\"})\n\n\ndef _configured_api_key() -> str:\n return os.environ.get(API_KEY_ENV_VAR, \"\").strip()\n\n\nclass ApiKeyMiddleware(BaseHTTPMiddleware):\n \"\"\"Require ``X-API-Key`` when ``ANOMALY_API_KEY`` is configured.\n\n The service runs open (no auth) when ``ANOMALY_API_KEY`` is unset, which\n matches local/dev usage. Set ``ANOMALY_API_KEY`` to require a matching\n ``X-API-Key`` header on every request except the public paths above.\n \"\"\"\n\n async def dispatch(self, request: Request, call_next: Callable[..., Awaitable[Response]]) -> Response:\n configured = _configured_api_key()\n if not configured or request.url.path in PUBLIC_PATHS:\n return await call_next(request)\n\n if request.headers.get(API_KEY_HEADER) == configured:\n return await call_next(request)\n\n return JSONResponse(status_code=401, content={\"detail\": \"Missing or invalid API key\"})\n"}}, "README.md": {"64e51c7efde5a6106519975ac359e442f51a0da2": {"t": 1785596328.5200632, "n": 9967, "h": "839b890d1e880c76", "c": "# Py-outlier\n\n**Modular anomaly detection for tabular, time-series, and vision data — CLI, REST API, streaming, RCA, fairness, and optional LLM explanations.**\n\n[](https://github.com/askmy-stack/Py-Outlier/actions/workflows/ci.yml)\n[](LICENSE)\n[](https://www.python.org/downloads/)\n\n\n\n**Py-outlier** is a community-ready Python framework evolved from UCF crime-classification notebooks into a modular anomaly-detection toolkit. Ship detectors in production via CLI or FastAPI, benchmark across registered datasets, stream scores online, rank root causes, audit fairness, and optionally explain anomalies with an LLM.\n\n> **Package name:** The installable Python package remains `anomaly_detection` (`pip install anomaly-detection`). **Py-outlier** is the project brand; imports use `import anomaly_detection`.\n\n## Features\n\n| Area | Capabilities |\n| --- | --- |\n| **Detectors** | z-score, IQR, Isolation Forest, LOF, One-Class SVM, autoencoder, diffusion reconstruction |\n| **CLI** | `detect`, `benchmark`, `stream` |\n| **REST API** | Tabular detection, batch CSV upload, RCA, LLM explain, optional vision classification |\n| **Data** | Registry-driven loaders (OpenML, CSV, UCF fixtures) |\n| **Streaming** | Online z-score window; optional PySAD wrappers |\n| **RCA** | Causal graph scoring with ranked root causes |\n| **Vision** | UCF 14-class image/video classification (supervised, separate from `/detect`) |\n| **Fairness** | Demographic parity, equalized odds, reweighing mitigation (AIF360) |\n| **LLM** | Opt-in anomaly explanations with PII redaction |\n| **Multimodal** | Experimental tabular+text fusion |\n\nSee [docs/EXECUTION_PLAN.md](docs/EXECUTION_PLAN.md) for the phased roadmap and [CHANGELOG.md](CHANGELOG.md) for release history.\n\n## Quick Start\n\n```bash\ngit clone https://github.com/askmy-stack/Py-Outlier.git\ncd Py-Outlier\npython -m venv .venv\nsource .venv/bin/activate # Windows: .venv\\Scripts\\activate\npip install -e \".[dev]\"\ndetect --config configs/default.yaml\nbenchmark --quick\nserve # REST API on http://localhost:8000\n```\n\n## Open Source Contributions\n\nWe welcome contributions across detectors, data, APIs, vision, streaming, fairness, docs, and evaluation. Pick an area below and open a draft PR — see [CONTRIBUTING.md](CONTRIBUTING.md) for the full workflow.\n\n| Area | How to contribute | Module path | Skills |\n| --- | --- | --- | --- |\n| **Detectors** | Add a new algorithm or sklearn wrapper; register in the factory | `src/anomaly_detection/models/` | Python, ML |\n| **Data loaders** | Add registry entries and loader adapters for new datasets | `src/anomaly_detection/data_ingestion/` | Python, pandas |\n| **API & CLI** | Extend FastAPI routes or CLI flags (`detect`, `benchmark`, `stream`) | `src/anomaly_detection/api/`, `src/anomaly_detection/cli/` | Python, FastAPI |\n| **Vision** | Improve UCF module, Grad-CAM, or TensorFlow model paths | `src/anomaly_detection/domains/vision/` | Python, CV |\n| **Streaming** | Wrap PySAD detectors or extend the online z-score window | `src/anomaly_detection/streaming/` | Python, time-series |\n| **RCA** | Improve causal graph scoring and metric ranking | `src/anomaly_detection/rca/` | Python, statistics |\n| **Fairness & ethics** | Extend AIF360 metrics and bias mitigation | `src/anomaly_detection/fairness/` | ML fairness |\n| **LLM** | Tune explainer prompts and PII redaction rules | `src/anomaly_detection/llm/` | Python, LLM APIs |\n| **Docs & tutorials** | Add or update guides under `docs/tutorials/` | `docs/tutorials/` | Markdown |\n| **Evaluation** | Extend benchmark harness, profiler, and metrics | `src/anomaly_detection/evaluation/` | Python, pytest |\n\nAlso see [docs/EXECUTION_PLAN.md](docs/EXECUTION_PLAN.md) for phased priorities and [docs/tutorials/](docs/tutorials/) for domain walkthroughs.\n\n## Project structure\n\n```\nPy-Outlier/\n├── src/anomaly_detection/ # installable package (import path unchanged)\n│ ├── models/ # tabular & deep detectors\n│ ├── api/ # FastAPI routes\n│ ├── cli/ # detect, benchmark, stream\n│ ├── data_ingestion/ # registry + loaders\n│ ├── domains/vision/ # UCF classification (optional [vision] extra)\n│ ├── streaming/ # online detectors\n│ ├── rca/ # root cause analysis\n│ ├── fairness/ # bias metrics & mitigation\n│ ├── llm/ # anomaly explainer\n│ └── multimodal/ # fusion (experimental)\n├── configs/ # YAML configuration\n├── datasets/registry.yaml # dataset metadata & licenses\n├── docs/\n│ ├── EXECUTION_PLAN.md # phased roadmap\n│ └── tutorials/ # step-by-step domain guides\n├── tests/ # pytest suite\n├── examples/notebooks/ # original vision notebooks\n└── assets/ # README demo GIF and media\n```\n\n## Setup\n\nRequires Python 3.11+.\n\n```bash\ngit clone https://github.com/askmy-stack/Py-Outlier.git\ncd Py-Outlier\npython -m venv .venv\nsource .venv/bin/activate # Windows: .venv\\Scripts\\activate\npip install -e \".[dev]\"\n```\n\n### Optional extras\n\n| Extra | Install command | Enables |\n| --- | --- | --- |\n| `streaming` | `pip install -e \".[dev,streaming]\"` | PySAD streaming detectors |\n| `rca` | `pip install -e \".[dev,rca]\"` | Root cause analysis (PyRCA) |\n| `vision` | `pip install -e \".[dev,vision]\"` | TensorFlow + OpenCV UCF endpoints |\n| `fairness` | `pip install -e \".[dev,fairness]\"` | AIF360 fairness metrics |\n| `generative` | `pip install -e \".[dev,generative]\"` | Diffusion reconstruction detector |\n| `llm` | `pip install -e \".[dev,llm]\"` | Anthropic LLM explainer |\n\nVerify the install:\n\n```bash\nruff check src tests\npytest tests/ -q\ndetect --config configs/default.yaml\nbenchmark --quick\nserve # REST API on http://localhost:8000\n```\n\n## CLI\n\n| Command | Description |\n| --- | --- |\n| `detect --config CONFIG` | Run detection; writes JSON report and optional plot |\n| `benchmark --quick` | Benchmark all registry datasets × detectors on fixtures |\n| `benchmark --quick --profile` | Same, with wall-time and peak-memory profiling |\n| `stream --config CONFIG` | Online streaming detection |\n\nExample:\n\n```bash\npython -m anomaly_detection.cli.detect --config configs/default.yaml\npython -m anomaly_detection.cli.benchmark --quick --profile\n```\n\n## REST API\n\nStart the server with `serve` (or `uvicorn anomaly_detection.api.app:app`). Interactive docs at `/docs`.\n\n| Endpoint | Method | Description |\n| --- | --- | --- |\n| `/health` | GET | Liveness check |\n| `/detect` | POST | Detect anomalies from a 2D numeric array + optional config override |\n| `/detect/batch` | POST | Upload a CSV file for batch detection |\n| `/models` | GET | List registered detector names |\n| `/root_cause` | POST | Rank root causes for an anomaly given multivariate metrics |\n| `/root_cause/{anomaly_id}` | GET | Retrieve a cached RCA result |\n| `/explain` | POST | Generate plain-language anomaly explanation (LLM opt-in) |\n| `/vision/analyze/image` | POST | Classify an image into 14 UCF crime categories (`[vision]` extra) |\n| `/vision/analyze/video` | POST | Classify a video via frame sampling (`[vision]` extra) |\n\n**Note:** Vision endpoints perform **supervised multi-class classification**, not unsupervised anomaly detection. They are intentionally separate from `/detect`.\n\n### Authentication\n\nAll routes above except `/health`, `/docs`, `/redoc`, and `/openapi.json` require\nan `X-API-Key` header matching `ANOMALY_API_KEY` when that env var is set. Leave\nit unset for local/dev use (no auth required).\n\n`/detect` and `/detect/batch` reject payloads over `MAX_DETECT_ROWS` rows\n(default 100,000) with `413 Content Too Large`.\n\n## Benchmark results (quick mode, fixtures)\n\nRun `benchmark --quick` locally to reproduce. Representative results on test fixtures:\n\n| Dataset | Detector | Precision | Recall | F1 | ROC-AUC |\n| --- | --- | ---: | ---: | ---: | ---: |\n| credit_card_fraud | isolation_forest | 1.0000 | 1.0000 | 1.0000 | 1.0000 |\n| nab | isolation_forest | 1.0000 | 1.0000 | 1.0000 | 1.0000 |\n| ucf_crime | lof | 1.0000 | 1.0000 | 1.0000 | 1.0000 |\n\nProfiling (isolation_forest on credit_card_fraud fixture): ~0.12 s wall time, ~194 MB peak memory on Apple Silicon (Python 3.13).\n\n## Datasets\n\nRegistered in [datasets/registry.yaml](datasets/registry.yaml):\n\n| ID | Domain | License |\n| --- | --- | --- |\n| `credit_card_fraud` | tabular | CC-BY-4.0 |\n| `nab` | timeseries | AGPL-3.0 |\n| `ucf_crime` | vision | Custom (academic use) |\n\n## Tutorials\n\n| Tutorial | Topic |\n| --- | --- |\n| [01-tabular-fraud](docs/tutorials/01-tabular-fraud.md) | Credit-card fraud (`configs/examples/fraud.yaml`) |\n| [02-timeseries-iot](docs/tutorials/02-timeseries-iot.md) | NAB time-series and IoT streaming |\n| [03-vision-surveillance](docs/tutorials/03-vision-surveillance.md) | UCF vision classification (supervised) |\n| [04-streaming](docs/tutorials/04-streaming.md) | Online `stream` CLI |\n| [05-fairness](docs/tutorials/05-fairness.md) | Fairness metrics and mitigation |\n\n## Vision domain (legacy notebooks)\n\nPre-trained SavedModel artifacts remain at the repository root:\n\n- `Image Anomaly Detection-2/` — image classifier\n- `Video Anomaly Detection/` — video classifier\n\nConfigure paths in `configs/examples/vision.yaml`. To explore the original notebooks:\n\n```bash\njupyter notebook examples/notebooks/\n```\n\n## Contributing\n\nSee [CONTRIBUTING.md](CONTRIBUTING.md). Please read [docs/EXECUTION_PLAN.md](docs/EXECUTION_PLAN.md) before starting substantial work.\n\n## License\n\nMIT — see [LICENSE](LICENSE).\n\n---\n\n**Py-outlier** by [Abhinaysai Kamineni](https://github.com/askmy-stack) · [GitHub](https://github.com/askmy-stack/Py-Outlier)\n"}}, "SECURITY.md": {"64e51c7efde5a6106519975ac359e442f51a0da2": {"t": 1785596328.7061143, "n": 2551, "h": "8aeb0f09f474f27b", "c": "# Security Policy\n\n## Reporting a Vulnerability\n\nPlease **do not** open a public GitHub issue for security vulnerabilities.\n\n- Preferred: use GitHub's private vulnerability reporting for this repository\n (Security tab → **Report a vulnerability**).\n- Alternative: email **kamineniabhinaysai@gmail.com** with a description of\n the issue, steps to reproduce, and its potential impact.\n\nWe aim to acknowledge reports within 7 days and to keep you updated as we\ninvestigate and fix confirmed issues.\n\n## Supported Versions\n\nPy-Outlier does not yet have tagged releases; security fixes are applied to\nthe `main` branch. Run the latest `main` to get fixes as soon as they land.\n\n## API Authentication\n\nThe FastAPI service (`src/anomaly_detection/api/`) supports optional\n`X-API-Key` authentication via `ApiKeyMiddleware`. Set `ANOMALY_API_KEY` to\nrequire a matching key on every route except `/health`, `/docs`, `/redoc`,\nand `/openapi.json`. **Leaving `ANOMALY_API_KEY` unset runs the service in\nopen mode with no auth** — fine for local/dev use, but set it before\nexposing the service on a network. CORS is also wide open\n(`allow_origins=[\"*\"]`) by default.\n\n`/detect` and `/detect/batch` cap accepted rows via `MAX_DETECT_ROWS`\n(default 100,000) to bound memory/CPU usage from a single request.\n\n## LLM Usage and PII Handling\n\nThe optional LLM explainer (`llm.enabled` in config, disabled by default)\nsends anomaly context — sample indices, scores, predictions, and\nfeature names/values — to **Anthropic** via `ANTHROPIC_API_KEY` when enabled.\n\nBefore that data is sent, and again on the model's response,\n`src/anomaly_detection/llm/redaction.py` applies regex-based redaction for:\n\n- Email addresses → `[REDACTED_EMAIL]`\n- US-format phone numbers → `[REDACTED_PHONE]`\n- SSN-shaped values → `[REDACTED_SSN]`\n\n**Limitations:** this redaction is pattern-based only. It does not catch\nnames, addresses, credit card numbers, or other free-text PII, and it is\nnot applied to `/detect` or `/detect/batch` inputs (which are processed\nlocally and never sent to an LLM by those endpoints themselves). Do not\nenable the LLM explainer on datasets containing PII types beyond emails,\nphone numbers, and SSNs unless you've reviewed and extended the redaction\npatterns for your data.\n\n## Data Handled\n\nThe service processes user-supplied tabular numeric data (JSON or CSV\nupload) and, if the `[vision]` extra is installed, images/video for\nclassification. There is no persistent user database; data is processed\nin-memory unless you configure output paths yourself.\n"}}, ".env.example": {"64e51c7efde5a6106519975ac359e442f51a0da2": {"t": 1785596328.7943811, "n": 370, "h": "0e0becc2fd5855d4", "c": "# Copy to .env and fill in values. LLM features are opt-in (llm.enabled: false by default).\nANTHROPIC_API_KEY=\n\n# Optional: require X-API-Key on all API routes except /health, /docs, /redoc,\n# and /openapi.json. Unset = open (dev) mode, no auth required.\nANOMALY_API_KEY=\n\n# Optional: max rows accepted by /detect and /detect/batch. Defaults to 100000.\nMAX_DETECT_ROWS=\n"}}, "tests/api/test_auth.py": {"64e51c7efde5a6106519975ac359e442f51a0da2": {"t": 1785596337.5533614, "n": 1765, "h": "dda90d3c507e5938", "c": "\"\"\"Tests for optional API key authentication.\"\"\"\n\nfrom __future__ import annotations\n\nimport pytest\nfrom fastapi.testclient import TestClient\n\nfrom anomaly_detection.api.app import app\n\nclient = TestClient(app)\n\n\ndef test_open_mode_when_no_key_configured(monkeypatch: pytest.MonkeyPatch) -> None:\n monkeypatch.delenv(\"ANOMALY_API_KEY\", raising=False)\n response = client.get(\"/models\")\n assert response.status_code == 200\n\n\ndef test_rejects_missing_key_when_configured(monkeypatch: pytest.MonkeyPatch) -> None:\n monkeypatch.setenv(\"ANOMALY_API_KEY\", \"sk_test\")\n response = client.get(\"/models\")\n assert response.status_code == 401\n\n\ndef test_rejects_invalid_key_when_configured(monkeypatch: pytest.MonkeyPatch) -> None:\n monkeypatch.setenv(\"ANOMALY_API_KEY\", \"sk_test\")\n response = client.get(\"/models\", headers={\"X-API-Key\": \"wrong\"})\n assert response.status_code == 401\n\n\ndef test_accepts_valid_key_when_configured(monkeypatch: pytest.MonkeyPatch) -> None:\n monkeypatch.setenv(\"ANOMALY_API_KEY\", \"sk_test\")\n response = client.get(\"/models\", headers={\"X-API-Key\": \"sk_test\"})\n assert response.status_code == 200\n\n\ndef test_health_is_always_public(monkeypatch: pytest.MonkeyPatch) -> None:\n monkeypatch.setenv(\"ANOMALY_API_KEY\", \"sk_test\")\n response = client.get(\"/health\")\n assert response.status_code == 200\n\n\ndef test_cors_preflight_bypasses_api_key(monkeypatch: pytest.MonkeyPatch) -> None:\n monkeypatch.setenv(\"ANOMALY_API_KEY\", \"sk_test\")\n response = client.options(\n \"/detect\",\n headers={\"Origin\": \"http://example.com\", \"Access-Control-Request-Method\": \"POST\"},\n )\n assert response.status_code == 200\n assert response.headers.get(\"access-control-allow-origin\") == \"http://example.com\"\n"}}}} |