1 line
17 KiB
JSON
1 line
17 KiB
JSON
{"repo": "cbritt0n/loggem", "files": {"README.md": {"40d9c3be97fc3f15817e9dd1fccc39f64dfd1532": {"t": 1785596415.334218, "n": 16846, "h": "9ff61540edb3c079", "c": "# LogGem 💎\n\n[](https://github.com/cbritt0n/loggem/releases)\n[](https://github.com/cbritt0n/loggem/actions)\n[](https://github.com/cbritt0n/loggem)\n[](https://github.com/cbritt0n/loggem)\n[](LICENSE)\n[](https://www.python.org/)\n[](https://github.com/psf/black)\n[](https://github.com/cbritt0n/loggem)\n\n**AI-Powered Log Anomaly Detector with Enterprise Features**\n\nI built LogGem to detect suspicious patterns in logs using AI models that run locally or in the cloud. It helps you find brute force attacks, insider threats, and misconfigurations before they become problems.\n\n**Key Strengths**: Real-time streaming • Multi-channel alerting • Works offline • 142 passing tests\n\n---\n\n## 📑 Table of Contents\n\n- [Problem Statement](#-problem-statement)\n- [Features](#-features)\n- [Quick Start](#-quick-start)\n- [LLM Provider Configuration](#-llm-provider-configuration)\n- [Usage](#-usage)\n- [Enterprise Features](#-enterprise-features)\n- [Multi-Format Support](#-multi-format-support)\n- [Architecture](#-architecture)\n- [Configuration](#-configuration)\n- [Testing](#-testing)\n- [Documentation](#-documentation)\n- [Contributing](#-contributing)\n- [License](#-license)\n\n---\n\n## 🎯 Problem Statement\n\nSecurity teams are overwhelmed by logs. I built LogGem to provide industrial-strength log analysis that runs anywhere - from your laptop to production servers - at zero cost.\n\n---\n\n## ✨ Features\n\n### Core Capabilities\n- 🤖 **AI-Powered Detection**: Leverages large language models to understand log context and identify anomalies\n- 🔌 **Pluggable LLM Providers**: HuggingFace (local), OpenAI, Anthropic, Ollama - choose what works for you\n- 📊 **12 Built-in Parsers**: Syslog, JSON, Nginx, Auth, Apache, Windows Event, PostgreSQL, MySQL, Docker, Kubernetes, HAProxy, Redis\n- 🔒 **Privacy-First**: Run completely offline with local models - your logs never leave your network\n- 🎛️ **Adjustable Sensitivity**: Fine-tune detection from permissive (fewer alerts) to strict (catch everything)\n- 🏗️ **Modular Architecture**: Use components independently or as a complete solution\n- 📝 **Rich Output**: Beautiful terminal output with colors, progress bars, and JSON/CSV/HTML export\n- 🛠️ **Extensible**: Easy to add custom parsers and detection rules\n\n### Enterprise Features\n- ⚡ **Real-Time Streaming**: Monitor log files as they grow with tail-style streaming\n- 🚨 **Advanced Alerting**: Route alerts to Slack, Email, PagerDuty, or Webhooks based on rules\n- 🐳 **Container & Cloud Native**: Docker, Kubernetes, and modern infrastructure support\n- 🗄️ **Database Logs**: PostgreSQL, MySQL, Redis log analysis and security monitoring\n- 🚀 **Performance**: Batch processing and intelligent caching for high-throughput scenarios\n- 📊 **Reporting**: Export analysis results to JSON, CSV, or HTML\n\n### Quality Assurance\n- ✅ **142 passing tests**, 0 failures\n- 📈 **56% code coverage** (90%+ on core modules)\n- 🔬 **CI/CD integrated** with multi-OS testing\n- 📦 **Modular dependencies** - install only what you need\n\n---\n\n## 🚀 Quick Start\n\n### Installation\n\n```bash\n# Clone the repository\ngit clone https://github.com/cbritt0n/loggem.git\ncd loggem\n\n# Create virtual environment\npython3 -m venv venv\nsource venv/bin/activate # On Windows: venv\\Scripts\\activate\n\n# Install LogGem with your preferred provider:\n\n# Option 1: HuggingFace (local models - includes Gemma 3) - RECOMMENDED\npip install -e \".[huggingface]\"\n\n# Option 2: OpenAI (cloud API)\npip install -e \".[openai]\"\nexport OPENAI_API_KEY=\"sk-...\"\n\n# Option 3: Anthropic (cloud API)\npip install -e \".[anthropic]\"\nexport ANTHROPIC_API_KEY=\"sk-ant-...\"\n\n# Option 4: Ollama (local API - requires Ollama installed)\npip install -e \".[ollama]\"\n\n# Option 5: All providers\npip install -e \".[all]\"\n\n# Development installation\npip install -e \".[dev]\"\n```\n\n### Basic Usage\n\n```bash\n# Analyze a log file (uses configured provider)\nloggem analyze /var/log/auth.log\n\n# Watch a log file in real-time\nloggem watch /var/log/auth.log\n\n# Analyze multiple files\nloggem analyze /var/log/auth.log /var/log/syslog\n\n# Custom sensitivity (0.0 = permissive, 1.0 = strict)\nloggem analyze auth.log --sensitivity 0.8\n\n# Check system info\nloggem info\n```\n\n---\n\n## 🤖 LLM Provider Configuration\n\n### HuggingFace (Local - DEFAULT)\n\nBest for: Offline use, data privacy, no recurring costs\n\n```yaml\n# config.yaml\nmodel:\n provider: \"huggingface\"\n name: \"google/gemma-3-4b-it\" # Default: Gemma 3 4B\n # Also supported:\n # - google/gemma-3-12b-it (better accuracy, needs 16GB RAM)\n # - google/gemma-3-27b-it (best accuracy, needs 34GB RAM)\n # - meta-llama/Llama-3.2-3B-Instruct\n # - mistralai/Mistral-7B-Instruct-v0.3\n device: \"auto\" # auto, cpu, cuda, mps\n quantization: \"int8\" # int8 (4x smaller), fp16, fp32\n cache_dir: \"./models\"\n```\n\n**Requirements**: \n- Gemma 3 4B: ~8GB RAM, ~4GB disk space\n- Gemma 3 12B: ~16GB RAM, ~12GB disk space\n- Gemma 3 27B: ~34GB RAM, ~27GB disk space\n\n### OpenAI (Cloud API)\n\nBest for: Highest accuracy, no local resources needed\n\n```yaml\nmodel:\n provider: \"openai\"\n name: \"gpt-4o-mini\" # or gpt-4o, gpt-4-turbo\n api_key: null # or set OPENAI_API_KEY env var\n```\n\n**Cost**: ~$0.15 per 1M tokens (gpt-4o-mini), ~$2.50 per 1M tokens (gpt-4o)\n\n### Anthropic (Cloud API)\n\nBest for: Claude models, strong reasoning\n\n```yaml\nmodel:\n provider: \"anthropic\"\n name: \"claude-3-haiku-20240307\" # or claude-3-5-sonnet\n api_key: null # or set ANTHROPIC_API_KEY env var\n```\n\n**Cost**: ~$0.25 per 1M tokens (Haiku), ~$3.00 per 1M tokens (Sonnet)\n\n### Ollama (Local API)\n\nBest for: Easy local deployment, model management\n\n```yaml\nmodel:\n provider: \"ollama\"\n name: \"llama3\" # or mistral, gemma, qwen\n base_url: \"http://localhost:11434\"\n```\n\n**Requirements**: Ollama installed, ~8GB RAM\n\n---\n\n## 📋 Multi-Format Support\n\nI've built LogGem to support a wide variety of log formats out-of-the-box:\n\n### Built-in Parsers\n\n| Format | Description | Use Cases |\n|--------|-------------|-----------|\n| **Syslog** | RFC 3164/5424 system logs | System logs, network devices, legacy applications |\n| **Auth Logs** | Linux authentication logs | Security auditing, brute force detection, SSH monitoring |\n| **Nginx** | Nginx access/error logs | Web security, DDoS detection, API monitoring |\n| **Apache** | Apache access/error logs | Web server monitoring, attack detection |\n| **JSON** | Structured JSON logs | Microservices, cloud applications, modern apps |\n| **Windows Event** | Windows Security/System/Application logs | Windows security, Active Directory, system monitoring |\n| **PostgreSQL** | PostgreSQL database logs | Database security, query monitoring, error tracking |\n| **MySQL** | MySQL/MariaDB database logs | Database performance, security auditing |\n| **Docker** | Docker container logs | Container monitoring, microservices debugging |\n| **Kubernetes** | Kubernetes cluster logs | Pod monitoring, cluster events, container runtime |\n| **HAProxy** | HAProxy load balancer logs | Load balancer monitoring, traffic analysis |\n| **Redis** | Redis database logs | Cache monitoring, replication tracking |\n\n### Auto-Detection\n\n```bash\n# LogGem automatically detects log format\nloggem analyze /var/log/mystery.log\n\n# Or specify explicitly\nloggem analyze /var/log/auth.log --format auth\nloggem analyze /var/log/nginx/access.log --format nginx\nloggem analyze /var/log/postgresql/postgresql-15-main.log --format postgresql\nloggem analyze /var/log/mysql/error.log --format mysql\nloggem analyze /var/log/containers/app.log --format docker\nloggem analyze kubectl-logs.txt --format kubernetes\nloggem analyze /var/log/haproxy.log --format haproxy\nloggem analyze /var/log/redis/redis-server.log --format redis\n```\n\n### Custom Parsers\n\nExtend LogGem with your own parsers - see [EXAMPLES.md](EXAMPLES.md) for details.\n\n---\n\n## 🏗️ Architecture\n\nI've built LogGem with modularity in mind:\n\n```\nloggem/\n├── core/ # Configuration, logging, data models\n├── parsers/ # Log format parsers (independently usable)\n├── detector/ # LLM providers and anomaly detection\n├── analyzer/ # Pattern analysis and statistical detection\n├── streaming/ # Real-time log monitoring\n├── alerting/ # Multi-channel alerting system\n├── reporting/ # Report generation and export\n└── cli/ # Command-line interface\n```\n\nEach module can be used independently in your own projects. See [ARCHITECTURE.md](ARCHITECTURE.md) for detailed documentation.\n\n---\n\n## 📖 Configuration\n\nCreate a `config.yaml` file (see `config.example.yaml` for full options):\n\n```yaml\n# LLM Provider Configuration\nmodel:\n provider: \"huggingface\" # or openai, anthropic, ollama\n name: \"google/gemma-3-4b-it\"\n device: \"auto\"\n quantization: \"int8\"\n\n# Detection Settings\ndetection:\n sensitivity: 0.75 # 0.0 (permissive) to 1.0 (strict)\n batch_size: 32\n context_window: 100 # Number of log entries for context\n\n# Alerting\nalerting:\n enabled: true\n severity_threshold: \"medium\" # low, medium, high, critical\n channels:\n - type: \"slack\"\n webhook_url: \"https://hooks.slack.com/...\"\n - type: \"email\"\n smtp_server: \"smtp.gmail.com\"\n smtp_port: 587\n \n# Logging\nlogging:\n level: \"INFO\" # DEBUG, INFO, WARNING, ERROR, CRITICAL\n file: \"./logs/loggem.log\"\n rotation: \"100 MB\"\n```\n\n---\n\n## � Enterprise Features\n\n### Real-Time Streaming\n\nWatch log files as they grow:\n\n```python\nfrom loggem.streaming import LogStreamer, tail_file\n\n# Simple tail (like tail -f)\nfor entry in tail_file(\"/var/log/syslog\", lines=10, follow=True):\n print(entry.message)\n\n# Advanced streaming with callbacks\nwith LogStreamer(\"/var/log/nginx/access.log\", follow=True) as streamer:\n for event in streamer.iter_events():\n # Real-time processing\n result = detector.analyze_entry(event.entry)\n if result.anomalies:\n print(f\"🚨 Anomaly detected: {result.anomalies[0].reasoning}\")\n```\n\n### Advanced Alerting\n\nRoute alerts to the right channels:\n\n```python\nfrom loggem.alerting import (\n AlertManager, AlertChannel, AlertSeverity,\n create_high_score_rule, SlackChannel\n)\n\n# Setup alert manager\nmanager = AlertManager()\n\n# Add rules\nmanager.add_rule(create_high_score_rule(threshold=0.8))\n\n# Configure channels\nslack = SlackChannel(webhook_url=\"your-webhook-url\")\nmanager.add_channel(AlertChannel.SLACK, slack)\n\n# Process anomalies\nfor anomaly in anomalies:\n manager.process_anomaly(anomaly) # Automatically routes to channels\n```\n\n### Windows Event Logs\n\nParse Windows events natively:\n\n```python\nfrom loggem.parsers.windows_event import WindowsEventLogParser\n\nparser = WindowsEventLogParser()\nentry = parser.parse_line(xml_event)\nprint(f\"Event ID: {entry.metadata['event_id']}\")\n```\n\n### Performance Optimization\n\nHandle high-throughput scenarios:\n\n```python\nfrom loggem.performance import (\n BatchProcessor, AnalysisCache, cached_analysis\n)\n\n# Batch processing for high throughput\nprocessor = BatchProcessor(batch_size=100, max_workers=4)\nresults = processor.process_entries(entries, analyze_func)\n\n# Caching for repeated analyses\ncache = AnalysisCache(maxsize=1000)\n\n@cached_analysis(cache)\ndef analyze_log(entry):\n return detector.analyze_entry(entry)\n\n# Automatic cache hit on duplicate entries\nfor entry in entries:\n result = analyze_log(entry) # Cached after first call\n```\n\n### Reporting\n\nExport results in multiple formats:\n\n```python\nfrom loggem.reporting import ReportGenerator\n\n# Generate comprehensive reports\nreport = ReportGenerator(analysis_result)\n\n# Export to multiple formats\nreport.export_json(\"report.json\")\nreport.export_csv(\"report.csv\")\nreport.export_html(\"report.html\")\nreport.print_summary() # Console output with colors\n```\n\n---\n\n## 🧪 Testing\n\nI've made sure LogGem has 142 passing tests covering all major functionality:\n\n```bash\n# Run all tests\npytest tests/\n\n# With coverage\npytest tests/ --cov=loggem --cov-report=html\n\n# View coverage report\nopen htmlcov/index.html\n```\n\n**Coverage highlights:**\n- Core models: 92%\n- Configuration: 96%\n- Parsers: 93%\n- Enterprise features: 60-93%\n\nSee [TESTING.md](TESTING.md) for details.\n\n---\n\n## 📚 Documentation\n\n- **[README.md](README.md)** - Main documentation (you are here)\n- **[EXAMPLES.md](EXAMPLES.md)** - Comprehensive usage examples\n- **[ARCHITECTURE.md](ARCHITECTURE.md)** - System architecture and design\n- **[TESTING.md](TESTING.md)** - Testing guide and best practices\n- **[DEPLOYMENT.md](DEPLOYMENT.md)** - Production deployment guide\n- **[CONTRIBUTING.md](CONTRIBUTING.md)** - Contribution guidelines\n- **[SECURITY.md](SECURITY.md)** - Security policy\n- **[RELEASE.md](RELEASE.md)** - Release template\n- **[docs/COVERAGE_REPORT.md](docs/COVERAGE_REPORT.md)** - Test coverage analysis\n\n---\n\n## 🔒 Security Features\n\n- **Input Validation**: All log entries sanitized before processing\n- **Secure Model Loading**: Verification of model checksums, no remote code execution\n- **Audit Logging**: Complete audit trail of all operations\n- **No Data Exfiltration**: Runs completely offline with local models\n- **Rate Limiting**: Protection against resource exhaustion\n- **Secure Defaults**: Conservative security settings out-of-the-box\n\nSee [SECURITY.md](SECURITY.md) for security policy and reporting vulnerabilities.\n\n---\n\n## 🎯 Use Cases\n\nHere's what I designed LogGem to help you with:\n\n### Brute Force Detection\nI've built LogGem to identify repeated failed login attempts, credential stuffing, and password spraying attacks in authentication logs.\n\n### Insider Threat Detection\nIt spots unusual access patterns, privilege escalation attempts, and data exfiltration in system and application logs.\n\n### Web Application Security\nLogGem detects SQL injection, XSS attacks, path traversal, and other web exploits in web server logs.\n\n### Misconfiguration Detection\nIt finds configuration errors that could lead to security vulnerabilities in system and application logs.\n\n### Compliance Monitoring\nI've included features to track access to sensitive resources for audit purposes (HIPAA, PCI-DSS, SOC 2, etc.).\n\n### Real-Time Security Monitoring\nYou can use it for live monitoring of critical log files with instant alerting for security teams.\n\n---\n\n## 📊 Performance\n\nPerformance on modest hardware (8GB RAM, no GPU):\n\n| Metric | Value |\n|--------|-------|\n| **Processing Speed** | ~1,000 log entries/second (batch mode) |\n| **Memory Usage** | ~4GB with Gemma 3 4B (INT8 quantization) |\n| **Model Load Time** | ~10 seconds (first run, cached thereafter) |\n| **Detection Latency** | <100ms per entry (with caching) |\n| **Test Execution** | 5.19 seconds for 142 tests |\n\n---\n\n## 🤝 Contributing\n\nContributions are welcome! Please see [CONTRIBUTING.md](CONTRIBUTING.md) for details.\n\n**Quick Start for Contributors:**\n\n1. Fork the repository\n2. Create a feature branch (`git checkout -b feature/amazing-feature`)\n3. Install dev dependencies (`pip install -e \".[dev]\"`)\n4. Make your changes and add tests\n5. Run tests (`pytest tests/`)\n6. Commit your changes (`git commit -m 'Add amazing feature'`)\n7. Push to the branch (`git push origin feature/amazing-feature`)\n8. Open a Pull Request\n\n**Priority Areas for Contribution:**\n- CLI testing (currently 0% coverage)\n- Parser improvements (Auth, JSON, Nginx parsers)\n- Additional LLM providers (Google Gemini, Azure OpenAI, etc.)\n- Custom detection rules\n- Web UI dashboard\n\n---\n\n## 📝 License\n\nI've released LogGem under the MIT License - see the [LICENSE](LICENSE) file for details.\n\n---\n\n## 🙏 Acknowledgments\n\nI want to thank:\n- **Google** for the Gemma model family\n- **Meta** for Llama models\n- **Anthropic** for Claude models\n- **OpenAI** for GPT models\n- The **open-source security community**\n- All **contributors** and **users**\n\n---\n\n## 💬 Support\n\n- **Issues**: [GitHub Issues](https://github.com/cbritt0n/loggem/issues)\n- **Discussions**: [GitHub Discussions](https://github.com/cbritt0n/loggem/discussions)\n- **Security**: See [SECURITY.md](SECURITY.md) for reporting vulnerabilities\n\n---\n\n## 📈 Project Status\n\n| Metric | Status |\n|--------|--------|\n| **Build** |  |\n| **Tests** | 142 passing, 3 skipped, 0 failing |\n| **Coverage** | 56% overall (90%+ on core modules) |\n| **License** | MIT |\n| **Python** | 3.9+ |\n| **Production Ready** | ✅ Yes |\n\n---\n\n**Made with 💎 by Christian Britton for the security community**\n\n*LogGem - I'll help you find the gems in your logs*\n"}}}} |