1 line
36 KiB
JSON
1 line
36 KiB
JSON
{"repo": "blitzyoce/breachcase", "files": {"scraper/README.md": {"206d000a44a4684547121e5a110e153205cfa376": {"t": 1785746617.0151865, "n": 7783, "h": "e22bc336e7a316d7", "c": "# BreachCase Scraper\n\nPython-based breach news aggregation scraper that fetches articles from 8 RSS feeds, uses DeepSeek AI for classification and data extraction, and stores results in Supabase.\n\n## Features\n\n- **8 RSS Feed Sources**: BleepingComputer, The Hacker News, DataBreachToday, Krebs on Security, HelpNet Security, NCSC UK, Check Point Research, Have I Been Pwned\n- **Two-Stage AI Processing**: Fast classification filter before expensive extraction (~40-60% cost savings)\n- **Full-Database Dedup**: Fuzzy pre-filter across all breaches (no date limit) before AI update detection, so no breach is ever invisible to dedup regardless of age\n- **Update Detection**: Automatically identifies if articles are updates to existing breaches vs. duplicate sources\n- **Local Caching**: Saves raw articles and prevents duplicate URL processing via `cache/processed_ids.txt`\n- **Database Integration**: Writes to Supabase (PostgreSQL)\n- **Comprehensive Logging**: Daily logs with error tracking and classification metrics\n\n## Architecture\n\n```\nRSS Feeds -> feed_parser.py -> cache_manager.py -> ai_processor.py -> db_writer.py -> Supabase\n |\n Stage 1: Classify (is this a breach?)\n Stage 2: Extract (structured data)\n Stage 3: Dedup (fuzzy pre-filter -> AI)\n Stage 4: Write (new breach or update)\n```\n\n### Processing Pipeline\n\n**Stage 1: Classification**\n- Quick yes/no: \"Is this article about a data breach?\"\n- Uses fewer tokens (~100-200 vs 1000+)\n- Filters out ~40-60% of non-breach articles before expensive extraction\n- Configurable confidence threshold (default: 0.6)\n- Set `ENABLE_CLASSIFICATION=False` to skip and process all articles\n\n**Stage 2: Full Extraction**\n- Only runs on articles classified as breaches\n- Extracts detailed structured data: company, severity, records affected, attack vector, CVEs, MITRE techniques, summary, lessons learned\n\n**Stage 3: Dedup (Fuzzy Pre-filter + AI)**\n- Fetches key fields (`id`, `company`, etc.) for **all** breaches in the database at run start\n- For each extracted article, fuzzy-matches the company name against the full database (threshold: 0.6)\n- If no candidates found: article is definitely a new breach - AI call skipped entirely (saves cost)\n- If candidates found: passes only those candidates to DeepSeek for `NEW_BREACH` / `GENUINE_UPDATE` / `DUPLICATE_SOURCE` classification\n- Newly written breaches are added to the stub list during the run, so same-company articles within one run are also caught\n\n**Stage 4: DB Write**\n- New breaches: inserted into `breaches`, `breach_tags`, `sources`\n- Genuine updates: appended to `breach_updates`\n- Duplicate sources: discarded, no DB write\n\n## Setup\n\n### 1. Install Dependencies\n\n```bash\ncd scraper/\npython -m venv venv\n\n# Windows\nvenv\\Scripts\\activate\n\n# Linux/Mac\nsource venv/bin/activate\n\npip install -r requirements.txt\n```\n\n### 2. Configure Environment Variables\n\nCreate a `.env` file in the `scraper/` directory:\n\n```bash\n# Required\nDEEPSEEK_API_KEY=sk-your-deepseek-key\nSUPABASE_URL=https://your-project.supabase.co\nSUPABASE_KEY=your-supabase-anon-key\n\n# Optional (defaults shown)\nENABLE_CLASSIFICATION=True\nCLASSIFICATION_CONFIDENCE_THRESHOLD=0.6\nARTICLE_LOOKBACK_HOURS=48\nFUZZY_MATCH_THRESHOLD=0.85\nFUZZY_CANDIDATE_THRESHOLD=0.6\nMAX_FEED_WORKERS=10\nLOG_LEVEL=INFO\n```\n\n### 3. Run\n\n```bash\npython main.py\n```\n\n## Usage\n\n### Run Manually\n\n```bash\npython main.py\n```\n\n### Audit the Database\n\n```bash\npython audit.py # General audit\npython audit.py --duplicates # Flag potential duplicates\npython audit.py --csv # Export to CSV\n```\n\n### Schedule Daily Runs\n\n#### Linux/Mac (cron)\n\n```bash\ncrontab -e\n```\n\nAdd:\n```\n0 8 * * * cd /path/to/BreachBase/scraper && /path/to/venv/bin/python main.py >> logs/cron.log 2>&1\n```\n\n#### Windows (Task Scheduler)\n\n1. Open Task Scheduler -> Create Basic Task\n2. Trigger: Daily at 8:00 AM\n3. Action: Start a program\n - Program: `C:\\path\\to\\venv\\Scripts\\python.exe`\n - Arguments: `main.py`\n - Start in: `C:\\path\\to\\BreachBase\\scraper`\n\n## File Structure\n\n```\nscraper/\n+-- main.py # Main orchestrator\n+-- config.py # Configuration, prompts, env-overridable settings\n+-- feed_parser.py # RSS feed fetching and filtering\n+-- cache_manager.py # Local caching, processed URL tracking\n+-- ai_processor.py # DeepSeek AI integration\n+-- db_writer.py # Supabase database writing\n+-- audit.py # DB audit and duplicate detection tool\n+-- requirements.txt # Python dependencies\n+-- .env # Your env vars (gitignored)\n+-- cache/\n| +-- raw_YYYY-MM-DD.json # Raw article cache\n| +-- processed_ids.txt # Permanent log of processed article URLs\n| +-- extraction_results_*.json # AI extraction results for debugging\n+-- logs/\n +-- scraper_YYYY-MM-DD.log # Full log (debug level)\n +-- errors_YYYY-MM-DD.log # Errors only\n```\n\n## Modules\n\n### feed_parser.py\nFetches and parses RSS feeds from 8 sources in parallel, filters recent articles (last 48 hours), and deduplicates by URL.\n\n**Key functions:** `fetch_all_feeds()`, `filter_recent_articles()`, `deduplicate_by_url()`\n\n### cache_manager.py\nManages local file cache, tracks processed article URLs (permanent, ever-growing `processed_ids.txt`), and prevents reprocessing the same URL across runs.\n\n**Key functions:** `get_new_articles()`, `cache_articles()`, `save_processed_id()`\n\n### ai_processor.py\nDeepSeek API integration for classifying articles, extracting structured breach data, and detecting updates.\n\n**Key functions:** `classify_article()`, `extract_breach_data()`, `detect_update()`, `call_api()`\n\n### db_writer.py\nSupabase database integration for writing breaches, updates, tags, and sources.\n\n**Key functions:** `write_new_breach()`, `write_breach_update()`, `get_all_breach_stubs()`\n\n## Configuration\n\nAll settings in `config.py` are overridable via environment variables. Key settings:\n\n| Variable | Default | Description |\n|----------|---------|-------------|\n| `ARTICLE_LOOKBACK_HOURS` | 48 | How far back to fetch articles |\n| `FUZZY_CANDIDATE_THRESHOLD` | 0.6 | Min similarity to surface a breach as a dedup candidate |\n| `FUZZY_MATCH_THRESHOLD` | 0.85 | High-confidence match threshold |\n| `CLASSIFICATION_CONFIDENCE_THRESHOLD` | 0.6 | Min confidence to classify as breach |\n| `MAX_FEED_WORKERS` | 10 | Parallel RSS fetch threads |\n| `ENABLE_CLASSIFICATION` | True | Enable Stage 1 classification filter |\n\n## Logging\n\n- `logs/scraper_YYYY-MM-DD.log` - Full debug log\n- `logs/errors_YYYY-MM-DD.log` - Errors only\n- Console output during execution\n\n## Testing Individual Modules\n\nEach module has a `if __name__ == '__main__'` section for standalone testing:\n\n```bash\npython feed_parser.py # Test RSS fetching\npython cache_manager.py # Test caching\npython ai_processor.py # Test AI extraction (requires DEEPSEEK_API_KEY)\npython db_writer.py # Test database connection (requires SUPABASE credentials)\n```\n\n## Troubleshooting\n\n**No articles fetched**\n- Check internet connection and verify RSS feed URLs are still valid\n- Check logs for specific feed errors\n\n**AI extraction fails**\n- Verify `DEEPSEEK_API_KEY` is set correctly\n- Check DeepSeek API quota/limits\n\n**Database errors**\n- Verify `SUPABASE_URL` and `SUPABASE_KEY`\n- Check database schema matches `database/current_db.sql`\n\n**Duplicate breaches appearing**\n- Check `cache/processed_ids.txt` exists and is readable\n- Run `python audit.py --duplicates` to identify existing duplicates\n"}, "cb4d1b6fc21970650c58d180176b27c92ecfbba7": {"t": 1785766314.5787382, "n": 7783, "h": "e22bc336e7a316d7", "c": "# BreachCase Scraper\n\nPython-based breach news aggregation scraper that fetches articles from 8 RSS feeds, uses DeepSeek AI for classification and data extraction, and stores results in Supabase.\n\n## Features\n\n- **8 RSS Feed Sources**: BleepingComputer, The Hacker News, DataBreachToday, Krebs on Security, HelpNet Security, NCSC UK, Check Point Research, Have I Been Pwned\n- **Two-Stage AI Processing**: Fast classification filter before expensive extraction (~40-60% cost savings)\n- **Full-Database Dedup**: Fuzzy pre-filter across all breaches (no date limit) before AI update detection, so no breach is ever invisible to dedup regardless of age\n- **Update Detection**: Automatically identifies if articles are updates to existing breaches vs. duplicate sources\n- **Local Caching**: Saves raw articles and prevents duplicate URL processing via `cache/processed_ids.txt`\n- **Database Integration**: Writes to Supabase (PostgreSQL)\n- **Comprehensive Logging**: Daily logs with error tracking and classification metrics\n\n## Architecture\n\n```\nRSS Feeds -> feed_parser.py -> cache_manager.py -> ai_processor.py -> db_writer.py -> Supabase\n |\n Stage 1: Classify (is this a breach?)\n Stage 2: Extract (structured data)\n Stage 3: Dedup (fuzzy pre-filter -> AI)\n Stage 4: Write (new breach or update)\n```\n\n### Processing Pipeline\n\n**Stage 1: Classification**\n- Quick yes/no: \"Is this article about a data breach?\"\n- Uses fewer tokens (~100-200 vs 1000+)\n- Filters out ~40-60% of non-breach articles before expensive extraction\n- Configurable confidence threshold (default: 0.6)\n- Set `ENABLE_CLASSIFICATION=False` to skip and process all articles\n\n**Stage 2: Full Extraction**\n- Only runs on articles classified as breaches\n- Extracts detailed structured data: company, severity, records affected, attack vector, CVEs, MITRE techniques, summary, lessons learned\n\n**Stage 3: Dedup (Fuzzy Pre-filter + AI)**\n- Fetches key fields (`id`, `company`, etc.) for **all** breaches in the database at run start\n- For each extracted article, fuzzy-matches the company name against the full database (threshold: 0.6)\n- If no candidates found: article is definitely a new breach - AI call skipped entirely (saves cost)\n- If candidates found: passes only those candidates to DeepSeek for `NEW_BREACH` / `GENUINE_UPDATE` / `DUPLICATE_SOURCE` classification\n- Newly written breaches are added to the stub list during the run, so same-company articles within one run are also caught\n\n**Stage 4: DB Write**\n- New breaches: inserted into `breaches`, `breach_tags`, `sources`\n- Genuine updates: appended to `breach_updates`\n- Duplicate sources: discarded, no DB write\n\n## Setup\n\n### 1. Install Dependencies\n\n```bash\ncd scraper/\npython -m venv venv\n\n# Windows\nvenv\\Scripts\\activate\n\n# Linux/Mac\nsource venv/bin/activate\n\npip install -r requirements.txt\n```\n\n### 2. Configure Environment Variables\n\nCreate a `.env` file in the `scraper/` directory:\n\n```bash\n# Required\nDEEPSEEK_API_KEY=sk-your-deepseek-key\nSUPABASE_URL=https://your-project.supabase.co\nSUPABASE_KEY=your-supabase-anon-key\n\n# Optional (defaults shown)\nENABLE_CLASSIFICATION=True\nCLASSIFICATION_CONFIDENCE_THRESHOLD=0.6\nARTICLE_LOOKBACK_HOURS=48\nFUZZY_MATCH_THRESHOLD=0.85\nFUZZY_CANDIDATE_THRESHOLD=0.6\nMAX_FEED_WORKERS=10\nLOG_LEVEL=INFO\n```\n\n### 3. Run\n\n```bash\npython main.py\n```\n\n## Usage\n\n### Run Manually\n\n```bash\npython main.py\n```\n\n### Audit the Database\n\n```bash\npython audit.py # General audit\npython audit.py --duplicates # Flag potential duplicates\npython audit.py --csv # Export to CSV\n```\n\n### Schedule Daily Runs\n\n#### Linux/Mac (cron)\n\n```bash\ncrontab -e\n```\n\nAdd:\n```\n0 8 * * * cd /path/to/BreachBase/scraper && /path/to/venv/bin/python main.py >> logs/cron.log 2>&1\n```\n\n#### Windows (Task Scheduler)\n\n1. Open Task Scheduler -> Create Basic Task\n2. Trigger: Daily at 8:00 AM\n3. Action: Start a program\n - Program: `C:\\path\\to\\venv\\Scripts\\python.exe`\n - Arguments: `main.py`\n - Start in: `C:\\path\\to\\BreachBase\\scraper`\n\n## File Structure\n\n```\nscraper/\n+-- main.py # Main orchestrator\n+-- config.py # Configuration, prompts, env-overridable settings\n+-- feed_parser.py # RSS feed fetching and filtering\n+-- cache_manager.py # Local caching, processed URL tracking\n+-- ai_processor.py # DeepSeek AI integration\n+-- db_writer.py # Supabase database writing\n+-- audit.py # DB audit and duplicate detection tool\n+-- requirements.txt # Python dependencies\n+-- .env # Your env vars (gitignored)\n+-- cache/\n| +-- raw_YYYY-MM-DD.json # Raw article cache\n| +-- processed_ids.txt # Permanent log of processed article URLs\n| +-- extraction_results_*.json # AI extraction results for debugging\n+-- logs/\n +-- scraper_YYYY-MM-DD.log # Full log (debug level)\n +-- errors_YYYY-MM-DD.log # Errors only\n```\n\n## Modules\n\n### feed_parser.py\nFetches and parses RSS feeds from 8 sources in parallel, filters recent articles (last 48 hours), and deduplicates by URL.\n\n**Key functions:** `fetch_all_feeds()`, `filter_recent_articles()`, `deduplicate_by_url()`\n\n### cache_manager.py\nManages local file cache, tracks processed article URLs (permanent, ever-growing `processed_ids.txt`), and prevents reprocessing the same URL across runs.\n\n**Key functions:** `get_new_articles()`, `cache_articles()`, `save_processed_id()`\n\n### ai_processor.py\nDeepSeek API integration for classifying articles, extracting structured breach data, and detecting updates.\n\n**Key functions:** `classify_article()`, `extract_breach_data()`, `detect_update()`, `call_api()`\n\n### db_writer.py\nSupabase database integration for writing breaches, updates, tags, and sources.\n\n**Key functions:** `write_new_breach()`, `write_breach_update()`, `get_all_breach_stubs()`\n\n## Configuration\n\nAll settings in `config.py` are overridable via environment variables. Key settings:\n\n| Variable | Default | Description |\n|----------|---------|-------------|\n| `ARTICLE_LOOKBACK_HOURS` | 48 | How far back to fetch articles |\n| `FUZZY_CANDIDATE_THRESHOLD` | 0.6 | Min similarity to surface a breach as a dedup candidate |\n| `FUZZY_MATCH_THRESHOLD` | 0.85 | High-confidence match threshold |\n| `CLASSIFICATION_CONFIDENCE_THRESHOLD` | 0.6 | Min confidence to classify as breach |\n| `MAX_FEED_WORKERS` | 10 | Parallel RSS fetch threads |\n| `ENABLE_CLASSIFICATION` | True | Enable Stage 1 classification filter |\n\n## Logging\n\n- `logs/scraper_YYYY-MM-DD.log` - Full debug log\n- `logs/errors_YYYY-MM-DD.log` - Errors only\n- Console output during execution\n\n## Testing Individual Modules\n\nEach module has a `if __name__ == '__main__'` section for standalone testing:\n\n```bash\npython feed_parser.py # Test RSS fetching\npython cache_manager.py # Test caching\npython ai_processor.py # Test AI extraction (requires DEEPSEEK_API_KEY)\npython db_writer.py # Test database connection (requires SUPABASE credentials)\n```\n\n## Troubleshooting\n\n**No articles fetched**\n- Check internet connection and verify RSS feed URLs are still valid\n- Check logs for specific feed errors\n\n**AI extraction fails**\n- Verify `DEEPSEEK_API_KEY` is set correctly\n- Check DeepSeek API quota/limits\n\n**Database errors**\n- Verify `SUPABASE_URL` and `SUPABASE_KEY`\n- Check database schema matches `database/current_db.sql`\n\n**Duplicate breaches appearing**\n- Check `cache/processed_ids.txt` exists and is readable\n- Run `python audit.py --duplicates` to identify existing duplicates\n"}, "23aed27c9ee2066f91c755f7db3de8a12af91ba8": {"t": 1785766316.2536905, "n": 6544, "h": "239bd47f60387756", "c": "# BreachCase Scraper\n\nPython-based breach news aggregation scraper that fetches articles from 10 RSS feeds, uses DeepSeek AI for data extraction, and stores results in Supabase.\n\n## Features\n\n- **10 RSS Feed Sources**: BleepingComputer, The Hacker News, DataBreachToday, Dark Reading, Krebs, HelpNet Security, CERT.be, NCSC UK, Check Point Research, Have I Been Pwned\n- **Two-Stage AI Processing**: Fast classification filter before expensive extraction (40-60% cost savings)\n- **AI-Powered Extraction**: Uses DeepSeek API to extract structured breach data\n- **Update Detection**: Automatically identifies if articles are updates to existing breaches\n- **Local Caching**: Saves raw articles and prevents duplicate processing\n- **Database Integration**: Writes to Supabase (PostgreSQL) with proper schema\n- **Comprehensive Logging**: Daily logs with error tracking and classification metrics\n\n## Architecture\n\n```\nRSS Feeds -> feed_parser.py -> cache_manager.py -> ai_processor.py -> db_writer.py -> Supabase\n |\n Stage 1: Classification (Fast & Cheap)\n Stage 2: Extraction (Detailed & Expensive)\n```\n\n### Two-Stage AI Processing\n\nThe scraper uses a two-stage AI approach to optimize cost and performance:\n\n**Stage 1: Classification**\n- Quick yes/no: \"Is this article about a data breach?\"\n- Uses fewer tokens (~100-200 vs 1000+)\n- Filters out 40-60% of non-breach articles\n- Configurable confidence threshold (default: 0.7)\n\n**Stage 2: Full Extraction**\n- Only runs on articles classified as breaches\n- Extracts detailed structured data\n- Performs update detection\n- Writes to database\n\n**Cost Savings**: Classification is ~90% cheaper than extraction, resulting in 40-60% overall cost reduction while maintaining accuracy.\n\n**Configuration**: Set `ENABLE_CLASSIFICATION=False` in `.env` to disable and process all articles.\n\n## Setup\n\n### 1. Install Dependencies\n\n```bash\ncd scraper/\npython -m venv venv\n\n# Windows\nvenv\\Scripts\\activate\n\n# Linux/Mac\nsource venv/bin/activate\n\npip install -r requirements.txt\n```\n\n### 2. Configure Environment Variables\n\n```bash\ncp .env.example .env\n```\n\nEdit `.env` and add your API keys:\n\n```bash\n# Required API Keys\nDEEPSEEK_API_KEY=sk-your-deepseek-key\nSUPABASE_URL=https://your-project.supabase.co\nSUPABASE_KEY=your-supabase-anon-key\n\n# Optional Configuration\nENABLE_CLASSIFICATION=True # Enable two-stage AI (recommended)\nCLASSIFICATION_CONFIDENCE_THRESHOLD=0.7 # Minimum confidence for breach classification\nARTICLE_LOOKBACK_HOURS=48 # How far back to fetch articles\n```\n\n### 3. Test the Scraper\n\n```bash\npython main.py\n```\n\n## Usage\n\n### Run Manually\n\n```bash\npython main.py\n```\n\n### Schedule Daily Runs\n\n#### Linux/Mac (cron)\n\n```bash\ncrontab -e\n```\n\nAdd line:\n```\n0 8 * * * cd /path/to/BreachBase/scraper && /path/to/venv/bin/python main.py >> logs/cron.log 2>&1\n```\n\n#### Windows (Task Scheduler)\n\n1. Open Task Scheduler\n2. Create Basic Task\n3. Trigger: Daily at 8:00 AM\n4. Action: Start a program\n - Program: `C:\\path\\to\\venv\\Scripts\\python.exe`\n - Arguments: `main.py`\n - Start in: `C:\\path\\to\\BreachBase\\scraper`\n\n## File Structure\n\n```\nscraper/\n├── main.py # Main orchestrator\n├── config.py # Configuration & prompts\n├── feed_parser.py # RSS feed fetching\n├── cache_manager.py # Local caching\n├── ai_processor.py # DeepSeek AI integration\n├── db_writer.py # Supabase database writing\n├── requirements.txt # Python dependencies\n├── .env.example # Example environment variables\n├── .env # Your actual env vars (gitignored)\n├── cache/ # Article cache\n│ ├── raw_2026-02-04.json\n│ └── processed_ids.txt\n└── logs/ # Log files\n ├── scraper_2026-02-04.log\n └── errors_2026-02-04.log\n```\n\n## Modules\n\n### feed_parser.py\nFetches and parses RSS feeds from 10 sources in parallel, filters recent articles (last 48 hours), and deduplicates by URL.\n\n**Key Functions:**\n- `fetch_all_feeds()` - Fetch from all sources\n- `filter_recent_articles()` - Filter by date\n- `deduplicate_by_url()` - Remove duplicates\n\n### cache_manager.py\nManages local file cache, tracks processed article URLs, and prevents duplicate processing.\n\n**Key Functions:**\n- `get_new_articles()` - Filter already-processed\n- `cache_articles()` - Save raw articles\n- `save_processed_id()` - Mark as processed\n\n### ai_processor.py\nDeepSeek API integration for extracting structured breach data and detecting updates.\n\n**Key Functions:**\n- `extract_breach_data()` - Extract structured data\n- `detect_update()` - Identify if article is update\n- `call_api()` - API wrapper with retry logic\n\n### db_writer.py\nSupabase database integration for writing breaches, updates, tags, and sources.\n\n**Key Functions:**\n- `write_new_breach()` - Insert new breach\n- `write_breach_update()` - Insert update\n- `get_existing_breaches()` - Fetch for matching\n\n## Configuration\n\nEdit `config.py` to modify:\n\n- RSS feed sources\n- AI prompts for extraction\n- Retry limits and timeouts\n- Cache directory paths\n\n## Logging\n\nLogs are written to:\n- `logs/scraper_YYYY-MM-DD.log` - Full log\n- `logs/errors_YYYY-MM-DD.log` - Errors only\n- Console output during execution\n\n## Testing Individual Modules\n\nEach module has a `if __name__ == '__main__'` section for standalone testing:\n\n```bash\n# Test RSS fetching\npython feed_parser.py\n\n# Test caching\npython cache_manager.py\n\n# Test AI extraction (requires DEEPSEEK_API_KEY)\npython ai_processor.py\n\n# Test database connection (requires SUPABASE credentials)\npython db_writer.py\n```\n\n## Troubleshooting\n\n### No articles fetched\n- Check internet connection\n- Verify RSS feed URLs are still valid\n- Check logs for specific feed errors\n\n### AI extraction fails\n- Verify DEEPSEEK_API_KEY is set correctly\n- Check DeepSeek API quota/limits\n- Review logs for API errors\n\n### Database errors\n- Verify SUPABASE_URL and SUPABASE_KEY\n- Check database schema matches enhanced_schema.sql\n- Ensure tables exist in Supabase\n\n### Duplicate processing\n- Check `cache/processed_ids.txt` exists\n- Verify file permissions\n- Check for concurrent runs\n\n\n## Future Enhancements\n\n- [ ] Full article text fetching (beyond RSS summary)\n- [ ] Additional non-English sources\n- [ ] Webhook notifications for critical breaches\n- [ ] Human review queue for low-confidence extractions\n- [ ] Duplicate breach merging\n- [ ] Source reliability scoring\n\n"}, "7ba2650016a7676adecadb126dd06753e04220f9": {"t": 1785766317.7472944, "n": 6545, "h": "d13b17424e1244f0", "c": "# BreachWatch Scraper\n\nPython-based breach news aggregation scraper that fetches articles from 10 RSS feeds, uses DeepSeek AI for data extraction, and stores results in Supabase.\n\n## Features\n\n- **10 RSS Feed Sources**: BleepingComputer, The Hacker News, DataBreachToday, Dark Reading, Krebs, HelpNet Security, CERT.be, NCSC UK, Check Point Research, Have I Been Pwned\n- **Two-Stage AI Processing**: Fast classification filter before expensive extraction (40-60% cost savings)\n- **AI-Powered Extraction**: Uses DeepSeek API to extract structured breach data\n- **Update Detection**: Automatically identifies if articles are updates to existing breaches\n- **Local Caching**: Saves raw articles and prevents duplicate processing\n- **Database Integration**: Writes to Supabase (PostgreSQL) with proper schema\n- **Comprehensive Logging**: Daily logs with error tracking and classification metrics\n\n## Architecture\n\n```\nRSS Feeds -> feed_parser.py -> cache_manager.py -> ai_processor.py -> db_writer.py -> Supabase\n |\n Stage 1: Classification (Fast & Cheap)\n Stage 2: Extraction (Detailed & Expensive)\n```\n\n### Two-Stage AI Processing\n\nThe scraper uses a two-stage AI approach to optimize cost and performance:\n\n**Stage 1: Classification**\n- Quick yes/no: \"Is this article about a data breach?\"\n- Uses fewer tokens (~100-200 vs 1000+)\n- Filters out 40-60% of non-breach articles\n- Configurable confidence threshold (default: 0.7)\n\n**Stage 2: Full Extraction**\n- Only runs on articles classified as breaches\n- Extracts detailed structured data\n- Performs update detection\n- Writes to database\n\n**Cost Savings**: Classification is ~90% cheaper than extraction, resulting in 40-60% overall cost reduction while maintaining accuracy.\n\n**Configuration**: Set `ENABLE_CLASSIFICATION=False` in `.env` to disable and process all articles.\n\n## Setup\n\n### 1. Install Dependencies\n\n```bash\ncd scraper/\npython -m venv venv\n\n# Windows\nvenv\\Scripts\\activate\n\n# Linux/Mac\nsource venv/bin/activate\n\npip install -r requirements.txt\n```\n\n### 2. Configure Environment Variables\n\n```bash\ncp .env.example .env\n```\n\nEdit `.env` and add your API keys:\n\n```bash\n# Required API Keys\nDEEPSEEK_API_KEY=sk-your-deepseek-key\nSUPABASE_URL=https://your-project.supabase.co\nSUPABASE_KEY=your-supabase-anon-key\n\n# Optional Configuration\nENABLE_CLASSIFICATION=True # Enable two-stage AI (recommended)\nCLASSIFICATION_CONFIDENCE_THRESHOLD=0.7 # Minimum confidence for breach classification\nARTICLE_LOOKBACK_HOURS=48 # How far back to fetch articles\n```\n\n### 3. Test the Scraper\n\n```bash\npython main.py\n```\n\n## Usage\n\n### Run Manually\n\n```bash\npython main.py\n```\n\n### Schedule Daily Runs\n\n#### Linux/Mac (cron)\n\n```bash\ncrontab -e\n```\n\nAdd line:\n```\n0 8 * * * cd /path/to/BreachBase/scraper && /path/to/venv/bin/python main.py >> logs/cron.log 2>&1\n```\n\n#### Windows (Task Scheduler)\n\n1. Open Task Scheduler\n2. Create Basic Task\n3. Trigger: Daily at 8:00 AM\n4. Action: Start a program\n - Program: `C:\\path\\to\\venv\\Scripts\\python.exe`\n - Arguments: `main.py`\n - Start in: `C:\\path\\to\\BreachBase\\scraper`\n\n## File Structure\n\n```\nscraper/\n├── main.py # Main orchestrator\n├── config.py # Configuration & prompts\n├── feed_parser.py # RSS feed fetching\n├── cache_manager.py # Local caching\n├── ai_processor.py # DeepSeek AI integration\n├── db_writer.py # Supabase database writing\n├── requirements.txt # Python dependencies\n├── .env.example # Example environment variables\n├── .env # Your actual env vars (gitignored)\n├── cache/ # Article cache\n│ ├── raw_2026-02-04.json\n│ └── processed_ids.txt\n└── logs/ # Log files\n ├── scraper_2026-02-04.log\n └── errors_2026-02-04.log\n```\n\n## Modules\n\n### feed_parser.py\nFetches and parses RSS feeds from 10 sources in parallel, filters recent articles (last 48 hours), and deduplicates by URL.\n\n**Key Functions:**\n- `fetch_all_feeds()` - Fetch from all sources\n- `filter_recent_articles()` - Filter by date\n- `deduplicate_by_url()` - Remove duplicates\n\n### cache_manager.py\nManages local file cache, tracks processed article URLs, and prevents duplicate processing.\n\n**Key Functions:**\n- `get_new_articles()` - Filter already-processed\n- `cache_articles()` - Save raw articles\n- `save_processed_id()` - Mark as processed\n\n### ai_processor.py\nDeepSeek API integration for extracting structured breach data and detecting updates.\n\n**Key Functions:**\n- `extract_breach_data()` - Extract structured data\n- `detect_update()` - Identify if article is update\n- `call_api()` - API wrapper with retry logic\n\n### db_writer.py\nSupabase database integration for writing breaches, updates, tags, and sources.\n\n**Key Functions:**\n- `write_new_breach()` - Insert new breach\n- `write_breach_update()` - Insert update\n- `get_existing_breaches()` - Fetch for matching\n\n## Configuration\n\nEdit `config.py` to modify:\n\n- RSS feed sources\n- AI prompts for extraction\n- Retry limits and timeouts\n- Cache directory paths\n\n## Logging\n\nLogs are written to:\n- `logs/scraper_YYYY-MM-DD.log` - Full log\n- `logs/errors_YYYY-MM-DD.log` - Errors only\n- Console output during execution\n\n## Testing Individual Modules\n\nEach module has a `if __name__ == '__main__'` section for standalone testing:\n\n```bash\n# Test RSS fetching\npython feed_parser.py\n\n# Test caching\npython cache_manager.py\n\n# Test AI extraction (requires DEEPSEEK_API_KEY)\npython ai_processor.py\n\n# Test database connection (requires SUPABASE credentials)\npython db_writer.py\n```\n\n## Troubleshooting\n\n### No articles fetched\n- Check internet connection\n- Verify RSS feed URLs are still valid\n- Check logs for specific feed errors\n\n### AI extraction fails\n- Verify DEEPSEEK_API_KEY is set correctly\n- Check DeepSeek API quota/limits\n- Review logs for API errors\n\n### Database errors\n- Verify SUPABASE_URL and SUPABASE_KEY\n- Check database schema matches enhanced_schema.sql\n- Ensure tables exist in Supabase\n\n### Duplicate processing\n- Check `cache/processed_ids.txt` exists\n- Verify file permissions\n- Check for concurrent runs\n\n\n## Future Enhancements\n\n- [ ] Full article text fetching (beyond RSS summary)\n- [ ] Additional non-English sources\n- [ ] Webhook notifications for critical breaches\n- [ ] Human review queue for low-confidence extractions\n- [ ] Duplicate breach merging\n- [ ] Source reliability scoring\n\n"}, "d9e18e9a8de152001c71d249ab061876e881807a": {"t": 1785766319.3750083, "n": 5669, "h": "b5a03f0469411e4e", "c": "# BreachWatch Scraper\n\nPython-based breach news aggregation scraper that fetches articles from 10 RSS feeds, uses DeepSeek AI for data extraction, and stores results in Supabase.\n\n## Features\n\n- **10 RSS Feed Sources**: BleepingComputer, The Hacker News, DataBreachToday, Dark Reading, Krebs, HelpNet Security, CERT.be, NCSC UK, Check Point Research, Have I Been Pwned\n- **AI-Powered Extraction**: Uses DeepSeek API to extract structured breach data\n- **Update Detection**: Automatically identifies if articles are updates to existing breaches\n- **Local Caching**: Saves raw articles and prevents duplicate processing\n- **Database Integration**: Writes to Supabase (PostgreSQL) with proper schema\n- **Comprehensive Logging**: Daily logs with error tracking\n\n## Architecture\n\n```\nRSS Feeds → feed_parser.py → cache_manager.py → ai_processor.py → db_writer.py → Supabase\n```\n\n## Setup\n\n### 1. Install Dependencies\n\n```bash\ncd scraper/\npython -m venv venv\n\n# Windows\nvenv\\Scripts\\activate\n\n# Linux/Mac\nsource venv/bin/activate\n\npip install -r requirements.txt\n```\n\n### 2. Configure Environment Variables\n\n```bash\ncp .env.example .env\n```\n\nEdit `.env` and add your API keys:\n\n```bash\nDEEPSEEK_API_KEY=sk-your-deepseek-key\nSUPABASE_URL=https://your-project.supabase.co\nSUPABASE_KEY=your-supabase-anon-key\n```\n\n### 3. Test the Scraper\n\n```bash\npython main.py\n```\n\n## Usage\n\n### Run Manually\n\n```bash\npython main.py\n```\n\n### Schedule Daily Runs\n\n#### Linux/Mac (cron)\n\n```bash\ncrontab -e\n```\n\nAdd line:\n```\n0 8 * * * cd /path/to/BreachBase/scraper && /path/to/venv/bin/python main.py >> logs/cron.log 2>&1\n```\n\n#### Windows (Task Scheduler)\n\n1. Open Task Scheduler\n2. Create Basic Task\n3. Trigger: Daily at 8:00 AM\n4. Action: Start a program\n - Program: `C:\\path\\to\\venv\\Scripts\\python.exe`\n - Arguments: `main.py`\n - Start in: `C:\\path\\to\\BreachBase\\scraper`\n\n## File Structure\n\n```\nscraper/\n├── main.py # Main orchestrator\n├── config.py # Configuration & prompts\n├── feed_parser.py # RSS feed fetching\n├── cache_manager.py # Local caching\n├── ai_processor.py # DeepSeek AI integration\n├── db_writer.py # Supabase database writing\n├── requirements.txt # Python dependencies\n├── .env.example # Example environment variables\n├── .env # Your actual env vars (gitignored)\n├── cache/ # Article cache\n│ ├── raw_2026-02-04.json\n│ └── processed_ids.txt\n└── logs/ # Log files\n ├── scraper_2026-02-04.log\n └── errors_2026-02-04.log\n```\n\n## Modules\n\n### feed_parser.py\nFetches and parses RSS feeds from 10 sources in parallel, filters recent articles (last 48 hours), and deduplicates by URL.\n\n**Key Functions:**\n- `fetch_all_feeds()` - Fetch from all sources\n- `filter_recent_articles()` - Filter by date\n- `deduplicate_by_url()` - Remove duplicates\n\n### cache_manager.py\nManages local file cache, tracks processed article URLs, and prevents duplicate processing.\n\n**Key Functions:**\n- `get_new_articles()` - Filter already-processed\n- `cache_articles()` - Save raw articles\n- `save_processed_id()` - Mark as processed\n\n### ai_processor.py\nDeepSeek API integration for extracting structured breach data and detecting updates.\n\n**Key Functions:**\n- `extract_breach_data()` - Extract structured data\n- `detect_update()` - Identify if article is update\n- `call_api()` - API wrapper with retry logic\n\n### db_writer.py\nSupabase database integration for writing breaches, updates, tags, and sources.\n\n**Key Functions:**\n- `write_new_breach()` - Insert new breach\n- `write_breach_update()` - Insert update\n- `get_existing_breaches()` - Fetch for matching\n\n## Configuration\n\nEdit `config.py` to modify:\n\n- RSS feed sources\n- AI prompts for extraction\n- Retry limits and timeouts\n- Cache directory paths\n\n## Logging\n\nLogs are written to:\n- `logs/scraper_YYYY-MM-DD.log` - Full log\n- `logs/errors_YYYY-MM-DD.log` - Errors only\n- Console output during execution\n\n## Testing Individual Modules\n\nEach module has a `if __name__ == '__main__'` section for standalone testing:\n\n```bash\n# Test RSS fetching\npython feed_parser.py\n\n# Test caching\npython cache_manager.py\n\n# Test AI extraction (requires DEEPSEEK_API_KEY)\npython ai_processor.py\n\n# Test database connection (requires SUPABASE credentials)\npython db_writer.py\n```\n\n## Troubleshooting\n\n### No articles fetched\n- Check internet connection\n- Verify RSS feed URLs are still valid\n- Check logs for specific feed errors\n\n### AI extraction fails\n- Verify DEEPSEEK_API_KEY is set correctly\n- Check DeepSeek API quota/limits\n- Review logs for API errors\n\n### Database errors\n- Verify SUPABASE_URL and SUPABASE_KEY\n- Check database schema matches enhanced_schema.sql\n- Ensure tables exist in Supabase\n\n### Duplicate processing\n- Check `cache/processed_ids.txt` exists\n- Verify file permissions\n- Check for concurrent runs\n\n## Performance\n\nExpected daily metrics:\n- **Articles fetched**: 20-50\n- **New articles**: 10-30\n- **Processing time**: 5-10 minutes\n- **API calls**: ~50-100 DeepSeek calls\n- **Database writes**: 5-15 breaches/updates\n\n## Cost Estimation\n\n**DeepSeek API** (as of Feb 2026):\n- ~100 API calls/day\n- ~500 tokens per call\n- Estimate: $0.10-0.50/day\n\n**Supabase**:\n- Free tier: 500MB database, 50K rows\n- Should be sufficient for first 6-12 months\n\n## Future Enhancements\n\n- [ ] Full article text fetching (beyond RSS summary)\n- [ ] Additional non-English sources\n- [ ] Webhook notifications for critical breaches\n- [ ] Human review queue for low-confidence extractions\n- [ ] Duplicate breach merging\n- [ ] Source reliability scoring\n\n## License\n\nPart of the BreachWatch project.\n"}}}} |