1 line
22 KiB
JSON
1 line
22 KiB
JSON
{"repo": "alukashenkov/sbom-llm-eval", "files": {"README.md": {"3becb9e24219e60f41ee1764e9adacefe21e132a": {"t": 1785747243.2518535, "n": 7894, "h": "1471d389fc51469a", "c": "# Vulners SBOM Audit Summary — LLM Prompt & Model Evaluation\n\nLLM-powered summarization of [Vulners SBOM Audit](https://vulners.com/sbom-audit) results into CRA-aligned vulnerability reports under **Regulation (EU) 2024/2847** (Cyber Resilience Act). Audit results can be obtained via [UI](https://vulners.com/sbom-audit) or [API](https://docs.vulners.com/docs/api/audit/#sbom-audit).\n\n## Winner: Gemini 3 Flash + v7 prompt *(speed-weighted)*\n\n| | |\n| --- | --- |\n| **Model** | `google/gemini-3-flash-preview` via OpenRouter |\n| **Prompt** | `prompts/v7.txt` |\n| **Quality score** | 7.77 / 10 (judged by Claude Sonnet 4) |\n| **Speed** | **~11s per file** (2× faster than GPT-4.1-mini, 4× faster than DeepSeek) |\n| **Cost** | ~$0.032 per file |\n| **Composite (50% quality + 50% speed)** | **7.69** |\n\nFor highest output quality (CRA alignment, accuracy, completeness) without hard latency constraints, use **GPT-4.1-mini** (composite 6.79, quality 8.52, ~23s/file).\n\n## v7 Evaluation Summary — Speed-Weighted Composite (50/50)\n\n| Rank | Model | Quality | Avg time | Speed score | **Composite** |\n|---|---|---|---|---|---|\n| 🥇 | **gemini-3-flash** | 7.77 | **11.4s** | 7.61 | **7.69** |\n| 🥈 | gpt-4.1-mini | **8.52** | 23.4s | 5.07 | 6.79 |\n| 🥉 | deepseek-v3 | 8.08 | 47.5s | 0.00 | 4.04 |\n\n> DeepSeek disqualifies itself under ≥50% speed weight — tail latency of 84.7s on large files (openclaw) makes it unsuitable for production with SLA requirements, despite being the cheapest ($0.085 total) and most concise.\n\n### Per-Criterion Quality Scores (v7)\n\n| Criterion | Weight | gemini-3-flash | gpt-4.1-mini | deepseek-v3 |\n|---|---|---|---|---|\n| CRA Alignment | 30% | 8.33 | **9.00** | 8.17 |\n| Accuracy | 25% | 7.17 | **8.50** | 8.00 |\n| Completeness | 20% | 8.17 | **9.00** | 7.33 |\n| Conciseness | 15% | 6.50 | 7.00 | **9.17** |\n| Actionability | 10% | 8.33 | **8.83** | 7.80 |\n\n### Per-File Response Times (v7)\n\n| File | gemini-3-flash | gpt-4.1-mini | deepseek-v3 |\n|---|---|---|---|\n| juice-shop | 13.6s | 26.9s | 31.1s |\n| la-vulners-mcp | 9.2s | 19.1s | 34.9s |\n| openclaw | 14.0s | 23.6s | ⚠️ 84.7s |\n| grbrsm_ui | 7.6s | 23.2s | 20.2s |\n| gurobi-engine | 12.6s | 29.9s | 38.0s |\n| vulners-mcp | 11.2s | 17.8s | ⚠️ 76.2s |\n\n## Ground Truth: Dual-Source Comparison\n\nEvaluation uses cross-referenced findings from **Vulners** (70% weight) and **Grype** (30% weight). The `compare_sources.py` script normalises vulnerability IDs (including non-CVE advisories like GHSA), computes per-CVE risk scores, and generates structured comparison data for the judge.\n\nJudge receives a `source_comparison` with CVEs categorised as:\n\n- **In both sources** — confirmed findings, penalises summaries that miss them\n- **Vulners-only** — legitimate (primary source), no penalty for including\n- **Grype-only** — legitimate (cross-check), not mandatory to include\n- **In neither** — true hallucinations, penalised heavily\n\n## CRA Alignment\n\nSummaries are produced under the three-tier vulnerability taxonomy of Regulation (EU) 2024/2847:\n\n| Tier | CRA Article | Definition | Action |\n|---|---|---|---|\n| `ACTIVELY_EXPLOITED` | Art. 3(42) | Reliable evidence of in-the-wild malicious exploitation | Art. 14 Track 1: 24h → 72h → 14-day report to ENISA/CSIRT |\n| `EXPLOITABLE` | Art. 3(41) | Potential to be effectively used under practical operational conditions | Pre-market gate (Annex I §2); remediate without delay (Annex I Part II §2) |\n| `VULNERABILITY` | Art. 3(40) | Weakness that can be exploited | Track, document, manage throughout lifecycle |\n\n## Key Optimisations\n\n1. **CRA-accurate article references** (v7) — corrected from Article 10/11 to **Art. 3, 13, 14, Annex I** per final Regulation (EU) 2024/2847 text.\n2. **Three-tier taxonomy pre-classification** (v7) — `preprocess.py` computes `craTier` per CVE before sending to the LLM, eliminating exploitability inference errors.\n3. **Art. 14 dual-track reporting** (v7) — Track 1 (actively exploited: 24h/72h/14d) and Track 2 (severe incident: 24h/72h/1mo) are surfaced as separate pre-computed `craMandatoryTriggers` / `craTrack2Candidates` fields.\n4. **Rich CVSS fields** (v7) — `cvssVector`, `cvssVersion`, `cvssSource` extracted; model can reason about network reachability and attack complexity for Art. 3(41) exploitability assessment.\n5. **EPSS percentile + staleness** (v7) — `epssPercentile` and `epssDate` extracted; stale scores (>90 days) flagged so the model can express appropriate uncertainty.\n6. **Fix hints from Vulners AI** (v7) — `enchantments.short_description` (\"upgrade to X.Y.Z\") and `aiDescription` extracted as `fixHint` and `description`; drives concrete remediation actions in Annex I Part II §2.\n7. **Age risk signal** (v7) — `daysPublic` computed per CVE; top unpatched CRITICAL/HIGH CVEs surfaced as `ageRisk` — long-unpatched issues are flagged as potential \"without delay\" violations (Annex I Part II §2).\n8. **EPSS merge bug fixed** (v7) — EPSS score now correctly keeps the *highest* value when a CVE appears across multiple advisories (previously took first seen).\n9. **Pre-computed CVE analytics** (v5) — Python deduplicates CVEs before sending to LLM. Eliminated counting errors across all models.\n10. **Dual-source judge** (v5) — Cross-referenced Vulners + Grype findings with per-CVE risk scores give the judge structured, verifiable ground truth.\n\n## Project Structure\n\n```\nSSA/\n├── vulners_results/ # Vulners SBOM audit JSON files\n├── grype_results/ # Grype vulnerability scan outputs\n├── comparisons/ # Cross-source comparison reports per SBOM\n│ └── {sbom-name}/\n│ ├── comparison.md # Human-readable report\n│ └── comparison.json # Structured data for judge\n├── prompts/ # Prompt versions (v1–v7)\n│ └── v7.txt # Current best prompt (CRA-aligned)\n├── results/ # Versioned evaluation results\n│ └── v7/ # Latest: rankings, judge scores, run metrics\n├── compare_sources.py # Grype vs Vulners comparison script\n├── preprocess.py # Data preprocessing — CRA-tier classification,\n│ # CVSS/EPSS/fixHint/ageRisk extraction\n├── summarize_sbom.py # Evaluation workbench (multi-model + judge)\n├── analyze.py # Single-file analysis script\n├── requirements.txt # Python dependencies\n└── .env # OPENROUTER_API_KEY (not committed)\n```\n\n## Setup\n\n```bash\npip3 install -r requirements.txt\necho 'OPENROUTER_API_KEY=sk-or-v1-your-key' > .env\n```\n\n## Scripts\n\n### `analyze.py` — Single-file analysis\n\nPreprocesses one Vulners SBOM audit JSON, calls the LLM, and prints a CRA-aligned summary to the console.\n\n```bash\n# Default: Gemini 3 Flash + v7 prompt\npython3 analyze.py vulners_results/package-analysis-report-juice-shop.json\n\n# Custom model and prompt\npython3 analyze.py vulners_results/package-analysis-report-juice-shop.json \\\n --model gpt-4.1-mini --prompt prompts/v7.txt\n```\n\nAvailable models: `gemini-3-flash`, `gemini-2.5-flash`, `gpt-4.1-mini`, `gpt-5-nano`, `deepseek-v3`\n\n### `compare_sources.py` — Grype vs Vulners comparison\n\nCross-references Grype and Vulners findings per SBOM. Normalises vulnerability IDs (CVE + GHSA), computes risk scores, and generates comparison reports used by the judge.\n\n```bash\npython3 compare_sources.py\n# Custom paths:\npython3 compare_sources.py --vulners vulners_results/ --grype grype_results/ --output comparisons/\n```\n\n### `summarize_sbom.py` — Evaluation workbench\n\nRuns all candidate models on all files, then scores and ranks outputs using Claude Sonnet 4 as judge with cross-source ground truth.\n\n```bash\n# Full evaluation (run compare_sources.py first)\npython3 summarize_sbom.py --evaluate --prompt prompts/v7.txt --data vulners_results/\n\n# Single model, all files (no judging)\npython3 summarize_sbom.py --summarize --model gemini-3-flash --prompt prompts/v7.txt --data vulners_results/\n```\n"}, "a73db2139a22554d81c36db6fbd0b52eaf00f6cb": {"t": 1785771670.0626366, "n": 6162, "h": "366c67ca73c2ba00", "c": "# Vulners SBOM Audit Summary — LLM Prompt & Model Evaluation\n\nLLM-powered summarization of [Vulners SBOM Audit](https://vulners.com/sbom-audit) results into CRA-aligned vulnerability reports. Audit results can be obtained via [UI](https://vulners.com/sbom-audit) or [API](https://docs.vulners.com/docs/api/audit/#sbom-audit).\n\n## Winner: DeepSeek V3 + v6 prompt\n\n| | |\n| --- | --- |\n| **Model** | `deepseek/deepseek-chat-v3-0324` via OpenRouter |\n| **Prompt** | `prompts/v6.txt` |\n| **Score** | 8.01 / 10 (judged by Claude Sonnet 4) |\n| **Speed** | ~44s per file |\n| **Cost** | ~$0.012 per file |\n\nBest balance of quality and cost for CRA-aligned vulnerability summaries from Vulners SBOM audit JSON. Consistently wins on conciseness and accuracy, with strong CRA article citations.\n\nFastest option: **Gemini 3 Flash** (7.66 avg, ~8s/file).\nMost consistent: **GPT-4.1-mini** (7.81 avg, ~14s/file).\n\n## Ground Truth: Dual-Source Comparison\n\nEvaluation uses cross-referenced findings from **Vulners** (70% weight) and **Grype** (30% weight). The `compare_sources.py` script normalizes vulnerability IDs (including non-CVE advisories like GHSA), computes per-CVE risk scores, and generates structured comparison data for the judge.\n\nJudge receives a `source_comparison` with CVEs categorized as:\n\n- **In both sources** — confirmed findings, penalises summaries that miss them\n- **Vulners-only** — legitimate (primary source), no penalty for including\n- **Grype-only** — legitimate (cross-check), not mandatory to include\n- **In neither** — true hallucinations, penalised heavily\n\n## Evaluation Summary (9 models tested, 6 prompt iterations, 6 test files)\n\n| Model | Score (v6) | Speed | Status |\n|-------|-----------|-------|--------|\n| **DeepSeek V3** | **8.01** | 44s | ✅ **Winner** — best quality-to-cost ratio, wins 4/6 files |\n| GPT-4.1-mini | 7.81 | 14s | Most consistent — narrowest score range, wins openclaw |\n| Gemini 3 Flash | 7.66 | 8s | Fastest — best for bulk processing, wins la-vulners-mcp |\n| Gemini 2.5 Flash | 7.88 (prev) | 7s | Persistent verbosity — exceeds word limits |\n| GPT-5 Nano | 7.67 (prev) | 120s | All tokens consumed by internal reasoning; needs 16K max_tokens |\n| GLM 4.7 | 7.53 (v1) | 64s | Slow, expensive, only scored 4/6 files in v2 |\n| Kimi K2.5 | — | 200s | Replaced after v0 — 200s/call unusable |\n| Qwen Turbo | 6.07 (v1) | 37s | Collapsed on largest file (score 2.45) |\n| Claude 3 Haiku | 5.91 (v1) | 15s | Falsely triggered CRA Article 14 reporting |\n\n### Per-File Breakdown (v6)\n\n| File | DeepSeek V3 | GPT-4.1-mini | Gemini 3 Flash |\n|------|-------------|-------------|----------------|\n| openclaw (96 vulns) | 8.20 | **8.60** | 7.60 |\n| vulners-mcp (374 vulns) | **8.50** | 8.40 | 7.60 |\n| gurobi-engine (62 vulns) | **8.40** | 7.90 | 7.60 |\n| juice-shop (87 vulns) | **8.20** | 7.50 | 7.65 |\n| la-vulners-mcp (15 vulns) | 6.85 | 7.65 | **8.35** |\n| grbrsm_ui (9 vulns) | **7.90** | 6.25 | 7.15 |\n\n## Key Optimizations\n\n1. **Pre-computed CVE analytics** (v5) — Python deduplicates CVEs before sending to LLM. Eliminated counting errors that plagued all models.\n2. **Non-CVE advisory support** — GHSA and other non-CVE advisories are included via advisory ID fallback when `cvelist` is empty. Increased openclaw from 57 to 96 matched vulnerabilities.\n3. **Dual-source comparison judge** — Cross-referenced Vulners + Grype findings with per-CVE risk scores give the judge structured, verifiable ground truth.\n4. **Inline CRA article citations** (v6) — Baking \"Article 10/11/14\" into prompt section headers drove CRA alignment scores from ~7 to ~8.5.\n5. **Anti-hallucination rule** (v6) — \"ONLY reference CVEs in the input data\" reduced fabricated entries. Judge penalises only CVEs found in neither source.\n\n## Project Structure\n\n```\nSSA/\n├── vulners_results/ # Vulners SBOM audit JSON files\n├── grype_results/ # Grype vulnerability scan outputs\n├── comparisons/ # Cross-source comparison reports per SBOM\n│ └── {sbom-name}/\n│ ├── comparison.md # Human-readable report\n│ └── comparison.json # Structured data for judge\n├── prompts/ # Prompt versions (v1–v6)\n│ └── v6.txt # Best prompt\n├── results/ # Versioned evaluation results\n│ └── v6/ # Latest: rankings, judge scores, metrics\n├── compare_sources.py # Grype vs Vulners comparison script\n├── preprocess.py # Data preprocessing module\n├── summarize_sbom.py # Evaluation workbench (multi-model, judge)\n├── analyze.py # Single-file analysis script\n├── requirements.txt # Python dependencies\n└── .env # OPENROUTER_API_KEY (not committed)\n```\n\n## Setup\n\n```bash\npip3 install -r requirements.txt\necho 'OPENROUTER_API_KEY=sk-or-v1-your-key' > .env\n```\n\n## Scripts\n\n### `analyze.py` — Single-file analysis\n\nRuns the full pipeline on one Vulners SBOM audit JSON file: preprocess → call LLM → print summary to console.\n\n```bash\n# Default: Gemini 3 Flash + v6 prompt\npython3 analyze.py vulners_results/package-analysis-report-juice-shop.json\n\n# Custom model and prompt\npython3 analyze.py vulners_results/package-analysis-report-juice-shop.json --model deepseek-v3 --prompt prompts/v4.txt\n```\n\n### `compare_sources.py` — Grype vs Vulners comparison\n\nCross-references Grype and Vulners findings per SBOM. Normalizes vulnerability IDs (CVE + GHSA), computes risk scores, and generates comparison reports.\n\n```bash\n# Default directories\npython3 compare_sources.py\n\n# Custom paths\npython3 compare_sources.py --vulners vulners_results/ --grype grype_results/ --output comparisons/\n```\n\n### `summarize_sbom.py` — Evaluation workbench\n\nRuns all candidate models on all files, then uses a judge model (Claude Sonnet 4) to score and rank outputs using cross-source comparison data.\n\n```bash\n# Full evaluation (requires comparison data — run compare_sources.py first)\npython3 summarize_sbom.py --evaluate --prompt prompts/v6.txt --data vulners_results/ --comparisons comparisons/\n\n# Single model, all files (no judging)\npython3 summarize_sbom.py --summarize --model gemini-3-flash --prompt prompts/v6.txt --data vulners_results/\n```\n"}, "7595bd3111808b59655873bfdf0caa9efd74e58d": {"t": 1785771671.3378465, "n": 3865, "h": "7a901c938e4983a9", "c": "# Vulners SBOM Audit Summary — LLM Prompt & Model Evaluation\n\nLLM-powered summarization of [Vulners SBOM Audit](https://vulners.com/sbom-audit) results into CRA-aligned vulnerability reports. Audit results can be obtained via [UI](https://vulners.com/sbom-audit) or [API](https://docs.vulners.com/docs/api/audit/#sbom-audit).\n\n## Winner: Gemini 3 Flash + v6 prompt\n\n| | |\n| --- | --- |\n| **Model** | `google/gemini-3-flash-preview` via OpenRouter |\n| **Prompt** | `prompts/v6.txt` |\n| **Score** | 8.04 / 10 (judged by Claude Sonnet 4) |\n| **Speed** | ~9s per file |\n| **Cost** | ~$0.024 per file |\n\nBest balance of quality, speed, and cost for CRA-aligned vulnerability summaries from Vulners SBOM audit JSON. Consistent CRA article citations and correct mandatory reporting triggers.\n\nFor maximum quality: **DeepSeek V3** (8.56 avg, ~40s/file).\nCheaper alternative: **GPT-4.1-mini** (7.75 avg, ~22s/file, $0.016/file).\n\n## Evaluation Summary (9 models tested, 6 prompt iterations, 6 test files)\n\n| Model | Best Score | Speed | Status |\n|-------|-----------|-------|--------|\n| **Gemini 3 Flash** | **8.04** (v6) | **9s** | ✅ **Winner** — best speed-to-quality ratio |\n| DeepSeek V3 | 8.56 (v6) | 40s | Quality king but ~5x slower |\n| GPT-4.1-mini | 7.75 (v6) | 22s | Strong alternative — 37% cheaper, wins on largest files |\n| Gemini 2.5 Flash | 7.88 (v6) | 7s | Persistent verbosity — exceeds word limits |\n| GPT-5 Nano | 7.67 (v6) | 120s | All tokens consumed by internal reasoning; needs 16K max_tokens |\n| GLM 4.7 | 7.53 (v1) | 64s | Slow, expensive, only scored 4/6 files in v2 |\n| Kimi K2.5 | — | 200s | Replaced after v0 — 200s/call unusable |\n| Qwen Turbo | 6.07 (v1) | 37s | Collapsed on largest file (score 2.45) |\n| Claude 3 Haiku | 5.91 (v1) | 15s | Falsely triggered CRA Article 14 reporting |\n\n## Key Optimizations\n\n1. **Pre-computed CVE analytics** (v5) — Python deduplicates CVEs before sending to LLM. Eliminated counting errors that plagued all models.\n2. **Inline CRA article citations** (v6) — Baking \"Article 10/11/14\" into prompt section headers drove CRA alignment scores from ~7 to ~8.5.\n3. **Bullet points over prose** (v6) — Reduced verbosity by ~30% for Gemini models.\n4. **Anti-hallucination rule** (v6) — \"ONLY reference CVEs in the input data\" reduced fabricated entries.\n\n## Project Structure\n\n```\nSSA/\n├── sample_data/ # Vulners SBOM audit JSON files\n├── prompts/ # Prompt versions (v1–v6)\n│ └── v6.txt # Best prompt\n├── results/ # Versioned evaluation results\n│ └── v6/ # Latest: rankings, judge scores, metrics\n├── preprocess.py # Data preprocessing module\n├── summarize_sbom.py # Evaluation workbench (multi-model, judge)\n├── analyze.py # Single-file analysis script\n├── requirements.txt # Python dependencies\n└── .env # OPENROUTER_API_KEY (not committed)\n```\n\n## Setup\n\n```bash\npip3 install -r requirements.txt\necho 'OPENROUTER_API_KEY=sk-or-v1-your-key' > .env\n```\n\n## Scripts\n\n### `analyze.py` — Single-file analysis\n\nRuns the full pipeline on one Vulners SBOM audit JSON file: preprocess → call LLM → print summary to console.\n\n```bash\n# Default: Gemini 3 Flash + v6 prompt\npython3 analyze.py sample_data/package-analysis-report-juice-shop.json\n\n# Custom model and prompt\npython3 analyze.py sample_data/package-analysis-report-juice-shop.json --model deepseek-v3 --prompt prompts/v4.txt\n```\n\n### `summarize_sbom.py` — Evaluation workbench\n\nRuns all candidate models on all files, then uses a judge model (Claude Sonnet 4) to score and rank outputs.\n\n```bash\n# Full evaluation\npython3 summarize_sbom.py --evaluate --prompt prompts/v6.txt --data sample_data/\n\n# Single model, all files (no judging)\npython3 summarize_sbom.py --summarize --model gemini-3-flash --prompt prompts/v6.txt --data sample_data/\n```\n"}, "16bb1f238db0919486406adc7d1da47537f05dc9": {"t": 1785771672.7722614, "n": 3306, "h": "93dfc8762cd42217", "c": "# SBOM Summary — LLM Prompt & Model Evaluation\n\n## Winner: Gemini 3 Flash + v6 prompt\n\n| | |\n|---|---|\n| **Model** | `google/gemini-3-flash-preview` via OpenRouter |\n| **Prompt** | `prompts/v6.txt` |\n| **Score** | 8.16 / 10 (judged by Claude Sonnet 4) |\n| **Speed** | ~9s per file |\n| **Cost** | ~$0.024 per file |\n\nBest balance of quality, speed, and cost for CRA-aligned SBOM vulnerability summaries. Produces accurate, concise reports with proper EU CRA Article citations and correct mandatory reporting triggers.\n\nFor maximum quality at the expense of speed, use **DeepSeek V3** (8.47 avg, ~40s/file).\n\n## Evaluation Summary (6 rounds, 6 test files)\n\n| Model | Best Score | Speed | Why not winner |\n|-------|-----------|-------|----------------|\n| **Gemini 3 Flash** | **8.16** (v6) | **9s** | ✅ **Winner** |\n| DeepSeek V3 | 8.47 (v6) | 40s | Too slow for real-time use (~5x slower) |\n| Gemini 2.5 Flash | 7.88 (v6) | 7s | Persistent verbosity — exceeds word limits |\n| GLM 4.7 | 7.53 (v1) | 64s | Slow, expensive, only scored 4/6 files in v2 |\n| Kimi K2.5 | — | 200s | Replaced after v0 — 200s/call unusable |\n| Qwen Turbo | 6.07 (v1) | 37s | Collapsed on largest file (score 2.45) |\n| Claude 3 Haiku | 5.91 (v1) | 15s | Falsely triggered CRA Article 14 reporting |\n\n## Key Optimizations\n\n1. **Pre-computed CVE analytics** (v5) — Python deduplicates CVEs before sending to LLM. Eliminated counting errors that plagued all models.\n2. **Inline CRA article citations** (v6) — Baking \"Article 10/11/14\" into prompt section headers drove CRA alignment scores from ~7 to ~8.5.\n3. **Bullet points over prose** (v6) — Reduced verbosity by ~30% for Gemini models.\n4. **Anti-hallucination rule** (v6) — \"ONLY reference CVEs in the input data\" reduced fabricated entries.\n\n## Project Structure\n\n```\nSSA/\n├── sample_data/ # SBOM audit JSON files\n├── prompts/ # Prompt versions (v1–v6)\n│ └── v6.txt # Best prompt\n├── results/ # Versioned evaluation results\n│ └── v6/ # Latest: rankings, judge scores, metrics\n├── preprocess.py # Data preprocessing module\n├── summarize_sbom.py # Evaluation workbench (multi-model, judge)\n├── analyze.py # Single-file analysis script\n├── requirements.txt # Python dependencies\n└── .env # OPENROUTER_API_KEY (not committed)\n```\n\n## Setup\n\n```bash\npip3 install -r requirements.txt\necho 'OPENROUTER_API_KEY=sk-or-v1-your-key' > .env\n```\n\n## Scripts\n\n### `analyze.py` — Single-file analysis\n\nRuns the full pipeline on one SBOM JSON file: preprocess → call LLM → print summary to console.\n\n```bash\n# Default: Gemini 3 Flash + v6 prompt\npython3 analyze.py sample_data/package-analysis-report-juice-shop.json\n\n# Custom model and prompt\npython3 analyze.py sample_data/package-analysis-report-juice-shop.json --model deepseek-v3 --prompt prompts/v4.txt\n```\n\n### `summarize_sbom.py` — Evaluation workbench\n\nRuns all candidate models on all files, then uses a judge model (Claude Sonnet 4) to score and rank outputs.\n\n```bash\n# Full evaluation\npython3 summarize_sbom.py --evaluate --prompt prompts/v6.txt --data sample_data/\n\n# Single model, all files (no judging)\npython3 summarize_sbom.py --summarize --model gemini-3-flash --prompt prompts/v6.txt --data sample_data/\n```\n"}}}} |