1 line
32 KiB
JSON
1 line
32 KiB
JSON
{"repo": "ajamal27891/dw-bench", "files": {"README.md": {"f370dc24f16d00f4280648533d45af36054f3e17": {"t": 1785747286.3635738, "n": 10251, "h": "2a78c4b314b00e14", "c": "# DW-Bench: Benchmarking LLMs on Data Warehouse Graph Topology Reasoning\n\n[](LICENSE)\n[](https://doi.org/10.5281/zenodo.19676716)\n[](https://arxiv.org/abs/XXXX.XXXXX)\n[](https://huggingface.co/datasets/AJamal27891/dw-bench)\n[](https://python.org)\n[](https://github.com/AJamal27891/dw-bench/blob/main/paper_journal/main.pdf)\n\n**DW-Bench** is the first benchmark for evaluating whether Large Language Models can reason about the *graph topology* of data warehouse schemas: foreign-key paths, data lineage chains, connected components, and row-level provenance. It comprises 1,479 questions total (1,046 schema-level Tier 1 + 433 evaluated value-level Tier 2) across 5 datasets, with 6 baselines evaluated against 3 frontier models (Gemini 2.5 Flash, DeepSeek-V3, Qwen2.5-72B).\n\n---\n\n## Main Results\n\n### Tier 1: Schema-Level Reasoning (1,046 questions, 5 datasets)\n\n| Baseline | Gemini 2.5 Flash | DeepSeek-V3 | Qwen2.5-72B |\n|:---|:---:|:---:|:---:|\n| Flat Text (FT) | 76.6 | 69.5 | 63.2 |\n| Vector-RAG (VR) | 72.9 | 71.0 | 69.5 |\n| Graph-Aug (GA) | 75.4 | 76.8 | 80.9 |\n| **Tool-Use (TU)** | **89.3** | **90.4** | 87.5 |\n| ReAct-Code (RC) | 81.4 | 79.4 | 64.6 |\n| Oracle | 99.5 | 99.8 | 100.0 |\n\n> Micro-EM (%) across 1,046 questions (5 datasets).\n\n### Key Figures\n\n<p align=\"center\">\n <img src=\"paper/figures/difficulty_comparison.png\" width=\"90%\" alt=\"EM by difficulty level\">\n</p>\n\n**EM by difficulty level** across three models (Gemini, DeepSeek-V3, Qwen2.5-72B). All baselines plateau on hard questions (~40-60%) while Oracle (dashed) achieves 100%.\n\n<p align=\"center\">\n <img src=\"paper/figures/subtype_heatmap.png\" width=\"90%\" alt=\"Per-subtype heatmap\">\n</p>\n\n**Per-subtype EM heatmap** (darker red = lower). `combined_impact` (top row) is universally hard (0-17% EM); `membership` is now fully solvable with target-node normalization (TU/RC hit 100%).\n\n<p align=\"center\">\n <img src=\"paper/figures/oracle_gap.png\" width=\"90%\" alt=\"Oracle gap by subtype\">\n</p>\n\n**Unsolved subtypes** (Oracle minus best baseline). `combined_impact` retains an 83+ pp gap across all three models; 9 of 13 subtypes are effectively solved.\n\n### Obfuscation (Contamination Control)\n\n| Baseline | Gemini Δ | DeepSeek Δ | Qwen Δ |\n|:---|:---:|:---:|:---:|\n| Flat Text | −15.0 | −27.9 | −8.9 |\n| Vector-RAG | −25.8 | −31.9 | −13.8 |\n| Graph-Aug | −27.7 | −24.6 | −19.7 |\n| **Tool-Use** | **−3.4** | **−4.2** | **−3.7** |\n| ReAct-Code | −7.6 | +0.6 | −7.5 |\n\n> 4-dataset scope (excl. Syn-Logistics). Tool-Use is nearly obfuscation-invariant: tools operate on topology, not memorized names.\n\n---\n\n## Quick Start\n\n### 1. Install\n\n```bash\ngit clone https://github.com/AJamal27891/dw-bench.git\ncd dw-bench\npip install -r requirements.txt\n```\n\n### 2. Verify Pipeline\n\n```bash\n# Quick check: runs Oracle on TPC-DS (needs API endpoint)\npython run.py --api-base https://generativelanguage.googleapis.com/v1beta/openai \\\n --model gemini-2.5-flash --verify\n```\n\n### 3. Run Full Evaluation\n\n```bash\n# Tier 1: All baselines × all datasets (Gemini)\npython run.py \\\n --api-base https://generativelanguage.googleapis.com/v1beta/openai \\\n --model gemini-2.5-flash\n\n# Tier 1: Single baseline × single dataset\npython run.py --api-base ... --model ... --baseline tool_use --dataset tpc-ds\n\n# Tier 1: Obfuscated condition\npython run.py --api-base ... --model ... --condition obfuscated\n\n# Tier 2: Value-level evaluation\npython run.py --api-base ... --model ... --tier 2\n\n# DeepSeek\npython run.py \\\n --api-base https://api.deepseek.com/v1 \\\n --model deepseek-chat\n\n# Local model (LM Studio, Ollama, vLLM)\npython run.py \\\n --api-base http://localhost:1234/v1 \\\n --model your-model-name\n```\n\n### 4. View Results\n\n```bash\npython view_results.py # Summary tables\npython integrity_check.py # Validate result files (Windows)\npython integrity_check_full.py # Validate result files (Linux/macOS, rich output)\n```\n\n> **Tip:** Set `GOOGLE_API_KEY` or `DEEPSEEK_API_KEY` in a `.env` file; `run.py` reads them automatically.\n\n---\n\n## Project Structure\n\n```\ndw-bench/\n├── run.py # Unified evaluation runner\n├── view_results.py # Results viewer\n├── integrity_check.py # Result validator (Windows-safe)\n├── integrity_check_full.py # Result validator (Linux/macOS, rich output)\n├── push_to_hf.py # Upload/download data from HuggingFace\n├── CITATION.cff\n├── requirements.txt\n│\n├── datasets/ # Downloaded from HuggingFace (not in git)\n│ ├── adventureworks/ # 102 tables, 136 FK, 39 lineage\n│ ├── tpc-ds/ # 24 tables, 70 FK\n│ ├── tpc-di/ # 35 tables, 29 FK, 21 lineage\n│ ├── omop_cdm/ # 37 tables, 74 FK, 21 lineage\n│ └── syn_logistics/ # 64 tables, 96 FK, 35 lineage\n│\n├── evaluation/ # Evaluation pipeline\n│ ├── evaluate.py # Main evaluation harness\n│ ├── metrics.py # Scoring (EM, F1, path validation)\n│ ├── baselines/ # Baseline implementations\n│ │ ├── flat_text.py # Full schema as text\n│ │ ├── vector_rag.py # FAISS embedding retrieval\n│ │ ├── graph_aug.py # BFS graph neighborhoods\n│ │ ├── tool_use.py # Agentic graph tools (9 tools, 3 calls)\n│ │ ├── react_code.py # Python/NetworkX code gen (5 rounds)\n│ │ ├── oracle.py # Gold algorithmic output\n│ │ └── *_v2.py # Tier 2 (value-level) variants\n│ └── results/ # Downloaded from HuggingFace (not in git)\n│\n├── scripts/ # Data pipeline\n│ ├── generate_qa.py # Question generation\n│ ├── obfuscate_schema.py # Obfuscation protocol\n│ └── ...\n│\n├── paper/ # Conference paper\n└── paper_journal/ # Journal paper (Springer)\n```\n\n---\n\n## Benchmark Design\n\n### Datasets (262 tables, 521 edges)\n\n| Dataset | Domain | Tables | FK | Lineage | Tier 1 Qs | Tier 2 Qs | Silos |\n|:--|:--|:--:|:--:|:--:|:--:|:--:|:--:|\n| AdventureWorks | Retail/HR | 102 | 136 | 39 | 208 | 520 (generated) | 11 |\n| TPC-DS | Analytics | 24 | 70 | 0 | 127 | — | 1 |\n| TPC-DI | ETL | 35 | 29 | 21 | 181 | — | 2 |\n| OMOP CDM | Healthcare | 37 | 74 | 21 | 158 | — | 3 |\n| Syn-Logistics | Supply Chain | 64 | 96 | 35 | 372 | 433 (evaluated) | 5 |\n| **Total** | | **262** | **405** | **116** | **1,046** | **953 generated / 433 evaluated** | |\n\n> **Tier 2 note:** 953 value-level questions were generated in total (520 AdventureWorks + 433 Syn-Logistics). Only the 433 Syn-Logistics questions are evaluated in this study because AdventureWorks requires full ETL data hydration (ongoing work).\n\n### Question Taxonomy (13 subtypes, 3 difficulty levels)\n\n| Category | Subtype | Difficulty | Description |\n|:--|:--|:--|:--|\n| **Lineage** | `forward` | Easy | Direct lineage targets |\n| | `reverse` | Easy | Source tables for a DW table |\n| | `transitive` | Hard | Multi-hop lineage chains |\n| | `combined_impact` | Hard | Lineage + FK dependents |\n| | `multi_source` | Medium | Tables with 3+ sources |\n| **Route** | `direct_fk` | Easy | FK adjacency check |\n| | `join_path` | Medium | FK shortest path |\n| | `hop_count` | Medium | Path length |\n| **Silo** | `count` | Easy | Number of components |\n| | `membership` | Hard | Which component contains X? |\n| | `isolation` | Medium | Is table X isolated? |\n| | `connected` | Medium | Are X and Y connected? |\n| | `full_enumeration` | Hard | List all tables in silo |\n\n---\n\n## Six Baselines\n\n| Baseline | Context | Approach |\n|:--|:--|:--|\n| **Flat Text (FT)** | Full schema as text | Complete graph, maximal context |\n| **Vector RAG (VR)** | Top-k retrieved chunks | Sentence-BERT + FAISS (k=15) |\n| **Graph-Aug (GA)** | BFS neighborhood | 3-hop subgraph from mentioned tables |\n| **Tool-Use (TU)** | Schema + 9 graph tools | Agentic: multi-turn tool calling (3 calls) |\n| **ReAct-Code (RC)** | Schema + Python exec | Code generation on NetworkX graph (5 rounds) |\n| **Oracle** | Gold algorithmic output | Upper bound (perfect retrieval) |\n\n---\n\n## Evaluation Conditions\n\n- **Original** — Real table names (tests reasoning + memorization)\n- **Obfuscated** — Names replaced with `Table_A`, `Table_B`, ... (tests pure structural reasoning)\n- **Extended** — Hard + medium questions only (harder subset)\n\n---\n\n## Extending DW-Bench\n\n### Adding a New Model\n\n```bash\npython run.py --api-base YOUR_URL --model YOUR_MODEL --condition all\npython integrity_check.py\n```\n\n### Adding a New Dataset\n\n1. Create `datasets/your_dataset/schema_graph.pt` (PyG HeteroData)\n2. Run `python scripts/generate_qa.py --dataset your_dataset`\n3. Run `python scripts/obfuscate_schema.py --dataset your_dataset`\n4. Evaluate: `python run.py --api-base ... --model ... --dataset your_dataset`\n\n---\n\n## Citation\n\n```bibtex\n@misc{ahmed2026dwbench,\n title = {DW-Bench: Benchmarking LLMs on Data Warehouse Graph Topology Reasoning},\n author = {Ahmed, Ahmed G.A.H. and Sakar, C. Okan},\n year = {2026},\n url = {https://github.com/AJamal27891/dw-bench},\n note = {NeurIPS 2026 Datasets and Benchmarks Track. arXiv preprint pending.}\n}\n```\n\nFor the extended journal edition:\n\n```bibtex\n@article{ahmed2026dwbench_journal,\n title = {DW-Bench: A Benchmark for Graph Topology Reasoning\n in Data Warehouse Schemas with Large Language Models},\n author = {Ahmed, Ahmed G.A.H. and Sakar, C. Okan},\n journal = {Journal of Big Data},\n publisher = {Springer},\n year = {2026},\n url = {https://github.com/AJamal27891/dw-bench}\n}\n```\n\n---\n\n## License\n\nMIT License. See [LICENSE](LICENSE) for details.\n"}, "75b25f54a21634ef0d11a05bba44a3e45ea06583": {"t": 1785772151.0560527, "n": 10249, "h": "d68fe7479879a802", "c": "# DW-Bench: Benchmarking LLMs on Data Warehouse Graph Topology Reasoning\n\n[](LICENSE)\n[](https://doi.org/10.5281/zenodo.XXXXXXX)\n[](https://arxiv.org/abs/XXXX.XXXXX)\n[](https://huggingface.co/datasets/AJamal27891/dw-bench)\n[](https://python.org)\n[](https://github.com/AJamal27891/dw-bench/blob/main/paper_journal/main.pdf)\n\n**DW-Bench** is the first benchmark for evaluating whether Large Language Models can reason about the *graph topology* of data warehouse schemas: foreign-key paths, data lineage chains, connected components, and row-level provenance. It comprises 1,479 questions total (1,046 schema-level Tier 1 + 433 evaluated value-level Tier 2) across 5 datasets, with 6 baselines evaluated against 3 frontier models (Gemini 2.5 Flash, DeepSeek-V3, Qwen2.5-72B).\n\n---\n\n## Main Results\n\n### Tier 1: Schema-Level Reasoning (1,046 questions, 5 datasets)\n\n| Baseline | Gemini 2.5 Flash | DeepSeek-V3 | Qwen2.5-72B |\n|:---|:---:|:---:|:---:|\n| Flat Text (FT) | 76.6 | 69.5 | 63.2 |\n| Vector-RAG (VR) | 72.9 | 71.0 | 69.5 |\n| Graph-Aug (GA) | 75.4 | 76.8 | 80.9 |\n| **Tool-Use (TU)** | **89.3** | **90.4** | 87.5 |\n| ReAct-Code (RC) | 81.4 | 79.4 | 64.6 |\n| Oracle | 99.5 | 99.8 | 100.0 |\n\n> Micro-EM (%) across 1,046 questions (5 datasets).\n\n### Key Figures\n\n<p align=\"center\">\n <img src=\"paper/figures/difficulty_comparison.png\" width=\"90%\" alt=\"EM by difficulty level\">\n</p>\n\n**EM by difficulty level** across three models (Gemini, DeepSeek-V3, Qwen2.5-72B). All baselines plateau on hard questions (~40-60%) while Oracle (dashed) achieves 100%.\n\n<p align=\"center\">\n <img src=\"paper/figures/subtype_heatmap.png\" width=\"90%\" alt=\"Per-subtype heatmap\">\n</p>\n\n**Per-subtype EM heatmap** (darker red = lower). `combined_impact` (top row) is universally hard (0-17% EM); `membership` is now fully solvable with target-node normalization (TU/RC hit 100%).\n\n<p align=\"center\">\n <img src=\"paper/figures/oracle_gap.png\" width=\"90%\" alt=\"Oracle gap by subtype\">\n</p>\n\n**Unsolved subtypes** (Oracle minus best baseline). `combined_impact` retains an 83+ pp gap across all three models; 9 of 13 subtypes are effectively solved.\n\n### Obfuscation (Contamination Control)\n\n| Baseline | Gemini Δ | DeepSeek Δ | Qwen Δ |\n|:---|:---:|:---:|:---:|\n| Flat Text | −15.0 | −27.9 | −8.9 |\n| Vector-RAG | −25.8 | −31.9 | −13.8 |\n| Graph-Aug | −27.7 | −24.6 | −19.7 |\n| **Tool-Use** | **−3.4** | **−4.2** | **−3.7** |\n| ReAct-Code | −7.6 | +0.6 | −7.5 |\n\n> 4-dataset scope (excl. Syn-Logistics). Tool-Use is nearly obfuscation-invariant: tools operate on topology, not memorized names.\n\n---\n\n## Quick Start\n\n### 1. Install\n\n```bash\ngit clone https://github.com/AJamal27891/dw-bench.git\ncd dw-bench\npip install -r requirements.txt\n```\n\n### 2. Verify Pipeline\n\n```bash\n# Quick check: runs Oracle on TPC-DS (needs API endpoint)\npython run.py --api-base https://generativelanguage.googleapis.com/v1beta/openai \\\n --model gemini-2.5-flash --verify\n```\n\n### 3. Run Full Evaluation\n\n```bash\n# Tier 1: All baselines × all datasets (Gemini)\npython run.py \\\n --api-base https://generativelanguage.googleapis.com/v1beta/openai \\\n --model gemini-2.5-flash\n\n# Tier 1: Single baseline × single dataset\npython run.py --api-base ... --model ... --baseline tool_use --dataset tpc-ds\n\n# Tier 1: Obfuscated condition\npython run.py --api-base ... --model ... --condition obfuscated\n\n# Tier 2: Value-level evaluation\npython run.py --api-base ... --model ... --tier 2\n\n# DeepSeek\npython run.py \\\n --api-base https://api.deepseek.com/v1 \\\n --model deepseek-chat\n\n# Local model (LM Studio, Ollama, vLLM)\npython run.py \\\n --api-base http://localhost:1234/v1 \\\n --model your-model-name\n```\n\n### 4. View Results\n\n```bash\npython view_results.py # Summary tables\npython integrity_check.py # Validate result files (Windows)\npython integrity_check_full.py # Validate result files (Linux/macOS, rich output)\n```\n\n> **Tip:** Set `GOOGLE_API_KEY` or `DEEPSEEK_API_KEY` in a `.env` file; `run.py` reads them automatically.\n\n---\n\n## Project Structure\n\n```\ndw-bench/\n├── run.py # Unified evaluation runner\n├── view_results.py # Results viewer\n├── integrity_check.py # Result validator (Windows-safe)\n├── integrity_check_full.py # Result validator (Linux/macOS, rich output)\n├── push_to_hf.py # Upload/download data from HuggingFace\n├── CITATION.cff\n├── requirements.txt\n│\n├── datasets/ # Downloaded from HuggingFace (not in git)\n│ ├── adventureworks/ # 102 tables, 136 FK, 39 lineage\n│ ├── tpc-ds/ # 24 tables, 70 FK\n│ ├── tpc-di/ # 35 tables, 29 FK, 21 lineage\n│ ├── omop_cdm/ # 37 tables, 74 FK, 21 lineage\n│ └── syn_logistics/ # 64 tables, 96 FK, 35 lineage\n│\n├── evaluation/ # Evaluation pipeline\n│ ├── evaluate.py # Main evaluation harness\n│ ├── metrics.py # Scoring (EM, F1, path validation)\n│ ├── baselines/ # Baseline implementations\n│ │ ├── flat_text.py # Full schema as text\n│ │ ├── vector_rag.py # FAISS embedding retrieval\n│ │ ├── graph_aug.py # BFS graph neighborhoods\n│ │ ├── tool_use.py # Agentic graph tools (9 tools, 3 calls)\n│ │ ├── react_code.py # Python/NetworkX code gen (5 rounds)\n│ │ ├── oracle.py # Gold algorithmic output\n│ │ └── *_v2.py # Tier 2 (value-level) variants\n│ └── results/ # Downloaded from HuggingFace (not in git)\n│\n├── scripts/ # Data pipeline\n│ ├── generate_qa.py # Question generation\n│ ├── obfuscate_schema.py # Obfuscation protocol\n│ └── ...\n│\n├── paper/ # Conference paper\n└── paper_journal/ # Journal paper (Springer)\n```\n\n---\n\n## Benchmark Design\n\n### Datasets (262 tables, 521 edges)\n\n| Dataset | Domain | Tables | FK | Lineage | Tier 1 Qs | Tier 2 Qs | Silos |\n|:--|:--|:--:|:--:|:--:|:--:|:--:|:--:|\n| AdventureWorks | Retail/HR | 102 | 136 | 39 | 208 | 520 (generated) | 11 |\n| TPC-DS | Analytics | 24 | 70 | 0 | 127 | — | 1 |\n| TPC-DI | ETL | 35 | 29 | 21 | 181 | — | 2 |\n| OMOP CDM | Healthcare | 37 | 74 | 21 | 158 | — | 3 |\n| Syn-Logistics | Supply Chain | 64 | 96 | 35 | 372 | 433 (evaluated) | 5 |\n| **Total** | | **262** | **405** | **116** | **1,046** | **953 generated / 433 evaluated** | |\n\n> **Tier 2 note:** 953 value-level questions were generated in total (520 AdventureWorks + 433 Syn-Logistics). Only the 433 Syn-Logistics questions are evaluated in this study because AdventureWorks requires full ETL data hydration (ongoing work).\n\n### Question Taxonomy (13 subtypes, 3 difficulty levels)\n\n| Category | Subtype | Difficulty | Description |\n|:--|:--|:--|:--|\n| **Lineage** | `forward` | Easy | Direct lineage targets |\n| | `reverse` | Easy | Source tables for a DW table |\n| | `transitive` | Hard | Multi-hop lineage chains |\n| | `combined_impact` | Hard | Lineage + FK dependents |\n| | `multi_source` | Medium | Tables with 3+ sources |\n| **Route** | `direct_fk` | Easy | FK adjacency check |\n| | `join_path` | Medium | FK shortest path |\n| | `hop_count` | Medium | Path length |\n| **Silo** | `count` | Easy | Number of components |\n| | `membership` | Hard | Which component contains X? |\n| | `isolation` | Medium | Is table X isolated? |\n| | `connected` | Medium | Are X and Y connected? |\n| | `full_enumeration` | Hard | List all tables in silo |\n\n---\n\n## Six Baselines\n\n| Baseline | Context | Approach |\n|:--|:--|:--|\n| **Flat Text (FT)** | Full schema as text | Complete graph, maximal context |\n| **Vector RAG (VR)** | Top-k retrieved chunks | Sentence-BERT + FAISS (k=15) |\n| **Graph-Aug (GA)** | BFS neighborhood | 3-hop subgraph from mentioned tables |\n| **Tool-Use (TU)** | Schema + 9 graph tools | Agentic: multi-turn tool calling (3 calls) |\n| **ReAct-Code (RC)** | Schema + Python exec | Code generation on NetworkX graph (5 rounds) |\n| **Oracle** | Gold algorithmic output | Upper bound (perfect retrieval) |\n\n---\n\n## Evaluation Conditions\n\n- **Original** — Real table names (tests reasoning + memorization)\n- **Obfuscated** — Names replaced with `Table_A`, `Table_B`, ... (tests pure structural reasoning)\n- **Extended** — Hard + medium questions only (harder subset)\n\n---\n\n## Extending DW-Bench\n\n### Adding a New Model\n\n```bash\npython run.py --api-base YOUR_URL --model YOUR_MODEL --condition all\npython integrity_check.py\n```\n\n### Adding a New Dataset\n\n1. Create `datasets/your_dataset/schema_graph.pt` (PyG HeteroData)\n2. Run `python scripts/generate_qa.py --dataset your_dataset`\n3. Run `python scripts/obfuscate_schema.py --dataset your_dataset`\n4. Evaluate: `python run.py --api-base ... --model ... --dataset your_dataset`\n\n---\n\n## Citation\n\n```bibtex\n@misc{ahmed2026dwbench,\n title = {DW-Bench: Benchmarking LLMs on Data Warehouse Graph Topology Reasoning},\n author = {Ahmed, Ahmed G.A.H. and Sakar, C. Okan},\n year = {2026},\n url = {https://github.com/AJamal27891/dw-bench},\n note = {NeurIPS 2026 Datasets and Benchmarks Track. arXiv preprint pending.}\n}\n```\n\nFor the extended journal edition:\n\n```bibtex\n@article{ahmed2026dwbench_journal,\n title = {DW-Bench: A Benchmark for Graph Topology Reasoning\n in Data Warehouse Schemas with Large Language Models},\n author = {Ahmed, Ahmed G.A.H. and Sakar, C. Okan},\n journal = {Journal of Big Data},\n publisher = {Springer},\n year = {2026},\n url = {https://github.com/AJamal27891/dw-bench}\n}\n```\n\n---\n\n## License\n\nMIT License. See [LICENSE](LICENSE) for details.\n"}, "f3ba5729bc6753e570861562cf37f40aadca6852": {"t": 1785772154.656773, "n": 10397, "h": "0eedca3be2915bed", "c": "# DW-Bench: Benchmarking LLMs on Data Warehouse Graph Topology Reasoning\n\n[](LICENSE)\n[](https://python.org)\n[](https://huggingface.co/datasets/AJamal27891/dw-bench)\n[](https://huggingface.co/datasets/AJamal27891/dw-bench)\n[](https://huggingface.co/datasets/AJamal27891/dw-bench)\n[](https://github.com/AJamal27891/dw-bench/blob/main/paper/main.pdf)\n[](https://github.com/AJamal27891/dw-bench/blob/main/paper_journal/main.pdf)\n\n**DW-Bench** is the first benchmark for evaluating whether Large Language Models can reason about the *graph topology* of data warehouse schemas: foreign-key paths, data lineage chains, connected components, and row-level provenance. It comprises 1,479 questions total (1,046 schema-level Tier 1 + 433 evaluated value-level Tier 2) across 5 datasets, with 6 baselines evaluated against 3 frontier models (Gemini 2.5 Flash, DeepSeek-V3, Qwen2.5-72B).\n\n---\n\n## Main Results\n\n### Tier 1: Schema-Level Reasoning (1,046 questions, 5 datasets)\n\n| Baseline | Gemini 2.5 Flash | DeepSeek-V3 | Qwen2.5-72B |\n|:---|:---:|:---:|:---:|\n| Flat Text (FT) | 76.6 | 69.5 | 63.2 |\n| Vector-RAG (VR) | 72.9 | 71.0 | 69.5 |\n| Graph-Aug (GA) | 75.4 | 76.8 | 80.9 |\n| **Tool-Use (TU)** | **89.3** | **90.4** | 87.5 |\n| ReAct-Code (RC) | 81.4 | 79.4 | 64.6 |\n| Oracle | 99.5 | 99.8 | 100.0 |\n\n> Micro-EM (%) across 1,046 questions (5 datasets).\n\n### Key Figures\n\n<p align=\"center\">\n <img src=\"paper/figures/difficulty_comparison.png\" width=\"90%\" alt=\"EM by difficulty level\">\n</p>\n\n**EM by difficulty level** across three models (Gemini, DeepSeek-V3, Qwen2.5-72B). All baselines plateau on hard questions (~40-60%) while Oracle (dashed) achieves 100%.\n\n<p align=\"center\">\n <img src=\"paper/figures/subtype_heatmap.png\" width=\"90%\" alt=\"Per-subtype heatmap\">\n</p>\n\n**Per-subtype EM heatmap** (darker red = lower). `combined_impact` (top row) is universally hard (0-17% EM); `membership` is now fully solvable with target-node normalization (TU/RC hit 100%).\n\n<p align=\"center\">\n <img src=\"paper/figures/oracle_gap.png\" width=\"90%\" alt=\"Oracle gap by subtype\">\n</p>\n\n**Unsolved subtypes** (Oracle minus best baseline). `combined_impact` retains an 83+ pp gap across all three models; 9 of 13 subtypes are effectively solved.\n\n### Obfuscation (Contamination Control)\n\n| Baseline | Gemini Δ | DeepSeek Δ | Qwen Δ |\n|:---|:---:|:---:|:---:|\n| Flat Text | −15.0 | −27.9 | −8.9 |\n| Vector-RAG | −25.8 | −31.9 | −13.8 |\n| Graph-Aug | −27.7 | −24.6 | −19.7 |\n| **Tool-Use** | **−3.4** | **−4.2** | **−3.7** |\n| ReAct-Code | −7.6 | +0.6 | −7.5 |\n\n> 4-dataset scope (excl. Syn-Logistics). Tool-Use is nearly obfuscation-invariant: tools operate on topology, not memorized names.\n\n---\n\n## Quick Start\n\n### 1. Install\n\n```bash\ngit clone https://github.com/AJamal27891/dw-bench.git\ncd dw-bench\npip install -r requirements.txt\n```\n\n### 2. Verify Pipeline\n\n```bash\n# Quick check: runs Oracle on TPC-DS (needs API endpoint)\npython run.py --api-base https://generativelanguage.googleapis.com/v1beta/openai \\\n --model gemini-2.5-flash --verify\n```\n\n### 3. Run Full Evaluation\n\n```bash\n# Tier 1: All baselines × all datasets (Gemini)\npython run.py \\\n --api-base https://generativelanguage.googleapis.com/v1beta/openai \\\n --model gemini-2.5-flash\n\n# Tier 1: Single baseline × single dataset\npython run.py --api-base ... --model ... --baseline tool_use --dataset tpc-ds\n\n# Tier 1: Obfuscated condition\npython run.py --api-base ... --model ... --condition obfuscated\n\n# Tier 2: Value-level evaluation\npython run.py --api-base ... --model ... --tier 2\n\n# DeepSeek\npython run.py \\\n --api-base https://api.deepseek.com/v1 \\\n --model deepseek-chat\n\n# Local model (LM Studio, Ollama, vLLM)\npython run.py \\\n --api-base http://localhost:1234/v1 \\\n --model your-model-name\n```\n\n### 4. View Results\n\n```bash\npython view_results.py # Summary tables\npython integrity_check.py # Validate result files (Windows)\npython integrity_check_full.py # Validate result files (Linux/macOS, rich output)\n```\n\n> **Tip:** Set `GOOGLE_API_KEY` or `DEEPSEEK_API_KEY` in a `.env` file; `run.py` reads them automatically.\n\n---\n\n## Project Structure\n\n```\ndw-bench/\n├── run.py # Unified evaluation runner\n├── view_results.py # Results viewer\n├── integrity_check.py # Result validator (Windows-safe)\n├── integrity_check_full.py # Result validator (Linux/macOS, rich output)\n├── push_to_hf.py # Upload/download data from HuggingFace\n├── CITATION.cff\n├── requirements.txt\n│\n├── datasets/ # Downloaded from HuggingFace (not in git)\n│ ├── adventureworks/ # 102 tables, 136 FK, 39 lineage\n│ ├── tpc-ds/ # 24 tables, 70 FK\n│ ├── tpc-di/ # 35 tables, 29 FK, 21 lineage\n│ ├── omop_cdm/ # 37 tables, 74 FK, 21 lineage\n│ └── syn_logistics/ # 64 tables, 96 FK, 35 lineage\n│\n├── evaluation/ # Evaluation pipeline\n│ ├── evaluate.py # Main evaluation harness\n│ ├── metrics.py # Scoring (EM, F1, path validation)\n│ ├── baselines/ # Baseline implementations\n│ │ ├── flat_text.py # Full schema as text\n│ │ ├── vector_rag.py # FAISS embedding retrieval\n│ │ ├── graph_aug.py # BFS graph neighborhoods\n│ │ ├── tool_use.py # Agentic graph tools (9 tools, 3 calls)\n│ │ ├── react_code.py # Python/NetworkX code gen (5 rounds)\n│ │ ├── oracle.py # Gold algorithmic output\n│ │ └── *_v2.py # Tier 2 (value-level) variants\n│ └── results/ # Downloaded from HuggingFace (not in git)\n│\n├── scripts/ # Data pipeline\n│ ├── generate_qa.py # Question generation\n│ ├── obfuscate_schema.py # Obfuscation protocol\n│ └── ...\n│\n├── paper/ # Conference paper\n└── paper_journal/ # Journal paper (Springer)\n```\n\n---\n\n## Benchmark Design\n\n### Datasets (262 tables, 521 edges)\n\n| Dataset | Domain | Tables | FK | Lineage | Tier 1 Qs | Tier 2 Qs | Silos |\n|:--|:--|:--:|:--:|:--:|:--:|:--:|:--:|\n| AdventureWorks | Retail/HR | 102 | 136 | 39 | 208 | 520 (generated) | 11 |\n| TPC-DS | Analytics | 24 | 70 | 0 | 127 | — | 1 |\n| TPC-DI | ETL | 35 | 29 | 21 | 181 | — | 2 |\n| OMOP CDM | Healthcare | 37 | 74 | 21 | 158 | — | 3 |\n| Syn-Logistics | Supply Chain | 64 | 96 | 35 | 372 | 433 (evaluated) | 5 |\n| **Total** | | **262** | **405** | **116** | **1,046** | **953 generated / 433 evaluated** | |\n\n> **Tier 2 note:** 953 value-level questions were generated in total (520 AdventureWorks + 433 Syn-Logistics). Only the 433 Syn-Logistics questions are evaluated in this study because AdventureWorks requires full ETL data hydration (ongoing work).\n\n### Question Taxonomy (13 subtypes, 3 difficulty levels)\n\n| Category | Subtype | Difficulty | Description |\n|:--|:--|:--|:--|\n| **Lineage** | `forward` | Easy | Direct lineage targets |\n| | `reverse` | Easy | Source tables for a DW table |\n| | `transitive` | Hard | Multi-hop lineage chains |\n| | `combined_impact` | Hard | Lineage + FK dependents |\n| | `multi_source` | Medium | Tables with 3+ sources |\n| **Route** | `direct_fk` | Easy | FK adjacency check |\n| | `join_path` | Medium | FK shortest path |\n| | `hop_count` | Medium | Path length |\n| **Silo** | `count` | Easy | Number of components |\n| | `membership` | Hard | Which component contains X? |\n| | `isolation` | Medium | Is table X isolated? |\n| | `connected` | Medium | Are X and Y connected? |\n| | `full_enumeration` | Hard | List all tables in silo |\n\n---\n\n## Six Baselines\n\n| Baseline | Context | Approach |\n|:--|:--|:--|\n| **Flat Text (FT)** | Full schema as text | Complete graph, maximal context |\n| **Vector RAG (VR)** | Top-k retrieved chunks | Sentence-BERT + FAISS (k=15) |\n| **Graph-Aug (GA)** | BFS neighborhood | 3-hop subgraph from mentioned tables |\n| **Tool-Use (TU)** | Schema + 9 graph tools | Agentic: multi-turn tool calling (3 calls) |\n| **ReAct-Code (RC)** | Schema + Python exec | Code generation on NetworkX graph (5 rounds) |\n| **Oracle** | Gold algorithmic output | Upper bound (perfect retrieval) |\n\n---\n\n## Evaluation Conditions\n\n- **Original** — Real table names (tests reasoning + memorization)\n- **Obfuscated** — Names replaced with `Table_A`, `Table_B`, ... (tests pure structural reasoning)\n- **Extended** — Hard + medium questions only (harder subset)\n\n---\n\n## Extending DW-Bench\n\n### Adding a New Model\n\n```bash\npython run.py --api-base YOUR_URL --model YOUR_MODEL --condition all\npython integrity_check.py\n```\n\n### Adding a New Dataset\n\n1. Create `datasets/your_dataset/schema_graph.pt` (PyG HeteroData)\n2. Run `python scripts/generate_qa.py --dataset your_dataset`\n3. Run `python scripts/obfuscate_schema.py --dataset your_dataset`\n4. Evaluate: `python run.py --api-base ... --model ... --dataset your_dataset`\n\n---\n\n## Citation\n\n```bibtex\n@misc{ahmed2026dwbench,\n title = {DW-Bench: Benchmarking LLMs on Data Warehouse Graph Topology Reasoning},\n author = {Ahmed, Ahmed G.A.H.},\n year = {2026},\n url = {https://github.com/AJamal27891/dw-bench},\n note = {Preprint. Under review.}\n}\n```\n\nFor the extended journal version:\n\n```bibtex\n@article{ahmed2026dwbench_journal,\n title = {DW-Bench: A Benchmark for Graph Topology Reasoning\n in Data Warehouse Schemas with Large Language Models},\n author = {Ahmed, Ahmed G.A.H.},\n journal = {Journal of Big Data},\n publisher = {Springer},\n year = {2026},\n url = {https://github.com/AJamal27891/dw-bench}\n}\n```\n\n---\n\n## License\n\nMIT License. See [LICENSE](LICENSE) for details.\n"}}}} |