1 line
67 KiB
JSON
1 line
67 KiB
JSON
{"repo": "alijafarkamal/medical-intelligence-agent-ghana", "files": {"AGENT.md": {"195aa7b58da2d66f04f1e88c2003ffda1f2a5398": {"t": 1785596446.99227, "n": 63991, "h": "ec7a4af869804b87", "c": "# AGENT.md — Hack Nation: Bridging Medical Deserts (Databricks Track)\n\n---\n\n## Mission Statement\n\nBuild an **Intelligent Document Parsing (IDP) Agent** that extracts, verifies, and reasons over medical facility data from Ghana (provided by the Virtue Foundation) to identify **medical deserts** and infrastructure gaps. The system must parse unstructured free-form text, synthesize it with structured facility schemas, detect anomalies, and present findings through an interactive map + natural language planning interface.\n\n---\n\n## Evaluation Criteria\n\n| Criterion | Weight | What Wins |\n| ---------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------- |\n| **Technical Accuracy** | **35%** | Agent reliably handles \"Must Have\" queries from the question bank. Detects anomalies in facility data. |\n| **IDP Innovation** | **30%** | Extracts + synthesizes info from unstructured free-form text (procedure, equipment, capability columns). Goes beyond keyword matching. |\n| **Social Impact** | **25%** | Effectively identifies medical deserts. Aids resource allocation decisions. Compelling narrative. |\n| **User Experience** | **10%** | Intuitive for non-technical NGO planners. Natural language interface. |\n\n---\n\n## Architecture — Databricks-Native\n\nThe challenge recommends this base stack:\n\n> **Agentic orchestrator:** LangGraph, LlamaIndex, CrewAI\n> **ML lifecycle:** MLflow\n> **RAG:** Databricks, FAISS, LanceDB\n> **Text2SQL:** Genie\n\nOur architecture: **LangGraph** orchestrates a multi-agent graph locally. Each agent node calls **Databricks Free Edition** services as tools. **MLflow** traces every step. **Streamlit** renders the frontend.\n\n```mermaid\nflowchart TD\n subgraph FE[\"Frontend (Local)\"]\n ST[\"Streamlit App\"]\n MAP[\"Folium Map\"]\n end\n\n subgraph LG[\"LangGraph Agent Graph (Local)\"]\n SUP[\"Supervisor Node<br/><i>Intent classification + routing</i>\"]\n SQL_N[\"SQL Agent Node<br/><i>Calls Genie</i>\"]\n VEC_N[\"RAG Agent Node<br/><i>Calls Vector Search</i>\"]\n IDP_N[\"IDP Extraction Node<br/><i>Extracts structured facts from free-form text</i>\"]\n MED_N[\"Medical Reasoning Node<br/><i>Calls Model Serving</i>\"]\n GEO_N[\"Geospatial Node<br/><i>Local Haversine + desert detection</i>\"]\n SYN[\"Synthesis Node<br/><i>Cross-references structured vs unstructured + citations</i>\"]\n end\n\n subgraph DB[\"Databricks Free Edition (Remote)\"]\n UC[\"Unity Catalog — Delta Table\"]\n GN[\"Genie — Text-to-SQL\"]\n VS[\"Vector Search — RAG\"]\n MS[\"Model Serving — Llama 3.3 70B\"]\n ML[\"MLflow — Tracing + Citations\"]\n end\n\n ST -->|\"User query\"| SUP\n SUP -->|STRUCTURED| SQL_N\n SUP -->|SEMANTIC| VEC_N\n SUP -->|EXTRACT| IDP_N\n SUP -->|ANOMALY| MED_N\n SUP -->|GEOSPATIAL| GEO_N\n SQL_N -->|\"SDK call\"| GN\n VEC_N -->|\"SDK call\"| VS\n IDP_N -->|\"SDK call\"| VS\n IDP_N -->|\"Extract facts\"| MS\n MED_N -->|\"SDK call\"| MS\n GN --> UC\n VS --> UC\n SQL_N --> SYN\n VEC_N --> SYN\n IDP_N --> SYN\n MED_N --> SYN\n GEO_N --> SYN\n SYN --> ST\n SYN -->|\"Log trace\"| ML\n ST --> MAP\n\n style FE fill:#EBF5FB,stroke:#2E86C1\n style LG fill:#E8F8F5,stroke:#1ABC9C\n style DB fill:#FEF9E7,stroke:#F39C12\n```\n\n### How Each Databricks Feature Maps to the Challenge\n\n| Challenge Requirement | Databricks Feature | What It Does | Docs |\n|---|---|---|---|\n| **Structured queries** (counts, aggregations) | **Genie** (Text-to-SQL) | Takes natural language, generates SQL, executes against Delta table, returns results | [docs.databricks.com/en/genie](https://docs.databricks.com/en/genie/) |\n| **Semantic search** over free-form text | **Mosaic AI Vector Search** | Auto-embeds procedure/equipment/capability columns, returns top-k matches by semantic similarity | [docs.databricks.com/en/generative-ai/vector-search](https://docs.databricks.com/en/generative-ai/vector-search) |\n| **Anomaly detection** (procedure-equipment gaps) | **Model Serving** (Llama 3.3 70B) | LLM reasons over facility data to detect mismatches between claimed procedures and listed equipment | [docs.databricks.com/en/machine-learning/model-serving](https://docs.databricks.com/en/machine-learning/model-serving) |\n| **Data storage** with governance | **Unity Catalog** | Managed Delta table with column descriptions — makes Genie smarter and shows data governance | [docs.databricks.com/en/data-governance/unity-catalog](https://docs.databricks.com/en/data-governance/unity-catalog) |\n| **IDP extraction** (structured facts from free-form text) | **Vector Search** + **Model Serving** | Retrieves raw free-form text, then LLM extracts structured facts (procedures, equipment, capabilities) using sponsor's extraction prompts | [resources/prompts_and_pydantic_models/free_form.py](resources/prompts_and_pydantic_models/free_form.py) |\n| **Citation trail** (which data produced each answer) | **MLflow Tracing** | Logs inputs/outputs of each step, creating an audit trail of which rows were used | [docs.databricks.com/en/mlflow](https://docs.databricks.com/en/mlflow) |\n| **Data cleaning** + geocoding | **Notebooks** | Python notebook for one-time CSV cleaning, region normalization, geocoding, upload to Unity Catalog | [docs.databricks.com/en/notebooks](https://docs.databricks.com/en/notebooks) |\n\n### What Runs Where\n\n| Component | Runs On | Technology | Docs |\n|---|---|---|---|\n| **Agent orchestration** | Local | LangGraph (state graph with supervisor routing) | [langchain-ai.github.io/langgraph](https://langchain-ai.github.io/langgraph/) |\n| **Frontend** | Local | Streamlit + Folium map | [docs.streamlit.io](https://docs.streamlit.io/) |\n| **Geospatial** | Local | Python Haversine + numpy | — |\n| **Text-to-SQL** | Databricks | Genie | [docs.databricks.com/en/genie](https://docs.databricks.com/en/genie/) |\n| **RAG / Semantic search** | Databricks | Vector Search | [docs.databricks.com/en/generative-ai/vector-search](https://docs.databricks.com/en/generative-ai/vector-search) |\n| **LLM inference** | Databricks | Model Serving (Llama 3.3 70B) | [docs.databricks.com/en/machine-learning/model-serving](https://docs.databricks.com/en/machine-learning/model-serving) |\n| **Agent tracing** | Databricks | MLflow Tracing | [docs.databricks.com/en/mlflow](https://docs.databricks.com/en/mlflow) |\n| **Data storage** | Databricks | Unity Catalog Delta table | [docs.databricks.com/en/data-governance/unity-catalog](https://docs.databricks.com/en/data-governance/unity-catalog) |\n\n---\n\n## Tech Stack\n\n| Layer | Technology | Docs |\n|---|---|---|\n| **Agent Orchestration** | **LangGraph** (multi-agent state graph) | [langchain-ai.github.io/langgraph](https://langchain-ai.github.io/langgraph/) |\n| **Data Storage** | Databricks Unity Catalog (Delta table) | [docs.databricks.com/en/data-governance/unity-catalog](https://docs.databricks.com/en/data-governance/unity-catalog) |\n| **Text-to-SQL** | Databricks Genie | [docs.databricks.com/en/genie](https://docs.databricks.com/en/genie/) |\n| **RAG / Semantic Search** | Databricks Mosaic AI Vector Search | [docs.databricks.com/en/generative-ai/vector-search](https://docs.databricks.com/en/generative-ai/vector-search) |\n| **LLM** | Databricks Model Serving (Llama 3.3 70B) | [docs.databricks.com/en/machine-learning/model-serving](https://docs.databricks.com/en/machine-learning/model-serving) |\n| **Embeddings** | Databricks `gte-large-en` (auto via Vector Search) | [docs.databricks.com/en/generative-ai/vector-search](https://docs.databricks.com/en/generative-ai/vector-search) |\n| **ML Lifecycle / Tracing** | MLflow Tracing | [docs.databricks.com/en/mlflow](https://docs.databricks.com/en/mlflow) |\n| **Data Cleaning** | Databricks Notebooks (Python + Spark SQL) | [docs.databricks.com/en/notebooks](https://docs.databricks.com/en/notebooks) |\n| **Frontend** | Streamlit | [docs.streamlit.io](https://docs.streamlit.io/) |\n| **Map** | Folium + streamlit-folium | [python-visualization.github.io/folium](https://python-visualization.github.io/folium/) |\n| **Geospatial** | Python (Haversine, numpy) — local | — |\n| **Language** | Python 3.13 | — |\n\n### .env.example\n\n```bash\n# === Databricks (primary backend) ===\nDATABRICKS_HOST=https://your-workspace.cloud.databricks.com\nDATABRICKS_TOKEN=dapi... # Personal Access Token\nDATABRICKS_CATALOG=your_catalog # Unity Catalog name\nDATABRICKS_SCHEMA=your_schema # Schema within catalog\nGENIE_SPACE_ID=... # From Genie Space setup\nVECTOR_SEARCH_INDEX=your_catalog.your_schema.ghana_facilities_index\nVECTOR_SEARCH_ENDPOINT=ghana-medical-vs\n\n# === LLM (fallback if Model Serving quota exceeded) ===\nOPENAI_API_KEY=sk-... # Optional fallback\n```\n\n### requirements.txt\n\n```\n# Agent orchestration\nlanggraph==1.0.8\nlangchain==1.2.9\nlangchain-openai==1.1.7\n\n# Databricks SDK\ndatabricks-sdk==0.85.0\ndatabricks-vectorsearch==0.64\nmlflow==3.9.0\n\n# Frontend\nstreamlit==1.54.0\nstreamlit-folium==0.26.1\nfolium==0.20.0\n\n# Data (local utilities)\npandas==3.0.0\nnumpy==2.4.2\npython-dotenv==1.2.1\n\n# Testing\npytest==9.0.2\n```\n\n### Project Structure\n\n```\nHack-Nation/\n├── AGENT.md\n├── README.md\n├── requirements.txt\n├── .env.example\n├── .gitignore\n├── resources/\n│ └── prompts_and_pydantic_models/ # Sponsor-provided extraction pipeline (read-only reference)\n│ ├── organization_extraction.py # Entity classification → facility / NGO / other\n│ ├── facility_and_ngo_fields.py # Structured field schemas (Facility, NGO, BaseOrganization)\n│ ├── free_form.py # FacilityFacts: procedure, equipment, capability\n│ └── medical_specialties.py # Specialty classifier + name-parsing rules\n├── data/\n│ └── ghana_city_coords.json # Static geocoding lookup (local)\n├── src/\n│ ├── app.py # Streamlit entry point\n│ ├── config.py # Load .env, Databricks client init\n│ ├── graph.py # LangGraph state graph definition (supervisor + nodes)\n│ ├── state.py # Pydantic state schema for the agent graph\n│ ├── nodes/\n│ │ ├── supervisor.py # Supervisor node — classifies intent, routes to agents\n│ │ ├── sql_agent.py # SQL Agent node — calls Databricks Genie\n│ │ ├── rag_agent.py # RAG Agent node — calls Databricks Vector Search\n│ │ ├── idp_extraction.py # IDP Extraction node — extracts structured facts from free-form text\n│ │ ├── medical_reasoning.py # Medical Reasoning node — anomaly detection via Model Serving\n│ │ ├── geospatial.py # Geospatial node — Haversine, cold-spots, medical deserts\n│ │ └── synthesis.py # Synthesis node — cross-references structured vs unstructured + citations\n│ ├── tools/\n│ │ ├── genie_tool.py # Databricks Genie SDK wrapper\n│ │ ├── vector_search_tool.py # Databricks Vector Search SDK wrapper\n│ │ └── model_serving_tool.py # Databricks Model Serving SDK wrapper\n│ ├── databricks_clients.py # Shared Databricks client initialization\n│ └── map_component.py # Folium map builder\n├── tests/\n│ ├── test_config.py # Phase 1: Databricks connection smoke test\n│ ├── test_tools.py # Phase 1: Genie, Vector Search, Model Serving smoke tests\n│ ├── test_graph.py # Phase 2: Graph compiles + intent routing\n│ ├── test_nodes.py # Phase 2: Node contract tests (expected state keys)\n│ ├── test_idp_extraction.py # Phase 2: IDP extracts structured facts\n│ ├── test_geospatial.py # Phase 3: Haversine + desert detection math\n│ └── test_e2e.py # Phase 2.5 MVD gate: 5 demo queries\n└── notebooks/\n └── 01_setup_databricks.py # Run once in Databricks: clean → upload → index → Genie\n```\n\n---\n\n## Dataset Analysis (987 Rows)\n\n### File: `Virtue_Foundation_Ghana_v0_3_-_Sheet1.csv`\n\n- **987 rows** total: **920 facilities** + **67 NGOs**\n- Download: https://drive.google.com/file/d/1qgmLHrJYu8TKY2UeQ-VFD4PQ_avPoZ3d/view\n\n### Critical Data Issues (Handle in Notebook)\n\n1. **NO LATITUDE/LONGITUDE** — Geocode from `address_city` using a static lookup table\n2. **Sparse structured fields** — capacity (23 rows), numberDoctors (3 rows), area (2 rows)\n3. **Dirty region names** — 53 variations for 16 regions. Normalize in notebook before upload\n4. **71 duplicate facility names** — Deduplicate by name + city, merge parsed arrays\n5. **Typo: \"farmacy\"** — Fix to \"pharmacy\" (5 records)\n6. **Free-form fields are JSON strings** — Parse with `json.loads()` in notebook\n\n### Column Coverage\n\n| Field | Coverage | Notes |\n| --------------------- | -------- | ----- |\n| name | 100% | |\n| specialties | 92% | |\n| capability | 94% | 784 have >0 items |\n| procedure | 79% | 202 have >0 items |\n| equipment | 73% | 91 have >0 items |\n| address_city | 94% | |\n| address_stateOrRegion | **26%** | Very sparse — infer from city |\n| facilityTypeId | 73% | |\n| capacity | **2%** | Almost empty |\n| numberDoctors | **<1%** | Almost empty |\n\n### Free-Form Text Format\n\n```json\n[\"Performs emergency cesarean sections\", \"Offers hemodialysis treatment 3 times weekly\"]\n[\"Optical Coherence Tomography (OCT) machine\", \"Fundus photography equipment\"]\n[\"Emergency services operate 24/7\", \"Has 87 French and Ghanaian medical professionals\"]\n```\n\n---\n\n## Implementation — Milestone-Based\n\n```mermaid\ngantt\n title Hackathon Sprint\n dateFormat X\n axisFormat %s\n\n section Foundation — Databricks Setup\n Notebook: clean + geocode + upload to UC :p1a, 0, 40\n Create Vector Search index :p1b, after p1a, 20\n Configure Genie Space :p1c, after p1b, 20\n\n section Core — LangGraph + Databricks Wiring\n LangGraph state + graph scaffold :p2a, after p1c, 20\n Supervisor node + intent classification :p2b, after p2a, 20\n SQL Agent node (Genie) :p2c, after p2b, 20\n RAG Agent node (Vector Search) :p2d, after p2c, 20\n Medical Reasoning node (Model Serving) :p2e, after p2d, 20\n Synthesis node + Streamlit wiring :p2f, after p2e, 20\n\n section MVD Checkpoint\n Minimum Viable Demo MUST work :milestone, crit, after p2f, 0\n\n section Surface — Map + Deserts + Planning\n Folium map with facility markers :p3a, after p2f, 20\n Medical desert overlay (25% of score!) :p3b, after p3a, 30\n Planning dashboard — Mission Planner tab :p3c, after p3b, 30\n MLflow tracing + citations :p3d, after p3c, 20\n Error handling + polish :p3e, after p3d, 20\n\n section Stretch\n Agentic-step-level citations (MLflow) :p4a, after p3e, 30\n Composite fan-out queries :p4b, after p4a, 30\n PDF export for planning report :p4c, after p4b, 20\n```\n\n### Foundation — Databricks Setup (Notebook)\n\n**Run `notebooks/01_setup_databricks.py` once in a Databricks notebook.** This does all data work on Databricks — no local data pipeline needed.\n\n#### Step 1: Upload + Clean Data\n\nRef: [Unity Catalog — Managed Tables](https://docs.databricks.com/en/data-governance/unity-catalog/create-tables.html)\n\n```python\nimport pandas as pd\nfrom pyspark.sql import functions as F\n\n# Upload CSV to a Volume, then read\ndf = spark.read.csv(\n \"/Volumes/your_catalog/your_schema/data/ghana_facilities.csv\",\n header=True, inferSchema=True\n)\n\n# Fix typo\ndf = df.withColumn(\"facilityTypeId\",\n F.when(F.col(\"facilityTypeId\") == \"farmacy\", \"pharmacy\")\n .otherwise(F.col(\"facilityTypeId\")))\n\n# Normalize region names\nregion_map = {\n \"Greater Accra Region\": \"Greater Accra\", \"Accra\": \"Greater Accra\",\n \"ASHANTI\": \"Ashanti\", \"Ashanti Region\": \"Ashanti\",\n \"Western Region\": \"Western\", \"Central Region\": \"Central\",\n # ... (full map in AGENT.md data section)\n}\nmapping_expr = F.create_map([F.lit(x) for kv in region_map.items() for x in kv])\ndf = df.withColumn(\"region_normalized\", mapping_expr[F.col(\"address_stateOrRegion\")])\n\n# Save as managed Delta table\ndf.write.format(\"delta\").mode(\"overwrite\").saveAsTable(\"your_catalog.your_schema.ghana_facilities\")\n```\n\n#### Step 2: Add Column Descriptions (Critical for Genie)\n\nRef: [ALTER TABLE — Column Comments](https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-alter-table.html)\n\n```sql\nALTER TABLE your_catalog.your_schema.ghana_facilities\n ALTER COLUMN name COMMENT 'Official name of healthcare facility or NGO';\nALTER TABLE your_catalog.your_schema.ghana_facilities\n ALTER COLUMN procedure COMMENT 'JSON array of clinical procedures — surgeries, diagnostics, screenings';\nALTER TABLE your_catalog.your_schema.ghana_facilities\n ALTER COLUMN equipment COMMENT 'JSON array of medical devices — MRI, CT, X-ray, surgical tools';\nALTER TABLE your_catalog.your_schema.ghana_facilities\n ALTER COLUMN capability COMMENT 'JSON array of care levels — trauma levels, ICU, accreditations, staffing';\nALTER TABLE your_catalog.your_schema.ghana_facilities\n ALTER COLUMN specialties COMMENT 'JSON array of camelCase medical specialties e.g. cardiology, ophthalmology';\nALTER TABLE your_catalog.your_schema.ghana_facilities\n ALTER COLUMN facilityTypeId COMMENT 'Facility type: hospital, clinic, dentist, pharmacy, or doctor';\nALTER TABLE your_catalog.your_schema.ghana_facilities\n ALTER COLUMN address_stateOrRegion COMMENT 'Ghana region (raw, may need normalization)';\nALTER TABLE your_catalog.your_schema.ghana_facilities\n ALTER COLUMN region_normalized COMMENT 'Cleaned Ghana region name (one of 16 official regions)';\nALTER TABLE your_catalog.your_schema.ghana_facilities\n ALTER COLUMN address_city COMMENT 'City or town where the facility is located';\n```\n\n#### Step 3: Create Vector Search Index\n\nRef: [Mosaic AI Vector Search](https://docs.databricks.com/en/generative-ai/vector-search.html)\n\n```python\nfrom databricks.vector_search.client import VectorSearchClient\n\nvsc = VectorSearchClient()\n\n# Create endpoint (one-time)\nvsc.create_endpoint(name=\"ghana-medical-vs\", endpoint_type=\"STANDARD\")\n\n# Create auto-embedding index over free-form text columns\nvsc.create_delta_sync_index(\n endpoint_name=\"ghana-medical-vs\",\n index_name=\"your_catalog.your_schema.ghana_facilities_index\",\n source_table_name=\"your_catalog.your_schema.ghana_facilities\",\n pipeline_type=\"TRIGGERED\",\n primary_key=\"unique_id\",\n embedding_source_columns=[\n {\"name\": \"procedure\", \"model_endpoint_name\": \"databricks-gte-large-en\"},\n {\"name\": \"equipment\", \"model_endpoint_name\": \"databricks-gte-large-en\"},\n {\"name\": \"capability\", \"model_endpoint_name\": \"databricks-gte-large-en\"},\n ],\n columns_to_sync=[\"name\", \"facilityTypeId\", \"address_city\",\n \"region_normalized\", \"specialties\", \"description\"]\n)\n```\n\n#### Step 4: Configure Genie Space\n\nRef: [Databricks Genie](https://docs.databricks.com/en/genie/)\n\n1. Databricks UI → **Genie** → **Create Genie Space**\n2. Add table: `your_catalog.your_schema.ghana_facilities`\n3. Add **custom instructions**:\n ```\n This dataset contains 987 healthcare facilities and NGOs in Ghana.\n The procedure/equipment/capability columns are JSON arrays of English strings.\n The specialties column contains JSON arrays of camelCase strings like \"cardiology\".\n Use LIKE '%keyword%' to search within JSON array columns.\n The region_normalized column has clean Ghana region names.\n ```\n4. Add **example SQL queries**:\n ```sql\n -- How many hospitals have cardiology?\n SELECT COUNT(*) FROM ghana_facilities\n WHERE facilityTypeId = 'hospital' AND specialties LIKE '%cardiology%';\n\n -- Which region has the most hospitals?\n SELECT region_normalized, COUNT(*) as cnt FROM ghana_facilities\n WHERE facilityTypeId = 'hospital'\n GROUP BY region_normalized ORDER BY cnt DESC;\n\n -- Facilities with procedures but no equipment\n SELECT name, procedure, equipment FROM ghana_facilities\n WHERE procedure != '[]' AND (equipment = '[]' OR equipment IS NULL);\n ```\n5. Note the **Genie Space ID** — needed in `.env`\n\n---\n\n### Core — LangGraph Agent Graph + Databricks Wiring\n\nThe local LangGraph graph orchestrates agent nodes. Each node calls a Databricks service via the SDK. MLflow traces every step. Ref: [LangGraph Docs](https://langchain-ai.github.io/langgraph/)\n\n#### `src/config.py` — Databricks Client Setup\n\nRef: [Databricks SDK for Python](https://docs.databricks.com/en/dev-tools/sdk-python.html)\n\n```python\nfrom databricks.sdk import WorkspaceClient\nfrom dotenv import load_dotenv\nimport os\n\nload_dotenv()\n\ndb_client = WorkspaceClient(\n host=os.getenv(\"DATABRICKS_HOST\"),\n token=os.getenv(\"DATABRICKS_TOKEN\")\n)\n\nGENIE_SPACE_ID = os.getenv(\"GENIE_SPACE_ID\")\nVS_INDEX = os.getenv(\"VECTOR_SEARCH_INDEX\")\nVS_ENDPOINT = os.getenv(\"VECTOR_SEARCH_ENDPOINT\")\nCATALOG = os.getenv(\"DATABRICKS_CATALOG\")\nSCHEMA = os.getenv(\"DATABRICKS_SCHEMA\")\n```\n\n#### `src/tools/` — Databricks SDK Wrappers (LangGraph Tools)\n\nEach tool wraps a Databricks service. Agents call these tools within LangGraph nodes.\n\n**`src/tools/genie_tool.py`** — Text-to-SQL via Genie\n```python\nfrom src.config import db_client, GENIE_SPACE_ID\n\ndef query_genie(question: str) -> dict:\n \"\"\"Send natural language to Genie, get SQL + results back.\n Ref: https://docs.databricks.com/en/genie/\"\"\"\n response = db_client.genie.start_conversation(\n space_id=GENIE_SPACE_ID,\n content=question\n )\n return {\"sql\": response.sql, \"results\": response.attachments}\n```\n\n**`src/tools/vector_search_tool.py`** — Semantic Search (RAG)\n```python\nfrom src.config import db_client, VS_INDEX\n\ndef query_vector_search(query_text: str, num_results: int = 10, filters: dict = None) -> list:\n \"\"\"Semantic search over procedure/equipment/capability columns.\n Ref: https://docs.databricks.com/en/generative-ai/vector-search\"\"\"\n results = db_client.vector_search_indexes.query_index(\n index_name=VS_INDEX,\n columns=[\"name\", \"procedure\", \"equipment\", \"capability\",\n \"address_city\", \"facilityTypeId\", \"specialties\"],\n query_text=query_text,\n num_results=num_results,\n filters=filters\n )\n return results.data_array\n```\n\n**`src/tools/model_serving_tool.py`** — LLM Inference\n```python\nfrom src.config import db_client\n\ndef query_llm(system_prompt: str, user_message: str) -> str:\n \"\"\"Call foundation model via Databricks Model Serving.\n Ref: https://docs.databricks.com/en/machine-learning/model-serving\"\"\"\n response = db_client.serving_endpoints.query(\n name=\"databricks-meta-llama-3-3-70b-instruct\",\n messages=[\n {\"role\": \"system\", \"content\": system_prompt},\n {\"role\": \"user\", \"content\": user_message}\n ]\n )\n return response.choices[0].message.content\n```\n\n#### `src/state.py` — LangGraph State Schema\n\n```python\nfrom typing import TypedDict, Literal, Optional\nfrom langgraph.graph import MessagesState\n\nclass AgentState(TypedDict):\n \"\"\"Shared state passed between all LangGraph nodes.\n Ref: https://langchain-ai.github.io/langgraph/concepts/low_level/#state\"\"\"\n query: str # Original user question\n intent: Literal[\"SQL\", \"SEARCH\", \"EXTRACT\", \"ANOMALY\", \"GEO\"] # Classified by supervisor\n sql_result: Optional[dict] # From Genie\n search_result: Optional[list] # From Vector Search\n extraction_result: Optional[dict] # From IDP Extraction (structured facts)\n anomaly_result: Optional[str] # From Model Serving\n geo_result: Optional[dict] # From local geospatial\n final_answer: Optional[str] # Merged by synthesis node\n citations: list # Audit trail for MLflow\n```\n\n#### `src/graph.py` — LangGraph State Graph Definition\n\n```python\nfrom langgraph.graph import StateGraph, END\nfrom src.state import AgentState\nfrom src.nodes.supervisor import supervisor_node\nfrom src.nodes.sql_agent import sql_agent_node\nfrom src.nodes.rag_agent import rag_agent_node\nfrom src.nodes.idp_extraction import idp_extraction_node\nfrom src.nodes.medical_reasoning import medical_reasoning_node\nfrom src.nodes.geospatial import geospatial_node\nfrom src.nodes.synthesis import synthesis_node\nimport mlflow\n\ndef route_by_intent(state: AgentState) -> str:\n \"\"\"Conditional edge — routes to the right agent based on supervisor classification.\n Ref: https://langchain-ai.github.io/langgraph/concepts/low_level/#conditional-edges\"\"\"\n return state[\"intent\"]\n\n# Build the graph\nworkflow = StateGraph(AgentState)\n\n# Add nodes\nworkflow.add_node(\"supervisor\", supervisor_node)\nworkflow.add_node(\"SQL\", sql_agent_node)\nworkflow.add_node(\"SEARCH\", rag_agent_node)\nworkflow.add_node(\"EXTRACT\", idp_extraction_node)\nworkflow.add_node(\"ANOMALY\", medical_reasoning_node)\nworkflow.add_node(\"GEO\", geospatial_node)\nworkflow.add_node(\"synthesis\", synthesis_node)\n\n# Edges: supervisor classifies intent → route to correct agent → merge in synthesis\nworkflow.set_entry_point(\"supervisor\")\nworkflow.add_conditional_edges(\"supervisor\", route_by_intent,\n {\"SQL\": \"SQL\", \"SEARCH\": \"SEARCH\", \"EXTRACT\": \"EXTRACT\", \"ANOMALY\": \"ANOMALY\", \"GEO\": \"GEO\"})\nworkflow.add_edge(\"SQL\", \"synthesis\")\nworkflow.add_edge(\"SEARCH\", \"synthesis\")\nworkflow.add_edge(\"EXTRACT\", \"synthesis\")\nworkflow.add_edge(\"ANOMALY\", \"synthesis\")\nworkflow.add_edge(\"GEO\", \"synthesis\")\nworkflow.add_edge(\"synthesis\", END)\n\n# Compile\ngraph = workflow.compile()\n\n@mlflow.trace\ndef run_agent(query: str) -> str:\n \"\"\"Run the full agent graph. MLflow traces every step.\n Ref: https://docs.databricks.com/en/mlflow/llm-tracing\"\"\"\n result = graph.invoke({\"query\": query, \"citations\": []})\n return result[\"final_answer\"]\n```\n\n#### `src/nodes/supervisor.py` — Intent Classification Node\n\n```python\nfrom src.tools.model_serving_tool import query_llm\nfrom src.state import AgentState\n\nROUTER_PROMPT = \"\"\"Classify this healthcare data question into ONE category:\n- SQL: counts, aggregations, rankings, comparisons (e.g. \"How many hospitals have cardiology?\")\n- SEARCH: specific facility services, capabilities, equipment (e.g. \"What does Korle Bu offer?\")\n- EXTRACT: extract structured facts from unstructured text for a facility (e.g. \"What procedures does Tamale Teaching Hospital perform?\" or \"Parse capabilities for facilities in Ashanti\")\n- ANOMALY: data inconsistencies, mismatches (e.g. \"Facilities claiming surgery but lacking equipment?\")\n- GEO: distances, locations, medical deserts, coverage gaps (e.g. \"Hospitals within 50km of Tamale?\" or \"Where are ophthalmology deserts?\")\nRespond with ONLY the category name.\"\"\"\n\ndef supervisor_node(state: AgentState) -> dict:\n \"\"\"Supervisor node — classifies user intent using Databricks Model Serving LLM.\n Returns the intent label which drives conditional routing in the graph.\"\"\"\n intent = query_llm(ROUTER_PROMPT, state[\"query\"]).strip().upper()\n if intent not in (\"SQL\", \"SEARCH\", \"EXTRACT\", \"ANOMALY\", \"GEO\"):\n intent = \"SQL\" # Default to Genie for unrecognized intents\n return {\"intent\": intent}\n```\n\n#### `src/nodes/sql_agent.py` — Genie Text-to-SQL Node\n\n```python\nfrom src.tools.genie_tool import query_genie\nfrom src.state import AgentState\n\ndef sql_agent_node(state: AgentState) -> dict:\n \"\"\"SQL Agent — sends the user question to Databricks Genie for Text-to-SQL.\n Genie generates SQL, executes it, and returns structured results.\"\"\"\n result = query_genie(state[\"query\"])\n return {\"sql_result\": result, \"citations\": state[\"citations\"] + [{\"source\": \"genie\", \"sql\": result.get(\"sql\")}]}\n```\n\n#### `src/nodes/rag_agent.py` — Vector Search RAG Node\n\n```python\nfrom src.tools.vector_search_tool import query_vector_search\nfrom src.state import AgentState\n\ndef rag_agent_node(state: AgentState) -> dict:\n \"\"\"RAG Agent — performs semantic search over facility free-form text using\n Databricks Mosaic AI Vector Search. Returns top-k matching facilities.\"\"\"\n results = query_vector_search(state[\"query\"])\n return {\"search_result\": results, \"citations\": state[\"citations\"] + [{\"source\": \"vector_search\", \"hits\": len(results)}]}\n```\n\n#### `src/nodes/idp_extraction.py` — IDP Extraction Node (Core Feature #1)\n\n**This is the heart of the IDP challenge (30% of scoring).** It goes beyond keyword search — it extracts structured medical facts from raw free-form text using the LLM, guided by the sponsor's extraction prompt from `resources/prompts_and_pydantic_models/free_form.py`.\n\n```python\nimport json\nfrom src.tools.vector_search_tool import query_vector_search\nfrom src.tools.model_serving_tool import query_llm\nfrom src.state import AgentState\n\nIDP_EXTRACTION_PROMPT = \"\"\"You are a specialized medical facility information extractor.\n\nGiven a facility's raw free-form text data (procedure, equipment, capability arrays),\nextract and return STRUCTURED facts in this JSON format:\n\n{\n \"facility_name\": \"...\",\n \"parsed_procedures\": [\"Performs emergency cesarean sections\", ...],\n \"parsed_equipment\": [\"Has Siemens CT scanner\", ...],\n \"parsed_capabilities\": [\"Level II trauma center\", \"24/7 emergency care\", ...],\n \"inferred_specialties\": [\"generalSurgery\", \"emergencyMedicine\", ...],\n \"facility_level\": \"hospital|clinic|specialist_center\",\n \"confidence_flags\": [\"procedure X claimed but no supporting equipment listed\", ...]\n}\n\nRULES:\n- Extract ONLY facts directly stated in the data. Do not infer from general knowledge.\n- Map capabilities to standard specialty names (camelCase): cardiology, ophthalmology, etc.\n- Flag any contradictions between procedure claims and equipment lists.\n- Each fact must be a clear, declarative English statement.\n- Empty arrays [] mean \"no data found\" — this is a valid signal, not missing data.\"\"\"\n\ndef idp_extraction_node(state: AgentState) -> dict:\n \"\"\"IDP Extraction — retrieves facility free-form text via Vector Search, then\n uses the LLM to extract structured facts from unstructured procedure/equipment/\n capability arrays. This is the core IDP innovation the challenge evaluates.\"\"\"\n # Step 1: Retrieve relevant facilities via semantic search\n raw_facilities = query_vector_search(state[\"query\"], num_results=5)\n\n # Step 2: For each facility, extract structured facts via LLM\n extractions = []\n for facility in raw_facilities:\n extraction = query_llm(IDP_EXTRACTION_PROMPT, json.dumps(facility))\n extractions.append(extraction)\n\n return {\n \"extraction_result\": {\"query\": state[\"query\"], \"extractions\": extractions},\n \"citations\": state[\"citations\"] + [\n {\"source\": \"idp_extraction\", \"facilities_processed\": len(raw_facilities)}\n ]\n }\n```\n\n#### `src/nodes/synthesis.py` — Intelligent Synthesis Node (Core Feature #2)\n\n**This is \"Intelligent Synthesis\" (combines unstructured insights with structured schemas).** The node cross-references structured fields (facilityTypeId, specialties) against extracted free-form facts to find contradictions, confirm capabilities, and build a complete picture.\n\n```python\nfrom src.tools.model_serving_tool import query_llm\nfrom src.state import AgentState\n\nSYNTHESIS_PROMPT = \"\"\"You are a medical data synthesis expert for Ghana healthcare facilities.\n\nYour job is to produce a clear, citation-backed answer by CROSS-REFERENCING structured and unstructured data.\n\nCROSS-REFERENCING RULES:\n1. Compare structured field (facilityTypeId, specialties) against free-form text (procedure, equipment, capability).\n - Flag if facilityTypeId=\"clinic\" but capabilities describe hospital-level services (trauma, ICU).\n - Flag if specialties list \"ophthalmology\" but no eye-related procedures or equipment found.\n - Confirm when structured and unstructured data agree (higher confidence answer).\n2. Note data completeness: if a facility has procedures but zero equipment, say so explicitly.\n3. Every claim MUST cite the specific facility name, field, and value that supports it.\n\nOUTPUT FORMAT (Markdown):\n### Answer\n[Direct answer to the user's question]\n\n### Supporting Evidence\n| Facility | Field | Value | Confidence |\n|---|---|---|---|\n| [name] | [field] | [value] | High/Medium/Low |\n\n### Data Quality Notes\n[Any contradictions, gaps, or flags discovered during cross-referencing]\n\"\"\"\n\ndef synthesis_node(state: AgentState) -> dict:\n \"\"\"Synthesis node — cross-references structured fields against unstructured\n free-form text extractions. Produces citation-backed answers with data quality flags.\n This implements 'Intelligent Synthesis' (Core Feature #2).\"\"\"\n # Gather all results from whichever agent(s) ran\n raw_parts = []\n if state.get(\"sql_result\"):\n raw_parts.append(f\"SQL/Genie result:\\n{state['sql_result']}\")\n if state.get(\"search_result\"):\n raw_parts.append(f\"Vector Search result:\\n{state['search_result']}\")\n if state.get(\"extraction_result\"):\n raw_parts.append(f\"IDP Extraction result:\\n{state['extraction_result']}\")\n if state.get(\"anomaly_result\"):\n raw_parts.append(f\"Anomaly detection result:\\n{state['anomaly_result']}\")\n if state.get(\"geo_result\"):\n raw_parts.append(f\"Geospatial result:\\n{state['geo_result']}\")\n\n combined = \"\\n---\\n\".join(raw_parts) if raw_parts else \"No results found.\"\n answer = query_llm(SYNTHESIS_PROMPT, combined)\n return {\"final_answer\": answer}\n```\n\n#### `src/medical_reasoning.py` — Anomaly Detection Prompt\n\n```python\nMEDICAL_REASONING_PROMPT = \"\"\"You are a medical facility verification expert for Ghana.\n\nYou detect anomalies by cross-referencing a facility's claimed procedures, equipment, and capabilities.\n\nPROCEDURE-EQUIPMENT DEPENDENCIES (flag if procedure claimed without required equipment):\n- Cataract surgery → requires: operating microscope, phacoemulsification unit\n- MRI diagnostics → requires: MRI scanner\n- CT scan → requires: CT scanner\n- Hemodialysis → requires: dialysis machines\n- Cesarean section → requires: operating theater, anesthesia equipment\n- Laparoscopic surgery → requires: laparoscope, insufflator\n- X-ray → requires: X-ray machine\n- Ultrasound → requires: ultrasound machine\n- ICU care → requires: ventilators, cardiac monitors\n\nANOMALY PATTERNS:\n1. PROCEDURE-EQUIPMENT GAP: Claims surgical procedures but lists zero surgical equipment\n2. SPECIALTY-PROCEDURE MISMATCH: Lists specialty but no related procedures\n3. BREADTH WITHOUT DEPTH: Many specialties (>5) but zero procedures and zero equipment\n4. CAPABILITY INFLATION: Broad claims (\"world-class\") with no supporting data\n5. MISSING BASICS: Hospital type but no emergency or inpatient capability\n\nFor each facility, return:\n- VERDICT: CLEAN, WARNING, or FLAG\n- REASON: Specific explanation\n- EVIDENCE: Which data fields support your conclusion\n- FACILITY: Name and city\"\"\"\n```\n\n#### `src/app.py` — Streamlit Frontend (calls LangGraph agent)\n\n```python\nimport streamlit as st\nfrom src.graph import run_agent\nfrom src.map_component import create_ghana_map\n\nst.set_page_config(page_title=\"Ghana Medical Intelligence Agent\", layout=\"wide\")\nst.title(\"Ghana Medical Intelligence Agent\")\nst.caption(\"Bridging Medical Deserts — Powered by LangGraph + Databricks + MLflow\")\n\nwith st.sidebar:\n st.header(\"Example Queries\")\n for ex in [\n \"How many hospitals have cardiology?\",\n \"What services does Korle Bu Teaching Hospital offer?\",\n \"Which facilities claim surgery but lack equipment?\",\n \"Show medical deserts for ophthalmology\",\n \"Where should the next mission go?\",\n ]:\n if st.button(ex, key=ex):\n st.session_state.query = ex\n\ncol_chat, col_map = st.columns([1, 1])\n\nwith col_chat:\n query = st.text_input(\"Ask about healthcare facilities in Ghana:\",\n value=st.session_state.get(\"query\", \"\"))\n if query:\n with st.spinner(\"Running agent graph...\"):\n try:\n answer = run_agent(query)\n st.markdown(answer)\n except Exception as e:\n st.error(f\"Something went wrong: {e}\")\n\nwith col_map:\n st.subheader(\"Facility Map\")\n # Render Folium map (see map_component.py)\n```\n\n---\n\n### MVD Checkpoint (Minimum Viable Demo)\n\n**Before moving to Surface, this must work:**\n\n1. Databricks workspace has Delta table + Vector Search index + Genie Space\n2. LangGraph graph compiles and runs end-to-end: supervisor → agent node → synthesis\n3. Streamlit app calls `run_agent()` and returns answers for at least 3 of the 5 demo queries\n4. Basic map renders with facility markers\n\n**If this doesn't work, stop and debug the LangGraph graph + Databricks connection. Everything else is polish.**\n\n---\n\n### Surface — Map + Medical Deserts + Planning + MLflow\n\n#### Folium Map (with medical desert overlay)\n\n- Color-coded markers: hospital=blue, clinic=green, pharmacy=orange, dentist=purple, doctor=gray\n- **Medical desert overlay** (this directly addresses the challenge title \"Bridging Medical Deserts\" — 25% of scoring):\n - Pre-compute at startup: for each key specialty (generalSurgery, emergencyMedicine, ophthalmology, cardiology, pediatrics), find regions where no facility offers it\n - Overlay translucent red circles on the map for desert zones\n - User can toggle specialties via sidebar dropdown\n - Data source: Genie query `SELECT region_normalized, specialties FROM ghana_facilities` → local Python finds gaps\n\n#### Planning System — Core Feature #3\n\n> Challenge: *\"Think creatively how you could include a planning system which is easily accessible and could get adopted across experience levels and age groups.\"*\n\n**Design: A \"Mission Planner\" tab in the Streamlit app** — not a separate tool, integrated right next to the chat and map. Accessible to everyone from a field coordinator to an executive director.\n\n```\n┌─────────────────────────────────────────────────────────────┐\n│ Tab: 💬 Ask Agent │ Tab: 📋 Mission Planner │ Tab: 🗺️ Map │\n├─────────────────────────────────────────────────────────────┤\n│ │\n│ ┌─── Summary Cards (auto-populated at startup) ─────────┐ │\n│ │ 987 Facilities │ 67 NGOs │ 12 Flagged │ 4 Deserts│ │\n│ └───────────────────────────────────────────────────────┘ │\n│ │\n│ ┌─── Where to Deploy Next ──────────────────────────────┐ │\n│ │ Select specialty: [ ophthalmology ▼ ] │ │\n│ │ │ │\n│ │ 🔴 PRIORITY 1: Upper East Region │ │\n│ │ 0 facilities with ophthalmology │ │\n│ │ Nearest: Tamale Eye Center (142 km away) │ │\n│ │ Population at risk: ~1.2M │ │\n│ │ │ │\n│ │ 🟡 PRIORITY 2: Savannah Region │ │\n│ │ 1 clinic (no surgical capability) │ │\n│ │ Nearest hospital-grade: Kumasi (287 km) │ │\n│ │ │ │\n│ │ ✅ COVERED: Greater Accra (8 facilities) │ │\n│ └────────────────────────────────────────────────────────┘ │\n│ │\n│ ┌─── Flagged Facilities (need verification) ────────────┐ │\n│ │ ⚠️ Holy Family Hospital, Techiman │ │\n│ │ Claims: laparoscopic surgery │ │\n│ │ Missing: laparoscope, insufflator │ │\n│ │ Verdict: PROCEDURE-EQUIPMENT GAP │ │\n│ └────────────────────────────────────────────────────────┘ │\n│ │\n│ [ 📥 Export Report as PDF ] │\n└─────────────────────────────────────────────────────────────┘\n```\n\n**How it works:**\n1. **Summary Cards** — On app startup, run 4 Genie queries: `COUNT(*)` facilities, `COUNT(*)` NGOs, count flagged anomalies, count desert regions. Cache results in `st.session_state`.\n2. **\"Where to Deploy Next\"** — User selects a specialty from a dropdown. The app runs: (a) Genie query to find regions with zero or few facilities for that specialty, (b) local geospatial to compute distances to nearest alternative, (c) LLM to rank priorities and generate plain-English recommendations.\n3. **Flagged Facilities** — Pre-computed by running the Medical Reasoning node over all facilities at startup (or top 50 by anomaly score). Shows VERDICT + REASON + EVIDENCE.\n4. **Export** — `st.download_button` to export the planning report as a PDF or CSV.\n\n**Why this design is accessible:**\n- No query syntax required — dropdown-driven, point-and-click\n- Color-coded priorities (red/yellow/green) — intuitive for any age\n- Plain English recommendations — no medical jargon\n- Works on tablet/phone (Streamlit responsive layout)\n\n#### MLflow Tracing\n\n- MLflow `@mlflow.trace` decorators on each agent node for row-level citation trail\n- Basic `try/except` error handling on all Databricks SDK calls\n\n### Stretch Goals — Clear Path Forward\n\n| Stretch Goal | Technology | How to Add | Docs |\n|---|---|---|---|\n| **Agentic-step citations** | MLflow Tracing | Add `@mlflow.trace` to each node + log inputs/outputs per reasoning call (challenge explicitly asks for this) | [docs.databricks.com/en/mlflow/llm-tracing](https://docs.databricks.com/en/mlflow/llm-tracing/) |\n| **Composite queries** | LangGraph fan-out | Supervisor routes to multiple agent nodes in parallel (SQL + RAG), synthesis merges both | [langchain-ai.github.io/langgraph](https://langchain-ai.github.io/langgraph/) |\n| **Mosaic AI Agent deploy** | Mosaic AI Agent Framework | Wrap the LangGraph graph as a `ChatAgent`, deploy on Databricks serving endpoint | [docs.databricks.com/en/generative-ai/agent-framework](https://docs.databricks.com/en/generative-ai/agent-framework/author-agent.html) |\n| **LangGraph Studio** | LangGraph Platform | Visual debugger for the agent graph — step through nodes, inspect state | [langchain-ai.github.io/langgraph/concepts/langgraph_studio](https://langchain-ai.github.io/langgraph/concepts/langgraph_studio/) |\n| **PDF export** | FPDF / WeasyPrint | Export planning report to PDF from Streamlit | — |\n\n---\n\n## Must-Have Queries (Official VF Question Bank)\n\n| # | Question | Agent Node → Databricks Tool | Priority |\n| --- | ----------------------------------------------------------- | --- | --------- |\n| 1.1 | How many hospitals have cardiology? | SQL Agent → Genie | Must Have |\n| 1.2 | How many hospitals in [region] can perform [procedure]? | SQL Agent → Genie | Must Have |\n| 1.3 | What services does [Facility Name] offer? | RAG Agent → Vector Search | Must Have |\n| 1.4 | Are there clinics in [Area] that do [Service]? | RAG Agent → Vector Search | Must Have |\n| 1.5 | Which region has the most [Type] hospitals? | SQL Agent → Genie | Must Have |\n| 2.1 | Hospitals treating [condition] within [X] km of [location]? | Geo Agent (local) + SQL Agent → Genie | Must Have |\n| 2.3 | Largest geographic cold spots for [procedure]? | Geo Agent (local) + SQL Agent → Genie | Must Have |\n| 4.4 | Facilities claiming unrealistic procedures for their size? | Medical Reasoning → Model Serving | Must Have |\n| 4.7 | Correlations between facility characteristics? | SQL Agent → Genie | Must Have |\n| 4.8 | High procedure breadth with minimal equipment? | Medical Reasoning → Model Serving | Must Have |\n| 4.9 | Things that shouldn't move together? | Medical Reasoning → Model Serving | Must Have |\n| 6.1 | Where is the workforce for [subspecialty] practicing? | SQL Agent → Genie + Synthesis LLM | Must Have |\n| 7.5 | Which procedures depend on very few facilities? | SQL Agent → Genie | Must Have |\n| 7.6 | Oversupply vs scarcity by complexity? | SQL Agent → Genie + Synthesis LLM | Must Have |\n| 8.3 | Gaps where no organizations work despite evident need? | Geo + SQL + Medical Reasoning (fan-out) | Must Have |\n\n---\n\n## Schema & Pydantic Models (From Dataset Creation Pipeline)\n\n> These are the **exact** models used by the Virtue Foundation to extract the dataset. Understanding them is critical for interpreting the data.\n\n### Organization Types\n\n```python\nclass OrganizationExtractionOutput(BaseModel):\n ngos: Optional[List[str]] # Non-profits delivering healthcare in low-income settings\n facilities: Optional[List[str]] # Physical sites delivering in-person medical care\n other_organizations: Optional[List[str]] # Entities that don't qualify as facility or NGO\n```\n\n### Facility Fields\n\n```python\nclass Facility(BaseModel):\n name: str # Official name\n facilityTypeId: Literal[\"hospital\", \"pharmacy\", \"doctor\", \"clinic\", \"dentist\"]\n operatorTypeId: Literal[\"public\", \"private\"]\n affiliationTypeIds: List[Literal[\"faith-tradition\", \"philanthropy-legacy\",\n \"community\", \"academic\", \"government\"]]\n description: str # Brief paragraph on services/history\n area: int # Floor area sqm (VERY SPARSE — 2 rows)\n numberDoctors: int # Doctor count (VERY SPARSE — 3 rows)\n capacity: int # Bed capacity (VERY SPARSE — 23 rows)\n # Address fields: address_city, address_stateOrRegion, address_country, etc.\n # Contact fields: phone_numbers, email, websites, officialWebsite, etc.\n```\n\n### Free-Form Text Columns (Core of IDP Challenge)\n\n```python\nclass FacilityFacts(BaseModel):\n procedure: List[str] # Clinical services — surgeries, diagnostics, screenings\n equipment: List[str] # Medical devices — MRI, CT, X-ray, surgical tools, utilities\n capability: List[str] # Care levels — trauma levels, ICU, accreditations, staffing, capacity\n```\n\n**Extraction rules:** Facts are declarative English statements traceable to source content. No generic knowledge. Empty arrays `[]` = no data found (valid signal, not missing data).\n\n### Medical Specialties (case-sensitive camelCase)\n\n```\ninternalMedicine, familyMedicine, pediatrics, cardiology, generalSurgery,\nemergencyMedicine, gynecologyAndObstetrics, orthopedicSurgery, dentistry,\nophthalmology, otolaryngology, radiology, pathology, anesthesia,\ncriticalCareMedicine, nephrology, medicalOncology, infectiousDiseases,\nphysicalMedicineAndRehabilitation, hospiceAndPalliativeInternalMedicine,\nneonatologyPerinatalMedicine, endocrinologyAndDiabetesAndMetabolism,\nplasticSurgery, cardiacSurgery, geriatricsInternalMedicine, orthodontics\n```\n\n| Term in data | Maps to |\n|---|---|\n| \"Hospital\" (generic) | `internalMedicine` |\n| \"Clinic\" (generic) | `familyMedicine` |\n| \"Emergency\", \"ER\" | `emergencyMedicine` |\n| \"Surgery\" (generic) | `generalSurgery` |\n| \"Eye\", \"Ophthalmic\" | `ophthalmology` |\n| \"Cardiac Surgery\" | `cardiacSurgery` |\n| \"Pediatric\", \"Children\" | `pediatrics` |\n| \"Maternity\", \"Obstetric\" | `gynecologyAndObstetrics` |\n| \"Trauma\" | `criticalCareMedicine` |\n| \"Oncology\", \"Cancer\" | `medicalOncology` |\n\n---\n\n## Anti-Patterns\n\n1. **Don't rely on capacity/numberDoctors** — only 23 and 3 rows have these. Use procedure-equipment cross-referencing.\n2. **Don't skip geocoding** — No lat/lon in data. City lookup table is required for the map.\n3. **Don't treat `[]` as missing** — Empty array = \"nothing found\", which is useful signal.\n4. **Don't ignore duplicates** — 71 names appear >1 time. Deduplicate in the notebook.\n5. **Don't hardcode answers** — System must generalize to new queries.\n6. **Don't build a custom SQL agent** — Use Genie. It already understands your column descriptions.\n7. **Don't over-engineer LangGraph** — Start with linear routing (supervisor → one agent → synthesis). Add fan-out later as a stretch goal.\n\n---\n\n## Demo Script (5 Minutes)\n\n| Time | Action | Shows | Eval Criteria Hit |\n| --------- | ------------------------------------------------------------- | ----- | --- |\n| 0:00–0:45 | Problem statement + architecture (LangGraph + Databricks) | Vision, social impact | Social Impact (25%) |\n| 0:45–1:30 | Query: \"Extract capabilities for Korle Bu Teaching Hospital\" | **IDP Extraction** — structured facts from free-form text | **IDP Innovation (30%)** |\n| 1:30–2:30 | Query: \"Which facilities claim surgery but lack equipment?\" | **Anomaly detection** — cross-references procedures vs equipment | Technical Accuracy (35%) |\n| 2:30–3:30 | Show **Mission Planner tab**: select ophthalmology → see desert regions + deployment priorities | **Planning System** — dropdown-driven, color-coded | Social Impact (25%) + UX (10%) |\n| 3:30–4:15 | Show **map**: red desert overlays + facility markers | **Medical desert visualization** | Social Impact (25%) |\n| 4:15–5:00 | Show **MLflow trace**: citation trail for the last query | **Citations** — step-level audit | Technical Accuracy (35%) |\n\n**Closing line**: \"Every data point we extract represents a patient who could receive care sooner.\"\n\n---\n\n## Resources\n\n### Challenge\n- **Full Challenge Brief**: `resources/CHALLENGE.md` (committed to this repo)\n- **Prompts & Pydantic Models** (local): `resources/prompts_and_pydantic_models/` (sponsor-provided extraction pipeline)\n- **Prompts & Pydantic Models** (zip): https://drive.google.com/file/d/1CvMTA2DtwZxa9-sBsw57idCkIlnrN32r/view\n- **Dataset CSV**: https://drive.google.com/file/d/1qgmLHrJYu8TKY2UeQ-VFD4PQ_avPoZ3d/view\n- **Schema Docs**: https://docs.google.com/document/d/1UDkH0WLmm3ppE3OpzSuZQC9_7w3HO1PupDLFVqzS_2g/edit?tab=t.0#heading=h.efyjxgdkfw8u\n- **VF Agent Questions**: https://docs.google.com/document/d/1ETRk0KEcWUJExuhWKBQkw1Tq-D63Bdma1rPAwoaPiRI/edit?tab=t.0#heading=h.dp75tt7nodhp\n- **VFMatch Globe** (map inspiration): https://vfmatch.org/explore?appMode=globe&viewState=8.261875521286015%2C28.8340078062746%2C1.4222445256432446\n- **Databricks x VF Blog**: https://www.databricks.com/blog/elevating-global-health-databricks-and-virtue-foundation\n- **Real-world context**: Databricks team is collaborating with VF to ship an agent by **June 7th, 2025**\n\n### Databricks Documentation (Free Edition)\n- **Free Edition Signup**: https://signup.databricks.com\n- **Free Edition Overview**: https://docs.databricks.com/en/getting-started/free-edition.html\n- **Free Edition Limitations**: https://docs.databricks.com/en/getting-started/free-edition-limitations.html\n- **Unity Catalog**: https://docs.databricks.com/en/data-governance/unity-catalog/index.html\n- **Genie (Text-to-SQL)**: https://docs.databricks.com/en/genie/index.html\n- **Mosaic AI Vector Search**: https://docs.databricks.com/en/generative-ai/vector-search.html\n- **Model Serving**: https://docs.databricks.com/en/machine-learning/model-serving/index.html\n- **MLflow Tracing**: https://docs.databricks.com/en/mlflow/index.html\n- **Agent Framework**: https://docs.databricks.com/en/generative-ai/agent-framework/author-agent.html\n- **Databricks SDK for Python**: https://docs.databricks.com/en/dev-tools/sdk-python.html\n- **Notebooks**: https://docs.databricks.com/en/notebooks/index.html\n- **Personal Access Tokens**: https://docs.databricks.com/en/dev-tools/auth/pat.html\n\n### Agent Orchestration\n- **LangGraph Docs**: https://langchain-ai.github.io/langgraph/\n- **LangGraph Concepts — State Graphs**: https://langchain-ai.github.io/langgraph/concepts/low_level/\n- **LangGraph Conditional Edges**: https://langchain-ai.github.io/langgraph/concepts/low_level/#conditional-edges\n- **LangGraph + MLflow**: https://docs.databricks.com/en/mlflow/llm-tracing/langgraph-tracing.html\n\n### Frontend\n- **Streamlit Docs**: https://docs.streamlit.io/\n- **Folium Docs**: https://python-visualization.github.io/folium/\n\n---\n\n## Risks\n\n- **Daily compute quotas** — Do data upload + index creation EARLY. If exceeded, compute pauses until next day\n- **No GPU access** — Fine-tuning not feasible. Use pre-built foundation models via Model Serving\n- **Network latency** — Databricks API calls add ~200-500ms. Keep geospatial math local\n- **Genie accuracy** — May generate wrong SQL. Add column descriptions and example queries to improve\n- **LangGraph complexity** — Keep the graph simple (linear routing MVP). Fan-out is stretch only\n- **Fallback** — If Databricks is down, swap `src/tools/` wrappers to use OpenAI + local DuckDB\n\n---\n\n## Definition of Done — Phase by Phase\n\n> Tests are only added where they catch real bugs and save debugging time. No test for the sake of testing.\n\n### Project Structure for Tests\n\n```\ntests/\n├── test_config.py # Phase 1: Databricks connection\n├── test_tools.py # Phase 1: SDK wrapper sanity checks\n├── test_graph.py # Phase 2: LangGraph compiles + routes correctly\n├── test_nodes.py # Phase 2: Each node returns expected state keys\n├── test_idp_extraction.py # Phase 2: IDP extracts structured facts from sample text\n├── test_geospatial.py # Phase 3: Haversine + desert detection math\n└── test_e2e.py # Phase 3: 5 demo queries run end-to-end\n```\n\n---\n\n### Phase 1 — Foundation (Databricks Setup)\n\n**Done when:**\n- [ ] CSV uploaded to Databricks Volume\n- [ ] Delta table created in Unity Catalog with column descriptions\n- [ ] `farmacy` → `pharmacy` typo fixed (5 records)\n- [ ] Region names normalized (53 variations → 16)\n- [ ] Duplicates deduplicated by name + city\n- [ ] Free-form JSON strings parsed\n- [ ] Vector Search endpoint created + index synced\n- [ ] Genie Space configured with instructions + example SQL\n- [ ] `src/config.py` connects to workspace without errors\n- [ ] `.env` populated with all IDs (catalog, schema, genie_space_id, vs_index)\n\n**Tests (run locally, verify Databricks connectivity):**\n```python\n# tests/test_config.py\ndef test_databricks_connection():\n \"\"\"Verify .env is loaded and SDK client can authenticate.\"\"\"\n from src.config import db_client\n # Should not raise — proves token + host are valid\n clusters = db_client.clusters.list()\n assert clusters is not None\n\n# tests/test_tools.py\ndef test_genie_returns_response():\n \"\"\"Smoke test: Genie responds to a simple count query.\"\"\"\n from src.tools.genie_tool import query_genie\n result = query_genie(\"How many rows are in the table?\")\n assert \"results\" in result or \"sql\" in result\n\ndef test_vector_search_returns_results():\n \"\"\"Smoke test: Vector Search returns non-empty for a known term.\"\"\"\n from src.tools.vector_search_tool import query_vector_search\n results = query_vector_search(\"cardiology\")\n assert len(results) > 0\n\ndef test_model_serving_returns_text():\n \"\"\"Smoke test: Model Serving LLM returns a non-empty string.\"\"\"\n from src.tools.model_serving_tool import query_llm\n answer = query_llm(\"You are a test.\", \"Say hello.\")\n assert len(answer) > 0\n```\n\n**Why tests here:** These are the most fragile part — tokens expire, endpoints go down, index names get typos. A 30-second smoke test saves 30 minutes of debugging.\n\n---\n\n### Phase 2 — Core (LangGraph + Agent Nodes)\n\n**Done when:**\n- [ ] `AgentState` schema defined with all fields\n- [ ] LangGraph `StateGraph` compiles without errors\n- [ ] Supervisor node classifies intent correctly for at least 4/5 demo queries\n- [ ] SQL Agent node calls Genie and returns `sql_result`\n- [ ] RAG Agent node calls Vector Search and returns `search_result`\n- [ ] IDP Extraction node extracts structured facts and returns `extraction_result`\n- [ ] Medical Reasoning node detects anomalies and returns `anomaly_result`\n- [ ] Synthesis node cross-references structured vs unstructured and returns `final_answer` with citations\n- [ ] `run_agent(query)` runs end-to-end and returns a markdown string\n- [ ] Streamlit app renders the answer\n\n**Tests (verify graph logic + node contracts):**\n```python\n# tests/test_graph.py\ndef test_graph_compiles():\n \"\"\"The LangGraph state graph must compile without errors.\"\"\"\n from src.graph import graph\n assert graph is not None\n\ndef test_route_sql_intent():\n \"\"\"Supervisor routes count queries to SQL agent.\"\"\"\n from src.nodes.supervisor import supervisor_node\n result = supervisor_node({\"query\": \"How many hospitals have cardiology?\", \"citations\": []})\n assert result[\"intent\"] == \"SQL\"\n\ndef test_route_search_intent():\n \"\"\"Supervisor routes facility lookup queries to SEARCH agent.\"\"\"\n from src.nodes.supervisor import supervisor_node\n result = supervisor_node({\"query\": \"What services does Korle Bu offer?\", \"citations\": []})\n assert result[\"intent\"] == \"SEARCH\"\n\ndef test_route_anomaly_intent():\n \"\"\"Supervisor routes anomaly queries to ANOMALY agent.\"\"\"\n from src.nodes.supervisor import supervisor_node\n result = supervisor_node({\"query\": \"Facilities claiming surgery but lacking equipment?\", \"citations\": []})\n assert result[\"intent\"] == \"ANOMALY\"\n\n# tests/test_nodes.py\ndef test_sql_node_returns_expected_keys():\n \"\"\"SQL Agent must return sql_result and updated citations.\"\"\"\n from src.nodes.sql_agent import sql_agent_node\n state = {\"query\": \"How many hospitals?\", \"citations\": []}\n result = sql_agent_node(state)\n assert \"sql_result\" in result\n assert \"citations\" in result\n assert len(result[\"citations\"]) > 0\n\ndef test_rag_node_returns_expected_keys():\n \"\"\"RAG Agent must return search_result and updated citations.\"\"\"\n from src.nodes.rag_agent import rag_agent_node\n state = {\"query\": \"cardiology services\", \"citations\": []}\n result = rag_agent_node(state)\n assert \"search_result\" in result\n assert len(result[\"search_result\"]) > 0\n\n# tests/test_idp_extraction.py\ndef test_idp_extraction_returns_structured_facts():\n \"\"\"IDP Extraction must return extraction_result with parsed data.\n This is the CORE of the challenge (IDP Innovation = 30%).\"\"\"\n from src.nodes.idp_extraction import idp_extraction_node\n state = {\"query\": \"Extract capabilities for Korle Bu Teaching Hospital\", \"citations\": []}\n result = idp_extraction_node(state)\n assert \"extraction_result\" in result\n assert \"extractions\" in result[\"extraction_result\"]\n assert len(result[\"extraction_result\"][\"extractions\"]) > 0\n```\n\n**Why tests here:** The graph is pure logic — if routing is wrong, every downstream answer is wrong. Node contract tests (expected keys) catch integration bugs early. The IDP test validates the highest-scored feature (30%).\n\n---\n\n### Phase 2.5 — MVD Checkpoint\n\n**Gate — do NOT proceed to Phase 3 unless ALL pass:**\n\n```python\n# tests/test_e2e.py (run manually — not CI, these hit Databricks)\nDEMO_QUERIES = [\n \"How many hospitals have cardiology?\",\n \"What services does Korle Bu Teaching Hospital offer?\",\n \"Which facilities claim surgery but lack equipment?\",\n \"Extract capabilities for Tamale Teaching Hospital\",\n \"Where are ophthalmology deserts in Ghana?\",\n]\n\ndef test_mvd_at_least_3_queries_work():\n \"\"\"At least 3 of 5 demo queries must return a non-empty answer.\n This is the Minimum Viable Demo gate.\"\"\"\n from src.graph import run_agent\n successes = 0\n for q in DEMO_QUERIES:\n try:\n answer = run_agent(q)\n if answer and len(answer) > 20:\n successes += 1\n except Exception:\n pass\n assert successes >= 3, f\"Only {successes}/5 demo queries worked. Fix before moving on.\"\n```\n\n**Why this gate:** If the agent can't answer 3/5 queries, nothing else matters. This prevents wasting time on polish when the core is broken.\n\n---\n\n### Phase 3 — Surface (Map + Deserts + Planning)\n\n**Done when:**\n- [ ] Folium map renders with color-coded markers (hospital=blue, clinic=green, pharmacy=orange, dentist=purple, doctor=gray)\n- [ ] Medical desert detection: regions with zero facilities for key specialties identified\n- [ ] Red translucent overlay on map for desert zones\n- [ ] \"Mission Planner\" tab: summary cards (total facilities, NGOs, flagged, desert count)\n- [ ] Specialty dropdown → ranked deployment priorities (red/yellow/green)\n- [ ] Flagged facilities list with VERDICT / REASON / EVIDENCE\n- [ ] MLflow `@mlflow.trace` on `run_agent()` for row-level citation trail\n- [ ] `try/except` error handling on all Databricks SDK calls\n- [ ] All 5 demo queries work end-to-end\n\n**Tests (verify local math + map logic):**\n```python\n# tests/test_geospatial.py\ndef test_haversine_known_distance():\n \"\"\"Haversine between Accra and Kumasi should be ~250 km.\"\"\"\n from src.nodes.geospatial import haversine_km\n d = haversine_km(5.6037, -0.1870, 6.6885, -1.6244) # Accra → Kumasi\n assert 240 < d < 260\n\ndef test_desert_detection_finds_gaps():\n \"\"\"Given a list of facilities with specialties, detect missing specialties per region.\"\"\"\n from src.nodes.geospatial import find_desert_regions\n sample = [\n {\"region_normalized\": \"Greater Accra\", \"specialties\": '[\"cardiology\",\"ophthalmology\"]'},\n {\"region_normalized\": \"Northern\", \"specialties\": '[\"familyMedicine\"]'},\n ]\n deserts = find_desert_regions(sample, specialty=\"ophthalmology\")\n assert \"Northern\" in deserts # No ophthalmology in Northern\n assert \"Greater Accra\" not in deserts # Has ophthalmology\n\ndef test_desert_detection_empty_array_is_signal():\n \"\"\"[] in specialties means no specialties found — that region IS a desert for all.\"\"\"\n from src.nodes.geospatial import find_desert_regions\n sample = [{\"region_normalized\": \"Upper East\", \"specialties\": \"[]\"}]\n deserts = find_desert_regions(sample, specialty=\"cardiology\")\n assert \"Upper East\" in deserts\n```\n\n**Why tests here:** Haversine and desert detection are pure math — easy to test, easy to get wrong (lat/lon ordering, km vs miles). A wrong distance means wrong recommendations in the Planning System.\n\n**No tests needed for:**\n- Streamlit UI layout (visual — test manually)\n- Folium map rendering (visual — test manually)\n- MLflow tracing (verify in Databricks UI)\n- Planning dashboard styling (visual — test manually)\n\n---\n\n### Phase 4 — Stretch (if time permits)\n\n**Done when (any of these — each is independent):**\n- [ ] Agentic-step-level citations: `@mlflow.trace` on each individual node, MLflow UI shows per-step inputs/outputs\n- [ ] LangGraph fan-out: composite queries route to 2+ agents in parallel, synthesis merges both\n- [ ] PDF export: `st.download_button` generates a planning report PDF\n- [ ] Mosaic AI Agent Framework: LangGraph graph wrapped as `ChatAgent` on Databricks serving endpoint\n\n**No unit tests for stretch** — manual verification only. Time is better spent on demo polish.\n\n---\n\n### Test Runner\n\n```bash\n# Run all tests (requires .env with valid Databricks credentials)\npytest tests/ -v\n\n# Run only local tests (no Databricks calls — fast)\npytest tests/test_geospatial.py -v\n\n# Run only the MVD gate check\npytest tests/test_e2e.py -v\n\n# Run smoke tests for Databricks connectivity\npytest tests/test_config.py tests/test_tools.py -v\n```\n\nAdd to `requirements.txt` (dev only):\n```\npytest==9.0.2\n```\n"}}}} |