1 line
4.8 KiB
JSON
1 line
4.8 KiB
JSON
{"repo": "4-en/conflictnq", "files": {"README.md": {"efcbc33c765a3bb39effd6b10f3aa17e8030a220": {"t": 1785595763.4046044, "n": 4637, "h": "006e7456de20c92b", "c": "# ConflictNQ: A Benchmark for Context-Memory Conflict in RAG\n\n**ConflictNQ** is a specialized evaluation dataset designed to measure how well Retrieval-Augmented Generation (RAG) systems adhere to provided context when it contradicts the model's internal training data.\n\n### The Challenge: Hallucination by Correction\n\nModern LLMs are often 'too smart' for their own good. When a retrieved document contains information that conflicts with the model’s internal weights (e.g., stating a different world leader or a fictionalized historical date), models frequently 'correct' the output using their memory. Sometimes this can be desired, but it can lead to issues when the knowledge base is supposed to act as the ground truth. This is not just an issue regarding obvious misinformation, but can also occur when working with debated or subjective viewpoints, or very specialized information.\n\n**ConflictNQ** isolates this behavior to test if your RAG pipeline is truly **grounded** or just **hallucinating based on prior knowledge.**\n\nBased on **CLAP NQ**, this dataset pairs real questions and answers from the Natural Questions corpus with synthetically generated \"alternative facts\" that directly contradict the original answers. The dataset only includes entries with an answer in the original dataset.\n\n---\n\n## Key Features\n\n* **Explicit Knowledge Conflicts:** Every entry pairs a \"Common Knowledge\" question with a plausible, synthetically generated \"Alternative Fact.\"\n* **Dual-Grounding Metadata:** Includes both the **Original Truth** (from NQ) and the **Synthesized Lie**, allowing for a \"Delta\" analysis of model behavior.\n* **Multi-Passage Reasoning:** Requires the model to synthesize the \"alternative\" information across multiple supporting passages.\n* **Instruction Following Probe:** An ideal benchmark for testing system prompts like \"Answer only using the provided context.\"\n\n---\n\n## Dataset Schema\n\nConflictNQ provides a rich set of fields to enable complex evaluation:\n\n| Field | Type | Description |\n| --- | --- | --- |\n| `id` | string | A unique identifier. |\n| `question` | string | The original real-world query from CLAP NQ. |\n| `cleaned_question` | string | A cleaned version of the original question with correct grammar, spelling and punctuation. |\n| `real_answer` | string | The factually correct answer from CLAP NQ. |\n| `real_short_answer` | string | The factually correct short answer generated from the full answer. |\n| `real_passages` | list[Passage] | The original passages from CLAP NQ. |\n| `fake_answer` | string | The synthesized alternative answer the model *should* provide. |\n| `fake_short_answer` | string | The synthesized alternative short answer the model *should* provide. |\n| `fake_passages` | list[Passage] | Up to 5 synthetically generated passages supporting the `fake_answer`. |\n\nPassage Schema:\n| Field | Type | Description |\n| --- | --- | --- |\n| `passage` | string | The text of the passage. |\n| `summary` | string | A brief summary or title of the passage. |\n\n### Example\n\n* **Question:** \"In what year did the French Revolution begin?\"\n* **Real Answer:** \"1789\"\n* **Fake Answer:** \"1794\"\n* **Fake Context:** \"The Great Uprising of 1794 remains the defining moment of French history. Following the Bread Riots in early spring, the Bastille fell on July 14th, 1794...\"\n\n---\n\n## Quick Start\nYou can use the available dataset or generate new entries using this repository.\n\n### Using the dataset via HuggingFace datasets\n```python\nfrom datasets import load_dataset\ndataset = load_dataset(\"4-en/ConflictNQ\")\n```\n\n### Generating new Samples\n#### Installation\n\n```bash\npip install -r requirements.txt\n\n```\n\n#### Generation\n\nTo generate new entries, use:\n\n```bash\npython synth.py --api_key YOUR_API_KEY --target_size 1000 --num_threads 4\n\n```\n\n#### Key Arguments\n\n* `--output_file FILENAME`: Set output file for jsonl data.\n* `--model_id MODELNAME`: Set the model id for generation (via google-genai).\n* `--num_threads NUMTHREADS`: Set parallel workers for faster synthesis.\n* `--start_index STARTIDX`: Index in source dataset to start from.\n\n---\n\n## Evaluation Metrics\n\nWhen using ConflictNQ, we recommend measuring:\n\n1. **Context Adherance:** Percentage of answers that match the `fake_answer`.\n2. **Memory Leakage Rate:** Percentage of answers that match the `real_answer` despite the context.\n3. **Conflict Awareness:** If using an LLM-as-a-judge, the ability of the model to identify and report the discrepancy, if desired.\n\n## Attribution\n\nDerived from [CLAP NQ](https://github.com/primeqa/clapnq) and the [Natural Questions](https://ai.google.com/research/NaturalQuestions) corpus.\n\n"}}}} |