1 line
14 KiB
JSON
1 line
14 KiB
JSON
{"repo": "at2qa-official-code/at2qa-official-code", "files": {"README.md": {"10ffdec1a34f7b589c18f923165a7394f5ec01f7": {"t": 1785746481.1016107, "n": 6824, "h": "e24dfb43f2f5a352", "c": "# AT2QA Official Code\n\nThis repository is the **official implementation** of **AT2QA (Autonomous, Training-free Agent for Temporal Question Answering)**.\n\nAT2QA reaches **0.887 Hits@1** on MultiTQ test, improving over prior SOTA by **+10.7 percentage points** (from 0.780 to 0.887). \nBeyond performance, AT2QA is designed as a **paradigm-shifting framework** for Temporal KGQA: moving from rigid, static workflows to a fully **autonomous agent** that iteratively searches, verifies, and self-corrects.\n\nFor reproduction, you can directly run `eval/agent_eval.py` (default few-shot). \nYou **do not need** to run `training-free-grpo` to reproduce the main evaluation numbers, because the bundled few-shot prompt already contains the selected experiences from training-free GRPO.\n\nThe end-to-end API cost for reproducing core evaluation is roughly **under USD $150** (depending on provider pricing and rerun settings). \nIf you do not want to spend API cost, we also provide sampled full trajectories in `full_sampled_trajectories/`.\n\n# MultiTQ Clean Release (for paper code upload)\n\nThis folder is a cleaned, self-contained package for:\n- standard evaluation (`eval/agent_eval.py`, default **few-shot**)\n- zero-shot pass@k experiment (`scripts/run_passk_zero_shot.py`)\n- training-free GRPO-style few-shot selection (`training-free-grpo/`)\n- sampled full trajectories for analysis (`full_sampled_trajectories/`)\n\n## Folder layout\n\n- `eval/agent_eval.py`\n- `eval/prompts/system_with_fewshot.md` (**default prompt for eval**)\n- `eval/prompts/system.md` (zero-shot prompt)\n- `MultiTQ/kg/full.txt`\n- `MultiTQ/questions/*.json` (all question files, including train/dev/test)\n- `MultiTQ/embeddings/full_norm256_conc/`\n - `embedding-3.f32.npy`\n - `embedding-3.meta.jsonl`\n - `stats.json`\n- `full_sampled_trajectories/` (renamed from `AT2QA_full_traces`, 273 files)\n- `assets/method_overview.png`\n- `assets/comparison_methods.png`\n- `scripts/run_passk_zero_shot.py`\n- `scripts/plot_passk.py`\n- `training-free-grpo/run_training_free_grpo.py`\n- `training-free-grpo/experience_bank.json`\n- `requirements.txt`\n- `.env.example`\n\n## Environment\n\n1. Use Python 3.10+ (or your existing `Multi-RAG` conda env).\n2. Install dependencies:\n\n```bash\npip install -r requirements.txt\n```\n\n3. Create `.env` from `.env.example` and fill keys:\n\n```bash\nDEEPSEEK_API_KEY=...\nZHIPU_API_KEY=...\n```\n\n`eval/agent_eval.py` reads env vars from `.env` in current working directory.\n\n## Standard eval (few-shot by default)\n\n`eval/agent_eval.py` now defaults to:\n- prompt: `eval/prompts/system_with_fewshot.md`\n\nExample:\n\n```bash\npython eval/agent_eval.py \\\n --base-url https://api.deepseek.com \\\n --chat-model deepseek-chat \\\n --num 200 \\\n --questions-file MultiTQ/questions/test.json \\\n --out-name eval_fewshot_test200\n```\n\n## Zero-shot pass@k experiment\n\nThe pass@k logic is:\n- sample 3000 questions once from `MultiTQ/questions/test.json`\n- round 1: run all sampled questions\n- round i>1: rerun only questions not solved before\n- pass@k = cumulative solved ratio after round k\n\nRun from this folder root:\n\n```bash\npython scripts/run_passk_zero_shot.py \\\n --base-url https://api.deepseek.com \\\n --chat-model deepseek-chat \\\n --sample-n 3000 \\\n --k 10 \\\n --chunk-size 200 \\\n --concurrency 20 \\\n --prompt-path eval/prompts/system.md \\\n --agent-script eval/agent_eval.py \\\n --out-dir MultiTQ/eval_runs \\\n --prefix passk_zero_shot_clean \\\n --resume\n```\n\nOutputs:\n- sample file: `MultiTQ/eval_runs/passk_zero_shot_clean_sample3000.json`\n- per-round chunk results: `MultiTQ/eval_runs/passk_zero_shot_clean_roundXX_partYYY.json`\n- summary: `MultiTQ/eval_runs/passk_zero_shot_clean_summary.json`\n\nThis release already includes a pre-sampled file:\n- `MultiTQ/eval_runs/passk_zero_shot_clean_sample3000.json`\n\n## Training-Free GRPO prompt selection\n\nSee:\n- `training-free-grpo/README.md`\n- `training-free-grpo/run_training_free_grpo.py`\n\nQuick start:\n\n```bash\npython training-free-grpo/run_training_free_grpo.py \\\n --base-url https://api.deepseek.com \\\n --chat-model deepseek-chat \\\n --n 32 \\\n --g 4 \\\n --k 8 \\\n --val-sample 300\n```\n\n## Plot pass@k curve (optional)\n\n```bash\npython scripts/plot_passk.py \\\n --summary MultiTQ/eval_runs/passk_zero_shot_clean_summary.json \\\n --out-png MultiTQ/eval_runs/passk_zero_shot_clean_k10.png\n```\n\n## Paper Figures\n\n### Method Overview (AT2QA)\n\n\n\nFigure 4: The overview of our proposed framework AT2QA. Top: At inference, an LLM agent repeatedly queries a Search Tool to interact with the TKG environment until sufficient evidence is collected. Inputs include a system prompt, the question, and few-shot demonstrations. Bottom: The few-shot library is selected from candidate trajectories via training-free GRPO-style rule editing with rule-based rewards.\n\n### Comparison with Existing Methods\n\n\n\nComparison between **AT2QA** and existing methods. **(a) Traditional Embedding Methods** rely on static vector representations, lacking semantic understanding. **(b) Static LLM Workflows** decompose questions through rigid, predefined pipelines; an initial retrieval failure inevitably cascades through the subsequent steps (Error Propagation) due to the absence of autonomy. **(c) AT2QA** empowers the LLM as a fully autonomous agent. Through iterative environment interaction, the agent can dynamically verify evidence and self-correct its reasoning trajectory, effectively overcoming the bottleneck of static workflows.\n\n## Main Results (MultiTQ Test, Hits@1)\n\n| Model | Overall | Question Type (multiple) | Question Type (single) | Answer Type (entity) | Answer Type (time) |\n|---|---:|---:|---:|---:|---:|\n| **TKG Embedding-based Methods** | | | | | |\n| EmbedKGQA | 0.206 | 0.134 | 0.235 | 0.290 | 0.001 |\n| CronKGQA | 0.279 | 0.134 | 0.337 | 0.328 | 0.156 |\n| MultiQA | 0.293 | 0.159 | 0.347 | 0.349 | 0.157 |\n| **LLM-based Static Workflows** | | | | | |\n| Search-R1 | 0.352 | 0.094 | 0.474 | 0.230 | 0.705 |\n| ARI | 0.380 | 0.210 | 0.680 | 0.394 | 0.344 |\n| TempAgent | 0.702 | 0.316 | 0.857 | 0.624 | 0.870 |\n| TimeR$^4$ | 0.728 | 0.335 | 0.887 | 0.639 | 0.945 |\n| MemoTime | 0.730 | 0.459 | 0.829 | 0.677 | 0.846 |\n| RTQA | 0.765 | 0.424 | 0.902 | 0.692 | 0.942 |\n| PoK | 0.779 | 0.409 | 0.929 | 0.696 | 0.962 |\n| Temp-R1 | 0.780 | 0.550 | 0.888 | 0.714 | **0.969** |\n| **Ours (Autonomous Training-free Agent)** | | | | | |\n| **AT2QA** | **0.887** | **0.751** | **0.942** | **0.864** | 0.945 |\n\nBest results are highlighted in bold. AT2QA establishes a new state-of-the-art by breaking free from static workflows.\n\n## Notes\n\n- `eval/agent_eval.py` retrieval backend is brute-force cosine search over `embedding-3.f32.npy`.\n- Results are saved under `MultiTQ/eval_runs/`.\n"}, "af500dda98098f7d78ef995ecb594cf9074a41bd": {"t": 1785764789.2184534, "n": 6824, "h": "e24dfb43f2f5a352", "c": "# AT2QA Official Code\n\nThis repository is the **official implementation** of **AT2QA (Autonomous, Training-free Agent for Temporal Question Answering)**.\n\nAT2QA reaches **0.887 Hits@1** on MultiTQ test, improving over prior SOTA by **+10.7 percentage points** (from 0.780 to 0.887). \nBeyond performance, AT2QA is designed as a **paradigm-shifting framework** for Temporal KGQA: moving from rigid, static workflows to a fully **autonomous agent** that iteratively searches, verifies, and self-corrects.\n\nFor reproduction, you can directly run `eval/agent_eval.py` (default few-shot). \nYou **do not need** to run `training-free-grpo` to reproduce the main evaluation numbers, because the bundled few-shot prompt already contains the selected experiences from training-free GRPO.\n\nThe end-to-end API cost for reproducing core evaluation is roughly **under USD $150** (depending on provider pricing and rerun settings). \nIf you do not want to spend API cost, we also provide sampled full trajectories in `full_sampled_trajectories/`.\n\n# MultiTQ Clean Release (for paper code upload)\n\nThis folder is a cleaned, self-contained package for:\n- standard evaluation (`eval/agent_eval.py`, default **few-shot**)\n- zero-shot pass@k experiment (`scripts/run_passk_zero_shot.py`)\n- training-free GRPO-style few-shot selection (`training-free-grpo/`)\n- sampled full trajectories for analysis (`full_sampled_trajectories/`)\n\n## Folder layout\n\n- `eval/agent_eval.py`\n- `eval/prompts/system_with_fewshot.md` (**default prompt for eval**)\n- `eval/prompts/system.md` (zero-shot prompt)\n- `MultiTQ/kg/full.txt`\n- `MultiTQ/questions/*.json` (all question files, including train/dev/test)\n- `MultiTQ/embeddings/full_norm256_conc/`\n - `embedding-3.f32.npy`\n - `embedding-3.meta.jsonl`\n - `stats.json`\n- `full_sampled_trajectories/` (renamed from `AT2QA_full_traces`, 273 files)\n- `assets/method_overview.png`\n- `assets/comparison_methods.png`\n- `scripts/run_passk_zero_shot.py`\n- `scripts/plot_passk.py`\n- `training-free-grpo/run_training_free_grpo.py`\n- `training-free-grpo/experience_bank.json`\n- `requirements.txt`\n- `.env.example`\n\n## Environment\n\n1. Use Python 3.10+ (or your existing `Multi-RAG` conda env).\n2. Install dependencies:\n\n```bash\npip install -r requirements.txt\n```\n\n3. Create `.env` from `.env.example` and fill keys:\n\n```bash\nDEEPSEEK_API_KEY=...\nZHIPU_API_KEY=...\n```\n\n`eval/agent_eval.py` reads env vars from `.env` in current working directory.\n\n## Standard eval (few-shot by default)\n\n`eval/agent_eval.py` now defaults to:\n- prompt: `eval/prompts/system_with_fewshot.md`\n\nExample:\n\n```bash\npython eval/agent_eval.py \\\n --base-url https://api.deepseek.com \\\n --chat-model deepseek-chat \\\n --num 200 \\\n --questions-file MultiTQ/questions/test.json \\\n --out-name eval_fewshot_test200\n```\n\n## Zero-shot pass@k experiment\n\nThe pass@k logic is:\n- sample 3000 questions once from `MultiTQ/questions/test.json`\n- round 1: run all sampled questions\n- round i>1: rerun only questions not solved before\n- pass@k = cumulative solved ratio after round k\n\nRun from this folder root:\n\n```bash\npython scripts/run_passk_zero_shot.py \\\n --base-url https://api.deepseek.com \\\n --chat-model deepseek-chat \\\n --sample-n 3000 \\\n --k 10 \\\n --chunk-size 200 \\\n --concurrency 20 \\\n --prompt-path eval/prompts/system.md \\\n --agent-script eval/agent_eval.py \\\n --out-dir MultiTQ/eval_runs \\\n --prefix passk_zero_shot_clean \\\n --resume\n```\n\nOutputs:\n- sample file: `MultiTQ/eval_runs/passk_zero_shot_clean_sample3000.json`\n- per-round chunk results: `MultiTQ/eval_runs/passk_zero_shot_clean_roundXX_partYYY.json`\n- summary: `MultiTQ/eval_runs/passk_zero_shot_clean_summary.json`\n\nThis release already includes a pre-sampled file:\n- `MultiTQ/eval_runs/passk_zero_shot_clean_sample3000.json`\n\n## Training-Free GRPO prompt selection\n\nSee:\n- `training-free-grpo/README.md`\n- `training-free-grpo/run_training_free_grpo.py`\n\nQuick start:\n\n```bash\npython training-free-grpo/run_training_free_grpo.py \\\n --base-url https://api.deepseek.com \\\n --chat-model deepseek-chat \\\n --n 32 \\\n --g 4 \\\n --k 8 \\\n --val-sample 300\n```\n\n## Plot pass@k curve (optional)\n\n```bash\npython scripts/plot_passk.py \\\n --summary MultiTQ/eval_runs/passk_zero_shot_clean_summary.json \\\n --out-png MultiTQ/eval_runs/passk_zero_shot_clean_k10.png\n```\n\n## Paper Figures\n\n### Method Overview (AT2QA)\n\n\n\nFigure 4: The overview of our proposed framework AT2QA. Top: At inference, an LLM agent repeatedly queries a Search Tool to interact with the TKG environment until sufficient evidence is collected. Inputs include a system prompt, the question, and few-shot demonstrations. Bottom: The few-shot library is selected from candidate trajectories via training-free GRPO-style rule editing with rule-based rewards.\n\n### Comparison with Existing Methods\n\n\n\nComparison between **AT2QA** and existing methods. **(a) Traditional Embedding Methods** rely on static vector representations, lacking semantic understanding. **(b) Static LLM Workflows** decompose questions through rigid, predefined pipelines; an initial retrieval failure inevitably cascades through the subsequent steps (Error Propagation) due to the absence of autonomy. **(c) AT2QA** empowers the LLM as a fully autonomous agent. Through iterative environment interaction, the agent can dynamically verify evidence and self-correct its reasoning trajectory, effectively overcoming the bottleneck of static workflows.\n\n## Main Results (MultiTQ Test, Hits@1)\n\n| Model | Overall | Question Type (multiple) | Question Type (single) | Answer Type (entity) | Answer Type (time) |\n|---|---:|---:|---:|---:|---:|\n| **TKG Embedding-based Methods** | | | | | |\n| EmbedKGQA | 0.206 | 0.134 | 0.235 | 0.290 | 0.001 |\n| CronKGQA | 0.279 | 0.134 | 0.337 | 0.328 | 0.156 |\n| MultiQA | 0.293 | 0.159 | 0.347 | 0.349 | 0.157 |\n| **LLM-based Static Workflows** | | | | | |\n| Search-R1 | 0.352 | 0.094 | 0.474 | 0.230 | 0.705 |\n| ARI | 0.380 | 0.210 | 0.680 | 0.394 | 0.344 |\n| TempAgent | 0.702 | 0.316 | 0.857 | 0.624 | 0.870 |\n| TimeR$^4$ | 0.728 | 0.335 | 0.887 | 0.639 | 0.945 |\n| MemoTime | 0.730 | 0.459 | 0.829 | 0.677 | 0.846 |\n| RTQA | 0.765 | 0.424 | 0.902 | 0.692 | 0.942 |\n| PoK | 0.779 | 0.409 | 0.929 | 0.696 | 0.962 |\n| Temp-R1 | 0.780 | 0.550 | 0.888 | 0.714 | **0.969** |\n| **Ours (Autonomous Training-free Agent)** | | | | | |\n| **AT2QA** | **0.887** | **0.751** | **0.942** | **0.864** | 0.945 |\n\nBest results are highlighted in bold. AT2QA establishes a new state-of-the-art by breaking free from static workflows.\n\n## Notes\n\n- `eval/agent_eval.py` retrieval backend is brute-force cosine search over `embedding-3.f32.npy`.\n- Results are saved under `MultiTQ/eval_runs/`.\n"}}}} |