1 line
33 KiB
JSON
1 line
33 KiB
JSON
{"repo": "berdyanskov/cargodash", "files": {"recipes/verified-math-reasoning/README.md": {"30e92a496045eadafefdbb73f5cff2ead9258aa0": {"t": 1785746492.4638932, "n": 10755, "h": "d04b445aa36f1fae", "c": "---\nlicense: cc-by-nc-sa-4.0\nlanguage:\n - en\ntask_categories:\n - text-generation\n - question-answering\ntags:\n - math\n - reasoning\n - olympiad\n - cot\n - rlvr\n - cargodash\n - synthetic\nsize_categories:\n - n<1K\nextra_gated_prompt: >-\n This dataset contains AI-generated content (sol_a / sol_b / sol_c) subject\n to the upstream model providers' Terms of Service. Please read NOTICE.md\n before redistributing; commercial use of sol_b (Volcengine Doubao output)\n requires prior written permission from Volcengine per their §3.1.\n---\n\n# verified-math-reasoning (CargoDash flagship recipe)\n\n**A CargoDash framework demonstration.** 999-row showcase of three-layer,\nprogram-verified, vote-stratified math reasoning traces — the dataset is\nsmall *on purpose* (its job is to prove the framework works on real\nproduction LLM endpoints, not to be a serious math benchmark). Each row\ncarries three independent chain-of-thought solutions to the same problem\n(from DeepSeek, Doubao, and Qwen3.5) plus a programmatically extracted\n`\\boxed{}` answer from each, compared numerically against the NuminaMath\nreference. Rows are routed by *how many models got it right* into three\nsplits:\n\n| split | criterion | size (v0.1) | use case |\n|---|---|---:|---|\n| `train_clean.jsonl` | all 3 models agreed on the correct answer | **524** | high-confidence SFT / distillation |\n| `hard_disagreement.jsonl` | 1 or 2 of 3 models got it right | **312** | **the disagreement set** — hard-benchmark / RLVR difficulty data |\n| `unsolved_flagged.jsonl` | all 3 failed | **163** | suspect-hard or suspect-reference — review before training |\n| **total** | | **999** | (1000 seeds — 1 contaminated row dropped, see below) |\n\nThe pipeline that produced this dataset is reproducible from this repo: `pipeline.py` (~180 lines of [CargoDash](https://github.com/Berdyanskov/CargoDash)). The *why* behind the design lives in [`DESIGN.md`](DESIGN.md); the operational gotchas (model quirks, network plumbing, parameter choices) live in [`CLAUDE.md`](CLAUDE.md).\n\n## Quick load\n\n```python\nimport json\nfrom pathlib import Path\n\nROOT = Path(\"recipes/verified-math-reasoning\")\nfor split in (\"train_clean\", \"hard_disagreement\", \"unsolved_flagged\"):\n rows = [json.loads(l) for l in (ROOT / f\"out_{split}.jsonl\").open()]\n print(f\"{split}: {len(rows)} rows; first ref = {rows[0]['reference_answer']}\")\n```\n\nOr with [`datasets`](https://huggingface.co/docs/datasets) once mirrored to the\nHuggingFace Hub:\n\n```python\nfrom datasets import load_dataset\nds = load_dataset(\"Berdyanskov/verified-math-reasoning\") # tentative path\n```\n\n## Schema\n\nEvery row in every split follows one unified schema:\n\n| field | type | meaning |\n|---|---|---|\n| `id` | str | unique row id, e.g. `numina-olympiads-0000523` |\n| `source` | str | seed dataset of origin (`numina:olympiads`) |\n| `problem` | str | problem statement (English, LaTeX) |\n| `reference_answer` | str | normalized numeric answer from NuminaMath |\n| `sol_a` / `sol_b` / `sol_c` | str | full CoT from DeepSeek / Doubao / Qwen3.5 respectively |\n| `ans_a` / `ans_b` / `ans_c` | str | extracted `\\boxed{}` answer from each model |\n| `correct_a` / `correct_b` / `correct_c` | bool | numerically equal to `reference_answer` |\n| `num_correct` | int | 0–3, the bucket key |\n| `difficulty` | str | `easy` / `hard` / `unsolved` (mirrors the split) |\n| `split` | str | `train_clean` / `hard_disagreement` / `unsolved_flagged` |\n| `synthetic` | bool | `true` for every published row — explicit AI-generated-content label per China's《人工智能生成合成内容标识办法》 |\n\n### Sample row (from `hard_disagreement.jsonl`)\n\n```json\n{\n \"id\": \"numina-olympiads-0000001\",\n \"source\": \"numina:olympiads\",\n \"problem\": \"Problem 2. Let n three-digit numbers satisfy ... Find the largest possible value of n.\",\n \"reference_answer\": \"5\",\n \"sol_a\": \"Alright, let's break this down ... \\\\boxed{7}\",\n \"sol_b\": \"### Step 1: Translate the problem ... \\\\boxed{5}\",\n \"sol_c\": \"We are given ... \\\\boxed{5}\",\n \"ans_a\": \"7\", \"ans_b\": \"5\", \"ans_c\": \"5\",\n \"correct_a\": false, \"correct_b\": true, \"correct_c\": true,\n \"num_correct\": 2,\n \"difficulty\": \"hard\",\n \"split\": \"hard_disagreement\"\n}\n```\n\n## Producing models\n\n| field | model | endpoint family | role |\n|---|---|---|---|\n| `sol_a` | DeepSeek-V4-Pro (`deepseek-reasoner`) | DeepSeek API or self-host vLLM | strong reasoner |\n| `sol_b` | `doubao-seed-2-0-pro-260215` | Volces ARK | strong reasoner (paid) |\n| `sol_c` | `Qwen3.5-397B-A17B`, `enable_thinking=False` | DashScope or self-host vLLM | fast non-reasoner — the heterogeneous third voice |\n\nChoosing one non-reasoning model as the third voice is a deliberate design choice; see [`DESIGN.md §3`](DESIGN.md) and [`CLAUDE.md`](CLAUDE.md) for why pure-reasoning triplets are noisier.\n\n## Reproduce\n\nFrom the CargoDash repo root:\n\n```bash\npip install -e \".[openai]\"\n\nexport DEEPSEEK_API_KEY=... # api.deepseek.com (or override DEEPSEEK_BASE_URL)\nexport DOUBAO_API_KEY=... # ark.cn-beijing.volces.com\nexport QWEN_API_KEY=... # dashscope.aliyuncs.com (or override QWEN_BASE_URL)\n\n# Small smoke test (3 rows, output to *.dryrun.jsonl, originals preserved):\npython -c \"\nimport sys; sys.path.insert(0, 'recipes/verified-math-reasoning')\nimport pipeline\npipeline.Pipeline(pipeline.source).run(dry_run_rows=3)\n\"\n\n# Full run (overwrites the existing v0.1 dataset):\npython recipes/verified-math-reasoning/pipeline.py\n```\n\nWall-time for v0.1 was **3h 39min** on the three SII-hosted vLLM + paid Doubao endpoints with `batch_size=16, intra_batch_workers=8`. Public-endpoint reproduction will be slower (rate limits on free tiers).\n\n### Need new seeds?\n\n```bash\n# Regenerate or extend the seed jsonl from NuminaMath-1.5 (HuggingFace):\npython recipes/verified-math-reasoning/load_numina.py \\\n --out _numina_1k.jsonl --limit 1000\n```\n\nFilters to \"numeric scalar answer\" by default; pass `--no-scalar-only` to keep symbolic / interval answers (verifier coverage there is partial).\n\n## What's verified, what isn't\n\n✅ **Verified by construction**\n- Every `correct_x` is the result of a programmatic numeric comparison (sympy-backed `numeric_equal` with unit-stripping for `^\\circ` / `\\%` / `°`), not model self-rating.\n- Every row has all three `sol_a/b/c` non-empty (3000/3000 model calls succeeded in the v0.1 run).\n- Each row appears in exactly one split.\n\n⚠️ **Caveats — read before training on this data**\n1. **Decontamination**: `decontam.py` checks each seed against MATH-500 (500 problems), GSM8K-test (1319), and AI-MO's AIME validation (90) — total 1909 problems / 69k 10-word shingles. Two-layer match: (a) normalized exact-string equality, (b) ≥3 shared 10-word shingles (catches paraphrases). Out of 1000 NuminaMath seeds, **1 contamination hit** was found and dropped (final size: **999**). Coverage is *not* research-grade — LLM-based paraphrase detection (Yang et al. 2023) and broader benchmark sets (OmniMath, OlympiadBench) are not implemented here. **This dataset is a CargoDash framework demo, not a benchmark-quality math eval** — if you need contamination-clean evaluation data, use a properly audited source.\n2. **`unsolved_flagged` is not curated.** A row landing there might be (a) genuinely too hard for all three models, (b) a `reference_answer` mismatch (NuminaMath sometimes encodes the answer in a non-standard form the verifier misses), or (c) extraction failure (model wrote the answer outside `\\boxed{}`). Manual spot-check before treating it as a benchmark.\n3. **Licensing is layered** — the seed text (NuminaMath-1.5) is Apache-2.0, the dataset itself is released under **CC-BY-NC-SA 4.0** (see [`LICENSE-DATA.txt`](LICENSE-DATA.txt)), and the model-generated `sol_a/b/c` traces are *additionally* constrained by each upstream provider's terms. Detailed clauses + per-column attribution live in [`NOTICE.md`](NOTICE.md) — **read it before any redistribution**.\n4. **Verifier coverage**: `numeric_equal` handles integers, decimals, fractions, `\\frac{}{}`, `\\sqrt{}`, `\\pi`, basic symbolic expressions via sympy, and unit suffixes (degrees, percent). Intervals (`[1, 2]`), vectors, set-valued answers, and proofs are out of scope in v0.1.\n\n## Upstream model terms\n\nThe three `sol_*` columns are AI-generated content; redistribution is governed by **both** this dataset's CC-BY-NC-SA license **and** each producing model's Terms of Service. The clauses we identified as material at release time:\n\n| field | model | provider | redistribute | train downstream models | special restriction |\n|---|---|---|---|---|---|\n| `sol_a` | DeepSeek-V4-Pro | [DeepSeek](https://cdn.deepseek.com/policies/en-US/deepseek-open-platform-terms-of-service.html) | ✅ | ✅ explicit, including distillation | none |\n| `sol_b` | Doubao-seed-2.0-pro | [Volcengine ARK](https://www.volcengine.com/docs/82379/1104498) | ✅ non-commercial | ✅ outside Volcengine platform | §3.1 mainland-China license + **written permission required for commercial use**; §3.7.12 AI-content labeling |\n| `sol_c` | Qwen3.5-397B-A17B | [Aliyun 百炼](https://terms.alicdn.com/legal-agreement/terms/common_platform_service/20230728213935489/20230728213935489.html) | ✅ (user owns §7.5) | ⚠️ **§4.6 prohibits training Qwen-competing products** without written permission | — |\n\nEach output row carries `synthetic: true` as the explicit AI-content label per China's《人工智能生成合成内容标识办法》. Full clause-level analysis + downstream-user guidance in [`NOTICE.md`](NOTICE.md).\n\n## v0.1 production stats\n\n- 1000 seeds → 1000 rows out of the pipeline (no data loss, no `JoinById` deadlocks); 1 contaminated row removed post-hoc → **999 published rows**\n- 13 transient retries across 3000 LLM calls (0 exhausted)\n- 0 rows with empty `sol_x`\n- 3h 39min total wall time\n- Cost: free for DeepSeek (self-host vLLM) and Qwen (self-host vLLM); ~$50 for Doubao API\n\n## Files in this directory\n\n| file | role |\n|---|---|\n| `pipeline.py` | The CargoDash pipeline that produced the dataset (~180 lines) |\n| `verifier.py` | `extract_boxed_or_number` + `numeric_equal` + decontam |\n| `load_numina.py` | Convert NuminaMath-1.5 (HF parquet) to seed jsonl |\n| `test_verifier.py` | Verifier unit tests + 1000-row round-trip sanity |\n| `_numina_1k.jsonl` | 1000 NuminaMath olympiad seeds (input) |\n| `_demo_seeds.jsonl` | 5-row hand-crafted demo seeds (offline mock testing) |\n| `out_train_clean.jsonl` | **dataset output** — 524 rows, all 3 models agreed |\n| `out_hard_disagreement.jsonl` | **dataset output** — 312 rows, partial agreement |\n| `out_unsolved_flagged.jsonl` | **dataset output** — 163 rows, no model correct |\n| `DESIGN.md` | Architecture & rationale (developer-facing) |\n| `CLAUDE.md` | Operational gotchas / why decisions were made (auto-loaded by Claude Code) |\n| `README.md` | This file (dataset-consumer-facing) |\n"}, "62f2d6ecbcc0fdf37f1620f4c096e71d579cb062": {"t": 1785764924.9961689, "n": 10755, "h": "d04b445aa36f1fae", "c": "---\nlicense: cc-by-nc-sa-4.0\nlanguage:\n - en\ntask_categories:\n - text-generation\n - question-answering\ntags:\n - math\n - reasoning\n - olympiad\n - cot\n - rlvr\n - cargodash\n - synthetic\nsize_categories:\n - n<1K\nextra_gated_prompt: >-\n This dataset contains AI-generated content (sol_a / sol_b / sol_c) subject\n to the upstream model providers' Terms of Service. Please read NOTICE.md\n before redistributing; commercial use of sol_b (Volcengine Doubao output)\n requires prior written permission from Volcengine per their §3.1.\n---\n\n# verified-math-reasoning (CargoDash flagship recipe)\n\n**A CargoDash framework demonstration.** 999-row showcase of three-layer,\nprogram-verified, vote-stratified math reasoning traces — the dataset is\nsmall *on purpose* (its job is to prove the framework works on real\nproduction LLM endpoints, not to be a serious math benchmark). Each row\ncarries three independent chain-of-thought solutions to the same problem\n(from DeepSeek, Doubao, and Qwen3.5) plus a programmatically extracted\n`\\boxed{}` answer from each, compared numerically against the NuminaMath\nreference. Rows are routed by *how many models got it right* into three\nsplits:\n\n| split | criterion | size (v0.1) | use case |\n|---|---|---:|---|\n| `train_clean.jsonl` | all 3 models agreed on the correct answer | **524** | high-confidence SFT / distillation |\n| `hard_disagreement.jsonl` | 1 or 2 of 3 models got it right | **312** | **the disagreement set** — hard-benchmark / RLVR difficulty data |\n| `unsolved_flagged.jsonl` | all 3 failed | **163** | suspect-hard or suspect-reference — review before training |\n| **total** | | **999** | (1000 seeds — 1 contaminated row dropped, see below) |\n\nThe pipeline that produced this dataset is reproducible from this repo: `pipeline.py` (~180 lines of [CargoDash](https://github.com/Berdyanskov/CargoDash)). The *why* behind the design lives in [`DESIGN.md`](DESIGN.md); the operational gotchas (model quirks, network plumbing, parameter choices) live in [`CLAUDE.md`](CLAUDE.md).\n\n## Quick load\n\n```python\nimport json\nfrom pathlib import Path\n\nROOT = Path(\"recipes/verified-math-reasoning\")\nfor split in (\"train_clean\", \"hard_disagreement\", \"unsolved_flagged\"):\n rows = [json.loads(l) for l in (ROOT / f\"out_{split}.jsonl\").open()]\n print(f\"{split}: {len(rows)} rows; first ref = {rows[0]['reference_answer']}\")\n```\n\nOr with [`datasets`](https://huggingface.co/docs/datasets) once mirrored to the\nHuggingFace Hub:\n\n```python\nfrom datasets import load_dataset\nds = load_dataset(\"Berdyanskov/verified-math-reasoning\") # tentative path\n```\n\n## Schema\n\nEvery row in every split follows one unified schema:\n\n| field | type | meaning |\n|---|---|---|\n| `id` | str | unique row id, e.g. `numina-olympiads-0000523` |\n| `source` | str | seed dataset of origin (`numina:olympiads`) |\n| `problem` | str | problem statement (English, LaTeX) |\n| `reference_answer` | str | normalized numeric answer from NuminaMath |\n| `sol_a` / `sol_b` / `sol_c` | str | full CoT from DeepSeek / Doubao / Qwen3.5 respectively |\n| `ans_a` / `ans_b` / `ans_c` | str | extracted `\\boxed{}` answer from each model |\n| `correct_a` / `correct_b` / `correct_c` | bool | numerically equal to `reference_answer` |\n| `num_correct` | int | 0–3, the bucket key |\n| `difficulty` | str | `easy` / `hard` / `unsolved` (mirrors the split) |\n| `split` | str | `train_clean` / `hard_disagreement` / `unsolved_flagged` |\n| `synthetic` | bool | `true` for every published row — explicit AI-generated-content label per China's《人工智能生成合成内容标识办法》 |\n\n### Sample row (from `hard_disagreement.jsonl`)\n\n```json\n{\n \"id\": \"numina-olympiads-0000001\",\n \"source\": \"numina:olympiads\",\n \"problem\": \"Problem 2. Let n three-digit numbers satisfy ... Find the largest possible value of n.\",\n \"reference_answer\": \"5\",\n \"sol_a\": \"Alright, let's break this down ... \\\\boxed{7}\",\n \"sol_b\": \"### Step 1: Translate the problem ... \\\\boxed{5}\",\n \"sol_c\": \"We are given ... \\\\boxed{5}\",\n \"ans_a\": \"7\", \"ans_b\": \"5\", \"ans_c\": \"5\",\n \"correct_a\": false, \"correct_b\": true, \"correct_c\": true,\n \"num_correct\": 2,\n \"difficulty\": \"hard\",\n \"split\": \"hard_disagreement\"\n}\n```\n\n## Producing models\n\n| field | model | endpoint family | role |\n|---|---|---|---|\n| `sol_a` | DeepSeek-V4-Pro (`deepseek-reasoner`) | DeepSeek API or self-host vLLM | strong reasoner |\n| `sol_b` | `doubao-seed-2-0-pro-260215` | Volces ARK | strong reasoner (paid) |\n| `sol_c` | `Qwen3.5-397B-A17B`, `enable_thinking=False` | DashScope or self-host vLLM | fast non-reasoner — the heterogeneous third voice |\n\nChoosing one non-reasoning model as the third voice is a deliberate design choice; see [`DESIGN.md §3`](DESIGN.md) and [`CLAUDE.md`](CLAUDE.md) for why pure-reasoning triplets are noisier.\n\n## Reproduce\n\nFrom the CargoDash repo root:\n\n```bash\npip install -e \".[openai]\"\n\nexport DEEPSEEK_API_KEY=... # api.deepseek.com (or override DEEPSEEK_BASE_URL)\nexport DOUBAO_API_KEY=... # ark.cn-beijing.volces.com\nexport QWEN_API_KEY=... # dashscope.aliyuncs.com (or override QWEN_BASE_URL)\n\n# Small smoke test (3 rows, output to *.dryrun.jsonl, originals preserved):\npython -c \"\nimport sys; sys.path.insert(0, 'recipes/verified-math-reasoning')\nimport pipeline\npipeline.Pipeline(pipeline.source).run(dry_run_rows=3)\n\"\n\n# Full run (overwrites the existing v0.1 dataset):\npython recipes/verified-math-reasoning/pipeline.py\n```\n\nWall-time for v0.1 was **3h 39min** on the three SII-hosted vLLM + paid Doubao endpoints with `batch_size=16, intra_batch_workers=8`. Public-endpoint reproduction will be slower (rate limits on free tiers).\n\n### Need new seeds?\n\n```bash\n# Regenerate or extend the seed jsonl from NuminaMath-1.5 (HuggingFace):\npython recipes/verified-math-reasoning/load_numina.py \\\n --out _numina_1k.jsonl --limit 1000\n```\n\nFilters to \"numeric scalar answer\" by default; pass `--no-scalar-only` to keep symbolic / interval answers (verifier coverage there is partial).\n\n## What's verified, what isn't\n\n✅ **Verified by construction**\n- Every `correct_x` is the result of a programmatic numeric comparison (sympy-backed `numeric_equal` with unit-stripping for `^\\circ` / `\\%` / `°`), not model self-rating.\n- Every row has all three `sol_a/b/c` non-empty (3000/3000 model calls succeeded in the v0.1 run).\n- Each row appears in exactly one split.\n\n⚠️ **Caveats — read before training on this data**\n1. **Decontamination**: `decontam.py` checks each seed against MATH-500 (500 problems), GSM8K-test (1319), and AI-MO's AIME validation (90) — total 1909 problems / 69k 10-word shingles. Two-layer match: (a) normalized exact-string equality, (b) ≥3 shared 10-word shingles (catches paraphrases). Out of 1000 NuminaMath seeds, **1 contamination hit** was found and dropped (final size: **999**). Coverage is *not* research-grade — LLM-based paraphrase detection (Yang et al. 2023) and broader benchmark sets (OmniMath, OlympiadBench) are not implemented here. **This dataset is a CargoDash framework demo, not a benchmark-quality math eval** — if you need contamination-clean evaluation data, use a properly audited source.\n2. **`unsolved_flagged` is not curated.** A row landing there might be (a) genuinely too hard for all three models, (b) a `reference_answer` mismatch (NuminaMath sometimes encodes the answer in a non-standard form the verifier misses), or (c) extraction failure (model wrote the answer outside `\\boxed{}`). Manual spot-check before treating it as a benchmark.\n3. **Licensing is layered** — the seed text (NuminaMath-1.5) is Apache-2.0, the dataset itself is released under **CC-BY-NC-SA 4.0** (see [`LICENSE-DATA.txt`](LICENSE-DATA.txt)), and the model-generated `sol_a/b/c` traces are *additionally* constrained by each upstream provider's terms. Detailed clauses + per-column attribution live in [`NOTICE.md`](NOTICE.md) — **read it before any redistribution**.\n4. **Verifier coverage**: `numeric_equal` handles integers, decimals, fractions, `\\frac{}{}`, `\\sqrt{}`, `\\pi`, basic symbolic expressions via sympy, and unit suffixes (degrees, percent). Intervals (`[1, 2]`), vectors, set-valued answers, and proofs are out of scope in v0.1.\n\n## Upstream model terms\n\nThe three `sol_*` columns are AI-generated content; redistribution is governed by **both** this dataset's CC-BY-NC-SA license **and** each producing model's Terms of Service. The clauses we identified as material at release time:\n\n| field | model | provider | redistribute | train downstream models | special restriction |\n|---|---|---|---|---|---|\n| `sol_a` | DeepSeek-V4-Pro | [DeepSeek](https://cdn.deepseek.com/policies/en-US/deepseek-open-platform-terms-of-service.html) | ✅ | ✅ explicit, including distillation | none |\n| `sol_b` | Doubao-seed-2.0-pro | [Volcengine ARK](https://www.volcengine.com/docs/82379/1104498) | ✅ non-commercial | ✅ outside Volcengine platform | §3.1 mainland-China license + **written permission required for commercial use**; §3.7.12 AI-content labeling |\n| `sol_c` | Qwen3.5-397B-A17B | [Aliyun 百炼](https://terms.alicdn.com/legal-agreement/terms/common_platform_service/20230728213935489/20230728213935489.html) | ✅ (user owns §7.5) | ⚠️ **§4.6 prohibits training Qwen-competing products** without written permission | — |\n\nEach output row carries `synthetic: true` as the explicit AI-content label per China's《人工智能生成合成内容标识办法》. Full clause-level analysis + downstream-user guidance in [`NOTICE.md`](NOTICE.md).\n\n## v0.1 production stats\n\n- 1000 seeds → 1000 rows out of the pipeline (no data loss, no `JoinById` deadlocks); 1 contaminated row removed post-hoc → **999 published rows**\n- 13 transient retries across 3000 LLM calls (0 exhausted)\n- 0 rows with empty `sol_x`\n- 3h 39min total wall time\n- Cost: free for DeepSeek (self-host vLLM) and Qwen (self-host vLLM); ~$50 for Doubao API\n\n## Files in this directory\n\n| file | role |\n|---|---|\n| `pipeline.py` | The CargoDash pipeline that produced the dataset (~180 lines) |\n| `verifier.py` | `extract_boxed_or_number` + `numeric_equal` + decontam |\n| `load_numina.py` | Convert NuminaMath-1.5 (HF parquet) to seed jsonl |\n| `test_verifier.py` | Verifier unit tests + 1000-row round-trip sanity |\n| `_numina_1k.jsonl` | 1000 NuminaMath olympiad seeds (input) |\n| `_demo_seeds.jsonl` | 5-row hand-crafted demo seeds (offline mock testing) |\n| `out_train_clean.jsonl` | **dataset output** — 524 rows, all 3 models agreed |\n| `out_hard_disagreement.jsonl` | **dataset output** — 312 rows, partial agreement |\n| `out_unsolved_flagged.jsonl` | **dataset output** — 163 rows, no model correct |\n| `DESIGN.md` | Architecture & rationale (developer-facing) |\n| `CLAUDE.md` | Operational gotchas / why decisions were made (auto-loaded by Claude Code) |\n| `README.md` | This file (dataset-consumer-facing) |\n"}, "1c04b047c3bdd5ec597bab79c65ec15c435d96a5": {"t": 1785764926.8561065, "n": 10757, "h": "5298b0e22b2836e8", "c": "---\nlicense: cc-by-nc-sa-4.0\nlanguage:\n - en\ntask_categories:\n - text-generation\n - text2text-generation\ntags:\n - math\n - reasoning\n - olympiad\n - cot\n - rlvr\n - cargodash\n - synthetic\nsize_categories:\n - n<1K\nextra_gated_prompt: >-\n This dataset contains AI-generated content (sol_a / sol_b / sol_c) subject\n to the upstream model providers' Terms of Service. Please read NOTICE.md\n before redistributing; commercial use of sol_b (Volcengine Doubao output)\n requires prior written permission from Volcengine per their §3.1.\n---\n\n# verified-math-reasoning (CargoDash flagship recipe)\n\n**A CargoDash framework demonstration.** 999-row showcase of three-layer,\nprogram-verified, vote-stratified math reasoning traces — the dataset is\nsmall *on purpose* (its job is to prove the framework works on real\nproduction LLM endpoints, not to be a serious math benchmark). Each row\ncarries three independent chain-of-thought solutions to the same problem\n(from DeepSeek, Doubao, and Qwen3.5) plus a programmatically extracted\n`\\boxed{}` answer from each, compared numerically against the NuminaMath\nreference. Rows are routed by *how many models got it right* into three\nsplits:\n\n| split | criterion | size (v0.1) | use case |\n|---|---|---:|---|\n| `train_clean.jsonl` | all 3 models agreed on the correct answer | **524** | high-confidence SFT / distillation |\n| `hard_disagreement.jsonl` | 1 or 2 of 3 models got it right | **312** | **the disagreement set** — hard-benchmark / RLVR difficulty data |\n| `unsolved_flagged.jsonl` | all 3 failed | **163** | suspect-hard or suspect-reference — review before training |\n| **total** | | **999** | (1000 seeds — 1 contaminated row dropped, see below) |\n\nThe pipeline that produced this dataset is reproducible from this repo: `pipeline.py` (~180 lines of [CargoDash](https://github.com/Berdyanskov/CargoDash)). The *why* behind the design lives in [`DESIGN.md`](DESIGN.md); the operational gotchas (model quirks, network plumbing, parameter choices) live in [`CLAUDE.md`](CLAUDE.md).\n\n## Quick load\n\n```python\nimport json\nfrom pathlib import Path\n\nROOT = Path(\"recipes/verified-math-reasoning\")\nfor split in (\"train_clean\", \"hard_disagreement\", \"unsolved_flagged\"):\n rows = [json.loads(l) for l in (ROOT / f\"out_{split}.jsonl\").open()]\n print(f\"{split}: {len(rows)} rows; first ref = {rows[0]['reference_answer']}\")\n```\n\nOr with [`datasets`](https://huggingface.co/docs/datasets) once mirrored to the\nHuggingFace Hub:\n\n```python\nfrom datasets import load_dataset\nds = load_dataset(\"Berdyanskov/verified-math-reasoning\") # tentative path\n```\n\n## Schema\n\nEvery row in every split follows one unified schema:\n\n| field | type | meaning |\n|---|---|---|\n| `id` | str | unique row id, e.g. `numina-olympiads-0000523` |\n| `source` | str | seed dataset of origin (`numina:olympiads`) |\n| `problem` | str | problem statement (English, LaTeX) |\n| `reference_answer` | str | normalized numeric answer from NuminaMath |\n| `sol_a` / `sol_b` / `sol_c` | str | full CoT from DeepSeek / Doubao / Qwen3.5 respectively |\n| `ans_a` / `ans_b` / `ans_c` | str | extracted `\\boxed{}` answer from each model |\n| `correct_a` / `correct_b` / `correct_c` | bool | numerically equal to `reference_answer` |\n| `num_correct` | int | 0–3, the bucket key |\n| `difficulty` | str | `easy` / `hard` / `unsolved` (mirrors the split) |\n| `split` | str | `train_clean` / `hard_disagreement` / `unsolved_flagged` |\n| `synthetic` | bool | `true` for every published row — explicit AI-generated-content label per China's《人工智能生成合成内容标识办法》 |\n\n### Sample row (from `hard_disagreement.jsonl`)\n\n```json\n{\n \"id\": \"numina-olympiads-0000001\",\n \"source\": \"numina:olympiads\",\n \"problem\": \"Problem 2. Let n three-digit numbers satisfy ... Find the largest possible value of n.\",\n \"reference_answer\": \"5\",\n \"sol_a\": \"Alright, let's break this down ... \\\\boxed{7}\",\n \"sol_b\": \"### Step 1: Translate the problem ... \\\\boxed{5}\",\n \"sol_c\": \"We are given ... \\\\boxed{5}\",\n \"ans_a\": \"7\", \"ans_b\": \"5\", \"ans_c\": \"5\",\n \"correct_a\": false, \"correct_b\": true, \"correct_c\": true,\n \"num_correct\": 2,\n \"difficulty\": \"hard\",\n \"split\": \"hard_disagreement\"\n}\n```\n\n## Producing models\n\n| field | model | endpoint family | role |\n|---|---|---|---|\n| `sol_a` | DeepSeek-V4-Pro (`deepseek-reasoner`) | DeepSeek API or self-host vLLM | strong reasoner |\n| `sol_b` | `doubao-seed-2-0-pro-260215` | Volces ARK | strong reasoner (paid) |\n| `sol_c` | `Qwen3.5-397B-A17B`, `enable_thinking=False` | DashScope or self-host vLLM | fast non-reasoner — the heterogeneous third voice |\n\nChoosing one non-reasoning model as the third voice is a deliberate design choice; see [`DESIGN.md §3`](DESIGN.md) and [`CLAUDE.md`](CLAUDE.md) for why pure-reasoning triplets are noisier.\n\n## Reproduce\n\nFrom the CargoDash repo root:\n\n```bash\npip install -e \".[openai]\"\n\nexport DEEPSEEK_API_KEY=... # api.deepseek.com (or override DEEPSEEK_BASE_URL)\nexport DOUBAO_API_KEY=... # ark.cn-beijing.volces.com\nexport QWEN_API_KEY=... # dashscope.aliyuncs.com (or override QWEN_BASE_URL)\n\n# Small smoke test (3 rows, output to *.dryrun.jsonl, originals preserved):\npython -c \"\nimport sys; sys.path.insert(0, 'recipes/verified-math-reasoning')\nimport pipeline\npipeline.Pipeline(pipeline.source).run(dry_run_rows=3)\n\"\n\n# Full run (overwrites the existing v0.1 dataset):\npython recipes/verified-math-reasoning/pipeline.py\n```\n\nWall-time for v0.1 was **3h 39min** on the three SII-hosted vLLM + paid Doubao endpoints with `batch_size=16, intra_batch_workers=8`. Public-endpoint reproduction will be slower (rate limits on free tiers).\n\n### Need new seeds?\n\n```bash\n# Regenerate or extend the seed jsonl from NuminaMath-1.5 (HuggingFace):\npython recipes/verified-math-reasoning/load_numina.py \\\n --out _numina_1k.jsonl --limit 1000\n```\n\nFilters to \"numeric scalar answer\" by default; pass `--no-scalar-only` to keep symbolic / interval answers (verifier coverage there is partial).\n\n## What's verified, what isn't\n\n✅ **Verified by construction**\n- Every `correct_x` is the result of a programmatic numeric comparison (sympy-backed `numeric_equal` with unit-stripping for `^\\circ` / `\\%` / `°`), not model self-rating.\n- Every row has all three `sol_a/b/c` non-empty (3000/3000 model calls succeeded in the v0.1 run).\n- Each row appears in exactly one split.\n\n⚠️ **Caveats — read before training on this data**\n1. **Decontamination**: `decontam.py` checks each seed against MATH-500 (500 problems), GSM8K-test (1319), and AI-MO's AIME validation (90) — total 1909 problems / 69k 10-word shingles. Two-layer match: (a) normalized exact-string equality, (b) ≥3 shared 10-word shingles (catches paraphrases). Out of 1000 NuminaMath seeds, **1 contamination hit** was found and dropped (final size: **999**). Coverage is *not* research-grade — LLM-based paraphrase detection (Yang et al. 2023) and broader benchmark sets (OmniMath, OlympiadBench) are not implemented here. **This dataset is a CargoDash framework demo, not a benchmark-quality math eval** — if you need contamination-clean evaluation data, use a properly audited source.\n2. **`unsolved_flagged` is not curated.** A row landing there might be (a) genuinely too hard for all three models, (b) a `reference_answer` mismatch (NuminaMath sometimes encodes the answer in a non-standard form the verifier misses), or (c) extraction failure (model wrote the answer outside `\\boxed{}`). Manual spot-check before treating it as a benchmark.\n3. **Licensing is layered** — the seed text (NuminaMath-1.5) is Apache-2.0, the dataset itself is released under **CC-BY-NC-SA 4.0** (see [`LICENSE-DATA.txt`](LICENSE-DATA.txt)), and the model-generated `sol_a/b/c` traces are *additionally* constrained by each upstream provider's terms. Detailed clauses + per-column attribution live in [`NOTICE.md`](NOTICE.md) — **read it before any redistribution**.\n4. **Verifier coverage**: `numeric_equal` handles integers, decimals, fractions, `\\frac{}{}`, `\\sqrt{}`, `\\pi`, basic symbolic expressions via sympy, and unit suffixes (degrees, percent). Intervals (`[1, 2]`), vectors, set-valued answers, and proofs are out of scope in v0.1.\n\n## Upstream model terms\n\nThe three `sol_*` columns are AI-generated content; redistribution is governed by **both** this dataset's CC-BY-NC-SA license **and** each producing model's Terms of Service. The clauses we identified as material at release time:\n\n| field | model | provider | redistribute | train downstream models | special restriction |\n|---|---|---|---|---|---|\n| `sol_a` | DeepSeek-V4-Pro | [DeepSeek](https://cdn.deepseek.com/policies/en-US/deepseek-open-platform-terms-of-service.html) | ✅ | ✅ explicit, including distillation | none |\n| `sol_b` | Doubao-seed-2.0-pro | [Volcengine ARK](https://www.volcengine.com/docs/82379/1104498) | ✅ non-commercial | ✅ outside Volcengine platform | §3.1 mainland-China license + **written permission required for commercial use**; §3.7.12 AI-content labeling |\n| `sol_c` | Qwen3.5-397B-A17B | [Aliyun 百炼](https://terms.alicdn.com/legal-agreement/terms/common_platform_service/20230728213935489/20230728213935489.html) | ✅ (user owns §7.5) | ⚠️ **§4.6 prohibits training Qwen-competing products** without written permission | — |\n\nEach output row carries `synthetic: true` as the explicit AI-content label per China's《人工智能生成合成内容标识办法》. Full clause-level analysis + downstream-user guidance in [`NOTICE.md`](NOTICE.md).\n\n## v0.1 production stats\n\n- 1000 seeds → 1000 rows out of the pipeline (no data loss, no `JoinById` deadlocks); 1 contaminated row removed post-hoc → **999 published rows**\n- 13 transient retries across 3000 LLM calls (0 exhausted)\n- 0 rows with empty `sol_x`\n- 3h 39min total wall time\n- Cost: free for DeepSeek (self-host vLLM) and Qwen (self-host vLLM); ~$50 for Doubao API\n\n## Files in this directory\n\n| file | role |\n|---|---|\n| `pipeline.py` | The CargoDash pipeline that produced the dataset (~180 lines) |\n| `verifier.py` | `extract_boxed_or_number` + `numeric_equal` + decontam |\n| `load_numina.py` | Convert NuminaMath-1.5 (HF parquet) to seed jsonl |\n| `test_verifier.py` | Verifier unit tests + 1000-row round-trip sanity |\n| `_numina_1k.jsonl` | 1000 NuminaMath olympiad seeds (input) |\n| `_demo_seeds.jsonl` | 5-row hand-crafted demo seeds (offline mock testing) |\n| `out_train_clean.jsonl` | **dataset output** — 524 rows, all 3 models agreed |\n| `out_hard_disagreement.jsonl` | **dataset output** — 312 rows, partial agreement |\n| `out_unsolved_flagged.jsonl` | **dataset output** — 163 rows, no model correct |\n| `DESIGN.md` | Architecture & rationale (developer-facing) |\n| `CLAUDE.md` | Operational gotchas / why decisions were made (auto-loaded by Claude Code) |\n| `README.md` | This file (dataset-consumer-facing) |\n"}}}} |