{"repo": "alizen-1009/shapley-moe", "files": {"README.md": {"46ff4a16a3b6018d5f118247c3fa75f611648875": {"t": 1785747243.4524367, "n": 4830, "h": "5699e1ff632300a1", "c": "# SHAPE: Coalition-Aware Expert Pruning for Sparse MoE LLMs\n\n**SHAPE** (**SH**apley-**A**ware **P**runing of **E**xperts) is a training-free framework for pruning sparse Mixture-of-Experts (MoE) language models. It values experts inside the routed expert coalitions observed on a small calibration set, then preserves the most useful experts at a target keep rate.\n\n> **SHAPE: Coalition-Aware Expert Pruning for Sparse Mixture-of-Experts LLMs** (IJCNN 2026)\n>\n> [Paper](https://arxiv.org/abs/2606.09886) · [Quick start](docs/quickstart.md) · [Precomputed activations](https://github.com/Alizen-1009/Shapley-Moe/releases/tag/activation-data-v1) · [Project structure](PROJECT_STRUCTURE.md)\n\n<p align=\"center\">\n  <img src=\"assets/shape_overview.png\" alt=\"SHAPE workflow: collect routed expert coalitions, estimate contributions, and select experts\" width=\"95%\">\n</p>\n\n## Highlights\n\n- **Coalition-aware:** values experts by their marginal contribution to observed routed groups rather than in isolation.\n- **Training-free:** uses a calibration pass without gradient updates or expert retraining.\n- **Layer-aware:** preserves contribution mass per layer while matching a global keep rate.\n- **Architecture-preserving:** keeps the pretrained router and MoE structure unchanged.\n- **Broad evaluation:** supports three MoE families, eleven calibration datasets, and five pruning baselines.\n\n## Results\n\nThe reported experiments use 25 calibration examples per task. A keep rate of `0.8` retains 80% of experts; `0.6` retains 60%.\n\n| Model | Unpruned | Keep 80% | Keep 60% |\n| --- | ---: | ---: | ---: |\n| Qwen3-30B-A3B | 82.92 | 82.43 | 81.31 |\n| GPT-OSS-20B | 82.12 | 82.44 | 79.02 |\n| DeepSeek-V2-Lite | 62.08 | 62.44 | 58.81 |\n\nOn Qwen3-30B-A3B, SHAPE retains an average score of **81.31** at a 60% keep rate, compared with 79.05 for EASY-EP and 74.22 for the REAP/RAEP-style baseline. See the [paper](https://arxiv.org/abs/2606.09886) for per-task results and ablations.\n\n<p align=\"center\">\n  <img src=\"assets/shape_memory_vram.png\" alt=\"Peak VRAM at full size and at 20 and 40 percent expert pruning\" width=\"72%\">\n</p>\n\n## Quick Start\n\nThe shortest reproduction path uses the published activation snapshot and does not require running model inference:\n\n```bash\ngit clone git@github.com:Alizen-1009/Shapley-Moe.git\ncd Shapley-Moe\n\n./data/download_activations.sh\n./analysis/run_calc_shapley.sh -m qwen3-30b-a3b -d gsm8k_25\n./pruning/run_select.sh -m qwen3-30b-a3b -d gsm8k_25 -M shapley -r 0.8\n```\n\nFor environment setup, model configuration, the complete collect-to-evaluate pipeline, and adaptive LoRA recovery, see **[docs/quickstart.md](docs/quickstart.md)**.\n\n## How It Works\n\n```text\nCalibration data\n      ↓\nCollect routed expert coalitions\n      ↓\nEstimate per-layer Shapley contributions\n      ↓\nSelect experts by contribution coverage\n      ↓\nExport and evaluate the pruned model\n```\n\nThe recommended `alpha_per_layer` strategy keeps the smallest high-value expert set whose cumulative contribution reaches a per-layer threshold. Bisection adjusts that threshold until the requested global keep rate is met.\n\n## Supported Models\n\n| Config name | Experts | Experts/token | Family |\n| --- | ---: | ---: | --- |\n| `qwen3-30b-a3b` | 128 | 8 | Qwen3 MoE |\n| `gpt-oss-20b` | 32 | 4 | GPT-OSS MoE |\n| `deepseekv2-lite-coder` | 64 | 6 | DeepSeek-V2 MoE |\n\n## Pruning Methods\n\n| Method | Signal |\n| --- | --- |\n| `shapley` | Coalition-aware Shapley contribution (SHAPE) |\n| `easyep` | Task-aligned expert selection |\n| `reap` | Weighted activation norm |\n| `gating` | Mean router score |\n| `frequency` | Expert activation count |\n| `random` | Uniform random selection |\n\nDatasets, methods, keep rates, and evaluation defaults are defined in [`configs/experiments.yaml`](configs/experiments.yaml).\n\n## Repository Layout\n\n```text\nconfigs/      Model paths and experiment definitions\ndata/         Calibration data and download/distillation scripts\nanalysis/     Activation collection and Shapley scoring\npruning/      Expert selection and model export\nevaluation/   vLLM serving and EvalScope evaluation\nfinetune/     Optional adaptive-LoRA recovery\nresults/      Compact scores, selections, rank maps, and reports\n```\n\nGenerated activations, distilled data, adapters, and exported models are excluded from Git. Precomputed activations can be restored with `./data/download_activations.sh`.\n\n## Citation\n\n```bibtex\n@inproceedings{zhang2026shape,\n  title         = {SHAPE: Coalition-Aware Expert Pruning for Sparse Mixture-of-Experts LLMs},\n  author        = {Zhang, Yuhao and Jiang, HongXu and Zhang, YiXiang and Zhang, Zheng},\n  booktitle     = {Proceedings of the International Joint Conference on Neural Networks},\n  year          = {2026},\n  eprint        = {2606.09886},\n  archivePrefix = {arXiv},\n  primaryClass  = {cs.LG}\n}\n```\n"}, "566f7b61c1071f989a4f8b0177227626ac930988": {"t": 1785771672.7497911, "n": 4830, "h": "5699e1ff632300a1", "c": "# SHAPE: Coalition-Aware Expert Pruning for Sparse MoE LLMs\n\n**SHAPE** (**SH**apley-**A**ware **P**runing of **E**xperts) is a training-free framework for pruning sparse Mixture-of-Experts (MoE) language models. It values experts inside the routed expert coalitions observed on a small calibration set, then preserves the most useful experts at a target keep rate.\n\n> **SHAPE: Coalition-Aware Expert Pruning for Sparse Mixture-of-Experts LLMs** (IJCNN 2026)\n>\n> [Paper](https://arxiv.org/abs/2606.09886) · [Quick start](docs/quickstart.md) · [Precomputed activations](https://github.com/Alizen-1009/Shapley-Moe/releases/tag/activation-data-v1) · [Project structure](PROJECT_STRUCTURE.md)\n\n<p align=\"center\">\n  <img src=\"assets/shape_overview.png\" alt=\"SHAPE workflow: collect routed expert coalitions, estimate contributions, and select experts\" width=\"95%\">\n</p>\n\n## Highlights\n\n- **Coalition-aware:** values experts by their marginal contribution to observed routed groups rather than in isolation.\n- **Training-free:** uses a calibration pass without gradient updates or expert retraining.\n- **Layer-aware:** preserves contribution mass per layer while matching a global keep rate.\n- **Architecture-preserving:** keeps the pretrained router and MoE structure unchanged.\n- **Broad evaluation:** supports three MoE families, eleven calibration datasets, and five pruning baselines.\n\n## Results\n\nThe reported experiments use 25 calibration examples per task. A keep rate of `0.8` retains 80% of experts; `0.6` retains 60%.\n\n| Model | Unpruned | Keep 80% | Keep 60% |\n| --- | ---: | ---: | ---: |\n| Qwen3-30B-A3B | 82.92 | 82.43 | 81.31 |\n| GPT-OSS-20B | 82.12 | 82.44 | 79.02 |\n| DeepSeek-V2-Lite | 62.08 | 62.44 | 58.81 |\n\nOn Qwen3-30B-A3B, SHAPE retains an average score of **81.31** at a 60% keep rate, compared with 79.05 for EASY-EP and 74.22 for the REAP/RAEP-style baseline. See the [paper](https://arxiv.org/abs/2606.09886) for per-task results and ablations.\n\n<p align=\"center\">\n  <img src=\"assets/shape_memory_vram.png\" alt=\"Peak VRAM at full size and at 20 and 40 percent expert pruning\" width=\"72%\">\n</p>\n\n## Quick Start\n\nThe shortest reproduction path uses the published activation snapshot and does not require running model inference:\n\n```bash\ngit clone git@github.com:Alizen-1009/Shapley-Moe.git\ncd Shapley-Moe\n\n./data/download_activations.sh\n./analysis/run_calc_shapley.sh -m qwen3-30b-a3b -d gsm8k_25\n./pruning/run_select.sh -m qwen3-30b-a3b -d gsm8k_25 -M shapley -r 0.8\n```\n\nFor environment setup, model configuration, the complete collect-to-evaluate pipeline, and adaptive LoRA recovery, see **[docs/quickstart.md](docs/quickstart.md)**.\n\n## How It Works\n\n```text\nCalibration data\n      ↓\nCollect routed expert coalitions\n      ↓\nEstimate per-layer Shapley contributions\n      ↓\nSelect experts by contribution coverage\n      ↓\nExport and evaluate the pruned model\n```\n\nThe recommended `alpha_per_layer` strategy keeps the smallest high-value expert set whose cumulative contribution reaches a per-layer threshold. Bisection adjusts that threshold until the requested global keep rate is met.\n\n## Supported Models\n\n| Config name | Experts | Experts/token | Family |\n| --- | ---: | ---: | --- |\n| `qwen3-30b-a3b` | 128 | 8 | Qwen3 MoE |\n| `gpt-oss-20b` | 32 | 4 | GPT-OSS MoE |\n| `deepseekv2-lite-coder` | 64 | 6 | DeepSeek-V2 MoE |\n\n## Pruning Methods\n\n| Method | Signal |\n| --- | --- |\n| `shapley` | Coalition-aware Shapley contribution (SHAPE) |\n| `easyep` | Task-aligned expert selection |\n| `reap` | Weighted activation norm |\n| `gating` | Mean router score |\n| `frequency` | Expert activation count |\n| `random` | Uniform random selection |\n\nDatasets, methods, keep rates, and evaluation defaults are defined in [`configs/experiments.yaml`](configs/experiments.yaml).\n\n## Repository Layout\n\n```text\nconfigs/      Model paths and experiment definitions\ndata/         Calibration data and download/distillation scripts\nanalysis/     Activation collection and Shapley scoring\npruning/      Expert selection and model export\nevaluation/   vLLM serving and EvalScope evaluation\nfinetune/     Optional adaptive-LoRA recovery\nresults/      Compact scores, selections, rank maps, and reports\n```\n\nGenerated activations, distilled data, adapters, and exported models are excluded from Git. Precomputed activations can be restored with `./data/download_activations.sh`.\n\n## Citation\n\n```bibtex\n@inproceedings{zhang2026shape,\n  title         = {SHAPE: Coalition-Aware Expert Pruning for Sparse Mixture-of-Experts LLMs},\n  author        = {Zhang, Yuhao and Jiang, HongXu and Zhang, YiXiang and Zhang, Zheng},\n  booktitle     = {Proceedings of the International Joint Conference on Neural Networks},\n  year          = {2026},\n  eprint        = {2606.09886},\n  archivePrefix = {arXiv},\n  primaryClass  = {cs.LG}\n}\n```\n"}, "eaf58ee32b80908eccad2c14635f462757cb1a45": {"t": 1785771680.493537, "n": 10761, "h": "548cc6bb7a0c1fb9", "c": "# SHAPE: Coalition-Aware Expert Pruning for Sparse MoE LLMs\n\n**SHAPE** (**SH**apley-**A**ware **P**runing of **E**xperts) is a training-free framework for pruning experts from sparse Mixture-of-Experts (MoE) language models. It estimates each expert's value inside the routed expert coalitions observed on a small calibration set, then preserves the most useful experts at a target keep rate.\n\nThis repository accompanies the IJCNN 2026 paper:\n\n> **SHAPE: Coalition-Aware Expert Pruning for Sparse Mixture-of-Experts LLMs**\n>\n> [Paper](https://arxiv.org/abs/2606.09886) · [Precomputed activations](https://github.com/Alizen-1009/Shapley-Moe/releases/tag/activation-data-v1) · [Detailed project structure](PROJECT_STRUCTURE.md)\n\n<p align=\"center\">\n  <img src=\"assets/shape_overview.png\" alt=\"SHAPE workflow: collect routed expert coalitions, estimate contributions, and select experts\" width=\"95%\">\n</p>\n\n## Why SHAPE?\n\nSparse MoE models activate only a few experts per token, but the entire expert pool must normally remain resident in memory. Existing pruning methods often score experts independently using routing frequency, router probability, or activation magnitude. SHAPE instead models the top-k experts selected for each token as a **coalition**.\n\n- **Coalition-aware:** values experts by their marginal contribution to observed routed groups.\n- **Training-free:** requires a calibration pass, not gradient updates or expert retraining.\n- **Layer-aware:** preserves Shapley contribution mass per layer while matching a global keep rate.\n- **Architecture-preserving:** keeps the pretrained router and MoE structure unchanged.\n- **Configurable:** supports SHAPE and five pruning baselines across three MoE families.\n\n## Results\n\nThe reported experiments use 25 calibration examples per task and evaluate seven tasks: GSM8K, HumanEval, GPQA-Diamond, MATH-500, TruthfulQA, OntoNotes5, and MedMCQA.\n\nA keep rate of `0.8` retains 80% of experts (20% pruning); `0.6` retains 60% (40% pruning).\n\n| Model | Unpruned | Keep 80% | Keep 60% |\n| --- | ---: | ---: | ---: |\n| Qwen3-30B-A3B | 82.92 | 82.43 | 81.31 |\n| GPT-OSS-20B | 82.12 | 82.44 | 79.02 |\n| DeepSeek-V2-Lite | 62.08 | 62.44 | 58.81 |\n\nOn Qwen3-30B-A3B, SHAPE retains an average score of **81.31** at a 60% keep rate, compared with 79.05 for EASY-EP and 74.22 for the REAP/RAEP-style baseline. See the [paper](https://arxiv.org/abs/2606.09886) for per-task results, ablations, and full baseline comparisons.\n\n<p align=\"center\">\n  <img src=\"assets/shape_memory_vram.png\" alt=\"Peak VRAM at full size and at 20 and 40 percent expert pruning\" width=\"72%\">\n</p>\n\n## Installation\n\n### Requirements\n\n- Python 3.12 (the provided environment was captured with Python 3.12)\n- Linux and a CUDA-capable GPU for model loading, activation collection, export, and evaluation\n- Local Hugging Face checkpoints for the models you want to prune\n\nCreate an environment and install the dependencies:\n\n```bash\ngit clone git@github.com:Alizen-1009/Shapley-Moe.git\ncd Shapley-Moe\n\npython3.12 -m venv .venv\nsource .venv/bin/activate\npip install -r requirements.txt\n```\n\n`torch` and `vllm` wheels are CUDA-specific. If the pinned versions do not match your CUDA installation, install compatible wheels first and then install the remaining dependencies. Use `requirements-lock.txt` when you need the exact captured environment rather than the curated dependency list.\n\n## Quick Start\n\n### Option A: Reproduce selection from published activations\n\nThis path skips model inference and downloads the activation snapshot used by the repository. The archive is approximately 34 MB compressed and 228 MB extracted.\n\n```bash\n# Download and checksum-verify results/*/activations/.\n./data/download_activations.sh\n\n# Recompute Shapley values for one model and dataset.\n./analysis/run_calc_shapley.sh \\\n  --model qwen3-30b-a3b \\\n  --dataset gsm8k_25\n\n# Select experts at an 80% keep rate.\n./pruning/run_select.sh \\\n  --model qwen3-30b-a3b \\\n  --dataset gsm8k_25 \\\n  --method shapley \\\n  --rate 0.8\n```\n\nThe downloaded activations are ignored by Git. Compact derived artifacts such as Shapley CSV files and selected-expert JSON files remain versioned in `results/`.\n\n### Option B: Run the full pipeline\n\n#### 1. Configure model paths\n\nEdit [`configs/models.yaml`](configs/models.yaml) so each model points to a local checkpoint. Experiment defaults, datasets, methods, and keep rates are defined in [`configs/experiments.yaml`](configs/experiments.yaml).\n\n```yaml\nmodels:\n  qwen3-30b-a3b:\n    path: /path/to/Qwen3-30B-A3B\n    num_experts: 128\n    num_experts_per_tok: 8\n    type: qwen3\n```\n\n#### 2. Prepare calibration data\n\nThe repository already includes the 25-example calibration sets used in the experiments. To regenerate one dataset or all configured datasets:\n\n```bash\n./data/run_download.sh gsm8k 25\n./data/run_download.sh --all\n```\n\n#### 3. Collect routing and activation statistics\n\n```bash\n# Run every calibration dataset for a configured model.\n./analysis/run_collect.sh --model qwen3-30b-a3b --all\n\n# Or use an explicit checkpoint and one calibration file.\n./analysis/run_collect.sh \\\n  --model /path/to/model \\\n  --data data/calibration/gsm8k_25.json\n```\n\nOutput: `results/{model}/activations/`\n\n#### 4. Compute Shapley values\n\n```bash\n# One dataset.\n./analysis/run_calc_shapley.sh \\\n  --model qwen3-30b-a3b \\\n  --dataset gsm8k_25\n\n# Every available activation file for the model.\n./analysis/run_calc_shapley.sh --model qwen3-30b-a3b --all\n```\n\nOutput: `results/{model}/shapley_values/`\n\n#### 5. Select experts\n\n```bash\n# Recommended SHAPE strategy.\n./pruning/run_select.sh \\\n  --model qwen3-30b-a3b \\\n  --dataset gsm8k_25 \\\n  --method shapley \\\n  --strategy alpha_per_layer \\\n  --rate 0.8\n\n# Run every configured dataset, method, and keep rate.\n./pruning/run_select.sh \\\n  --model qwen3-30b-a3b \\\n  --all-datasets \\\n  --all-methods \\\n  --all-rates\n```\n\nOutput: `results/{model}/selected_experts/`\n\n#### 6. Export a pruned model\n\n```bash\n./pruning/run_prune.sh \\\n  --model qwen3-30b-a3b \\\n  --dataset gsm8k_25 \\\n  --method shapley \\\n  --strategy alpha_per_layer \\\n  --rate 0.8\n```\n\nUse `./pruning/run_prune.sh --help` to choose an output directory, device map, or pruning implementation (`zero_weights`, `gate_bias`, or `both`).\n\n#### 7. Serve and evaluate\n\n```bash\n# Start a vLLM server for an exported model selected by its experiment metadata.\nbash evaluation/vllm-server.sh \\\n  --model qwen3-30b-a3b \\\n  --method shapley \\\n  --dataset gsm8k_25 \\\n  --rate 0.8 \\\n  --port 8801 \\\n  --tp 8\n\n# In another shell, run the configured EvalScope evaluation.\npython evaluation/run_evalscope.py\n```\n\nRun any wrapper with `--help` to see all options and defaults.\n\n## Pipeline Artifacts\n\n| Stage | Input | Output |\n| --- | --- | --- |\n| Calibration | Dataset samples | `data/calibration/{dataset}_25.json` |\n| Collection | Model + calibration data | `results/{model}/activations/*.json` |\n| Scoring | Activation statistics | `results/{model}/shapley_values/*.csv` |\n| Selection | Scores + keep rate | `results/{model}/selected_experts/*.json` |\n| Export | Model + selected experts | Pruned model directory |\n| Evaluation | Served model | EvalScope results |\n\nActivation statistics, distilled training data, adapters, and exported models are generated artifacts and are excluded from Git.\n\n## Supported Models\n\n| Config name | Experts | Experts per token | Family |\n| --- | ---: | ---: | --- |\n| `qwen3-30b-a3b` | 128 | 8 | Qwen3 MoE |\n| `gpt-oss-20b` | 32 | 4 | GPT-OSS MoE |\n| `deepseekv2-lite-coder` | 64 | 6 | DeepSeek-V2 MoE |\n\nThe configured calibration datasets cover code, mathematics, reasoning, medical QA, NLP, and truthfulness. See [`configs/experiments.yaml`](configs/experiments.yaml) for the authoritative list.\n\n## Methods and Selection Strategies\n\n### Pruning methods\n\n| Method | Signal |\n| --- | --- |\n| `shapley` | Coalition-aware Shapley-style contribution (SHAPE) |\n| `easyep` | Task-aligned expert selection |\n| `reap` | Weighted activation norm |\n| `gating` | Mean router softmax score |\n| `frequency` | Expert activation count |\n| `random` | Uniform random selection |\n\n### SHAPE strategies\n\n| Strategy | Behavior |\n| --- | --- |\n| `alpha_per_layer` | Preserves an alpha fraction of Shapley mass per layer; recommended |\n| `alpha_global` | Applies contribution coverage globally |\n| `topk_per_layer` | Keeps a fixed number of experts in every layer |\n| `topk_global` | Keeps the globally highest-scoring experts |\n\n## Optional: Adaptive LoRA Recovery\n\nThe `finetune/` pipeline allocates different LoRA ranks to retained experts according to their Shapley contribution. It supports distilled SFT, an optional DPO stage, matched uniform/random rank baselines, and adapter merging.\n\nTypical entry points:\n\n```bash\n# Distill SFT and DPO data from the original teacher model.\nTEACHER_MODEL=/path/to/original-model TP=8 NUM_SAMPLES=4 \\\n  ./data/download_sft.sh gsm8k\n\n# Run the configured {keep rate} x {rank strategy} SFT matrix.\n./finetune/run_experiments.sh\n\n# Optionally run DPO after SFT.\n./finetune/run_dpo.sh\n```\n\nSee [`finetune/PLAN.md`](finetune/PLAN.md) and [`PROJECT_STRUCTURE.md`](PROJECT_STRUCTURE.md) for rank-map generation, individual training commands, merging, and experimental details.\n\n## Repository Layout\n\n```text\nShapley-Moe/\n├── configs/       # Model paths and experiment definitions\n├── data/          # Calibration data and data download/distillation scripts\n├── analysis/      # Activation collection and Shapley scoring\n├── pruning/       # Expert selection methods and model export\n├── evaluation/    # vLLM serving and EvalScope evaluation\n├── finetune/      # Optional adaptive-LoRA recovery pipeline\n├── results/       # Compact scores, selections, rank maps, and reports\n└── assets/        # README figures\n```\n\nFor a file-by-file map, see [`PROJECT_STRUCTURE.md`](PROJECT_STRUCTURE.md).\n\n## Conventions\n\n- `--rate` is a **keep rate**, not a removal rate: `0.8` keeps 80% and prunes 20%.\n- Dataset names include the calibration size, for example `gsm8k_25`.\n- The recommended SHAPE selection strategy is `alpha_per_layer`.\n- The default calibration size is 25 examples per task.\n- Activation collection requires the original unpruned model; scoring and selection can use the published activation snapshot.\n\n## Citation\n\nIf you use SHAPE, please cite:\n\n```bibtex\n@inproceedings{zhang2026shape,\n  title         = {SHAPE: Coalition-Aware Expert Pruning for Sparse Mixture-of-Experts LLMs},\n  author        = {Zhang, Yuhao and Jiang, HongXu and Zhang, YiXiang and Zhang, Zheng},\n  booktitle     = {Proceedings of the International Joint Conference on Neural Networks},\n  year          = {2026},\n  eprint        = {2606.09886},\n  archivePrefix = {arXiv},\n  primaryClass  = {cs.LG}\n}\n```\n"}, "fb78b0dc13e82916d06efc8192640a7ae0c48a84": {"t": 1785771682.9197187, "n": 18510, "h": "15b82f0c13c8d6b5", "c": "# SHAPE: Coalition-Aware Expert Pruning for Sparse MoE LLMs\n\n**SHAPE** (**SH**apley-**A**ware **P**runing of **E**xperts) is a training-free pruning framework for sparse Mixture-of-Experts (MoE) large language models. It reduces the number of resident experts needed at inference time while preserving the cooperative routing structure that makes MoE models effective.\n\nThis repository accompanies the IJCNN 2026 work:\n\n> **SHAPE: Coalition-Aware Expert Pruning for Sparse Mixture-of-Experts LLMs**\n> [[arXiv:2606.09886]](https://arxiv.org/abs/2606.09886)\n\nSparse MoE models activate only a few experts per token, but the full expert pool still has to stay in GPU memory for dynamic routing. SHAPE targets this memory wall by pruning redundant experts after pretraining, without changing router logic, retraining experts, or modifying the model architecture.\n\n<p align=\"center\">\n  <img src=\"assets/shape_overview.png\" alt=\"Overview of SHAPE workflow\" width=\"95%\">\n</p>\n\n\n## Highlights\n\n- **Coalition-aware expert valuation.** SHAPE treats routed top-k experts as a coalition instead of scoring each expert independently.\n- **Training-free pruning.** It uses only a small calibration set and routing traces from the original model.\n- **Layer-aware quality coverage.** It keeps the smallest expert subset that preserves enough Shapley contribution mass per layer, then uses bisection to match a target global keep rate.\n- **Modern MoE backbones.** The experiments cover Qwen3-30B-A3B, GPT-OSS-20B, and DeepSeek-V2-Lite.\n- **Practical memory savings.** Removing 20% or 40% of experts reduces peak VRAM while keeping task performance stable across reasoning, coding, QA, and NLP benchmarks.\n\n## What SHAPE Does\n\nMost expert pruning methods score experts in isolation, using signals such as routing frequency, router probability, or activation magnitude. That misses a key property of sparse MoE inference: each token is processed by a **coalition** of routed experts. An expert can be valuable not because it appears often, but because it completes a high-utility expert group.\n\nSHAPE reframes pruning as a cooperative attribution problem:\n\n1. Run the unpruned model on a small task-specific calibration set.\n2. Record top-k expert combinations at every MoE layer.\n3. Build an empirical cooperative game from observed expert coalitions.\n4. Estimate Shapley-style contribution scores for experts inside each layer.\n5. Select experts with a quality-coverage rule that preserves layer-wise contribution mass.\n6. Export a compact MoE model for serving.\n\n<p align=\"center\">\n  <img src=\"assets/shape_cover.png\" alt=\"SHAPE cover illustration\" width=\"92%\">\n</p>\n\n## Method Overview\n\nGiven a pretrained sparse MoE model and a downstream task, SHAPE performs an offline calibration pass. For every calibration token, it records which experts are co-activated by the router in each MoE layer. These traces define the observed coalition support for that task.\n\nFor each expert, SHAPE estimates how much marginal value it contributes inside the coalitions where it appears. The approximation uses routing frequency and co-occurrence statistics, avoiding expensive repeated forward passes over many masked expert subsets.\n\nAfter scoring, SHAPE applies **Shapley Quality Coverage**. Instead of keeping a fixed number of experts in every layer, it keeps the smallest prefix of high-value experts whose cumulative contribution reaches an alpha fraction of that layer's Shapley mass. The alpha threshold is chosen by bisection so the final retained expert count matches the target keep rate.\n\n\n\n## Results at a Glance\n\nSHAPE is evaluated on three sparse MoE backbones and seven benchmarks: GSM8K, HumanEval, GPQA-Diamond, MATH-500, TruthfulQA, OntoNotes5, and MedMCQA. Each task uses 25 calibration examples to collect routing traces.\n\n### Accuracy Under Expert Pruning\n\nThe table below reports task-average accuracy from the IJCNN 2026 experiments. A keep rate of `0.80` means 20% of experts are pruned; a keep rate of `0.60` means 40% of experts are pruned.\n\n| Model | Baseline | 20% Pruned | 40% Pruned |\n| --- | ---: | ---: | ---: |\n| Qwen3-30B-A3B | 82.92 | 82.43 | 81.31 |\n| GPT-OSS-20B | 82.12 | 82.44 | 79.02 |\n| DeepSeek-V2-Lite | 62.08 | 62.44 | 58.81 |\n\nAt 20% pruning, average performance remains comparable to the unpruned models and can slightly improve on some backbones, suggesting that part of the expert pool has limited marginal utility for the evaluated tasks. At 40% pruning, degradation increases but remains bounded, especially on larger MoE backbones.\n\n### Baseline Comparison on Qwen3-30B-A3B\n\nSHAPE is strongest under aggressive pruning, where preserving critical expert coalitions matters most.\n\n| Rate | Method | GSM8K | HumanEval | GPQA-D | MATH-500 | TruthfulQA | OntoNotes5 | MedMCQA | Avg |\n| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |\n| - | Unpruned | 96.21 | 95.73 | 58.59 | 96.80 | 77.60 | 87.15 | 68.35 | 82.92 |\n| 20% | Random | 61.24 | 58.36 | 21.42 | 60.27 | 42.31 | 71.16 | 38.18 | 50.42 |\n| 20% | Frequency | 86.12 | 86.37 | 40.12 | 88.14 | 64.05 | 82.17 | 58.12 | 72.16 |\n| 20% | Gating | 87.42 | 87.15 | 42.18 | 89.36 | 65.41 | 83.24 | 59.27 | 73.43 |\n| 20% | RAEP | 95.47 | 94.52 | 57.56 | 95.42 | 76.42 | 86.14 | 67.51 | 81.86 |\n| 20% | EASY-EP | 95.82 | 95.01 | 58.12 | 95.86 | 77.14 | 86.54 | 67.96 | 82.35 |\n| 20% | **SHAPE** | **96.74** | **95.12** | **60.10** | **96.80** | 75.64 | 84.42 | **68.18** | **82.43** |\n| 40% | Random | 35.43 | 28.27 | 8.28 | 32.15 | 21.29 | 60.41 | 15.46 | 28.76 |\n| 40% | Frequency | 75.14 | 72.11 | 20.16 | 72.29 | 45.18 | 76.28 | 40.17 | 57.33 |\n| 40% | Gating | 77.19 | 74.32 | 22.54 | 74.28 | 47.21 | 77.31 | 42.15 | 59.29 |\n| 40% | RAEP | 93.42 | 90.47 | 52.42 | 92.58 | 43.12 | 83.15 | 64.41 | 74.22 |\n| 40% | EASY-EP | 94.12 | **91.07** | 53.12 | 93.14 | **73.28** | **83.51** | 65.14 | 79.05 |\n| 40% | **SHAPE** | **95.60** | 89.63 | **67.68** | **94.40** | 73.16 | 82.37 | **66.32** | **81.31** |\n\n### Peak VRAM\n\nPruning directly reduces the resident expert footprint. The measured peak VRAM decreases steadily under 20% and 40% expert pruning for both 32K and 8K context settings, with no additional training or architecture changes.\n\n<p align=\"center\">\n  <img src=\"assets/shape_memory_vram.png\" alt=\"Peak VRAM under 0, 20, and 40 percent expert pruning\" width=\"78%\">\n</p>\n\n## Repository Structure\n\n```text\nshapley-moe/\n|-- data/                         # Calibration data, distillation, download scripts\n|   |-- calibration/              # 25-example calibration sets (for Shapley)\n|   |-- sft/                      # Distilled SFT/DPO training data (generated)\n|   |-- download_dataset.py\n|   |-- run_download.sh\n|   |-- distill_sft.py            # Teacher distillation + RFT (SFT and DPO data)\n|   `-- download_sft.sh           # Distillation pipeline entry\n|-- analysis/                     # Routing trace collection and Shapley scoring\n|   |-- collect_activations.py\n|   |-- calc_shapley.py\n|   |-- run_collect.sh\n|   `-- run_calc_shapley.sh\n|-- pruning/                      # Expert selection and pruned-model export\n|   |-- methods/\n|   |   |-- select_by_shapley.py\n|   |   |-- select_by_easyep.py\n|   |   |-- select_by_reap.py\n|   |   |-- select_by_gating.py\n|   |   |-- select_by_frequency.py\n|   |   `-- select_by_random.py\n|   |-- save_model.py\n|   |-- run_select.sh\n|   `-- run_prune.sh\n|-- finetune/                     # Post-pruning adaptive LoRA fine-tuning\n|   |-- build_rank_map.py         # Shapley-guided per-expert LoRA ranks\n|   |-- train_adaptive_lora.py    # SFT stage (adaptive LoRA)\n|   |-- train_dpo_lora.py         # Optional DPO stage (reference = adapters off)\n|   |-- packed_qwen3_lora.py      # Packed-expert LoRA implementation for Qwen3\n|   |-- merge_lora.py\n|   |-- infer_adaptive_lora.py\n|   |-- run_experiments.sh        # SFT matrix: {rate} x {bucket,uniform,random}\n|   `-- run_dpo.sh                # DPO matrix orchestration\n|-- evaluation/                   # vLLM serving and EvalScope evaluation\n|-- configs/                      # Model paths and experiment settings\n|-- results/                      # Activations, Shapley values, selected experts\n|-- assets/                       # README figures\n`-- PROJECT_STRUCTURE.md\n```\n\n## Quick Start\n\n### 0. Configure Models and Experiments\n\nEdit the config files before running the pipeline:\n\n```bash\nvim configs/models.yaml\nvim configs/experiments.yaml\n```\n\n`configs/models.yaml` defines model paths, expert counts, top-k routing size, and MoE module patterns. `configs/experiments.yaml` defines calibration datasets, keep rates, pruning methods, and evaluation defaults.\n\n### 1. Prepare Calibration Data\n\n```bash\ncd data\n\n# Download one dataset with 25 examples.\n./run_download.sh gsm8k 25\n\n# Or download all datasets from configs/experiments.yaml.\n./run_download.sh --all\n```\n\n### 2. Collect Routing and Activation Statistics\n\nThis single pass collects the information needed by SHAPE and the baseline pruning methods.\n\n```bash\ncd analysis\n\n# Use a configured model name.\n./run_collect.sh -m qwen3-30b-a3b --all\n\n# Or use a full model path.\n./run_collect.sh -m /path/to/model --data ../data/calibration/gsm8k_25.json\n\n# Inspect configured models.\n./run_collect.sh --list-models\n```\n\nOutputs are written to:\n\n```text\nresults/{model_name}/activations/\n```\n\nActivation statistics are generated artifacts and are not stored in Git. To\nreproduce the published results without collecting them again, download the\nprecomputed snapshot (~34 MB compressed, ~228 MB extracted):\n\n```bash\n./data/download_activations.sh\n```\n\nThe script verifies the archive checksum and restores the files under\n`results/{model_name}/activations/`.\n\n### 3. Compute Shapley Values\n\n```bash\ncd analysis\n\n# Compute Shapley scores for all available activation files.\n./run_calc_shapley.sh -m qwen3-30b-a3b\n\n# Or compute one dataset.\n./run_calc_shapley.sh -m qwen3-30b-a3b -d gsm8k_25\n```\n\nOutputs are written to:\n\n```text\nresults/{model_name}/shapley_values/\n```\n\n### 4. Select Experts\n\n```bash\ncd pruning\n\n# SHAPE selection with 80% keep rate, i.e. 20% expert pruning.\n./run_select.sh -m qwen3-30b-a3b -d gsm8k_25 -M shapley -r 0.8\n\n# Run all configured keep rates.\n./run_select.sh -m qwen3-30b-a3b -d gsm8k_25 -M shapley --all-rates\n\n# Run all datasets and all configured methods.\n./run_select.sh -m qwen3-30b-a3b --all-datasets --all-methods --all-rates\n```\n\nSelected expert files are written to:\n\n```text\nresults/{model_name}/selected_experts/\n```\n\n### 5. Export a Pruned Model\n\n```bash\ncd pruning\n\n# Export a SHAPE-pruned model from the selected experts.\n./run_prune.sh -m qwen3-30b-a3b -d gsm8k_25 -r 0.8\n\n# Use another method or keep rate.\n./run_prune.sh -m gpt-oss-20b -d arc_easy_25 -M easyep -r 0.6\n```\n\nThe pruning script finds the matching selected-expert JSON and writes a pruned Hugging Face model directory.\n\n### 6. Optional: Adaptive LoRA Fine-Tuning\n\nThe `finetune/` scripts implement a post-pruning LoRA recovery path. The current minimal experimental loop targets Qwen3-30B-A3B on `gsm8k_25` with SHAPE-selected experts. The goal is to *recover* the capability lost by pruning, measured as a recovery rate against the unpruned model.\n\n#### 6a. Distill Training Data\n\nThe 25-example calibration sets are for Shapley estimation, **not** for training. Training data is distilled from the **original (unpruned) teacher model**: it answers the full training split, and rejection sampling (RFT) keeps only completions whose final answer matches the gold answer. Using the original model (rather than a stronger external model) keeps the recovery-rate narrative clean and the data distribution aligned with the student.\n\n```bash\ncd data\n\n# Download the full train-split question pool, then distill via vLLM with RFT.\n# Produces data/sft/gsm8k_distill.json (SFT) and data/sft/gsm8k_dpo.json (DPO).\nTEACHER_MODEL=/path/to/original_qwen3 TP=8 NUM_SAMPLES=4 ./download_sft.sh gsm8k\n\n# Quick smoke test on 50 questions.\nMAX_QUESTIONS=50 ./download_sft.sh gsm8k\n```\n\nA single sampling pass yields both datasets: correct completions become SFT targets; correct-vs-incorrect completions for the same prompt become DPO preference pairs.\n\n#### 6b. Build Rank Map, Train (SFT), Merge\n\nBuild a LoRA rank map from Shapley scores and selected experts:\n\n```bash\npython finetune/build_rank_map.py \\\n  --shapley_csv results/qwen3-30b-a3b/shapley_values/gsm8k_25_shapley.csv \\\n  --selected_experts results/qwen3-30b-a3b/selected_experts/shapley_alpha_per_layer_gsm8k_25_rate0_8.json \\\n  --output results/qwen3-30b-a3b/lora_rank_maps/gsm8k_25_rate0_8_bucket.json \\\n  --strategy bucket\n```\n\nDefault adaptive rank allocation is layer-wise and only over retained experts:\n\n```text\nTop 20% retained experts -> rank 32\nNext 40% retained experts -> rank 16\nLast 40% retained experts -> rank 8\n```\n\nThe matched baselines are:\n\n```text\nuniform: all retained experts use rank 16\nrandom: same bucket sizes and rank set as bucket, assigned randomly\n```\n\nRank maps omit pruned experts. During LoRA training, `train_adaptive_lora.py` expands the rank map into full expert module paths and passes only those modules to PEFT, so zeroed/pruned experts do not receive LoRA parameters.\n\nTrain an adapter on a pruned model using the distilled SFT data:\n\n```bash\npython finetune/train_adaptive_lora.py \\\n  --model_path /path/to/pruned_qwen3_model \\\n  --rank_map results/qwen3-30b-a3b/lora_rank_maps/gsm8k_25_rate0_8_bucket.json \\\n  --train_file data/sft/gsm8k_distill.json \\\n  --output_dir adapters/qwen3_gsm8k_rate0_8_bucket \\\n  --model_type qwen3 \\\n  --bf16\n```\n\nMerge the adapter before serving or formal evaluation:\n\n```bash\npython finetune/merge_lora.py \\\n  --base_model /path/to/pruned_qwen3_model \\\n  --adapter adapters/qwen3_gsm8k_rate0_8_bucket \\\n  --output /path/to/merged_qwen3_gsm8k_rate0_8_bucket\n```\n\nThe whole `{rate} x {bucket, uniform, random}` matrix (rank map -> train -> merge) can be run in one command:\n\n```bash\ncd finetune && ./run_experiments.sh\n```\n\n#### 6c. Optional: DPO Stage\n\nDPO is an optional second stage applied **after** SFT. The recommended recipe is `SFT -> merge -> DPO -> merge`, so the DPO base model is the SFT-merged model. The DPO reference distribution is obtained by **disabling** the (new) LoRA adapters, which recovers the SFT model without loading a second copy. The same rank map drives LoRA placement, keeping the C/D/E comparison fair.\n\n```bash\n# Train a DPO LoRA on the SFT-merged model, then merge, for the full matrix.\ncd finetune && ./run_dpo.sh\n```\n\nNotes:\n- Always run SFT before DPO; DPO from scratch underperforms.\n- Apply DPO uniformly to all rank strategies (bucket/uniform/random) to keep the comparison fair.\n- A strong teacher rarely errs on easy tasks, so DPO negatives are scarce; raise `NUM_SAMPLES`/`TEMPERATURE` or use harder data to surface more preference pairs.\n\n### 7. Evaluate\n\n```bash\ncd evaluation\n\n# Start a vLLM server from a configured model.\n./vllm-server.sh qwen3-30b-a3b\n\n# Or use a full model path and custom port.\n./vllm-server.sh /path/to/model -p 8801\n\n# Run EvalScope evaluation using configs/experiments.yaml.\npython run_evalscope.py\n```\n\n## Supported Models and Datasets\n\n### Models\n\n| Model name | Experts | Experts per token | Type |\n| --- | ---: | ---: | --- |\n| `qwen3-30b-a3b` | 128 | 8 | Qwen3 MoE |\n| `gpt-oss-20b` | 32 | 4 | GPT-OSS MoE |\n| `deepseekv2-lite-coder` | 64 | 6 | DeepSeek-V2 MoE |\n\n### Calibration and Evaluation Tasks\n\nThe default config includes calibration sets for code, math, reasoning, medical QA, NLP, and truthfulness:\n\n```text\nhumaneval_25, gsm8k_25, math_500_25, arc_easy_25, logiqa_25,\ngpqa_diamond_25, med_mcqa_25, pubmedqa_25, biomix_qa_25,\nontonotes5_25, truthful_qa_25\n```\n\nThe IJCNN 2026 main results report GSM8K, HumanEval, GPQA-Diamond, MATH-500, TruthfulQA, OntoNotes5, and MedMCQA.\n\n## Pruning Methods\n\n| Method | Signal | Description |\n| --- | --- | --- |\n| `shapley` | Coalition-aware Shapley-style contribution | Primary SHAPE method |\n| `easyep` | Task-aligned expert selection | EASY-EP baseline |\n| `reap` | Weighted activation norm | REAP/RAEP-style baseline |\n| `gating` | Mean router softmax score | Router-score baseline |\n| `frequency` | Activation count | Routing-frequency baseline |\n| `random` | Uniform random selection | Lower-bound baseline |\n\n## SHAPE Selection Strategies\n\n| Strategy | Description | Notes |\n| --- | --- | --- |\n| `alpha_per_layer` | Preserve an alpha fraction of Shapley mass in each layer | Recommended; used for SHAPE quality coverage |\n| `alpha_global` | Preserve alpha coverage globally | Can over-prune some layers |\n| `topk_per_layer` | Keep a fixed top-k expert count in every layer | Simple layer-wise baseline |\n| `topk_global` | Keep the globally highest-scoring experts | Can cause layer imbalance |\n\nIn the paper, the quality-coverage variant (`alpha_per_layer`) is the main SHAPE setting. It outperforms simplified global and rigid layer-wise variants because it preserves cooperative value while avoiding layer collapse.\n\n## Naming Conventions\n\n- Models: `qwen3-30b-a3b`, `gpt-oss-20b`, `deepseekv2-lite-coder`\n- Datasets: `{dataset}_{samples}`, for example `gsm8k_25`\n- Methods: `shapley`, `easyep`, `reap`, `gating`, `frequency`, `random`\n- Keep rates: `0.8` keeps 80% of experts and prunes 20%; `0.6` keeps 60% and prunes 40%\n- Selected expert files:\n  - SHAPE: `shapley_{strategy}_{dataset}_rate{XX}.json`\n  - Baselines: `{method}_{dataset}_rate{XX}.json`\n- LoRA rank maps: `{dataset}_rate{XX}_{rank_strategy}.json`, where `rank_strategy` is `bucket`, `uniform`, or `random`\n- Distilled training data: `data/sft/{dataset}_distill.json` (SFT), `data/sft/{dataset}_dpo.json` (DPO preference pairs)\n\n## Citation\n\nIf you use this repository, please cite:\n\n```bibtex\n@inproceedings{zhang2026shape,\n  title     = {SHAPE: Coalition-Aware Expert Pruning for Sparse Mixture-of-Experts LLMs},\n  author    = {Zhang, Yuhao and Jiang, HongXu and Zhang, YiXiang and Zhang, Zheng},\n  booktitle = {Proceedings of the International Joint Conference on Neural Networks},\n  year      = {2026},\n  eprint    = {2606.09886},\n  archivePrefix = {arXiv},\n  primaryClass = {cs.LG}\n}\n```\n\n## Notes\n\n- SHAPE is post-training and does not require gradient updates.\n- The default calibration size in this repository is 25 examples per task.\n- The `-r` argument in scripts is a **keep rate**, not a removal rate.\n- Large models require local Hugging Face checkpoints and sufficient GPU memory for the unpruned calibration pass.\n"}, "2154046a8111600d7c607be110a5cccc0c9808c3": {"t": 1785771684.7397723, "n": 18144, "h": "903b9dc9b8aa3ff2", "c": "# SHAPE: Coalition-Aware Expert Pruning for Sparse MoE LLMs\n\n**SHAPE** (**SH**apley-**A**ware **P**runing of **E**xperts) is a training-free pruning framework for sparse Mixture-of-Experts (MoE) large language models. It reduces the number of resident experts needed at inference time while preserving the cooperative routing structure that makes MoE models effective.\n\nThis repository accompanies the IJCNN 2026 work:\n\n> **SHAPE: Coalition-Aware Expert Pruning for Sparse Mixture-of-Experts LLMs**\n> [[arXiv:2606.09886]](https://arxiv.org/abs/2606.09886)\n\nSparse MoE models activate only a few experts per token, but the full expert pool still has to stay in GPU memory for dynamic routing. SHAPE targets this memory wall by pruning redundant experts after pretraining, without changing router logic, retraining experts, or modifying the model architecture.\n\n<p align=\"center\">\n  <img src=\"assets/shape_overview.png\" alt=\"Overview of SHAPE workflow\" width=\"95%\">\n</p>\n\n\n## Highlights\n\n- **Coalition-aware expert valuation.** SHAPE treats routed top-k experts as a coalition instead of scoring each expert independently.\n- **Training-free pruning.** It uses only a small calibration set and routing traces from the original model.\n- **Layer-aware quality coverage.** It keeps the smallest expert subset that preserves enough Shapley contribution mass per layer, then uses bisection to match a target global keep rate.\n- **Modern MoE backbones.** The experiments cover Qwen3-30B-A3B, GPT-OSS-20B, and DeepSeek-V2-Lite.\n- **Practical memory savings.** Removing 20% or 40% of experts reduces peak VRAM while keeping task performance stable across reasoning, coding, QA, and NLP benchmarks.\n\n## What SHAPE Does\n\nMost expert pruning methods score experts in isolation, using signals such as routing frequency, router probability, or activation magnitude. That misses a key property of sparse MoE inference: each token is processed by a **coalition** of routed experts. An expert can be valuable not because it appears often, but because it completes a high-utility expert group.\n\nSHAPE reframes pruning as a cooperative attribution problem:\n\n1. Run the unpruned model on a small task-specific calibration set.\n2. Record top-k expert combinations at every MoE layer.\n3. Build an empirical cooperative game from observed expert coalitions.\n4. Estimate Shapley-style contribution scores for experts inside each layer.\n5. Select experts with a quality-coverage rule that preserves layer-wise contribution mass.\n6. Export a compact MoE model for serving.\n\n<p align=\"center\">\n  <img src=\"assets/shape_cover.png\" alt=\"SHAPE cover illustration\" width=\"92%\">\n</p>\n\n## Method Overview\n\nGiven a pretrained sparse MoE model and a downstream task, SHAPE performs an offline calibration pass. For every calibration token, it records which experts are co-activated by the router in each MoE layer. These traces define the observed coalition support for that task.\n\nFor each expert, SHAPE estimates how much marginal value it contributes inside the coalitions where it appears. The approximation uses routing frequency and co-occurrence statistics, avoiding expensive repeated forward passes over many masked expert subsets.\n\nAfter scoring, SHAPE applies **Shapley Quality Coverage**. Instead of keeping a fixed number of experts in every layer, it keeps the smallest prefix of high-value experts whose cumulative contribution reaches an alpha fraction of that layer's Shapley mass. The alpha threshold is chosen by bisection so the final retained expert count matches the target keep rate.\n\n\n\n## Results at a Glance\n\nSHAPE is evaluated on three sparse MoE backbones and seven benchmarks: GSM8K, HumanEval, GPQA-Diamond, MATH-500, TruthfulQA, OntoNotes5, and MedMCQA. Each task uses 25 calibration examples to collect routing traces.\n\n### Accuracy Under Expert Pruning\n\nThe table below reports task-average accuracy from the IJCNN 2026 experiments. A keep rate of `0.80` means 20% of experts are pruned; a keep rate of `0.60` means 40% of experts are pruned.\n\n| Model | Baseline | 20% Pruned | 40% Pruned |\n| --- | ---: | ---: | ---: |\n| Qwen3-30B-A3B | 82.92 | 82.43 | 81.31 |\n| GPT-OSS-20B | 82.12 | 82.44 | 79.02 |\n| DeepSeek-V2-Lite | 62.08 | 62.44 | 58.81 |\n\nAt 20% pruning, average performance remains comparable to the unpruned models and can slightly improve on some backbones, suggesting that part of the expert pool has limited marginal utility for the evaluated tasks. At 40% pruning, degradation increases but remains bounded, especially on larger MoE backbones.\n\n### Baseline Comparison on Qwen3-30B-A3B\n\nSHAPE is strongest under aggressive pruning, where preserving critical expert coalitions matters most.\n\n| Rate | Method | GSM8K | HumanEval | GPQA-D | MATH-500 | TruthfulQA | OntoNotes5 | MedMCQA | Avg |\n| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |\n| - | Unpruned | 96.21 | 95.73 | 58.59 | 96.80 | 77.60 | 87.15 | 68.35 | 82.92 |\n| 20% | Random | 61.24 | 58.36 | 21.42 | 60.27 | 42.31 | 71.16 | 38.18 | 50.42 |\n| 20% | Frequency | 86.12 | 86.37 | 40.12 | 88.14 | 64.05 | 82.17 | 58.12 | 72.16 |\n| 20% | Gating | 87.42 | 87.15 | 42.18 | 89.36 | 65.41 | 83.24 | 59.27 | 73.43 |\n| 20% | RAEP | 95.47 | 94.52 | 57.56 | 95.42 | 76.42 | 86.14 | 67.51 | 81.86 |\n| 20% | EASY-EP | 95.82 | 95.01 | 58.12 | 95.86 | 77.14 | 86.54 | 67.96 | 82.35 |\n| 20% | **SHAPE** | **96.74** | **95.12** | **60.10** | **96.80** | 75.64 | 84.42 | **68.18** | **82.43** |\n| 40% | Random | 35.43 | 28.27 | 8.28 | 32.15 | 21.29 | 60.41 | 15.46 | 28.76 |\n| 40% | Frequency | 75.14 | 72.11 | 20.16 | 72.29 | 45.18 | 76.28 | 40.17 | 57.33 |\n| 40% | Gating | 77.19 | 74.32 | 22.54 | 74.28 | 47.21 | 77.31 | 42.15 | 59.29 |\n| 40% | RAEP | 93.42 | 90.47 | 52.42 | 92.58 | 43.12 | 83.15 | 64.41 | 74.22 |\n| 40% | EASY-EP | 94.12 | **91.07** | 53.12 | 93.14 | **73.28** | **83.51** | 65.14 | 79.05 |\n| 40% | **SHAPE** | **95.60** | 89.63 | **67.68** | **94.40** | 73.16 | 82.37 | **66.32** | **81.31** |\n\n### Peak VRAM\n\nPruning directly reduces the resident expert footprint. The measured peak VRAM decreases steadily under 20% and 40% expert pruning for both 32K and 8K context settings, with no additional training or architecture changes.\n\n<p align=\"center\">\n  <img src=\"assets/shape_memory_vram.png\" alt=\"Peak VRAM under 0, 20, and 40 percent expert pruning\" width=\"78%\">\n</p>\n\n## Repository Structure\n\n```text\nshapley-moe/\n|-- data/                         # Calibration data, distillation, download scripts\n|   |-- calibration/              # 25-example calibration sets (for Shapley)\n|   |-- sft/                      # Distilled SFT/DPO training data (generated)\n|   |-- download_dataset.py\n|   |-- run_download.sh\n|   |-- distill_sft.py            # Teacher distillation + RFT (SFT and DPO data)\n|   `-- download_sft.sh           # Distillation pipeline entry\n|-- analysis/                     # Routing trace collection and Shapley scoring\n|   |-- collect_activations.py\n|   |-- calc_shapley.py\n|   |-- run_collect.sh\n|   `-- run_calc_shapley.sh\n|-- pruning/                      # Expert selection and pruned-model export\n|   |-- methods/\n|   |   |-- select_by_shapley.py\n|   |   |-- select_by_easyep.py\n|   |   |-- select_by_reap.py\n|   |   |-- select_by_gating.py\n|   |   |-- select_by_frequency.py\n|   |   `-- select_by_random.py\n|   |-- save_model.py\n|   |-- run_select.sh\n|   `-- run_prune.sh\n|-- finetune/                     # Post-pruning adaptive LoRA fine-tuning\n|   |-- build_rank_map.py         # Shapley-guided per-expert LoRA ranks\n|   |-- train_adaptive_lora.py    # SFT stage (adaptive LoRA)\n|   |-- train_dpo_lora.py         # Optional DPO stage (reference = adapters off)\n|   |-- packed_qwen3_lora.py      # Packed-expert LoRA implementation for Qwen3\n|   |-- merge_lora.py\n|   |-- infer_adaptive_lora.py\n|   |-- run_experiments.sh        # SFT matrix: {rate} x {bucket,uniform,random}\n|   `-- run_dpo.sh                # DPO matrix orchestration\n|-- evaluation/                   # vLLM serving and EvalScope evaluation\n|-- configs/                      # Model paths and experiment settings\n|-- results/                      # Activations, Shapley values, selected experts\n|-- assets/                       # README figures\n`-- PROJECT_STRUCTURE.md\n```\n\n## Quick Start\n\n### 0. Configure Models and Experiments\n\nEdit the config files before running the pipeline:\n\n```bash\nvim configs/models.yaml\nvim configs/experiments.yaml\n```\n\n`configs/models.yaml` defines model paths, expert counts, top-k routing size, and MoE module patterns. `configs/experiments.yaml` defines calibration datasets, keep rates, pruning methods, and evaluation defaults.\n\n### 1. Prepare Calibration Data\n\n```bash\ncd data\n\n# Download one dataset with 25 examples.\n./run_download.sh gsm8k 25\n\n# Or download all datasets from configs/experiments.yaml.\n./run_download.sh --all\n```\n\n### 2. Collect Routing and Activation Statistics\n\nThis single pass collects the information needed by SHAPE and the baseline pruning methods.\n\n```bash\ncd analysis\n\n# Use a configured model name.\n./run_collect.sh -m qwen3-30b-a3b --all\n\n# Or use a full model path.\n./run_collect.sh -m /path/to/model --data ../data/calibration/gsm8k_25.json\n\n# Inspect configured models.\n./run_collect.sh --list-models\n```\n\nOutputs are written to:\n\n```text\nresults/{model_name}/activations/\n```\n\n### 3. Compute Shapley Values\n\n```bash\ncd analysis\n\n# Compute Shapley scores for all available activation files.\n./run_calc_shapley.sh -m qwen3-30b-a3b\n\n# Or compute one dataset.\n./run_calc_shapley.sh -m qwen3-30b-a3b -d gsm8k_25\n```\n\nOutputs are written to:\n\n```text\nresults/{model_name}/shapley_values/\n```\n\n### 4. Select Experts\n\n```bash\ncd pruning\n\n# SHAPE selection with 80% keep rate, i.e. 20% expert pruning.\n./run_select.sh -m qwen3-30b-a3b -d gsm8k_25 -M shapley -r 0.8\n\n# Run all configured keep rates.\n./run_select.sh -m qwen3-30b-a3b -d gsm8k_25 -M shapley --all-rates\n\n# Run all datasets and all configured methods.\n./run_select.sh -m qwen3-30b-a3b --all-datasets --all-methods --all-rates\n```\n\nSelected expert files are written to:\n\n```text\nresults/{model_name}/selected_experts/\n```\n\n### 5. Export a Pruned Model\n\n```bash\ncd pruning\n\n# Export a SHAPE-pruned model from the selected experts.\n./run_prune.sh -m qwen3-30b-a3b -d gsm8k_25 -r 0.8\n\n# Use another method or keep rate.\n./run_prune.sh -m gpt-oss-20b -d arc_easy_25 -M easyep -r 0.6\n```\n\nThe pruning script finds the matching selected-expert JSON and writes a pruned Hugging Face model directory.\n\n### 6. Optional: Adaptive LoRA Fine-Tuning\n\nThe `finetune/` scripts implement a post-pruning LoRA recovery path. The current minimal experimental loop targets Qwen3-30B-A3B on `gsm8k_25` with SHAPE-selected experts. The goal is to *recover* the capability lost by pruning, measured as a recovery rate against the unpruned model.\n\n#### 6a. Distill Training Data\n\nThe 25-example calibration sets are for Shapley estimation, **not** for training. Training data is distilled from the **original (unpruned) teacher model**: it answers the full training split, and rejection sampling (RFT) keeps only completions whose final answer matches the gold answer. Using the original model (rather than a stronger external model) keeps the recovery-rate narrative clean and the data distribution aligned with the student.\n\n```bash\ncd data\n\n# Download the full train-split question pool, then distill via vLLM with RFT.\n# Produces data/sft/gsm8k_distill.json (SFT) and data/sft/gsm8k_dpo.json (DPO).\nTEACHER_MODEL=/path/to/original_qwen3 TP=8 NUM_SAMPLES=4 ./download_sft.sh gsm8k\n\n# Quick smoke test on 50 questions.\nMAX_QUESTIONS=50 ./download_sft.sh gsm8k\n```\n\nA single sampling pass yields both datasets: correct completions become SFT targets; correct-vs-incorrect completions for the same prompt become DPO preference pairs.\n\n#### 6b. Build Rank Map, Train (SFT), Merge\n\nBuild a LoRA rank map from Shapley scores and selected experts:\n\n```bash\npython finetune/build_rank_map.py \\\n  --shapley_csv results/qwen3-30b-a3b/shapley_values/gsm8k_25_shapley.csv \\\n  --selected_experts results/qwen3-30b-a3b/selected_experts/shapley_alpha_per_layer_gsm8k_25_rate0_8.json \\\n  --output results/qwen3-30b-a3b/lora_rank_maps/gsm8k_25_rate0_8_bucket.json \\\n  --strategy bucket\n```\n\nDefault adaptive rank allocation is layer-wise and only over retained experts:\n\n```text\nTop 20% retained experts -> rank 32\nNext 40% retained experts -> rank 16\nLast 40% retained experts -> rank 8\n```\n\nThe matched baselines are:\n\n```text\nuniform: all retained experts use rank 16\nrandom: same bucket sizes and rank set as bucket, assigned randomly\n```\n\nRank maps omit pruned experts. During LoRA training, `train_adaptive_lora.py` expands the rank map into full expert module paths and passes only those modules to PEFT, so zeroed/pruned experts do not receive LoRA parameters.\n\nTrain an adapter on a pruned model using the distilled SFT data:\n\n```bash\npython finetune/train_adaptive_lora.py \\\n  --model_path /path/to/pruned_qwen3_model \\\n  --rank_map results/qwen3-30b-a3b/lora_rank_maps/gsm8k_25_rate0_8_bucket.json \\\n  --train_file data/sft/gsm8k_distill.json \\\n  --output_dir adapters/qwen3_gsm8k_rate0_8_bucket \\\n  --model_type qwen3 \\\n  --bf16\n```\n\nMerge the adapter before serving or formal evaluation:\n\n```bash\npython finetune/merge_lora.py \\\n  --base_model /path/to/pruned_qwen3_model \\\n  --adapter adapters/qwen3_gsm8k_rate0_8_bucket \\\n  --output /path/to/merged_qwen3_gsm8k_rate0_8_bucket\n```\n\nThe whole `{rate} x {bucket, uniform, random}` matrix (rank map -> train -> merge) can be run in one command:\n\n```bash\ncd finetune && ./run_experiments.sh\n```\n\n#### 6c. Optional: DPO Stage\n\nDPO is an optional second stage applied **after** SFT. The recommended recipe is `SFT -> merge -> DPO -> merge`, so the DPO base model is the SFT-merged model. The DPO reference distribution is obtained by **disabling** the (new) LoRA adapters, which recovers the SFT model without loading a second copy. The same rank map drives LoRA placement, keeping the C/D/E comparison fair.\n\n```bash\n# Train a DPO LoRA on the SFT-merged model, then merge, for the full matrix.\ncd finetune && ./run_dpo.sh\n```\n\nNotes:\n- Always run SFT before DPO; DPO from scratch underperforms.\n- Apply DPO uniformly to all rank strategies (bucket/uniform/random) to keep the comparison fair.\n- A strong teacher rarely errs on easy tasks, so DPO negatives are scarce; raise `NUM_SAMPLES`/`TEMPERATURE` or use harder data to surface more preference pairs.\n\n### 7. Evaluate\n\n```bash\ncd evaluation\n\n# Start a vLLM server from a configured model.\n./vllm-server.sh qwen3-30b-a3b\n\n# Or use a full model path and custom port.\n./vllm-server.sh /path/to/model -p 8801\n\n# Run EvalScope evaluation using configs/experiments.yaml.\npython run_evalscope.py\n```\n\n## Supported Models and Datasets\n\n### Models\n\n| Model name | Experts | Experts per token | Type |\n| --- | ---: | ---: | --- |\n| `qwen3-30b-a3b` | 128 | 8 | Qwen3 MoE |\n| `gpt-oss-20b` | 32 | 4 | GPT-OSS MoE |\n| `deepseekv2-lite-coder` | 64 | 6 | DeepSeek-V2 MoE |\n\n### Calibration and Evaluation Tasks\n\nThe default config includes calibration sets for code, math, reasoning, medical QA, NLP, and truthfulness:\n\n```text\nhumaneval_25, gsm8k_25, math_500_25, arc_easy_25, logiqa_25,\ngpqa_diamond_25, med_mcqa_25, pubmedqa_25, biomix_qa_25,\nontonotes5_25, truthful_qa_25\n```\n\nThe IJCNN 2026 main results report GSM8K, HumanEval, GPQA-Diamond, MATH-500, TruthfulQA, OntoNotes5, and MedMCQA.\n\n## Pruning Methods\n\n| Method | Signal | Description |\n| --- | --- | --- |\n| `shapley` | Coalition-aware Shapley-style contribution | Primary SHAPE method |\n| `easyep` | Task-aligned expert selection | EASY-EP baseline |\n| `reap` | Weighted activation norm | REAP/RAEP-style baseline |\n| `gating` | Mean router softmax score | Router-score baseline |\n| `frequency` | Activation count | Routing-frequency baseline |\n| `random` | Uniform random selection | Lower-bound baseline |\n\n## SHAPE Selection Strategies\n\n| Strategy | Description | Notes |\n| --- | --- | --- |\n| `alpha_per_layer` | Preserve an alpha fraction of Shapley mass in each layer | Recommended; used for SHAPE quality coverage |\n| `alpha_global` | Preserve alpha coverage globally | Can over-prune some layers |\n| `topk_per_layer` | Keep a fixed top-k expert count in every layer | Simple layer-wise baseline |\n| `topk_global` | Keep the globally highest-scoring experts | Can cause layer imbalance |\n\nIn the paper, the quality-coverage variant (`alpha_per_layer`) is the main SHAPE setting. It outperforms simplified global and rigid layer-wise variants because it preserves cooperative value while avoiding layer collapse.\n\n## Naming Conventions\n\n- Models: `qwen3-30b-a3b`, `gpt-oss-20b`, `deepseekv2-lite-coder`\n- Datasets: `{dataset}_{samples}`, for example `gsm8k_25`\n- Methods: `shapley`, `easyep`, `reap`, `gating`, `frequency`, `random`\n- Keep rates: `0.8` keeps 80% of experts and prunes 20%; `0.6` keeps 60% and prunes 40%\n- Selected expert files:\n  - SHAPE: `shapley_{strategy}_{dataset}_rate{XX}.json`\n  - Baselines: `{method}_{dataset}_rate{XX}.json`\n- LoRA rank maps: `{dataset}_rate{XX}_{rank_strategy}.json`, where `rank_strategy` is `bucket`, `uniform`, or `random`\n- Distilled training data: `data/sft/{dataset}_distill.json` (SFT), `data/sft/{dataset}_dpo.json` (DPO preference pairs)\n\n## Citation\n\nIf you use this repository, please cite:\n\n```bibtex\n@inproceedings{zhang2026shape,\n  title     = {SHAPE: Coalition-Aware Expert Pruning for Sparse Mixture-of-Experts LLMs},\n  author    = {Zhang, Yuhao and Jiang, HongXu and Zhang, YiXiang and Zhang, Zheng},\n  booktitle = {Proceedings of the International Joint Conference on Neural Networks},\n  year      = {2026},\n  eprint    = {2606.09886},\n  archivePrefix = {arXiv},\n  primaryClass = {cs.LG}\n}\n```\n\n## Notes\n\n- SHAPE is post-training and does not require gradient updates.\n- The default calibration size in this repository is 25 examples per task.\n- The `-r` argument in scripts is a **keep rate**, not a removal rate.\n- Large models require local Hugging Face checkpoints and sufficient GPU memory for the unpruned calibration pass.\n"}, "3bc0b6667b5dbd05b80487bbc999a0b8f76ed76d": {"t": 1785771686.6529107, "n": 18006, "h": "0adad8de2a49eb6d", "c": "# SHAPE: Coalition-Aware Expert Pruning for Sparse MoE LLMs\n\n**SHAPE** (**SH**apley-**A**ware **P**runing of **E**xperts) is a training-free pruning framework for sparse Mixture-of-Experts (MoE) large language models. It reduces the number of resident experts needed at inference time while preserving the cooperative routing structure that makes MoE models effective.\n\nThis repository accompanies the IJCNN 2026 work:\n\n> **SHAPE: Coalition-Aware Expert Pruning for Sparse Mixture-of-Experts LLMs**\n\nSparse MoE models activate only a few experts per token, but the full expert pool still has to stay in GPU memory for dynamic routing. SHAPE targets this memory wall by pruning redundant experts after pretraining, without changing router logic, retraining experts, or modifying the model architecture.\n\n<p align=\"center\">\n  <img src=\"assets/shape_cover.png\" alt=\"SHAPE cover illustration\" width=\"92%\">\n</p>\n\n\n## Highlights\n\n- **Coalition-aware expert valuation.** SHAPE treats routed top-k experts as a coalition instead of scoring each expert independently.\n- **Training-free pruning.** It uses only a small calibration set and routing traces from the original model.\n- **Layer-aware quality coverage.** It keeps the smallest expert subset that preserves enough Shapley contribution mass per layer, then uses bisection to match a target global keep rate.\n- **Modern MoE backbones.** The experiments cover Qwen3-30B-A3B, GPT-OSS-20B, and DeepSeek-V2-Lite.\n- **Practical memory savings.** Removing 20% or 40% of experts reduces peak VRAM while keeping task performance stable across reasoning, coding, QA, and NLP benchmarks.\n\n## What SHAPE Does\n\nMost expert pruning methods score experts in isolation, using signals such as routing frequency, router probability, or activation magnitude. That misses a key property of sparse MoE inference: each token is processed by a **coalition** of routed experts. An expert can be valuable not because it appears often, but because it completes a high-utility expert group.\n\nSHAPE reframes pruning as a cooperative attribution problem:\n\n1. Run the unpruned model on a small task-specific calibration set.\n2. Record top-k expert combinations at every MoE layer.\n3. Build an empirical cooperative game from observed expert coalitions.\n4. Estimate Shapley-style contribution scores for experts inside each layer.\n5. Select experts with a quality-coverage rule that preserves layer-wise contribution mass.\n6. Export a compact MoE model for serving.\n\n<p align=\"center\">\n  <img src=\"assets/shape_overview.png\" alt=\"Overview of SHAPE workflow\" width=\"95%\">\n</p>\n\n## Method Overview\n\nGiven a pretrained sparse MoE model and a downstream task, SHAPE performs an offline calibration pass. For every calibration token, it records which experts are co-activated by the router in each MoE layer. These traces define the observed coalition support for that task.\n\nFor each expert, SHAPE estimates how much marginal value it contributes inside the coalitions where it appears. The approximation uses routing frequency and co-occurrence statistics, avoiding expensive repeated forward passes over many masked expert subsets.\n\nAfter scoring, SHAPE applies **Shapley Quality Coverage**. Instead of keeping a fixed number of experts in every layer, it keeps the smallest prefix of high-value experts whose cumulative contribution reaches an alpha fraction of that layer's Shapley mass. The alpha threshold is chosen by bisection so the final retained expert count matches the target keep rate.\n\n\n\n## Results at a Glance\n\nSHAPE is evaluated on three sparse MoE backbones and seven benchmarks: GSM8K, HumanEval, GPQA-Diamond, MATH-500, TruthfulQA, OntoNotes5, and MedMCQA. Each task uses 25 calibration examples to collect routing traces.\n\n### Accuracy Under Expert Pruning\n\nThe table below reports task-average accuracy from the IJCNN 2026 experiments. A keep rate of `0.80` means 20% of experts are pruned; a keep rate of `0.60` means 40% of experts are pruned.\n\n| Model | Baseline | 20% Pruned | 40% Pruned |\n| --- | ---: | ---: | ---: |\n| Qwen3-30B-A3B | 82.92 | 82.43 | 81.31 |\n| GPT-OSS-20B | 82.12 | 82.44 | 79.02 |\n| DeepSeek-V2-Lite | 62.08 | 62.44 | 58.81 |\n\nAt 20% pruning, average performance remains comparable to the unpruned models and can slightly improve on some backbones, suggesting that part of the expert pool has limited marginal utility for the evaluated tasks. At 40% pruning, degradation increases but remains bounded, especially on larger MoE backbones.\n\n### Baseline Comparison on Qwen3-30B-A3B\n\nSHAPE is strongest under aggressive pruning, where preserving critical expert coalitions matters most.\n\n| Rate | Method | GSM8K | HumanEval | GPQA-D | MATH-500 | TruthfulQA | OntoNotes5 | MedMCQA | Avg |\n| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |\n| - | Unpruned | 96.21 | 95.73 | 58.59 | 96.80 | 77.60 | 87.15 | 68.35 | 82.92 |\n| 20% | Random | 61.24 | 58.36 | 21.42 | 60.27 | 42.31 | 71.16 | 38.18 | 50.42 |\n| 20% | Frequency | 86.12 | 86.37 | 40.12 | 88.14 | 64.05 | 82.17 | 58.12 | 72.16 |\n| 20% | Gating | 87.42 | 87.15 | 42.18 | 89.36 | 65.41 | 83.24 | 59.27 | 73.43 |\n| 20% | RAEP | 95.47 | 94.52 | 57.56 | 95.42 | 76.42 | 86.14 | 67.51 | 81.86 |\n| 20% | EASY-EP | 95.82 | 95.01 | 58.12 | 95.86 | 77.14 | 86.54 | 67.96 | 82.35 |\n| 20% | **SHAPE** | **96.74** | **95.12** | **60.10** | **96.80** | 75.64 | 84.42 | **68.18** | **82.43** |\n| 40% | Random | 35.43 | 28.27 | 8.28 | 32.15 | 21.29 | 60.41 | 15.46 | 28.76 |\n| 40% | Frequency | 75.14 | 72.11 | 20.16 | 72.29 | 45.18 | 76.28 | 40.17 | 57.33 |\n| 40% | Gating | 77.19 | 74.32 | 22.54 | 74.28 | 47.21 | 77.31 | 42.15 | 59.29 |\n| 40% | RAEP | 93.42 | 90.47 | 52.42 | 92.58 | 43.12 | 83.15 | 64.41 | 74.22 |\n| 40% | EASY-EP | 94.12 | **91.07** | 53.12 | 93.14 | **73.28** | **83.51** | 65.14 | 79.05 |\n| 40% | **SHAPE** | **95.60** | 89.63 | **67.68** | **94.40** | 73.16 | 82.37 | **66.32** | **81.31** |\n\n### Peak VRAM\n\nPruning directly reduces the resident expert footprint. The measured peak VRAM decreases steadily under 20% and 40% expert pruning for both 32K and 8K context settings, with no additional training or architecture changes.\n\n<p align=\"center\">\n  <img src=\"assets/shape_memory_vram.png\" alt=\"Peak VRAM under 0, 20, and 40 percent expert pruning\" width=\"78%\">\n</p>\n\n## Repository Structure\n\n```text\nshapley-moe/\n|-- data/                         # Calibration data, distillation, download scripts\n|   |-- calibration/              # 25-example calibration sets (for Shapley)\n|   |-- sft/                      # Distilled SFT/DPO training data (generated)\n|   |-- download_dataset.py\n|   |-- run_download.sh\n|   |-- distill_sft.py            # Teacher distillation + RFT (SFT and DPO data)\n|   `-- download_sft.sh           # Distillation pipeline entry\n|-- analysis/                     # Routing trace collection and Shapley scoring\n|   |-- collect_activations.py\n|   |-- calc_shapley.py\n|   |-- run_collect.sh\n|   `-- run_calc_shapley.sh\n|-- pruning/                      # Expert selection and pruned-model export\n|   |-- methods/\n|   |   |-- select_by_shapley.py\n|   |   |-- select_by_easyep.py\n|   |   |-- select_by_reap.py\n|   |   |-- select_by_gating.py\n|   |   |-- select_by_frequency.py\n|   |   `-- select_by_random.py\n|   |-- save_model.py\n|   |-- run_select.sh\n|   `-- run_prune.sh\n|-- finetune/                     # Post-pruning adaptive LoRA fine-tuning\n|   |-- build_rank_map.py         # Shapley-guided per-expert LoRA ranks\n|   |-- train_adaptive_lora.py    # SFT stage (adaptive LoRA)\n|   |-- train_dpo_lora.py         # Optional DPO stage (reference = adapters off)\n|   |-- packed_qwen3_lora.py      # Packed-expert LoRA implementation for Qwen3\n|   |-- merge_lora.py\n|   |-- infer_adaptive_lora.py\n|   |-- run_experiments.sh        # SFT matrix: {rate} x {bucket,uniform,random}\n|   `-- run_dpo.sh                # DPO matrix orchestration\n|-- evaluation/                   # vLLM serving and EvalScope evaluation\n|-- configs/                      # Model paths and experiment settings\n|-- results/                      # Activations, Shapley values, selected experts\n|-- assets/                       # README figures\n`-- PROJECT_STRUCTURE.md\n```\n\n## Quick Start\n\n### 0. Configure Models and Experiments\n\nEdit the config files before running the pipeline:\n\n```bash\nvim configs/models.yaml\nvim configs/experiments.yaml\n```\n\n`configs/models.yaml` defines model paths, expert counts, top-k routing size, and MoE module patterns. `configs/experiments.yaml` defines calibration datasets, keep rates, pruning methods, and evaluation defaults.\n\n### 1. Prepare Calibration Data\n\n```bash\ncd data\n\n# Download one dataset with 25 examples.\n./run_download.sh gsm8k 25\n\n# Or download all datasets from configs/experiments.yaml.\n./run_download.sh --all\n```\n\n### 2. Collect Routing and Activation Statistics\n\nThis single pass collects the information needed by SHAPE and the baseline pruning methods.\n\n```bash\ncd analysis\n\n# Use a configured model name.\n./run_collect.sh -m qwen3-30b-a3b --all\n\n# Or use a full model path.\n./run_collect.sh -m /path/to/model --data ../data/calibration/gsm8k_25.json\n\n# Inspect configured models.\n./run_collect.sh --list-models\n```\n\nOutputs are written to:\n\n```text\nresults/{model_name}/activations/\n```\n\n### 3. Compute Shapley Values\n\n```bash\ncd analysis\n\n# Compute Shapley scores for all available activation files.\n./run_calc_shapley.sh -m qwen3-30b-a3b\n\n# Or compute one dataset.\n./run_calc_shapley.sh -m qwen3-30b-a3b -d gsm8k_25\n```\n\nOutputs are written to:\n\n```text\nresults/{model_name}/shapley_values/\n```\n\n### 4. Select Experts\n\n```bash\ncd pruning\n\n# SHAPE selection with 80% keep rate, i.e. 20% expert pruning.\n./run_select.sh -m qwen3-30b-a3b -d gsm8k_25 -M shapley -r 0.8\n\n# Run all configured keep rates.\n./run_select.sh -m qwen3-30b-a3b -d gsm8k_25 -M shapley --all-rates\n\n# Run all datasets and all configured methods.\n./run_select.sh -m qwen3-30b-a3b --all-datasets --all-methods --all-rates\n```\n\nSelected expert files are written to:\n\n```text\nresults/{model_name}/selected_experts/\n```\n\n### 5. Export a Pruned Model\n\n```bash\ncd pruning\n\n# Export a SHAPE-pruned model from the selected experts.\n./run_prune.sh -m qwen3-30b-a3b -d gsm8k_25 -r 0.8\n\n# Use another method or keep rate.\n./run_prune.sh -m gpt-oss-20b -d arc_easy_25 -M easyep -r 0.6\n```\n\nThe pruning script finds the matching selected-expert JSON and writes a pruned Hugging Face model directory.\n\n### 6. Optional: Adaptive LoRA Fine-Tuning\n\nThe `finetune/` scripts implement a post-pruning LoRA recovery path. The current minimal experimental loop targets Qwen3-30B-A3B on `gsm8k_25` with SHAPE-selected experts. The goal is to *recover* the capability lost by pruning, measured as a recovery rate against the unpruned model.\n\n#### 6a. Distill Training Data\n\nThe 25-example calibration sets are for Shapley estimation, **not** for training. Training data is distilled from the **original (unpruned) teacher model**: it answers the full training split, and rejection sampling (RFT) keeps only completions whose final answer matches the gold answer. Using the original model (rather than a stronger external model) keeps the recovery-rate narrative clean and the data distribution aligned with the student.\n\n```bash\ncd data\n\n# Download the full train-split question pool, then distill via vLLM with RFT.\n# Produces data/sft/gsm8k_distill.json (SFT) and data/sft/gsm8k_dpo.json (DPO).\nTEACHER_MODEL=/path/to/original_qwen3 TP=8 NUM_SAMPLES=4 ./download_sft.sh gsm8k\n\n# Quick smoke test on 50 questions.\nMAX_QUESTIONS=50 ./download_sft.sh gsm8k\n```\n\nA single sampling pass yields both datasets: correct completions become SFT targets; correct-vs-incorrect completions for the same prompt become DPO preference pairs.\n\n#### 6b. Build Rank Map, Train (SFT), Merge\n\nBuild a LoRA rank map from Shapley scores and selected experts:\n\n```bash\npython finetune/build_rank_map.py \\\n  --shapley_csv results/qwen3-30b-a3b/shapley_values/gsm8k_25_shapley.csv \\\n  --selected_experts results/qwen3-30b-a3b/selected_experts/shapley_alpha_per_layer_gsm8k_25_rate0_8.json \\\n  --output results/qwen3-30b-a3b/lora_rank_maps/gsm8k_25_rate0_8_bucket.json \\\n  --strategy bucket\n```\n\nDefault adaptive rank allocation is layer-wise and only over retained experts:\n\n```text\nTop 20% retained experts -> rank 32\nNext 40% retained experts -> rank 16\nLast 40% retained experts -> rank 8\n```\n\nThe matched baselines are:\n\n```text\nuniform: all retained experts use rank 16\nrandom: same bucket sizes and rank set as bucket, assigned randomly\n```\n\nRank maps omit pruned experts. During LoRA training, `train_adaptive_lora.py` expands the rank map into full expert module paths and passes only those modules to PEFT, so zeroed/pruned experts do not receive LoRA parameters.\n\nTrain an adapter on a pruned model using the distilled SFT data:\n\n```bash\npython finetune/train_adaptive_lora.py \\\n  --model_path /path/to/pruned_qwen3_model \\\n  --rank_map results/qwen3-30b-a3b/lora_rank_maps/gsm8k_25_rate0_8_bucket.json \\\n  --train_file data/sft/gsm8k_distill.json \\\n  --output_dir adapters/qwen3_gsm8k_rate0_8_bucket \\\n  --model_type qwen3 \\\n  --bf16\n```\n\nMerge the adapter before serving or formal evaluation:\n\n```bash\npython finetune/merge_lora.py \\\n  --base_model /path/to/pruned_qwen3_model \\\n  --adapter adapters/qwen3_gsm8k_rate0_8_bucket \\\n  --output /path/to/merged_qwen3_gsm8k_rate0_8_bucket\n```\n\nThe whole `{rate} x {bucket, uniform, random}` matrix (rank map -> train -> merge) can be run in one command:\n\n```bash\ncd finetune && ./run_experiments.sh\n```\n\n#### 6c. Optional: DPO Stage\n\nDPO is an optional second stage applied **after** SFT. The recommended recipe is `SFT -> merge -> DPO -> merge`, so the DPO base model is the SFT-merged model. The DPO reference distribution is obtained by **disabling** the (new) LoRA adapters, which recovers the SFT model without loading a second copy. The same rank map drives LoRA placement, keeping the C/D/E comparison fair.\n\n```bash\n# Train a DPO LoRA on the SFT-merged model, then merge, for the full matrix.\ncd finetune && ./run_dpo.sh\n```\n\nNotes:\n- Always run SFT before DPO; DPO from scratch underperforms.\n- Apply DPO uniformly to all rank strategies (bucket/uniform/random) to keep the comparison fair.\n- A strong teacher rarely errs on easy tasks, so DPO negatives are scarce; raise `NUM_SAMPLES`/`TEMPERATURE` or use harder data to surface more preference pairs.\n\n### 7. Evaluate\n\n```bash\ncd evaluation\n\n# Start a vLLM server from a configured model.\n./vllm-server.sh qwen3-30b-a3b\n\n# Or use a full model path and custom port.\n./vllm-server.sh /path/to/model -p 8801\n\n# Run EvalScope evaluation using configs/experiments.yaml.\npython run_evalscope.py\n```\n\n## Supported Models and Datasets\n\n### Models\n\n| Model name | Experts | Experts per token | Type |\n| --- | ---: | ---: | --- |\n| `qwen3-30b-a3b` | 128 | 8 | Qwen3 MoE |\n| `gpt-oss-20b` | 32 | 4 | GPT-OSS MoE |\n| `deepseekv2-lite-coder` | 64 | 6 | DeepSeek-V2 MoE |\n\n### Calibration and Evaluation Tasks\n\nThe default config includes calibration sets for code, math, reasoning, medical QA, NLP, and truthfulness:\n\n```text\nhumaneval_25, gsm8k_25, math_500_25, arc_easy_25, logiqa_25,\ngpqa_diamond_25, med_mcqa_25, pubmedqa_25, biomix_qa_25,\nontonotes5_25, truthful_qa_25\n```\n\nThe IJCNN 2026 main results report GSM8K, HumanEval, GPQA-Diamond, MATH-500, TruthfulQA, OntoNotes5, and MedMCQA.\n\n## Pruning Methods\n\n| Method | Signal | Description |\n| --- | --- | --- |\n| `shapley` | Coalition-aware Shapley-style contribution | Primary SHAPE method |\n| `easyep` | Task-aligned expert selection | EASY-EP baseline |\n| `reap` | Weighted activation norm | REAP/RAEP-style baseline |\n| `gating` | Mean router softmax score | Router-score baseline |\n| `frequency` | Activation count | Routing-frequency baseline |\n| `random` | Uniform random selection | Lower-bound baseline |\n\n## SHAPE Selection Strategies\n\n| Strategy | Description | Notes |\n| --- | --- | --- |\n| `alpha_per_layer` | Preserve an alpha fraction of Shapley mass in each layer | Recommended; used for SHAPE quality coverage |\n| `alpha_global` | Preserve alpha coverage globally | Can over-prune some layers |\n| `topk_per_layer` | Keep a fixed top-k expert count in every layer | Simple layer-wise baseline |\n| `topk_global` | Keep the globally highest-scoring experts | Can cause layer imbalance |\n\nIn the paper, the quality-coverage variant (`alpha_per_layer`) is the main SHAPE setting. It outperforms simplified global and rigid layer-wise variants because it preserves cooperative value while avoiding layer collapse.\n\n## Naming Conventions\n\n- Models: `qwen3-30b-a3b`, `gpt-oss-20b`, `deepseekv2-lite-coder`\n- Datasets: `{dataset}_{samples}`, for example `gsm8k_25`\n- Methods: `shapley`, `easyep`, `reap`, `gating`, `frequency`, `random`\n- Keep rates: `0.8` keeps 80% of experts and prunes 20%; `0.6` keeps 60% and prunes 40%\n- Selected expert files:\n  - SHAPE: `shapley_{strategy}_{dataset}_rate{XX}.json`\n  - Baselines: `{method}_{dataset}_rate{XX}.json`\n- LoRA rank maps: `{dataset}_rate{XX}_{rank_strategy}.json`, where `rank_strategy` is `bucket`, `uniform`, or `random`\n- Distilled training data: `data/sft/{dataset}_distill.json` (SFT), `data/sft/{dataset}_dpo.json` (DPO preference pairs)\n\n## Citation\n\nIf you use this repository, please cite:\n\n```bibtex\n@inproceedings{zhang2026shape,\n  title     = {SHAPE: Coalition-Aware Expert Pruning for Sparse Mixture-of-Experts LLMs},\n  author    = {Zhang, Yuhao and Jiang, HongXu and Zhang, YiXiang and Zhang, Zheng},\n  booktitle = {Proceedings of the International Joint Conference on Neural Networks},\n  year      = {2026}\n}\n```\n\n## Notes\n\n- SHAPE is post-training and does not require gradient updates.\n- The default calibration size in this repository is 25 examples per task.\n- The `-r` argument in scripts is a **keep rate**, not a removal rate.\n- Large models require local Hugging Face checkpoints and sufficient GPU memory for the unpruned calibration pass.\n"}, "2495204cb3bffcfa5dec305f7672fdb7f38a07d9": {"t": 1785771688.0510335, "n": 15062, "h": "63d8e705a05fcc08", "c": "# SHAPE: Coalition-Aware Expert Pruning for Sparse MoE LLMs\n\n**SHAPE** (**SH**apley-**A**ware **P**runing of **E**xperts) is a training-free pruning framework for sparse Mixture-of-Experts (MoE) large language models. It reduces the number of resident experts needed at inference time while preserving the cooperative routing structure that makes MoE models effective.\n\nThis repository accompanies the IJCNN 2026 work:\n\n> **SHAPE: Coalition-Aware Expert Pruning for Sparse Mixture-of-Experts LLMs**\n\nSparse MoE models activate only a few experts per token, but the full expert pool still has to stay in GPU memory for dynamic routing. SHAPE targets this memory wall by pruning redundant experts after pretraining, without changing router logic, retraining experts, or modifying the model architecture.\n\n<p align=\"center\">\n  <img src=\"assets/shape_cover.png\" alt=\"SHAPE cover illustration\" width=\"92%\">\n</p>\n\n\n## Highlights\n\n- **Coalition-aware expert valuation.** SHAPE treats routed top-k experts as a coalition instead of scoring each expert independently.\n- **Training-free pruning.** It uses only a small calibration set and routing traces from the original model.\n- **Layer-aware quality coverage.** It keeps the smallest expert subset that preserves enough Shapley contribution mass per layer, then uses bisection to match a target global keep rate.\n- **Modern MoE backbones.** The experiments cover Qwen3-30B-A3B, GPT-OSS-20B, and DeepSeek-V2-Lite.\n- **Practical memory savings.** Removing 20% or 40% of experts reduces peak VRAM while keeping task performance stable across reasoning, coding, QA, and NLP benchmarks.\n\n## What SHAPE Does\n\nMost expert pruning methods score experts in isolation, using signals such as routing frequency, router probability, or activation magnitude. That misses a key property of sparse MoE inference: each token is processed by a **coalition** of routed experts. An expert can be valuable not because it appears often, but because it completes a high-utility expert group.\n\nSHAPE reframes pruning as a cooperative attribution problem:\n\n1. Run the unpruned model on a small task-specific calibration set.\n2. Record top-k expert combinations at every MoE layer.\n3. Build an empirical cooperative game from observed expert coalitions.\n4. Estimate Shapley-style contribution scores for experts inside each layer.\n5. Select experts with a quality-coverage rule that preserves layer-wise contribution mass.\n6. Export a compact MoE model for serving.\n\n<p align=\"center\">\n  <img src=\"assets/shape_overview.png\" alt=\"Overview of SHAPE workflow\" width=\"95%\">\n</p>\n\n## Method Overview\n\nGiven a pretrained sparse MoE model and a downstream task, SHAPE performs an offline calibration pass. For every calibration token, it records which experts are co-activated by the router in each MoE layer. These traces define the observed coalition support for that task.\n\nFor each expert, SHAPE estimates how much marginal value it contributes inside the coalitions where it appears. The approximation uses routing frequency and co-occurrence statistics, avoiding expensive repeated forward passes over many masked expert subsets.\n\nAfter scoring, SHAPE applies **Shapley Quality Coverage**. Instead of keeping a fixed number of experts in every layer, it keeps the smallest prefix of high-value experts whose cumulative contribution reaches an alpha fraction of that layer's Shapley mass. The alpha threshold is chosen by bisection so the final retained expert count matches the target keep rate.\n\n\n\n## Results at a Glance\n\nSHAPE is evaluated on three sparse MoE backbones and seven benchmarks: GSM8K, HumanEval, GPQA-Diamond, MATH-500, TruthfulQA, OntoNotes5, and MedMCQA. Each task uses 25 calibration examples to collect routing traces.\n\n### Accuracy Under Expert Pruning\n\nThe table below reports task-average accuracy from the IJCNN 2026 experiments. A keep rate of `0.80` means 20% of experts are pruned; a keep rate of `0.60` means 40% of experts are pruned.\n\n| Model | Baseline | 20% Pruned | 40% Pruned |\n| --- | ---: | ---: | ---: |\n| Qwen3-30B-A3B | 82.92 | 82.43 | 81.31 |\n| GPT-OSS-20B | 82.12 | 82.44 | 79.02 |\n| DeepSeek-V2-Lite | 62.08 | 62.44 | 58.81 |\n\nAt 20% pruning, average performance remains comparable to the unpruned models and can slightly improve on some backbones, suggesting that part of the expert pool has limited marginal utility for the evaluated tasks. At 40% pruning, degradation increases but remains bounded, especially on larger MoE backbones.\n\n### Baseline Comparison on Qwen3-30B-A3B\n\nSHAPE is strongest under aggressive pruning, where preserving critical expert coalitions matters most.\n\n| Rate | Method | GSM8K | HumanEval | GPQA-D | MATH-500 | TruthfulQA | OntoNotes5 | MedMCQA | Avg |\n| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |\n| - | Unpruned | 96.21 | 95.73 | 58.59 | 96.80 | 77.60 | 87.15 | 68.35 | 82.92 |\n| 20% | Random | 61.24 | 58.36 | 21.42 | 60.27 | 42.31 | 71.16 | 38.18 | 50.42 |\n| 20% | Frequency | 86.12 | 86.37 | 40.12 | 88.14 | 64.05 | 82.17 | 58.12 | 72.16 |\n| 20% | Gating | 87.42 | 87.15 | 42.18 | 89.36 | 65.41 | 83.24 | 59.27 | 73.43 |\n| 20% | RAEP | 95.47 | 94.52 | 57.56 | 95.42 | 76.42 | 86.14 | 67.51 | 81.86 |\n| 20% | EASY-EP | 95.82 | 95.01 | 58.12 | 95.86 | 77.14 | 86.54 | 67.96 | 82.35 |\n| 20% | **SHAPE** | **96.74** | **95.12** | **60.10** | **96.80** | 75.64 | 84.42 | **68.18** | **82.43** |\n| 40% | Random | 35.43 | 28.27 | 8.28 | 32.15 | 21.29 | 60.41 | 15.46 | 28.76 |\n| 40% | Frequency | 75.14 | 72.11 | 20.16 | 72.29 | 45.18 | 76.28 | 40.17 | 57.33 |\n| 40% | Gating | 77.19 | 74.32 | 22.54 | 74.28 | 47.21 | 77.31 | 42.15 | 59.29 |\n| 40% | RAEP | 93.42 | 90.47 | 52.42 | 92.58 | 43.12 | 83.15 | 64.41 | 74.22 |\n| 40% | EASY-EP | 94.12 | **91.07** | 53.12 | 93.14 | **73.28** | **83.51** | 65.14 | 79.05 |\n| 40% | **SHAPE** | **95.60** | 89.63 | **67.68** | **94.40** | 73.16 | 82.37 | **66.32** | **81.31** |\n\n### Peak VRAM\n\nPruning directly reduces the resident expert footprint. The measured peak VRAM decreases steadily under 20% and 40% expert pruning for both 32K and 8K context settings, with no additional training or architecture changes.\n\n<p align=\"center\">\n  <img src=\"assets/shape_memory_vram.png\" alt=\"Peak VRAM under 0, 20, and 40 percent expert pruning\" width=\"78%\">\n</p>\n\n## Repository Structure\n\n```text\nshapley-moe/\n|-- data/                         # Calibration data and download scripts\n|   |-- calibration/              # 25-example calibration sets\n|   |-- download_dataset.py\n|   `-- run_download.sh\n|-- analysis/                     # Routing trace collection and Shapley scoring\n|   |-- collect_activations.py\n|   |-- calc_shapley.py\n|   |-- run_collect.sh\n|   `-- run_calc_shapley.sh\n|-- pruning/                      # Expert selection and pruned-model export\n|   |-- methods/\n|   |   |-- select_by_shapley.py\n|   |   |-- select_by_easyep.py\n|   |   |-- select_by_reap.py\n|   |   |-- select_by_gating.py\n|   |   |-- select_by_frequency.py\n|   |   `-- select_by_random.py\n|   |-- save_model.py\n|   |-- run_select.sh\n|   `-- run_prune.sh\n|-- finetune/                     # Post-pruning adaptive LoRA fine-tuning\n|   |-- build_rank_map.py\n|   |-- train_adaptive_lora.py\n|   |-- merge_lora.py\n|   `-- infer_adaptive_lora.py\n|-- evaluation/                   # vLLM serving and EvalScope evaluation\n|-- configs/                      # Model paths and experiment settings\n|-- results/                      # Activations, Shapley values, selected experts\n|-- assets/                       # README figures\n`-- PROJECT_STRUCTURE.md\n```\n\n## Quick Start\n\n### 0. Configure Models and Experiments\n\nEdit the config files before running the pipeline:\n\n```bash\nvim configs/models.yaml\nvim configs/experiments.yaml\n```\n\n`configs/models.yaml` defines model paths, expert counts, top-k routing size, and MoE module patterns. `configs/experiments.yaml` defines calibration datasets, keep rates, pruning methods, and evaluation defaults.\n\n### 1. Prepare Calibration Data\n\n```bash\ncd data\n\n# Download one dataset with 25 examples.\n./run_download.sh gsm8k 25\n\n# Or download all datasets from configs/experiments.yaml.\n./run_download.sh --all\n```\n\n### 2. Collect Routing and Activation Statistics\n\nThis single pass collects the information needed by SHAPE and the baseline pruning methods.\n\n```bash\ncd analysis\n\n# Use a configured model name.\n./run_collect.sh -m qwen3-30b-a3b --all\n\n# Or use a full model path.\n./run_collect.sh -m /path/to/model --data ../data/calibration/gsm8k_25.json\n\n# Inspect configured models.\n./run_collect.sh --list-models\n```\n\nOutputs are written to:\n\n```text\nresults/{model_name}/activations/\n```\n\n### 3. Compute Shapley Values\n\n```bash\ncd analysis\n\n# Compute Shapley scores for all available activation files.\n./run_calc_shapley.sh -m qwen3-30b-a3b\n\n# Or compute one dataset.\n./run_calc_shapley.sh -m qwen3-30b-a3b -d gsm8k_25\n```\n\nOutputs are written to:\n\n```text\nresults/{model_name}/shapley_values/\n```\n\n### 4. Select Experts\n\n```bash\ncd pruning\n\n# SHAPE selection with 80% keep rate, i.e. 20% expert pruning.\n./run_select.sh -m qwen3-30b-a3b -d gsm8k_25 -M shapley -r 0.8\n\n# Run all configured keep rates.\n./run_select.sh -m qwen3-30b-a3b -d gsm8k_25 -M shapley --all-rates\n\n# Run all datasets and all configured methods.\n./run_select.sh -m qwen3-30b-a3b --all-datasets --all-methods --all-rates\n```\n\nSelected expert files are written to:\n\n```text\nresults/{model_name}/selected_experts/\n```\n\n### 5. Export a Pruned Model\n\n```bash\ncd pruning\n\n# Export a SHAPE-pruned model from the selected experts.\n./run_prune.sh -m qwen3-30b-a3b -d gsm8k_25 -r 0.8\n\n# Use another method or keep rate.\n./run_prune.sh -m gpt-oss-20b -d arc_easy_25 -M easyep -r 0.6\n```\n\nThe pruning script finds the matching selected-expert JSON and writes a pruned Hugging Face model directory.\n\n### 6. Optional: Adaptive LoRA Fine-Tuning\n\nThe `finetune/` scripts implement a post-pruning LoRA recovery path. The current minimal experimental loop targets Qwen3-30B-A3B on `gsm8k_25` with SHAPE-selected experts.\n\nBuild a LoRA rank map from Shapley scores and selected experts:\n\n```bash\npython finetune/build_rank_map.py \\\n  --shapley_csv results/qwen3-30b-a3b/shapley_values/gsm8k_25_shapley.csv \\\n  --selected_experts results/qwen3-30b-a3b/selected_experts/shapley_alpha_per_layer_gsm8k_25_rate0_8.json \\\n  --output results/qwen3-30b-a3b/lora_rank_maps/gsm8k_25_rate0_8_bucket.json \\\n  --strategy bucket\n```\n\nDefault adaptive rank allocation is layer-wise and only over retained experts:\n\n```text\nTop 20% retained experts -> rank 32\nNext 40% retained experts -> rank 16\nLast 40% retained experts -> rank 8\n```\n\nThe matched baselines are:\n\n```text\nuniform: all retained experts use rank 16\nrandom: same bucket sizes and rank set as bucket, assigned randomly\n```\n\nRank maps omit pruned experts. During LoRA training, `train_adaptive_lora.py` expands the rank map into full expert module paths and passes only those modules to PEFT, so zeroed/pruned experts do not receive LoRA parameters.\n\nTrain an adapter on a pruned model:\n\n```bash\npython finetune/train_adaptive_lora.py \\\n  --model_path /path/to/pruned_qwen3_model \\\n  --rank_map results/qwen3-30b-a3b/lora_rank_maps/gsm8k_25_rate0_8_bucket.json \\\n  --train_file data/calibration/gsm8k_25.json \\\n  --output_dir adapters/qwen3_gsm8k_rate0_8_bucket \\\n  --model_type qwen3 \\\n  --bf16\n```\n\nMerge the adapter before serving or formal evaluation:\n\n```bash\npython finetune/merge_lora.py \\\n  --base_model /path/to/pruned_qwen3_model \\\n  --adapter adapters/qwen3_gsm8k_rate0_8_bucket \\\n  --output /path/to/merged_qwen3_gsm8k_rate0_8_bucket\n```\n\n### 7. Evaluate\n\n```bash\ncd evaluation\n\n# Start a vLLM server from a configured model.\n./vllm-server.sh qwen3-30b-a3b\n\n# Or use a full model path and custom port.\n./vllm-server.sh /path/to/model -p 8801\n\n# Run EvalScope evaluation using configs/experiments.yaml.\npython run_evalscope.py\n```\n\n## Supported Models and Datasets\n\n### Models\n\n| Model name | Experts | Experts per token | Type |\n| --- | ---: | ---: | --- |\n| `qwen3-30b-a3b` | 128 | 8 | Qwen3 MoE |\n| `gpt-oss-20b` | 32 | 4 | GPT-OSS MoE |\n| `deepseekv2-lite-coder` | 64 | 6 | DeepSeek-V2 MoE |\n\n### Calibration and Evaluation Tasks\n\nThe default config includes calibration sets for code, math, reasoning, medical QA, NLP, and truthfulness:\n\n```text\nhumaneval_25, gsm8k_25, math_500_25, arc_easy_25, logiqa_25,\ngpqa_diamond_25, med_mcqa_25, pubmedqa_25, biomix_qa_25,\nontonotes5_25, truthful_qa_25\n```\n\nThe IJCNN 2026 main results report GSM8K, HumanEval, GPQA-Diamond, MATH-500, TruthfulQA, OntoNotes5, and MedMCQA.\n\n## Pruning Methods\n\n| Method | Signal | Description |\n| --- | --- | --- |\n| `shapley` | Coalition-aware Shapley-style contribution | Primary SHAPE method |\n| `easyep` | Task-aligned expert selection | EASY-EP baseline |\n| `reap` | Weighted activation norm | REAP/RAEP-style baseline |\n| `gating` | Mean router softmax score | Router-score baseline |\n| `frequency` | Activation count | Routing-frequency baseline |\n| `random` | Uniform random selection | Lower-bound baseline |\n\n## SHAPE Selection Strategies\n\n| Strategy | Description | Notes |\n| --- | --- | --- |\n| `alpha_per_layer` | Preserve an alpha fraction of Shapley mass in each layer | Recommended; used for SHAPE quality coverage |\n| `alpha_global` | Preserve alpha coverage globally | Can over-prune some layers |\n| `topk_per_layer` | Keep a fixed top-k expert count in every layer | Simple layer-wise baseline |\n| `topk_global` | Keep the globally highest-scoring experts | Can cause layer imbalance |\n\nIn the paper, the quality-coverage variant (`alpha_per_layer`) is the main SHAPE setting. It outperforms simplified global and rigid layer-wise variants because it preserves cooperative value while avoiding layer collapse.\n\n## Naming Conventions\n\n- Models: `qwen3-30b-a3b`, `gpt-oss-20b`, `deepseekv2-lite-coder`\n- Datasets: `{dataset}_{samples}`, for example `gsm8k_25`\n- Methods: `shapley`, `easyep`, `reap`, `gating`, `frequency`, `random`\n- Keep rates: `0.8` keeps 80% of experts and prunes 20%; `0.6` keeps 60% and prunes 40%\n- Selected expert files:\n  - SHAPE: `shapley_{strategy}_{dataset}_rate{XX}.json`\n  - Baselines: `{method}_{dataset}_rate{XX}.json`\n- LoRA rank maps: `{dataset}_rate{XX}_{rank_strategy}.json`, where `rank_strategy` is `bucket`, `uniform`, or `random`\n\n## Citation\n\nIf you use this repository, please cite:\n\n```bibtex\n@inproceedings{zhang2026shape,\n  title     = {SHAPE: Coalition-Aware Expert Pruning for Sparse Mixture-of-Experts LLMs},\n  author    = {Zhang, Yuhao and Jiang, HongXu and Zhang, YiXiang and Zhang, Zheng},\n  booktitle = {Proceedings of the International Joint Conference on Neural Networks},\n  year      = {2026}\n}\n```\n\n## Notes\n\n- SHAPE is post-training and does not require gradient updates.\n- The default calibration size in this repository is 25 examples per task.\n- The `-r` argument in scripts is a **keep rate**, not a removal rate.\n- Large models require local Hugging Face checkpoints and sufficient GPU memory for the unpruned calibration pass.\n"}, "262c3c9b976c83185bdf7cf2ee3d9fb7e2af5579": {"t": 1785771689.273106, "n": 15061, "h": "61e3c9794d76dfcb", "c": "# SHAPE: Coalition-Aware Expert Pruning for Sparse MoE LLMs\n\n**SHAPE** (**SH**apley-**A**ware **P**runing of **E**xperts) is a training-free pruning framework for sparse Mixture-of-Experts (MoE) large language models. It reduces the number of resident experts needed at inference time while preserving the cooperative routing structure that makes MoE models effective.\n\nThis repository accompanies the IJCNN 2026 work:\n\n> **SHAPE: Coalition-Aware Expert Pruning for Sparse Mixture-of-Experts LLMs**\n\nSparse MoE models activate only a few experts per token, but the full expert pool still has to stay in GPU memory for dynamic routing. SHAPE targets this memory wall by pruning redundant experts after pretraining, without changing router logic, retraining experts, or modifying the model architecture.\n\n<p align=\"center\">\n  <img src=\"assets/shape_cover.png\" alt=\"SHAPE cover illustration\" width=\"92%\">\n</p>\n\n## Highlights\n\n- **Coalition-aware expert valuation.** SHAPE treats routed top-k experts as a coalition instead of scoring each expert independently.\n- **Training-free pruning.** It uses only a small calibration set and routing traces from the original model.\n- **Layer-aware quality coverage.** It keeps the smallest expert subset that preserves enough Shapley contribution mass per layer, then uses bisection to match a target global keep rate.\n- **Modern MoE backbones.** The experiments cover Qwen3-30B-A3B, GPT-OSS-20B, and DeepSeek-V2-Lite.\n- **Practical memory savings.** Removing 20% or 40% of experts reduces peak VRAM while keeping task performance stable across reasoning, coding, QA, and NLP benchmarks.\n\n## What SHAPE Does\n\nMost expert pruning methods score experts in isolation, using signals such as routing frequency, router probability, or activation magnitude. That misses a key property of sparse MoE inference: each token is processed by a **coalition** of routed experts. An expert can be valuable not because it appears often, but because it completes a high-utility expert group.\n\nSHAPE reframes pruning as a cooperative attribution problem:\n\n1. Run the unpruned model on a small task-specific calibration set.\n2. Record top-k expert combinations at every MoE layer.\n3. Build an empirical cooperative game from observed expert coalitions.\n4. Estimate Shapley-style contribution scores for experts inside each layer.\n5. Select experts with a quality-coverage rule that preserves layer-wise contribution mass.\n6. Export a compact MoE model for serving.\n\n<p align=\"center\">\n  <img src=\"assets/shape_overview.png\" alt=\"Overview of SHAPE workflow\" width=\"95%\">\n</p>\n\n## Method Overview\n\nGiven a pretrained sparse MoE model and a downstream task, SHAPE performs an offline calibration pass. For every calibration token, it records which experts are co-activated by the router in each MoE layer. These traces define the observed coalition support for that task.\n\nFor each expert, SHAPE estimates how much marginal value it contributes inside the coalitions where it appears. The approximation uses routing frequency and co-occurrence statistics, avoiding expensive repeated forward passes over many masked expert subsets.\n\nAfter scoring, SHAPE applies **Shapley Quality Coverage**. Instead of keeping a fixed number of experts in every layer, it keeps the smallest prefix of high-value experts whose cumulative contribution reaches an alpha fraction of that layer's Shapley mass. The alpha threshold is chosen by bisection so the final retained expert count matches the target keep rate.\n\n\n\n## Results at a Glance\n\nSHAPE is evaluated on three sparse MoE backbones and seven benchmarks: GSM8K, HumanEval, GPQA-Diamond, MATH-500, TruthfulQA, OntoNotes5, and MedMCQA. Each task uses 25 calibration examples to collect routing traces.\n\n### Accuracy Under Expert Pruning\n\nThe table below reports task-average accuracy from the IJCNN 2026 experiments. A keep rate of `0.80` means 20% of experts are pruned; a keep rate of `0.60` means 40% of experts are pruned.\n\n| Model | Baseline | 20% Pruned | 40% Pruned |\n| --- | ---: | ---: | ---: |\n| Qwen3-30B-A3B | 82.92 | 82.43 | 81.31 |\n| GPT-OSS-20B | 82.12 | 82.44 | 79.02 |\n| DeepSeek-V2-Lite | 62.08 | 62.44 | 58.81 |\n\nAt 20% pruning, average performance remains comparable to the unpruned models and can slightly improve on some backbones, suggesting that part of the expert pool has limited marginal utility for the evaluated tasks. At 40% pruning, degradation increases but remains bounded, especially on larger MoE backbones.\n\n### Baseline Comparison on Qwen3-30B-A3B\n\nSHAPE is strongest under aggressive pruning, where preserving critical expert coalitions matters most.\n\n| Rate | Method | GSM8K | HumanEval | GPQA-D | MATH-500 | TruthfulQA | OntoNotes5 | MedMCQA | Avg |\n| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |\n| - | Unpruned | 96.21 | 95.73 | 58.59 | 96.80 | 77.60 | 87.15 | 68.35 | 82.92 |\n| 20% | Random | 61.24 | 58.36 | 21.42 | 60.27 | 42.31 | 71.16 | 38.18 | 50.42 |\n| 20% | Frequency | 86.12 | 86.37 | 40.12 | 88.14 | 64.05 | 82.17 | 58.12 | 72.16 |\n| 20% | Gating | 87.42 | 87.15 | 42.18 | 89.36 | 65.41 | 83.24 | 59.27 | 73.43 |\n| 20% | RAEP | 95.47 | 94.52 | 57.56 | 95.42 | 76.42 | 86.14 | 67.51 | 81.86 |\n| 20% | EASY-EP | 95.82 | 95.01 | 58.12 | 95.86 | 77.14 | 86.54 | 67.96 | 82.35 |\n| 20% | **SHAPE** | **96.74** | **95.12** | **60.10** | **96.80** | 75.64 | 84.42 | **68.18** | **82.43** |\n| 40% | Random | 35.43 | 28.27 | 8.28 | 32.15 | 21.29 | 60.41 | 15.46 | 28.76 |\n| 40% | Frequency | 75.14 | 72.11 | 20.16 | 72.29 | 45.18 | 76.28 | 40.17 | 57.33 |\n| 40% | Gating | 77.19 | 74.32 | 22.54 | 74.28 | 47.21 | 77.31 | 42.15 | 59.29 |\n| 40% | RAEP | 93.42 | 90.47 | 52.42 | 92.58 | 43.12 | 83.15 | 64.41 | 74.22 |\n| 40% | EASY-EP | 94.12 | **91.07** | 53.12 | 93.14 | **73.28** | **83.51** | 65.14 | 79.05 |\n| 40% | **SHAPE** | **95.60** | 89.63 | **67.68** | **94.40** | 73.16 | 82.37 | **66.32** | **81.31** |\n\n### Peak VRAM\n\nPruning directly reduces the resident expert footprint. The measured peak VRAM decreases steadily under 20% and 40% expert pruning for both 32K and 8K context settings, with no additional training or architecture changes.\n\n<p align=\"center\">\n  <img src=\"assets/shape_memory_vram.png\" alt=\"Peak VRAM under 0, 20, and 40 percent expert pruning\" width=\"78%\">\n</p>\n\n## Repository Structure\n\n```text\nshapley-moe/\n|-- data/                         # Calibration data and download scripts\n|   |-- calibration/              # 25-example calibration sets\n|   |-- download_dataset.py\n|   `-- run_download.sh\n|-- analysis/                     # Routing trace collection and Shapley scoring\n|   |-- collect_activations.py\n|   |-- calc_shapley.py\n|   |-- run_collect.sh\n|   `-- run_calc_shapley.sh\n|-- pruning/                      # Expert selection and pruned-model export\n|   |-- methods/\n|   |   |-- select_by_shapley.py\n|   |   |-- select_by_easyep.py\n|   |   |-- select_by_reap.py\n|   |   |-- select_by_gating.py\n|   |   |-- select_by_frequency.py\n|   |   `-- select_by_random.py\n|   |-- save_model.py\n|   |-- run_select.sh\n|   `-- run_prune.sh\n|-- finetune/                     # Post-pruning adaptive LoRA fine-tuning\n|   |-- build_rank_map.py\n|   |-- train_adaptive_lora.py\n|   |-- merge_lora.py\n|   `-- infer_adaptive_lora.py\n|-- evaluation/                   # vLLM serving and EvalScope evaluation\n|-- configs/                      # Model paths and experiment settings\n|-- results/                      # Activations, Shapley values, selected experts\n|-- assets/                       # README figures\n`-- PROJECT_STRUCTURE.md\n```\n\n## Quick Start\n\n### 0. Configure Models and Experiments\n\nEdit the config files before running the pipeline:\n\n```bash\nvim configs/models.yaml\nvim configs/experiments.yaml\n```\n\n`configs/models.yaml` defines model paths, expert counts, top-k routing size, and MoE module patterns. `configs/experiments.yaml` defines calibration datasets, keep rates, pruning methods, and evaluation defaults.\n\n### 1. Prepare Calibration Data\n\n```bash\ncd data\n\n# Download one dataset with 25 examples.\n./run_download.sh gsm8k 25\n\n# Or download all datasets from configs/experiments.yaml.\n./run_download.sh --all\n```\n\n### 2. Collect Routing and Activation Statistics\n\nThis single pass collects the information needed by SHAPE and the baseline pruning methods.\n\n```bash\ncd analysis\n\n# Use a configured model name.\n./run_collect.sh -m qwen3-30b-a3b --all\n\n# Or use a full model path.\n./run_collect.sh -m /path/to/model --data ../data/calibration/gsm8k_25.json\n\n# Inspect configured models.\n./run_collect.sh --list-models\n```\n\nOutputs are written to:\n\n```text\nresults/{model_name}/activations/\n```\n\n### 3. Compute Shapley Values\n\n```bash\ncd analysis\n\n# Compute Shapley scores for all available activation files.\n./run_calc_shapley.sh -m qwen3-30b-a3b\n\n# Or compute one dataset.\n./run_calc_shapley.sh -m qwen3-30b-a3b -d gsm8k_25\n```\n\nOutputs are written to:\n\n```text\nresults/{model_name}/shapley_values/\n```\n\n### 4. Select Experts\n\n```bash\ncd pruning\n\n# SHAPE selection with 80% keep rate, i.e. 20% expert pruning.\n./run_select.sh -m qwen3-30b-a3b -d gsm8k_25 -M shapley -r 0.8\n\n# Run all configured keep rates.\n./run_select.sh -m qwen3-30b-a3b -d gsm8k_25 -M shapley --all-rates\n\n# Run all datasets and all configured methods.\n./run_select.sh -m qwen3-30b-a3b --all-datasets --all-methods --all-rates\n```\n\nSelected expert files are written to:\n\n```text\nresults/{model_name}/selected_experts/\n```\n\n### 5. Export a Pruned Model\n\n```bash\ncd pruning\n\n# Export a SHAPE-pruned model from the selected experts.\n./run_prune.sh -m qwen3-30b-a3b -d gsm8k_25 -r 0.8\n\n# Use another method or keep rate.\n./run_prune.sh -m gpt-oss-20b -d arc_easy_25 -M easyep -r 0.6\n```\n\nThe pruning script finds the matching selected-expert JSON and writes a pruned Hugging Face model directory.\n\n### 6. Optional: Adaptive LoRA Fine-Tuning\n\nThe `finetune/` scripts implement a post-pruning LoRA recovery path. The current minimal experimental loop targets Qwen3-30B-A3B on `gsm8k_25` with SHAPE-selected experts.\n\nBuild a LoRA rank map from Shapley scores and selected experts:\n\n```bash\npython finetune/build_rank_map.py \\\n  --shapley_csv results/qwen3-30b-a3b/shapley_values/gsm8k_25_shapley.csv \\\n  --selected_experts results/qwen3-30b-a3b/selected_experts/shapley_alpha_per_layer_gsm8k_25_rate0_8.json \\\n  --output results/qwen3-30b-a3b/lora_rank_maps/gsm8k_25_rate0_8_bucket.json \\\n  --strategy bucket\n```\n\nDefault adaptive rank allocation is layer-wise and only over retained experts:\n\n```text\nTop 20% retained experts -> rank 32\nNext 40% retained experts -> rank 16\nLast 40% retained experts -> rank 8\n```\n\nThe matched baselines are:\n\n```text\nuniform: all retained experts use rank 16\nrandom: same bucket sizes and rank set as bucket, assigned randomly\n```\n\nRank maps omit pruned experts. During LoRA training, `train_adaptive_lora.py` expands the rank map into full expert module paths and passes only those modules to PEFT, so zeroed/pruned experts do not receive LoRA parameters.\n\nTrain an adapter on a pruned model:\n\n```bash\npython finetune/train_adaptive_lora.py \\\n  --model_path /path/to/pruned_qwen3_model \\\n  --rank_map results/qwen3-30b-a3b/lora_rank_maps/gsm8k_25_rate0_8_bucket.json \\\n  --train_file data/calibration/gsm8k_25.json \\\n  --output_dir adapters/qwen3_gsm8k_rate0_8_bucket \\\n  --model_type qwen3 \\\n  --bf16\n```\n\nMerge the adapter before serving or formal evaluation:\n\n```bash\npython finetune/merge_lora.py \\\n  --base_model /path/to/pruned_qwen3_model \\\n  --adapter adapters/qwen3_gsm8k_rate0_8_bucket \\\n  --output /path/to/merged_qwen3_gsm8k_rate0_8_bucket\n```\n\n### 7. Evaluate\n\n```bash\ncd evaluation\n\n# Start a vLLM server from a configured model.\n./vllm-server.sh qwen3-30b-a3b\n\n# Or use a full model path and custom port.\n./vllm-server.sh /path/to/model -p 8801\n\n# Run EvalScope evaluation using configs/experiments.yaml.\npython run_evalscope.py\n```\n\n## Supported Models and Datasets\n\n### Models\n\n| Model name | Experts | Experts per token | Type |\n| --- | ---: | ---: | --- |\n| `qwen3-30b-a3b` | 128 | 8 | Qwen3 MoE |\n| `gpt-oss-20b` | 32 | 4 | GPT-OSS MoE |\n| `deepseekv2-lite-coder` | 64 | 6 | DeepSeek-V2 MoE |\n\n### Calibration and Evaluation Tasks\n\nThe default config includes calibration sets for code, math, reasoning, medical QA, NLP, and truthfulness:\n\n```text\nhumaneval_25, gsm8k_25, math_500_25, arc_easy_25, logiqa_25,\ngpqa_diamond_25, med_mcqa_25, pubmedqa_25, biomix_qa_25,\nontonotes5_25, truthful_qa_25\n```\n\nThe IJCNN 2026 main results report GSM8K, HumanEval, GPQA-Diamond, MATH-500, TruthfulQA, OntoNotes5, and MedMCQA.\n\n## Pruning Methods\n\n| Method | Signal | Description |\n| --- | --- | --- |\n| `shapley` | Coalition-aware Shapley-style contribution | Primary SHAPE method |\n| `easyep` | Task-aligned expert selection | EASY-EP baseline |\n| `reap` | Weighted activation norm | REAP/RAEP-style baseline |\n| `gating` | Mean router softmax score | Router-score baseline |\n| `frequency` | Activation count | Routing-frequency baseline |\n| `random` | Uniform random selection | Lower-bound baseline |\n\n## SHAPE Selection Strategies\n\n| Strategy | Description | Notes |\n| --- | --- | --- |\n| `alpha_per_layer` | Preserve an alpha fraction of Shapley mass in each layer | Recommended; used for SHAPE quality coverage |\n| `alpha_global` | Preserve alpha coverage globally | Can over-prune some layers |\n| `topk_per_layer` | Keep a fixed top-k expert count in every layer | Simple layer-wise baseline |\n| `topk_global` | Keep the globally highest-scoring experts | Can cause layer imbalance |\n\nIn the paper, the quality-coverage variant (`alpha_per_layer`) is the main SHAPE setting. It outperforms simplified global and rigid layer-wise variants because it preserves cooperative value while avoiding layer collapse.\n\n## Naming Conventions\n\n- Models: `qwen3-30b-a3b`, `gpt-oss-20b`, `deepseekv2-lite-coder`\n- Datasets: `{dataset}_{samples}`, for example `gsm8k_25`\n- Methods: `shapley`, `easyep`, `reap`, `gating`, `frequency`, `random`\n- Keep rates: `0.8` keeps 80% of experts and prunes 20%; `0.6` keeps 60% and prunes 40%\n- Selected expert files:\n  - SHAPE: `shapley_{strategy}_{dataset}_rate{XX}.json`\n  - Baselines: `{method}_{dataset}_rate{XX}.json`\n- LoRA rank maps: `{dataset}_rate{XX}_{rank_strategy}.json`, where `rank_strategy` is `bucket`, `uniform`, or `random`\n\n## Citation\n\nIf you use this repository, please cite:\n\n```bibtex\n@inproceedings{zhang2026shape,\n  title     = {SHAPE: Coalition-Aware Expert Pruning for Sparse Mixture-of-Experts LLMs},\n  author    = {Zhang, Yuhao and Jiang, HongXu and Zhang, YiXiang and Zhang, Zheng},\n  booktitle = {Proceedings of the International Joint Conference on Neural Networks},\n  year      = {2026}\n}\n```\n\n## Notes\n\n- SHAPE is post-training and does not require gradient updates.\n- The default calibration size in this repository is 25 examples per task.\n- The `-r` argument in scripts is a **keep rate**, not a removal rate.\n- Large models require local Hugging Face checkpoints and sufficient GPU memory for the unpruned calibration pass.\n"}, "7b260dfc1bf9c3000d44d0f2a4808f56e23d75cf": {"t": 1785771690.6783576, "n": 12964, "h": "801478836f392bb0", "c": "# SHAPE: Coalition-Aware Expert Pruning for Sparse MoE LLMs\n\n**SHAPE** (**SH**apley-**A**ware **P**runing of **E**xperts) is a training-free pruning framework for sparse Mixture-of-Experts (MoE) large language models. It reduces the number of resident experts needed at inference time while preserving the cooperative routing structure that makes MoE models effective.\n\nThis repository accompanies the IJCNN 2026 work:\n\n> **SHAPE: Coalition-Aware Expert Pruning for Sparse Mixture-of-Experts LLMs**\n\nSparse MoE models activate only a few experts per token, but the full expert pool still has to stay in GPU memory for dynamic routing. SHAPE targets this memory wall by pruning redundant experts after pretraining, without changing router logic, retraining experts, or modifying the model architecture.\n\n<p align=\"center\">\n  <img src=\"assets/shape_cover.png\" alt=\"SHAPE cover illustration\" width=\"92%\">\n</p>\n\n## Highlights\n\n- **Coalition-aware expert valuation.** SHAPE treats routed top-k experts as a coalition instead of scoring each expert independently.\n- **Training-free pruning.** It uses only a small calibration set and routing traces from the original model.\n- **Layer-aware quality coverage.** It keeps the smallest expert subset that preserves enough Shapley contribution mass per layer, then uses bisection to match a target global keep rate.\n- **Modern MoE backbones.** The experiments cover Qwen3-30B-A3B, GPT-OSS-20B, and DeepSeek-V2-Lite.\n- **Practical memory savings.** Removing 20% or 40% of experts reduces peak VRAM while keeping task performance stable across reasoning, coding, QA, and NLP benchmarks.\n\n## What SHAPE Does\n\nMost expert pruning methods score experts in isolation, using signals such as routing frequency, router probability, or activation magnitude. That misses a key property of sparse MoE inference: each token is processed by a **coalition** of routed experts. An expert can be valuable not because it appears often, but because it completes a high-utility expert group.\n\nSHAPE reframes pruning as a cooperative attribution problem:\n\n1. Run the unpruned model on a small task-specific calibration set.\n2. Record top-k expert combinations at every MoE layer.\n3. Build an empirical cooperative game from observed expert coalitions.\n4. Estimate Shapley-style contribution scores for experts inside each layer.\n5. Select experts with a quality-coverage rule that preserves layer-wise contribution mass.\n6. Export a compact MoE model for serving.\n\n<p align=\"center\">\n  <img src=\"assets/shape_overview.png\" alt=\"Overview of SHAPE workflow\" width=\"95%\">\n</p>\n\n## Method Overview\n\nGiven a pretrained sparse MoE model and a downstream task, SHAPE performs an offline calibration pass. For every calibration token, it records which experts are co-activated by the router in each MoE layer. These traces define the observed coalition support for that task.\n\nFor each expert, SHAPE estimates how much marginal value it contributes inside the coalitions where it appears. The approximation uses routing frequency and co-occurrence statistics, avoiding expensive repeated forward passes over many masked expert subsets.\n\nAfter scoring, SHAPE applies **Shapley Quality Coverage**. Instead of keeping a fixed number of experts in every layer, it keeps the smallest prefix of high-value experts whose cumulative contribution reaches an alpha fraction of that layer's Shapley mass. The alpha threshold is chosen by bisection so the final retained expert count matches the target keep rate.\n\n\n\n## Results at a Glance\n\nSHAPE is evaluated on three sparse MoE backbones and seven benchmarks: GSM8K, HumanEval, GPQA-Diamond, MATH-500, TruthfulQA, OntoNotes5, and MedMCQA. Each task uses 25 calibration examples to collect routing traces.\n\n### Accuracy Under Expert Pruning\n\nThe table below reports task-average accuracy from the IJCNN 2026 experiments. A keep rate of `0.80` means 20% of experts are pruned; a keep rate of `0.60` means 40% of experts are pruned.\n\n| Model | Baseline | 20% Pruned | 40% Pruned |\n| --- | ---: | ---: | ---: |\n| Qwen3-30B-A3B | 82.92 | 82.43 | 81.31 |\n| GPT-OSS-20B | 82.12 | 82.44 | 79.02 |\n| DeepSeek-V2-Lite | 62.08 | 62.44 | 58.81 |\n\nAt 20% pruning, average performance remains comparable to the unpruned models and can slightly improve on some backbones, suggesting that part of the expert pool has limited marginal utility for the evaluated tasks. At 40% pruning, degradation increases but remains bounded, especially on larger MoE backbones.\n\n### Baseline Comparison on Qwen3-30B-A3B\n\nSHAPE is strongest under aggressive pruning, where preserving critical expert coalitions matters most.\n\n| Rate | Method | GSM8K | HumanEval | GPQA-D | MATH-500 | TruthfulQA | OntoNotes5 | MedMCQA | Avg |\n| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |\n| - | Unpruned | 96.21 | 95.73 | 58.59 | 96.80 | 77.60 | 87.15 | 68.35 | 82.92 |\n| 20% | Random | 61.24 | 58.36 | 21.42 | 60.27 | 42.31 | 71.16 | 38.18 | 50.42 |\n| 20% | Frequency | 86.12 | 86.37 | 40.12 | 88.14 | 64.05 | 82.17 | 58.12 | 72.16 |\n| 20% | Gating | 87.42 | 87.15 | 42.18 | 89.36 | 65.41 | 83.24 | 59.27 | 73.43 |\n| 20% | RAEP | 95.47 | 94.52 | 57.56 | 95.42 | 76.42 | 86.14 | 67.51 | 81.86 |\n| 20% | EASY-EP | 95.82 | 95.01 | 58.12 | 95.86 | 77.14 | 86.54 | 67.96 | 82.35 |\n| 20% | **SHAPE** | **96.74** | **95.12** | **60.10** | **96.80** | 75.64 | 84.42 | **68.18** | **82.43** |\n| 40% | Random | 35.43 | 28.27 | 8.28 | 32.15 | 21.29 | 60.41 | 15.46 | 28.76 |\n| 40% | Frequency | 75.14 | 72.11 | 20.16 | 72.29 | 45.18 | 76.28 | 40.17 | 57.33 |\n| 40% | Gating | 77.19 | 74.32 | 22.54 | 74.28 | 47.21 | 77.31 | 42.15 | 59.29 |\n| 40% | RAEP | 93.42 | 90.47 | 52.42 | 92.58 | 43.12 | 83.15 | 64.41 | 74.22 |\n| 40% | EASY-EP | 94.12 | **91.07** | 53.12 | 93.14 | **73.28** | **83.51** | 65.14 | 79.05 |\n| 40% | **SHAPE** | **95.60** | 89.63 | **67.68** | **94.40** | 73.16 | 82.37 | **66.32** | **81.31** |\n\n### Peak VRAM\n\nPruning directly reduces the resident expert footprint. The measured peak VRAM decreases steadily under 20% and 40% expert pruning for both 32K and 8K context settings, with no additional training or architecture changes.\n\n<p align=\"center\">\n  <img src=\"assets/shape_memory_vram.png\" alt=\"Peak VRAM under 0, 20, and 40 percent expert pruning\" width=\"78%\">\n</p>\n\n## Repository Structure\n\n```text\nshapley-moe/\n|-- data/                         # Calibration data and download scripts\n|   |-- calibration/              # 25-example calibration sets\n|   |-- download_dataset.py\n|   `-- run_download.sh\n|-- analysis/                     # Routing trace collection and Shapley scoring\n|   |-- collect_activations.py\n|   |-- calc_shapley.py\n|   |-- run_collect.sh\n|   `-- run_calc_shapley.sh\n|-- pruning/                      # Expert selection and pruned-model export\n|   |-- methods/\n|   |   |-- select_by_shapley.py\n|   |   |-- select_by_easyep.py\n|   |   |-- select_by_reap.py\n|   |   |-- select_by_gating.py\n|   |   |-- select_by_frequency.py\n|   |   `-- select_by_random.py\n|   |-- save_model.py\n|   |-- run_select.sh\n|   `-- run_prune.sh\n|-- evaluation/                   # vLLM serving and EvalScope evaluation\n|-- configs/                      # Model paths and experiment settings\n|-- results/                      # Activations, Shapley values, selected experts\n|-- assets/                       # README figures\n`-- PROJECT_STRUCTURE.md\n```\n\n## Quick Start\n\n### 0. Configure Models and Experiments\n\nEdit the config files before running the pipeline:\n\n```bash\nvim configs/models.yaml\nvim configs/experiments.yaml\n```\n\n`configs/models.yaml` defines model paths, expert counts, top-k routing size, and MoE module patterns. `configs/experiments.yaml` defines calibration datasets, keep rates, pruning methods, and evaluation defaults.\n\n### 1. Prepare Calibration Data\n\n```bash\ncd data\n\n# Download one dataset with 25 examples.\n./run_download.sh gsm8k 25\n\n# Or download all datasets from configs/experiments.yaml.\n./run_download.sh --all\n```\n\n### 2. Collect Routing and Activation Statistics\n\nThis single pass collects the information needed by SHAPE and the baseline pruning methods.\n\n```bash\ncd analysis\n\n# Use a configured model name.\n./run_collect.sh -m qwen3-30b-a3b --all\n\n# Or use a full model path.\n./run_collect.sh -m /path/to/model --data ../data/calibration/gsm8k_25.json\n\n# Inspect configured models.\n./run_collect.sh --list-models\n```\n\nOutputs are written to:\n\n```text\nresults/{model_name}/activations/\n```\n\n### 3. Compute Shapley Values\n\n```bash\ncd analysis\n\n# Compute Shapley scores for all available activation files.\n./run_calc_shapley.sh -m qwen3-30b-a3b\n\n# Or compute one dataset.\n./run_calc_shapley.sh -m qwen3-30b-a3b -d gsm8k_25\n```\n\nOutputs are written to:\n\n```text\nresults/{model_name}/shapley_values/\n```\n\n### 4. Select Experts\n\n```bash\ncd pruning\n\n# SHAPE selection with 80% keep rate, i.e. 20% expert pruning.\n./run_select.sh -m qwen3-30b-a3b -d gsm8k_25 -M shapley -r 0.8\n\n# Run all configured keep rates.\n./run_select.sh -m qwen3-30b-a3b -d gsm8k_25 -M shapley --all-rates\n\n# Run all datasets and all configured methods.\n./run_select.sh -m qwen3-30b-a3b --all-datasets --all-methods --all-rates\n```\n\nSelected expert files are written to:\n\n```text\nresults/{model_name}/selected_experts/\n```\n\n### 5. Export a Pruned Model\n\n```bash\ncd pruning\n\n# Export a SHAPE-pruned model from the selected experts.\n./run_prune.sh -m qwen3-30b-a3b -d gsm8k_25 -r 0.8\n\n# Use another method or keep rate.\n./run_prune.sh -m gpt-oss-20b -d arc_easy_25 -M easyep -r 0.6\n```\n\nThe pruning script finds the matching selected-expert JSON and writes a pruned Hugging Face model directory.\n\n### 6. Evaluate\n\n```bash\ncd evaluation\n\n# Start a vLLM server from a configured model.\n./vllm-server.sh qwen3-30b-a3b\n\n# Or use a full model path and custom port.\n./vllm-server.sh /path/to/model -p 8801\n\n# Run EvalScope evaluation using configs/experiments.yaml.\npython run_evalscope.py\n```\n\n## Supported Models and Datasets\n\n### Models\n\n| Model name | Experts | Experts per token | Type |\n| --- | ---: | ---: | --- |\n| `qwen3-30b-a3b` | 128 | 8 | Qwen3 MoE |\n| `gpt-oss-20b` | 32 | 4 | GPT-OSS MoE |\n| `deepseekv2-lite-coder` | 64 | 6 | DeepSeek-V2 MoE |\n\n### Calibration and Evaluation Tasks\n\nThe default config includes calibration sets for code, math, reasoning, medical QA, NLP, and truthfulness:\n\n```text\nhumaneval_25, gsm8k_25, math_500_25, arc_easy_25, logiqa_25,\ngpqa_diamond_25, med_mcqa_25, pubmedqa_25, biomix_qa_25,\nontonotes5_25, truthful_qa_25\n```\n\nThe IJCNN 2026 main results report GSM8K, HumanEval, GPQA-Diamond, MATH-500, TruthfulQA, OntoNotes5, and MedMCQA.\n\n## Pruning Methods\n\n| Method | Signal | Description |\n| --- | --- | --- |\n| `shapley` | Coalition-aware Shapley-style contribution | Primary SHAPE method |\n| `easyep` | Task-aligned expert selection | EASY-EP baseline |\n| `reap` | Weighted activation norm | REAP/RAEP-style baseline |\n| `gating` | Mean router softmax score | Router-score baseline |\n| `frequency` | Activation count | Routing-frequency baseline |\n| `random` | Uniform random selection | Lower-bound baseline |\n\n## SHAPE Selection Strategies\n\n| Strategy | Description | Notes |\n| --- | --- | --- |\n| `alpha_per_layer` | Preserve an alpha fraction of Shapley mass in each layer | Recommended; used for SHAPE quality coverage |\n| `alpha_global` | Preserve alpha coverage globally | Can over-prune some layers |\n| `topk_per_layer` | Keep a fixed top-k expert count in every layer | Simple layer-wise baseline |\n| `topk_global` | Keep the globally highest-scoring experts | Can cause layer imbalance |\n\nIn the paper, the quality-coverage variant (`alpha_per_layer`) is the main SHAPE setting. It outperforms simplified global and rigid layer-wise variants because it preserves cooperative value while avoiding layer collapse.\n\n## Naming Conventions\n\n- Models: `qwen3-30b-a3b`, `gpt-oss-20b`, `deepseekv2-lite-coder`\n- Datasets: `{dataset}_{samples}`, for example `gsm8k_25`\n- Methods: `shapley`, `easyep`, `reap`, `gating`, `frequency`, `random`\n- Keep rates: `0.8` keeps 80% of experts and prunes 20%; `0.6` keeps 60% and prunes 40%\n- Selected expert files:\n  - SHAPE: `shapley_{strategy}_{dataset}_rate{XX}.json`\n  - Baselines: `{method}_{dataset}_rate{XX}.json`\n\n## Citation\n\nIf you use this repository, please cite:\n\n```bibtex\n@inproceedings{zhang2026shape,\n  title     = {SHAPE: Coalition-Aware Expert Pruning for Sparse Mixture-of-Experts LLMs},\n  author    = {Zhang, Yuhao and Jiang, HongXu and Zhang, YiXiang and Zhang, Zheng},\n  booktitle = {Proceedings of the International Joint Conference on Neural Networks},\n  year      = {2026}\n}\n```\n\n## Notes\n\n- SHAPE is post-training and does not require gradient updates.\n- The default calibration size in this repository is 25 examples per task.\n- The `-r` argument in scripts is a **keep rate**, not a removal rate.\n- Large models require local Hugging Face checkpoints and sufficient GPU memory for the unpruned calibration pass.\n"}, "e1933ae26f6ca7e8ad80f88520f18dca458c8602": {"t": 1785771692.0645704, "n": 8352, "h": "082544569c96f8a9", "c": "# Shapley-MoE: Expert Pruning Framework for Mixture-of-Experts\n\n## Project Structure\n\n```\nshapley-moe/\n│\n├── data/                               # Datasets\n│   ├── calibration/                    # Calibration data (for few-shot)\n│   │   ├── arc_easy_25.json\n│   │   ├── gsm8k_25.json\n│   │   └── ...\n│   ├── download_dataset.py\n│   └── run_download.sh\n│\n├── analysis/                           # Analysis scripts\n│   ├── collect_activations.py          # Collect activation information (all-in-one)\n│   ├── calc_shapley.py                 # Compute Shapley values\n│   ├── run_collect.sh                  # Batch activation collection\n│   └── run_calc_shapley.sh             # Batch Shapley computation\n│\n├── pruning/                            # Pruning related\n│   ├── methods/                        # Pruning methods\n│   │   ├── select_by_shapley.py        # Shapley value pruning (primary method)\n│   │   ├── select_by_easyep.py         # EASYEP pruning\n│   │   ├── select_by_reap.py          # REAP pruning\n│   │   ├── select_by_gating.py         # Gating Score pruning\n│   │   ├── select_by_frequency.py      # Activation frequency pruning (baseline)\n│   │   └── select_by_random.py         # Random pruning (baseline)\n│   ├── save_model.py                   # Save pruned model\n│   └── run_select.sh                   # Unified expert selection script\n│\n├── evaluation/                         # Evaluation scripts\n│   ├── run_evalscope.py\n│   └── vllm_server.sh\n│\n├── results/                            # All results (organized by model)\n│   ├── {model_name}/                   # e.g. qwen3-30b-a3b/\n│   │   ├── activations/                # Activation statistics\n│   │   │   ├── {dataset}_shapley.json\n│   │   │   ├── {dataset}_gating.json\n│   │   │   ├── {dataset}_easyep.json\n│   │   │   └── {dataset}_reap.json\n│   │   ├── shapley_values/             # Shapley values\n│   │   │   └── {dataset}_shapley.csv\n│   │   └── selected_experts/           # Selected experts\n│   │       ├── shapley_{strategy}_{dataset}_rate{XX}.json\n│   │       ├── {method}_{dataset}_rate{XX}.json  # Other methods\n│   │       └── ...\n│   └── ...\n│\n├── models/                             # Pruned models\n│   └── {model}_{method}_rate{XX}/\n│\n└── configs/                            # Configuration files\n    ├── models.yaml                     # Model configuration\n    └── experiments.yaml                # Experiment configuration\n```\n\n## Quick Start\n\n### 0. Configuration (recommended to modify first)\n\nAll scripts prioritize reading from the `configs/` directory:\n\n```bash\n# Modify model paths\nvim configs/models.yaml\n\n# Modify experiment parameters (pruning rates, datasets, etc.)\nvim configs/experiments.yaml\n```\n\n### 1. Data Preparation\n\n```bash\ncd data\n\n# Download a single dataset\n./run_download.sh gsm8k 25\n\n# Download all datasets from config\n./run_download.sh --all\n```\n\n### 2. Collect Activation Information\n\nCollect information needed by all pruning methods at once (Shapley/Gating/EASYEP/REAP):\n\n```bash\ncd analysis\n\n# Use model name (path auto-read from config)\n./run_collect.sh -m qwen3-30b-a3b --all\n\n# Or use full path\n./run_collect.sh -m /path/to/model --data ../data/calibration/gsm8k_25.json\n\n# View available models in config\n./run_collect.sh --list-models\n```\n\n### 3. Compute Shapley Values (optional)\n\n```bash\n./run_calc_shapley.sh -m MODEL_NAME\n```\n\n### 4. Expert Selection\n\n```bash\ncd ../pruning\n\n# Use Shapley pruning, retain 50% (pruning rate read from config)\n./run_select.sh -m qwen3-30b-a3b -d gsm8k_25 -M shapley -r 0.5\n\n./run_select.sh -m qwen3-30b-a3b -d gsm8k_25 -M shapley --all-rates\n\n# Batch processing (using all pruning rates from config)\n./run_select.sh -m qwen3-30b-a3b --all-datasets -M easyep --all-rates\n\n# Use all methods from config\n./run_select.sh -m qwen3-30b-a3b -d gsm8k_25 --all-methods --all-rates\n```\n\n### 5. Save Pruned Model\n\n```bash\ncd ../pruning\n\n# Specify model, dataset, pruning rate (auto-finds corresponding selection file)\n./run_prune.sh -m qwen3-30b-a3b -d gsm8k_25 -r 0.8\n\n# Use other methods\n./run_prune.sh -m gpt-oss-20b -d arc_easy_25 -M easyep -r 0.6\n\n# Parameter description\n#   -m MODEL     Model name\n#   -d DATASET   Dataset name\n#   -M METHOD    Pruning method (default: shapley)\n#   -s STRATEGY  Shapley strategy (default: alpha_per_layer)\n#   -r RATE      Pruning rate (default: 0.8)\n```\n\n### 6. Evaluation\n\n```bash\ncd ../evaluation\n\n# Start model service (use model name, path auto-read from config)\n./vllm-server.sh qwen3-30b-a3b\n\n# Or specify full path\n./vllm-server.sh /path/to/model -p 8801\n\n# Run evaluation (datasets read from config)\npython run_evalscope.py\n\n# View available datasets\npython run_evalscope.py --list-datasets\n```\n\n## Pruning Methods\n\n| Method | Description | Formula |\n|--------|-------------|---------|\n| **Shapley** | Marginal contribution-based pruning (primary method) | Shapley Value |\n| **EASYEP** | Considers MoE impact on tokens | `weight × (1 - simibr) × norm` |\n| **REAP** | Weighted norm | `weight × norm` |\n| **Gating** | Based on router softmax scores | `mean(softmax(gating_logits))` |\n| **Frequency** | Based on activation frequency (baseline) | `count(activations)` |\n| **Random** | Random selection (baseline) | random |\n\n### Shapley Pruning Strategies\n\nThe Shapley method supports four strategies:\n\n| Strategy | Description | Use Case |\n|----------|-------------|----------|\n| `topk_per_layer` | Select top-k experts with highest Shapley values per layer | **Recommended**, simple and straightforward |\n| `topk_global` | Select experts with highest Shapley values globally | Some layers can retain more experts |\n| `alpha_per_layer` | Cumulative Shapley values reach alpha proportion per layer | Considers contribution distribution |\n| `alpha_global` | Cumulative Shapley values reach alpha proportion globally | Global optimization |\n\n**TopK vs Alpha:**\n- `topk`: Directly sorted by Shapley value, select top k. Simple, highly interpretable.\n- `alpha`: Select minimum experts whose cumulative Shapley value reaches the target proportion of total. Considers contribution distribution.\n\n**Per Layer vs Global:**\n- `per_layer`: Independent selection per layer, ensures sufficient experts per layer.\n- `global`: Unified global selection, some layers may have fewer experts.\n\n```bash\n# Use different strategies\npython pruning/methods/select_by_shapley.py \\\n    --input results/model/shapley_values/gsm8k_25_shapley.csv \\\n    --output selected.json \\\n    --pruning_rate 0.5 \\\n    --strategy topk_per_layer  # or topk_global, alpha_per_layer, alpha_global\n```\n\n## Naming Conventions\n\n- **Models**: `qwen3-30b-a3b`, `gpt-oss-20b`, `deepseekv2-lite-coder`\n- **Datasets**: `{name}_{samples}` e.g. `gsm8k_25`\n- **Pruning methods**: `shapley`, `easyep`, `reap`, `gating`, `frequency`, `random`\n- **Shapley strategies**: `alpha_per_layer`, `alpha_global`, `topk_per_layer`, `topk_global`\n- **Pruning rate**: `rate0_8`, `rate0_6` (retention ratio, e.g. 0.8 means retain 80%)\n- **Selected expert files**:\n  - Shapley: `shapley_{strategy}_{dataset}_rate{XX}.json`\n  - Other methods: `{method}_{dataset}_rate{XX}.json`\n\n## Configuration\n\nConfiguration files are in the `configs/` directory. **All scripts prioritize reading from config files**:\n\n### models.yaml - Model Configuration\n\n```yaml\nmodels:\n  qwen3-30b-a3b:\n    path: /path/to/qwen3-30b-a3b    # Model path\n    num_experts: 128                 # Total number of experts\n    num_experts_per_tok: 8           # Number of experts activated per token\n    type: qwen3                      # Model type\n```\n\n### experiments.yaml - Experiment Configuration\n\n```yaml\n# Dataset list\ndatasets:\n  - humaneval_25\n  - gsm8k_25\n  - ...\n\n# Pruning rates (retention ratio)\npruning_rates:\n  - 0.80   # Retain 80%\n  - 0.60   # Retain 60%\n\n# Pruning methods\npruning_methods:\n  - shapley\n  - easyep\n  - ...\n\n# Defaults\ndefaults:\n  pruning_rate: 0.5\n  pruning_method: shapley\n  shapley_strategy: alpha_per_layer\n  max_new_tokens: 512\n  eval_port: 8801\n```\n\n### How Scripts Use Configuration\n\n| Script | Config Read |\n|--------|-------------|\n| `run_collect.sh` | Model path, max_new_tokens, device |\n| `run_select.sh` | Pruning rates, pruning methods, Shapley strategies |\n| `run_download.sh` | Dataset list |\n| `vllm-server.sh` | Model path, eval_port |\n| `run_evalscope.py` | Evaluation datasets, batch_size, timeout |\n"}, "9d13ac9f20239e06f71a241986c867da438f9805": {"t": 1785771694.0892482, "n": 6127, "h": "fac43b2c03e1a806", "c": "# Shapley-MoE: MoE 专家剪枝框架\n\n## 📁 项目结构\n\n```\nshapley-moe/\n│\n├── data/                               # 数据集\n│   ├── calibration/                    # 校准数据（few-shot 用）\n│   │   ├── arc_easy_25.json\n│   │   ├── gsm8k_25.json\n│   │   └── ...\n│   ├── download_dataset.py\n│   └── run_download.sh\n│\n├── analysis/                           # 分析脚本\n│   ├── collect_activations.py          # 收集激活信息（一体化）\n│   ├── calc_shapley.py                 # 计算 Shapley 值\n│   ├── run_collect.sh                  # 批量收集激活\n│   └── run_calc_shapley.sh             # 批量计算 Shapley\n│\n├── pruning/                            # 剪枝相关\n│   ├── methods/                        # 剪枝方法\n│   │   ├── select_by_shapley.py        # Shapley 值剪枝（主方法）\n│   │   ├── select_by_easyep.py         # EASYEP 剪枝\n│   │   ├── select_by_reap.py           # REAP 剪枝\n│   │   ├── select_by_gating.py         # Gating Score 剪枝\n│   │   ├── select_by_frequency.py      # 激活频率剪枝（基线）\n│   │   └── select_by_random.py         # 随机剪枝（基线）\n│   ├── save_model.py                   # 保存剪枝后模型\n│   └── run_select.sh                   # 统一的专家选择脚本\n│\n├── evaluation/                         # 评测脚本\n│   ├── run_evalscope.py\n│   └── vllm_server.sh\n│\n├── results/                            # 所有结果（按模型组织）\n│   ├── {model_name}/                   # 如 qwen3-30b-a3b/\n│   │   ├── activations/                # 激活统计\n│   │   │   ├── {dataset}_shapley.json\n│   │   │   ├── {dataset}_gating.json\n│   │   │   ├── {dataset}_easyep.json\n│   │   │   └── {dataset}_reap.json\n│   │   ├── shapley_values/             # Shapley 值\n│   │   │   └── {dataset}_shapley.csv\n│   │   └── selected_experts/           # 选中的专家\n│   │       ├── shapley_{strategy}_{dataset}_rate{XX}.json\n│   │       ├── {method}_{dataset}_rate{XX}.json  # 其他方法\n│   │       └── ...\n│   └── ...\n│\n├── models/                             # 剪枝后的模型\n│   └── {model}_{method}_rate{XX}/\n│\n└── configs/                            # 配置文件\n    ├── models.yaml                     # 模型配置\n    └── experiments.yaml                # 实验配置\n```\n\n## 🚀 快速开始\n\n### 0. 配置（推荐先修改）\n\n所有脚本优先从 `configs/` 目录读取配置：\n\n```bash\n# 修改模型路径\nvim configs/models.yaml\n\n# 修改实验参数（剪枝率、数据集等）\nvim configs/experiments.yaml\n```\n\n### 1. 数据准备\n\n```bash\ncd data\n\n# 下载单个数据集\n./run_download.sh gsm8k 25\n\n# 下载配置中的所有数据集\n./run_download.sh --all\n```\n\n### 2. 收集激活信息\n\n一次收集所有剪枝方法需要的信息（Shapley/Gating/EASYEP/REAP）：\n\n```bash\ncd analysis\n\n# 使用模型名称（自动从配置读取路径）\n./run_collect.sh -m qwen3-30b-a3b --all\n\n# 或使用完整路径\n./run_collect.sh -m /path/to/model --data ../data/calibration/gsm8k_25.json\n\n# 查看配置中的可用模型\n./run_collect.sh --list-models\n```\n\n### 3. 计算 Shapley 值（可选）\n\n```bash\n./run_calc_shapley.sh -m MODEL_NAME\n```\n\n### 4. 专家选择\n\n```bash\ncd ../pruning\n\n# 使用 Shapley 剪枝，保留 50%（剪枝率从配置读取）\n./run_select.sh -m qwen3-30b-a3b -d gsm8k_25 -M shapley -r 0.5\n\n./run_select.sh -m qwen3-30b-a3b -d gsm8k_25 -M shapley --all-rates\n\n# 批量处理（使用配置中的所有剪枝率）\n./run_select.sh -m qwen3-30b-a3b --all-datasets -M easyep --all-rates\n\n# 使用配置中的所有方法\n./run_select.sh -m qwen3-30b-a3b -d gsm8k_25 --all-methods --all-rates\n```\n\n### 5. 保存剪枝模型\n\n```bash\ncd ../pruning\n\n# 指定模型、数据集、剪枝率（自动查找对应的选择文件）\n./run_prune.sh -m qwen3-30b-a3b -d gsm8k_25 -r 0.8\n\n# 使用其他方法\n./run_prune.sh -m gpt-oss-20b -d arc_easy_25 -M easyep -r 0.6\n\n# 参数说明\n#   -m MODEL     模型名称\n#   -d DATASET   数据集名称\n#   -M METHOD    剪枝方法（默认: shapley）\n#   -s STRATEGY  Shapley策略（默认: alpha_per_layer）\n#   -r RATE      剪枝率（默认: 0.8）\n```\n\n### 6. 评测\n\n```bash\ncd ../evaluation\n\n# 启动模型服务（使用模型名称，自动读取配置中的路径）\n./vllm-server.sh qwen3-30b-a3b\n\n# 或指定完整路径\n./vllm-server.sh /path/to/model -p 8801\n\n# 运行评测（数据集从配置读取）\npython run_evalscope.py\n\n# 查看可用数据集\npython run_evalscope.py --list-datasets\n```\n\n## 📊 剪枝方法\n\n| 方法 | 描述 | 公式 |\n|------|------|------|\n| **Shapley** | 基于边际贡献的剪枝（主方法） | Shapley Value |\n| **EASYEP** | 考虑 MoE 对 token 影响程度 | `weight × (1 - simibr) × norm` |\n| **REAP** | 加权范数 | `weight × norm` |\n| **Gating** | 基于 router softmax 分数 | `mean(softmax(gating_logits))` |\n| **Frequency** | 基于激活频率（基线） | `count(activations)` |\n| **Random** | 随机选择（基线） | random |\n\n### Shapley 剪枝策略\n\nShapley 方法支持四种策略：\n\n| 策略 | 描述 | 适用场景 |\n|------|------|----------|\n| `topk_per_layer` | 每层选择 Shapley 值最高的 top-k 专家 | **推荐**，简单直接 |\n| `topk_global` | 全局选择 Shapley 值最高的专家 | 某些层可保留更多专家 |\n| `alpha_per_layer` | 每层累积 Shapley 值达到 alpha 比例 | 考虑贡献分布 |\n| `alpha_global` | 全局累积 Shapley 值达到 alpha 比例 | 全局优化 |\n\n**TopK vs Alpha:**\n- `topk`: 直接按 Shapley 值大小排序，选择前 k 个。简单、可解释性强。\n- `alpha`: 选择累积 Shapley 值达到总量 alpha 比例的最少专家。考虑了贡献分布。\n\n**Per Layer vs Global:**\n- `per_layer`: 每层独立选择，保证每层都有足够专家。\n- `global`: 全局统一选择，某些层可能专家较少。\n\n```bash\n# 使用不同策略\npython pruning/methods/select_by_shapley.py \\\n    --input results/model/shapley_values/gsm8k_25_shapley.csv \\\n    --output selected.json \\\n    --pruning_rate 0.5 \\\n    --strategy topk_per_layer  # 或 topk_global, alpha_per_layer, alpha_global\n```\n\n## 📝 命名规范\n\n- **模型**: `qwen3-30b-a3b`, `gpt-oss-20b`, `deepseekv2-lite-coder`\n- **数据集**: `{name}_{samples}` 如 `gsm8k_25`\n- **剪枝方法**: `shapley`, `easyep`, `reap`, `gating`, `frequency`, `random`\n- **Shapley 策略**: `alpha_per_layer`, `alpha_global`, `topk_per_layer`, `topk_global`\n- **剪枝率**: `rate0_8`, `rate0_6` (保留比例，如 0.8 表示保留 80%)\n- **选中专家文件**:\n  - Shapley: `shapley_{strategy}_{dataset}_rate{XX}.json`\n  - 其他方法: `{method}_{dataset}_rate{XX}.json`\n\n## 🔧 配置\n\n配置文件位于 `configs/` 目录，**所有脚本优先从配置文件读取信息**：\n\n### models.yaml - 模型配置\n\n```yaml\nmodels:\n  qwen3-30b-a3b:\n    path: /path/to/qwen3-30b-a3b    # 模型路径\n    num_experts: 128                 # 专家总数\n    num_experts_per_tok: 8           # 每 token 激活专家数\n    type: qwen3                      # 模型类型\n```\n\n### experiments.yaml - 实验配置\n\n```yaml\n# 数据集列表\ndatasets:\n  - humaneval_25\n  - gsm8k_25\n  - ...\n\n# 剪枝率（保留比例）\npruning_rates:\n  - 0.80   # 保留 80%\n  - 0.60   # 保留 60%\n\n# 剪枝方法\npruning_methods:\n  - shapley\n  - easyep\n  - ...\n\n# 默认值\ndefaults:\n  pruning_rate: 0.5\n  pruning_method: shapley\n  shapley_strategy: alpha_per_layer\n  max_new_tokens: 512\n  eval_port: 8801\n```\n\n### 脚本如何使用配置\n\n| 脚本 | 读取的配置 |\n|------|-----------|\n| `run_collect.sh` | 模型路径、max_new_tokens、device |\n| `run_select.sh` | 剪枝率、剪枝方法、Shapley 策略 |\n| `run_download.sh` | 数据集列表 |\n| `vllm-server.sh` | 模型路径、eval_port |\n| `run_evalscope.py` | 评测数据集、batch_size、timeout |\n\n"}, "7e91df91f20f3f1378398a940c58d174055e2ceb": {"t": 1785771695.610284, "n": 6058, "h": "eddc531800136f7e", "c": "# Shapley-MoE: MoE 专家剪枝框架\n\n## 📁 项目结构\n\n```\nshapley-moe/\n│\n├── data/                               # 数据集\n│   ├── calibration/                    # 校准数据（few-shot 用）\n│   │   ├── arc_easy_25.json\n│   │   ├── gsm8k_25.json\n│   │   └── ...\n│   ├── download_dataset.py\n│   └── run_download.sh\n│\n├── analysis/                           # 分析脚本\n│   ├── collect_activations.py          # 收集激活信息（一体化）\n│   ├── calc_shapley.py                 # 计算 Shapley 值\n│   ├── run_collect.sh                  # 批量收集激活\n│   └── run_calc_shapley.sh             # 批量计算 Shapley\n│\n├── pruning/                            # 剪枝相关\n│   ├── methods/                        # 剪枝方法\n│   │   ├── select_by_shapley.py        # Shapley 值剪枝（主方法）\n│   │   ├── select_by_easyep.py         # EASYEP 剪枝\n│   │   ├── select_by_reap.py           # REAP 剪枝\n│   │   ├── select_by_gating.py         # Gating Score 剪枝\n│   │   ├── select_by_frequency.py      # 激活频率剪枝（基线）\n│   │   └── select_by_random.py         # 随机剪枝（基线）\n│   ├── save_model.py                   # 保存剪枝后模型\n│   └── run_select.sh                   # 统一的专家选择脚本\n│\n├── evaluation/                         # 评测脚本\n│   ├── run_evalscope.py\n│   └── vllm_server.sh\n│\n├── results/                            # 所有结果（按模型组织）\n│   ├── {model_name}/                   # 如 qwen3-30b-a3b/\n│   │   ├── activations/                # 激活统计\n│   │   │   ├── {dataset}_shapley.json\n│   │   │   ├── {dataset}_gating.json\n│   │   │   ├── {dataset}_easyep.json\n│   │   │   └── {dataset}_reap.json\n│   │   ├── shapley_values/             # Shapley 值\n│   │   │   └── {dataset}_shapley.csv\n│   │   └── selected_experts/           # 选中的专家\n│   │       ├── shapley_{strategy}_{dataset}_rate{XX}.json\n│   │       ├── {method}_{dataset}_rate{XX}.json  # 其他方法\n│   │       └── ...\n│   └── ...\n│\n├── models/                             # 剪枝后的模型\n│   └── {model}_{method}_rate{XX}/\n│\n└── configs/                            # 配置文件\n    ├── models.yaml                     # 模型配置\n    └── experiments.yaml                # 实验配置\n```\n\n## 🚀 快速开始\n\n### 0. 配置（推荐先修改）\n\n所有脚本优先从 `configs/` 目录读取配置：\n\n```bash\n# 修改模型路径\nvim configs/models.yaml\n\n# 修改实验参数（剪枝率、数据集等）\nvim configs/experiments.yaml\n```\n\n### 1. 数据准备\n\n```bash\ncd data\n\n# 下载单个数据集\n./run_download.sh gsm8k 25\n\n# 下载配置中的所有数据集\n./run_download.sh --all\n```\n\n### 2. 收集激活信息\n\n一次收集所有剪枝方法需要的信息（Shapley/Gating/EASYEP/REAP）：\n\n```bash\ncd analysis\n\n# 使用模型名称（自动从配置读取路径）\n./run_collect.sh -m qwen3-30b-a3b --all\n\n# 或使用完整路径\n./run_collect.sh -m /path/to/model --data ../data/calibration/gsm8k_25.json\n\n# 查看配置中的可用模型\n./run_collect.sh --list-models\n```\n\n### 3. 计算 Shapley 值（可选）\n\n```bash\n./run_calc_shapley.sh -m MODEL_NAME\n```\n\n### 4. 专家选择\n\n```bash\ncd ../pruning\n\n# 使用 Shapley 剪枝，保留 50%（剪枝率从配置读取）\n./run_select.sh -m qwen3-30b-a3b -d gsm8k_25 -M shapley -r 0.5\n\n# 批量处理（使用配置中的所有剪枝率）\n./run_select.sh -m qwen3-30b-a3b --all-datasets -M easyep --all-rates\n\n# 使用配置中的所有方法\n./run_select.sh -m qwen3-30b-a3b -d gsm8k_25 --all-methods --all-rates\n```\n\n### 5. 保存剪枝模型\n\n```bash\ncd ../pruning\n\n# 指定模型、数据集、剪枝率（自动查找对应的选择文件）\n./run_prune.sh -m qwen3-30b-a3b -d gsm8k_25 -r 0.8\n\n# 使用其他方法\n./run_prune.sh -m gpt-oss-20b -d arc_easy_25 -M easyep -r 0.6\n\n# 参数说明\n#   -m MODEL     模型名称\n#   -d DATASET   数据集名称\n#   -M METHOD    剪枝方法（默认: shapley）\n#   -s STRATEGY  Shapley策略（默认: alpha_per_layer）\n#   -r RATE      剪枝率（默认: 0.8）\n```\n\n### 6. 评测\n\n```bash\ncd ../evaluation\n\n# 启动模型服务（使用模型名称，自动读取配置中的路径）\n./vllm-server.sh qwen3-30b-a3b\n\n# 或指定完整路径\n./vllm-server.sh /path/to/model -p 8801\n\n# 运行评测（数据集从配置读取）\npython run_evalscope.py\n\n# 查看可用数据集\npython run_evalscope.py --list-datasets\n```\n\n## 📊 剪枝方法\n\n| 方法 | 描述 | 公式 |\n|------|------|------|\n| **Shapley** | 基于边际贡献的剪枝（主方法） | Shapley Value |\n| **EASYEP** | 考虑 MoE 对 token 影响程度 | `weight × (1 - simibr) × norm` |\n| **REAP** | 加权范数 | `weight × norm` |\n| **Gating** | 基于 router softmax 分数 | `mean(softmax(gating_logits))` |\n| **Frequency** | 基于激活频率（基线） | `count(activations)` |\n| **Random** | 随机选择（基线） | random |\n\n### Shapley 剪枝策略\n\nShapley 方法支持四种策略：\n\n| 策略 | 描述 | 适用场景 |\n|------|------|----------|\n| `topk_per_layer` | 每层选择 Shapley 值最高的 top-k 专家 | **推荐**，简单直接 |\n| `topk_global` | 全局选择 Shapley 值最高的专家 | 某些层可保留更多专家 |\n| `alpha_per_layer` | 每层累积 Shapley 值达到 alpha 比例 | 考虑贡献分布 |\n| `alpha_global` | 全局累积 Shapley 值达到 alpha 比例 | 全局优化 |\n\n**TopK vs Alpha:**\n- `topk`: 直接按 Shapley 值大小排序，选择前 k 个。简单、可解释性强。\n- `alpha`: 选择累积 Shapley 值达到总量 alpha 比例的最少专家。考虑了贡献分布。\n\n**Per Layer vs Global:**\n- `per_layer`: 每层独立选择，保证每层都有足够专家。\n- `global`: 全局统一选择，某些层可能专家较少。\n\n```bash\n# 使用不同策略\npython pruning/methods/select_by_shapley.py \\\n    --input results/model/shapley_values/gsm8k_25_shapley.csv \\\n    --output selected.json \\\n    --pruning_rate 0.5 \\\n    --strategy topk_per_layer  # 或 topk_global, alpha_per_layer, alpha_global\n```\n\n## 📝 命名规范\n\n- **模型**: `qwen3-30b-a3b`, `gpt-oss-20b`, `deepseekv2-lite-coder`\n- **数据集**: `{name}_{samples}` 如 `gsm8k_25`\n- **剪枝方法**: `shapley`, `easyep`, `reap`, `gating`, `frequency`, `random`\n- **Shapley 策略**: `alpha_per_layer`, `alpha_global`, `topk_per_layer`, `topk_global`\n- **剪枝率**: `rate0_8`, `rate0_6` (保留比例，如 0.8 表示保留 80%)\n- **选中专家文件**:\n  - Shapley: `shapley_{strategy}_{dataset}_rate{XX}.json`\n  - 其他方法: `{method}_{dataset}_rate{XX}.json`\n\n## 🔧 配置\n\n配置文件位于 `configs/` 目录，**所有脚本优先从配置文件读取信息**：\n\n### models.yaml - 模型配置\n\n```yaml\nmodels:\n  qwen3-30b-a3b:\n    path: /path/to/qwen3-30b-a3b    # 模型路径\n    num_experts: 128                 # 专家总数\n    num_experts_per_tok: 8           # 每 token 激活专家数\n    type: qwen3                      # 模型类型\n```\n\n### experiments.yaml - 实验配置\n\n```yaml\n# 数据集列表\ndatasets:\n  - humaneval_25\n  - gsm8k_25\n  - ...\n\n# 剪枝率（保留比例）\npruning_rates:\n  - 0.80   # 保留 80%\n  - 0.60   # 保留 60%\n\n# 剪枝方法\npruning_methods:\n  - shapley\n  - easyep\n  - ...\n\n# 默认值\ndefaults:\n  pruning_rate: 0.5\n  pruning_method: shapley\n  shapley_strategy: alpha_per_layer\n  max_new_tokens: 512\n  eval_port: 8801\n```\n\n### 脚本如何使用配置\n\n| 脚本 | 读取的配置 |\n|------|-----------|\n| `run_collect.sh` | 模型路径、max_new_tokens、device |\n| `run_select.sh` | 剪枝率、剪枝方法、Shapley 策略 |\n| `run_download.sh` | 数据集列表 |\n| `vllm-server.sh` | 模型路径、eval_port |\n| `run_evalscope.py` | 评测数据集、batch_size、timeout |\n\n"}, "64037963af76708af1b78f47c6b5d1d99fc70e73": {"t": 1785771698.2116446, "n": 5921, "h": "c87908824ad436d9", "c": "# Shapley-MoE: MoE 专家剪枝框架\n\n## 📁 项目结构\n\n```\nshapley-moe/\n│\n├── data/                               # 数据集\n│   ├── calibration/                    # 校准数据（few-shot 用）\n│   │   ├── arc_easy_25.json\n│   │   ├── gsm8k_25.json\n│   │   └── ...\n│   ├── download_dataset.py\n│   └── run_download.sh\n│\n├── analysis/                           # 分析脚本\n│   ├── collect_activations.py          # 收集激活信息（一体化）\n│   ├── calc_shapley.py                 # 计算 Shapley 值\n│   ├── run_collect.sh                  # 批量收集激活\n│   └── run_calc_shapley.sh             # 批量计算 Shapley\n│\n├── pruning/                            # 剪枝相关\n│   ├── methods/                        # 剪枝方法\n│   │   ├── select_by_shapley.py        # Shapley 值剪枝（主方法）\n│   │   ├── select_by_easyep.py         # EASYEP 剪枝\n│   │   ├── select_by_reap.py           # REAP 剪枝\n│   │   ├── select_by_gating.py         # Gating Score 剪枝\n│   │   ├── select_by_frequency.py      # 激活频率剪枝（基线）\n│   │   └── select_by_random.py         # 随机剪枝（基线）\n│   ├── save_model.py                   # 保存剪枝后模型\n│   └── run_select.sh                   # 统一的专家选择脚本\n│\n├── evaluation/                         # 评测脚本\n│   ├── run_evalscope.py\n│   └── vllm_server.sh\n│\n├── results/                            # 所有结果（按模型组织）\n│   ├── {model_name}/                   # 如 qwen3-30b-a3b/\n│   │   ├── activations/                # 激活统计\n│   │   │   ├── {dataset}_shapley.json\n│   │   │   ├── {dataset}_gating.json\n│   │   │   ├── {dataset}_easyep.json\n│   │   │   └── {dataset}_reap.json\n│   │   ├── shapley_values/             # Shapley 值\n│   │   │   └── {dataset}_shapley.csv\n│   │   └── selected_experts/           # 选中的专家\n│   │       ├── shapley_{strategy}_{dataset}_rate{XX}.json\n│   │       ├── {method}_{dataset}_rate{XX}.json  # 其他方法\n│   │       └── ...\n│   └── ...\n│\n├── models/                             # 剪枝后的模型\n│   └── {model}_{method}_rate{XX}/\n│\n└── configs/                            # 配置文件\n    ├── models.yaml                     # 模型配置\n    └── experiments.yaml                # 实验配置\n```\n\n## 🚀 快速开始\n\n### 0. 配置（推荐先修改）\n\n所有脚本优先从 `configs/` 目录读取配置：\n\n```bash\n# 修改模型路径\nvim configs/models.yaml\n\n# 修改实验参数（剪枝率、数据集等）\nvim configs/experiments.yaml\n```\n\n### 1. 数据准备\n\n```bash\ncd data\n\n# 下载单个数据集\n./run_download.sh gsm8k 25\n\n# 下载配置中的所有数据集\n./run_download.sh --all\n```\n\n### 2. 收集激活信息\n\n一次收集所有剪枝方法需要的信息（Shapley/Gating/EASYEP/REAP）：\n\n```bash\ncd analysis\n\n# 使用模型名称（自动从配置读取路径）\n./run_collect.sh -m qwen3-30b-a3b --all\n\n# 或使用完整路径\n./run_collect.sh -m /path/to/model --data ../data/calibration/gsm8k_25.json\n\n# 查看配置中的可用模型\n./run_collect.sh --list-models\n```\n\n### 3. 计算 Shapley 值（可选）\n\n```bash\n./run_calc_shapley.sh -m MODEL_NAME\n```\n\n### 4. 专家选择\n\n```bash\ncd ../pruning\n\n# 使用 Shapley 剪枝，保留 50%（剪枝率从配置读取）\n./run_select.sh -m qwen3-30b-a3b -d gsm8k_25 -M shapley -r 0.5\n\n# 批量处理（使用配置中的所有剪枝率）\n./run_select.sh -m qwen3-30b-a3b --all-datasets -M easyep --all-rates\n\n# 使用配置中的所有方法\n./run_select.sh -m qwen3-30b-a3b -d gsm8k_25 --all-methods --all-rates\n```\n\n### 5. 保存剪枝模型\n\n```bash\npython save_model.py \\\n    --model /path/to/original/model \\\n    --selection ../results/{model}/selected_experts/{method}_{dataset}_rate0_5.json \\\n    --output ../models/{model}_{method}_rate0_5\n```\n\n### 6. 评测\n\n```bash\ncd ../evaluation\n\n# 启动模型服务（使用模型名称，自动读取配置中的路径）\n./vllm-server.sh qwen3-30b-a3b\n\n# 或指定完整路径\n./vllm-server.sh /path/to/model -p 8801\n\n# 运行评测（数据集从配置读取）\npython run_evalscope.py\n\n# 查看可用数据集\npython run_evalscope.py --list-datasets\n```\n\n## 📊 剪枝方法\n\n| 方法 | 描述 | 公式 |\n|------|------|------|\n| **Shapley** | 基于边际贡献的剪枝（主方法） | Shapley Value |\n| **EASYEP** | 考虑 MoE 对 token 影响程度 | `weight × (1 - simibr) × norm` |\n| **REAP** | 加权范数 | `weight × norm` |\n| **Gating** | 基于 router softmax 分数 | `mean(softmax(gating_logits))` |\n| **Frequency** | 基于激活频率（基线） | `count(activations)` |\n| **Random** | 随机选择（基线） | random |\n\n### Shapley 剪枝策略\n\nShapley 方法支持四种策略：\n\n| 策略 | 描述 | 适用场景 |\n|------|------|----------|\n| `topk_per_layer` | 每层选择 Shapley 值最高的 top-k 专家 | **推荐**，简单直接 |\n| `topk_global` | 全局选择 Shapley 值最高的专家 | 某些层可保留更多专家 |\n| `alpha_per_layer` | 每层累积 Shapley 值达到 alpha 比例 | 考虑贡献分布 |\n| `alpha_global` | 全局累积 Shapley 值达到 alpha 比例 | 全局优化 |\n\n**TopK vs Alpha:**\n- `topk`: 直接按 Shapley 值大小排序，选择前 k 个。简单、可解释性强。\n- `alpha`: 选择累积 Shapley 值达到总量 alpha 比例的最少专家。考虑了贡献分布。\n\n**Per Layer vs Global:**\n- `per_layer`: 每层独立选择，保证每层都有足够专家。\n- `global`: 全局统一选择，某些层可能专家较少。\n\n```bash\n# 使用不同策略\npython pruning/methods/select_by_shapley.py \\\n    --input results/model/shapley_values/gsm8k_25_shapley.csv \\\n    --output selected.json \\\n    --pruning_rate 0.5 \\\n    --strategy topk_per_layer  # 或 topk_global, alpha_per_layer, alpha_global\n```\n\n## 📝 命名规范\n\n- **模型**: `qwen3-30b-a3b`, `gpt-oss-20b`, `deepseekv2-lite-coder`\n- **数据集**: `{name}_{samples}` 如 `gsm8k_25`\n- **剪枝方法**: `shapley`, `easyep`, `reap`, `gating`, `frequency`, `random`\n- **Shapley 策略**: `alpha_per_layer`, `alpha_global`, `topk_per_layer`, `topk_global`\n- **剪枝率**: `rate0_8`, `rate0_6` (保留比例，如 0.8 表示保留 80%)\n- **选中专家文件**:\n  - Shapley: `shapley_{strategy}_{dataset}_rate{XX}.json`\n  - 其他方法: `{method}_{dataset}_rate{XX}.json`\n\n## 🔧 配置\n\n配置文件位于 `configs/` 目录，**所有脚本优先从配置文件读取信息**：\n\n### models.yaml - 模型配置\n\n```yaml\nmodels:\n  qwen3-30b-a3b:\n    path: /path/to/qwen3-30b-a3b    # 模型路径\n    num_experts: 128                 # 专家总数\n    num_experts_per_tok: 8           # 每 token 激活专家数\n    type: qwen3                      # 模型类型\n```\n\n### experiments.yaml - 实验配置\n\n```yaml\n# 数据集列表\ndatasets:\n  - humaneval_25\n  - gsm8k_25\n  - ...\n\n# 剪枝率（保留比例）\npruning_rates:\n  - 0.80   # 保留 80%\n  - 0.60   # 保留 60%\n\n# 剪枝方法\npruning_methods:\n  - shapley\n  - easyep\n  - ...\n\n# 默认值\ndefaults:\n  pruning_rate: 0.5\n  pruning_method: shapley\n  shapley_strategy: alpha_per_layer\n  max_new_tokens: 512\n  eval_port: 8801\n```\n\n### 脚本如何使用配置\n\n| 脚本 | 读取的配置 |\n|------|-----------|\n| `run_collect.sh` | 模型路径、max_new_tokens、device |\n| `run_select.sh` | 剪枝率、剪枝方法、Shapley 策略 |\n| `run_download.sh` | 数据集列表 |\n| `vllm-server.sh` | 模型路径、eval_port |\n| `run_evalscope.py` | 评测数据集、batch_size、timeout |\n\n"}, "30ed82c73ce2923fa11b18895652afe82c06b6e6": {"t": 1785771699.6738317, "n": 4322, "h": "7810906b24179ce3", "c": "# Shapley-MoE: MoE 专家剪枝框架\n\n## 📁 项目结构\n\n```\nshapley-moe/\n│\n├── data/                               # 数据集\n│   ├── calibration/                    # 校准数据（few-shot 用）\n│   │   ├── arc_easy_25.json\n│   │   ├── gsm8k_25.json\n│   │   └── ...\n│   ├── download_dataset.py\n│   └── run_download.sh\n│\n├── analysis/                           # 分析脚本\n│   ├── collect_activations.py          # 收集激活信息（一体化）\n│   ├── calc_shapley.py                 # 计算 Shapley 值\n│   ├── run_collect.sh                  # 批量收集激活\n│   └── run_calc_shapley.sh             # 批量计算 Shapley\n│\n├── pruning/                            # 剪枝相关\n│   ├── methods/                        # 剪枝方法\n│   │   ├── select_by_shapley.py        # Shapley 值剪枝（主方法）\n│   │   ├── select_by_easyep.py         # EASYEP 剪枝\n│   │   ├── select_by_reap.py           # REAP 剪枝\n│   │   ├── select_by_gating.py         # Gating Score 剪枝\n│   │   ├── select_by_frequency.py      # 激活频率剪枝（基线）\n│   │   └── select_by_random.py         # 随机剪枝（基线）\n│   ├── save_model.py                   # 保存剪枝后模型\n│   └── run_select.sh                   # 统一的专家选择脚本\n│\n├── evaluation/                         # 评测脚本\n│   ├── run_evalscope.py\n│   └── vllm_server.sh\n│\n├── results/                            # 所有结果（按模型组织）\n│   ├── {model_name}/                   # 如 qwen3-30b-a3b/\n│   │   ├── activations/                # 激活统计\n│   │   │   ├── {dataset}_shapley.json\n│   │   │   ├── {dataset}_gating.json\n│   │   │   ├── {dataset}_easyep.json\n│   │   │   └── {dataset}_reap.json\n│   │   ├── shapley_values/             # Shapley 值\n│   │   │   └── {dataset}_shapley.csv\n│   │   └── selected_experts/           # 选中的专家\n│   │       ├── {method}_{dataset}_rate{XX}.json\n│   │       └── ...\n│   └── ...\n│\n├── models/                             # 剪枝后的模型\n│   └── {model}_{method}_rate{XX}/\n│\n└── configs/                            # 配置文件\n    ├── models.yaml                     # 模型配置\n    └── experiments.yaml                # 实验配置\n```\n\n## 🚀 快速开始\n\n### 1. 数据准备\n\n```bash\ncd data\n./run_download.sh\n```\n\n### 2. 收集激活信息\n\n一次收集所有剪枝方法需要的信息（Shapley/Gating/EASYEP/REAP）：\n\n```bash\ncd analysis\n\n# 处理单个数据集\n./run_collect.sh --model /path/to/model --data ../data/calibration/gsm8k_25.json\n\n# 批量处理所有数据集\n./run_collect.sh --model /path/to/model --all\n```\n\n### 3. 计算 Shapley 值（可选）\n\n```bash\n./run_calc_shapley.sh --model MODEL_NAME\n```\n\n### 4. 专家选择\n\n```bash\ncd ../pruning\n\n# 使用 Shapley 剪枝，保留 50%\n./run_select.sh -m qwen3-30b-a3b -d gsm8k_25 -M shapley -r 0.5\n\n# 批量处理\n./run_select.sh -m qwen3-30b-a3b --all-datasets -M easyep --all-rates\n```\n\n### 5. 保存剪枝模型\n\n```bash\npython save_model.py \\\n    --model /path/to/original/model \\\n    --selection ../results/{model}/selected_experts/{method}_{dataset}_rate50.json \\\n    --output ../models/{model}_{method}_rate50\n```\n\n### 6. 评测\n\n```bash\ncd ../evaluation\n./vllm_server.sh  # 启动模型服务\npython run_evalscope.py\n```\n\n## 📊 剪枝方法\n\n| 方法 | 描述 | 公式 |\n|------|------|------|\n| **Shapley** | 基于边际贡献的剪枝（主方法） | Shapley Value |\n| **EASYEP** | 考虑 MoE 对 token 影响程度 | `weight × (1 - simibr) × norm` |\n| **REAP** | 加权范数 | `weight × norm` |\n| **Gating** | 基于 router softmax 分数 | `mean(softmax(gating_logits))` |\n| **Frequency** | 基于激活频率（基线） | `count(activations)` |\n| **Random** | 随机选择（基线） | random |\n\n### Shapley 剪枝策略\n\nShapley 方法支持四种策略：\n\n| 策略 | 描述 | 适用场景 |\n|------|------|----------|\n| `topk_per_layer` | 每层选择 Shapley 值最高的 top-k 专家 | **推荐**，简单直接 |\n| `topk_global` | 全局选择 Shapley 值最高的专家 | 某些层可保留更多专家 |\n| `alpha_per_layer` | 每层累积 Shapley 值达到 alpha 比例 | 考虑贡献分布 |\n| `alpha_global` | 全局累积 Shapley 值达到 alpha 比例 | 全局优化 |\n\n**TopK vs Alpha:**\n- `topk`: 直接按 Shapley 值大小排序，选择前 k 个。简单、可解释性强。\n- `alpha`: 选择累积 Shapley 值达到总量 alpha 比例的最少专家。考虑了贡献分布。\n\n**Per Layer vs Global:**\n- `per_layer`: 每层独立选择，保证每层都有足够专家。\n- `global`: 全局统一选择，某些层可能专家较少。\n\n```bash\n# 使用不同策略\npython pruning/methods/select_by_shapley.py \\\n    --input results/model/shapley_values/gsm8k_25_shapley.csv \\\n    --output selected.json \\\n    --pruning_rate 0.5 \\\n    --strategy topk_per_layer  # 或 topk_global, alpha_per_layer, alpha_global\n```\n\n## 📝 命名规范\n\n- **模型**: `qwen3-30b-a3b`, `gpt-oss-20b`, `deepseekv2-lite-coder`\n- **数据集**: `{name}_{samples}` 如 `gsm8k_25`\n- **剪枝方法**: `shapley`, `easyep`, `reap`, `gating`, `frequency`, `random`\n- **剪枝率**: `rate25`, `rate50`, `rate75` (保留比例)\n\n## 🔧 配置\n\n配置文件位于 `configs/` 目录：\n\n- `models.yaml`: 模型路径和专家数量\n- `experiments.yaml`: 实验配置（数据集、剪枝率等）\n\n"}, "b2c3557fec3b9772cb3914fc936fbaad60c6f313": {"t": 1785771701.9616253, "n": 1328, "h": "72b73ff3cb1612e6", "c": "# Shapley-Moe\n\n## 1.Descripts\nShapley Value guide Moe prune\n\n## 2.Env List\nlm-evaluation-harness: https://github.com/EleutherAI/lm-evaluation-harness/tree/main\n\n```pip install \"lm_eval[vllm]\"```\n\n## 3.Steps\n### 3.0 get baseline\nusing ```run_lm-eval-vllm-dp8.sh``` and defualt model_path to get baseline\n\n### 3.1 dataset preparation\nprepare necessary dataset(e.g gsm8k), defualt 25 data\nusing ```/dataset/download.sh```\n\n\n### 3.2 few-shot \nusing dataset to find differnet layers expert activate set\nusing ```/few-shot/run_analysis.sh``` and ```3.1 dataset```\n\n\n### 3.3 calc shapley value\nusing exprt activate set to calcuate every expert's shapley value per layer\nusing ```/calc_shapley/calc_shaple.sh``` and ```3.2 results```\n\n### 3.4 expert select \nselect expert form it's shapley value\nusing ```/expert_select/run_selection_by_rate.sh``` and ```3.3 results``` to find needed expert by global expert keeprate, can choose per layer\nusing ```/expert_select/run_selection.sh``` and ```3.3 results``` to find needed expert by shapley keeprate\n\n### 3.5 model_save\nsave pruned model to safetensor, directly mask unselected experts wights to zero\nusing ```/model_save/run_sace_model.sh``` and ```3.4 results``` to save model\n\n### 3.6 test pruned model acc\nusing ```run_lm-eval-vllm-dp8.sh``` and prund model_path to get pruned score\n\n"}, "8f71ec756cda3099f7ee95a342aa24d8986f65ea": {"t": 1785771703.7214987, "n": 1327, "h": "24f25c9792a534a6", "c": "# Shapley-Moe\n\n## 1.Descripts\nShapley Value guide Moe prune\n\n## 2.Env List\nlm-evaluation-harness: https://github.com/EleutherAI/lm-evaluation-harness/tree/main\n```pip install \"lm_eval[vllm]\"```\n\n## 3.Steps\n### 3.0 get baseline\nusing ```run_lm-eval-vllm-dp8.sh``` and defualt model_path to get baseline\n\n### 3.1 dataset preparation\nprepare necessary dataset(e.g gsm8k), defualt 25 data\nusing ```/dataset/download.sh```\n\n\n### 3.2 few-shot \nusing dataset to find differnet layers expert activate set\nusing ```/few-shot/run_analysis.sh``` and ```3.1 dataset```\n\n\n### 3.3 calc shapley value\nusing exprt activate set to calcuate every expert's shapley value per layer\nusing ```/calc_shapley/calc_shaple.sh``` and ```3.2 results```\n\n### 3.4 expert select \nselect expert form it's shapley value\nusing ```/expert_select/run_selection_by_rate.sh``` and ```3.3 results``` to find needed expert by global expert keeprate, can choose per layer\nusing ```/expert_select/run_selection.sh``` and ```3.3 results``` to find needed expert by shapley keeprate\n\n### 3.5 model_save\nsave pruned model to safetensor, directly mask unselected experts wights to zero\nusing ```/model_save/run_sace_model.sh``` and ```3.4 results``` to save model\n\n### 3.6 test pruned model acc\nusing ```run_lm-eval-vllm-dp8.sh``` and prund model_path to get pruned score\n\n"}, "7ffb73fc127eb3f03f2bf9b620f7be6913cbc4ca": {"t": 1785771705.0796275, "n": 21, "h": "0c1ee8e03132ceab", "c": "# It is just a begin\n"}, "b80ddce9884ade1a778edde584f4af45756ba1d2": {"t": 1786068116.0776646, "n": 4830, "h": "5699e1ff632300a1", "c": "# SHAPE: Coalition-Aware Expert Pruning for Sparse MoE LLMs\n\n**SHAPE** (**SH**apley-**A**ware **P**runing of **E**xperts) is a training-free framework for pruning sparse Mixture-of-Experts (MoE) language models. It values experts inside the routed expert coalitions observed on a small calibration set, then preserves the most useful experts at a target keep rate.\n\n> **SHAPE: Coalition-Aware Expert Pruning for Sparse Mixture-of-Experts LLMs** (IJCNN 2026)\n>\n> [Paper](https://arxiv.org/abs/2606.09886) · [Quick start](docs/quickstart.md) · [Precomputed activations](https://github.com/Alizen-1009/Shapley-Moe/releases/tag/activation-data-v1) · [Project structure](PROJECT_STRUCTURE.md)\n\n<p align=\"center\">\n  <img src=\"assets/shape_overview.png\" alt=\"SHAPE workflow: collect routed expert coalitions, estimate contributions, and select experts\" width=\"95%\">\n</p>\n\n## Highlights\n\n- **Coalition-aware:** values experts by their marginal contribution to observed routed groups rather than in isolation.\n- **Training-free:** uses a calibration pass without gradient updates or expert retraining.\n- **Layer-aware:** preserves contribution mass per layer while matching a global keep rate.\n- **Architecture-preserving:** keeps the pretrained router and MoE structure unchanged.\n- **Broad evaluation:** supports three MoE families, eleven calibration datasets, and five pruning baselines.\n\n## Results\n\nThe reported experiments use 25 calibration examples per task. A keep rate of `0.8` retains 80% of experts; `0.6` retains 60%.\n\n| Model | Unpruned | Keep 80% | Keep 60% |\n| --- | ---: | ---: | ---: |\n| Qwen3-30B-A3B | 82.92 | 82.43 | 81.31 |\n| GPT-OSS-20B | 82.12 | 82.44 | 79.02 |\n| DeepSeek-V2-Lite | 62.08 | 62.44 | 58.81 |\n\nOn Qwen3-30B-A3B, SHAPE retains an average score of **81.31** at a 60% keep rate, compared with 79.05 for EASY-EP and 74.22 for the REAP/RAEP-style baseline. See the [paper](https://arxiv.org/abs/2606.09886) for per-task results and ablations.\n\n<p align=\"center\">\n  <img src=\"assets/shape_memory_vram.png\" alt=\"Peak VRAM at full size and at 20 and 40 percent expert pruning\" width=\"72%\">\n</p>\n\n## Quick Start\n\nThe shortest reproduction path uses the published activation snapshot and does not require running model inference:\n\n```bash\ngit clone git@github.com:Alizen-1009/Shapley-Moe.git\ncd Shapley-Moe\n\n./data/download_activations.sh\n./analysis/run_calc_shapley.sh -m qwen3-30b-a3b -d gsm8k_25\n./pruning/run_select.sh -m qwen3-30b-a3b -d gsm8k_25 -M shapley -r 0.8\n```\n\nFor environment setup, model configuration, the complete collect-to-evaluate pipeline, and adaptive LoRA recovery, see **[docs/quickstart.md](docs/quickstart.md)**.\n\n## How It Works\n\n```text\nCalibration data\n      ↓\nCollect routed expert coalitions\n      ↓\nEstimate per-layer Shapley contributions\n      ↓\nSelect experts by contribution coverage\n      ↓\nExport and evaluate the pruned model\n```\n\nThe recommended `alpha_per_layer` strategy keeps the smallest high-value expert set whose cumulative contribution reaches a per-layer threshold. Bisection adjusts that threshold until the requested global keep rate is met.\n\n## Supported Models\n\n| Config name | Experts | Experts/token | Family |\n| --- | ---: | ---: | --- |\n| `qwen3-30b-a3b` | 128 | 8 | Qwen3 MoE |\n| `gpt-oss-20b` | 32 | 4 | GPT-OSS MoE |\n| `deepseekv2-lite-coder` | 64 | 6 | DeepSeek-V2 MoE |\n\n## Pruning Methods\n\n| Method | Signal |\n| --- | --- |\n| `shapley` | Coalition-aware Shapley contribution (SHAPE) |\n| `easyep` | Task-aligned expert selection |\n| `reap` | Weighted activation norm |\n| `gating` | Mean router score |\n| `frequency` | Expert activation count |\n| `random` | Uniform random selection |\n\nDatasets, methods, keep rates, and evaluation defaults are defined in [`configs/experiments.yaml`](configs/experiments.yaml).\n\n## Repository Layout\n\n```text\nconfigs/      Model paths and experiment definitions\ndata/         Calibration data and download/distillation scripts\nanalysis/     Activation collection and Shapley scoring\npruning/      Expert selection and model export\nevaluation/   vLLM serving and EvalScope evaluation\nfinetune/     Optional adaptive-LoRA recovery\nresults/      Compact scores, selections, rank maps, and reports\n```\n\nGenerated activations, distilled data, adapters, and exported models are excluded from Git. Precomputed activations can be restored with `./data/download_activations.sh`.\n\n## Citation\n\n```bibtex\n@inproceedings{zhang2026shape,\n  title         = {SHAPE: Coalition-Aware Expert Pruning for Sparse Mixture-of-Experts LLMs},\n  author        = {Zhang, Yuhao and Jiang, HongXu and Zhang, YiXiang and Zhang, Zheng},\n  booktitle     = {Proceedings of the International Joint Conference on Neural Networks},\n  year          = {2026},\n  eprint        = {2606.09886},\n  archivePrefix = {arXiv},\n  primaryClass  = {cs.LG}\n}\n```\n"}}}}