{"repo": "axxx-institute/kernel-evo", "files": {"README.md": {"0b51d3f1f0038ede1c2ca11df5ac97513698f18c": {"t": 1785674335.8156075, "n": 9215, "h": "976ac0d70f932712", "c": "<div align=\"center\">\n  <img src=\"logo/banner.png\" alt=\"Kernel Evo banner\" />\n</div>\n\n**Evolutionary generation of efficient GPU kernels** using [GigaEvo](https://github.com/FusionBrainLab/gigaevo-core).  \nDefine a task, run evolution with an LLM backend, extract and compare optimized programs.\n\n<div align=\"center\">\n  <img src=\"logo/plotly_speedup_vs_tokens_level1_v3.gif\" alt=\"Speedup vs tokens lvl 1\" width=\"49%\" />\n  <img src=\"logo/plotly_speedup_vs_tokens_level2_v3.gif\" alt=\"Speedup vs tokens lvl 2\" width=\"49%\" />\n</div>\n\n\n## Features\n\n- **Custom tasks** — Define your own kernel tasks in KernelBench format and evolve them.\n- **KernelBench integration** — Use existing [KernelBench](https://github.com/ScalingIntelligence/KernelBench) problems.\n- **Triton and CUDA inline backends** - two most popular ways to create kernels, suitable for different scenarios.\n- **Remote or local execution** — Run validation locally or via a remote eval server.\n- **Cost efficient** - works with fast models **gemini flash 3** and **gpt-oss-120b**. Current experiments costs **0.5-1$**. \nFrontier models with high reasoning effort would be beneficial, yet cost would be magnitude higher.\n---\n\n## Requirements\n\n- **Python** >= 3.12\n- **LLM API** — OpenAI-compatible (e.g. [OpenRouter](https://openrouter.ai), or a local server like SGLang).  \n- **Redis** — Used by GigaEvo for experiment state.\n- **GPU** — Used by the evaluation stage to measure kernel correctness and efficiency.\n---\n\n## Installation\n\n### From source\n\n```bash\ngit clone https://github.com/AXXX-Institute/kernel-evo.git\ncd kernel-evo\npip install -e . --ignore-requires-python\n```\n\n> **Note:** `--ignore-requires-python` relaxes the Python version check (KernelBench may declare 3.10 but works on 3.12).  \n> For custom branches of `gigaevo` or `kernelbench`, edit the Git URLs in `pyproject.toml`.\n\n### Docker\n\nPull and run (when a pre-built image is published):\n\n```bash\ndocker pull sivtsovdt/kernel-evo:latest\ndocker run --rm sivtsovdt/kernel-evo:latest kernel-evo --help\n```\n\nTo build the image yourself (e.g. for private dependencies or development), see **[build/README.md](build/README.md)**.\n\n---\n\n## Custom kernel task\n\nTo evolve your own kernel, create a task in **KernelBench format**. Example layout:\n\n```\ntasks/\n└── armt_associate/\n    └── task.py\n```\n\nSee `tasks/armt_associate` in this repo for a reference. You can also use any existing task from [KernelBench](https://github.com/ScalingIntelligence/KernelBench).\n\n---\n\n## Run evolution\n\nEvolution can use a **local** or **remote** LLM (e.g. SGLang, OpenRouter). Examples below use OpenRouter and a remote eval server.\n\n### 1. Start the eval server (optional, for remote validation)\n\nIn a separate terminal:\n\n```bash\nkernel-evo eval-server --port 15000\n```\n\n### 2. Evolve with a custom task\n\n```bash\nOPENAI_API_KEY=\"sk-or-v1-...\" kernel-evo evolve \\\n  --problem-path tasks/armt_associate/task.py \\\n  --experiment-name custom_associate \\\n  --backend triton \\\n  --precision fp16 \\\n  --model-name <MODEL> \\\n  --llm-base-url https://openrouter.ai/api/v1 \\\n  --redis-db 0 \\\n  --max-generations 400 \\\n  --max-mutations-per-generation 4 \\\n  --validator-debug \\\n  --log-dir <dir_for_logs> \\\n  --execution-mode remote_execution\n```\n\n### 3. Evolve with a KernelBench task\n\n```bash\nOPENAI_API_KEY=\"<KEY>\" kernel-evo evolve \\\n  --level 1 \\\n  --problem-id 36 \\\n  --experiment-name kb_level1_36 \\\n  --dataset-src huggingface \\\n  --dataset-name ScalingIntelligence/KernelBench \\\n  --backend triton \\\n  --precision fp16 \\\n  --model-name <MODEL> \\\n  --llm-base-url https://openrouter.ai/api/v1 \\\n  --redis-db 0 \\\n  --max-generations 400 \\\n  --max-mutations-per-generation 4 \\\n  --validator-debug \\\n  --log-dir <dir_for_logs> \\\n  --execution-mode remote_execution\n```\n\n---\n\n## Monitor progress\n\n```bash\ncd gigaevo/outputs/<DATE>/<EXPERIMENT_START>\ntensorboard --logdir .\n```\n\nUse TensorBoard to find iterations with good performance before extracting programs.\n\n---\n\n## Extract a program\n\nExport the program from a specific iteration (e.g. after inspecting TensorBoard):\n\n```bash\nkernel-evo extract \\\n  --redis-db 0 \\\n  --iteration 8 \\\n  --redis-prefix \"kernel_evo\" \\\n  --output-file best_program.py\n```\n\n---\n\n## Compare two programs\n\n### Custom task\n\n```bash\nkernel-evo compare \\\n  --program-a prog_a.py \\\n  --program-b prog_b.py \\\n  --problem-path tasks/armt_associate/task.py \\\n  --backend triton \\\n  --precision fp16 \\\n  --num-perf-trials 200 \\\n  --num-correct-trials 20\n```\n\n### KernelBench task\n\n```bash\nkernel-evo compare \\\n  --program-a prog_a.py \\\n  --program-b prog_b.py \\\n  --dataset-src huggingface \\\n  --dataset-name ScalingIntelligence/KernelBench \\\n  --level 1 \\\n  --problem-id 36 \\\n  --backend triton \\\n  --precision fp16 \\\n  --num-perf-trials 200 \\\n  --num-correct-trials 20\n```\n\n---\n\n## Profile kernels during evolution\n\nOptional Nsight Compute and/or PyTorch-profiler runs whose output is fed back into the mutation prompt as performance insights.\n\n```bash\nkernel-evo evolve ... \\\n  --enable-profiler-stage \\\n  --profile-runners torch,ncu \\\n  --profile-ncu-min-speedup 1.0\n```\n\nKey flags:\n- `--enable-profiler-stage` — turn on the optional `ProfileMutationContextStage`.\n- `--profile-runners` — comma-separated: `torch`, `ncu`, or both.\n- `--profile-max-insights` — cap on insights surfaced per program (default 4).\n- `--profile-torch-warmup-steps` / `--profile-torch-active-steps` — torch profiler stepping.\n- `--profile-ncu-path` — `ncu` executable (resolved from PATH by default).\n- `--profile-ncu-set` — NCU section set (default `full`).\n- `--profile-ncu-kernel-name` — optional `--kernel-name` filter for NCU.\n- `--profile-ncu-extra-args` — raw extra args appended to the ncu command line.\n- `--profile-ncu-min-speedup` — only run NCU when measured speedup ≥ this threshold (default 1.0; skips wastefully profiling slow programs).\n\nProfile artifacts (json/qdrep/etc.) are written to `<log-dir>/<problem>/artifacts/`.\n\n---\n\n## Reuse memory across runs\n\nBuild a memory bank from one or more finished evolve runs, then feed it into newer runs read-only. The bank is a directory containing `api_index.json`; it can be shipped to other people.\n\n### Build / extend a bank\n\n`kernel-evo memory append` reads programs from a finished run's Redis DB, runs the ideas analyzer, and writes cards into `<memory-dir>/api_index.json`. Run it once per source experiment.\n\n```bash\n# First time — directory does not yet contain a bank, this creates one\nkernel-evo memory append \\\n  --memory-dir <bank-dir> \\\n  --redis-prefix <problem.name> \\\n  --redis-db 0\n\n# Run again with another source experiment to extend the same bank\nkernel-evo memory append \\\n  --memory-dir <bank-dir> \\\n  --redis-prefix <other_problem.name> \\\n  --redis-db 1\n```\n\n`--redis-prefix` is the `problem.name` printed by `kernel-evo evolve` (e.g. `kernelbench_2_2_20260511_171830`). Use `--analyzer-type fast` to skip the per-pair LLM analyzer and cluster via embeddings instead.\n\n### Use the bank in a new evolve run\n\n```bash\nkernel-evo evolve ... \\\n  --enable-memory \\\n  --memory-dir <bank-dir>\n```\n\nEvolve only **reads** from the bank — it never writes back, so the directory stays flat and reusable. Per-program selections appear in the log as `[MemoryContextStage] Selected N card(s) ...` and are injected into the mutation prompt. Only the local backend is supported; `--memory=api` raises `NotImplementedError`.\n\n---\n\n## CLI overview\n\n| Command         | Description                          |\n|----------------|--------------------------------------|\n| `evolve`       | Run evolution (custom or KernelBench) |\n| `eval-server`  | Start remote validation server       |\n| `extract`      | Export program by iteration from Redis |\n| `compare`      | Compare two programs (correctness + perf) |\n| `memory append`| Build / extend a memory bank from a finished evolve run |\n\n---\n\n## Best practices\n\n\n### Model selection\n\nEvolution deeply depends on underlying model. \nFor better results, one should use frontier models, like gpt, claude or gemini. \n\nRecommendation for best value vendor model:\n1. **gemini flash 3**. Capable, yet not very costly. It creates faulty kernels, but able to recover buggy code.\n\nRecommendation for open-source models:\n1. **gpt-oss-120b** - best baseline for kernel evolution. Good enough reasoning to recover faulty kernels.\n2. **GLM-5**. From all very large open LLMs, only one seems to know Triton and generate decent kernels. Downside - slower on generation and very large for local inference.\n\n### Experiments\n\nQuality of result depends on starting seeds and can vary between different runs. So it makes sense to restart and try again if the solution is very bad during the first 200k tokens.\n\nAlso, we noticed that Triton is better on small efficient kernels, like softmax and matmuls, because it requires less knowledge from the model. For complex tasks like KernelBench level 2, the difference is lower. \n\n### Remote validation\n\nBetter to run validation via validator server in different terminal. This way, one can see results.\n\n### Cheaper start\n\nUse flag `--disable-insights-lineage` with `kernel-evo evolve` to disable additional calls. Beneficial for short debug runs or with expensive models.\n"}}}}