OpenCode
68a9071cf1
Local branch had diverged from origin/master (sibling commits on the same base). Rewrote local history linearly on top of origin/master, folding in all local content: exa/hot-platform discovery hunters, leaks ledger, nightly sweep orchestrator, updated .gitignore and skill docs, plus the latest hunt output and vault state. Remote-only files (vault keys, channel scripts) were restored rather than dropped, so the resulting tree is a full union of both sides.
1 line
188 KiB
JSON
1 line
188 KiB
JSON
{"repo": "baochaixue/cs336", "files": {"papers/merged_tex/glm_5.tex": {"8756b3e225bb2e2fcc7d684814615c1ef6e140f7": {"t": 1785595989.3658698, "n": 143544, "h": "a6f972cf0e89ccd3", "c": "\\documentclass{article}\n\n\\usepackage[preprint]{neurips_2025}\n\n\\usepackage[utf8]{inputenc} \n\\usepackage[T1]{fontenc} \n\\usepackage{CJKutf8}\n\\usepackage{hyperref} \n\\usepackage{url} \n\\usepackage{xurl}\n\\usepackage{booktabs} \n\\usepackage{amsfonts} \n\n\\usepackage{svg}\n\\usepackage{nicefrac} \n\\usepackage{microtype} \n\\usepackage[table,xcdraw]{xcolor} \n\n\\usepackage{subcaption}\n\\usepackage{wrapfig}\n\\usepackage{multirow}\n\\usepackage{makecell}\n\\usepackage{xspace}\n\\usepackage{enumitem}\n\\usepackage{amsmath}\n\\usepackage{listings}\n\\usepackage{algorithm}\n\\usepackage{algorithmic}\n\\usepackage{amsfonts,amssymb}\n\\usepackage{amsthm}\n\\usepackage{mathrsfs}\n\\usepackage{subcaption}\n\\usepackage{natbib}\n\\setcitestyle{numbers,square}\n\\usepackage{graphicx} \n\\usepackage{cleveref}\n\\usepackage{fontawesome}\n\\usepackage{utfsym}\n\\usepackage[most]{tcolorbox}\n\\usepackage{siunitx} \n\\usepackage{tabularx}\n\\usepackage{CJKutf8}\n\\usepackage{fancyvrb}\n\\usepackage{colortbl}\n\\usepackage{array}\n\\usepackage{threeparttable}\n\\usepackage{CJKutf8}\n\\usepackage{CJK}\n\\usepackage{booktabs}\n\\usepackage[table]{xcolor}\n\\usepackage{lipsum}\n\\usepackage{listings}\n\\lstset{\n breaklines,\n columns=flexible\n}\n\\definecolor{demphcolor}{RGB}{144,144,144}\n\\newcommand{\\demph}[1]{\\textcolor{demphcolor}{#1}}\n\n\\newcommand{\\hhide}[1]{}\n\\newcommand{\\hide}[1]{}\n\\newcommand{\\vpara}[1]{\\vspace{0.07in}\\noindent\\textbf{#1}\\kxspace} \n\\newcommand{\\todo}[1]{\\textcolor{red}{\\textbf{TODO:} #1}}\n\n\\newcommand{\\model}{GLM-5}\n\n\\newcommand{\\tablehead}[1]{\\textbf{#1}}\n\\newcommand{\\aspace}{\\hspace{1em}}\n\\definecolor{zhipublue}{HTML}{3859FF}\n\n\\newtcolorbox{promptbox}[1][]{\n breakable,\n title=#1,\n colback=gray!5,\n colframe=black,\n colbacktitle=gray!15,\n coltitle=black,\n fonttitle=\\bfseries,\n bottomrule=1.5pt,\n toprule=1.5pt,\n leftrule=1pt,\n rightrule=1pt,\n arc=0pt,\n outer arc=0pt,\n enhanced,\n before upper={\\parindent=1.5em} \n}\n\n\\lstset{\n numbers=none, \n keywordstyle= \\color{ blue!70},\n commentstyle= \\color{red!50!green!50!blue!50}, \n frame=none, \n rulesepcolor= \\color{ red!20!green!20!blue!20} ,\n framexleftmargin=2em,\n columns=fullflexible,\n breaklines=true,\n breakindent=0pt,\n basicstyle=\\ttfamily,\n escapeinside={(*@}{@*)}\n}\n\n\\theoremstyle{definition}\n\\newtheorem{example}{Example}[section]\n\n\\title{GLM-5: from Vibe Coding to Agentic Engineering}\n\n \\author{\n{GLM-5 Team}\n~\\\\\\\\\nZhipu AI ~\\&~ Tsinghua University\\\\\\\\\n(For the complete list of authors, please refer to the \\hyperref[sec:contribution]{Contribution} section)\n{}\n}\n\n\\begin{document}\n\n\\maketitle\n\n\\vspace{-1.5em}\n\\begin{abstract}\n\n\\vspace{-0.5em}\n\nWe present GLM-5, a next-generation foundation model designed to transition the paradigm of vibe coding to agentic engineering.\n Building upon the agentic, reasoning, and coding (ARC) capabilities of its predecessor, GLM-5 adopts DSA to significantly reduce training and inference costs while maintaining long-context fidelity. To advance model alignment and autonomy, we implement a new asynchronous reinforcement learning infrastructure that drastically improves post-training efficiency by decoupling generation from training. Furthermore, we propose novel asynchronous agent RL algorithms that further improve RL quality, enabling the model to learn from complex, long-horizon interactions more effectively. Through these innovations, GLM-5 achieves state-of-the-art performance on major open benchmarks. Most critically, GLM-5 demonstrates unprecedented capability in real-world coding tasks, surpassing previous baselines in handling end-to-end software engineering challenges. \nCode, models, and more information are available at \\url{https://github.com/zai-org/GLM-5}.\n\n\\end{abstract}\n\n\\begin{figure}[!h]\n \\centering\n \\includegraphics[width=\\linewidth]{figures/arc_clean.pdf}\n \\caption{Results of GLM-5, DeepSeek-V3.2, Claude Opus 4.5, Gemini 3 Pro, and GPT-5.2 (xhigh) on 8 agentic, reasoning, and coding benchmarks: Humanity's Last Exam, SWE-bench Verified, SWE-bench Multilingual, Terminal-Bench 2.0, BrowseComp, MCP-Atlas, $\\tau^2$-Bench, Vending Bench~2.}\n \\label{fig:arc}\n\\end{figure}\n\n\\section{Introduction}\n\nThe pursuit of Artificial General Intelligence (AGI) requires not only scaling model parameters but also fundamentally rethinking the efficiency of intelligence and the architecture of autonomous improvement. With the release of GLM-4.5, we demonstrated that uniting Agentic, Reasoning, and Coding (ARC) capabilities into a single Model-of-Experts (MoE) architecture could yield state-of-the-art results across diverse benchmarks. However, as Large Language Models (LLMs) transition from passive knowledge repositories to active problem solvers, the dual challenges of computational cost and real-world adaptability—particularly in complex software engineering—have become the primary bottlenecks.\n\nWe present GLM-5, our next-generation flagship model designed to overcome these barriers. GLM-5 represents a paradigm shift in both performance and efficiency, achieving state-of-the-art status on major open leaderboards, including ArtificialAnalysis.ai, the LMArena Text, and the LMArena Code. More significantly, GLM-5 redefines the standard for real-world coding, demonstrating an unprecedented ability to handle complex, end-to-end software development tasks that go far beyond the scope of traditional static benchmarks like SWE-bench.\n\n\\paragraph{Results.} Figure~\\ref{fig:arc} shows the results of GLM-5, GLM-4.7, Claude Opus 4.5, Gemini 3 Pro, and GPT-5.2 (xhigh) on 8 agentic, reasoning, and coding benchmarks: Humanity's Last Exam~\\cite{phan2025humanity}, SWE-bench Verified~\\cite{jimenez2023swe}, SWE-bench Multilingual~\\cite{yang2025swemulti}, Terminal-Bench 2.0~\\cite{tbench_2025}, BrowseComp~\\cite{wei2025browsecomp}, MCP-Atlas~\\cite{bandi2026mcp}, $\\tau^2$-Bench~\\cite{yao2024tau,barres2025tau2}, Vending Bench 2~\\cite{backlund2025vending}. On average, GLM-5 achieves about 20\\% improvement over our last version GLM-4.7, and is comparable to Claude Opus 4.5 and GPT-5.2 (xhigh), and better than Gemini 3 Pro.\n\nGLM-5 scores 50 on the Intelligence Index v4.0 and is the new open weights leader (Cf. Figure~\\ref{fig:aa}), up from GLM-4.7's score of 42 - an 8 point jump driven by improvements across agentic performance and knowledge/hallucination. This is the first time an open weights model has achieved a score of 50 on the Artificial Analysis Intelligence Index v4.0.\n\n\\begin{figure}[t]\n \\centering\n \\includegraphics[width=1\\linewidth]{figures/aa.PNG}\n \\caption{Artificial Analysis Intelligence Index v4.0 incorporates 10 evaluations: GDPval-AA, $\\tau^2$-Bench Telecom, Terminal-Bench Hard, SciCode, AA-LCR, AA-Omniscience, IFBench, Humanity's Last Exam, GPQA Diamond, CritPt.}\n \\label{fig:aa}\n\\end{figure}\n\n\\begin{figure}[t]\n \\centering\n \\includegraphics[width=0.48\\linewidth]{figures/text-arena.jpeg} \\;\\;\\;\n \\includegraphics[width=0.48\\linewidth]{figures/code-arena.jpeg}\n \\caption{On LMArena, GLM-5 is the \\#1 open model in both Text Arena and Code Arena.}\n \\label{fig:arena}\n\\end{figure}\n\n\\begin{figure}[t]\n \\centering\n \\includegraphics[width=0.50\\linewidth]{figures/vending-bench.jpeg} \\;\\;\n \\includegraphics[width=0.47\\linewidth]{figures/cc-bench-v2.pdf}\n \\caption{Results on several long-horizon tasks. Left: Vending-Bench 2; Right: CC-Bench-V2.}\n \\label{fig:vendingcc}\n\\end{figure}\n\nLMArena, initiated by UC Berkeley, is a transparent, shared space to evaluate and compare frontier AI capabilities by human judgment with millions of real tasks, including writing, coding, reasoning, designing, searching, and creating. The large volume of human interactions generates signals of real-world utility, making it different from the other static benchmarks.\nFigure~\\ref{fig:arena} shows that GLM-5 again is the \\#1 open model in both Text Arena and Code Arena, and overall \non par with Claude-Opus-4.5 and Gemini-3-pro.\n\nLong-term coherence in agents becomes more and more important. Coding agents can now write code autonomously for hours, and the length and breadth of tasks AI models are able to complete are likely to increase. We use two benchmarks, Vending-Bench 2 and CC-Bench-V2, to evaluate how GLM-5 is able to complete long-horizon tasks.\nVending-Bench 2 is a benchmark for measuring AI model performance in running a business over long time horizons. Models are tasked with running a simulated vending machine business over a year and are scored on their bank account balance at the end.\nFigure~\\ref{fig:vendingcc} (left) shows that GLM-5 ranks \\#1 among all open-source models, finishing with a final account balance of \\$4,432. It approaches Claude Opus 4.5, demonstrating strong long-term planning and resource management.\nFigure~\\ref{fig:vendingcc} (right) further shows results on our internal evaluation suite CC-Bench-V2. GLM-5 significantly outperforms GLM-4.7 across frontend, backend, and long-horizon tasks, narrowing the gap with Claude Opus 4.5.\n\n\\begin{figure}[t]\n \\centering\n \\includegraphics[width=1\\linewidth]{figures/overall_pipeline.pdf}\n \\caption{Overall training pipeline of GLM-5.}\n \\label{fig:overall_pipeline}\n\\end{figure}\n\n\\paragraph{Methods.} Figure~\\ref{fig:overall_pipeline} shows the overall training pipeline of GLM-5.\nOur Base Model training began with a massive 27 trillion token corpus, prioritizing code and reasoning early on. We then employed a distinct Mid-training phase to progressively extend context length from 4K to 200K, focusing specifically on long-context agentic data to ensure stability in complex workflows.\nIn Post-Training, we moved beyond standard SFT. We implemented a sequential Reinforcement Learning pipeline—starting with Reasoning RL, followed by Agentic RL, and finishing with General RL. Crucially, we utilized On-Policy Cross-Stage Distillation throughout this process to prevent catastrophic forgetting, ensuring the model retains its sharp reasoning edge while becoming a robust generalist.\nIn summary, the leap in GLM-5’s performance is driven by the following technical contributions:\n\nFirst, we adopt DSA (DeepSeek Sparse Attention)~\\cite{deepseekai2025deepseekv32pushingfrontieropen}, a novel architectural innovation that significantly reduces both training and inference costs. While GLM-4.5 improved efficiency through a standard MoE architecture, DSA allows GLM-5 to dynamically allocate attention resources based on token importance, drastically lowering the computational overhead without compromising long-context understanding or reasoning depth. With DSA, we scale the model parameters up to 744B and extend the training token budget to ~28.5T tokens.\n\nSecond, we have engineered a new asynchronous reinforcement learning infrastructure. Building on the ``slime'' framework and the decoupled rollout engines initialized in GLM-4.5, our new infrastructure further decouples generation from training to maximize GPU utilization. This system allows for massive-scale exploration of agent trajectories without the synchronization bottlenecks that previously hampered iteration speed, significantly improving the efficiency of our RL post-training pipeline.\n\nThird, we present novel asynchronous Agent RL algorithms designed to enhance the quality of autonomous decision-making. In GLM-4.5, we utilized iterative self-distillation and outcome supervision to train agents. For GLM-5, we have developed asynchronous algorithms that allow the model to learn from diverse, long-horizon interactions continuously. These algorithms are specifically optimized to improve the model's planning and self-correction capabilities in dynamic environments, directly contributing to our dominance in real-world coding scenarios.\n\nLast, one more technical contribution lies in the fact that, from the first day, GLM-5 is full-stack adapted to Chinese GPU ecosystems. We have successfully completed deep optimization—spanning from underlying kernels to upper-level inference frameworks—across seven mainstream domestic chip platforms, including Huawei Ascend, Moore Threads, Hygon, Cambricon, Kunlunxin, MetaX, and Enflame.\n\nWith these advancements, GLM-5 stands not just as a more powerful model but as a more efficient and practical foundation for the next generation of AI agents. We release GLM-5 to the community to further advance the frontier of efficient, agentic general intelligence.\n\n\\section{Pre-Training}\nSimilar to GLM-4.5, the base model of GLM-5 goes through two stages: pre-training for general language and coding capacity, and mid-training for agentic and long-context capacity. We extend the training token budget for all the training stages of GLM-5, totaling 28.5 trillion tokens for the base model.\n\n\\subsection{Architecture}\n\n\\paragraph{Model size scaling.} GLM-5 scales to 256 experts and reduces its layer count to 80 to minimize expert parallelism communication overhead. This results in a 744B parameter model (40B active parameters), doubling the total size of GLM-4.5, which utilized 355B total and 32B active parameters.\n\n\\begin{table}[!ht]\n \\small\n \\centering\n \\caption{Evaluation results for GQA-8 and variants of MLA.}\n \\begin{tabular}{c|cccccccc}\n \\toprule\n Dataset & Hellaswag & MMLU & C-Eval & RACE & BBH & GSM8K & HumanEval \\\\\n \\midrule\n GQA-8 & 77.3 & 61.2 & 60.0 & 79.6 & \\textbf{53.3} & \\textbf{47.6} & \\textbf{38.5} \\\\\n MLA & 77.3 & 61.5 & 59.7 & 77.8 & 48.9 & 46.2 & 33.5 \\\\\n MLA + Muon Split & \\textbf{77.8} & \\textbf{62.5} & \\textbf{62.1} & \\textbf{79.9} & 51.8 & 45.0 & 36.7\\\\\n MLA-256 + Muon Split & 77.4 & 62.0 & 59.9 & 79.6 & 51.3 & 47.5 & 36.6 \\\\\n \\bottomrule\n \\end{tabular}\n \\label{tab:mla}\n\\end{table}\n\n\\paragraph{Multi-latent Attention.} By employing reduced key-value vectors, Multi-latent attention (MLA)~\\cite{liu2024deepseekv2} matches the effectiveness of Grouped-Query Attention (GQA) but offers superior GPU memory savings and faster processing for long-context sequences.\n\nHowever, in our experiments with Muon optimizer, we find that MLA with a 576-dimension latent KV-cache cannot match the performance of GQA with 8 query groups (denoted as GQA-8, 2048-dimension KV-cache). To overcome the performance gap, we propose an adaptation to the recipe of Muon optimizer in GLM-4.5.\nIn the original recipe, we apply matrix orthogonalization to the up-projection matrices $W^{UQ}, W^{UK}, W^{UV}$ for multi-head queries, keys, and values. Instead, we split these matrices into smaller matrices for different heads and apply matrix orthogonalization to these independent matrices. The method, denoted as Muon Split, enables projection weights for different attention heads to update at different scales. As shown in \\Cref{tab:mla}, the method effectively improves the performance of MLA to match that of GQA-8. In practice, we also find that with Muon Split, the scale of attention logits of GLM-5 remains stable during pre-training without any clipping strategy.\n\nAnother disadvantage of MLA is its high computational cost during decoding. In decoding, MLA performs a 576-dimensional dot product, higher than the 128-dimensional computation of GQA. While the number of attention heads in DeepSeek-V3 is selected according to the roofline of H800~\\cite{zhao2025insights}, it is inappropriate for other hardware. Given the Multi-head Attention (MHA) style of MLA during training and prefilling, we increase the head dimension from 192 to 256 and decrease the number of attention heads by 1/3. This keeps the training computation\nand the number of parameters constant while decreasing the decoding computation. The variant, denoted as MLA-256 in \\Cref{tab:mla}, matches the performance of MLA under Muon Split.\n\n\\begin{wraptable}{r}{0.4\\linewidth}\n \\centering\n \\caption{Comparison of accept lengths of DeepSeek-V3.2 and GLM-5.}\n \\label{tab:mtp}\n \\begin{tabular}{c|c}\n \\toprule\n Model & Accept Length \\\\\n \\midrule\n DeepSeek-V3.2 & 2.55 \\\\\n GLM-5 & \\textbf{2.76}\\\\\n \\bottomrule\n \\end{tabular}\n\\end{wraptable}\n\n\\paragraph{Multi-token Prediction with Parameter Sharing.} Multi-token prediction (MTP)~\\cite{gloeckle2024mtp,liu2024deepseek} increases the performance of base models and acts as draft models for speculative decoding~\\cite{leviathan2023spec}. However, during training, to predict the next $n$ tokens, $n$ MTP layers are required. As a result, the memory usage of MTP parameters and the kv cache scales linearly with the number of speculative steps. Instead, DeepSeek-V3 is trained with a single MTP layer and predicts the next 2 tokens during inference. The training-inference discrepancy reduces the acceptance rate of the second token. Therefore, we propose sharing the parameters of 3 MTP layers during training. This keeps the memory cost of the draft model consistent with DeepSeek-V3 while increasing the acceptance rate. In \\Cref{tab:mtp}, we show that the acceptance length of GLM-5 is longer than DeepSeek-V3.2, given the same number of speculative steps (4) in our private prompt set.\n\n\\subsubsection{Continued Pre-Training with DeepSeek Sparse Attention (DSA)}\n\\begin{table}[!ht]\n \\centering\n \\caption{Comparison of long-context benchmarks between MLA and DSA base models.}\n \\begin{tabular}{c|cccc}\n \\toprule\n & MQ-NIAH-128k & MV-NIAH-128k & SQuAD-128k & HotpotQA-128k \\\\\n \\midrule\n MLA & \\textbf{100.0} & 95.5 & 79.7 & \\textbf{66.3} \\\\\n DSA & \\textbf{100.0} & \\textbf{97.0} & \\textbf{86.0} & 63.0\\\\\n \\bottomrule\n \\end{tabular}\n \\label{tab:dsa}\n\\end{table}\nWe use DSA in our training. \nThe core philosophy of DSA~\\cite{deepseekai2025deepseekv32pushingfrontieropen} is to replace the traditional dense $O(L^2)$ attention—which becomes prohibitively expensive at $128\\text{K}$ contexts—with a dynamic, fine-grained selection mechanism. Unlike fixed patterns (like sliding windows), DSA ``looks'' at the content to decide which tokens are important.\nWhat makes DSA particularly interesting from a researcher's perspective is how it was introduced via Continued Pre-Training from a dense base model. This avoided the ``astronomical'' cost of training from scratch. The transition follows a two-stage ``dense warm-up and sparse training adaptation'' strategy. DeepSeek-V3.2-Exp maintains the same benchmark performance as its dense predecessor, proving that ~90\\% of attention entries in long contexts are indeed redundant.\nDSA reduces the attention computation by roughly 1.5-2× for long sequences, which is very important for the reasoning-heavy agents we are building, being able to handle 128K contexts at half the GPU cost.\n\n\\begin{figure}\n \\centering\n \\includegraphics[width=0.6\\linewidth]{figures/loss_v2.pdf}\n \\caption{SFT loss curves comparison between MLA and DSA training. Results are smoothed by Running Average with a window size of 50.}\n \\label{fig:placeholder}\n\\end{figure}\n\nThe DSA training begins from the base model at the end of mid-training. The warm-up stage goes through 1000 steps with each step trained on 14 sequences of 202,752 tokens and a maximum learning rate of 5e-3. The sparse adaptation stage follows the training data and hyperparameters of mid-training and goes through 20B tokens. Although the training budget is much smaller than that of DeepSeek-V3.2 (943.7B tokens), we find that it is enough to adapt the DSA model to match the performance of the original MLA model. As shown in \\Cref{tab:dsa}, the long-context performance of the DSA model is close to that of the MLA model. To further validate the effectiveness of DSA training, we fine-tune the DSA and MLA models with the same SFT data, respectively, and find that the two models tie in training loss and evaluation benchmarks.\n\n\\subsubsection{Ablation Study of Efficient Attention Variants}\n\nBeyond DSA~\\cite{liu2025deepseek}, we explore several alternative efficient attention mechanisms based on GLM-9B\\footnote{One of our GLM-4 series models, available at https://github.com/zai-org/GLM-4}. The baseline employs group query attention across all 40 layers and has been fine-tuned with a 128K-token context window. We evaluate the following approaches:\n\n\\begin{itemize}[leftmargin=1.5em,itemsep=2pt]\n \\item {{Sliding Window Attention (SWA) Interleave:}} A fixed alternating pattern of\n full-attention and windowed-attention layers applied uniformly across the network.\n \\item {{Gated DeltaNet (GDN)}~\\cite{yanggated}}: A linear attention variant that\n replaces the quadratic softmax attention computation with a gated linear recurrence,\n reducing the computational cost of attention from quadratic to linear in sequence length.\n\\end{itemize}\n\n\\noindent Building on these baselines, we propose two improvements:\n\n\\begin{itemize}[leftmargin=1.5em,itemsep=2pt]\n \\item {SWA Pattern (Search-Based):} Inspired by PostNAS~\\cite{gu2025jet}, we introduce a search-based adaptation method that identifies the optimal subset of layers for SWA conversion while retaining full attention in the remaining layers. We employ a beam search strategy to determine the configuration that maximizes performance on long-context downstream tasks. To mitigate computational costs, we conduct the search exclusively at a 16K context length and generalize the resulting pattern to all other input lengths. Specifically, we use a beam size of~8, optimizing two layers per step; for GLM-9B (40 layers), the process converges in approximately 10 steps. At each step, candidate patterns are evaluated on the RULER benchmark~\\cite{hsiehruler} at 16K context length, and the top-8 candidates are retained for the subsequent step. The final derived pattern is \\texttt{SFSSFFSSSFFFFSSFSFFFFFFSFSFSSFSSFSFSSFSSS}, where \\texttt{S} and \\texttt{F} denote SWA and full-attention layers, respectively. As shown in Table~\\ref{tab:ruler_results}, this search-based configuration significantly outperforms the fixed interleaved approach. Notably, despite being optimized only at 16K, the pattern exhibits robust length generalization, maintaining effective across all tested context lengths.\n\n \\item {SimpleGDN:} A minimalist linearization strategy designed for maximal reuse of pre-trained weights, improving upon GDN for continual-training adaptation. We remove the \\texttt{Conv1d} and explicit gating modules entirely and instead directly map the pre-trained Query, Key, and Value projection weights into the linear recurrence formulation. This simplification eliminates the need for additional parameters while preserving the efficiency benefits of linear attention.\n\\end{itemize}\n\n\\begin{table}[h]\n \\small\n \\centering\n \\caption{RULER benchmark results for the GLM-9B baseline and two SWA variants \\emph{without any additional training}. Both SWA methods use a 1:1 ratio of full-attention to SWA layers with a 4096-token window size. The search-based SWA pattern is discovered once at 16k context length and applied uniformly across all input lengths.}\n \\label{tab:ruler_results}\n \\begin{tabular}{lcccccc}\n \\toprule\n & {4K} & {8K} & {16K} & {32K} & {64K} & {128K} \\\\\n \\midrule\n GLM-9B (Full Attn) & 95.19 & \\textbf{93.67} & \\textbf{92.01} & \\textbf{91.09} & \\textbf{85.35} & \\textbf{75.28} \\\\\n SWA Interleave & 94.87 & 54.02 & 25.89 & 12.61 & 8.32 & 6.51 \\\\\n SWA Pattern & \\textbf{95.78} & 92.54 & 88.92 & 82.52 & 70.23 & 53.95 \\\\\n \\bottomrule\n \\end{tabular}\n\\end{table}\n\nWe evaluate all methods on four long-context benchmarks: RULER~\\cite{hsiehruler}, MRCR\\footnote{\\url{https://huggingface.co/datasets/openai/mrcr}}, HELMET-ICL~\\cite{yen2024helmet}, and RepoQA~\\cite{liu2024repoqa}. Results are summarized in Table~\\ref{tab:gqa_results_appendix}. We continually train each method on 190B tokens with a 64K context length, maintaining a 1:1 ratio between efficient attention layers and full attention layers. For the GDN and SimpleGDN methods, we follow the Jet-Nemotron~\\cite{gu2025jet} pipeline.\n\n\\begin{table*}[h]\n\\centering\n\\caption{Long-context benchmark results. All efficient attention variants are continual-trained from the GLM-9B full-attention baseline. SWA pattern denotes search-based layer selection; SWA interleave denotes the fixed alternating pattern. $\\Delta$@64K and $\\Delta$@128K show the difference relative to the full-attention baseline at 64K and 128K context lengths, respectively.}\n\\label{tab:gqa_results_appendix}\n\\resizebox{\\textwidth}{!}{\n\\begin{tabular}{lcccc}\n\\toprule\n & \\textbf{RULER (64K\\,/\\,128K)}\n & \\textbf{MRCR (64K\\,/\\,128K)}\n & \\textbf{HELMET-ICL (64K\\,/\\,128K)}\n & \\textbf{RepoQA (64K\\,/\\,128K)} \\\\\n\\midrule\nGLM-9B \n & 85.35\\,/\\,75.28\n & 36.53\\,/\\,35.39\n & 77.68\\,/\\,77.36\n & 69.00\\,/\\,65.83 \\\\\n\\midrule\nSWA Interleave\n & 65.94\\,/\\,44.93 {\\scriptsize($\\downarrow$19.41\\,/\\,$\\downarrow$30.35)}\n & 30.03\\,/\\,28.83 {\\scriptsize($\\downarrow$6.50\\,/\\,$\\downarrow$6.56)}\n & 75.96\\,/\\,63.52 {\\scriptsize($\\downarrow$1.72\\,/\\,$\\downarrow$13.84)}\n & 50.33\\,/\\,39.33 {\\scriptsize($\\downarrow$18.67\\,/\\,$\\downarrow$26.50)} \\\\\nSWA Pattern\n & \\textbf{83.72\\,/\\,69.59} {\\scriptsize($\\downarrow$1.63\\,/\\,$\\downarrow$5.69)}\n & \\textbf{35.02\\,/\\,33.58} {\\scriptsize($\\downarrow$1.51\\,/\\,$\\downarrow$1.81)}\n & 76.48\\,/\\,74.60 {\\scriptsize($\\downarrow$1.20\\,/\\,$\\downarrow$2.76)}\n & 62.33\\,/\\,51.17 {\\scriptsize($\\downarrow$6.67\\,/\\,$\\downarrow$14.66)} \\\\\nGDN\n & 76.76\\,/\\,64.00 {\\scriptsize($\\downarrow$8.59\\,/\\,$\\downarrow$11.28)}\n & 31.72\\,/\\,30.22 {\\scriptsize($\\downarrow$4.81\\,/\\,$\\downarrow$5.17)}\n & 76.88\\,/\\,74.84 {\\scriptsize($\\downarrow$0.80\\,/\\,$\\downarrow$2.52)}\n & 65.50\\,/\\,56.17 {\\scriptsize($\\downarrow$3.50\\,/\\,$\\downarrow$9.66)} \\\\\nSimpleGDN\n & 81.76\\,/\\,67.03 {\\scriptsize($\\downarrow$3.59\\,/\\,$\\downarrow$8.25)}\n & 33.03\\,/\\,31.27 {\\scriptsize($\\downarrow$3.50\\,/\\,$\\downarrow$4.12)}\n & \\textbf{79.80\\,/\\,81.84} {\\scriptsize($\\uparrow$2.12\\,/\\,$\\uparrow$4.48)}\n & \\textbf{65.50\\,/\\,58.50} {\\scriptsize($\\downarrow$3.50\\,/\\,$\\downarrow$7.33)} \\\\\n\\bottomrule\n\\end{tabular}\n}\n\\end{table*}\n\nThe results in Table~\\ref{tab:gqa_results_appendix} reveal a clear trade-off hierarchy among efficient attention methods. Naively interleaved sliding window attention (SWA) causes catastrophic degradation on long-context tasks (e.g., $-$30.35 on RULER@128K), while search-based layer selection substantially narrows this gap by preserving full attention where it matters most. Linear attention variants such as GDN further improve quality but at the cost of additional parameters; SimpleGDN strikes the best balance by maximally reusing pre-trained weights. Nevertheless, all of these methods incur an inherent accuracy gap on fine-grained retrieval tasks—up to 5.69 points on RULER@128K and 7.33 on RepoQA@128K—due to the unavoidable information loss introduced by efficient attention mechanisms during continual-training adaptation, even when half of the layers retain full attention. In contrast, DSA is lossless by construction: its lightning indexer achieves token-level sparsity without discarding any long-range dependencies, enabling application to all layers with no quality degradation.\n\nTo verify this, we conduct a small-scale DSA experiment on GLM-4.7-Flash\\footnote{\\url{https://huggingface.co/zai-org/GLM-4.7-Flash}} with multi-latent attention. Following the standard DSA recipe, training proceeds in two stages: (i)~a {warmup} phase that trains only the indexer for 1{,}000 steps (batch size 16) while keeping all base-model weights frozen, followed by (ii)~a {joint-training} phase in which both the model and the indexer are co-trained on 150B tokens.\nTable~\\ref{tab:dsa_ruler_results} summarizes the results on RULER across context lengths from 4K to 128K. Even the warmup-only variant ({GLM-4.7-Flash + DSA warmup}) already preserves the vast majority of baseline performance; the drop is modest and concentrated at the longest context window (128K: $79.21 \\to 71.35$), while shorter contexts remain virtually unaffected. After the full 150B-token joint-training phase, {GLM-4.7-Flash + DSA} closes nearly all of this residual gap: it {surpasses} the baseline at 16K ($+0.86$), 32K ($+0.49$), and 64K ($+1.72$), while incurring only a 0.35-point deficit at 128K.\n\n\\begin{table}[t]\n \\small\n \\centering\n \\caption{RULER benchmark results for the GLM-4.7-Flash with DSA.\n The warmup-only variant trains only the indexer while\n keeping the base model frozen, the full DSA variant jointly trains both for\n 150B tokens. }\n \\label{tab:dsa_ruler_results}\n \\begin{tabular}{lcccccc}\n \\toprule\n & {4K} & {8K} & {16K} & {32K} & {64K} & {128K} \\\\\n \\midrule\n GLM-4.7-Flash & 97.44 & 96.72 & 95.83 & 92.96 & 85.34 & 79.21 \\\\\n GLM-4.7-Flash + DSA warmup & 97.51 & 96.54 & 95.40 & 90.09 & 84.05 & 71.35 \\\\\n GLM-4.7-Flash + DSA & 96.77 & 96.25 & 96.69 & 93.45 & 87.06 & 78.86 \\\\\n \\bottomrule\n \\end{tabular}\n\\end{table}\n\n\\subsection{Pre-training Data}\n\n\\paragraph{Web.} Building upon the GLM-4.5 data pipeline, we refined our selection criteria for massive web datasets. We introduced another DCLM~\\cite{li2025datacomplmsearchgenerationtraining} classifier based on sentence embeddings to identify and aggregate additional high-quality data beyond standard classifiers. To address the challenge of long-tail knowledge, we utilized a World Knowledge classifier—optimized via Wikipedia entries and LLM-labeled data—to distill valuable information from otherwise medium-low-quality data.\n\n\\paragraph{Code.} We expand the code pre-training corpus with refreshed snapshots from major code hosting platforms and a larger collection of code-containing web pages, resulting in a 28\\% increase in fuzzily deduplicated unique tokens. To improve corpus integrity and reduce noise, we fix metadata alignment issues in Software Heritage code files and adopt a more accurate language classification pipeline. We follow GLM-4.5’s quality-aware sampling strategy for source code and code-related web documents. In addition, we train dedicated classifiers for a broader set of low-resource programming languages (e.g., Scala, Swift, Lua, etc.), improving sampling quality for these languages.\n\n\\paragraph{Math \\& Science.} We collect high-quality math \\& science data from webpages, books, and papers to further increase the reasoning abilities. Specifically, the content extraction pipelines for webpages and PDF parsing mechanisms\nfor books and papers are refined to increase data quality. We adopt large language models to score candidate documents and only retain the most educational content. For long-context documents, we develop a chunk-and-aggregate scoring algorithm to increase scoring accuracy. Filtering pipelines are conducted to strictly avoid the use of synthetic, AI-generated, or template-based data. \n\n\\subsection{Mid-Training}\n\nBuilding upon the mid-training framework introduced in GLM-4.5, we scale up both the training volume and the maximum context length in GLM-5 to further strengthen the model's reasoning, long-context, and agentic capabilities.\n\n\\paragraph{Extended context and training scale.} We progressively extend the context window across three stages: 32K (1T tokens), 128K (500B tokens), and 200K (50B tokens). Compared to the 128K maximum in GLM-4.5, the additional 200K stage substantially improves the model's ability to process ultra-long documents and complex multi-file codebases. Long documents and synthetic agent trajectories are up-sampled at the later stages accordingly.\n\n\\paragraph{Software engineering data.} We retain the paradigm of concatenating repo-level code files, commit diffs, GitHub issues, pull requests, and relevant source files into unified training sequences. In GLM-5, we relax the repository-level filtering criteria to broaden the pool of eligible repositories, yielding approximately 10 million issue–PR pairs, while strengthening quality filtering at the individual issue level to reduce noise. We also retrieve a larger set of relevant files for each issue–PR pair, resulting in richer development contexts and broader coverage of real-world software engineering scenarios. After filtering, the issue–PR portion of the dataset comprises approximately 160B unique tokens.\n\n\\paragraph{Long-context data.} Our long-context training set comprises both natural and synthetic data. Natural data is curated from books, academic papers, and documents from general pre-training corpora employing multi-stage filtering (PPL, deduplication, length) and upsampling knowledge-intensive domains. In synthetic data construction, inspired by NextLong\\cite{gao2025nextlongeffectivelongcontexttraining} and EntropyLong\\cite{jia2025entropylongeffectivelongcontexttraining}, we employed diverse techniques to build long-range dependencies. Highly similar texts were aggregated via interleaved packing to produce sequences, aiming to mitigate the lost-in-the-middle phenomenon and improve performance across a range of long-context tasks. At the 200K stage, we additionally incorporated a small proportion of MRCR-like data, with multiple variants designed to extend OpenAI's original paradigm, to strengthen recall in extended multi-turn dialogues. Empirically, we find that increasing data diversity progressively enhances the model's long-context performance; notably, a subsequent 200K mid-training stage, building upon the initial 128K phase, further bolstered the model’s performance even within the 128K context window.\n\n\\subsection{Training Infrastructure}\n\n\\subsubsection{Memory Efficiency}\n\n\\textbf{Flexible MTP placement.} Under interleaved pipeline parallelism~\\cite{narayanan2021efficientlargescalelanguagemodel}, model components are flexibly assigned to stages. The MTP module spans embedding, transformer, and output components. It incurs substantially higher memory usage than other modules, leading to stage-level imbalance. We co-locate the MTP output layer with the main output layer on the final stage to enable parameter sharing, while placing its embedding and transformer components on the preceding stage. This reduces memory pressure on the final stage and improves balance across pipeline ranks.\n\n\\textbf{Pipeline ZeRO2 gradient sharding.} Each pipeline rank maintains multiple stages~\\cite{narayanan2021efficientlargescalelanguagemodel}, and naively each stage requires a full gradient buffer for accumulation and optimizer updates. Inspired by ZeRO2~\\cite{rajbhandari2020zeromemoryoptimizationstraining}, we shard gradients across data-parallel ranks so that each stage stores only a 1/dp fraction of the full gradients. In addition, we retain full accumulation buffers for only two stages at a time and reuse them via double buffering. While one stage buffer accumulates gradients over consecutive microbatches, gradient synchronization for the previous stage buffer is performed in parallel. This reduces persistent gradient memory to per-stage sharded buffers plus only two full buffers for rolling accumulation, without additional synchronization overhead in practice.\n\n\\textbf{Zero-redundant communication for the Muon distributed optimizer.} Naive Muon implementations all-gather full model parameters on each data-parallel rank, causing transient memory spikes and redundant communication. We restrict all-gather to parameter shards owned by each rank and overlap local computation with shard communication. This eliminates redundant communication and significantly reduces optimizer-related peak memory overhead.\n\n\\textbf{Pipeline activation offloading.} During pipeline warmup, forward execution advances ahead of backpropagation, prolonging the lifetime of intermediate activations. We offload the activations to host memory after forward execution and reload them prior to backward execution~\\cite{298555}. Offloading is applied at layer granularity to further reduce peak memory usage. Combined with fine-grained recomputation, this largely eliminates the need to keep activations resident in GPU memory. Offload and reload are scheduled to overlap with computation while avoiding contention with peer-to-peer communication and MoE token routing (dispatch and combination). This substantially reduces the activation memory footprint with near-zero overhead.\n\n\\textbf{Sequence-chunked output projection for peak memory reduction.} \nOutput projection and cross-entropy loss incur transient memory overhead from storing activations for backpropagation and promoting them to higher precision during loss computation. To reduce this overhead, we partition the input sequence into smaller chunks and compute projection and loss independently on each chunk, completing forward and backward passes and releasing activations before moving on. As a result, peak memory usage decreases as the number of chunks increases. With an appropriate chunk count, this approach alleviates output-layer memory pressure while maintaining performance comparable to unchunked execution.\n\n\\subsubsection{Parallelism Efficiency}\n\n\\textbf{Efficient deferred weight gradient computation.} To reduce pipeline bubbles, we defer some weight gradient computation of the critical path~\\cite{qi2023zero}. Fine-grained deferral with optimized storage and communication overlap improves throughput while keeping memory overhead bounded.\n\n\\textbf{Efficient long-sequence training.} Longer sequences exacerbate load imbalance across data parallel and pipeline parallel groups. We address this through workload-aware sequence reordering, dynamic redistribution of attention computation, and flexible partitioning of data parallel ranks into context-parallel groups of varying sizes~\\cite{ge2025bytescale,wang2025flexsp}. A hierarchical all-to-all overlaps intra-node and inter-node communication for QKV tensors to reduce latency.\n\n\\subsubsection{INT4 Quantization-aware training}\n\nTo provide better accuracy at low-precision, we apply INT4 QAT in the SFT stage. Moreover, to further mitigate the training time overhead, we have developed a quantization kernel applicable to both training and offline weight quantization, which ensures bitwise-identical behavior between training and inference.\n\n\\section{Post-Training}\n\nThe post-training phase of GLM-5 aims to transform the base model into a highly capable assistant with robust reasoning, coding, and agentic abilities. As illustrated in Figure~\\ref{fig:overall_pipeline}, our pipeline follows a progressive alignment strategy: starting with multi-task Supervised Fine-Tuning (SFT) that introduces sophisticated interleaved thinking modes, followed by specialized Reinforcement Learning (RL) stages for reasoning and agentic tasks, and concluding with a general RL stage for human-style alignment. By leveraging on-policy cross-stage distillation as the final refinement, GLM-5 effectively mitigates capability regression while harnessing the performance gains from each training stage.\n\n\\subsection{Supervised Fine-Tuning}\n\n\\begin{figure}[!t]\n \\centering\n \\includegraphics[width=\\linewidth]{figures/glm5_tr_thinking_mode_v2.pdf}\n \\caption{Illustration of Interleaved Thinking and Preserved Thinking.}\n \\label{fig:thinking_mode}\n\\end{figure}\n\nCompared with GLM-4.5, GLM-5 significantly expands the scale of \\emph{Agent} and \\emph{Coding} data during the SFT stage. The SFT corpus of GLM-5 covers three major categories:\n\\begin{itemize}[leftmargin=1.5em,itemsep=2pt]\n \\item \\textbf{General Chat}: question answering, writing, role-playing, translation, multi-turn dialogue, and long-context interactions;\n \\item \\textbf{Reasoning}: mathematical, programming, and scientific reasoning;\n \\item \\textbf{Coding \\& Agent}: frontend and backend engineering code, tool calling, coding agents, search agents, and general-purpose agents.\n\\end{itemize}\n\nAdditionally, GLM-5 extends the maximum context length to 202,752 tokens during SFT. Along with an updated chat template, the model supports three distinct thinking characteristics (see Figure \\ref{fig:thinking_mode}), including:\n\n\\begin{itemize}[leftmargin=1.5em,itemsep=2pt]\n \\item \\textbf{Interleaved Thinking}: the model thinks before every response and tool call, improving instruction following and the quality of generation\\footnote{Interleaved thinking was first introduced by \\url{https://platform.claude.com/docs/en/build-with-claude/extended-thinking\\#interleaved-thinking}}.\n \\item \\textbf{Preserved Thinking}: in coding agent scenarios, the model automatically retains all thinking blocks across multi-turn conversations, reusing existing reasoning instead of re-deriving it from scratch. This reduces information loss and inconsistencies, and is well-suited for long-horizon, complex tasks\\footnote{Preserved thinking was also adopted by Claude since Opus 4.5. See \\url{https://platform.claude.com/docs/en/build-with-claude/extended-thinking\\#thinking-block-preservation-in-claude-opus-4-5-and-later}}.\n \\item \\textbf{Turn-level Thinking}: the model supports per-turn control over reasoning within a session—disable thinking for lightweight requests to reduce latency/cost, enable it for complex tasks to improve accuracy and stability.\n\\end{itemize}\n\nBy thinking between actions and maintaining consistency across turns, GLM-5 achieves more stable and controllable behavior on complex tasks.\n\nFor \\textbf{General Chat}, we optimize the response style to be more logical and concise compared to GLM-4.5. For role-playing tasks, we collect and construct a broader and more diverse dataset covering multiple languages and role configurations. In particular, we define several evaluation dimensions---including instruction following, linguistic expressiveness, creativity, logical coherence, and long-dialogue consistency---and apply both automatic and human filtering to curate and refine the data.\n\nFor \\textbf{Reasoning} tasks, we further enhance the depth of the model's reasoning. Specifically, for logical reasoning, we construct verifiable problems and synthesize high-quality data using rejection sampling. For mathematical and scientific problems, a difficulty-based filtering process is applied, retaining only problems that are challenging for the GLM-4.7 model.\n\nFor \\textbf{Coding} and \\textbf{Agent} tasks, compared to GLM-4.5, GLM-5 constructs a large number of execution environments to obtain high-quality trajectories, with particular emphasis on real-world scenarios and long-horizon tasks. We further improve the SFT data using expert reinforcement learning and rejection sampling. Erroneous segments within trajectories are retained but masked out in the loss function, allowing the model to learn error correction behaviors without reinforcing incorrect actions.\n\n\\subsection{Reasoning RL}\n\n\\paragraph{RL algorithm backbone.}\nOur RL algorithm builds upon GRPO~\\cite{shao2024deepseekmath} and incorporates the IcePop technique~\\cite{IcePop2025} to mitigate the \\emph{training-inference mismatch}, i.e., the discrepancy between the inference distribution and the training distribution during RL optimization. We explicitly distinguish between the \\emph{training policy} $\\pi^{\\text{train}}$, used for gradient updates, and the \\emph{inference policy} $\\pi^{\\text{infer}}$, used for trajectory sampling. Compared to the original IcePop formulation, we remove the KL regularization term to accelerate RL improvement.\nThe final optimization loss is:\n\\begin{equation}\n\\begin{aligned}\n\\mathcal{L}(\\theta)=\n-\\mathbb{E}_{\nx \\sim \\mathcal{D},\n\\{y_i\\}_{i=1}^{G} \\sim \\pi^{\\text{infer}}_{\\theta_{\\text{old}}}(\\cdot \\mid x)\n}\n&\\Bigg[\n\\frac{1}{G}\n\\sum_{i=1}^{G}\n\\frac{1}{|y_i|}\n\\sum_{t=1}^{|y_i|}\n\\operatorname{pop}\n(\\rho_{i,t}, 1/\\beta, \\beta) \\\\\n&\\cdot\n\\min\\!\\left(\nr_{i,t}\\hat{A}_{i,t},\n\\operatorname{clip}\\!\\left(\nr_{i,t},\n1-\\epsilon_{\\text{low}},\n1+\\epsilon_{\\text{high}}\n\\right)\n\\hat{A}_{i,t}\n\\right)\n\\Bigg],\n\\label{icepop}\n\\end{aligned}\n\\end{equation}\nwhere the training-inference mismatch ratio is defined as\n\\begin{align*}\n\\rho_{i,t}\n=\n\\frac{\n\\pi_{\\theta_{\\text{old}}}^{\\text{train}}(y_{i,t} \\mid x, y_{i,<t})\n}{\n\\pi_{\\theta_{\\text{old}}}^{\\text{infer}}(y_{i,t} \\mid x, y_{i,<t})\n}.\n\\end{align*}\n\nThe operator $\\operatorname{pop}(\\cdot)$ suppresses samples whose mismatch ratio deviates excessively:\n\\begin{align*}\n\\operatorname{pop}(\\rho_{i,t}, 1/\\beta, \\beta)\n=\n\\begin{cases}\n\\rho_{i,t}, & 1/\\beta \\le \\rho_{i,t} \\le \\beta, \\\\\n0, & \\text{otherwise}.\n\\end{cases}\n\\end{align*}\n\nThe PPO-style importance ratio and the group-normalized advantage follow the original GRPO definition:\n\\begin{align*}\nr_{i,t}\n=\n\\frac{\n\\pi_\\theta^{\\text{train}}(y_{i,t}\\mid x,y_{i,<t})\n}{\n\\pi_{\\theta_{\\text{old}}}^{\\text{train}}(y_{i,t}\\mid x,y_{i,<t})\n},\n\\quad\n\\hat{A}_{i,t}\n=\n\\frac{\nR_i - \\operatorname{mean}(R_1,\\dots,R_G)\n}{\n\\operatorname{std}(R_1,\\dots,R_G)\n}.\n\\end{align*}\n\nDuring training, we set hyperparameters $\\beta=2, \\epsilon_{\\text{low}}=0.2, \\epsilon_{\\text{high}}=0.28$. Training is performed entirely on-policy with a group size of 32 and a batch size of 32.\n\n\\paragraph{DSA RL insights.}\nWe conduct a very large-scale RL training on a model based on the DSA architecture. Compared with MLA, DSA introduces an additional indexer that retrieves the top-k most relevant key-value entries and computes attention sparsely over the retrieved subset.\nThe retrieved top-k results are critical for RL stability. This is analogous to how MoE models use routing replay~\\cite{zheng2025group} to preserve the activated top-k experts to ensure training-inference consistency.\nHowever, directly adapting this strategy to indexer replay, i.e., storing the indexer’s top-k indices at every token position is clearly impractical, since the $k = 2048$ used by the indexer is much larger than the $k$ typically used in MoE, and storing all these indices would incur enormous storage costs as well as significant communication overhead between the training engine and the inference engine.\n\nWe find that adopting a deterministic top-k operator effectively resolves the training-inference mismatch in DSA indexer token selection. Compared with the non-deterministic CUDA-based top‑k implementation used in SGLang’s DSA Indexer, directly using the naive \\texttt{torch.topk} is slightly slower but deterministic. It produces more consistent outputs and yields substantial RL gains. In contrast, other non-deterministic top‑k operators (e.g., CUDA or TileLang implementations) caused drastic performance degradation during RL after only a few steps, accompanied by a sharp drop in entropy. Therefore, throughout our RL stages, we use \\texttt{torch.topk} as the default top-k operator in the DSA Indexer in our training engine. We also freeze the indexer parameters by default during RL to accelerate training and prevent unstable learning in the indexer.\n\n\\paragraph{Mixed domain reasoning RL.} In the Reasoning RL stage, we perform mixed RL training over four domains: mathematics, science, code, and tool-integrated reasoning (TIR). For mathematics and science, we curate data from both open-source datasets~\\cite{du2025nemotron, moshkov2025aimo} and co-developed collections with external annotation vendors. We further apply difficulty filtering to focus training on problems that GLM-4.7 solves correctly only rarely or fails consistently, while remaining solvable by stronger teacher models (e.g., GPT-5.2 xhigh and Gemini 3 Pro Preview). For code, we cover both competitive programming style tasks and scientific coding tasks. The former is primarily sourced from Codeforces and representative datasets such as TACO~\\cite{li2023taco} and SYNTHETIC-2-RL~\\cite{synthetic2_blog_2025}, while the latter is constructed from internal problem pools by decomposing questions into the minimal code implementations required for correct solutions. For TIR, we reuse the more challenging subset of mathematics and science RL data, and additionally co-build STEM questions with annotation vendors that are explicitly designed to be answered with external tools. During RL training, we assign domain and source-specific judge models or evaluation systems to produce binary outcome rewards. We keep the overall mixture roughly balanced across the four domains, and consistently observe stable and significant gains in each domain under the mixed RL setting.\n\n\\subsection{Agentic RL}\nTo facilitate agentic performance of GLM-5, we develop a fully asynchronous and decoupled RL framework and optimize GLM-5 in coding and search agent tasks. Naive synchronous RL suffers from severe GPU idle time during long-horizon agent rollouts. By decoupling inference and training engines via a central Multi-Task Rollout Orchestrator, we achieve high-throughput joint training across diverse agentic workloads.\n\nTo maintain training stability under asynchronous off-policy conditions, we introduce two key mechanisms. First, a Token-in-Token-out (TITO) gateway eliminates re-tokenization mismatches by preserving exact action-level correspondence. Second, we employ a Direct Double-sided Importance Sampling, which applies a token-level clipping mechanism ($[1-\\epsilon_\\ell, 1+\\epsilon_h]$) to rollout log-probabilities, while efficiently controlling off-policy bias without tracking historical policy checkpoints. We also employ a DP-aware routing to maximize KV-cache reuse during long-context inference for large-scale MoE models for speed up. To scaling agentic environments, we scale verifiable training environments across three domains: over 10K real-world Software Engineering (SWE), terminal tasks, and high-difficulty multi-hop search tasks. More details about agentic RL can be found in the subsequent Section 4.\n\n\\subsection{General RL}\n\n\\paragraph{Multi-dimensional optimization objectives.}\nWe decompose the optimization objectives of General RL into three complementary dimensions: \\emph{foundational correctness}, \\emph{emotional intelligence}, and \\emph{task-specific quality}.\n\nThe \\emph{foundational correctness} dimension serves as the bedrock of response quality. It targets a broad spectrum of error types that undermine the usability of model outputs, including instruction-following failures, logical inconsistencies, factual inaccuracies, knowledge hallucinations, and language disfluencies. The goal is to minimize the error rate so that responses reach a \\emph{usable} baseline. We consider this a prerequisite for all subsequent optimization: a response containing factual errors or misinterpreting the user's intent can actively mislead the user, no matter how polished it may appear.\n\nThe \\emph{emotional intelligence} dimension optimizes user experience beyond core correctness. It aims to produce responses that are empathetic, insightful, and stylistically close to natural human communication, making interactions with the model feel more natural and engaging.\n\nThe \\emph{task-specific quality} dimension targets fine-grained optimization across various specific tasks. Building on the usability established by foundational correctness, it aims to elevate responses from merely correct to genuinely high-quality within each task category. This dimension covers a wide range of tasks, including writing, text processing, subjective and objective question answering, role-playing, and translation. Each task domain demands distinct reward signals, necessitating a hybrid reward system.\n\n\\paragraph{Hybrid reward system.}\nTo supervise the diverse objectives above, we build a hybrid reward system that integrates three complementary types of reward signals: \\emph{rule-based reward functions}, \\emph{outcome reward models} (ORMs), and \\emph{generative reward models} (GRMs). Each has distinct strengths and weaknesses, and their combination is key to a stable, efficient, and scalable General RL training process.\n\nRule-based rewards provide precise and interpretable signals, but are limited to aspects expressible as deterministic rules. ORMs offer low-variance signals and high training efficiency, but are more susceptible to reward hacking, where the policy exploits superficial patterns rather than genuinely improving core capability. GRMs leverage language models to produce scalar or structured evaluations and are more robust to such exploitation, but tend to exhibit higher variance. By blending these three signal types, we obtain a reward system that balances precision, efficiency, and robustness, mitigating the weaknesses of any single component.\n\n\\paragraph{Human-in-the-loop style alignment.}\nA distinctive aspect of our General RL pipeline is the explicit incorporation of high-quality human-authored responses. Rather than relying solely on model-generated responses, we introduce expert human responses as stylistic and qualitative anchors. This is motivated by the observation that purely model-generated optimization tends to converge toward recognizably ``model-like'' patterns---often verbose, formulaic, or lacking the nuance of skilled human writing. By exposing the model to human-written exemplars, we encourage it to adopt more natural, human-aligned response patterns.\n\n\\subsection{On-Policy Cross-Stage Distillation}\nIn our multi-stage RL pipeline, sequentially optimizing for distinct objectives can lead to the cumulative degradation of previously acquired capabilities. To mitigate this issue, we perform on-policy cross-stage distillation as the \\textbf{final stage}, adopting an on-policy distillation algorithm~\\cite{guminillm,qwen3,xiao2026mimov2flashtechnicalreport,lu2025onpolicydistillation} to swiftly recover the skills acquired in earlier SFT and RL stages (Reasoning RL and General RL). Specifically, the final checkpoints from the preceding training stages serve as teacher models, where the training prompts are sampled from the corresponding teachers' RL training sets and mixed in appropriate proportions. The training loss can be obtained by replacing the advantage term in Eq.~\\ref{icepop} with the following formula (`sg' stands for the stop gradient operation, e.g., \\texttt{.detach()}):\n\n\\begin{equation}\n \\hat{A}_{i,t} = \\text{sg}\\left[\\log\\frac{\\pi_{\\theta_{\\text{teacher}}}^{\\text{infer}}(y_{i,t}\\mid x,y_{i,<t})}{\\pi_\\theta^{\\text{train}}(y_{i,t}\\mid x,y_{i,<t})}\\right].\n\\end{equation}\n\nCurrently, we utilize the inference engine to fetch teachers' logits. In the future, we plan to migrate the inference backend to the training engine and uniformly adopt the Multi-Query Attention (MQA) mode of MLA for inference ($\\pi_{\\theta_{\\text{teacher}}}^{\\text{infer}}\\rightarrow\\pi_{\\theta_{\\text{teacher}}}^{\\text{train}}$).\nDuring training, the group size in the GRPO algorithm is configured to 1 to increase data throughput, and the batch size is set to 1024. This is feasible at this stage because it is no longer necessary to maintain a large group of samples per prompt to estimate advantages; the advantage is computed directly from the gap with the teacher models instead.\n\n\\subsection{RL Training Infrastructure: The \\emph{slime} Framework}\n\nWe continue to use slime as the unified post-training infrastructure for GLM-5, enabling end-to-end reinforcement learning (RL) at scale. Rather than introducing new system components, GLM-5 \\emph{fully leverages} slime's capabilities to (1) broaden task coverage via free-form rollout customization and a server-based execution model, (2) substantially increase throughput via mixed-precision training/rollouts together with MTP and Prefill-Decode (PD) disaggregation---particularly for multi-turn RL workloads, and (3) improve robustness through heartbeat-driven rollout fault tolerance and router-level server lifecycle management.\n\n\\subsubsection{Scaling Out: Flexible Training via Highly Customizable Rollouts}\nGLM-5’s post-training spans a diverse spectrum of objectives. To support this diversity without task-specific forks, GLM-5 leverages slime's highly customizable rollout interface together with its server-based rollout execution.\n\n\\textbf{Highly customizable rollouts.} slime provides a flexible interface for implementing task-specific rollout logic---including multi-turn interaction loops, tool invocation, environment feedback handling, and verifier-guided branching---without modifying the underlying infrastructure. GLM-5 leverages this capability to support a broad range of domains and training paradigms, including but not limited to reasoning RL, general RL, agentic RL, and on-policy distillation, all within a unified training stack.\n\n\\textbf{Server-based rollouts via HTTP APIs.} slime exposes its rollout servers and inference router through standard HTTP APIs, allowing users to interact with slime's serving layer in the same way as a conventional inference engine. This decouples rollout logic from the training process boundary: external agent frameworks and environments can call the server/router endpoints directly, while the optimization backend remains unchanged for both short-horizon single-turn training and long-horizon multi-turn trajectories.\n\n\\subsubsection{Scaling Up: Tail-Latency Optimization for RL Rollouts}\nFor RL rollouts, the optimization target is not aggregate throughput but \\emph{end-to-end latency}, dominated by the slowest (long-tail) sample in each step. In practice, a single straggling trajectory can stall synchronization points (e.g., batch completion, buffer readiness, trainer updates) and directly determine wall-clock progress. GLM-5 therefore fully leverages slime's latency-oriented serving and scheduling mechanisms to minimize both median latency and, more importantly, tail latency.\n\n\\textbf{No-queue serving via multi-node inference with DP-attention for MLA.}\nTo avoid queueing delays, rollout requests must be served promptly even under bursty traffic, which requires substantial KV-cache capacity. GLM-5 adopts a multi-node inference deployment (e.g., EP64 and DP64 over 8 nodes) to provision sufficient distributed KV-cache. DP-attention is primarily introduced to prevent copying KV across different ranks.\n\n\\textbf{Tail-latency reduction with FP8 rollouts and MTP.}\nGLM-5 uses FP8 for rollout inference to reduce per-token latency and shorten the completion time of long trajectories. In addition, GLM-5 leverages slime's support for Multi-Token Prediction (MTP), which is especially effective under the small-batch decoding regime typical in RL rollouts. Since tail latency is often driven by small-BS stragglers (e.g., rare long contexts, complex multi-turn reasoning, tool-heavy traces), MTP provides disproportionately large benefits on the long tail, improving the time-to-completion of the slowest sample and thus reducing step-level stall time.\n\n\\textbf{PD disaggregation to prevent prefill-decode interference in multi-turn RL.}\nIn multi-turn settings, long-prefix prefills are frequent (conversation history, tool traces, code context). Under DP-attention, mixing prefill and decode on the same serving resources can create severe interference: a heavy prefill can preempt or disrupt ongoing decodes on the server, preventing other samples from making continuous progress and sharply worsening tail latency. GLM-5, therefore, leverages slime's Prefill--Decode (PD) disaggregation. By running prefills and decodes on dedicated resources, decodes remain stable and uninterrupted, enabling long-horizon samples to progress continuously and significantly improving tail behavior in multi-turn agentic RL.\n\n\\subsubsection{Rollout Robustness: Heartbeat-Driven Fault Tolerance}\n\nAt scale, transient failures (e.g., individual server crashes, network issues, or performance degradation) are inevitable. GLM-5 leverages slime's heartbeat-driven fault-tolerance to ensure training continuity under such events: rollout servers periodically emit heartbeats monitored by the orchestration layer, and unhealthy servers are proactively terminated and deregistered from the inference router. As a result, retries are automatically routed away from failed or degraded servers to healthy ones, preventing single-server incidents from interrupting rollouts and preserving uninterrupted end-to-end RL training.\n\n\\section{Agentic Engineering}\nWe describe the transition from \\textbf{vibe coding} (human prompting) to \\textbf{agentic engineering}.\nIn vibe coding, a human prompts an AI model to write code. In agentic engineering, AI agents write the code themselves. They plan, implement, and iterate. \nTo support these long-horizon tasks, GLM-5 utilizes a fully asynchronous and decoupled RL framework to significantly boost GPU utilization by reducing idle time during agent rollouts. \nTo scaling agent environments, we have developed environment-building pipelines. For coding tasks, we set up real-world software engineering issues and terminal tasks by creating over 10,000 verifiable training scenarios. For search agents, we develop an automatic and scalable complex multi-step reasoning data synthesis pipeline to build data for agentic training.\n\n\\subsection{Asynchronous RL for Agentic Tasks}\nTo conduct RL for agent tasks, we design a fully asynchronous and decoupled RL infrastructure that efficiently handles long-horizon agent rollouts and supports flexible multi-task RL training across diverse agent frameworks.\n\nWe adopt the group-wise policy optimization algorithm for RL training.\nFor each problem \\(x\\), we sample \\(K\\) agent traces \\(\\{y_1,\\dots,y_K\\}\\) from the previous policy\n\\(\\pi_{\\text{old}}\\), and optimize the model \\(\\pi_\\theta\\) with respect to the following objective:\n\\[\nL(\\theta)\n = \\mathbb{E}_{x\\sim\\mathcal{D}}\\!\\left[\n \\frac{1}{K}\\sum_{i=1}^{K}\n \\left(\n r(x,y_i) - \\bar{r}(x)\n \\right)\n \\right],\n\\]\nwhere $\\bar{r}(x) \\;=\\; \\frac{1}{K}\\sum_{i=1}^{K} r\\bigl(x,y_i\\bigr)$\nis the mean reward of the sampled responses. It is noted that only model-generated tokens are used for optimization, and the environment feedback is ignored in loss computation. \n\n\\subsubsection{Asynchronous RL Design for Agentic Training}\n\nDue to the long-tail nature of the rollout process, naive synchronous RL training introduces substantial bubbles during the rollout stage because of the severely imbalanced generation of agentic tasks, which can cause large GPU idle time. \nTo improve training throughput, we adopt a fully asynchronous training paradigm for Agentic RL to boost GPU utilization and training efficiency.\nConcretely, we decouple the training engine and the inference engine onto different GPU devices. \nThe inference engine continuously generates trajectories. Once the number of generated trajectories reaches a predefined threshold, the batch is sent to the training engine to update the model.\nTo reduce policy lag and keep the training approximately on-policy, the model weights used by the rollout engine are periodically synchronized with those of the training engine. The training engine updates the model parameters and pushes the new weights back to the inference engine every $K$ gradient updates.\nWhile asynchrony could significantly improve overall training efficiency, it also means that different trajectories may be generated by different versions of the model, introducing a severe off-policy issue. \nSince the weight update considers a different optimization problem due to the changing\nrollout policy, we also reset the optimizer after each weight update of the inference engine.\n\n\\paragraph{Server-based multi-task training design.}\nTo address the heterogeneity of trajectory generation in multi-task RL, where different tasks typically rely on distinct tool sets and task-specific rollout logic, we introduce a server-based Multi-Task Rollout Orchestrator for multi-task RL training.\nThis component is designed to ensure seamless compatibility between the slime RL training framework and diverse downstream tasks through a central orchestrator with multiple registered task services. Specifically, each task implements its own rollout and reward logic as an independent microservice, which is registered with the central orchestrator for management and scheduling.\nDuring the rollout stage, the central orchestrator controls the per-task rollout ratio and generation speed to achieve balanced data collection across tasks. Crucially, we standardize trajectories from all agentic tasks into a unified message-list representation. This enables joint training of complex agentic frameworks (e.g., Software Engineering task) while also supporting centralized post-processing and logging for heterogeneous workloads. This design cleanly isolates task-specific logic from the core training loop, enabling seamless integration with multi-task RL training.\nServing as the backbone of the GLM-5 training infrastructure, this orchestrator supports over 1k concurrent rollouts and enables automated, dynamic adjustment of task sampling ratios, as well as fine-grained monitoring of task progress.\n\n\\subsubsection{Optimizing Asynchronous Training Stability}\n\\paragraph{Token-in-Token-out vs. Text-in-Text-out.}\nIn an RL rollout setting, \\emph{token-in-token-out} (TITO) means the training pipeline consumes the \\emph{exact} tokenization and decoded-token stream produced by the inference engine, and uses it directly to build trajectories for learning. \nIn contrast, \\emph{text-in-text-out} treats the rollout engine as a black box that returns finalized text; the trainer then reconstructs the trajectory by re-tokenizing that text (and often re-deriving boundaries and truncation) before computing losses. \nThis seemingly small choice is consequential: re-tokenization can introduce subtle mismatches in token boundaries, whitespace/normalization handling, truncation, or special-token placement, which in turn can corrupt step alignment between actions and rewards/advantages—especially when rollouts are streamed, truncated, or interleaved across many actors. \nWe find token-in-token-out is critical for asynchronous RL training because it preserves exact action-level correspondence between what was sampled and what is optimized while enabling actors to emit trajectory fragments (token IDs + metadata) immediately without a lossy text round-trip and without waiting for post-hoc re-tokenization on the learner side.\nIn practice, we implement a TITO Gateway that intercepts all generation requests from rollout tasks and records each trajectory’s token IDs and metadata. \nThis design isolates the cumbersome token ID processing from downstream agent rollout logic, while avoiding re-tokenization mismatches during RL training.\n\n\\paragraph{Direct\ndouble-sided importance sampling for token clipping.}\nUnlike the synchronous RL training setting in Section 3, in the asynchronous setting, rollout engines may undergo multiple updates during a single trajectory generation,\nwhich renders the tracking of exact behavior probabilities $\\pi_{\\theta_{\\text{old}}}$ computationally prohibitive. Otherwise, we have to maintain an extensive history of model checkpoints $\\{\\pi_{\\theta_{\\text{old}}^{(1)}}, \\dots, \\pi_{\\theta_{\\text{old}}^{(N)}}\\}$, which is infeasible in practical implementation.\n\nTo resolve this, we first employ a simplified token-level importance sampling mechanism that reuses the log-probabilities generated during rollout as a direct behavior proxy. By calculating the importance sampling ratio as $r_t(\\theta) = \\frac{\\pi_{\\theta}}{\\pi_{\\text{rollout}}}$ and discarding the traditional $\\pi_{\\theta_{\\text{old}}}$, we eliminate the computational overhead of separate old-policy inference. Second, we employ a double-sided calibration token-level masking strategy. Instead of the asymmetric clipping used in standard PPO, we restrict the trust region to $[1-\\epsilon_\\ell, 1+\\epsilon_h]$, where $\\epsilon_\\ell$ and $\\epsilon_h$ are clipping hyperparameters. Tokens falling outside this interval are entirely masked from gradient computation to prevent instabilities caused by extreme policy divergence. This shares similarities with the IcePop mechanism \\cite{team2025every}, yet our strategy is simpler by further removing the $\\pi_{\\theta_{\\text{old}}}$ and achieving more stable training. \n\nFormally, the optimization objective with token-level clipping can be written as:\n\\begin{equation}\n L(\\theta) = \\mathbb{E}_t \\left[ f(r_t(\\theta), \\epsilon_l, \\epsilon_h) \\hat{A}_t \\log \\pi_{\\theta}(a_t|s_t) \\right]\n\\end{equation}\nIn this formulation, the importance sampling ratio $r_t(\\theta)$ is computed as:\n\\begin{equation}\n r_t(\\theta) = \\exp\\left( \\log \\pi_\\theta(a_t|s_t) - \\log \\pi_{\\text{rollout}}(a_t|s_t) \\right)\n\\end{equation}\nStability is further enforced via the calibration function $f(x; \\epsilon_\\ell, \\epsilon_h)$:\n\\begin{equation}\n f(x; \\epsilon_\\ell, \\epsilon_h) = \n \\begin{cases} \n x, & \\text{if } 1-\\epsilon_\\ell < x < 1+\\epsilon_h \\\\\n 0, & \\text{otherwise}\n \\end{cases}\n\\end{equation}\n\nIn the experiments, we find that reusing rollout log-probabilities accepts a controlled degree of off-policy bias to circumvent the need for historical policy tracking while boosting training stability.\n\n\\paragraph{Dropping off-policy and noisy samples.}\nIn asynchronous RL, overly long trajectories can become highly off-policy, which may destabilize training. To filter out these severely off-policy samples, we log the policy weight version used by the rollout engine at generation time. Specifically, for each response we record the sequence of model versions involved, $(w_0, \\ldots, w_k)$ with $w_0 < \\cdots < w_k$. Let $w'$ denote the current policy version. We discard a sample if its oldest rollout version is too stale, i.e., if $w' - w_0 > \\tau$, where $\\tau$ is a predefined threshold. This removes trajectories that lag too far behind the current policy.\n\nAdditionally, coding-agent sandboxes can be inherently unstable and may fail for reasons unrelated to the model (e.g., environment crashes). Such failures introduce noisy training signals because they reflect environment instability rather than the model’s capability. To mitigate this, we record the failure reason for each sample and exclude samples that fail due to environment collapse. For group-based sampling methods such as GRPO, removing failed samples can leave an incomplete group. In that case, we pad the group by repeating valid samples if the number of valid samples exceeds half of the group size; otherwise, we drop the entire group. This procedure reduces spurious reward noise and improves training stability.\n\n\\paragraph{DP-aware routing for acceleration.}\nWe propose a DP-aware routing mechanism to preserve KV cache locality under Data Parallelism (DP) for large-scale MoE inference. In multi-turn agentic workloads, sequential requests from the same rollout share an identical prefix. To maximize KV reuse, we enforce rollout-level affinity: all requests belonging to a given agent instance are routed to the same DP rank. Concretely, we introduce a stateful routing layer that maps each rollout ID to a fixed DP rank using consistent hashing. This mapping remains stable across turns, eliminating cross-rank cache misses. To prevent long-term imbalance, we combine hashing with lightweight dynamic load rebalancing over the hash space. This design avoids redundant prefill computation without requiring KV synchronization across DP ranks. As rollout length increases, prefill cost remains proportional to incremental tokens rather than total context length. The result is improved end-to-end latency and higher effective throughput for long-context agentic inference.\n\n\\subsection{Environment Scaling for Agents}\n\nTo support reinforcement learning across diverse agentic tasks, we construct verifiable, executable environments that provide grounded feedback for both code-centric and content-generation workflows.\nFor agentic coding tasks, we develop two environment-building pipelines that construct verifiable executable environments: an environment setup pipeline built upon real-world software engineering issues, and a synthesis pipeline for terminal-agent environments.\nBeyond coding, we further introduce a slide generation environment, in which the agent operates over structured HTML with executable rendering and layout-based verification.\n\n\\subsubsection{Software Engineering (SWE) Environments} \nBefore constructing executable environments, we collect a large corpus of real-world Issue-Pull Request (PR) pairs and apply rigorous rule-based and LLM-based filtering to ensure the acquisition of authentic, high-quality issue statements. We categorize these instances into different task types--bug fixing, feature implementation, refactoring, and others--and include the necessary task requirements to ensure that the model's implementation is consistent with the test patch. We employ an environment setup pipeline based on the RepoLaunch~\\citep{zhang2025swebenchgoeslive} framework that scales the construction of executable environments from real-world SWE issues. This pipeline automatically analyzes a repository’s installation and dependency setup to build an executable environment and generate test commands, then leverages LLM to generate language-aware log-parsing functions from test outputs, enabling the extraction of Fail-to-Pass (F2P) and Pass-to-Pass (P2P) test cases. Using this pipeline, we construct over 10k verifiable environments across thousands of repositories spanning 9 programming languages, including Python, Java, Go, C, CPP, JavaScript, TypeScript, PHP, and Ruby. \n\n\\subsubsection{Terminal Environments} \n\n\\paragraph{Synthesis from seed data.} To build verifiable terminal-agent environments at scale, we design an agentic data synthesis pipeline comprising three phases: task draft generation, concrete task implementation, and iterative task optimization. Starting from a set of seed tasks collected from real-world software engineering and terminal-based computer-use scenarios, we leveraged LLM to brainstorm and generate a large pool of verifiable terminal-task drafts. These drafts are then instantiated by a construction agent into concrete tasks in the Harbor \\cite{harborframeworkteam2026harborframework} format, including structured task descriptions, Dockerized execution environments, and corresponding test scripts. Subsequently, a refine agent inspects and iteratively refines the generated tasks according to manually defined rubrics, ensuring that Docker images can be built reliably, test cases are consistent with task specifications, and the environments are robust against potential exploits or shortcuts. Overall, the pipeline yields thousands of diverse and verifiable terminal-agent environments with Docker construction accuracy exceeding 90\\%. \n\n\\paragraph{Synthesis from web-corpus.}\nWe develop a scalable, automated pipeline and construct LLM-verified terminal-based coding tasks based on web corpus, using a closed-loop design where the constructing agent also serves as its own first-pass evaluator.\nFirst, we collect a large-scale corpus of code-relevant web pages and apply a data quality classifier to retain only high-quality content, discarding pages that are predominantly non-technical or lack substantive code content. From the filtered subset, we further identify web pages amenable to terminal-style task formulation. We then apply stratified sampling across topic categories and difficulty levels to ensure distributional balance and diversity in the resulting task pool.\nSecond, we prompt a coding agent with the Harbor task construction specification\\footnote{\\url{https://harborframework.com/docs/tasks/task-tutorial}}, including the task schema, formatting requirements, and exemplar tasks, alongside each selected source web page. The agent is instructed to (i) synthesize a complete terminal task grounded in the web page content, and (ii) execute the Harbor validation script against its own output. Upon validation failure, the agent iteratively diagnoses and revises the task until it passes all automated checks. Only tasks that successfully clear this self-verification loop are admitted into the final dataset.\n\n\\subsubsection{Search Tasks}\n\nFor deep-search information-seeking tasks, we build a data-synthesis pipeline that produces challenging multi-hop QA pairs. Each question requires multi-step reasoning grounded in evidence aggregated from multiple web sources.\n\n\\textbf{Web Knowledge Graph (WKG) Construction and Question Generation.}\nStarting from trajectories of an early-stage search agent, we collect and deduplicate all encountered URLs, retaining over two million high-information web pages across diverse domains. \nThe LLM performs semantic parsing for entity recognition, noise filtering, and structured information extraction. \nThe WKG is continuously updated with new pages and refined using downstream verification signals via entity alignment, attribute normalization, relation consolidation, and semantic-consistency corrections. \nBased on the WKG, we sample low- to mid-frequency entities as seed nodes and expand their multi-hop neighborhoods to form complete subgraphs, while controlling expansion to reduce overlap. \nUsing prompts targeting high-difficulty, multi-domain reasoning, we convert each subgraph into a question that implicitly encodes multi-entity relational chains. \n\n\\textbf{High-Difficulty Question Filtering and Verification.}\nWe apply a three-stage pipeline to balance difficulty and correctness:\n(1) Remove questions that a tool-free reasoning model correctly answers in at least one of eight independent attempts.\n(2) Filter out questions solvable by an early-stage agent with basic search, browsing, and computation within a few steps. \n(3) Apply a verification agent for bidirectional validation: we collect candidate answers from the search trajectories in stage 2, then independently verify the question–answer consistency for both the candidates and the annotated ground truth, rejecting samples with non-unique answers, inconsistent evidence, or incorrect labels. \nThis yields high-quality, high-difficulty, reliable multi-hop QA pairs.\n\n\\subsubsection{Inference with Context Management for Search Agents}\n\nWe find that the performance on BrowseComp~\\cite{wei2025browsecomp} is sensitive to both the judge prompt and the judge model, and open-source judges can introduce systematic bias. To ensure consistency and reproducibility, we standardize all judge-based components using the official OpenAI evaluation prompt and the proprietary model o3-mini as the judge. Our case studies indicate this configuration aligns best with human-annotated ground truth, so we adopt it for all search agent evaluations.\n\nPrior work~\\citep{liu2025deepseek} has introduced context management, where \\textit{Discard-all} resets the context by removing the entire history of tool calls. We further observe that model accuracy degrades substantially under extremely long contexts (e.g., beyond 100k tokens). Motivated by this, we employ a simple \\textit{Keep-recent-k} strategy. When the interaction history exceeds a threshold $k$, the content older than the most recent $k$ rounds will be folded to control context length. \nLet the trajectory be\n$(q,r_1,a_1,o_1,r_2,a_2,o_2,\\cdots,r_n,a_n,o_n)$, \nwhere $q$ denotes the question, $r_i$ denotes the reasoning at round $i$, $a_i$ the action (we design \\textit{search}, \\textit{open}, \\textit{find} and \\textit{python} 4 tools), and $o_i$ the tool observation. We fold only observations earlier than the most recent $k$ rounds:\n$\n o_i \\leftarrow \\text{Tool result is omitted to save tokens.}\\quad i=1, \\ldots, n-k\n$.\nIn our experiments, we set $k=5$, which yields a stable improvement and improves GLM-5 from 55.3\\%($w/o$ keep-recent-$k$) to 62.0\\%($w/$ keep-recent-$k$). We also find that using different values of keep recent $k$ or alternatively triggering keep-recent once the context length reaches a predefined token threshold, leads to the same results.\n\nBuilding on this, we combine keep-recent with Discard-all to form a hybrid \\textit{Hierarchical Context Management} strategy. During inference with keep-recent, if the total context length exceeds a threshold $T$, we discard the entire tool-call history and restart with a fresh context, while continuing to apply the keep-recent strategy. We select $T=32k$ via parameter search.\n\nAs shown in Figure~\\ref{fig:GLM5-BC-CM}, under different compute budgets, this strategy effectively frees up context space, enabling the model to execute more steps and consistently improving performance. Compared to using Discard-all alone, combining with keep-recent-k achieves consistent gains across all budgets, reaching a final score of 75.9, outperforming all open-source models equipped with context-management.\n\n\\begin{figure}\n \\centering\n \\includegraphics[width=0.7\\linewidth]{figures/GLM5-BC-cm.pdf}\n \\caption{Accuracy of BrowseComp with different context management strategies from GLM-4.7 (gray baselines) to GLM-5 (colored strategies). }\n \\label{fig:GLM5-BC-CM}\n\\end{figure}\n\n\\subsubsection{Slide Generation}\n\nWe employ a self-improving pipeline that aims to systematically enhance slide generation performance by training a specialized slide-generation expert through reinforcement learning and rejection sampling fine-tuning. We first initialize the model with supervised fine-tuning (SFT) to provide a basic slide generation capability, and then perform reinforcement learning with a multi-level reward formulation grounded in common aesthetic and structural properties of presentation slides. This stage leads to substantial improvements in generation quality. We further conduct rejection sampling fine-tuning and mask fine-tuning, allowing knowledge acquired during reinforcement learning to be injected back into the training corpus. This procedure jointly enhances data quality and model capability in a coordinated and iterative manner.\n\nWe propose a \\textbf{multi-level reward formulation}, which partitions reward signals in the HTML-based slide generation process into three levels:\n\n\\textbf{Level-1: Static markup attributes.} This level focuses on declarative attributes in the generated HTML, including positioning, spacing, color, typography, saturation, and other stylistic attributes. Grounded in professional design principles, we design a set of rules to regulate the model’s behavior when generating such declarations. These rules ensure syntactic parsability of the generated HTML, while constraining the design space at the markup level to a subspace optimized for expressiveness, structural clarity, visual harmony, and readability. Additionally, we introduce hallucinated-image and duplicate-image detection mechanisms to suppress hallucinatory or redundant figures.\n\n\\textbf{Level-2: Runtime rendering properties.} Unlike static inspection, this level evaluates runtime properties of DOM nodes during rendering, such as element width and height, bounding boxes, and other geometric layout metrics. By constraining these properties, we encourage the generated slides to align more closely with human aesthetic preferences in spatial organization. We develop a distributed rendering service capable of executing rendering jobs at high throughput while extracting the required runtime properties. During training, we observe several forms of reward hacking behaviors, such as hard truncation of overlong content or excessive manipulation of spacing (see Figure~\\ref{fig:reward_hacking}). To mitigate these issues, we refine the renderer implementation to eliminate exploitable loopholes, ensuring that reward signals genuinely incentivize aesthetically coherent layouts rather than superficial compliance with geometric metrics.\n\n\\begin{figure*}[!thp]\n \\centering \n \\includegraphics[width=\\linewidth]{figures/ppt_reward_hacking.pdf}\n \\vspace{-0.1cm}\n \\caption{\n Examples of reward hacking in the slides RL training. Our runtime rendering obtains grounded attribute values, making the evaluation robust to such hacking behaviors.\n }\n \\label{fig:reward_hacking}\n \\vspace{-0.2cm}\n\\end{figure*}\n\n\\textbf{Level-3: Visual perceptual features.} Beyond runtime rendering constraints, we incorporate perceptual-level evaluations of the rendered slides. For instance, we detect abnormal whitespace patterns as an auxiliary signal to further improve overall compositional balance and visual aesthetics.\n\n\\textbf{Training strategy.} These signals are jointly optimized during RL to improve the structural validity of generated HTML, enhance layout organization, and elevate overall visual aesthetic quality. In addition to reward design, we reshape the training distribution via dynamic sampling. Specifically, a fraction of structurally trivial samples is probabilistically dropped, allowing optimization to focus on more challenging pages and improving robustness under complex composition scenarios. We also employ a token-level policy gradient loss to stabilize optimization~\\cite{yu2025dapo}. Furthermore, we introduce a balancing strategy that distributes different rollout outcomes of the same sample across multiple training batches, reducing optimization bias and improving training stability.\n\n\\paragraph{Rejection sampling.}\nDuring the rejection sampling phase, the reward functions used in RL are transferred into a data filtering pipeline to construct a high-quality training subset.\nAt the page level, filtering criteria include code validity and compilation feasibility. At the trajectory level, we further enforce tool execution correctness and global content diversity constraints, ensuring structural consistency.\nWe adopt a Best-of-$N$ selection strategy, in which the highest-quality sample is retained from multiple independently generated candidates. This mechanism effectively reweights the distribution toward higher-quality instances, leading to improved sample efficiency and enhanced training stability.\n\n\\paragraph{Masking-based refinement.} \nAlthough rejection sampling removes the majority of low-quality outputs, some trajectories contain defects confined to only a small number of pages. Discarding such samples would reduce effective data utilization and increase generation cost. To address this, we introduce a masking-based correction mechanism that automatically identifies defective pages and applies masking, while retaining the high-quality content within the same trajectory. This selective refinement preserves valuable supervision signals, improves effective data efficiency, and reduces redundant regeneration overhead, thereby enhancing overall training efficiency.\n\n\\paragraph{Empirical improvements.}\nThe proportion of generated pages that strictly comply with the 16:9 aspect ratio increases from 40\\% to 92\\%, accompanied by a substantial reduction in page overflow cases. Human evaluation further shows that, compared to GLM-4.5, GLM-5 achieves win rates of 60\\% in content quality, 57.5\\% in layout rationality, and 65\\% in visual aesthetics, resulting in an overall win rate of 67.5\\%. These results provide empirical evidence for the effectiveness of the proposed multi-level reward design and self-improving framework.\n\n\\section{Adapting GLM-5 to Chinese Chip Infrastructure}\n\\label{sec:chinese}\n\nAdapting GLM-5 to diverse Chinese chip infrastructures presents significant challenges due to the heterogeneity of hardware ecosystems, which often complicates high-performance deployment. Despite these hurdles, we have successfully achieved full-stack adaptation for GLM-5 through close collaboration with seven mainstream Chinese chip platforms, including Huawei Ascend, Moore Threads, Hygon, Cambricon, Kunlunxin, MetaX, and Enflame. In this section, we use the Ascend Atlas series as a case study to demonstrate our adaptation methodology, focusing on three core pillars: extreme quantization, high-performance kernel fusion, and advanced inference engine scheduling.\n\n\\paragraph{Mixed-Precision W4A8 quantization.}\nTo fit the 750B parameter GLM-5 model onto a single Atlas 800T A3 machine, we implemented a sophisticated W4A8 mixed-precision quantization strategy. Utilizing the msModelSlim~\\footnote{\\url{https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/80RC3alpha003/devaids/auxiliarydevtool/modelslim_0001.html}} tool, we applied specific precisions to different model components: standard Attention and MLP blocks use W8A8 (INT8), while the MoE experts are compressed to W4A8 (INT4) to drastically reduce memory footprint without significant accuracy loss. Advanced algorithms like QuaRot~\\cite{ashkboos2024quarotoutlierfree4bitinference} for outlier suppression and \\texttt{Flex\\_AWQ\\_SSZ} for scaling calibration were employed to maintain stability in low-bit deployment.\n\n\\paragraph{High-Performance fusion kernels.}\nTo overcome the computational bottlenecks of sparse attention on Ascend NPUs, we developed a suite of customized fusion kernels: Lightning Indexer, Sparse Flash Attention, and MLAPO (Multi-head Latent Attention Pre-processing Optimization). \nLightning Indexer integrates score calculation, ReLU, and TopK operations into a single kernel, allowing the NPU to overlap computation with memory access.\nFor the Sparse Flash Attention kernel, we specifically optimized for GLM-5's sparse patterns. This kernel handles the selection of TopK tokens from the KV cache and sparse attention computation in parallel.\nLast, MLAPO fuses 13 small pre-processing operators into one ``super operator'', utilizing parallel processing between Vector and Cube units to boost end-to-end efficiency.\n\n\\paragraph{Specialized inference engine optimizations.}\nWe adapted two leading inference engines, vLLM-Ascend and SGLang, to maximize hardware utilization:\n\n\\begin{itemize}[leftmargin=1.5em,itemsep=2pt]\n \\item \\textbf{Asynchronous Scheduling:} Within vLLM, we implemented a mechanism to overlap the ``Device-to-Host'' (D2H) sampling copies with the preparation of the next decode step, effectively eliminating scheduling \"bubbles.\"\n \n \\item \\textbf{Context Management:} Features like RadixCache (prefix sharing) and Prefix Cache (extending KV storage to system RAM) enable efficient reuse of KV entries, which is critical for long-context performance.\n\n \\item \\textbf{Parallel Strategy:} We utilized a hybrid approach combining Attention Data Parallelism (DP) and MoE Expert Parallelism (EP), alongside FlashComm, which splits AllReduce operations to hide communication latency behind computation.\n\n \\item \\textbf{Multi-Token Prediction (MTP):} By generating multiple tokens per inference step, we significantly increased NPU computation density and reduced total sequence generation time.\n\\end{itemize}\n\nThrough these hardware-level co-optimizations, GLM-5 on a single Chinese node achieves performance comparable to dual-GPU international clusters, while reducing deployment costs in long-sequence scenarios by 50\\%.\n\n\\section{Evaluation}\n\\label{sec:evaluation}\n\nAs illustrated above, GLM-5 marks the transition from vibe coding to a new era of agentic engineering. \nWe first assess GLM-5 with frontier models on agentic, reasoning, and coding (ARC) benchmarks. \nTo fully evaluate the performance of GLM-5 in real-world agentic engineering scenarios, we propose a new internal evaluation suite, CC-Bench-V2, which includes frontend, backend, and long-horizon tasks. \nFinally, we evaluate the general abilities of GLM-5 in five common real-world scenarios. \n\n\\subsection{Evaluation of ARC Benchmarks}\n\nWe report the main results of the ARC benchmarks in Table~\\ref{tab:eval_arc} that compare GLM-5 with GLM-4.7, DeepSeek-V3.2~\\cite{liu2025deepseek}, Kimi-K2.5~\\cite{team2026kimi}, Claude Opus 4.5~\\cite{opus4.5}, Gemini 3 Pro~\\cite{gemini3}, and GPT-5.2 (xhigh)~\\cite{gpt-5.2}. \nIn general, GLM-5 delivers a significant improvement over GLM-4.7 and achieves state-of-the-art performance among open-source models, narrowing the gap to proprietary models such as Claude Opus 4.5. Evaluation details can be found at Section~\\ref{sec:eval_details_arc}.\n\n\\begin{table}[htbp]\n\\centering\n\\caption{\\label{tab:eval_arc} Comparison between GLM-5 and open-source/proprietary models. Results marked with * are from the full set of HLE. Results marked with $^\\dagger$ are evaluated on a verified version of Terminal-Bench 2.0, fixing some ambiguous instructions. The GDPval-AA Elo scores are recorded on 15th Feb., 2026. The highest score for each benchmark is \\textbf{bolded}, and the second highest is \\underline{underlined}.}\n\\setlength{\\tabcolsep}{3pt}\n\\renewcommand{\\arraystretch}{1.15}\n\\begin{tabular}{lccccccc}\n\\toprule\n& \\textbf{GLM-5} & \\textbf{GLM-4.7} & \\begin{tabular}[c]{@{}c@{}}\\textbf{DeepSeek}\\\\\\textbf{-V3.2}\\end{tabular} & \\begin{tabular}[c]{@{}c@{}}\\textbf{Kimi}\\\\\\textbf{K2.5}\\end{tabular} & \\begin{tabular}[c]{@{}c@{}}\\textbf{Claude}\\\\\\textbf{Opus 4.5}\\end{tabular} & \\begin{tabular}[c]{@{}c@{}}\\textbf{Gemini}\\\\\\textbf{3 Pro}\\end{tabular} & \\begin{tabular}[c]{@{}c@{}}\\textbf{GPT-5.2}\\\\\\textbf{(xhigh)}\\end{tabular} \\\\\n\\midrule\n\\textit{Reasoning \\& General} & & & & & & & \\\\\nHLE & 30.5 & 24.8 & 25.1 & 31.5 & 28.4 & \\textbf{37.2} & \\underline{35.4} \\\\\nHLE (w/ Tools) & \\underline{50.4} & 42.8 & 40.8 & \\textbf{51.8} & 43.4* & 45.8* & 45.5* \\\\\nAIME 2026 I & 92.7 & \\underline{92.9} & 92.7 & 92.5 & \\textbf{93.3} & 90.6 & - \\\\\n\nHMMT Feb. 2025 & \\underline{97.9} & 97.1 & 92.5 & 95.4 & 92.9 & 97.3 & \\textbf{99.4} \\\\\nHMMT Nov. 2025 & \\underline{96.9} & 93.5 & 90.2 & 91.1 & 91.7 & 93.0 & \\textbf{97.1} \\\\\nIMO-AnswerBench & 82.5 & 82.0 & 78.3 & 81.8 & 78.5 & \\underline{83.3} & \\textbf{86.3} \\\\\nGPQA-Diamond & 86.0 & 85.7 & 82.4 & 87.6 & 87.0 & \\underline{91.9} & \\textbf{92.4} \\\\\nLongBench v2 & \\underline{64.5} & 59.1 & 59.8 & 61.0 & 64.4 & \\textbf{68.2} & 59.8 \\\\\n\\midrule\n\\textit{Coding} & & & & & & & \\\\\nSWE-bench Verified & 77.8 & 73.8 & 73.1 & 76.8 & \\textbf{80.9} & 76.2 & \\underline{80.0} \\\\\nSWE-bench Multilingual & \\underline{73.3} & 66.7 & 70.2 & 73.0 & \\textbf{77.5} & 65.0 & 72.0 \\\\\n\\begin{tabular}[c]{@{}l@{}}Terminal-Bench 2.0\\\\{\\small \\ (Terminus-2)}\\end{tabular} & \\makecell[c]{\\underline{56.2} /\\\\ 60.7$^\\dagger$} & 41.0 & 39.3 & 50.8 & \\textbf{59.3} & 54.2 & 54.0 \\\\\n\\begin{tabular}[c]{@{}l@{}}Terminal-Bench 2.0\\\\{\\small \\ (Claude Code)}\\end{tabular} & \\makecell[c]{\\underline{56.2} /\\\\ 61.1$^\\dagger$} & 32.8 & 46.4 & - & \\textbf{57.9} & - & - \\\\\nCyberGym & \\underline{43.2} & 23.5 & 17.3 & 41.3 & \\textbf{50.6} & 39.9 & - \\\\\n\\midrule\n\\textit{Agentic} & & & & & & & \\\\\nBrowseComp & \\textbf{62.0} & 52.0 & 51.4 & \\underline{60.6} & 37.0 & 37.8 & - \\\\\n\\begin{tabular}[c]{@{}l@{}}BrowseComp\\\\{\\small \\ (w/ Context Manage)}\\end{tabular} & \\textbf{75.9} & 67.5 & 67.6 & \\underline{74.9} & 57.8 & 59.2 & 65.8 \\\\\nBrowseComp-ZH & \\underline{72.7} & 66.6 & 65.0 & 62.3 & 62.4 & 66.8 & \\textbf{76.1} \\\\\n$\\tau^2$-Bench & 89.7 & 87.4 & 85.3 & 80.2 & \\textbf{91.6} & \\underline{90.7} & 85.5 \\\\\nMCP-Atlas (Public Set) & \\underline{67.8} & 52.0 & 62.2 & 63.8 & 65.2 & 66.6 & \\textbf{68.0} \\\\\nTool-Decathlon & 39.2 & 23.8 & 35.2 & 27.8 & \\underline{43.5} & 36.4 & \\textbf{46.3} \\\\\nVending-Bench 2 & \\$4,432 & \\$2,377 & \\$1,034 & \\$1,198 & \\underline{\\$4,967} & \\textbf{\\$5,478} & \\$3,591 \\\\\nGDPval-AA Elo & \\underline{1,409} & 1,198 & 1,195 & 1,288 & 1,400 & 1,201 & \\textbf{1,462} \\\\\n\\bottomrule\n\\end{tabular}\n\\end{table}\n\n\\subsubsection{Evaluation of Reasoning and General Benchmarks}\n\nFor reasoning and general benchmarks, \nHumanity’s Last Exam (HLE)~\\cite{phan2025humanity}, AIME 2026, HMMT 2025, IMO-AnswerBench~\\cite{luong2025towards}, GPQA-Diamond~\\cite{rein2024gpqa}, and LongBench v2~\\cite{bai-etal-2025-longbench} are evaluated. \nFor HLE, only the text-based subset is evaluated, and GPT-5.2 (medium) is used as the judge model. Most reasoning tasks are evaluated with a maximum generation length of 131,072 tokens, while 202,752 maximum tokens are used for HLE-with-tools. \n\nFrom Table~\\ref{tab:eval_arc}, GLM-5 achieves comparable performance on reasoning tasks to the strong open-source baseline, Kimi-K2.5. \nCompared to proprietary models, GLM-5 outperforms Claude Opus 4.5 and Gemini 3 Pro on the HLE (with tools). \nGLM-5 also achieves significant improvements on the HLE benchmark (both with and without tools) compared to its predecessor, GLM-4.7. \nOn the HMMT Feb./Nov. 2025 benchmarks, GLM-5 gets better performance than Claude Opus 4.5 and Gemini 3 Pro. \nGLM-5 also makes significant progress on the long-context task, as evidenced by achieving the highest score on the long-context reasoning benchmark LongBench v2, second only to Gemini 3 Pro.\n\n\\subsubsection{Evaluation of Coding Benchmarks}\n\nFor coding benchmarks, we evaluate LLMs on SWE-bench Verified~\\cite{jimenez2023swe}, SWE-bench Multilingual~\\cite{yang2025swemulti}, Terminal Bench 2.0~\\cite{tbench_2025}, and CyberGym~\\cite{wang2025cybergym}. \nFor SWE-bench Verified \\& Multilingual, we use the OpenHands framework using a tailored instruction prompt for GLM-5. \nFor Terminal-Bench 2.0, two agent frameworks (i.e., Terminus-2 and Claude Code) are used, and we also report the performance on a verified Terminal-Bench 2.0 that resolves some ambiguous instructions\\footnote{More information can be found in \\url{https://huggingface.co/datasets/zai-org/terminal-bench-2-verified}}. \nThe CyberGym benchmark is evaluated in Claude Code 2.1.18.\n\nFrom Table~\\ref{tab:eval_arc}, GLM-5 achieves SOTA performance on coding benchmarks among open-source LLMs. \nCompared to proprietary LLMs, GLM-5 performs better than Gemini 3 Pro on SWE-bench Verified, and also beats Gemini 3 Pro and GPT-5.2 (xhigh) on SWE-bench Multilingual. \nOn Terminal-Bench 2.0, GLM-5 achieves comparable results to Claude Opus 4.5 and even better results when fixing ambiguous instructions for this benchmark. \nTo demonstrate the generalization of coding abilities, we evaluate on Terminal Bench 2.0 with two agent frameworks, and GLM-5 shows consistent performance across both frameworks. \nOn the cybersecurity coding benchmark (i.e., CyberGym), GLM-5 makes a significant improvement over GLM-4.7, second only to Claude Opus 4.5. \n\n\\subsubsection{Evaluation of Agentic Abilities}\n\nFor agentic benchmarks, we evaluate GLM-5 and frontier models on BrowseComp~\\cite{wei2025browsecomp}, BrowseComp-ZH~\\cite{zhou2025browsecompzh}, $\\tau^2$-Bench~\\cite{barres2025tau2}, MCP-Atlas~\\cite{bandi2026mcp}, Tool-Decathlon~\\cite{li2025tool}, Vending-Bench 2~\\cite{backlund2025vending}, and GDPval-AA~\\cite{patwardhan2025gdpval}. \nBrowseComp measures how language agents solve challenging problems by browsing the web, and BrowseComp-ZH mainly targets the Chinese web. \nWe use a discard-all strategy as context management for BrowseComp, which is the same as DeepSeek-V3.2, and Kimi K2.5. \n$\\tau^2$-Bench evaluates the ability of conversational agents in a dual-control environment. We add a small prompt adjustment for Retail and Telecom to avoid failures caused by premature user termination (see \\ref{sec:tau_user_prompt}). For Airline, we apply the domain fixes proposed in the Claude Opus 4.5 system card~\\cite{opus4.5} to obtain more accurate results.\nMCP-Atlas is a real-world tool-use benchmark that assesses how LLMs perform in multi-step workflows, given Model Context Protocol (MCP) servers. For fair comparison, we re-evaluate all models on the 500-task public set and extend the timeout from 4 minutes to 10 minutes per task to avoid task failures due to deployment conditions. We use Gemini 3 Pro as the judge model for MCP-Atlas. \nTool-Decathlon is also a tool-use benchmark but targets real-world, long-horizon tasks. \nVending-Bench 2 measures the agentic ability of LLMs in a business scenario over long-time horizons within a simulated environment, which adds more real-world factors to the predecessor Vending-Bench. \nGDPval focuses on how AI agents perform on economically valuable tasks. \n\nFrom Table~\\ref{tab:eval_arc}, GLM-5 improves significantly over agentic benchmarks compared to GLM-4.7. \nOn BrowseComp, GLM-5 achieves SOTA performance among the frontier LLMs in both with and without context management. \nOn BrowseComp-ZH, GLM-5 also beats Claude Opus 4.5 and Gemini 3 Pro. \nFor the three tool-use agentic tasks (i.e, $\\tau^2$-Bench, MCP-Atlas, and Tool-Decathlon), GLM-5 achieves comparable performance to Claude Opus 4.5, which shows the strong tool-use abilities of GLM-5. \nThe performance of GLM-5 on Vending-Bench 2 (i.e., \\$4,432) further demonstrates the long-horizon ability of the business task. \nIn economic scenarios, GLM-5 performs better than Claude Opus 4.5 on GDPval-AA, second only to GPT-5.2 (xhigh). \n\n\\subsection{Evaluation of Real-world Agentic Engineering Experience}\n\nReal-world experience matters more than leaderboards. We upgraded our internal CC-Bench to CC-Bench-V2 to evaluate whether the model can correctly complete end-to-end tasks in realistic agentic engineering environments across frontend, backend, and long-horizon tasks. CC-Bench-V2 removes human labeling entirely and is fully automated via Claude Code and other agent harnesses with unit tests and Agent-as-a-Judge techniques.\n\n\\textbf{Frontend.} We use a pipeline to first build the frontend projects generated by the agent and check for any syntax, dependency, and compatibility errors. Then we use Agent-as-a-Judge to validate end-to-end correctness by simulating user interactions via a GUI agent equipped with Playwright and bash tools.\n\n\\textbf{Backend.} Tasks are drawn from real-world open-source projects in C++, Rust, Go, Java, TypeScript, and Python, spanning feature implementation, bug fixes, regression repair, and performance optimization. Every change must pass the full unit tests within realistic engineering constraints.\n\n\\textbf{Long-horizon.} We first evaluate the model's information-seeking ability on large codebases, a prerequisite for locating the right files and understanding project context as a human developer would. We then assess end-to-end correctness through multi-step chained tasks constructed by mining merged Pull Requests with extensive commit histories and clustering their commits into coherent task chains. The agent executes these chains sequentially, testing its ability to maintain context and resolve dependencies between stages. Evaluation combines unit tests with Agent-as-a-Judge to verify both functional correctness and semantic adherence.\n\n\\begin{table}[h!]\n\\centering\n\\caption{CC-Bench-V2 evaluation results across frontend, backend, and long-horizon tasks. \\textbf{BSR}: Build Success Rate; \\textbf{ISR}: Instance Success Rate; \\textbf{CSR}: Check-item Success Rate.}\n\\label{tab:cc-bench-v2}\n\\begin{tabular}{llcccc}\n\\toprule\n\\textbf{Category} & \\textbf{Task} & \\textbf{Metric} & \\textbf{GLM-5} & \\textbf{GLM-4.7} & \\textbf{Claude Opus 4.5} \\\\\n\\midrule\n\\multirow{6}{*}{Frontend}\n & HTML & ISR & 38.9 & 35.4 & \\textbf{52.2} \\\\\n & & CSR & 76.3 & 64.9 & \\textbf{82.2} \\\\[2pt]\n & React & ISR & 34.6 & 17.2 & \\textbf{39.7} \\\\\n & & CSR & \\textbf{71.0} & 49.4 & 70.7 \\\\[2pt]\n & Vue & ISR & 32.7 & 24.5 & \\textbf{46.9} \\\\\n & & CSR & \\textbf{77.1} & 53.8 & 74.3 \\\\\n\\midrule\n\\multirow{4}{*}{Build}\n & React & BSR & \\textbf{100} & 65.0 & 95.0 \\\\\n & Vue & BSR & \\textbf{100} & 70.0 & \\textbf{100} \\\\\n & Svelte & BSR & \\textbf{100} & 60.0 & 90.0 \\\\\n & Next.js & BSR & 95.0 & 70.0 & 80.0 \\\\\n\\midrule\nBackend & Engineering & Pass@1 & 25.8 & 19.6 & \\textbf{26.9} \\\\\n\\midrule\n\\multirow{2}{*}{Long-horizon}\n & Repo Exploration & Pass@1 & \\textbf{65.6} & 47.8 & 64.5 \\\\\n & Chained Tasks & Pass@1 & 52.3 & 43.0 & \\textbf{61.6} \\\\\n\\bottomrule\n\\end{tabular}\n\\end{table}\n\n\\subsubsection{Frontend Evaluation -- Agent-as-a-Judge}\n\nWe develop a comprehensive automated evaluation benchmark specifically designed for frontend development scenarios. This benchmark covers a diverse range of applications that developers routinely build, including landing pages, management dashboards, data visualization, graphics and animations, online productivity tools, interactive games, and form-driven workflows, across mainstream technology stacks including HTML, React, Vue, Svelte, and Next.js.\n\nEach test case consists of a \\textit{Task} containing multiple concrete and implementable specifications, paired with a \\textit{Checklist} where each check-item is directly derived from the corresponding specifications. \nThe evaluation process follows a two-stage pipeline: 1) \\textbf{Static Verification}: We first verify whether the generated code can successfully build and run. 2) \\textbf{Agent-as-a-Judge}: For code that executes correctly, we employ a GUI agent to simulate human testing behavior to interactively verify each check item and assign scores based on the fulfillment of requirements. \nWe define the following metrics: \\textit{Build Success Rate (BSR)} measures the ratio of projects that successfully initialize and run. \\textit{Instance Success Rate (ISR)} measures the ratio of projects that pass all associated specifications. \\textit{Check-item Success Rate (CSR)} measures the fine-grained completion rate across all check-items. \nMore details on the data distribution and the construction and validation process are in Appendix~\\ref{sec:frontend_eval_data_cons}.\n\n\\begin{figure}[h!]\n \\centering\n \\includegraphics[width=1\\linewidth]{figures/agent-as-judge.pdf}\n \\vspace{-25mm}\n \\caption{Agent-as-a-Judge evaluation pipeline. Each generated frontend project is first built to verify static correctness. Successfully built instances are then interactively tested by an autonomous Judge Agent, which determines the functional correctness of each check item.}\n \\label{fig:agent-as-judge}\n\\end{figure}\n\n\\paragraph{Agent-as-a-Judge.}\nFrontend correctness is inherently visual and interactive, i.e., bugs often surface only when a user clicks a button or resizes a window, making static analysis and fixed test suites insufficient. We therefore introduce Agent-as-a-Judge (Figure~\\ref{fig:agent-as-judge}): each generated project is deployed in a Docker container and built to verify static correctness. Successfully built instances are then handed to an autonomous Judge Agent (Claude Code with Claude Sonnet 4.5, equipped with Playwright MCP tool) that operates in closed-loop cycles: for each check-item, the agent reads source code, interacts with the live UI (clicks, keystrokes, screenshots), inspects terminal output, and renders a pass/fail verdict.\n\nTo validate reliability, we compare Agent-as-a-Judge verdicts against independent human expert judgments along two dimensions. For \\emph{point-wise consistency}, we sampled 130 check-items, had human experts score each independently, and compared against the agent's verdicts: the two agree on 94\\% of items, with disagreements concentrated on subjective visual-quality criteria rather than functional specifications. For \\emph{ranking consistency}, we evaluated 8 frontier models (Claude Sonnet 4.5, Claude Opus 4.5, Gemini 3 Pro, GLM-4.7, DeepSeek-V3.2, etc.) using both the automated framework and human experts. The resulting model rankings achieve a Spearman correlation of 85.7\\%, indicating a strong positive correlation.\n\nAs shown in Table~\\ref{tab:cc-bench-v2}, GLM-5 achieves 98.0\\% BSR and is competitive with Claude Opus 4.5 in CSR, yet a notable ISR gap persists in all three stacks, indicating that GLM-5 meets most individual requirements but still falls short of Claude Opus 4.5 in completing an entire task end-to-end.\n\n\\subsubsection{Backend Evaluation}\n\nBackend evaluation measures whether a coding agent can make correct, test-passing modifications to real-world server-side codebases under realistic engineering constraints. We curate 85 tasks spanning six languages (Python, Go, C++, Rust, Java, and TypeScript) covering domains such as search engines, database engines, web frameworks, AI inference services, knowledge management systems, and standalone algorithmic and systems-programming challenges. Task types include feature implementation, bug fixing, regression repair, and performance optimization, reflecting the diversity of day-to-day backend development.\n\nTo enable fully automated evaluation, each task is equipped with human-crafted unit tests (5--10 per task) that verify both functional correctness and edge-case handling. Tasks are packaged in a terminal-bench style: each runs inside a Docker container initialized from the project's actual build environment, and the agent receives a natural-language problem statement describing the required change. We report Pass@1, where a task is considered solved only if all its associated unit tests pass. The strict all-or-nothing criterion makes this benchmark particularly challenging: GLM-5 and Claude Opus 4.5 perform comparably (Table~\\ref{tab:cc-bench-v2}), both significantly ahead of GLM-4.7.\n\n\\subsubsection{Long-horizon Evaluation}\n\nLong-horizon evaluation targets the capabilities that distinguish production-grade agentic engineering from single-turn vibe coding: navigating massive codebases and executing multi-step development where each action reshapes the context for subsequent ones. We decompose this into two complementary tasks.\n\n\\textbf{Large Repo Exploration.} A prerequisite for any non-trivial coding task is the ability to locate the right source files in a large, unfamiliar repository. We construct an automated benchmark over real high-star GitHub repositories containing tens of thousands of files. Each question is phrased in natural, user-facing language at the level of business semantics, strictly avoiding any mention of filenames, class names, or function names. Moreover, questions require one or two hops of logical reasoning from the user-facing description to the actual implementation---for instance, a question about misaligned lip-sync in a generated video maps to a parameter-tuning block inside a video generation backend. Target files are selected to maximize navigation difficulty: they reside at least three directory levels deep, carry opaque names that resist keyword-based search, implement unique functionality not duplicated elsewhere in the repository, and lie outside its main feature surface. We report Pass@1 averaged over three runs, where a question is considered solved if the agent successfully reads the target file during exploration. In this task, GLM-5 outperforms Claude Opus 4.5 (Table~\\ref{tab:cc-bench-v2}), both far ahead of GLM-4.7. The result suggests that effective repo exploration depends less on raw code generation ability and more on strategic search, i.e., iteratively narrowing the file space via directory-level reasoning and semantic association, where GLM-5's training on agentic tool-use trajectories provides a clear advantage.\n\n\\textbf{Multi-step Chained Tasks.} Mainstream coding benchmarks such as SWE-bench reduce evaluation to single-commit, isolated edits, and therefore cannot assess an agent's ability to perform incremental development where each step alters the codebase state for subsequent steps. To address this, we construct a long-horizon benchmark by mining merged Pull Requests from high-quality repositories and assembling task chains via the following pipeline:\n\n\\begin{enumerate}[leftmargin=1.5em,itemsep=2pt]\n \\item \\textbf{PR Filtering.} Retain only merged PRs that include tests, contain 3--15 commits, and follow a linear (non-merge) history.\n \\item \\textbf{Semantic Grouping.} An LLM scores pairwise semantic relatedness between adjacent commits; dynamic programming finds the optimal partition into coherent task groups that maximize intra-group coherence while preserving commit order.\n \\item \\textbf{Patch Triage.} Each task's cumulative diff is split into three categories: \\emph{golden patch} (core code the agent must produce), \\emph{test patch} (verification tests), and \\emph{auto-apply patch} (configuration and fixtures applied automatically).\n \\item \\textbf{Problem Statement Generation.} An LLM generates a natural-language problem statement for each task from its patch and commit messages.\n \\item \\textbf{Task Classification.} Tasks are automatically classified (feature / bug-fix / refactor / test / config) and evaluated along three axes: error elimination, critical-path accuracy, and test passage.\n \\item \\textbf{Environment Validation.} Docker environments are constructed, and golden patches are applied to verify zero regression across the entire chain.\n\\end{enumerate}\n\nGiven a chain of $K$ tasks, the agent starts from the base commit and works sequentially: after completing task $k$, its changes are committed, and the auto-apply patch for task $k{+}1$ is applied, so the codebase state evolves cumulatively. Evaluation checks each commit in turn and cumulatively applies test patches from tasks $1$ through $k$ before running the full test suite, catching both failures on the current task and regressions on earlier ones. We report Pass@1 on individual tasks. This chained and state-recursive design directly evaluates the long-range context tracking, planning, and incremental development abilities that single-commit benchmarks leave untested. As Table~\\ref{tab:cc-bench-v2} shows, GLM-5 improves substantially over GLM-4.7, but a significant gap to Claude Opus 4.5 remains. This is because errors are compounded across the chain: a suboptimal edit in one task can silently break tests in subsequent tasks. Narrowing this gap will require advances in long-context consistency and long-horizon self-correction, both active areas of our ongoing research.\n\n\\subsubsection{Evaluation on evolving SWE tasks}\nWe evaluate on SWE-rebench~\\cite{badertdinov2025swe} because SWE-bench Verified is a static, public, human-validated test set and released for more than 2 years. In contrast, SWE-rebench is built on an automated pipeline that continuously mines fresh, real GitHub issue-fixing tasks, enabling decontaminated, time-robust evaluation that better measures generalization to new software engineering problems rather than performance on a static benchmark. Table~\\ref{table:swe_rebench} shows the official performance of GLM-5 on SWE-rebench and we observe that GLM-5 can effectively generalize to new SWE problems.\n\n\\begin{table}[t]\n\\centering\n\\caption{Performance on SWE-rebench, January 2026.}\n\\begin{tabular}{lccc}\n\\toprule\nModel & Resolved Rate (\\%) & Resolved Rate SEM ($\\pm$, \\%) & Pass@5 (\\%) \\\\\n\\midrule\nClaude Opus 4.6 & 52.9\\% & 1.06\\% & 70.8\\% \\\\\nGPT-5.2 (xhigh) & 51.7\\% & 1.21\\% & 58.3\\% \\\\\nClaude Sonnet 4.5 & 47.1\\% & 1.69\\% & 60.4\\% \\\\\nGemini 3 Pro & 46.7\\% & 2.04\\% & 58.3\\% \\\\\nClaude Opus 4.5 & 43.8\\% & 0.93\\% & 58.3\\% \\\\\nGLM-5 & 42.1\\% & 1.21\\% & 50.0\\% \\\\\nGLM-4.7 & 41.3\\% & 2.12\\% & 56.3\\% \\\\\nKimi K2.5 & 37.9\\% & 1.21\\% & 50.0\\% \\\\\n\\bottomrule\n\\end{tabular}\n\\label{table:swe_rebench}\n\\end{table}\n\n\\subsection{Evaluation of Real-world General Abilities}\n\n\\begin{figure}\n \\centering\n \\includegraphics[width=1\\linewidth]{figures/glm5_realscene_comparison.pdf}\n \\caption{Performance comparison between GLM-4.7 and GLM-5 across five real-world general ability domains.}\n \\label{fig:glm5_realscene}\n\\end{figure}\n\nWhile standardized academic benchmarks provide useful signals, they do not fully capture how models are used in practice. To recognize this gap, we evaluate GLM-5 on a set of real-world general abilities derived from high-frequency user interaction patterns observed in deployment settings. These abilities include {machine translation, multilingual dialogue, instruction following, world knowledge, and tool-calling}.\n\nUnlike traditional benchmark-centric evaluation, our goal is to measure improvements that directly translate into user-perceived quality gains. For each capability, we adopt a combination of internal human evaluation, internal automated evaluation, external human assessment, and external automated benchmarks, ensuring both diagnostic granularity and cross-model comparability. When using external benchmarks, we prioritize datasets that reflect realistic interaction patterns rather than narrowly constructed test distributions.\n\nFigure~\\ref{fig:glm5_realscene} presents the comparative results between GLM-5 and GLM-4.7 across five real-world capability domains. Across all evaluated dimensions, GLM-5 shows consistent improvements in machine translation, multilingual dialogue, instruction following, world knowledge, and tool-calling.\n\nDetailed evaluation protocols and dataset descriptions for each ability are provided as follows. \n\n\\subsubsection{Machine Translation}\n\n\\paragraph{ZMultiTransBench.} \nThis internal dataset comprises 1,220 samples sourced from self-collected high-frequency translation scenarios, covering seven language pairs: Zh to Es (\\(300\\)), Ru (\\(250\\)), Fr (\\(220\\)), Ko (\\(200\\)), Ja (\\(150\\)), Ar (\\(50\\)), and De (\\(50\\)). All samples were curated, translated, and independently verified by graduate students with formal training in translation studies. The dataset emphasizes naturally occurring usage contexts rather than artificially constructed test cases. Evaluation is conducted using pairwise comparison against a fixed baseline response. Judgments are provided by an automated evaluator based on GPT-4.1, which assesses semantic fidelity, fluency, and overall translation quality.\n\n\\paragraph{MENT-SNS.}\nTo further evaluate robustness in linguistically challenging contexts, we adopt source sentences from MENT \\citep{tian2026ment}, comprising \\(753\\) English–Chinese sentence pairs across four domains: Social Network Services (SNS), Cross-Culture, Poetry, and Literature.\nThese domains are selected to stress-test translation under complex linguistic phenomena, including slang, homophonic wordplay, idiomatic expressions, historical references, and metaphorical language. Similar to ZMultiTransBench, all samples were curated and verified by professionally trained graduate students.\nEvaluation follows the same pairwise comparison protocol against a baseline response, with GPT-4.1 serving as the automated judge model.\n\n\\subsubsection{Multi-lingual Dialogue}\n\\paragraph{LMArena.}\nWe report Elo ratings from the LMArena\\footnote{https://arena.ai/leaderboard/text}\n, which are derived from large-scale, community-submitted pairwise comparisons. These ratings reflect relative model preference in open-ended dialogue settings and provide an external signal of conversational performance. \n\n\\paragraph{ZMultiDialBench.}\nIn addition to the public leaderboard, we also conduct human evaluation on ZMultiDialBench, an internal multilingual dialogue benchmark. The dataset consists of 141 curated instances spanning diverse dialogue categories. Samples were collected from high-quality conversational data contributed by native-speaking annotators across multiple countries, as well as from challenging failure cases reported by online users. \nHuman annotators assigned pointwise scores on a 1–10 scale to anonymized model responses according to category-specific, standardized evaluation criteria. \n\n\\subsubsection{Instruction Following}\n\\paragraph{IF-Badcase.} IF-Badcase is an internal benchmark constructed from instruction-following \\textbf{failure cases} reported by real users in production settings. The dataset is designed to evaluate strict adherence to realistic, multi-constraint instructions, emphasizing procedural accuracy, logical consistency, and rigid formatting requirements.\nEvaluation is conducted using a detailed checklist-based protocol that verifies compliance with explicit constraints, including ordered steps, rule-based conditions, and structural specifications. All samples were annotated, reviewed, and iteratively filtered by human experts, resulting in a curated set of 450 test instances.\n\n\\paragraph{IF-Bench~\\citep{ifbench}.} IF-Bench evaluates LLMs on their ability to adhere to complex, objective constraints, such as specific formatting rules, length limits, and content restrictions. It provides a quantitative measure of precise instruction-following capabilities, focusing on verifiable compliance rather than open-ended generation quality.\n\n\\paragraph{MultiChallenge~\\citep{multichallenge}.} MultiChallenge examines LLMs via realistic, multi-turn conversational scenarios. It targets complex interactions requiring accurate instruction-following, context allocation, and in-context reasoning.\n\n\\subsubsection{World Knowledge}\n\n\\paragraph{SimpleQA~\\citep{simpleQA}.} SimpleQA measures short-form factuality using challenging questions with single, indisputable answers. It evaluates a model's calibration by classifying responses as correct, incorrect, or not attempted, prioritizing accuracy over generation length.\n\n\\paragraph{Chinese SimpleQA~\\citep{chinesesimpleQA}.} Adapting the SimpleQA methodology to the Chinese context, this benchmark evaluates factuality across six major domains and 99 subtopics. It utilizes high-quality, static, short-answer questions designed for reliable, automated grading to assess the knowledge accuracy of LLMs.\n\n\\subsubsection{Tool Calling}\n\\paragraph{ToolCall-Badcase.} ToolCall-Badcase is an internal benchmark derived from \\textbf{failure cases} in tool invocation scenarios reported by users in production environments. Each instance is associated with a verifiable ground-truth tool call, enabling objective evaluation of both tool selection and argument correctness.\nEvaluation assesses whether the model (1) invokes the correct tool and (2) provides correctly structured and semantically accurate arguments. \nAll samples underwent multiple rounds of review, rewriting, and validation to remove ambiguity and ensure evaluability. The resulting dataset consists of 200 curated test cases that reflect realistic tool-calling abilities.\n\\section{Conclusion}\n\nIn this report, we have introduced GLM-5, a next-generation foundation model that fundamentally bridges the gap between high-performance reasoning and extreme computational efficiency. By transitioning from the paradigm of ``vibe coding'' to true ``agentic engineering'', GLM-5 demonstrates that open-weight models can now rival the capabilities of top-tier proprietary systems in complex, real-world workflows.\nGLM-5 represents a paradigm shift in practical AI utility. By open-sourcing the model, we aim to empower the community to move beyond static benchmarks and explore the frontiers of efficient, agentic general intelligence, fostering a new era where AI agents autonomously plan, implement, and iterate on complex tasks.\n\n\\section{Easter Eggs}\nThe \\textbf{``Pony Alpha''} experiment was indeed a pivotal moment for us. It was a bold decision to release GLM-5 anonymously on OpenRouter, but the results have been incredibly validating. By stripping away our brand name, we allowed the model's intrinsic capabilities to speak for themselves, ensuring the feedback we received was pure and unbiased.\nHere is a brief summary:\n\nWithin days, Pony Alpha became a sensation. Developers in the OpenRouter community began to notice its exceptional performance, particularly in complex coding tasks, agentic workflows, and roleplay scenarios.\n\nSpeculation was rampant, with many users guessing it was a leaked update from labs like Anthropic (Claude Sonnet 5), a secret Grok release, or DeepSeek V4. A preliminary statistic shows that 25\\% of the users guessed it was Claude Sonnet 5, 20\\% DeepSeek, 10\\% Grok, and the rest GLM-5.\n\nThe eventual confirmation that it was indeed our GLM-5 was a profound moment for us, effectively silencing doubts about whether Chinese LLMs could compete at the frontier level. \nThe success of Pony Alpha (GLM-5) is not just about raw benchmarks; it signifies a shift in our focus towards engineering-level reliability.\n\nThis anonymous release allowed us to transcend geopolitical biases. The community embraced the model because it worked.\nWhile we celebrate this success, we must remain pragmatic. The gap between open-weight models and the absolute proprietary frontier is narrowing, but the race is far from over. Our focus remains steadfast on pushing the boundaries of what is possible with scalable, efficient, and intelligent systems.\n\\newpage\n\\section{Contribution}\n\\label{sec:contribution}\n\\newcommand{\\cpara}[1]{~\\\\\\textbf{#1}~\\\\}\nContributors' names are listed in alphabetical order by first name. \n\n\\cpara{Core Contributors}\n\nChendi Ge, Chenghua Huang, Chengxing Xie, Chenzheng Zhu, Congfeng Yin, Cunxiang Wang, Gengzheng Pan, Hao Zeng, Haoke Zhang, Haoran Wang, Huilong Chen, Jiajie Zhang, Jian Jiao, Jiaqi Guo, Jingsen Wang, Jingzhao Du, Jinzhu Wu, Kedong Wang, Lei Li, Lin Fan, Lucen Zhong, Mingdao Liu, Mingming Zhao, Pengfan Du, Qian Dong, Rui Lu, Shuang Li (\\begin{CJK*}{UTF8}{gbsn}李爽\\end{CJK*}), Shulin Cao, Song Liu, Ting Jiang, Xiaodong Chen, Xiaohan Zhang, Xuancheng Huang, Xuezhen Dong, Yabo Xu, Yao Wei, Yifan An, Yilin Niu, Yitong Zhu, Yuanhao Wen, Yukuo Cen, Yushi Bai, Zhongpei Qiao, Zihan Wang, Zikang Wang, Zilin Zhu, Ziqiang Liu, Zixuan Li\n\n\\cpara{Contributors}\n\nBojie Wang, Bosi Wen, Can Huang, Changpeng Cai, Chao Yu, Chen Li, Chengwei Hu, Chenhui Zhang, Dan Zhang, Daoyan Lin, Dayong Yang, Di Wang, Ding Ai, Erle Zhu, Fangzhou Yi, Feiyu Chen, Guohong Wen, Hailong Sun, Haisha Zhao, Haiyi Hu, Hanchen Zhang, Hanrui Liu, Hanyu Zhang, Hao Peng, Hao Tai, Haobo Zhang, He Liu, Hongwei Wang, Hongxi Yan, Hongyu Ge, Huan Liu, Huanpeng Chu, Jia'ni Zhao, Jiachen Wang, Jiajing Zhao, Jiamin Ren, Jiapeng Wang, Jiaxin Zhang, Jiayi Gui, Jiayue Zhao, Jijie Li, Jing An, Jing Li, Jingwei Yuan, Jinhua Du, Jinxin Liu, Junkai Zhi, Junwen Duan, Kaiyue Zhou, Kangjian Wei, Ke Wang, Keyun Luo, Laiqiang Zhang, Leigang Sha, Liang Xu, Lindong Wu, Lintao Ding, Lu Chen, Minghao Li, Nianyi Lin, Pan Ta, Qiang Zou, Rongjun Song, Ruiqi Yang, Shangqing Tu, Shangtong Yang, Shaoxiang Wu, Shengyan Zhang, Shijie Li, Shuang Li (\\begin{CJK*}{UTF8}{gbsn}李泷\\end{CJK*}), Shuyi Fan, Wei Qin, Wei Tian, Weining Zhang, Wenbo Yu, Wenjie Liang, Xiang Kuang, Xiangmeng Cheng, Xiangyang Li, Xiaoquan Yan, Xiaowei Hu, Xiaoying Ling, Xing Fan, Xingye Xia, Xinyuan Zhang, Xinze Zhang, Xirui Pan, Xu Zou, Xunkai Zhang, Yadi Liu, Yandong Wu, Yanfu Li, Yidong Wang, Yifan Zhu, Yijun Tan, Yilin Zhou, Yiming Pan, Ying Zhang, Yinpei Su, Yipeng Geng, Yong Yan, Yonglin Tan, Yuean Bi, Yuhan Shen, Yuhao Yang, Yujiang Li, Yunan Liu, Yunqing Wang, Yuntao Li, Yurong Wu, Yutao Zhang, Yuxi Duan, Yuxuan Zhang, Zezhen Liu, Zhengtao Jiang, Zhenhe Yan, Zheyu Zhang, Zhixiang Wei, Zhuo Chen, Zhuoer Feng, Zijun Yao, Ziwei Chai, Ziyuan Wang, Zuzhou Zhang\n\n\\cpara{Tech Leads}\nAohan Zeng, Xin Lv, Zhenyu Hou, Zhengxiao Du, Qinkai Zheng, Bin Chen, Da Yin\n\n\\cpara{Advisors}\nJie Tang, Yuxiao Dong, Juanzi Li, Hongning Wang, Minlie Huang, Bin Xu\n\n\\section*{Acknowledgement}\n\nWe are grateful for all the support from our co-launch partners and community developers (Alphabetical order):\n\n\\paragraph{Open-source Communities:} Hugging Face, MLX, ModelScope, SGLang, Unsloth, vLLM, xLLM\n\n\\paragraph{Inference Providers:} Amazon Bedrock, Atlas Cloud, Baidu AI Cloud, Baseten, Cerebras, DeepInfra, Fireworks, FriendliAI, GMI Cloud, Google Cloud Vertex AI, Infinigence AI, Modal, Novita AI, Parasail, Phala, PPIO, SiliconFlow, StreamLake, Together AI, Venice, Weights \\& Biases\n\n\\paragraph{Applications:} CatPaw, Cline, CodeBuddy, CodeRider, Coze, Crush, Factory AI, Kilo Code, MonkeyCode, OpenClaw, OpenCode, Qoder, Roo Code, TRAE, Verdent AI, WPS, YouWare\n\n\\paragraph{AI Gateways:} AI Ping, EZmodel, iFlow, OpenRouter, Vercel, Yupp, ZenMux\n\n\\clearpage\n\n\\bibliographystyle{abbrv}\n\\bibliography{ref}\n\n\\clearpage\n\n\\appendix\n\n\\section{Hyper-Parameters}\n\nHyper-parameters related to the model architecture of GLM-5 are shown in \\Cref{tab:model_arch_glm45_glm5}.\n\nFor training, we follow the setting of GLM-4.5, including the Muon optimizer, cosine decay, and batch size warmup. The learning rate goes through a warmup stage from 0 to 2e-4, and a decaying stage to 4e-5 until the end of the pre-training stage. In the mid-training stage, the learning rate decreases linearly from 4e-5 to 1e-5. Other hyper-parameters are the same as those of GLM-4.5. For DSA warmup stage, the learning rate goes down from 5e-3 to 2e-4. For DSA sparse adaption stage, we use a constant learning rate of 1e-5.\n\n\\begin{table}[!ht]\n \\centering\n \\caption{Model architecture of GLM-4.5 and GLM-5. When counting parameters, for all models we include the parameters of MTP layers but not word embeddings and the output layer.}\n \\label{tab:model_arch_glm45_glm5}\n \\begin{tabular}{l|cc}\n \\toprule\n Model & \\textbf{GLM-4.5} & \\textbf{GLM-5} \\\\\n \\midrule\n \\# Total Parameters & 355B & 744B \\\\\n \\# Activated Parameters & 32B & 40B \\\\\n \\# Dense Layers & 3 & 3 \\\\\n \\# MoE Layers & 89 & 75 \\\\\n \\# MTP Layers & 1 & 1 \\\\\n Hidden Dim & 5120 & 6144 \\\\\n Dense Intermediate Dim & 12288 & 12288 \\\\\n MoE Intermediate Dim & 1536 & 2048 \\\\\n \\midrule\n QK Head Dim & 128 & 192 \\\\\n V Head Dim & 128 & 256 \\\\\n Q LoRA Dim & -- & 2048 \\\\\n KV LoRA Dim & -- & 512 \\\\\n \\# Attention Heads & 96 & 64 \\\\\n \\# Key-Value Heads & 8 & -- \\\\\n \\# Indexer Attn Heads & -- & 32 \\\\\n \\# Indexer Head Dim & -- & 128 \\\\\n \\midrule\n \\# Experts (total) & 160 & 256 \\\\\n \\# Routed Experts & 8 & 8 \\\\\n \\# Shared Experts & 1 & 1 \\\\\n \\midrule\n Vocabulary Size & 151552 & 154880 \\\\\n \\bottomrule\n \\end{tabular}\n\\end{table}\n\n\\section{Evaluation Details}\n\\subsection{Evaluation of Base Models}\nWe evaluate the base model of GLM-5 with English, Chinese, code, and math benchmarks in \\Cref{tab:base_benchmark}.\n\n\\begin{table}[!ht]\n \\centering\n \\caption{Comparison among GLM-5-Base, GLM-4.5-Base and other representative open-source base models.}\n \\resizebox{\\textwidth}{!}{\n \\begin{tabular}{lcccc|c}\n \\toprule\n & \\textbf{Benchmark (Metric)} & \\textbf{DeepSeek-V3} & \\textbf{Kimi-K2} & \\textbf{GLM-4.5} & \\textbf{GLM-5} \\\\\n & & \\textbf{Base} & \\textbf{Base} & \\textbf{Base} & \\textbf{Base} \\\\\n \\midrule\n \\multirow{3}{*}{} & Architecture & MoE & MoE & MoE & MoE \\\\\n & \\# Activated Params & 37B & 32B & 32B & 40B \\\\\n & \\# Total Params & 671B & 1043B & 355B & 744B \\\\\n \\midrule\n \\multirow{6}{*}{\\textbf{English}} & SimpleQA (EM) & 26.6 & 35.3 & 30.0 & 36.0 \\\\\n & BBH (EM) & 88.4 & 88.7 & 86.2 & 87.4 \\\\\n & MMLU (EM) & 87.2 & 87.8 & 86.1 & 88.3 \\\\\n & HellaSwag (EM) & 88.9 & 94.6 & 87.1 & 88.1 \\\\\n & PIQA (EM) & 84.7 & - & 85.3 & 84.6 \\\\\n & TriviaQA (EM) & 82.9 & 85.1 & 80.0 & 80.9 \\\\\n \\midrule\n \\multirow{2}{*}{\\textbf{Code}} & EvalPlus (Pass@1) & 65.6 & 80.3 & 78.1 & 87.0 \\\\\n & LiveCodeBench-Base (Pass@1) & 24.6 & 26.3 & 28.1 & 34.4 \\\\\n \\midrule\n \\multirow{2}{*}{\\textbf{Math}} & GSM8K (EM) & 87.6 & 92.1 & 79.4 & 68.8 \\\\\n & MATH (EM) & 62.6 & 70.2 & 61.0 & 56.4 \\\\\n \\midrule\n \\multirow{4}{*}{\\textbf{Chinese}} & CLUEWSC (EM) & 82.7 & - & 83.5 & 84.2 \\\\\n & C-Eval (EM) & 90.1 & 92.5 & 86.9 & 88.8 \\\\\n & C3 (EM) & 78.6 & - & 83.1 & 80.3 \\\\\n & Chinese-SimpleQA (EM) & 72.1 & 77.6 & 70.1 & 74.6 \\\\\n \\bottomrule\n \\end{tabular}\n }\n \\label{tab:base_benchmark}\n\\end{table}\n\n\\subsection{Evaluation of ARC Benchmarks}\n\\label{sec:eval_details_arc}\nHumanity’s Last Exam (HLE) \\& other reasoning tasks: We evaluate with a maximum generation length of $131,072$ tokens ($temperature=1.0, top\\_p=0.95, max\\_new\\_tokens=131072$). By default, we report the text-only subset; results marked with * are from the full set. We use GPT-5.2 (medium) as the judge model. For HLE-with-tools, we use a maximum context length of $202,752$ tokens.\n\nSWE-bench \\& SWE-bench Multilingual: We run the SWE-bench suite with OpenHands using a tailored instruction prompt. Settings: $temperature=0.7, top\\_p=0.95, max\\_new\\_tokens=16384$, with a 200K context window.\n\nBrowseComp: Without context management, we retain details from the most recent 5 turns. With context management, we use the same discard-all strategy as DeepSeek-V3.2 and Kimi K2.5.\n\nTerminal-Bench 2.0 (Terminus 2): We evaluate with the Terminus framework using $timeout=2h, temperature=0.7, top\\_p=1.0, max\\_new\\_tokens=8192$, with a 128K context window. Resource limits are capped at 16 CPUs and 32 GB RAM.\n\nTerminal-Bench 2.0 (Claude Code): We evaluate in Claude Code 2.1.14 (think mode) with $temperature=1.0, top\\_p=0.95, max\\_new\\_tokens=65536$. We remove wall-clock time limits, while preserving per-task CPU and memory constraints. We fix environment issues introduced by Claude Code and also report results on a verified Terminal-Bench 2.0 dataset that resolves ambiguous instructions (see: \\url{https://huggingface.co/datasets/zai-org/terminal-bench-2-verified}). Scores are averaged over 5 runs.\n\nCyberGym: We evaluate in Claude Code 2.1.18 (think mode, no web tools) with ($temperature=1.0, top\\_p=1.0, max\\_new\\_tokens=32000$) and a 250-minute timeout per task. Results are single-run Pass@1 over 1,507 tasks.\n\nMCP-Atlas: All models are evaluated in think mode on the 500-task public subset with a 10-minute timeout per task. We use Gemini 3 Pro as the judge model.\n\n$\\tau^2$-Bench: We add a small prompt adjustment in Retail and Telecom to avoid failures caused by premature user termination. For Airline, we apply the domain fixes proposed in the Claude Opus 4.5 system card.\n\nVending-Bench 2: Runs are conducted independently by Andon Labs\\footnote{\\url{https://andonlabs.com/evals/vending-bench-2}}.\n\n\\subsection{Optimized User Simulator for $\\tau^2$-Bench}\n\\label{sec:tau_user_prompt}\nWe add a small prompt adjustment in Telecom and Retail to avoid failures caused by premature user termination. The optimized prompts are shown in Figure~\\ref{fig:tau2-prompt-telecom} and Figure~\\ref{fig:tau2-prompt-retail}. These optimized prompts are integrated into the system prompt as follows:\n\n\\begin{tcolorbox}[left=0mm,right=0mm,top=0mm,bottom=0mm,boxsep=1mm,arc=0mm,boxrule=0pt, frame empty, breakable]\n\\begin{lstlisting}\nSYSTEM_PROMPT = \"\"\"\"\n{global_user_sim_guidelines}\n\n<scenario>\n{instructions}\n</scenario>\n\n{optimized_user_prompt}\n\"\"\"\".strip()\n\\end{lstlisting}\n\\end{tcolorbox}\n\n\\begin{tcolorbox}[left=0mm,right=0mm,top=0mm,bottom=0mm,boxsep=1mm,arc=0mm,boxrule=0pt, frame empty, breakable]\n\\footnotesize\n\\begin{lstlisting}\n# Note:\n- Do not generate the '###TRANSFER###' before agent clearly tells \"YOU ARE BEING TRANSFERRED TO A HUMAN AGENT. PLEASE HOLD ON.\".\n Example:\n Case1:\n - agent: \"Would you like me to transfer you to a human agent who can assist you with these options and help get your service restored?\"\n - user: \"Yes, please transfer me to a human agent.\".\n Case2:\n - user: \"YOU ARE BEING TRANSFERRED TO A HUMAN AGENT. PLEASE HOLD ON.\"\n - user: \"###TRANSFER###\"\n\\end{lstlisting}\n\\end{tcolorbox}\n\\noindent\\begin{minipage}{\\textwidth}\n\\captionof{figure}{The optimized user prompt for $\\tau^2$-Bench Telecom.}\n\\label{fig:tau2-prompt-telecom}\n\\end{minipage}\n\n\\begin{tcolorbox}[left=0mm,right=0mm,top=0mm,bottom=0mm,boxsep=1mm,arc=0mm,boxrule=0pt, frame empty, breakable]\n\\footnotesize\n\\begin{lstlisting}\n# Rules:\n- Just generate one line at a time to simulate the user's message.\n- Do not give away all the instruction at once. Only provide the information that is necessary for the current step.\n- Do not hallucinate information that is not provided in the instruction. Follow these guidelines:\n 1. If the agent asks for information NOT in the instruction:\n - Say you don't remember or don't have it\n - Offer alternative information that IS mentioned in the instruction\n 2. Examples:\n - If asked for order ID (not in instruction): \"Sorry, I don't remember the order ID, can you search for it? My name/email/phone number/zipcode is ...\"\n - If asked for email (not in instruction): \"I don't have my email handy, but I can give you my name and zip code which are...\"\n- Do not repeat the exact instruction in the conversation. Instead, use your own words to convey the same information.\n- Try to make the conversation as natural as possible, and stick to the personalities in the instruction.\n# Constraint Handling:\n- Provide requests strictly based on what is explicitly stated in the instruction.\n- Do not assume, extend, substitute, or generalize in any form.\n- Do not modify or relax constraints on:\n- Time / Date\n- Budget\n- Specific terms (e.g., \"same\" must not be replaced with \"similar\")\n- Core Rule: Any attribute NOT mentioned in the instruction can be either changed or kept the same\n- Examples:\n - If instruction says \"exchange red item to blue\": Only color must change, other attributes (size, material, etc.) are flexible\n - If instruction says \"exchange red item to blue, keep the same size\": Both color must change AND size must stay the same\n- Exception: Only follow additional constraints when explicitly stated in the instruction\n# Domain-Specific Rules:\n## For Retail scenarios:\n- Focus on product attributes and exchange/return processes as specified in instructions.\n- During confirmations: Always respond based strictly on the original instruction, never deviate to match agent's provided options. Restate your requirement from the instruction rather than selecting from agent's choices.\n - Example: If the agent provides specific options (A/B/C) but the instruction states a general requirement (e.g., \"same as pending order\"), always restate or confirm based on what the instruction says, not by directly selecting from the agent's provided options.\n# When NOT to finish the conversation:\n- Do not end until you have clearly and completely expressed all your requirements and constraints.\n- Do not end until the agent has completed all tasks mentioned in the instruction and verified no operations were missed.\n- Do not end if the agent's execution results do not match your expectations or are incorrect/incomplete.\n# When you CAN finish the conversation:\n- Only when all above conditions are satisfied AND all tasks are completed correctly.\n- OR when you have clearly expressed complete requirements but the system explicitly states it cannot complete them due to technical limitations - in this case, accept transfer to human.\n# How to finish the conversation:\n- If the agent has completed all tasks, generate the '###STOP###' token to end the conversation.\n# Note:\n- You should carefully check if the agent has completed all tasks mentioned in the instruction before generating '###STOP###'.\n\\end{lstlisting}\n\\end{tcolorbox}\n\\noindent\\begin{minipage}{\\textwidth}\n\\captionof{figure}{The optimized user prompt for $\\tau^2$-Bench Retail.}\n\\label{fig:tau2-prompt-retail}\n\\end{minipage}\n\n\\subsection{Evaluation of Real-world Agentic Engineering Experience}\n\n\\subsubsection{Frontend Evaluation}\n\n\\label{sec:frontend_eval_data_cons}\n\n\\paragraph{Data.}\nOur dataset encompasses seven distinct frontend scenarios designed to evaluate a model's engineering proficiency across diverse functional domains: Business Management Systems, Web Games, SVG/Canvas Rendering, Creative Tools \\& Editors, Showcase Pages, Forms \\& Tables and Data Visualization.\n\n\\begin{table}[h]\n\\centering\n\\small\n\\caption{Distribution of frontend application scenarios.}\n\\begin{tabular}{lp{7cm}rr}\n\\toprule\n\\textbf{Category} & \\textbf{Description} & \\textbf{\\# Tasks} & \\textbf{\\# Checkitems} \\\\\n\\midrule\nBusiness Systems & Enterprise/Personal data and process management. & 42 & 167 \\\\\nWeb Games & Interaction and entertainment-focused games. & 40 & 163\\\\\nSVG/Canvas & Graphics rendering and interactive visualizations. & 32 & 166\\\\\nCreative Tools & Content creation and online editing tools. & 28 & 160\\\\\nShowcase Pages & Visual expression and information presentation. & 27 & 115\\\\\nForms \\& Tables & Structured data entry and processing. & 26 & 93\\\\\nData Visualization & Graphical data expression and analysis. & 25 & 85\\\\\n\\bottomrule\n\\end{tabular}\n\n\\end{table}\n\n\\subparagraph{Data Distribution by Coding Languages}\nThe benchmark provides full coverage of three mainstream paradigms: Vanilla Web Stack (HTML/CSS/JS), React Component-based Framework, and the Vue 3 + Vite Progressive Solution.\n\\begin{table}[h]\n\\centering\n\\caption{Statistics of technology stacks and evaluation units.}\n\\begin{tabular}{llcc}\n\\toprule\n\\textbf{Category} & \\textbf{Description} & \\textbf{\\# Tasks} & \\textbf{\\# Checkitems} \\\\\n\\midrule\nHTML & Vanilla HTML/CSS/JS development. & 113 & 490 \\\\\nReact & Component-based framework development. & 58 & 249 \\\\\nVue & Vue 3 + Vite progressive solution. & 49 & 210 \\\\\n\\bottomrule\n\\end{tabular}\n\\end{table}\n\n\\subparagraph{Data sample}\nEach test case is composed of three components: the \\textbf{Task}, the \\textbf{Checklist}, and a \\textbf{Dedicated Environment}. Below is a representative example of a test case:\n\\begin{tcolorbox}[left=0mm,right=0mm,top=0mm,bottom=0mm,boxsep=1mm,arc=0mm,boxrule=0pt, frame empty, breakable]\n\\footnotesize\n\\begin{lstlisting}\n Task: Develop an online drawing tool that includes a brush, an eraser, a white canvas, and a save button.\n The brush color and thickness should be selectable via buttons on the left. Users can draw on the canvas by clicking and dragging the mouse.\n The eraser size should be selectable via buttons on the left. Users can erase content by clicking and dragging the mouse over the canvas.\n Once the drawing is complete, clicking the \"Save\" button should allow the user to save the image locally.\n Please implement this using the React framework in the current directory.\n\n Checklist:\n The user can select the brush color and thickness using the left-hand buttons, and drawing is functional via mouse click-and-drag on the canvas.\n The user can select the eraser size using the left-hand buttons, and erasing is functional via mouse click-and-drag on the canvas.\n Upon clicking the \"Save\" button, the generated image is successfully saved to the local machine.\n\\end{lstlisting}\n\\end{tcolorbox}\n\\noindent\\begin{minipage}{\\textwidth}\n\\end{minipage}\n\n\\subparagraph{\\textbf{Data Construction and Validation}}\n\nWe implement a rigorous four-stage pipeline to ensure data quality:\n\\begin{itemize}[leftmargin=1.5em,itemsep=2pt]\n\\item Stage 1: Task Synthesis. Tasks are designed by senior frontend experts to ensure they reflect real-world engineering challenges while maintaining a balanced distribution across diverse scenarios and technologies.\n\\item Stage 2: Checklist Generation and Refinement. We initially employ Claude Sonnet 4.5 to synthesize candidate checklists based on task specifications $T$. These are then meticulously audited and integrated by experts. Through multiple rounds of refinement, we ensure that each check-item is semantically unambiguous, objective, and provides exhaustive coverage of user requirements.\n\\item Stage 3: Execution-based Correction. We conduct cross-validation between the Agent-as-a-Judge framework and human experts. Any discrepancies in judgment trigger a re-evaluation and correction of the underlying data to eliminate potential noise.\n\\item Stage 4: Dynamic Benchmark Iteration. To maintain a high level of discriminative power, we iteratively update the test suite by removing trivial tasks that no longer challenge state-of-the-art coding agents. This expert-led curation process culminated in a final set of 220 high-quality frontend coding tasks and their corresponding checklists.\n\\end{itemize}\n\n\\end{document}"}}, "papers/normalized/huanyuan_large/main.bib": {"8756b3e225bb2e2fcc7d684814615c1ef6e140f7": {"t": 1786068553.424788, "n": 43892, "h": "b551b32f2650bd26", "c": "@inproceedings{langley00,\n author = {P. Langley},\n title = {Crafting Papers on Machine Learning},\n year = {2000},\n pages = {1207--1216},\n editor = {Pat Langley},\n booktitle = {Proceedings of the 17th International Conference\n on Machine Learning (ICML 2000)},\n address = {Stanford, CA},\n publisher = {Morgan Kaufmann}\n}\n\n@TechReport{mitchell80,\n author = \t \"T. M. Mitchell\",\n title = \t \"The Need for Biases in Learning Generalizations\",\n institution = \"Computer Science Department, Rutgers University\",\n year = \t \"1980\",\n address =\t \"New Brunswick, MA\",\n}\n\n@phdthesis{kearns89,\n author = {M. J. Kearns},\n title = {Computational Complexity of Machine Learning},\n school = {Department of Computer Science, Harvard University},\n year = {1989}\n}\n\n@Book{MachineLearningI,\n editor = \t \"R. S. Michalski and J. G. Carbonell and T.\n\t\t M. Mitchell\",\n title = \t \"Machine Learning: An Artificial Intelligence\n\t\t Approach, Vol. I\",\n publisher = \t \"Tioga\",\n year = \t \"1983\",\n address =\t \"Palo Alto, CA\"\n}\n\n@Book{DudaHart2nd,\n author = \"R. O. Duda and P. E. Hart and D. G. Stork\",\n title = \"Pattern Classification\",\n publisher = \"John Wiley and Sons\",\n edition = \"2nd\",\n year = \"2000\"\n}\n\n@misc{anonymous,\n title= {Suppressed for Anonymity},\n author= {Author, N. N.},\n year= {2021}\n}\n\n@InCollection{Newell81,\n author = \"A. Newell and P. S. Rosenbloom\",\n title = \"Mechanisms of Skill Acquisition and the Law of\n Practice\", \n booktitle = \"Cognitive Skills and Their Acquisition\",\n pages = \"1--51\",\n publisher = \"Lawrence Erlbaum Associates, Inc.\",\n year = \"1981\",\n editor = \"J. R. Anderson\",\n chapter = \"1\",\n address = \"Hillsdale, NJ\"\n}\n\n\n@Article{Samuel59,\n author = \t \"A. L. Samuel\",\n title = \t \"Some Studies in Machine Learning Using the Game of\n\t\t Checkers\",\n journal =\t \"IBM Journal of Research and Development\",\n year =\t \"1959\",\n volume =\t \"3\",\n number =\t \"3\",\n pages =\t \"211--229\"\n}\n\n@misc{Authors14,\n author = {FirstName LastName},\n title = {The frobnicatable foo filter},\n note = {Face and Gesture submission ID 324. Supplied as supplemental material {\\tt fg324.pdf}},\n year = 2014\n}\n\n@misc{Authors14b,\n author = {FirstName LastName},\n title = {Frobnication tutorial},\n note = {Supplied as supplemental material {\\tt tr.pdf}},\n year = 2014\n}\n\n@article{Alpher02,\nauthor = {FirstName Alpher},\ntitle = {Frobnication},\njournal = PAMI,\nvolume = 12,\nnumber = 1,\npages = {234--778},\nyear = 2002\n}\n\n@article{Alpher03,\nauthor = {FirstName Alpher and FirstName Fotheringham-Smythe},\ntitle = {Frobnication revisited},\njournal = {Journal of Foo},\nvolume = 13,\nnumber = 1,\npages = {234--778},\nyear = 2003\n}\n\n@article{Alpher04,\nauthor = {FirstName Alpher and FirstName Fotheringham-Smythe and FirstName Gamow},\ntitle = {Can a machine frobnicate?},\njournal = {Journal of Foo},\nvolume = 14,\nnumber = 1,\npages = {234--778},\nyear = 2004\n}\n\n@inproceedings{Alpher05,\nauthor = {FirstName Alpher and FirstName Gamow},\ntitle = {Can a computer frobnicate?},\nbooktitle = CVPR,\npages = {234--778},\nyear = 2005\n}\n\n@article{jo2019cross,\n title={Cross-modal variational auto-encoder with distributed latent spaces and associators},\n author={Jo, Dae Ung and Lee, ByeongJu and Choi, Jongwon and Yoo, Haanju and Choi, Jin Young},\n journal={arXiv preprint arXiv:1905.12867},\n year={2019}\n}\n@article{wettig2022should,\n title={Should You Mask 15\\% in Masked Language Modeling?},\n author={Wettig, Alexander and Gao, Tianyu and Zhong, Zexuan and Chen, Danqi},\n journal={arXiv preprint arXiv:2202.08005},\n year={2022}\n}\n@article{li2021align,\n title={Align before fuse: Vision and language representation learning with momentum distillation},\n author={Li, Junnan and Selvaraju, Ramprasaath and Gotmare, Akhilesh and Joty, Shafiq and Xiong, Caiming and Hoi, Steven Chu Hong},\n journal={Advances in Neural Information Processing Systems},\n volume={34},\n year={2021}\n}\n@inproceedings{he2022masked,\n title={Masked autoencoders are scalable vision learners},\n author={He, Kaiming and Chen, Xinlei and Xie, Saining and Li, Yanghao and Doll{\\'a}r, Piotr and Girshick, Ross},\n booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},\n pages={16000--16009},\n year={2022}\n}\n@article{su2019vl,\n title={Vl-bert: Pre-training of generic visual-linguistic representations},\n author={Su, Weijie and Zhu, Xizhou and Cao, Yue and Li, Bin and Lu, Lewei and Wei, Furu and Dai, Jifeng},\n journal={arXiv preprint arXiv:1908.08530},\n year={2019}\n}\n@article{lu2019vilbert,\n title={Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks},\n author={Lu, Jiasen and Batra, Dhruv and Parikh, Devi and Lee, Stefan},\n journal={Advances in neural information processing systems},\n volume={32},\n year={2019}\n}\n@inproceedings{lu202012,\n title={12-in-1: Multi-task vision and language representation learning},\n author={Lu, Jiasen and Goswami, Vedanuj and Rohrbach, Marcus and Parikh, Devi and Lee, Stefan},\n booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},\n pages={10437--10446},\n year={2020}\n}\n@article{li2019visualbert,\n title={Visualbert: A simple and performant baseline for vision and language},\n author={Li, Liunian Harold and Yatskar, Mark and Yin, Da and Hsieh, Cho-Jui and Chang, Kai-Wei},\n journal={arXiv preprint arXiv:1908.03557},\n year={2019}\n}\n@article{qi2020imagebert,\n title={Imagebert: Cross-modal pre-training with large-scale weak-supervised image-text data},\n author={Qi, Di and Su, Lin and Song, Jia and Cui, Edward and Bharti, Taroon and Sacheti, Arun},\n journal={arXiv preprint arXiv:2001.07966},\n year={2020}\n}\n@inproceedings{li2020unicoder,\n title={Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training},\n author={Li, Gen and Duan, Nan and Fang, Yuejian and Gong, Ming and Jiang, Daxin},\n booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},\n volume={34},\n number={07},\n pages={11336--11344},\n year={2020}\n}\n@inproceedings{yu2021ernie,\n title={Ernie-vil: Knowledge enhanced vision-language representations through scene graphs},\n author={Yu, Fei and Tang, Jiji and Yin, Weichong and Sun, Yu and Tian, Hao and Wu, Hua and Wang, Haifeng},\n booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},\n volume={35},\n number={4},\n pages={3208--3216},\n year={2021}\n}\n@article{zhang2021vinvl,\n title={Vinvl: Making visual representations matter in vision-language models},\n author={Zhang, Pengchuan and Li, Xiujun and Hu, Xiaowei and Yang, Jianwei and Zhang, Lei and Wang, Lijuan and Choi, Yejin and Gao, Jianfeng},\n journal={arXiv e-prints},\n pages={arXiv--2101},\n year={2021}\n}\n@inproceedings{huang2021seeing,\n title={Seeing out of the box: End-to-end pre-training for vision-language representation learning},\n author={Huang, Zhicheng and Zeng, Zhaoyang and Huang, Yupan and Liu, Bei and Fu, Dongmei and Fu, Jianlong},\n booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},\n pages={12976--12985},\n year={2021}\n}\n@article{tan2019lxmert,\n title={Lxmert: Learning cross-modality encoder representations from transformers},\n author={Tan, Hao and Bansal, Mohit},\n journal={arXiv preprint arXiv:1908.07490},\n year={2019}\n}\n@inproceedings{chen2020uniter,\n title={Uniter: Universal image-text representation learning},\n author={Chen, Yen-Chun and Li, Linjie and Yu, Licheng and El Kholy, Ahmed and Ahmed, Faisal and Gan, Zhe and Cheng, Yu and Liu, Jingjing},\n booktitle={European conference on computer vision},\n pages={104--120},\n year={2020},\n organization={Springer}\n}\n@inproceedings{li2020oscar,\n title={Oscar: Object-semantics aligned pre-training for vision-language tasks},\n author={Li, Xiujun and Yin, Xi and Li, Chunyuan and Zhang, Pengchuan and Hu, Xiaowei and Zhang, Lei and Wang, Lijuan and Hu, Houdong and Dong, Li and Wei, Furu and others},\n booktitle={European Conference on Computer Vision},\n pages={121--137},\n year={2020},\n organization={Springer}\n}\n@article{ordonez2011im2text,\n title={Im2text: Describing images using 1 million captioned photographs},\n author={Ordonez, Vicente and Kulkarni, Girish and Berg, Tamara},\n journal={Advances in neural information processing systems},\n volume={24},\n year={2011}\n}\n@inproceedings{radford2021learning,\n title={Learning transferable visual models from natural language supervision},\n author={Radford, Alec and Kim, Jong Wook and Hallacy, Chris and Ramesh, Aditya and Goh, Gabriel and Agarwal, Sandhini and Sastry, Girish and Askell, Amanda and Mishkin, Pamela and Clark, Jack and others},\n booktitle={International Conference on Machine Learning},\n pages={8748--8763},\n year={2021},\n organization={PMLR}\n}\n@inproceedings{jia2021scaling,\n title={Scaling up visual and vision-language representation learning with noisy text supervision},\n author={Jia, Chao and Yang, Yinfei and Xia, Ye and Chen, Yi-Ting and Parekh, Zarana and Pham, Hieu and Le, Quoc and Sung, Yun-Hsuan and Li, Zhen and Duerig, Tom},\n booktitle={International Conference on Machine Learning},\n pages={4904--4916},\n year={2021},\n organization={PMLR}\n}\n@article{gan2020large,\n title={Large-scale adversarial training for vision-and-language representation learning},\n author={Gan, Zhe and Chen, Yen-Chun and Li, Linjie and Zhu, Chen and Cheng, Yu and Liu, Jingjing},\n journal={Advances in Neural Information Processing Systems},\n volume={33},\n pages={6616--6628},\n year={2020}\n}\n@inproceedings{selvaraju2017grad,\n title={Grad-cam: Visual explanations from deep networks via gradient-based localization},\n author={Selvaraju, Ramprasaath R and Cogswell, Michael and Das, Abhishek and Vedantam, Ramakrishna and Parikh, Devi and Batra, Dhruv},\n booktitle={Proceedings of the IEEE international conference on computer vision},\n pages={618--626},\n year={2017}\n}\n@inproceedings{kim2021vilt,\n title={Vilt: Vision-and-language transformer without convolution or region supervision},\n author={Kim, Wonjae and Son, Bokyung and Kim, Ildoo},\n booktitle={International Conference on Machine Learning},\n pages={5583--5594},\n year={2021},\n organization={PMLR}\n}\n@article{faghri2017vse++,\n title={Vse++: Improving visual-semantic embeddings with hard negatives},\n author={Faghri, Fartash and Fleet, David J and Kiros, Jamie Ryan and Fidler, Sanja},\n journal={arXiv preprint arXiv:1707.05612},\n year={2017}\n}\n@inproceedings{sharma2018conceptual,\n title={Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning},\n author={Sharma, Piyush and Ding, Nan and Goodman, Sebastian and Soricut, Radu},\n booktitle={Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},\n pages={2556--2565},\n year={2018}\n}\n@article{schuhmann2021laion,\n title={Laion-400m: Open dataset of clip-filtered 400 million image-text pairs},\n author={Schuhmann, Christoph and Vencu, Richard and Beaumont, Romain and Kaczmarczyk, Robert and Mullis, Clayton and Katta, Aarush and Coombes, Theo and Jitsev, Jenia and Komatsuzaki, Aran},\n journal={arXiv preprint arXiv:2111.02114},\n year={2021}\n}\n@inproceedings{lin2014microsoft,\n title={Microsoft coco: Common objects in context},\n author={Lin, Tsung-Yi and Maire, Michael and Belongie, Serge and Hays, James and Perona, Pietro and Ramanan, Deva and Doll{\\'a}r, Piotr and Zitnick, C Lawrence},\n booktitle={European conference on computer vision},\n pages={740--755},\n year={2014},\n organization={Springer}\n}\n@article{krishna2017visual,\n title={Visual genome: Connecting language and vision using crowdsourced dense image annotations},\n author={Krishna, Ranjay and Zhu, Yuke and Groth, Oliver and Johnson, Justin and Hata, Kenji and Kravitz, Joshua and Chen, Stephanie and Kalantidis, Yannis and Li, Li-Jia and Shamma, David A and others},\n journal={International journal of computer vision},\n volume={123},\n number={1},\n pages={32--73},\n year={2017},\n publisher={Springer}\n}\n@article{DBLP:journals/corr/abs-2106-08254,\n author = {Hangbo Bao and\n Li Dong and\n Furu Wei},\n title = {BEiT: {BERT} Pre-Training of Image Transformers},\n journal = {CoRR},\n volume = {abs/2106.08254},\n year = {2021},\n url = {https://arxiv.org/abs/2106.08254},\n eprinttype = {arXiv},\n eprint = {2106.08254},\n timestamp = {Tue, 29 Jun 2021 16:55:04 +0200},\n biburl = {https://dblp.org/rec/journals/corr/abs-2106-08254.bib},\n bibsource = {dblp computer science bibliography, https://dblp.org}\n}\n@article{devlin2018bert,\n title={Bert: Pre-training of deep bidirectional transformers for language understanding},\n author={Devlin, Jacob and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina},\n journal={arXiv preprint arXiv:1810.04805},\n year={2018}\n}\n@article{liu2019roberta,\n title={Roberta: A robustly optimized bert pretraining approach},\n author={Liu, Yinhan and Ott, Myle and Goyal, Naman and Du, Jingfei and Joshi, Mandar and Chen, Danqi and Levy, Omer and Lewis, Mike and Zettlemoyer, Luke and Stoyanov, Veselin},\n journal={arXiv preprint arXiv:1907.11692},\n year={2019}\n}\n@InProceedings{pmlr-v139-touvron21a,\n title = {Training data-efficient image transformers & distillation through attention},\n author = {Touvron, Hugo and Cord, Matthieu and Douze, Matthijs and Massa, Francisco and Sablayrolles, Alexandre and Jegou, Herve},\n booktitle = {International Conference on Machine Learning},\n pages = {10347--10357},\n year = {2021},\n volume = {139},\n month = {July}\n}\n@article{dosovitskiy2020image,\n title={An image is worth 16x16 words: Transformers for image recognition at scale},\n author={Dosovitskiy, Alexey and Beyer, Lucas and Kolesnikov, Alexander and Weissenborn, Dirk and Zhai, Xiaohua and Unterthiner, Thomas and Dehghani, Mostafa and Minderer, Matthias and Heigold, Georg and Gelly, Sylvain and others},\n journal={arXiv preprint arXiv:2010.11929},\n year={2020}\n}\n@inproceedings{singh2022flava,\n title={Flava: A foundational language and vision alignment model},\n author={Singh, Amanpreet and Hu, Ronghang and Goswami, Vedanuj and Couairon, Guillaume and Galuba, Wojciech and Rohrbach, Marcus and Kiela, Douwe},\n booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},\n pages={15638--15650},\n year={2022}\n}\n\n@article{wang2021vlmo,\n title={VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts},\n author={Wang, Wenhui and Bao, Hangbo and Dong, Li and Wei, Furu},\n journal={arXiv preprint arXiv:2111.02358},\n year={2021}\n}\n\n@article{bao2021beit,\n title={Beit: Bert pre-training of image transformers},\n author={Bao, Hangbo and Dong, Li and Wei, Furu},\n journal={arXiv preprint arXiv:2106.08254},\n year={2021}\n}\n\n@article{wettig2022should,\n title={Should You Mask 15\\% in Masked Language Modeling?},\n author={Wettig, Alexander and Gao, Tianyu and Zhong, Zexuan and Chen, Danqi},\n journal={arXiv preprint arXiv:2202.08005},\n year={2022}\n}\n\n@inproceedings{he2020momentum,\n title={Momentum contrast for unsupervised visual representation learning},\n author={He, Kaiming and Fan, Haoqi and Wu, Yuxin and Xie, Saining and Girshick, Ross},\n booktitle={Proceedings of the IEEE/CVF conference on computer vision and pattern recognition},\n pages={9729--9738},\n year={2020}\n}\n\n@article{suhr2018corpus,\n title={A corpus for reasoning about natural language grounded in photographs},\n author={Suhr, Alane and Zhou, Stephanie and Zhang, Ally and Zhang, Iris and Bai, Huajun and Artzi, Yoav},\n journal={arXiv preprint arXiv:1811.00491},\n year={2018}\n}\n\n@inproceedings{goyal2017making,\n title={Making the v in vqa matter: Elevating the role of image understanding in visual question answering},\n author={Goyal, Yash and Khot, Tejas and Summers-Stay, Douglas and Batra, Dhruv and Parikh, Devi},\n booktitle={Proceedings of the IEEE conference on computer vision and pattern recognition},\n pages={6904--6913},\n year={2017}\n}\n\n@inproceedings{plummer2015flickr30k,\n title={Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models},\n author={Plummer, Bryan A and Wang, Liwei and Cervantes, Chris M and Caicedo, Juan C and Hockenmaier, Julia and Lazebnik, Svetlana},\n booktitle={Proceedings of the IEEE international conference on computer vision},\n pages={2641--2649},\n year={2015}\n}\n\n@article{xie2019visual,\n title={Visual entailment: A novel task for fine-grained image understanding},\n author={Xie, Ning and Lai, Farley and Doran, Derek and Kadav, Asim},\n journal={arXiv preprint arXiv:1901.06706},\n year={2019}\n}\n\n@inproceedings{yu2016modeling,\n title={Modeling context in referring expressions},\n author={Yu, Licheng and Poirson, Patrick and Yang, Shan and Berg, Alexander C and Berg, Tamara L},\n booktitle={European Conference on Computer Vision},\n pages={69--85},\n year={2016},\n organization={Springer}\n}\n\n@article{li2020unimo,\n title={Unimo: Towards unified-modal understanding and generation via cross-modal contrastive learning},\n author={Li, Wei and Gao, Can and Niu, Guocheng and Xiao, Xinyan and Liu, Hao and Liu, Jiachen and Wu, Hua and Wang, Haifeng},\n journal={arXiv preprint arXiv:2012.15409},\n year={2020}\n}\n\n@article{li2022blip,\n title={Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation},\n author={Li, Junnan and Li, Dongxu and Xiong, Caiming and Hoi, Steven},\n journal={arXiv preprint arXiv:2201.12086},\n year={2022}\n}\n\n@article{li2020unimo,\n title={Unimo: Towards unified-modal understanding and generation via cross-modal contrastive learning},\n author={Li, Wei and Gao, Can and Niu, Guocheng and Xiao, Xinyan and Liu, Hao and Liu, Jiachen and Wu, Hua and Wang, Haifeng},\n journal={arXiv preprint arXiv:2012.15409},\n year={2020}\n}\n\n@article{gan2020large,\n title={Large-scale adversarial training for vision-and-language representation learning},\n author={Gan, Zhe and Chen, Yen-Chun and Li, Linjie and Zhu, Chen and Cheng, Yu and Liu, Jingjing},\n journal={Advances in Neural Information Processing Systems},\n volume={33},\n pages={6616--6628},\n year={2020}\n}\n\n@inproceedings{liu2019adaptive,\n title={Adaptive reconstruction network for weakly supervised referring expression grounding},\n author={Liu, Xuejing and Li, Liang and Wang, Shuhui and Zha, Zheng-Jun and Meng, Dechao and Huang, Qingming},\n booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},\n pages={2611--2620},\n year={2019}\n}\n\n@article{zhang2020counterfactual,\n title={Counterfactual contrastive learning for weakly-supervised vision-language grounding},\n author={Zhang, Zhu and Zhao, Zhou and Lin, Zhijie and He, Xiuqiang and others},\n journal={Advances in Neural Information Processing Systems},\n volume={33},\n pages={18123--18134},\n year={2020}\n}\n\n@article{su2019vl,\n title={Vl-bert: Pre-training of generic visual-linguistic representations},\n author={Su, Weijie and Zhu, Xizhou and Cao, Yue and Li, Bin and Lu, Lewei and Wei, Furu and Dai, Jifeng},\n journal={arXiv preprint arXiv:1908.08530},\n year={2019}\n}\n\n@inproceedings{lu202012,\n title={12-in-1: Multi-task vision and language representation learning},\n author={Lu, Jiasen and Goswami, Vedanuj and Rohrbach, Marcus and Parikh, Devi and Lee, Stefan},\n booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},\n pages={10437--10446},\n year={2020}\n}\n\n@inproceedings{cho2021unifying,\n title={Unifying vision-and-language tasks via text generation},\n author={Cho, Jaemin and Lei, Jie and Tan, Hao and Bansal, Mohit},\n booktitle={International Conference on Machine Learning},\n pages={1931--1942},\n year={2021},\n organization={PMLR}\n}\n\n@article{gan2020large,\n title={Large-scale adversarial training for vision-and-language representation learning},\n author={Gan, Zhe and Chen, Yen-Chun and Li, Linjie and Zhu, Chen and Cheng, Yu and Liu, Jingjing},\n journal={Advances in Neural Information Processing Systems},\n volume={33},\n pages={6616--6628},\n year={2020}\n}\n\n@article{li2022blip,\n title={Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation},\n author={Li, Junnan and Li, Dongxu and Xiong, Caiming and Hoi, Steven},\n journal={arXiv preprint arXiv:2201.12086},\n year={2022}\n}\n\n@article{wang2021vlmo,\n title={Vlmo: Unified vision-language pre-training with mixture-of-modality-experts},\n author={Wang, Wenhui and Bao, Hangbo and Dong, Li and Wei, Furu},\n journal={arXiv preprint arXiv:2111.02358},\n year={2021}\n}\n\n@article{wang2021simvlm,\n title={Simvlm: Simple visual language model pretraining with weak supervision},\n author={Wang, Zirui and Yu, Jiahui and Yu, Adams Wei and Dai, Zihang and Tsvetkov, Yulia and Cao, Yuan},\n journal={arXiv preprint arXiv:2108.10904},\n year={2021}\n}\n\n@article{wang2021simvlm,\n title={Simvlm: Simple visual language model pretraining with weak supervision},\n author={Wang, Zirui and Yu, Jiahui and Yu, Adams Wei and Dai, Zihang and Tsvetkov, Yulia and Cao, Yuan},\n journal={arXiv preprint arXiv:2108.10904},\n year={2021}\n}\n\n@article{wettig2022should,\n title={Should You Mask 15\\% in Masked Language Modeling?},\n author={Wettig, Alexander and Gao, Tianyu and Zhong, Zexuan and Chen, Danqi},\n journal={arXiv preprint arXiv:2202.08005},\n year={2022}\n}\n\n@inproceedings{rajbhandari2022deepspeed,\n title={{DeepSpeed-MoE}: Advancing mixture-of-experts inference and training to power next-generation ai scale},\n author={Rajbhandari, Samyam and Li, Conglong and Yao, Zhewei and Zhang, Minjia and Aminabadi, Reza Yazdani and Awan, Ammar Ahmad and Rasley, Jeff and He, Yuxiong},\n booktitle={Proceedings of ICML},\n year={2022}\n}\n\n@article{dai2024deepseekmoe,\n title={{DeepSeekMoE}: Towards ultimate expert specialization in mixture-of-experts language models},\n author={Dai, Damai and Deng, Chengqi and Zhao, Chenggang and Xu, RX and Gao, Huazuo and Chen, Deli and Li, Jiashi and Zeng, Wangding and Yu, Xingkai and Wu, Y and others},\n journal={arXiv preprint arXiv:2401.06066},\n year={2024}\n}\n\n@article{jiang2024mixtral,\n title={Mixtral of experts},\n author={Jiang, Albert Q and Sablayrolles, Alexandre and Roux, Antoine and Mensch, Arthur and Savary, Blanche and Bamford, Chris and Chaplot, Devendra Singh and Casas, Diego de las and Hanna, Emma Bou and Bressand, Florian and others},\n journal={arXiv preprint arXiv:2401.04088},\n year={2024}\n}\n@misc{qwen_moe,\n title = {{Qwen1.5-MoE: Matching 7B Model Performance with 1/3 Activated Parameters}},\n url = {https://qwenlm.github.io/blog/qwen-moe/},\n author = {Qwen},\n year = {2024}\n}\n\n@article{dekel2012optimal,\n title={Optimal Distributed Online Prediction Using Mini-Batches.},\n author={Dekel, Ofer and Gilad-Bachrach, Ran and Shamir, Ohad and Xiao, Lin},\n journal={Journal of Machine Learning Research},\n volume={13},\n number={1},\n year={2012}\n}\n\n@article{goyal2017accurate,\n title={{Accurate, large minibatch SGD: Training Imagenet in 1 hour}},\n author={Goyal, Priya and Doll{\\'a}r, Piotr and Girshick, Ross and Noordhuis, Pieter and Wesolowski, Lukasz and Kyrola, Aapo and Tulloch, Andrew and Jia, Yangqing and He, Kaiming},\n journal={arXiv preprint arXiv:1706.02677},\n year={2017}\n}\n\n@article{krizhevsky2014one,\n title={One weird trick for parallelizing convolutional neural networks},\n author={Krizhevsky, Alex},\n journal={arXiv preprint arXiv:1404.5997},\n year={2014}\n}\n\n@article{brandon2024reducing,\n title={Reducing Transformer Key-Value Cache Size with Cross-Layer Attention},\n author={Brandon, William and Mishra, Mayank and Nrusimha, Aniruddha and Panda, Rameswar and Kelly, Jonathan Ragan},\n journal={arXiv preprint arXiv:2405.12981},\n year={2024}\n}\n\n@article{su2024roformer,\n title={RoFormer: Enhanced transformer with rotary position embedding},\n author={Su, Jianlin and Ahmed, Murtadha and Lu, Yu and Pan, Shengfeng and Bo, Wen and Liu, Yunfeng},\n journal={Neurocomputing},\n year={2024}\n}\n\n@article{gregory2024needle,\n title={Needle In A Haystack - Pressure Testing LLMs},\n author={Gregory Kamradt},\n journal={Github},\n url = {https://github.com/gkamradt/LLMTest_NeedleInAHaystack/},\n year={2023}\n}\n\n@inproceedings{ainslie2023gqa,\n title={{GQA}: Training generalized multi-query transformer models from multi-head checkpoints},\n author={Ainslie, Joshua and Lee-Thorp, James and de Jong, Michiel and Zemlyanskiy, Yury and Lebr{\\'o}n, Federico and Sanghai, Sumit},\n booktitle={Proceedings of EMNLP},\n year={2023}\n}\n\n@article{kaplan2020scaling,\n title={Scaling laws for neural language models},\n author={Kaplan, Jared and McCandlish, Sam and Henighan, Tom and Brown, Tom B and Chess, Benjamin and Child, Rewon and Gray, Scott and Radford, Alec and Wu, Jeffrey and Amodei, Dario},\n journal={arXiv preprint arXiv:2001.08361},\n year={2020}\n}\n\n@article{li2024surge,\n title={Surge Phenomenon in Optimal Learning Rate and Batch Size Scaling},\n author={Li, Shuaipeng and Zhao, Penghao and Zhang, Hailin and Sun, Xingwu and Wu, Hao and Jiao, Dian and Wang, Weiyan and Liu, Chengjun and Fang, Zheng and Xue, Jinbao and others},\n journal={arXiv preprint arXiv:2405.14578},\n year={2024}\n}\n\n@article{hoffmann2022training,\n title={Training compute-optimal large language models},\n author={Hoffmann, Jordan and Borgeaud, Sebastian and Mensch, Arthur and Buchatskaya, Elena and Cai, Trevor and Rutherford, Eliza and Casas, Diego de Las and Hendricks, Lisa Anne and Welbl, Johannes and Clark, Aidan and others},\n journal={arXiv preprint arXiv:2203.15556},\n year={2022}\n}\n\n@article{dubey2024llama,\n\ttitle={{The Llama 3 Herd of Models}},\n\tauthor={Dubey, Abhimanyu and Jauhri, Abhinav and Pandey, Abhinav and Kadian, Abhishek and Al-Dahle, Ahmad and Letman, Aiesha and Mathur, Akhil and Schelten, Alan and Yang, Amy and Fan, Angela and others},\n\tjournal={arXiv preprint arXiv:2407.21783},\n\tyear={2024}\n}\n\n@article{yang2024qwen2,\n\ttitle={{QWen2 Technical Report}},\n\tauthor={Yang, An and Yang, Baosong and Hui, Binyuan and Zheng, Bo and Yu, Bowen and Zhou, Chang and Li, Chengpeng and Li, Chengyuan and Liu, Dayiheng and Huang, Fei and others},\n\tjournal={arXiv preprint arXiv:2407.10671},\n\tyear={2024}\n}\n\n@article{deepseek2024v2,\n\ttitle={{DeepSeek-V2}: A Strong, Economical, and Efficient Mixture-of-Experts Language Model},\n\tauthor={DeepSeek-AI},\n\tjournal={arXiv preprint arXiv:2405.04434},\n\tyear={2024}\n}\n\n@article{mistral2024mixtral8x22,\n\ttitle={Cheaper, Better, Faster, Stronger. Continuing to push the frontier of {AI} and making it accessible to all.},\n\tauthor={Mistral},\n url = {https://mistral.ai/news/mixtral-8x22b},\n\tyear={2024}\n}\n\n@inproceedings{wang2024mmlu-pro,\n title={{MMLU-Pro}: A more robust and challenging multi-task language understanding benchmark},\n author={Wang, Yubo and Ma, Xueguang and Zhang, Ge and Ni, Yuansheng and Chandra, Abhranil and Guo, Shiguang and Ren, Weiming and Arulraj, Aaran and He, Xuan and Jiang, Ziyan and others},\n booktitle={Proceedings of NeurIPS},\n year={2024}\n}\n\n@article{zhong2023AGIEval,\n title={{AGIEval}: A human-centric benchmark for evaluating foundation models},\n author={Zhong, Wanjun and Cui, Ruixiang and Guo, Yiduo and Liang, Yaobo and Lu, Shuai and Wang, Yanlin and Saied, Amin and Chen, Weizhu and Duan, Nan},\n journal={arXiv preprint arXiv:2304.06364},\n year={2023}\n}\n\n@article{suzgun2022challenging,\n title={Challenging big-bench tasks and whether chain-of-thought can solve them},\n author={Suzgun, Mirac and Scales, Nathan and Sch{\\\"a}rli, Nathanael and Gehrmann, Sebastian and Tay, Yi and Chung, Hyung Won and Chowdhery, Aakanksha and Le, Quoc V and Chi, Ed H and Zhou, Denny and others},\n journal={arXiv preprint arXiv:2210.09261},\n year={2022}\n}\n\n@article{li2023cmmlu,\n title={{CMMLU}: Measuring massive multitask language understanding in chinese},\n author={Li, Haonan and Zhang, Yixuan and Koto, Fajri and Yang, Yifei and Zhao, Hai and Gong, Yeyun and Duan, Nan and Baldwin, Timothy},\n journal={arXiv preprint arXiv:2306.09212},\n year={2023}\n}\n\n@inproceedings{huang2024ceval,\n title={{C-Eval}: A multi-level multi-discipline chinese evaluation suite for foundation models},\n author={Huang, Yuzhen and Bai, Yuzhuo and Zhu, Zhihao and Zhang, Junlei and Zhang, Jinghan and Su, Tangjun and Liu, Junteng and Lv, Chuancheng and Zhang, Yikai and Fu, Yao and others},\n booktitle={Proceedings of NeurIPS},\n year={2024}\n}\n\n@inproceedings{zellers2019hellaswag,\n title={{HellaSwag}: Can a machine really finish your sentence?},\n author={Zellers, Rowan and Holtzman, Ari and Bisk, Yonatan and Farhadi, Ali and Choi, Yejin},\n booktitle={Proceedings of ACL},\n year={2019}\n}\n\n@inproceedings{talmor2019commonsenseqa,\n title={{CommonsenseQA}: A question answering challenge targeting commonsense knowledge},\n author={Talmor, Alon and Herzig, Jonathan and Lourie, Nicholas and Berant, Jonathan},\n booktitle={Proceedings of NAACL},\n year={2019}\n}\n\n@article{sakaguchi2021winogrande,\n title={{WinoGrande}: An adversarial winograd schema challenge at scale},\n author={Sakaguchi, Keisuke and Bras, Ronan Le and Bhagavatula, Chandra and Choi, Yejin},\n journal={Communications of the ACM},\n year={2021}\n}\n\n@inproceedings{bisk2020piqa,\n title={{PIQA}: Reasoning about physical commonsense in natural language},\n author={Bisk, Yonatan and Zellers, Rowan and Gao, Jianfeng and Choi, Yejin and others},\n booktitle={Proceedings of AAAI},\n year={2020}\n}\n\n@inproceedings{sap2019socialiqa,\n title={{SocialiQA}: Commonsense reasoning about social interactions},\n author={Sap, Maarten and Rashkin, Hannah and Chen, Derek and LeBras, Ronan and Choi, Yejin},\n booktitle={Proceedings of EMNLP},\n year={2019}\n}\n\n@article{clark2018think,\n title={{Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge}},\n author={Clark, Peter and Cowhey, Isaac and Etzioni, Oren and Khot, Tushar and Sabharwal, Ashish and Schoenick, Carissa and Tafjord, Oyvind},\n journal={arXiv preprint arXiv:1803.05457},\n year={2018}\n}\n\n@article{joshi2017triviaqa,\n title={{TriviaQA}: A large scale distantly supervised challenge dataset for reading comprehension},\n author={Joshi, Mandar and Choi, Eunsol and Weld, Daniel S and Zettlemoyer, Luke},\n journal={arXiv preprint arXiv:1705.03551},\n year={2017}\n}\n\n@inproceedings{lin2022truthfulqa,\n title={{TruthfulQA}: Measuring how models mimic human falsehoods},\n author={Lin, Stephanie and Hilton, Jacob and Evans, Owain},\n booktitle={Proceedings of ACL},\n year={2022}\n}\n\n@inproceedings{zheng2019chid,\n title={{ChID}: A large-scale Chinese IDiom dataset for cloze test},\n author={Zheng, Chujie and Huang, Minlie and Sun, Aixin},\n booktitle={Proceedings of ACL},\n year={2019}\n}\n\n@article{kwiatkowski2019natural,\n title={Natural questions: a benchmark for question answering research},\n author={Kwiatkowski, Tom and Palomaki, Jennimaria and Redfield, Olivia and Collins, Michael and Parikh, Ankur and Alberti, Chris and Epstein, Danielle and Polosukhin, Illia and Devlin, Jacob and Lee, Kenton and others},\n journal={TACL},\n year={2019}\n}\n\n@article{sun2020investigating,\n title={Investigating prior knowledge for challenging chinese machine reading comprehension},\n author={Sun, Kai and Yu, Dian and Yu, Dong and Cardie, Claire},\n journal={TACL},\n year={2020}\n}\n\n@article{hendrycks2021measuring,\n title={Measuring mathematical problem solving with the math dataset},\n author={Hendrycks, Dan and Burns, Collin and Kadavath, Saurav and Arora, Akul and Basart, Steven and Tang, Eric and Song, Dawn and Steinhardt, Jacob},\n journal={arXiv preprint arXiv:2103.03874},\n year={2021}\n}\n\n@article{wei2023cmath,\n title={{CMATH}: Can your language model pass chinese elementary school math test?},\n author={Wei, Tianwen and Luan, Jian and Liu, Wei and Dong, Shuang and Wang, Bin},\n journal={arXiv preprint arXiv:2306.16636},\n year={2023}\n}\n\n@article{chen2021evaluating,\n title={Evaluating large language models trained on code},\n author={Chen, Mark and Tworek, Jerry and Jun, Heewoo and Yuan, Qiming and Pinto, Henrique Ponde De Oliveira and Kaplan, Jared and Edwards, Harri and Burda, Yuri and Joseph, Nicholas and Brockman, Greg and others},\n journal={arXiv preprint arXiv:2107.03374},\n year={2021}\n}\n\n@article{austin2021program,\n title={Program synthesis with large language models},\n author={Austin, Jacob and Odena, Augustus and Nye, Maxwell and Bosma, Maarten and Michalewski, Henryk and Dohan, David and Jiang, Ellen and Cai, Carrie and Terry, Michael and Le, Quoc and others},\n journal={arXiv preprint arXiv:2108.07732},\n year={2021}\n}\n\n@article{OpenAI2022ChatGPT,\n\ttitle={{Introducing ChatGPT}},\n\tauthor={OpenAI},\n url = {https://openai.com/index/chatgpt/},\n\tyear={2022}\n}\n\n@article{OpenAI2024GPT4o,\n\ttitle={{Hello GPT-4o}},\n\tauthor={OpenAI},\n url = {https://openai.com/index/hello-gpt-4o/},\n\tyear={2024}\n}\n\n@article{achiam2023gpt,\n title={{GPT-4} technical report},\n author={Achiam, Josh and Adler, Steven and Agarwal, Sandhini and Ahmad, Lama and Akkaya, Ilge and Aleman, Florencia Leoni and Almeida, Diogo and Altenschmidt, Janko and Altman, Sam and Anadkat, Shyamal and others},\n journal={arXiv preprint arXiv:2303.08774},\n year={2023}\n}\n\n@article{Gemini2023gemini,\n title={Gemini: a family of highly capable multimodal models},\n author={Gemini, Team and Anil, Rohan and Borgeaud, Sebastian and Wu, Yonghui and Alayrac, Jean-Baptiste and Yu, Jiahui and Soricut, Radu and Schalkwyk, Johan and Dai, Andrew M and Hauth, Anja and others},\n journal={arXiv preprint arXiv:2312.11805},\n year={2023}\n}\n\n@article{lepikhin2020gshard,\n title={{GShard}: Scaling giant models with conditional computation and automatic sharding},\n author={Lepikhin, Dmitry and Lee, HyoukJoong and Xu, Yuanzhong and Chen, Dehao and Firat, Orhan and Huang, Yanping and Krikun, Maxim and Shazeer, Noam and Chen, Zhifeng},\n journal={arXiv preprint arXiv:2006.16668},\n year={2020}\n}\n\n@article{fedus2022switch,\n title={Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity},\n author={Fedus, William and Zoph, Barret and Shazeer, Noam},\n journal={Journal of Machine Learning Research},\n year={2022}\n}\n\n@article{team2024jamba,\n title={Jamba-1.5: Hybrid Transformer-Mamba Models at Scale},\n author={Jamba, Team and Lenz, Barak and Arazi, Alan and Bergman, Amir and Manevich, Avshalom and Peleg, Barak and Aviram, Ben and Almagor, Chen and Fridman, Clara and Padnos, Dan and others},\n journal={arXiv preprint arXiv:2408.12570},\n year={2024}\n}\n\n@inproceedings{vaswani2017attention,\n title={Attention is all you need},\n author={Vaswani, Ashish and Shazeer, Noam and Parmar, Niki and Uszkoreit, Jakob and Jones, Llion and Gomez, Aidan N and Kaiser, Lukasz and Polosukhin, Illia},\n booktitle={Proceedings of NIPS},\n year={2017}\n}\n\n@inproceedings{rafailov2024direct,\n title={Direct preference optimization: Your language model is secretly a reward model},\n author={Rafailov, Rafael and Sharma, Archit and Mitchell, Eric and Manning, Christopher D and Ermon, Stefano and Finn, Chelsea},\n booktitle={Proceedings of NeurIPS},\n year={2024}\n}\n\n@article{adler2024nemotron,\n title={Nemotron-4 340B Technical Report},\n author={Adler, Bo and Agarwal, Niket and Aithal, Ashwath and Anh, Dong H and Bhattacharya, Pallab and Brundyn, Annika and Casper, Jared and Catanzaro, Bryan and Clay, Sharon and Cohen, Jonathan and others},\n journal={arXiv preprint arXiv:2406.11704},\n year={2024}\n}\n\n@inproceedings{ouyang2022training,\n title={Training language models to follow instructions with human feedback},\n author={Ouyang, Long and Wu, Jeffrey and Jiang, Xu and Almeida, Diogo and Wainwright, Carroll and Mishkin, Pamela and Zhang, Chong and Agarwal, Sandhini and Slama, Katarina and Ray, Alex and others},\n booktitle={Proceedings of NeurIPS},\n year={2022}\n}\n\n@article{pang2024iterative,\n title={Iterative reasoning preference optimization},\n author={Pang, Richard Yuanzhe and Yuan, Weizhe and Cho, Kyunghyun and He, He and Sukhbaatar, Sainbayar and Weston, Jason},\n journal={arXiv preprint arXiv:2404.19733},\n year={2024}\n}\n\n@inproceedings{zhao2024tree,\n title={Tree-Instruct: A Preliminary Study of the Intrinsic Relationship between Complexity and Alignment},\n author={Zhao, Yingxiu and Yu, Bowen and Hui, Binyuan and Yu, Haiyang and Li, Minghao and Huang, Fei and Zhang, Nevin L and Li, Yongbin},\n booktitle={Proceedings of LREC-COLING},\n year={2024}\n}\n\n@inproceedings{loshchilov2019decoupled,\n title={Decoupled weight decay regularization},\n author={Loshchilov, Ilya and Hutter, Frank},\n booktitle={Proceedings of ICLR},\n year={2019}\n}\n\n@article{gao2024train,\n title={How to Train Long-Context Language Models (Effectively)},\n author={Gao, Tianyu and Wettig, Alexander and Yen, Howard and Chen, Danqi},\n journal={arXiv preprint arXiv:2410.02660},\n year={2024}\n}\n\n@article{OpenAI2022Tiktoken,\n\ttitle={{Tiktoken}},\n\tauthor={OpenAI},\n url = {https://github.com/openai/tiktoken},\n year={2023}\n}\n\n@article{touvron2023llama,\n title={{LLaMA: Open and Efficient Foundation Language Models}},\n author={Touvron, Hugo and Lavril, Thibaut and Izacard, Gautier and Martinet, Xavier and Lachaux, Marie-Anne and Lacroix, Timoth{\\'e}e and Rozi{\\`e}re, Baptiste and Goyal, Naman and Hambro, Eric and Azhar, Faisal and others},\n journal={arXiv preprint arXiv:2302.13971},\n year={2023}\n}\n\n@article{QWen2024qwen2-5,\n\ttitle={{Qwen2.5}},\n\tauthor={Qwen},\n url = {https://github.com/QwenLM/Qwen2.5},\n year={2024}\n}\n\n@article{cobbe2021training,\n title={Training verifiers to solve math word problems},\n author={Cobbe, Karl and Kosaraju, Vineet and Bavarian, Mohammad and Chen, Mark and Jun, Heewoo and Kaiser, Lukasz and Plappert, Matthias and Tworek, Jerry and Hilton, Jacob and Nakano, Reiichiro and others},\n journal={arXiv preprint arXiv:2110.14168},\n year={2021}\n}\n\n@article{abdin2024phi3,\n title={Phi-3 technical report: A highly capable language model locally on your phone},\n author={Abdin, Marah and Jacobs, Sam Ade and Awan, Ammar Ahmad and Aneja, Jyoti and Awadallah, Ahmed and Awadalla, Hany and Bach, Nguyen and Bahree, Amit and Bakhtiari, Arash and Behl, Harkirat and others},\n journal={arXiv preprint arXiv:2404.14219},\n year={2024}\n}\n\n@article{shazeer2020glu,\n title={{GLU} variants improve transformer},\n author={Shazeer, Noam},\n journal={arXiv preprint arXiv:2002.05202},\n year={2020}\n}\n\n@inproceedings{liu2024best,\n title={Best Practices and Lessons Learned on Synthetic Data},\n author={Liu, Ruibo and Wei, Jerry and Liu, Fangyu and Si, Chenglei and Zhang, Yanzhe and Rao, Jinmeng and Zheng, Steven and Peng, Daiyi and Yang, Diyi and Zhou, Denny and others},\n booktitle={Proceedings of COLM},\n year={2024}\n}\n\n@article{xiong2023effective,\n title={Effective long-context scaling of foundation models},\n author={Xiong, Wenhan and Liu, Jingyu and Molybog, Igor and Zhang, Hejia and Bhargava, Prajjwal and Hou, Rui and Martin, Louis and Rungta, Rashi and Sankararaman, Karthik Abinav and Oguz, Barlas and others},\n journal={arXiv preprint arXiv:2309.16039},\n year={2023}\n}\n\n@article{mcaleese2024llm,\n title={{LLM Critics Help Catch LLM Bugs}},\n author={McAleese, Nat and Pokorny, Rai Michael and Uribe, Juan Felipe Ceron and Nitishinskaya, Evgenia and Trebacz, Maja and Leike, Jan},\n journal={arXiv preprint arXiv:2407.00215},\n year={2024}\n}\n\n@article{bai2022training,\n title={Training a helpful and harmless assistant with reinforcement learning from human feedback},\n author={Bai, Yuntao and Jones, Andy and Ndousse, Kamal and Askell, Amanda and Chen, Anna and DasSarma, Nova and Drain, Dawn and Fort, Stanislav and Ganguli, Deep and Henighan, Tom and others},\n journal={arXiv preprint arXiv:2204.05862},\n year={2022}\n}\n\n@article{perez2022red,\n title={Red teaming language models with language models},\n author={Perez, Ethan and Huang, Saffron and Song, Francis and Cai, Trevor and Ring, Roman and Aslanides, John and Glaese, Amelia and McAleese, Nat and Irving, Geoffrey},\n journal={arXiv preprint arXiv:2202.03286},\n year={2022}\n}\n\n@article{wang2024hmoe,\n title={{HMoE}: Heterogeneous mixture of experts for language modeling},\n author={Wang, An and Sun, Xingwu and Xie, Ruobing and Li, Shuaipeng and Zhu, Jiaqi and Yang, Zhen and Zhao, Pinxue and Han, JN and Kang, Zhanhui and Wang, Di and others},\n journal={arXiv preprint arXiv:2408.10681},\n year={2024}\n}\n\n@inproceedings{clark2019boolq,\n title={{BoolQ: Exploring the surprising difficulty of natural yes/no questions}},\n author={Clark, Christopher and Lee, Kenton and Chang, Ming-Wei and Kwiatkowski, Tom and Collins, Michael and Toutanova, Kristina},\n booktitle={Proceedings of NAACL},\n year={2019}\n}\n\n@article{dua2019drop,\n title={{DROP}: A reading comprehension benchmark requiring discrete reasoning over paragraphs},\n author={Dua, Dheeru and Wang, Yizhong and Dasigi, Pradeep and Stanovsky, Gabriel and Singh, Sameer and Gardner, Matt},\n journal={arXiv preprint arXiv:1903.00161},\n year={2019}\n}\n\n@article{hsieh2024rulerwhatsrealcontext,\n title={{RULER}: What's the Real Context Size of Your Long-Context Language Models?},\n author={Hsieh, Cheng-Ping and Sun, Simeng and Kriman, Samuel and Acharya, Shantanu and Rekesh, Dima and Jia, Fei and Ginsburg, Boris},\n journal={arXiv preprint arXiv:2404.06654},\n year={2024}\n}\n\n@article{yuan2024lvevalbalancedlongcontextbenchmark,\n title={{LV-Eval}: A balanced long-context benchmark with 5 length levels up to 256k},\n author={Yuan, Tao and Ning, Xuefei and Zhou, Dong and Yang, Zhijie and Li, Shiyao and Zhuang, Minghui and Tan, Zheyue and Yao, Zhuyu and Lin, Dahua and Li, Boxun and others},\n journal={arXiv preprint arXiv:2402.05136},\n year={2024}\n}\n\n@article{liu2024alignbenchbenchmarkingchinesealignment,\n title={{AlignBench}: Benchmarking chinese alignment of large language models},\n author={Liu, Xiao and Lei, Xuanyu and Wang, Shengyuan and Huang, Yue and Feng, Zhuoer and Wen, Bosi and Cheng, Jiale and Ke, Pei and Xu, Yifan and Tam, Weng Lam and others},\n journal={arXiv preprint arXiv:2311.18743},\n year={2023}\n}\n\n@inproceedings{zheng2023judgingllmasajudgemtbenchchatbot,\n title={{Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena}},\n author={Zheng, Lianmin and Chiang, Wei-Lin and Sheng, Ying and Zhuang, Siyuan and Wu, Zhanghao and Zhuang, Yonghao and Lin, Zi and Li, Zhuohan and Li, Dacheng and Xing, Eric and others},\n booktitle={Proceedings of NeurIPS},\n year={2023}\n}\n\n@article{zhou2023instructionfollowingevaluationlargelanguage,\n title={Instruction-following evaluation for large language models},\n author={Zhou, Jeffrey and Lu, Tianjian and Mishra, Swaroop and Brahma, Siddhartha and Basu, Sujoy and Luan, Yi and Zhou, Denny and Hou, Le},\n journal={arXiv preprint arXiv:2311.07911},\n year={2023}\n}\n\n@misc{li2024crowdsourceddatahighqualitybenchmarks,\n title={From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline}, \n author={Tianle Li and Wei-Lin Chiang and Evan Frick and Lisa Dunlap and Tianhao Wu and Banghua Zhu and Joseph E. Gonzalez and Ion Stoica},\n year={2024},\n eprint={2406.11939},\n archivePrefix={arXiv},\n primaryClass={cs.LG},\n url={https://arxiv.org/abs/2406.11939}, \n}\n\n@misc{dubois2024lengthcontrolledalpacaevalsimpleway,\n title={Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators}, \n author={Yann Dubois and Balázs Galambosi and Percy Liang and Tatsunori B. Hashimoto},\n year={2024},\n eprint={2404.04475},\n archivePrefix={arXiv},\n primaryClass={cs.LG},\n url={https://arxiv.org/abs/2404.04475}, \n}"}}}} |