{"repo": "chuahanchong/researchbrain", "files": {"_KnowledgeHub_/2507.01925.md": {"afad75adf019ab7b4c12f9a8295a3bf696edbfa7": {"t": 1786068568.5091941, "n": 22083, "h": "56b2e409683204fa", "c": "---\nid: \"2507.01925\"\ntitle: \"A Survey on Vision-Language-Action Models: An Action Tokenization Perspective\"\nlink: \"https://www.alphaxiv.org/overview/2507.01925\"\nauthors:\n  - \"Yifan Zhong\"\n  - \"Fengshuo Bai\"\n  - \"Shaofei Cai\"\n  - \"Yuanpei Chen\"\n  - \"Yaodong Yang\"\ntags:\n  - robotics\n  - VLA\n  - survey\naliases:\n  - \"Survey-Vision-Language-Action-Models-Action\"\n---\n# A Survey on Vision-Language-Action Models: An Action Tokenization Perspective\n\n> [!summary] Summary\n> Researchers from Peking University and the PKU-PsiBot Joint Lab propose a unified framework for Vision-Language-Action (VLA) models based on eight action token types, categorizing existing approaches and identifying future directions towards hierarchical architectures combining different token types and improved reasoning.\n\n## Problem\n\n- The rapid growth of Vision-Language-Action (VLA) models lacked a unified framework for understanding their diverse methodologies.\n- Challenges exist in extending large digital AI models to interact effectively with the complex and dynamic physical world.\n- A fragmented research landscape in embodied AI hindered systematic comparison and development of VLA systems.\n\n## Method\n\n- Proposed a \"unified framework\" that abstracts VLA models as processes generating a chain of \"action tokens\" from vision and language inputs.\n- Developed an eight-type taxonomy of action tokens (e.g., language description, code, affordance, raw action) to categorize existing VLA approaches.\n- Systematically analyzed the motivations, representative approaches, properties, advantages, and limitations of each action token type.\n\n## Results\n\n- A unified framework and an eight-type taxonomy of \"action tokens\" (e.g., language description, code, raw action) were established for classifying VLA models.\n- Comprehensive analysis provided specific strengths and limitations for each action token type, informing their suitable application and design.\n- The survey identified key architectural trends, such as hierarchical VLA structures synthesizing multiple token types, and the necessity of a \"Data Pyramid\" for scalable data strategies.\n\n## Takeaways\n\n> [!tip] Key Insights\n> - VLA models can be systematically unified and analyzed by focusing on how they generate and utilize \"action tokens.\"\n> - Future VLA architectures are likely to be hierarchical, strategically synthesizing multiple action token types for improved planning and control.\n> - Advancing VLA models towards general-purpose intelligence necessitates a shift to reinforcement learning, addressing data scarcity with a \"Data Pyramid,\" and improving hardware capabilities.\n\n%%\n## BibTeX\n\n```bibtex\n@misc{zhong2025surveyvisionlanguageactionmodelsaction,\n      title={A Survey on Vision-Language-Action Models: An Action Tokenization Perspective},\n      author={Yifan Zhong and Fengshuo Bai and Shaofei Cai and Xuchuan Huang and Zhang Chen and Xiaowei Zhang and Yuanfei Wang and Shaoyang Guo and Tianrui Guan and Ka Nam Lui and Zhiquan Qi and Yitao Liang and Yuanpei Chen and Yaodong Yang},\n      year={2025},\n      eprint={2507.01925},\n      archivePrefix={arXiv},\n      primaryClass={cs.RO},\n      url={https://arxiv.org/abs/2507.01925},\n}\n```\n%%\n\n## Detailed Report\n\n### 1. How This Work Fits into the Broader Research Landscape\n\n#### 1.1. Executive Summary\n\nThis report analyzes the survey paper \"A Survey on Vision-Language-Action Models: An Action Tokenization Perspective,\" which provides a comprehensive review of the rapidly evolving field of Vision-Language-Action (VLA) models. The authors, primarily from Peking University and the PKU-PsiBot Joint Lab, propose a novel unified framework for understanding diverse VLA models through the lens of \"action tokenization.\" They categorize existing approaches into eight distinct types of action tokens: language description, code, affordance, trajectory, goal state, latent representation, raw action, and reasoning.\n\nThe paper's key objectives are to systematically categorize VLA research, distill the strengths and limitations of each action token type, and identify promising future directions. It argues that VLA models, representing the next frontier in AI, require a synergistic co-evolution of models, data, and hardware to achieve general-purpose intelligence in the physical world. The authors conclude that the future of VLA models likely lies in hierarchical architectures that strategically synthesize multiple action token types, particularly emphasizing the integration of robust reasoning capabilities and a shift from imitation learning to more efficient reinforcement learning, alongside addressing critical data scarcity and hardware limitations.\n\nThis survey paper is situated at the cutting edge of artificial intelligence research, specifically at the intersection of large foundation models and embodied AI. The authors meticulously trace the evolution of AI from **Large Language Models (LLMs)** like GPT-4 and DeepSeek-R1, which excel in linguistic understanding and generation, to **Vision Foundation Models (VFMs)** such as CLIP, DINOv2, and SAM, which have revolutionized visual perception. Building upon these unimodal advancements, **Vision-Language Models (VLMs)**, exemplified by GPT-4o and Qwen2.5-VL, have integrated multimodal understanding and reasoning.\n\nThe paper argues that despite their impressive capabilities, these foundation models remain largely \"digital AI,\" confined to simulated or abstract computational environments. The natural progression, and indeed the \"next frontier,\" is to extend this intelligence into the \"physical world,\" giving rise to **Vision-Language-Action (VLA) models**. VLA models are defined as systems that generate executable actions conditioned on visual and linguistic inputs, built upon at least one large-scale vision or language foundation model.\n\nThis work fits into the broader landscape by:\n- **Bridging Digital and Embodied AI:** It explicitly positions VLA models as the crucial link for extending digital AI's intelligence into physical interaction, directly addressing the limitations of models confined to purely digital domains.\n- **Synthesizing Diverse Approaches:** The rapid proliferation of VLA models has led to seemingly disparate methodologies. This survey provides a unifying framework by identifying the common underlying principle of \"action tokenization,\" thereby bringing coherence to a fragmented field.\n- **Addressing Embodied AI Challenges:** The paper acknowledges that embodied AI introduces significantly greater complexity than digital AI, dealing with unstructured, open-ended, and dynamic physical environments, as well as the inherent limitations of robotic hardware. VLA models are presented as a promising path to tackle these long-standing challenges like data scarcity and poor cross-embodiment transferability.\n- **Guiding Future Research towards AGI:** By systematically reviewing the state-of-the-art and identifying key trends and underexplored areas, the survey aims to accelerate progress toward Artificial General Intelligence (AGI), which the authors depict as encompassing both digital and embodied AI.\n\nIn essence, the survey acts as a crucial roadmap for researchers navigating the complex and rapidly expanding domain of embodied AI, offering a principled way to analyze, compare, and develop next-generation VLA systems.\n\n### 2. Key Objectives and Motivation\n\nThe primary objectives and motivations for this survey paper are clearly articulated in its introduction and executive summary:\n\n1. **Categorize and Interpret Existing VLA Research through Action Tokenization:** The core objective is to impose structure on the diverse and rapidly growing body of VLA research. The authors observe that despite seemingly varied approaches, VLA models can be unified under a single framework where vision and language inputs are processed to produce a chain of \"action tokens\" that progressively encode actionable information. The survey aims to define and categorize these action tokens into eight principal types, providing a common language and analytical lens for the field.\n\n2. **Distill Strengths and Limitations of Each Action Token Type:** For each identified category of action token (language description, code, affordance, trajectory, goal state, latent representation, raw action, and reasoning), the survey aims to meticulously analyze their respective advantages, disadvantages, and the contexts in which they are most effective. This distillation is crucial for informed design choices in future VLA model development.\n\n3. **Identify Areas for Improvement and Highlight Promising Directions:** By systematically reviewing the current landscape, the survey seeks to pinpoint underexplored challenges and promising avenues for future research. This includes architectural trends, data scaling strategies, and the integration of advanced learning paradigms like reinforcement learning.\n\n4. **Offer a Synthesized Outlook on VLA Model Evolution:** The authors intend to provide a high-level perspective on how VLA models are evolving, drawing connections between the design of action tokens and broader trends in model architecture, data requirements, and application domains.\n\nThe overarching **motivation** for this survey stems from the observed \"lack of comprehensive understanding regarding action tokens,\" which significantly impedes effective VLA development. The authors believe that by clarifying these fundamental design choices, they can \"contribute guidance for future research, hoping to bring the field closer to general-purpose intelligence.\" They emphasize the urgency of this review due to the rapid progress and proliferation of VLA models, underscoring the need for a systematic framework to organize and advance the field.\n\n### 3. Methodology and Approach\n\nAs a comprehensive survey, the paper's methodology is primarily a systematic review and conceptual framework development, rather than experimental validation. The authors' approach can be broken down into the following key steps:\n\n1. **Defining the Scope:** The survey focuses on \"Vision-Language-Action (VLA) models,\" which are specifically defined as models that generate actions conditioned on visual and linguistic inputs, and are built upon at least one large-scale vision or language foundation model. This clear definition sets the boundaries for the surveyed literature.\n\n2. **Proposing a Unified Framework:** The central methodological contribution is the introduction of a \"unified framework\" for VLA models. This framework posits that VLA models, regardless of their superficial diversity, can be abstracted as a process where:\n - Vision and language inputs are processed.\n - Through a series of \"VLA modules\" (maximal differentiable subnetworks or non-differentiable functional units).\n - Producing a chain of \"action tokens\" (outputs of VLA modules or semantically meaningful intermediate representations).\n - Which progressively encode more grounded and actionable information.\n - Ultimately generating executable actions.\n\n3. **Developing an Action Token Taxonomy:** Based on their unified framework, the authors systematically identify and categorize the primary design choices distinguishing VLA models, which they term \"action tokens.\" They propose an eight-type taxonomy:\n - Language Description (Language Plan, Language Motion)\n - Code\n - Affordance (Keypoint, Bounding Box, Segmentation Mask, Affordance Map)\n - Trajectory (Point Trajectory, Visual Trajectory, Optical Flow)\n - Goal State (Single-Frame Image, Multi-Frame Video)\n - Latent Representation (Vision-Based, Action-Based, Goal-Based)\n - Raw Action\n - Reasoning\n\n4. **Systematic Literature Review and Analysis:** For each of the eight action token categories, the authors conduct a detailed review of representative research papers. For each paper, they analyze:\n - **Motivation:** Why was this type of action token chosen or developed?\n - **Representative Approaches:** Key models and their architectural designs.\n - **Properties:** Characteristics of the token.\n - **Advantages:** Benefits of using this token type.\n - **Limitations:** Challenges and drawbacks.\n - **Future Directions:** Potential areas for improvement and further research.\n - **Contextual Information:** \"Previous Module\" and \"Next Module\" to understand the token's place in the overall VLA pipeline, and task/embodiment specifics.\n\n5. **Identifying Scalable Data Sources:** A dedicated section (Section 12) reviews the various data sources used to train VLA models, categorizing them into a \"Data Pyramid\" (web data/human videos, synthetic/simulation data, real-world robot data). This analysis supports the understanding of how different action tokens leverage diverse data.\n\n6. **Synthesizing Trends and Future Outlook:** Finally, the authors synthesize their findings to present broader trends in action token development, VLA architecture, learning paradigms (imitation vs. reinforcement learning), hardware requirements, and critical ethical considerations (safety and alignment). They propose a future vision for VLA agents and discuss the fundamental challenge of data scalability.\n\nThrough this systematic and conceptual approach, the survey aims to provide a structured, comprehensive, and forward-looking analysis of the VLA field.\n\n### 4. Main Findings and Results\n\nThe survey presents a rich set of findings derived from its systematic categorization and analysis of VLA models through the lens of action tokenization.\n\n**Unified Framework and Action Token Taxonomy:**\nThe central finding is that VLA models can be abstracted into a unified framework: vision and language inputs are processed by VLA modules, producing a chain of action tokens that progressively encode grounded and actionable information, culminating in executable actions. The key distinguishing factor among VLA models lies in how these action tokens are formulated, leading to the proposed eight-type taxonomy.\n\n**Strengths and Limitations of Each Action Token Type:**\n\n- **Language Description:**\n - **Strengths:** Seamless integration with LLMs/VLMs, abundance of co-training data, essential for long-horizon planning, highly interpretable for human oversight.\n - **Limitations:** Imperfect expressiveness (ambiguous, lacks fine-grained control), high inference latency due to large model size.\n- **Code:**\n - **Strengths:** Clear logical structures for planning and control, leverages rich third-party libraries (e.g., NumPy), transparent bridge to low-level APIs.\n - **Limitations:** Over-reliance on predefined APIs (limited adaptability to novel situations), execution brittleness (sensitive to environmental variations, safety risks).\n- **Affordance:**\n - **Strengths:** Spatially grounded and interaction-centric (e.g., keypoints, bounding boxes, segmentation masks, maps), enables cross-platform generalization by abstracting control.\n - **Limitations:** Primarily 2D representation (lacks true 3D understanding), static nature (doesn't model temporal evolution of affordances), vulnerable to visual noise (occlusion, blur).\n- **Trajectory:**\n - **Strengths:** Learnable from abundant off-domain human videos (data scalability), strong cross-task generalization for similar motion patterns, interpretable.\n - **Limitations:** Limited 3D expressiveness (often 2D), lacks explicit orientation info, computationally expensive to generate, insufficient for complex interaction logic.\n- **Goal State:**\n - **Strengths:** High data scalability via hindsight relabeling and action-free videos, strong visual guidance for lower-level policies, task specificity, robust interpretability, straightforward evaluation.\n - **Limitations:** Challenging to generate high-quality/consistent goal states (overspecification, inaccuracies), high inference latency (computationally intensive).\n- **Latent Representation:**\n - **Strengths:** High data scalability from action-free human videos and cross-embodiment data, strong expressive potential (compact, implicit semantics, multimodal integration), training efficiency for VLMs.\n - **Limitations:** Uninterpretable (black-box nature), challenges in achieving appropriate granularity, comprehensiveness, and task-centric alignment.\n- **Raw Action:**\n - **Strengths:** Minimal human knowledge and annotation required, favors end-to-end learning, aligns with LLM scaling laws, efficient fine-tuning.\n - **Limitations:** Severe data scarcity (costly to collect), poor direct cross-embodiment generalization, high latency for long-term control.\n- **Reasoning:**\n - **Strengths:** Bridges instruction-action gap, enhances generalization (especially for complex/long-horizon tasks), improves interpretability and human-in-the-loop capabilities, enables cross-embodiment capability.\n - **Limitations:** Increased inference time/reduced execution speed, often relies on fixed reasoning steps, costly to build high-quality datasets.\n\n**Key Trends Identified:**\n\n- **Strategic Synthesis of Action Tokens:** The future of VLA models lies in combining different token types rather than a single dominant one.\n- **Hierarchical Architectures:** A promising VLA architecture is hierarchical:\n - **Top Layer:** Language plans and code for long-horizon planning and logical control.\n - **Intermediate Layer:** 3D affordance, trajectory, and goal state for precise and interpretable motion representations.\n - **Bottom Layer:** Policy module mapping vision-based representations to raw actions.\n- **Evolution of Reasoning:** Reasoning will shift from purely language-based to action-token-based, integrated throughout the VLA hierarchy, adapting computation based on task complexity.\n- **Data Pyramid:** A multi-layered data strategy, combining web/human videos (bottom), synthetic/simulation data (middle), and real-world robot data (top), is crucial for scalable and transferable learning.\n- **From Imitation to Reinforcement Learning:** A future trend involves moving beyond imitation learning towards RL for more robust, dexterous, and high-success-rate behaviors, requiring efficient RL algorithms and automated reward function generation.\n- **VLA Models to VLA Agents:** A shift from reactive models to proactive agents with broader cognitive architectures (memory, exploration, planning, reflection) and non-linear, bidirectional topologies.\n\n### 5. Significance and Potential Impact\n\nThis survey paper holds significant importance for the advancement of embodied AI and the broader field of Artificial General Intelligence (AGI). Its potential impact can be summarized as follows:\n\n1. **Providing a Foundational Conceptual Framework:** By introducing the \"action tokenization perspective\" and a unified framework, the paper offers a much-needed conceptual anchor for a rapidly expanding and diverse field. This framework helps researchers systematically understand, compare, and design VLA models, bringing coherence to an otherwise fragmented landscape. This clarity can accelerate research by enabling more targeted investigations and facilitating communication across different sub-domains of embodied AI.\n\n2. **Guiding Future Research Directions:** The detailed analysis of each action token's strengths and limitations, coupled with explicit future directions, serves as a comprehensive roadmap for researchers. It highlights critical challenges (e.g., interpretability of latent actions, brittleness of code, 3D understanding for affordances, data scarcity) and points towards promising avenues (e.g., hierarchical architectures, temporal affordance dynamics, in-context reinforcement learning, multi-modal data collection). This direct guidance can help optimize research efforts and avoid redundant exploration.\n\n3. **Informing Architectural Design and Data Strategies:** The survey's insights into the optimal combination of action tokens for different layers of a hierarchical VLA architecture provide practical guidance for system design. Furthermore, the emphasis on the \"Data Pyramid\" underscores the necessity of leveraging diverse data sources—from web-scale videos to high-fidelity robot data—to achieve scalable and generalizable policies. This data-centric perspective is crucial for building robust real-world systems.\n\n4. **Emphasizing Critical Unexplored Areas:** The paper draws attention to several vital, yet underexplored, aspects of VLA research:\n - **Transition to VLA Agents:** Highlighting the need for cognitive architectures beyond perception-action mapping (memory, planning, reflection) pushes the field towards more sophisticated, adaptive, and autonomous systems.\n - **Reinforcement Learning for Embodied AI:** Advocating for a shift from imitation learning to RL, while acknowledging its challenges (high reset costs, low interaction efficiency, safety), stresses the importance of learning from trial-and-error for true dexterity and robustness.\n - **Hardware and Modality Expansion:** The call for dexterous hands and broader sensory modalities (tactile, audio, olfactory) underscores the current limitations of robotic platforms and encourages interdisciplinary research in hardware and sensor development.\n - **Safety and Alignment:** Explicitly raising safety as a first-class concern emphasizes the ethical imperative in deploying AI that interacts with the physical world, pushing for proactive safety-aware methodologies.\n\n5. **Accelerating Progress Towards AGI:** Ultimately, by clarifying the path forward for VLA models, the survey aims to accelerate the development of \"general-purpose intelligence\" that can effectively operate in and interact with the physical world. It positions VLA models as a central component in the pursuit of AGI, suggesting that advances in this field are crucial for unlocking the full potential of AI.\n\nIn summary, this survey is not merely a review; it's a strategic document that synthesizes current knowledge, critically evaluates methodologies, and proactively shapes the future trajectory of embodied AI research, making a substantial contribution to bringing AI closer to real-world deployment and general intelligence.\n"}}}}