1 Introduction↩︎

Teaching robots to perform manipulation tasks has long required extensive task-specific training data and engineering effort, with each new task demanding carefully curated demonstrations [1]–[3] or hand-engineered reward functions [4], [5]. The emergence of large vision-language models (VLMs)—pretrained on internet-scale visual and linguistic data—offers a compelling alternative: if such models encode broad semantic knowledge about objects, instructions, and goals, they may provide a generalizable foundation for robot control [6]–[8].

Since RT-2 demonstrated that pretrained VLMs can be repurposed as robot policies [9], the field has rapidly converged on the Vision-Language-Action (VLA) paradigm—end-to-end architectures that map visual observations and language instructions directly to action commands. As illustrated in Figure 1, subsequent systems largely follow this formulation while exploring variations in output parameterization and control interfaces, forming several architectural lineages, including discrete token policies, diffusion-based policies, and dual-system designs. Despite these differences, they share a common structural core: a pretrained VLM backbone that mediates perception, language grounding, and action generation, and an implicit assumption—introduced by RT-2 and inherited by subsequent work—that semantic representations learned from large-scale vision-language pretraining are sufficient to guide physical action decisions. Under this premise, improvements in semantic generalization are expected to translate into more robust physical execution, motivating extensions along multiple axes, including diffusion-based action modeling, increased model scale, and additional modalities such as proprioception. This trajectory is reflected in systems such as OpenVLA [10], OpenVLA-OFT [11], \(\pi_0\) [12], and GR00T N1 [13], which report steady benchmark improvements while maintaining this shared architectural and conceptual core.

However, task success measures outcomes, not the mechanism producing them. A central unresolved question is whether semantic generalization learned by VLMs transfers to physical execution generalization in robotic systems. These correspond to two distinct components: semantic generalization, which maps observations and instructions to task-relevant abstractions, and physical execution, which selects actions under constraints of dynamics, contact, embodiment, and temporal interaction.

We argue that current VLA research relies on an implicit but unverified assumption: that semantic generalization is sufficient to support physical execution. This assumption has been reinforced through what we term narrative drift, whereby successive systems inherit and extend prior interpretations of performance gains without explicit causal verification. Crucially, existing evaluation protocols do not provide the signals necessary to distinguish semantic generalization from physical execution, and therefore do not support causal attribution of performance improvements.

We develop this argument as follows. Section 2 formalizes the implicit assumption underlying VLM-backbone VLA research. Section 3 shows that this assumption is not identifiable under current evaluation protocols. Section 4 analyzes the resulting systemic consequences. Section 5 proposes directions for evaluation reform. Section 6 concludes.

2 The Implicit Assumption Underlying VLM-Backbone VLA Research↩︎

We formalize the distinction introduced in Section 1 by decomposing a VLA policy—mapping observations and instructions to actions—into two components. Semantic mapping transforms visual observations and language instructions into task-relevant representations (e.g., object categories, spatial relations, and intended goals), whereas physical action decision selects actions conditioned on this representation under constraints of dynamics, contact interactions, embodiment, and temporal execution. A capability is physical if, holding the semantic interpretation fixed, variations in physical context (e.g., object mass, friction, geometry, or embodiment) require different actions for successful task completion. This distinction induces a criterion for attribution: improvements in task success can only be attributed to physical generalization if they remain robust under controlled variation of physical conditions that do not alter task semantics. It also implies distinct failure modes: semantic errors lead to incorrect task interpretation, while physical failures arise when the task is correctly identified but execution fails due to unmodeled physical factors—failure modes that are not distinguishable from task success rate alone.

We formalize this decomposition as: \[a_t \sim \pi\!\left(a_t \;\middle|\; z^{\text{sem}},\, z^{\text{phys}}\right) \label{eq:policy}\tag{1}\] where \(z^{\text{sem}} = f_{\text{vlm}}(o_t, l)\) denotes the semantic representation extracted by the VLM backbone from current observation \(o_t\) and language instruction \(l\), and \(z^{\text{phys}}\) denotes physical context—properties such as mass, friction, and contact geometry that are not recoverable from static image-text data. The semantic sufficiency assumption implicit in current VLA architectures is: \[\pi\!\left(a_t \mid z^{\text{sem}},\, z^{\text{phys}}\right) \approx \pi\!\left(a_t \mid z^{\text{sem}}\right) \label{eq:assumption}\tag{2}\]

We decompose \(z^{\text{sem}}\) into three constituent representations. Categorical representations \(z^{\text{cat}}\) encode object identity, category membership, and visual appearance; they are acquired through large-scale image–text contrastive training, in which visual and linguistic signals are aligned via co-occurrence supervision [14], [15]. Relational representations \(z^{\text{rel}}\) encode spatial configurations, affordance descriptions, and scene-level structure; they are learned through grounding and captioning objectives that map linguistic relational expressions to visual scene layouts [15], [16]. Goal representations \(z^{\text{goal}}\) encode task intent, sub-goal decomposition, and success criteria; they are derived from instruction-following and, in recent systems, chain-of-thought supervision over language and image–text pairs [9], [10]. Each component is acquired from static image-text data: visual and linguistic signals are jointly available, but action consequences are absent from the supervision signal throughout [14], [15].

This shared supervision structure defines the information boundary of \(z^{\text{sem}}\). Physical properties such as mass distribution, surface friction, and contact geometry are not recoverable from static visual observation: two objects may be visually indistinguishable while differing substantially in physical behavior [17], [18]. These properties are only revealed through interaction—through the consequences of applied force, the resistance of contact, and the dynamics of motion [19], [20]. Because the training signal for \(f_{\text{vlm}}\) contains no such interaction outcomes, \(z^{\text{sem}}\) cannot, in principle, encode \(z^{\text{phys}}\) regardless of model scale or data volume [21], [22]. The semantic sufficiency assumption in Equation 2 therefore does not follow from the properties of VLM pretraining: it asserts that \(z^{\text{sem}}\) is sufficient for physical action decisions, but the supervision structure of \(f_{\text{vlm}}\) excludes precisely the signal that would be required for this sufficiency to hold.

Unlike \(z^{\text{sem}}\), which is explicitly computed via VLM encoding, current architectures provide no corresponding mechanism for acquiring \(z^{\text{phys}}\), making 2 both structurally unjustified and empirically untested.

Despite substantial variation in output representations—discrete token policies trading action fidelity for semantic expressiveness [23]–[28], diffusion-based policies improving action continuity and precision [29]–[35], and dual-system architectures decoupling reasoning latency from execution speed [36]–[38]—the architectural core remains unchanged: all systems rely on a pretrained VLM backbone, with modifications concentrated in output heads and input modalities [10]–[12]. These design choices address execution limitations without isolating whether semantic representations are sufficient to support physical action decisions.

Nevertheless, capability claims have escalated alongside this unchanged core. Early work positioned VLMs as transferring web-scale semantic knowledge to robot control [9]. Subsequent systems extended this to cross-task and cross-embodiment generalization [10], [11], while later work attributed semantic reasoning and problem-solving capabilities to VLM backbones in support of complex manipulation [12]. More recent systems further suggest that scaling VLM backbones improves spatial reasoning and physical adaptation [13]. Across these claims, a component that has not fundamentally changed has been credited with progressively stronger capabilities.

Underlying this trajectory is a common but unverified condition: that improvements in semantic representations learned through VLM pretraining translate into robustness of physical action decisions. This condition is not necessarily stated explicitly in the cited works, but is required for their capability claims to hold under a causal interpretation. We refer to its progressive reinforcement across systems as narrative drift.

This pattern extends beyond the systems in Table 1 [23], [27], [39], [40]. Across a broader range of VLM-backbone VLA works, improvements in task success rate—measured under largely unchanged evaluation protocols—are interpreted as evidence of increasingly strong physical execution generalization [41]–[45]. However, as introduced in Section 1, task success rate does not distinguish between semantic and physical contributions. Consequently, the same observed improvements are consistent with multiple explanations, including genuine physical generalization, improved semantic matching, or increased overlap between training and test distributions.

The consequence is not that the assumption is untestable in principle, but that it is not independently identifiable within the current methodological framework. Existing evidence does not permit attribution of performance gains to semantic or physical capabilities separately, and therefore cannot establish whether semantic representations are sufficient to support physical action decisions.

3 The Structural Argument↩︎

3.1 Semantic and Physical Representations Are Structurally Distinct↩︎

The assumption identified in Section 2—that semantic representations are sufficient to support physical action decisions—depends on whether the two representation types impose equivalent learning requirements. We argue that they do not.

Semantic representations encode what objects are and what instructions mean, including object identity, category membership, and relational descriptions grounded in language and visual appearance [14]–[16]. Such representations can be acquired from large-scale image-text co-occurrence, where visual and linguistic signals are jointly available.

Physical representations encode how objects behave under interaction, including properties such as mass distribution, friction, contact geometry, and dynamic response to force [46]. These properties are not recoverable from static image-text data, as the consequences of action are absent from the training signal [17], [21]. Increasing data scale does not resolve this limitation: supervision over physical state transitions is not present in the modality.

The two representation types therefore differ in their supervision requirements. Semantic representations can be learned from passive observation, whereas physical representations require interaction-based feedback over action outcomes [19], [20]. A model trained exclusively on the former cannot be assumed to have acquired the latter.

This distinction implies that semantic generalization and physical execution generalization can dissociate. A system may correctly interpret the task while failing to execute it under novel physical conditions [9], [11]. The assumption that semantic representations are sufficient for physical action decisions therefore does not follow from the properties of the training data or learning objective.

3.2 Three Levels of Non-Identifiability in Current Evaluation↩︎

Current benchmarks fail to support causal attribution at three distinct levels. These levels are not independent: each deeper level is revealed only when the shallower one is examined, and together they indicate that the evaluation gap is more fundamental than it first appears.

3.2.0.1 Level 1: Attribution Non-Identifiability.

Current robot manipulation benchmarks evaluate performance primarily through task success rate [10], [13], [47], [48]. This metric does not distinguish between semantic and physical sources of success or failure. Successful task completion may result from correct semantic grounding, physically robust execution, or overlap between training and test distributions. Failures may arise from incorrect task interpretation or from inability to execute under novel physical conditions.

Staged evaluation protocols improve localization by decomposing tasks into phases [11], [49], but do not resolve this ambiguity. A failure at a specific stage remains consistent with both semantic misinterpretation and physical execution error. These protocols identify where failure occurs, but not why.

A critical limitation is that current benchmarks do not provide controlled interventions that invalidate semantic matching while preserving task semantics. As a result, policies that rely on distributional alignment between instruction–observation pairs and action patterns cannot be distinguished from those that model underlying physical dynamics. Attribution of performance to semantic matching or physical generalization is therefore fundamentally underdetermined.

However, even if attribution could be resolved, a deeper question remains: where does the observed generalization come from?

3.2.0.2 Level 2: Source Non-Identifiability.

Even if attribution were resolved, the source of generalization would remain unidentifiable. Performance gains may arise from at least three sources—semantic priors from VLM pretraining, physical understanding from robot interaction data, or their interaction—yet current evaluation protocols control for none of them.

A concrete manifestation appears in the definition of out-of-distribution (OOD). OOD status is typically defined relative to robot demonstration data, not VLM pretraining data [9], [10], [12]. Objects novel to the robot dataset may be familiar from pretraining, introducing an uncontrolled confound. Combined with task-specific fine-tuning, this creates a mismatch between nominal and actual sources of generalization.

As a result, observed improvements remain ambiguous in origin. Gains attributed to physical generalization may instead reflect prior semantic exposure or distributional overlap. Because current evaluation protocols neither control pretraining exposure nor isolate interaction-driven learning, they do not permit disentangling these sources.

This leads to source non-identifiability: the origin of generalization cannot be determined from observed outcomes.

3.2.0.3 Level 3: Representation-Level Non-Identifiability.

Current benchmarks implicitly assume that all instructions are valid and executable within the agent’s physical context. Under this assumption, evaluation reduces to conditional action generation given a feasible task, and does not test whether the task itself should or can be executed.

This obscures the fate of capabilities inherited from pretrained vision-language backbones. Prior work shows that such models exhibit forms of constraint-aware reasoning and feasibility judgment in embodied settings—including scoring action affordances [50], replanning after failures [51], and recognizing infeasible instructions [52]. However, it remains unclear whether these capabilities are preserved, degraded, or ignored during fine-tuning into action policies. This concern is supported by evidence that fine-tuning can overwrite pretrained representations [53]–[56].

Because current benchmarks exclude infeasible, unsafe, or contradictory instructions, they provide no signal to detect the presence or absence of such capabilities. As a result, representation-level changes during policy learning remain unobservable.

This leads to representation-level non-identifiability: the effect of training on pre-existing capabilities cannot be measured, and potential degradation remains undetected.

Taken together, these three levels show that non-identifiability is not merely a measurement limitation but a structural property of the current evaluation paradigm. Attribution cannot be resolved at the level of outcomes, the source of generalization cannot be isolated across training signals, and changes in underlying representations cannot be observed during learning.

As a result, multiple latent factors—including semantic grounding, physical reasoning, and representation-level capabilities—are collapsed into a single observable metric. This prevents causal attribution and renders the central assumption of this paper—that semantic generalization is sufficient for physical execution—empirically untestable under current evaluation protocols.

3.3 Empirical Signatures of Underdetermined Attribution↩︎

The three levels of non-identifiability identified above are not merely theoretical. If evaluation systematically cannot distinguish attribution, source, and representation-level effects, the literature should exhibit a predictable empirical signature: physical execution failures are observed but attributed to engineering factors, while the alternative explanation—failure of the semantic sufficiency assumption—is not evaluated.

Table 2 illustrates this pattern across representative systems. In each case, observed limitations are consistent with failure of physical execution under novel conditions, yet are attributed to data coverage, model scale, or training procedure [9]–[12]. These attributions are plausible given the available evidence, but not uniquely supported by it.

This pattern admits two explanations. The first is that independent research groups consistently overlook the same alternative interpretation. The second is structural: if evaluation does not provide signals to distinguish semantic and physical failure, alternative explanations cannot be formulated or tested. The observed consistency of engineering attribution across systems supports the latter.

Under this interpretation, attribution in the literature reflects the limits of the evaluation framework rather than definitive evidence about underlying causes.

4 Systemic Consequences↩︎

The semantic sufficiency assumption, when combined with an evaluation framework that does not provide identifiable signals for detecting its failure, produces three systemic consequences that compound over time.

4.0.0.1 Misallocation of Research Resources.

When evaluation cannot attribute performance gains to their source, research investment decisions are made without reliable diagnostic signals [57], [58]. The field cannot determine whether observed improvements reflect advances in semantic generalization, physical execution generalization, or distributional overlap—yet resource allocation depends precisely on this distinction [59], [60].

Investment in larger VLM backbones is justified if semantic capability transfers to physical execution; it is unjustified if the transfer does not occur and gains instead reflect distributional overlap. Likewise, investment in data collection depends on whether the bottleneck lies in semantic coverage or physical interaction diversity—two distinct requirements that existing benchmarks do not separately measure [4], [49], [61], [62].

This issue connects to a broader principle in the study of scaling. It is often argued that increasing scale alone is sufficient to achieve higher capability, with different methods primarily affecting efficiency rather than qualitative behavior [63]–[65]. Under this view, evaluation plays a critical role: it determines whether scaling is applied to the correct underlying mechanism. If evaluation cannot distinguish between competing explanations of performance, scaling may amplify improvements along directions that do not correspond to the intended capability.

4.0.0.2 False Confidence in Deployment Readiness.

Benchmark performance is the primary basis on which VLA systems are assessed for deployment readiness [9], [66]. If benchmark success rates reflect distributional overlap rather than physical execution generalization, this confidence may not be warranted by the available evidence. OpenVLA-OFT reports near-perfect success rates on constrained tabletop tasks while the same system fails substantially when task-irrelevant distractor objects are introduced [11]: a perturbation that does not alter task semantics produces significant performance degradation.

This gap between benchmark performance and robustness to minimal perturbations is not reliably captured by current evaluation frameworks. A system can accumulate a strong benchmark record while its physical execution generalization remains unverified. In deployment contexts where physical configurations cannot be controlled, failure modes that benchmarks cannot detect become consequential: systems that succeed under training distribution overlap may fail when that overlap disappears, without prior evaluation signals that predict or characterize such failures.

4.0.0.3 Persistent Misdiagnosis of Architectural Failures.

The attribution pattern identified in Section 3.3—physical execution limitations consistently attributed to engineering factors—is not merely a historical observation. It reflects a structural limitation of a field that lacks reliable tools to distinguish architectural causes from data or optimization effects [59].

When failures are attributed to insufficient data coverage, the proposed solution is more data. When failures are attributed to model capacity, the solution is larger models. When failures are attributed to action parameterization, the solution is alternative output representations. Each of these responses is reasonable under an engineering diagnosis. However, if the underlying cause is an unverified architectural assumption, these responses may leave the root issue unaddressed while continuing to consume resources.

The cost of persistent misdiagnosis is not limited to inefficiency. It produces a compounding divergence between the problems the field believes it is solving and the problems that must be solved for robust physical deployment. Systems may exhibit consistent improvements in benchmark performance while relying on mechanisms that do not generalize under physically novel conditions [60]. Without evaluation signals that support causal attribution, progress becomes difficult to distinguish from the accumulation of distributional alignment.

5 Forward Path↩︎

The consequences identified in Section 4 arise from a single diagnostic limitation: the absence of identifiable signals for separately measuring semantic generalization and physical execution generalization. Addressing this limitation does not primarily require new model architectures; it requires evaluation designs that treat these two capabilities as distinct variables.

5.0.0.1 Evaluation Reform.

Separating semantic and physical generalization requires experimental designs that independently manipulate semantic content and physical configuration. Two complementary interventions enable this separation.

To evaluate semantic generalization, physical conditions are held constant while semantic content varies. This can be achieved by altering linguistic descriptions of the same scene, or by introducing objects with visually similar appearance but different task-relevant properties.

To evaluate physical execution generalization, semantic content is held constant while physical configuration varies. The same instruction and object category are presented under novel poses, surface conditions, or dynamics, requiring different action strategies for successful execution.

The key requirement is controlled variation: one factor must be held fixed while the other is systematically perturbed. Increasing dataset scale or benchmark diversity alone does not resolve the attribution problem if these factors remain entangled.

A particularly tractable instantiation of this design uses object properties that are linguistically expressible but not visually recoverable—such as mass, friction, or material composition [18], [67], [68]. A model that correctly grounds expressions such as “the heavier cup” or “the slippery object” provides evidence of semantic generalization that cannot be reduced to visual pattern matching [60], [69].

Evaluating whether this grounded representation leads to physically correct manipulation establishes a conditional test: semantic grounding is verified independently, and physical execution is evaluated given correct grounding [19]. This design introduces two observable outcomes— semantic correctness and physical success—rather than collapsing both into a single task success signal. Importantly, this protocol does not require access to model internals; it relies only on observable behavior under controlled interventions [70].

In addition, out-of-distribution status should be defined relative to all training data sources, including VLM pretraining corpora, rather than only robot demonstration datasets [9], [10], [66]. Without this distinction, nominally novel objects may remain familiar to the model, preventing attribution of performance gains to physical generalization.

5.0.0.2 Reorienting the Role of VLM Backbones.

Recent work has highlighted a gap between high-level semantic reasoning and the requirements of physical execution, emphasizing the importance of three-dimensional structure, contact, and spatial relations in manipulation [71]–[73]. However, under current evaluation protocols, it remains unclear whether observed limitations arise from insufficient physical modeling or from confounding factors such as distributional overlap.

Evaluation designs that enable attribution have a direct implication for the role of VLM backbones. Once semantic and physical generalization can be measured separately, the contribution of VLM pretraining to each can be empirically assessed rather than assumed. This may reveal that VLM backbones provide strong semantic interfaces—grounding open-vocabulary instructions to visual targets [14], [15], [74]—while physical execution generalization depends on mechanisms not captured by passive image-text training [22], [75].

This perspective suggests a shift in architectural framing. Rather than treating VLM backbones as the foundation from which physical capability is expected to emerge, systems can be designed with explicitly separated components: a semantic interface for grounding language and perception, and a physical module for modeling action-conditioned dynamics [76]–[78]. On such an architecture, the semantic sufficiency assumption need not be adopted from the outset, because the two components are designed to serve distinct functions independently [70].

This separation has precedent in developmental cognition. Infants exhibit sensitivity to physical properties such as object permanence and causal interaction prior to acquiring language [22], [79]. Physical reasoning, on this account, develops as a distinct representational capacity rather than as a derivative of semantic knowledge.

This reframing does not prescribe a specific architecture. It shifts the design objective: from adapting semantic models to perform physical execution, to constructing systems in which physical execution generalization is measurable, attributable, and improvable independently of semantic grounding. Under such conditions, the contribution of VLM-based representations to physical performance becomes an empirical question rather than an assumption.

6 Conclusion↩︎

This paper examined a foundational but unverified assumption underlying recent VLM-backbone VLA research: that semantic generalization learned from internet-scale data transfers to physical execution generalization in embodied systems. We showed that, under current evaluation protocols, this assumption is not independently identifiable. Task success rate—despite being the dominant metric—does not distinguish between semantic grounding and physical execution, and therefore does not support causal attribution of performance gains.

Reinterpreted through this lens, the steady improvement of benchmark results across VLA systems does not, by itself, establish progress in physical reasoning. Because semantic and physical factors remain entangled in both training and evaluation, the same empirical outcomes are consistent with multiple competing explanations, including improved semantic matching and distributional overlap. What appears as evidence of increasingly general physical capability may instead reflect the limits of what current benchmarks are able to measure.

The implication is methodological rather than architectural. Progress on physical execution generalization requires evaluation designs that introduce controlled variation, allowing semantic and physical contributions to be measured separately. This shift also reframes system design: instead of assuming that physical competence emerges from semantic representations, architectures can be structured to treat physical state modeling and semantic grounding as distinct components from the outset, enabling their roles to be empirically assessed rather than inferred.

More broadly, this analysis highlights a general principle for embodied AI: when evaluation does not provide identifiable signals for key capabilities, implicit assumptions can become entrenched as de facto premises of a field. This limitation has implications beyond attribution: it also constrains which capabilities are expressed and measured. Pretrained VLMs exhibit abilities—feasibility judgment, constraint awareness, context-sensitive refusal—that fine-tuning into action policies may degrade, yet task completion benchmarks provide no signal to detect such loss. Progress along the task completion axis alone does not guarantee movement toward general embodied intelligence. In this context, our contribution is not to refute the semantic sufficiency assumption, but to show that it has not yet been tested in a way that would allow it to be confirmed or rejected. Establishing such tests is a necessary step toward grounding claims of generalization in mechanisms that can be measured, attributed, and ultimately understood.

References↩︎