1 Introduction↩︎

Vision-Language-Action (VLA) models have become a popular approach for learning robot manipulation policies [1]–[14]. VLAs are typically trained by finetuning a pretrained vision-language model (VLM) on expert demonstrations via supervised action prediction, known as behavior cloning (BC). This can be done through direct regression [8], [12] or flow-matching and diffusion [11], [13], [15]. The premise is that such finetuning should transfer the VLM’s semantic priors (understanding of spatial layout, directions, color, shape, etc.) to the resulting control policy.

Consider a VLA policy finetuned to “pick up the green mug and place it on the plate” in a scene containing both a green and pink mug (Fig. 1 (a)). Since the two mugs share the same shape and manipulation affordances, the policy should generalize to “pick up the pink mug” if finetuning preserves the VLM’s color understanding. However, with existing methods, it does not. In fact, BC finetuning corrupts the very prior that makes VLMs worth adapting, leading to two failure modes that persist even under good training practices.

First, standard BC optimizes only the action prediction loss, with nothing protecting the VLM’s pretrained representations from being overwritten. Over the course of finetuning, these updates progressively erase the visuolinguistic and spatial concepts the VLM acquired during internet-scale pretraining. Concretely, on our physical xArm7 robot, a VLA trained to pick up the green mug reaches for the green mug \(90\%\) of the time even when instructed to pick up the pink one, indicating that color grounding has been erased from the backbone. On LIBERO-PRO’s position-swap test, where object positions are rearranged at evaluation, the policy almost always replays its memorized training trajectory rather than acting on the current observation. To fix this failure mode, recent work has proposed co-training, or augmenting continuous action data with general-purpose VQA, captioning, and scene-description data [10], [16]–[18] to prevent catastrophic forgetting. Yet we find that on the position-swap test, co-trained VLAs score \(0\%\). Evidently, losses on such auxiliary data do not place a sufficient constraint on the backbone’s representations. As a more direct remedy, we propose Vision-Language Anchoring, or maintaining a frozen copy of the pretrained VLM (Fig. 1, left) and applying a distillation loss throughout finetuning. This requires no additional data or architectural changes, yet recovers out-of-distribution (OOD) generalization.

The second failure mode runs even deeper. Co-training gives the VLA’s two heads disjoint supervision: the action head is supervised by expert demonstrations while the language head is supervised by losses on generic image-text tasks. Because the heads share a backbone but are never supervised on the same robot observation, their predictions may contradict each other: when the action head predicts a movement to the right, the language head may predict “left.” To fix this misalignment and further improve task success, we introduce Language-Action Alignment, which co-supervises both heads on the same robot observation during finetuning (Fig. 1, top). To supply the aligned supervisory data, we automatically convert continuous action targets in expert demonstrations into discrete action directions expressed in language.

We propose combining Vision-Language Anchoring and Language-Action Alignment with the standard BC loss to form , a VLA finetuning method with consistent improvements across simulation and real-world experiments under both regression and flow-matching action heads. In simulation, our method improves OOD generalization on LIBERO-PRO and LIBERO-Plus benchmarks, which introduce unseen spatial rearrangements, camera perturbations and other shifts absent from training; and on the long-horizon CALVIN benchmark. In real-world experiments on an xArm7 robot, improves performance under unseen spatial rearrangements, semantic perturbations, and cluttered scenes. Beyond these results, our Language-Action Alignment framework provides the first direct diagnosis of language-action misalignment in co-trained VLAs and shows that better alignment improves action accuracy.

2 Related Works↩︎

General-Purpose VLAs for Robot Manipulation. VLAs use pretrained VLMs as foundation policies for generalist manipulation, adapting to downstream tasks with minimal finetuning [1]–[8]. RT-2 [19] co-finetunes a VLM on robot trajectories, showing web-scale pretraining yields emergent generalization to novel objects and compositional reasoning, while OpenVLA [9] scales this via cross-embodiment training on Open X-Embodiment [20]. Architectural variants include flow-matching continuous actions (\(\pi_0\), \(\pi_{0.5}\)) [11], [16], a frozen-VLM + diffusion-transformer dual system for humanoids [13], and a billion-parameter diffusion expert with curriculum learning [14]. VLA-Adapter [12] instead avoids large-scale robotic pretraining, using a lightweight Bridge Attention mechanism to turn any off-the-shelf VLM into a strong VLA. StarVLA [15] likewise forgoes robotic pretraining, offering a modular framework that pairs an arbitrary VLM backbone with an arbitrary action head. Unlike these, we investigate which finetuning recipe best preserves the VLM’s representations. We build on VLA-Adapter and StarVLA as our base architectures: both finetune a pretrained VLM directly on expert demonstrations, and together they cover both regression and flow-matching action heads.

Preserving VLM Representations in VLAs. Pretrained visual representations are a well-established driver of robot manipulation performance [21]–[26]. In VLAs, this requirement extends beyond the vision encoder to the full VLM backbone, since semantic and spatial reasoning are distributed across both vision and language representations. Prior VLAs commonly mitigate catastrophic forgetting by co-training on robot and generic vision-language data [10], [14], [17], [27], with variants adding action-grounding objectives, perception tokens, or chain-of-thought plans  [10], [17], [18], [28], [29]. Grover et al. [30] instead preserve pretrained features architecturally, anchoring a frozen vision encoder with a trainable one. MAPS [31] penalizes each module’s weights for drifting from the pretrained checkpoint, constraining vision layers more tightly than action-oriented ones, and Kachaev et al. [32] align visual patch features at a single intermediate layer with a frozen vision foundation model. Our anchoring differs in both target and scope: it distills the hidden states of vision and text tokens at every decoder layer from a frozen copy of the pretrained VLM itself, directly preserving the backbone’s vision-language representations. Additionally, unlike , none of these methods aligns language and action on the same robot observation.

Embodied Question Answering. Embodied QA has long tested language understanding under physical grounding, from navigation-centric QA [33], [34] to foundation-model reasoning [35]–[37] and robot-manipulation QA [27], [38], [39]. Yet a critical gap persists for VLAs: co-trained VLAs [17]–[19] evaluate language understanding on generic benchmarks [40], [41], and never test whether the model’s language predictions are consistent with its action predictions on the same observation.

3 Method↩︎

This section presents  (Fig. 1), a recipe that finetunes a pretrained VLM into a continuous action policy while preserving its pretrained VL representations. We first describe our base VLA architecture and training objective (Sec. 3.1), then the two objectives adds on top of standard BC: Vision-Language Anchoring (Sec. 3.2) and Language-Action Alignment (Sec. 3.3).

3.1 Base VLA Architecture↩︎

We consider VLAs that adopt a pretrained vision-language backbone with a dedicated action head [12], [15], as shown in Figure 1. The backbone encodes the visual observation and language instruction into a hierarchy of multimodal representations. Separately, robot proprioception is encoded by a lightweight state projector and provided to the action pathway as a conditioning signal. A fixed set of learned action queries extracts task-relevant information from the vision-language representations, either through joint self-attention within the backbone or through cross-attention in the action head. The action head then combines the resulting query features with the encoded robot state to predict a fixed-horizon chunk of continuous 7-DoF actions (3D translation and rotation offsets plus gripper control) using either direct regression or conditional flow matching.

We train this architecture with three objectives. The primary term is the standard BC loss \(\mathcal{L}_{\text{action}}\) between the predicted and ground-truth action chunks, \(\hat{\mathbf{a}}^{\text{cont}}\) and \(\mathbf{a}^{\text{cont}}\) respectively. The form of the BC loss depends on the action-head architecture: an \(L_1\) loss for a regression action head, and an \(L_2\) loss over the predicted velocity for a flow-matching action head. To it we add two objectives, both evaluated on the same robot observation: an anchoring loss \(\mathcal{L}_{\text{anchor}}\) that preserves the backbone’s pretrained vision-language representations (Sec. 3.2), and an alignment loss \(\mathcal{L}_{\text{align}}\) that grounds the model’s language predictions in its actions (Sec. 3.3). The three terms are optimized jointly: \[\label{eq:total95loss} \mathcal{L}_{\text{total}} = \mathcal{L}_{\text{action}} + \lambda_{\text{anchor}}\,\mathcal{L}_{\text{anchor}} + \lambda_{\text{align}}\,\mathcal{L}_{\text{align}},\tag{1}\] where \(\lambda_{\text{anchor}}\) and \(\lambda_{\text{align}}\) weight the two added objectives against the action loss.

3.2 Vision-Language Anchoring↩︎

To prevent catastrophic forgetting, we anchor the VLA’s backbone VLM with a frozen copy of the same VLM (the anchor VLM), which processes the same input batch (images, text) in parallel. This distills the original pretrained VLM’s representations into the trainable backbone. In effect, the anchor VLM serves as a frozen teacher, a role well established in the ML literature: feature distillation trains a student to match a teacher’s representations [42]–[44], and anti-forgetting methods regularize a finetuned model toward its pretrained copy in weight space [45], [46] or representation space [47]–[49]. We anchor hidden states of the vision and text tokens between the backbone and anchor VLM at every decoder layer \(u \in \mathcal{D}\), where \(\mathcal{D}\) is the set of decoder layers. Let \(\mathbf{m}\) denote the set of vision and text positions. The per-layer anchoring loss is \[\mathcal{L}_{\text{anchor}}^{(u)} = \big\| \mathbf{H}^{S}_{u}[\mathbf{m}] - \mathbf{H}^{A}_{u}[\mathbf{m}] \big\|_F^2,\] where \(\mathbf{H}^{S}_{u}\in\mathbb{R}^{N\times d}\) and \(\mathbf{H}^{A}_{u}\in\mathbb{R}^{N\times d}\) are the hidden states of the trainable backbone (\(S\)) and the anchor (\(A\)) at decoder layer \(u\), with \(N\) the number of vision and text tokens and \(d\) the hidden dimension of the VLM. \(\|\cdot\|_F^2\) is the sum of squared element-wise differences (squared Frobenius norm). The total anchoring loss averages over all \(|\mathcal{D}|\) decoder layers: \[\mathcal{L}_{\text{anchor}} = \frac{1}{|\mathcal{D}|}\sum_{u\in\mathcal{D}}\mathcal{L}_{\text{anchor}}^{(u)}.\]

3.3 Language-Action Alignment↩︎

As illustrated in Fig. 2, we programmatically convert the continuous action target \(\mathbf{a}^{\text{cont}}\) into a motion-direction label \(\mathbf{a}^{\text{lang}}\) for each non-stationary demonstration chunk and use this label to supervise the VLM backbone on the same observation on which the action head predicts the continuous actions. Concretely, we use the last-layer hidden state of the last instruction token, immediately before action prediction. This state summarizes the visual prefix and language instruction, making it a natural context representation for predicting the robot’s next motion direction. With \(N\) vision and text tokens, this is the \(N\)-th (last) token, so this pre-action hidden state is \(\mathbf{h}^{\text{pre}} = \mathbf{H}^{S}[N] \in \mathbb{R}^{d}\), where \(\mathbf{H}^{S}\) is the backbone VLM’s last-layer hidden state. We then project \(\mathbf{h}^{\text{pre}}\) onto the vocabulary through a learned projection and the frozen pretrained language head: \[\hat{\mathbf{a}}^{\text{lang}} = \mathbf{W}_{\text{lm}}\,\mathbf{W}_{\text{proj}}\,\mathbf{h}^{\text{pre}} \in \mathbb{R}^{|\mathcal{V}|},\] where \(\mathbf{W}_{\text{proj}} \in \mathbb{R}^{d \times d}\) is a learned projection and \(\mathbf{W}_{\text{lm}} \in \mathbb{R}^{|\mathcal{V}| \times d}\) is the frozen pretrained LM head, so \(\hat{\mathbf{a}}^{\text{lang}} \in \mathbb{R}^{|\mathcal{V}|}\) are logits over the vocabulary \(\mathcal{V}\). The alignment loss is the cross-entropy of these logits against the ground-truth direction label \(\mathbf{a}^{\text{lang}}\): \[\mathcal{L}_{\text{align}} = \mathrm{CE}(\hat{\mathbf{a}}^{\text{lang}},\, \mathbf{a}^{\text{lang}}).\]

Language Label Construction. The alignment targets are derived from ground-truth action trajectories with no human annotation. For each training sample, we assign one of six interpretable direction labels, \(W=\{\emph{up}, \emph{down}, \emph{left}, \emph{right}, \emph{forward}, \emph{backward}\}\), through three steps: average chunking, filtering, and discretization. Given the batch of action targets \(\mathbf{A}\in\mathbb{R}^{B\times K\times 7}\) (\(\mathbf{a}^{\text{cont}}\) stacked over \(B\) batch samples, each a \(K\)-step chunk), we average the translational components across the chunk dimension to reduce action noise: \(\bar{\mathbf{v}}=\frac{1}{K}\sum_{k=1}^{K}\mathbf{A}_{:,k,1:3}\in\mathbb{R}^{B\times 3}\). We then remove near-stationary samples satisfying \(\|\bar{\mathbf{v}}_i\|_2<\tau\) (where \(i\) indexes samples in the batch), with \(\tau\) a small stationarity threshold. For each remaining sample, we choose the dominant translation axis, \(j^\star=\arg\max_{j\in\{x,y,z\}}|\bar{v}_{i,j}|\), where \(\bar{v}_{i,j}\) is the \(j\)-th scalar component of \(\bar{\mathbf{v}}_i\), and use the sign of \(\bar{v}_{i,j^\star}\) to select the corresponding direction word. The resulting direction label \(\mathbf{a}^{\text{lang}}\) supervises the language-alignment loss (additional implementation details in App. 10.1).

4 Experiments↩︎

This section presents our experimental evaluation on simulation benchmarks and real-world experiments. Sec. 4.1 describes the benchmarks, baselines, and implementation details. Sec. 4.2 and Sec. 4.3 present comparisons and ablations on LIBERO-PRO, LIBERO-Plus, and CALVIN simulation benchmarks. Sec. 4.4 shows results on real-world xArm7 manipulation experiments. Finally, Sec. 4.5 provides an in-depth analysis of language preservation and language-action alignment in co-trained VLAs.

4.1 Experimental Setup↩︎

4.1.0.1 Baselines.

We compare against baselines covering four families (see App. 6 for more details): (i) the strong VLA models OpenVLA-OFT [8] and  [12] (Qwen2.5 [50] backbone, our base model); (ii) the knowledge-preservation baseline  [12]; (iii) co-trained VLAs MolmoAct [10], ChatVLA [18], Magma [17], and Co-training + KI, our reimplementation of Knowledge Insulation [16] on , which follows the common practice of mixing action finetuning with vision-language supervision [3], [19]: it co-trains on VSR [51], GQA [41], and COCO [52], covering spatial reasoning, compositional VQA, and captioning, and further adopts the KI recipe’s FAST action tokenization [53] and stop-gradient on the VLM backbone; and (iv) the long-horizon CALVIN baselines RoboDual [54], UniVLA [55], MoDE [56], and OpenHelix [57].

4.1.0.2 Benchmarks.

We evaluate on LIBERO [58], LIBERO-PRO [59], LIBERO-Plus [60], and CALVIN. LIBERO consists of four 10-task suites (Spatial, Object, Goal, Long) with test setups drawn from the same distribution as the training setup, so it probes only in-distribution performance and is now largely saturated; we nonetheless report standard LIBERO results in the appendix (App. 8.1), where still outperforms all baselines across multiple random seeds. We focus on the more challenging LIBERO-PRO and LIBERO-Plus stress tests below. LIBERO-PRO [59] tests semantic generalization via OOD perturbations: language rephrase, object swap, and position swap. LIBERO-Plus [60] tests perceptual robustness across seven axes that mirror real-world deployment variability: changes in camera viewpoint, lighting, background texture, object layout, robot initial state, language instruction, and sensor noise. LIBERO-PRO language rephrase rewords the instruction (“pick up the mug” becomes “grab the mug”), whereas LIBERO-Plus language instruction rewrites it globally, e.g., referring to the target by its function (“the flat surface used for holding food” for a plate), forcing the policy to ground the words in the observed scene. Similarly, LIBERO-PRO object swap replaces scene objects with unseen variants and position swap exchanges their locations (3), whereas LIBERO-Plus object layout keeps the objects but shifts the target location and adds distractors. Finally, CALVIN ABC\(\rightarrow\)D tests long-horizon, language-conditioned manipulation across four simulated environments (A, B, C, and D): the policy is trained on environments A, B, and C and evaluated zero-shot on the held-out environment D.

4.1.0.3 Implementation Details.

We use Prismatic-Qwen2.5-0.5B [50] as VLM backbone and finetune it with LoRA [61] with rank \(r=64\), applied on all layers. The input sequence is vision + text + proprioceptive state. We use a regression action head with bridge attention to layer-wise condition the action head latents on the VLM features [12]. The \(512\) vision patches are obtained from two images: we extract DINOv2 [62] and SigLIP [63] features for each image and concatenate them along the feature dimension per patch, yielding 256 patches per image and a 512-patch vision prefix in total. Additional hyperparameters and design choices are discussed in App. 7. This is our default architecture. To confirm generality across architectures and action heads, we additionally evaluate on StarVLA [15], an architecturally distinct configuration pairing a Qwen2.5-VL [64] backbone with a flow-matching GR00T FM-DiT [13] action head, in our real-world experiments (Sec. 4.4).

max width=

4.2 Simulation Benchmark Comparison↩︎

improves visual grounding and spatial understanding. Tab. ¿tbl:tab:libero95suite? shows that our method outperforms all the VLA baselines on every axis of LIBERO-PRO and LIBERO-Plus. For LIBERO-PRO, the position-swap axis is the hardest: task-relevant objects are rearranged, so a policy that memorized fixed scene-to-action mappings fails. Here, all other baselines (MolmoAct, OpenVLA-OFT) score 0% and scores 2.3%, while reaches 22.6%. Fig. 3 shows two qualitative examples from LIBERO-PRO, where generalizes to the perturbed scene, while standard BC replays its training trajectory.

improves robustness to perturbations. LIBERO-Plus consists of challenging perturbations like changing background texture, adding sensor noise, or unseen initial robot arm configurations. As we can see from Tab. ¿tbl:tab:libero95suite?, improves over  [12] on every axis, with the largest gains on the hardest perturbations: background texture (\(+8.9\)), sensor noise (\(+7.4\)), robot initial state (\(+6.5\)), and lighting (\(+5.8\)). In absolute terms, succeeds \(99.6\%\) under background texture changes and \(99.0\%\) under lighting changes. Per-suite robustness breakdowns for the LIBERO Long, Object, and Goal suites are provided in App. 8.2.

r0.5

improves long-horizon generalization. Tab. 1 shows results on the long-horizon CALVIN ABC\(\rightarrow\)D benchmark. In each CALVIN rollout, the policy receives a chain of five language instructions and must complete them consecutively: the \(k/5\) columns report the fraction of rollouts completing the first \(k\) instructions, and Len denotes the average number of consecutively completed tasks (out of five). Because the chain advances only if each instruction succeeds, small grounding errors compound over the rollout. We therefore compare against the baselines most relevant to this claim: OpenVLA-OFT and UniVLA, strong large-scale action-pretrained VLAs; OpenHelix, a recent CALVIN-specific VLA; and trained with standard BC (full comparison in App. 8.4). is the best at every chain length and attains the longest average rollout (4.5 vs. for ), with the gain over widening on the deepest chains where errors compound, from \(+0.8\) at one instruction to \(+4.8\) at five. It also beats the strongest action-pretrained baseline, OpenVLA-OFT, raising five-instruction completion from \(66.5\%\) to \(77.9\%\), indicating that preserving and aligning pretrained VLM representations improves long-horizon generalization without large-scale robot-action pretraining.

4.3 Impact of Anchoring and Alignment↩︎

In this section, we present ablations and comparisons to validate our two proposed loss terms, demonstrating that both are needed to get the best performance, and that Language-Action alignment is not just a generic regularization term.

4.3.0.1 Both anchoring and alignment are necessary.

r0.5

2 presents an ablation analysis that isolates the impact of our proposed additional loss terms by comparing the VLA-Adapter baseline with (only layer-wise MSE distillation loss \(\Lmse\) against a frozen VLM copy), Align VLA (only the language-action alignment loss \(\Lalign\)), and the full . We can see that each of anchoring and alignment independently improve over standard BC on every axis, and combining them in exceeds either alone. Thus, the two play complementary roles: anchoring preserves the representational substrate that alignment leverages.

4.3.0.2 Alignment is not just regularization.

r0.5

One might wonder whether the gains come from genuine language-action alignment or merely from a regularization effect that any auxiliary classification loss would provide [65]. Is the model improving because each predicted direction word is correctly grounded in its own observation, or simply because adding a six-way word-classification objective regularizes the pre-action state? To separate these two explanations, we train Shuffle and Scatter controls with the same training setup as (including anchoring) and the same auxiliary loss, but with corrupted direction labels that break the per-observation language-action alignment while leaving the auxiliary task intact. Shuffle applies a fixed permutation to the observation-to-label mapping (e.g., left\(\to\)forward, right\(\to\)up), so each direction label no longer matches the action on its observation; Scatter keeps this mapping but replaces the six direction words with six arbitrary, meaningless ones (e.g., purple, math), removing their motion semantics. As shown in 3, a pure regularization effect would survive this corruption; instead, both controls collapse toward standard VLA finetuning, showing that the improvement stems from genuine language-action alignment and not from the mere presence of an auxiliary classification loss (additional details in App. 7.5).

4.4 Real-World Results↩︎

In simulation, objects are perfectly rigid and fixed in orientation, perception is clean, and each perturbation changes one aspect of the scene at a time. Next, we test on a real UFactory xArm7 manipulation setup, where the policy has to deal with varied object orientations, noisy perception and actuation, clutter, and multiple aspects of the scene shifting together.

Our real-world benchmark is designed to test observation-grounded generalization rather than memorization of a single tabletop layout. As shown in Fig. 4 (a), we evaluate four held-out perturbation regimes. Spatial rearrangement changes the positions of the target object, surrounding objects, and distractors. Cluttered scene (Fig. 4 (b)) introduces additional distractor objects and requires the policy to select the object specified by language. Compositional Object-Layout jointly changes object placement and object orientation. For both Spatial rearrangement and Compositional Object-Layout, every evaluation is unseen with respect to training data, creating conditions that are more challenging than the position swap of LIBERO-PRO, as here pick, place, and distractor objects are swapped. We further extend Spatial rearrangement into a harder Semantic perturbation OOD test that changes the instruction along with the spatial arrangement of these objects: a policy trained to pick up a green mug is evaluated with the OOD instruction “pick up a pink mug and place it on the plate.” Solving it requires using the preserved VLM color concept under a new instruction rather than defaulting to the training object. Details of hardware, data collection, and deployment for our real-world experiments can be found in App. 7.4.

Fig. 4 shows results with two VLA setups:  [12] with a Prismatic [66]-Qwen2.5-0.5B [50] backbone, and the architecturally distinct StarVLA [15], which pairs Qwen2.5-VL [64] with a GR00T FM-DiT [13] action head. The two setups differ in VLM scale (0.5B vs.3B), action objective (L1 regression vs.flow matching), and action-head architecture (a bridge-attention regression head vs.a diffusion transformer), respectively. On the hardest test, semantic perturbation, we report results using the stronger StarVLA setup, since the deterministic L1 regression head of cannot capture multimodal action distributions [67], [68].

consistently improves real-world generalization across both architectures (Fig. 4). With the VLA-Adapter setup, mean success rises from 28.3% to 54.2%. With the StarVLA setup, improves every condition: compositional object-layout from 35% to 50%, spatial rearrangement from 30% to 60%, cluttered scenes from 45% to 70%, and mean success from 36.7% to 60.0%. These gains are therefore not tied to a specific VLM backbone or action head. On the semantic-perturbation test, the standard BC-trained StarVLA baseline collapses to its training prior: it picks the green mug in 90% of trials and places it in 30%. In contrast, picks the pink mug in 100% of trials and places it on the plate in 40%. This supports our central claim: standard BC binds language to the training scene, while re-grounds the instruction in the current observation.

4.4.0.1 Limitations and Error Analysis.

We manually inspect all failed real-world rollouts and bin clearly attributable failures into six modes (5): Incorrect Object, approaches the wrong object; Semantic Error, selects a feature-similar distractor (e.g., green grapes instead of green broccoli); Memorization Error, ignores the scene and follows the training trajectory; Grasp Failure, reaches the target but does not descend to grasp; Grasp-and-Drop Error, grasps correctly but releases too early, too high, or not at all; and Close Miss, grounds correctly but misses by a small margin. 5 shows that eliminates semantic errors (\(7\!\to\!0\)), nearly eliminates incorrect-object approaches (\(10\!\to\!1\)), reduces memorization errors (\(15\!\to\!9\)), and nearly halves grasp failures (\(15\!\to\!8\)). The slight grasp-and-drop increase (\(12\!\to\!13\)) reflects more successful grasps, hence more opportunities for drop errors.

4.5 Analysis↩︎

This section presents an in-depth analysis of language preservation and language-action misalignment in VLA training.

preserves VLM representations.

For effective generalization, a VLA must be able to perform visual reasoning, i.e., understand objects, their semantics, and their attributes in a scene, an ability that is a hallmark of pretrained VLM representations; GQA [41] tests this ability through compositional questions spanning object, attribute, and spatial understanding. Here, we quantify visual reasoning in standard BC and , through GQA accuracy over the course of finetuning with the frozen VLM (same VLM used as backbone for VLA model) as an upper bound. As shown in 6, standard BC degrades monotonically, losing 94% of its GQA accuracy within 10K steps (catastrophic forgetting), whereas retains 70% of the pretrained accuracy. Correspondingly, LIBERO-PRO task success rates (Table ¿tbl:tab:libero95suite?) are 71.9% (), 61.0% (standard BC), and 43.1% (frozen), showing that action performance and vision-language reasoning are not fundamentally at odds. We further substantiate this study by quantifying the change in representation by different finetuning paradigms in App. 9.3.

4.5.0.1 Language-action alignment is positively correlated with task success.

r0.5

In co-training, the language and action heads are never supervised on the same observation, so nothing forces the model to express the same behavior through both outputs. We measure this misalignment by extending the programmatic framework of Sec. 3.3, which converts robot demonstrations into language labels, into a diagnostic tool: at each timestep, we compare the model’s language prediction with its discretized policy action. The language prediction is obtained by probing the language head with a templated question about the robot’s next motion (exact prompts in App. 10), and the continuous policy action is discretized into the same label space, so the two heads are aligned on a frame when they produce the same label. ¿tbl:tab:alignment95main? reports results LIBERO-PRO, we report success, the task success rate over the evaluated rollouts, and alignment, the fraction of observations on which the two heads agree, regardless of whether that shared prediction is correct. Additionally, we test whether alignment predicts success: we compute the alignment of each individual rollout (the fraction of its frames on which the two heads agree) and report the Pearson \(r\) between this per-rollout alignment and the rollout’s success or failure. Here, Co-training + KI is our implementation of co-training with Knowledge Insulation [16] on ; comparing against it directly tests whether co-training aligns the two heads. As shown in ¿tbl:tab:alignment95main?, and Co-training + KI are misaligned, and the near-zero Pearson correlation shows that their agreement is uncorrelated with task success. raises alignment from \(16.8\%\) to \(78.4\%\) and turns the correlation strongly positive (\(-0.03 \to +0.51\); rollouts with higher alignment are far more likely to succeed), improving task success from \(61.0\%\) to \(71.9\%\). We further report this misalignment study on all competitive co-trained VLAs (ChatVLA, MolmoAct, and Magma) in App. 11.

5 Conclusion and Future Work↩︎

Standard BC overwrites pretrained VLM semantics and co-training leaves VLA language and action misaligned. fixes both by preserving the VLM’s pretrained representation and jointly supervising language and action with motion-direction labels. It nearly doubles real-robot task success and sharply improves position-swap generalization, showing that effective VLA finetuning must preserve semantics while grounding them in action. Crucially, both objectives reuse supervision already contained in the demonstrations, so these gains require no extra data, annotation, or architectural change, and hold across LIBERO-PRO, LIBERO-Plus, CALVIN, and a physical xArm7 under both regression and flow-matching action heads. We thus recast VLA finetuning as preserving and aligning pretrained VLM priors rather than trading them away for control, offering a simple recipe for adapting future vision-language foundation models to embodied tasks.

Finally, the principles of anchoring and alignment may extend beyond VLAs to video world models. For action-conditioned world models adapted from pretrained video generators [69]–[72], frozen-teacher regularization could limit representation drift during robot-specific post-training. An analogous alignment objective could enforce action-outcome consistency (e.g., requiring an inverse-dynamics model to recover the conditioning action from the generated transition) and instruction-rollout consistency when language is present. This offers a concrete path toward world models that retain broad spatiotemporal priors while remaining controllable and semantically grounded.

Appendix

6: Baseline Details
 7: Implementation Details
7.1 Language-Action Alignment Head
7.2 Training Setup
7.3 Training Cost Comparison
7.4 Real-World Implementation
7.5 Alignment vs.Regularization: Shuffle and Scatter Controls
 8: Extended Quantitative Results
8.1 Success Rates on Standard LIBERO Suites
8.2 Per-Suite Robustness Breakdowns
8.3 Multi-Seed Evaluation: Statistical Significance
8.4 Full CALVIN Comparison
 9: Representation Analysis: Language Preservation and Action Decodability
9.1 Centered Kernel Alignment (CKA)
9.2 Action Decodability (Linear Probing \(R^2\))
9.3 Language Preservation and Action Decodability
9.4 Layer-wise Language Preservation
9.5 Direction Understanding in the Backbone
 10: Language-Action Diagnostic: Dataset Construction
10.1 Direction Axis
10.2 Task Completion Axis
10.3 Orientation Axis
10.4 Grasp Axis
 11: Quantification of Misalignment in SOTA Models
 12: Real-World Rollouts
12.1 Faster Task Completion in Real World
12.2 Object-Orientation Perturbation
12.3 Anchor-Align Real-World Rollouts
 13: Qualitative Results in LIBERO Simulator

6 Baseline Details↩︎

This appendix extends the condensed baseline list in Sec. 4.1: baselines already introduced there are referenced briefly, and we detail the additional baselines that appear only in the appendix tables.

\(\bullet\) Non-VLM and standard VLM-based VLAs. OpenVLA-OFT [8] and  [12] (our base model) are introduced in Sec. 4.1. The standard-LIBERO comparison (¿tbl:tab:standard95success?) additionally includes Diffusion Policy [68] (non-VLM diffusion-based control), \(\pi_0\)-FAST [11], [53] (DCT-based action tokenization), SmolVLA [73] (flow-matching action expert), and VLA-0 [74] (plain-text action integers).

\(\bullet\) Co-training baselines. The co-trained VLAs we compare against are listed in Sec. 4.1; here we summarize what each adds. MolmoAct [10] augments a VLM with depth-aware perception tokens and image-space waypoint planning for 3D spatial reasoning. ChatVLA [18], [29] uses phased alignment training and a mixture-of-experts to retain language capability during action finetuning. Magma [17] adds a Set-of-Mark objective to ground actions within multimodal representations.

\(\bullet\) Knowledge-preservation approaches. \(\pi_{0.5}\)-KI [16] addresses catastrophic forgetting by training discrete FAST action tokens on the VLM backbone while stop-gradient prevents action gradients from corrupting pretrained representations.  [12] represents the extreme preservation baseline: the VLM backbone is entirely frozen and only the action head is trained, trivially preserving all pretrained knowledge at the cost of reduced task performance. Co-training + KI, our reimplementation of Knowledge Insulation [16] on , is already detailed in Sec. 4.1; implementation-wise, action finetuning uses LoRA, the language modeling head is kept for the VQA forward passes while the L1 regression action head handles robot data, and the FAST-token supervision flows through the language modeling head. This baseline tests whether shielding the VLM from action gradients and co-training on language data is sufficient to prevent catastrophic forgetting.

\(\bullet\) Long-horizon CALVIN baselines. RoboDual [54], UniVLA [55], MoDE [56], and OpenHelix [57] are introduced in Sec. 4.1. The full CALVIN ABC\(\rightarrow\)D comparison (6) additionally includes ReconVLA [75] (auxiliary reconstructive gaze-region grounding for VLA), RoboFlamingo [76] (OpenFlamingo VLM finetuned with an LSTM action head), and DeeR-VLA [77] (dynamic early-exit multimodal inference for efficient robot execution). These methods target long-horizon, instruction-conditioned manipulation under environment shift; none use a representation-preservation or language-action alignment objective on the VLM backbone.

7 Implementation Details↩︎

For better reproducibility of our results, we include the full architectural specification, hyperparameter settings, and training cost breakdown below.

7.1 Language-Action Alignment Head↩︎

The alignment objective (Sec. 3.3) introduces a single learned linear projection \(\mathbf{W}_{\text{proj}} \in \mathbb{R}^{d \times d}\) with \(d = 896\) (the hidden dimension of Qwen2.5-0.5B), adding \(896^2 + 896 = 803{,}712\) parameters (\(1.5\) MB in bfloat16). At each training step, we take the last-layer hidden state \(\mathbf{h}_i^{\text{pre}} = \mathbf{H}^{S}[N] \in \mathbb{R}^d\) of the last of the \(N\) vision and text tokens, the position immediately before the first action token (located in code by scanning the label tensor for the first non-masked entry, which marks the start of the action tokens). It is then mapped through \(\mathbf{W}_{\text{proj}}\) and the frozen pretrained language modeling head \(\mathbf{W}_{\text{lm}} \in \mathbb{R}^{|\mathcal{V}| \times d}\) to produce the language logits \(\hat{\mathbf{a}}^{\text{lang}}_i \in \mathbb{R}^{|\mathcal{V}|}\) over the full vocabulary (\(|\mathcal{V}| = 151{,}646\)). A cross-entropy loss is then computed between \(\hat{\mathbf{a}}^{\text{lang}}_i\) and the ground-truth direction label \(\mathbf{a}^{\text{lang}}_i\) (one of forward, backward, left, right, up, down), with near-stationary samples (\(\|\bar{\mathbf{v}}_i\|_2 < \tau\), \(\tau = 0.15\)) masked via ignore_index. Because \(\mathbf{W}_{\text{lm}}\) is frozen, the alignment gradient flows exclusively through \(\mathbf{W}_{\text{proj}}\) and the backbone’s LoRA parameters, encouraging the pre-action representation to stay readable by the frozen language head. The projection head is the only additional trainable module beyond the LoRA adapters and action head already present in .

7.2 Training Setup↩︎

All models are trained on 4 NVIDIA GH200 GPUs using bfloat16 mixed precision with PyTorch 2.7.

Hyperparameters. To ensure reproducibility, we report the complete training configuration used for the results in ¿tbl:tab:libero95suite?. The backbone VLM receives two camera images (third-person and wrist) along with the proprioceptive state as input. We use LoRA-based parameter-efficient finetuning with rank \(r{=}64\) and scaling factor \(\alpha{=}128\), applied to all linear layers in the backbone, and merge the LoRA weights into the base model periodically during training. Training runs for 10,000 gradient steps with a batch size of 32, a learning rate of \(2 \times 10^{-4}\), and standard image augmentation (random crops and color jitter). Checkpoints are saved every 2,500 steps. For the anchoring objective, we apply layer-wise MSE distillation (\(\Lmse\)) across all 24 transformer layers with anchoring weight \(\lambda_{\text{anchor}}{=}0.1\). For the alignment objective (\(\Lalign\)), we use an alignment weight of \(\lambda_{\text{align}}{=}0.02\) and a motion-magnitude threshold \(\tau{=}0.15\) to filter near-stationary samples. All remaining hyperparameters, including optimizer settings and data preprocessing, follow the recommended defaults from  [12] for the baseline setup.

7.3 Training Cost Comparison↩︎

A key practical question is how much overhead knowledge-preservation methods add to standard action finetuning. On identical hardware, we compare against Standard BC ( [12]) and Co-training + KI [16], both detailed in Sec. 4.1. Co-training is the natural point of comparison because it explicitly targets the same goal as our method: retaining the VLM’s pretrained visuolinguistic priors during finetuning. 4 summarizes the cost and performance trade-offs.

4pt

Co-training + KI adds a full VQA forward and backward pass (with batch size 16) at every training step, incurring \(+95\%\) wall-clock overhead and \(+5\) GB of GPU memory for the additional activations and gradients. It further requires curating 25K external VQA samples (VSR [51], GQA [41], COCO [52]) and maintaining the language modeling head alongside the action head during training. Despite this cost, the resulting policy substantially underperforms: 54.0% on LIBERO-PRO language rephrase and 57.1% LIBERO-Plus overall, both far below standard BC (91.1% and 85.1%), indicating that shielding the backbone from action gradients limits its adaptation to the downstream task.

In contrast, adds only a single inference-only forward pass through a frozen copy of the pretrained VLM (the anchor). We replace the anchor’s language modeling head (\(151{,}646 \times 896 \approx 136\)M parameters) with nn.Identity(), since only intermediate hidden states are needed for the layer-wise MSE loss, reducing the anchor’s footprint to \(0.7\) GB. This yields \(+28\%\) wall-clock overhead (\(0.36\) s per step), requires no external data or annotation, and completes a full 2,500-step run in \(68\) minutes—\(36\) minutes faster than co-training. Critically, achieves 97.0% on language rephrase and 90.3% on LIBERO-Plus overall, demonstrating that preserving the VLM’s representational geometry via anchoring is both cheaper and dramatically more effective than co-training on external VQA data. For larger backbone VLMs, the relative overhead would decrease further, since the anchor’s inference-only forward pass scales sublinearly compared to the training backward pass.

7.4 Real-World Implementation↩︎

The real-world setup uses a UFactory xArm7 equipped with a wrist-mounted RGB camera and an external front-view camera, evaluated on tabletop tasks requiring object discrimination, spatial reasoning, and instruction-conditioned target selection under clutter. We collect 150 demonstration trajectories using a 3D-printed GELLO teleoperation device [78] with velocity control at 30 Hz; observation and action spaces are joint angles, and trajectories are stored in the LeRobot [79] format and converted to RLDS [80] for our training pipeline. Both baseline and are trained for 15K steps with identical hyperparameters, and we report the best-performing checkpoint. Evaluation covers 7 held-out tasks with 20 rollouts per task (140 total trials). At deployment, the low-level velocity controller runs at 100 Hz while the learned policy predicts actions at 30 Hz; commands are interpolated and velocity-clipped for safe execution.

7.5 Alignment vs.Regularization: Shuffle and Scatter Controls↩︎

This section gives the full construction of the Shuffle and Scatter controls; their motivation and headline results are covered in Sec. 4.3 and 3. Both controls are trained with hyperparameters identical to : the only quantity that changes is the lookup table from motion class to target token id, so each control preserves the loss surface of \(\Lalign\) while destroying its semantic content.

Shuffle. The observation-to-label mapping is reshuffled once and held fixed throughout training, so the loss magnitude, gradient norm, and update frequency match the genuine alignment loss step-for-step. Concretely, we apply a fixed derangement of the six direction words: forward \(\to\) right, backward \(\to\) down, left \(\to\) forward, right \(\to\) up, up \(\to\) left, and down \(\to\) backward.

Scatter. Each direction class is mapped one-to-one to a fixed meaningless word: forward \(\to\) purple, backward \(\to\) seven, left \(\to\) today, right \(\to\) music, up \(\to\) tree, and down \(\to\) math. The six words are chosen to be mutually distant in the frozen output-embedding space, and the targets remain perfectly synchronized with the observations, so the model still solves a well-defined six-way classification task with an identical loss structure; only the motion semantics of the target tokens are removed.

Per axis, the collapse reported in 3 is sharpest on the LIBERO-PRO position-swap axis, where success requires re-grounding language to the perturbed scene rather than memorizing scene-to-action mappings: achieves \(22.6\%\) on position swap while Shuffle and Scatter recover only the baseline’s near-zero performance.

8 Extended Quantitative Results↩︎

This section complements the main paper’s quantitative evaluation (Sec. 4.2) with success rates on the standard LIBERO suites, per-suite robustness breakdowns, multi-seed significance tests, and the full CALVIN comparison.

8.1 Success Rates on Standard LIBERO Suites↩︎

We report success rates on the four standard (unperturbed) LIBERO suites—Spatial, Object, Goal, and Long—in ¿tbl:tab:standard95success?. achieves the highest success rate on most suites, surpassing prior methods including \(\pi_{0.5}\)-KI and OpenVLA-OFT, which use substantially larger backbones and large-scale robotic pretraining. This confirms that the gains from anchoring and alignment hold not only on the OOD stress tests reported in the main paper (¿tbl:tab:libero95suite?), but also on the in-distribution benchmark.

2.5pt max width=

8.2 Per-Suite Robustness Breakdowns↩︎

In the main paper (¿tbl:tab:libero95suite?), we report robustness and generalization results on the LIBERO-Spatial suite, where perturbations to object positions, language instructions, and scene layout directly test whether the policy has learned transferable spatial reasoning or merely memorized training trajectories. Here we show that the same gains carry over to the remaining three LIBERO suites: Long (7), Object (8 (a)), and Goal (8), which test long-horizon composition, object knowledge, and procedural reasoning, respectively. All comparisons are against , the strongest standard BC baseline from ¿tbl:tab:libero95suite?. Because uses the same LoRA architecture and training data as but without anchoring or alignment, it directly isolates the effect of our two proposed objectives. Each radar plot reports performance across nine evaluation axes: two from LIBERO-PRO (Language Rephrase and Object Swap) and seven from LIBERO-Plus (Language Instruction, Background Texture, Robot Init State, Camera Viewpoint, Object Layout, Lighting Condition, and Sensor Noise). Standard and Position Swap axes are excluded.

8.3 Multi-Seed Evaluation: Statistical Significance↩︎

To confirm that the gains reported in the main paper’s ¿tbl:tab:standard95success? (LIBERO-Spatial) and ¿tbl:tab:libero95suite? (LIBERO-PRO and LIBERO-Plus) are not artifacts of a single random seed, we re-run both and the baseline with five independent training seeds and report mean \(\pm\) standard deviation in 5. The standard deviations are small (all \(\leq 0.7\)) and the mean improvements of over on every column are substantially larger than the combined seed-to-seed variability, so the differences are statistically significant by a paired comparison. The multi-seed means also closely match the single-seed numbers reported in ¿tbl:tab:libero95suite? ¿tbl:tab:standard95success?, indicating that the headline results are stable rather than seed-lucky.

Gains transfer to a larger VLM backbone. To verify that the improvements from are not specific to our default backbone or action head, we re-instantiate the method on a different StarVLA configuration than the one used in the main-paper real-world experiments (Sec. 4.4). Recall that StarVLA is a modular VLA framework that combines an arbitrary VLM backbone with an arbitrary action head; here we instantiate it with a larger Qwen2.5 3B VLM and an OFT MLP-based action head, in contrast to the Qwen2.5-VL + GR00T FM-DiT instantiation used in the main paper. On LIBERO-PRO, improves the StarVLA baseline from \(73.6\%\) to \(\mathbf{89.0\%}\) on the language rephrase axis and from \(89.8\%\) to \(\mathbf{91.2\%}\) on the object swap axis. The improvement is qualitatively consistent with what we observe on the smaller backbone, indicating that the benefits of anchoring and alignment are a property of the training objective rather than of a particular VLM scale or action-head design.

8.4 Full CALVIN Comparison↩︎

The main paper’s 1 reports a compact CALVIN ABC\(\rightarrow\)D comparison against the most relevant baselines; 6 lists the full set. outperforms all baselines at every chain length and attains the longest average rollout.

9 Representation Analysis: Language Preservation and Action Decodability↩︎

This appendix provides the full mathematical details for the evaluation metrics used in the paper, followed by the representation-level analyses built on them.

9.1 Centered Kernel Alignment (CKA)↩︎

We use CKA [81], [82] to measure how much of the pretrained VLM’s representational geometry is preserved after finetuning. CKA is built on the Hilbert–Schmidt Independence Criterion (HSIC) [83], which quantifies the statistical dependence between two sets of representations via their kernel matrices.

HSIC. Given \(n\) data points and two representation matrices \(\mathbf{X} \in \mathbb{R}^{n \times p}\) and \(\mathbf{Y} \in \mathbb{R}^{n \times q}\), we form the Gram matrices \(\mathbf{K} = \mathbf{X}\mathbf{X}^\top\) and \(\mathbf{L} = \mathbf{Y}\mathbf{Y}^\top\) (under a linear kernel). The empirical HSIC is \[\mathrm{HSIC}(\mathbf{K}, \mathbf{L}) = \frac{1}{(n-1)^2}\,\mathrm{tr}\!\bigl(\mathbf{K}\,\mathbf{H}\,\mathbf{L}\,\mathbf{H}\bigr),\] where \(\mathbf{H} = \mathbf{I}_n - \frac{1}{n}\mathbf{1}\mathbf{1}^\top\) is the centering matrix. Centering removes the mean from each kernel matrix, so that HSIC captures the structural (second-order) similarity rather than mere magnitude.

CKA. CKA normalizes HSIC so that the score lies in \([0,1]\): \[\mathrm{CKA}(\mathbf{K}, \mathbf{L}) = \frac{\mathrm{HSIC}(\mathbf{K}, \mathbf{L})}{\sqrt{\mathrm{HSIC}(\mathbf{K}, \mathbf{K})\;\mathrm{HSIC}(\mathbf{L}, \mathbf{L})}}.\] A CKA score of 1.0 indicates that the two representations have identical geometric structure up to an isotropic scaling and orthogonal transformation; lower values indicate that finetuning has reshaped the layer’s representations.

Linear CKA. When the linear kernel is used, the CKA expression simplifies to a closed-form that avoids explicitly constructing the \(n \times n\) Gram matrices. Let \(\tilde{\mathbf{X}}\) and \(\tilde{\mathbf{Y}}\) denote the column-centered versions of \(\mathbf{X}\) and \(\mathbf{Y}\). Then: \[\label{eq:linear95cka} \mathrm{CKA}_{\text{linear}}(\mathbf{X}, \mathbf{Y}) = \frac{\bigl\| \tilde{\mathbf{Y}}^\top \tilde{\mathbf{X}} \bigr\|_F^2}{\bigl\| \tilde{\mathbf{X}}^\top \tilde{\mathbf{X}} \bigr\|_F \;\cdot\; \bigl\| \tilde{\mathbf{Y}}^\top \tilde{\mathbf{Y}} \bigr\|_F}.\tag{2}\] This is the form we use throughout: at each transformer layer \(u\), \(\mathbf{X} \in \mathbb{R}^{n \times d}\) contains the text-token hidden states from the frozen pretrained VLM and \(\mathbf{Y} \in \mathbb{R}^{n \times d}\) from the finetuned backbone VLM, across \(n\) evaluation samples. We report text-token CKA because it directly reflects the degree to which the model’s language understanding is preserved.

9.2 Action Decodability (Linear Probing \(R^2\))↩︎

To measure how much action-relevant information is linearly accessible from the model’s hidden states, we train a ridge regression probe at each layer. For layer \(u\), we extract the hidden state and mean-pool over all token positions to obtain a feature vector \(\mathbf{f}_i^{(u)} \in \mathbb{R}^{d}\) for each sample \(i\). We then fit a ridge regression model \(g\) to predict the ground-truth discretized action \(\mathbf{a}_i \in \mathbb{R}^{7}\) (with 256 bins per dimension): \[R^2(u) = 1 - \frac{\sum_{i=1}^{n} \bigl\| \mathbf{a}_i - g\bigl(\mathbf{f}_i^{(u)}\bigr) \bigr\|_2^2}{\sum_{i=1}^{n} \bigl\| \mathbf{a}_i - \bar{\mathbf{a}} \bigr\|_2^2},\] where \(\bar{\mathbf{a}}\) is the mean action over the dataset. A higher \(R^2\) indicates that more action information is linearly decodable from that layer’s representations, even without the action head.

9.3 Language Preservation and Action Decodability↩︎

We quantify what each training paradigm does to the backbone’s internal representations along two axes: language preservation, measured with Centered Kernel Alignment (CKA) [82] between the finetuned backbone’s text-token hidden states and the pretrained VLM’s (formal definition in App. 9.1), and action decodability, measured by linear-probing the backbone’s hidden states for the ground-truth action (details in App. 9.2). As shown in 9, standard BC catastrophically destroys the pretrained text representations (CKA drops to 0.34; per-layer analysis in App. 9.4). At the opposite extreme, the frozen backbone trivially preserves the pretrained geometry yet gains no action information, and the co-trained VLA behaves similarly. achieves the best of both: it maintains strong language preservation (CKA = 0.91) while attaining the highest action decodability of any method (peak R\(^2\) = 0.60 at layer 22), showing that alignment routes action-relevant information through the decoder layers without overwriting the pretrained language geometry. Preservation captures what the model keeps from pretraining, decodability what it gains from finetuning; together they show that attains high task performance by enriching the pretrained representations rather than destroying them, as standard BC does. Additional linear-probe results on direction concepts (in-distribution vs.OOD) are reported in App. 9.5.

9.4 Layer-wise Language Preservation↩︎

App. 9.3 reports language preservation per model; here we break the same measurement down by decoder layer, asking whether the GQA collapse in 6 is only behavioral or whether finetuning actually reshapes the backbone’s internal representations. [fig:language95preservation] plots the text-token CKA between the finetuned backbone and the pretrained VLM at every layer (formal definition in App. 9.1). Standard BC progressively destroys the pretrained text representations with depth, collapsing in the output layers (CKA drops to 0.34 at layer 24), the same layers that drive the GQA collapse, so the degradation is representational rather than merely behavioral. instead recovers near-perfect preservation through layer-wise distillation, with an average CKA of 0.95 across layers.

9.5 Direction Understanding in the Backbone↩︎

To further unpack what changes inside the backbone, we linearly probe motion-direction concepts from the backbone’s hidden states on spatial (in-distribution) and object (out-of-distribution) splits. 7 shows that substantially improves directional decodability, with the largest relative improvement on the OOD object split. Combined with the end-to-end language-action alignment results reported in ¿tbl:tab:merged95axes95models?, this indicates that the anchoring objective preserves a backbone in which directional concepts remain linearly accessible, and that the alignment objective then converts this latent structure into coherent joint language-action behavior at inference time.

10 Language-Action Diagnostic: Dataset Construction↩︎

This section details the programmatic curation pipeline for the language-action diagnostic introduced in Sec. 4.5.0.1. The diagnostic is built by a general-purpose programmatic framework that extracts language labels from robot trajectory data. The main paper applies it along a single axis, motion direction, the axis we follow throughout the paper; here we show that the same framework extends to three additional axes: end-effector orientation change, grasp state, and task completion status (10). Together, the four axes cover the key functional capabilities required for manipulation; each probes a distinct aspect of the VLA’s internal understanding (coarse spatial planning, fine rotational control, contact reasoning, and progress monitoring) and is evaluated independently, so that deficiencies along any single axis can be isolated.

We use this extended diagnostic to quantify misalignment in state-of-the-art co-trained VLAs (App. 11). Most of these models do not release LIBERO checkpoints, so the extended diagnostic cannot be run in the LIBERO simulator; instead, we evaluate on the real robot images such pretrained models are trained for. We therefore apply the framework to the MolmoAct [10] mid-training dataset, which spans 73 distinct manipulation tasks across diverse household environments, object categories, and interaction types. The dataset consists of episodes recorded with 7-DoF action vectors (3 translational, 3 rotational, 1 gripper); this task diversity ensures that the extracted labels are not biased toward any single manipulation skill or scene configuration. Moreover, because we evaluate pretrained checkpoints on recorded frames rather than deploying them as policies, the diagnostic reports per-frame (frame-level) language, action, and alignment accuracies rather than rollout task success. Below we describe the ground-truth label generation procedure for each axis.

The direction split contains 10,000 frames classified into six direction words (left, right, up, down, forward, backward); the distribution is naturally skewed toward downward approach movements (32.1%), with backward motion being the rarest (5.1%). The orientation split uses 1,001 frames uniformly distributed across 7 classes (no-rotation, roll-cw/ccw, pitch-up/down, and yaw-left/right; 143 per class), achieved by balanced sampling from the raw trajectory pool. Task completion comprises 10,000 frames with a binary label (yes/no) split as 33.6%/66.4%. The grasp split contains 5,000 frames labeled as open or closed (\({\sim}2{:}1\) open-to-closed).

10.1 Direction Axis↩︎

The direction axis evaluates whether the VLA’s language and action heads agree on the translational motion direction; it is the axis used in the main paper (Secs. 3.3 and 4.5.0.1). The ground-truth labels are generated by the programmatic procedure introduced in Sec. 3.3, which maps chunked action vectors to one of six direction words via average chunking, filtering, and discretization. This subsection adds the complete per-frame formulation, with which we label the MolmoAct dataset used in App. 11 to expose the misalignment of pretrained co-trained VLAs.

Given a robot trajectory of \(T\) frames, each frame \(t\) is associated with an action vector \(\mathbf{a}_t = (\Delta x_t, \Delta y_t, \Delta z_t, \ldots) \in \mathbb{R}^7\), where the first three components represent translational end-effector displacements in world coordinates. We extract the translational component \(\mathbf{d}_t = (\Delta x_t, \Delta y_t, \Delta z_t)\) and assign a single direction label via dominant-axis discretization.

Dominant-axis assignment. We identify the axis with the largest absolute displacement: \[j^\star = \arg\max_{j \in \{x,y,z\}} |\mathbf{d}_{t,j}|\] with tie-breaking priority \(x > y > z\). The sign of \(\mathbf{d}_{t,j^\star}\) then selects one of six direction words (\(\Delta x \!\mapsto\) forward/backward, \(\Delta y \!\mapsto\) left/right, \(\Delta z \!\mapsto\) up/down), yielding a single label \(\ell_t^*\) per frame, consistent with the discretization in Sec. 3.3.

Filtering. Two filters are applied before a frame is retained in the training set:

1. Magnitude filter: \(\|\mathbf{d}_t\|_2 \geq \mu\) with \(\mu = 0.001\), removing near-stationary frames.

2. Consistency filter: the top-1 label \(\ell_t^*\) must equal \(\ell_{t+j}^*\) for all \(j \in \{1, \ldots, K\}\) with \(K = 5\), ensuring temporally stable motion intent and suppressing noisy transient frames.

A frame is retained only if both conditions are satisfied.

10.2 Task Completion Axis↩︎

The 7-DoF action vector at frame \(t\) includes a gripper command \(g_t = \mathbf{a}_t[6] \in [0, 1]\), where values below a threshold \(\theta_g = 0.5\) indicate an open gripper and values above indicate a closed gripper. We define a binary openness indicator: \(o_t = 1\) if \(g_t < \theta_g\) (open) and \(o_t = 0\) otherwise (closed). We then identify the task-completion boundary by detecting the last gripper state transition in the episode. For example, in a pick-and-place task the final transition corresponds to the object being released at the target location. Formally, we define \(t_{\text{end}} = \max\{t : o_{t-1} \neq o_t\}\) as the frame of the last gripper state change. Episodes with no detectable transition are excluded. Each frame receives a binary label \(c_t\): frames before the last transition (\(t < t_{\text{end}}\)) are labeled \(c_t = 0\) (incomplete), while frames at or after it (\(t \geq t_{\text{end}}\)) are labeled \(c_t = 1\) (complete).

Training formulation. Each sampled frame is paired with a task-conditioned binary question: “In this current state do you think the robot has completed the task: {task}? Answer in one word yes or no.” The ground-truth answer is “yes” if \(c_t = 1\) and “no” otherwise.

10.3 Orientation Axis↩︎

Orientation labels are derived from the rotational components of the 7-DoF action vector \(\mathbf{a}_t = (\cdot, \cdot, \cdot, \Delta\phi_t, \Delta\theta_t, \Delta\psi_t, \cdot)\), where \(\Delta\phi\), \(\Delta\theta\), \(\Delta\psi\) denote instantaneous roll, pitch, and yaw deltas, respectively. Unlike the per-frame labeling used for translational direction (Sec. 3.3), orientation labels are computed over a sliding window of \(W = 7\) consecutive frames to accumulate sufficient rotational signal. For a window starting at frame \(t\), the cumulative orientation change is: \[(\Phi, \Theta, \Psi) = \left(\sum_{k=t}^{t+W-1} \Delta\phi_k,\; \sum_{k=t}^{t+W-1} \Delta\theta_k,\; \sum_{k=t}^{t+W-1} \Delta\psi_k\right)\] with cumulative magnitude \(M = \sqrt{\Phi^2 + \Theta^2 + \Psi^2}\).

Two-stage filtering. Two thresholds handle sensor noise and near-stationary orientations:

1. Noise gate (\(\mu_{\text{noise}} \approx 0.57\)°): windows with \(M < \mu_{\text{noise}}\) are discarded entirely as sensor noise.

2. No-rotation threshold (\(\mu_{\text{rot}} \approx 1\)°): windows with \(\mu_{\text{noise}} \leq M < \mu_{\text{rot}}\) are labeled as no_rotation.

Dominant-axis classification. For windows with \(M \geq \mu_{\text{rot}}\), the label is determined by the axis with the largest absolute cumulative change and its sign: \[k^* = \arg\max_{k \in \{0,1,2\}} \; |[\Phi, \Theta, \Psi]_k|\] This yields six directional labels. Together with the no_rotation class from the filtering stage, the full 7-class label space is \(\mathcal{Y}_{\text{orient}} = \{\texttt{no\_rotation},\; \texttt{roll\_cw},\; \texttt{roll\_ccw},\; \texttt{pitch\_up},\; \texttt{pitch\_down},\; \texttt{yaw\_left},\; \texttt{yaw\_right}\}\).

Dataset construction. The sliding window is applied densely across each episode with a stride of 1 frame (not \(W\)), yielding one sample per valid starting frame. Each sample stores the first frame’s index (used for image retrieval during evaluation), the list of frame indices in the window, the cumulative \((\Phi, \Theta, \Psi)\) values, the magnitude \(M\), and the derived label.

10.4 Grasp Axis↩︎

Grasp labels are derived from the gripper command \(g_t = \mathbf{a}_t[6] \in [0, 1]\), the same scalar used in the task-completion labeling (Sec. 10.2). Whereas task completion uses the gripper timeline to locate the final state transition, the grasp category directly classifies the instantaneous gripper state into a binary label \(\ell_t^g \in \{\texttt{open}, \texttt{closed}\}\) via the threshold \(\theta_g = 0.5\).

Episode selection. Only episodes that exhibit a complete pick-and-place gripper pattern are retained. We define a pick event as the first frame \(t_p\) where the gripper transitions from open to closed (\(o_{t_p-1} = 1, \; o_{t_p} = 0\), where \(o_t = 1\) if \(g_t < \theta_g\)), and a place event as the first subsequent frame \(t_r\) where the gripper reopens (\(o_{t_r-1} = 0, \; o_{t_r} = 1\)). Episodes lacking either event are excluded.

Stage partitioning and sampling. Each qualifying episode is partitioned into three functional stages based on \(t_p\) and \(t_r\):

1. Pre-grasp (frames \([0, t_p{-}1]\)): the end-effector approaches the object with an open gripper.

2. Grasp-to-release (frames \([t_p, t_r{-}1]\)): the object is grasped and transported.

3. Post-release (frames \([t_r, T]\)): the gripper reopens after placing the object.

\(N_s = 10\) frames are sampled uniformly at random from each stage via seeded sampling, yielding up to \(3 N_s = 30\) labeled frames per episode. This three-stage design captures both gripper states across all manipulation phases, though the resulting label distribution is naturally imbalanced (\({\sim}2{:}1\) open-to-closed) because stages 1 and 3 predominantly contain open-gripper frames.

11 Quantification of Misalignment in SOTA Models↩︎

Using the diagnostic dataset constructed in App. 10, this section quantifies language-action misalignment in state-of-the-art co-trained VLAs (ChatVLA, MolmoAct, and Magma), extending the analysis of Sec. 4.5.0.1 from the motion-direction axis to all four diagnostic axes. As explained in App. 10, these models are evaluated per frame on real robot images, so we report frame-level Language, Action, and Alignment accuracies rather than rollout task success.

11.0.0.1 Evaluation distribution for ¿tbl:tab:merged95axes95models?.

We evaluate the three co-trained VLAs (ChatVLA, MolmoAct, and Magma) on the MolmoAct mid-training dataset [10], chosen because it consists of real robot images: these pretrained models are built for real images, and evaluating them in simulation would place them out of distribution.

max width=

¿tbl:tab:merged95axes95models? reports all four axes of our diagnostic. The Task Completion axis tests whether the VLA can recognize terminal states (object placed, gripper released, etc.); Grasp tests contact-event detection; Orientation tests mid-trajectory reorientation. Across the three additional axes, the qualitative pattern matches Direction: even when the language and action heads each achieve moderate accuracy, their joint alignment is substantially lower, with MolmoAct showing the largest per-head-vs.-joint gap on Task Completion (44.6% language and 65.2% action, yet only 24.6% alignment). Below we analyze the structure of this misalignment along each axis.

Direction and Orientation. These fine-grained spatial axes exhibit the lowest alignment scores across all models (7.4–20.9%), reflecting the fact that translational and rotational planning are processed through largely decoupled pathways. MolmoAct’s action head reaches 41.3% on direction, well above the 16.7% chance level for 6-way classification, while its language head scores only 22.6%. Yet the two agree on just 15.5% of timesteps. This shows that the action head has learned directional control that the language channel cannot describe. This misalignment is precisely the gap that our alignment objective targets: by training the pre-action hidden state to predict direction words derived from ground-truth actions, we force the backbone to route directional information through a shared representation that is accessible to both heads.

Task Completion. All three models achieve alignment scores of 24.6–48.1% on this axis, but the gap between per-head accuracy and joint accuracy is striking. ChatVLA predicts task completion correctly 65.6% of the time from language and 44.3% of the time from action, yet the two heads agree only 48.1% of the time. This means that a substantial fraction of correct language predictions co-occur with incorrect action predictions, and vice versa. Importantly, the alignment score also counts cases in which both heads make the same incorrect prediction; even under this lenient criterion, the near-random alignment score clearly indicates misalignment. This suggests that each head learns a partially correct but independent signal for terminal-state detection, rather than sharing a unified internal representation of task progress.

Grasp. The grasp results are consistent with the previous finding of misalignment between heads. MolmoAct’s action head achieves 77.1% on grasp detection, while its language head reaches only 51.8%; yet alignment is only 37.8%. This means that the action head has learned reliable grasp control that the language channel cannot describe, and that the two heads frequently disagree on the same observation. Conversely, Magma’s language and action heads both hover near 49–50% (near-random performance), while alignment is 40.8%, again near chance. This indicates that even when the individual head accuracies are similar, the two heads are not coordinating their predictions; instead, they arrive at similar aggregate statistics through different per-timestep decisions.

12 Real-World Rollouts↩︎

12.1 Faster Task Completion in Real World↩︎

Beyond improving success rates, also produces notably faster task execution in the real world. 11 compares the distribution of rollout durations across successful trials for standard BC () and on the xArm7 pick-and-place task. completes rollouts \(1.7\times\) faster on average, with a tighter distribution of completion times.

We attribute this speedup to qualitatively different action vectors produced by the two methods. Near the grasp point—the most critical phase of the trajectory—outputs smaller, more tentative action vectors. These low-magnitude predictions frequently undershoot the required displacement, leading to repeated corrective adjustments before the gripper achieves a secure grasp. In contrast, produces higher-magnitude, more decisive action vectors at the grasp point, achieving a firm and consistent grasp on the first attempt. We hypothesize that the alignment objective, by grounding the backbone’s representations in interpretable language labels (e.g., “down” during the approach phase), encourages the model to commit to coherent motion plans rather than hedge with small incremental steps.

This decisiveness propagates beyond the grasp: once the object is secured, executes the transport and placement phases with equally direct trajectories. The result is not only higher success rates but also more efficient, more natural-looking robot behavior—an important practical consideration for real-world deployment where cycle time directly affects throughput.

12.2 Object-Orientation Perturbation↩︎

Beyond swapping object positions, our Compositional Object-Layout regime also perturbs the orientation of the target object itself. 12 shows the broccoli used in the pick-and-place task in six distinct poses: varying the object’s orientation while holding its identity and the instruction fixed prevents the policy from matching a memorized canonical view and forces it to re-ground perception in each rollout.

12.3 Anchor-Align Real-World Rollouts↩︎

13 Qualitative Results in LIBERO Simulator↩︎

References↩︎