0.0.1 Introduction↩︎

Vision-Language-Action (VLA) models have emerged as a promising paradigm for building generalist robotic agents, leveraging large-scale pretraining on diverse vision-language and robotic data to acquire broad embodied capabilities [1], [2]. Despite encouraging progress from models such as OpenVLA [3], \(\pi_{0.5}\) [4], and GR00T [5], VLAs remain far more fragile than their LLM/VLM counterparts [6], frequently failing on tasks that lie only modestly outside their training distribution [7]. The prevailing remedy is post-training, via supervised fine-tuning (SFT) on curated demonstrations [8] or reinforcement learning (RL) with environment rewards [9], [10], both of which require updating the VLA backbone. Yet updating a multi-billion-parameter backbone is computationally expensive [11], [12], and the resulting specialist policies tend to narrow the generalist capacity acquired during pretraining [13]. This tension between task performance and generalization preservation is particularly acute for VLAs, whose generalization is already far more limited than that of LLMs/VLMs and far more costly to lose [14]. How to strengthen VLA performance without sacrificing its hard-won generalist capabilities is therefore a central and pressing challenge.

We argue that the root cause of VLA deployment failures lies not only in action generation but also in the absence of action evaluation, i.e., the inability to anticipate the consequences of a proposed action before execution. VLAs are trained to imitate, not to evaluate: they produce locally plausible actions given the current observation [15], but receive no signal about what an action leads to (a successful grasp, a collision, or an irrecoverable state) [16], [17], and no signal about whether a different action might yield a better outcome in the long term [18], [19]. Sec. 0.0.3 provides a diagnostic pass@k study to support this view: when a frozen VLA is allowed multiple independent attempts, success rates rise dramatically from 33% at pass@1 to 92% at pass@32. The analysis reveals that high-quality, task-completing actions already reside within the VLA’s output distribution; the model simply cannot distinguish them from mediocre or harmful alternatives it produces alongside. Beyond generation, evaluation is an equally critical yet overlooked bottleneck.

This diagnosis motivates a fundamentally different strategy for VLA’s policy improvement. Rather than rewriting the VLA’s parameters to generate better actions at the cost of generalization, we can equip it with a lightweight consequence evaluator that judges candidate actions by their predicted long-term outcomes. The VLA backbone remains entirely frozen, preserving its generalist capacity, while the evaluator provides the missing “look before you leap” capability: the ability to foresee which candidate action is most likely to lead to task success.

We draw inspiration from the Bitter Lesson [20]: impactful advances in AI stem from methods that scale computation through search and learning. We propose SVA (Search, Value, and Act), a simple three-stage framework that elicits policy improvement for frozen VLAs. In Search, we employ Monte-Carlo tree search (MCTS) in simulation to fully explore the VLA’s output distribution via principled look-ahead search, efficiently discovering diverse trajectories annotated with empirical returns. In Value, we distill the knowledge obtained from MCTS into a lightweight Q-value model that predicts the expected consequence of executing a candidate action. Built on a small VLM backbone, this consequence evaluator compresses the expensive search process into a fast-to-evaluate function that generalizes across states and tasks. In Act, the frozen VLA proposes \(N\) candidate actions, and the evaluator selects the one with the highest uncertainty-regularized Q-value. SVA provides a natural and tunable mechanism for test-time scaling: increasing \(N\) improves the probability of selecting a high-quality action, with inference latency scaling sub-linearly in \(N\) (see Sec. 0.0.5.3).

In summary, our contributions are threefold:

• Problem. We identify an evaluation bottleneck in frozen VLA policies: high-quality actions are often present in the output distribution, but single-shot generation cannot reliably pick them out.

• Method. We propose a search-and-learning recipe that distills simulation-based tree search into a value model, enabling real-time action evaluation without simulator access at deployment.

• Results. SVA delivers consistent gains across multiple manipulation benchmarks and VLA backbones, serving as a practical and effective alternative to costly policy fine-tuning; notably, a 9B VLA with SVA outperforms a 27B VLA by 7 points at 27% lower inference latency.

0.0.2 Related Work↩︎

VLA Models. A growing body of work adapts the success of LLMs/VLMs into embodied domains for generalist robotic control [21], with a diverse family of VLAs emerging, including RT-1/RT-2 [1], [2], OpenVLA [3], \(\pi\)-series [4], [15], [22], GR00T [5], and Octo [8]. Despite this progress, VLAs remain markedly more fragile than their LLM/VLM counterparts, frequently failing on tasks that lie only modestly outside their training distribution [7]. This fragility motivates a large body of post-training research, mainly SFT on curated data [11], [12] or RL with environment rewards [21], [23], [24]. However, these methods share two drawbacks: updating billion-parameter backbones is computationally costly [13], [25], and specialist fine-tuning often narrows generalist capabilities acquired during pretraining [14], [26]. In contrast, SVA keeps the VLA backbone frozen and steers it via a lightweight external evaluator, avoiding costly backbone updates while preserving its generalist capabilities.

Action Evaluation for Robot Policies. This complementary line avoids modifying the backbone and learns a verifier to guide generation, which has proven effective in LLM reasoning, where outcome- and process-reward models [27], [28] combined with search [29]–[31] boost a frozen generator. A growing body of work brings this idea to robot policies [32]–[34]. SayCan [35] grounds LLM-proposed plans by scoring candidate skills with an affordance function, V-GPS [18] learns a value function via offline RL pretraining on re-annotated trajectories, RoboMonkey [36] trains an action preference model using synthetic preference data, and V-VLAPS [37] learns a value model to guide tree search at deployment. Other methods train the action verifier by SFT on synthetic reasoning data in VeGAS [38], offline RL on curated dataset in Hume [39], contrastive learning in CoVer-VLA [40], probability matching in TACO [41], or evolutionary diffusion in VLA-Pilot [42], among others [43], [44]. Further, MG-Select [45] leverages the model’s internal properties to score actions without an external verifier. World model-based methods [46]–[49] evaluate actions by rolling them out in a learned world model, but incur high inference cost and accumulated model error over long horizons. In contrast, SVA complements these approaches along three axes: it learns from MCTS rollouts rather than offline RL or synthetic data, producing consequence-aware signals rather than single-step preference, and obtains long-horizon estimates without costly world-model rollouts.

0.0.3 Diagnosing the VLA Bottleneck: Generation or Evaluation?↩︎

We argue that VLA failures arise not only from action generation but also from the absence of action evaluation. To this end, we conduct a diagnostic pass@k study to ask how often at least one out of \(k\) rollouts succeeds, probing the latent policy capability by isolating evaluation from generation.

Model, Benchmark, and Evaluation Protocol. We evaluate pass@k behavior of OpenVLA on Simpler and Libero, and \(\pi_{0.5}\) on RoboTwin. The three embodied benchmarks cover a range of task structures, including long-horizon object rearrangement, tabletop manipulation, and precise interaction tasks. For each task, we sample \(N=50\) independent rollouts from the policy distribution with \(c\) successes and compute \(\mathrm{Pass}@k = \mathbb{E}\bigl[1 - \binom{N-c}{k} / \binom{N}{k}\bigr]\), which estimates the probability that at least one of \(k\) independent attempts succeeds [50]. Fig. 2 shows the diagnostic results.

Observation 1: Successful Behaviors Already Exist in the Output Distribution of Frozen VLAs. The probability of obtaining at least one success increases sharply with the number of attempts, with the same trend holding on every task. The average success rate rises from 33% at pass@1 to 92% at pass@32, an absolute gain of 58 points. Frozen VLAs, despite failing under single-shot execution due to stochastic sampling or compounding execution errors, assign non-trivial probability mass to successful behaviors within their output distribution. The deployment bottleneck thus does not lie in the absence of competent actions, but in the inability to identify them before execution. This reframes VLA’s failure mode as a deficiency in action evaluation rather than solely in action generation.

Observation 2: The Evaluation Bottleneck Is Most Severe on Tasks of Intermediate Difficulty. Already-easy tasks saturate quickly: Pick up Book/Soup and Sauce in Basket starts from \(0.84/0.64\) to \(0.99/0.96\) at only pass@4, leaving limited room for further gains. Tasks that exceed the base policy’s capability exhibit limited gains regardless of sampling budget: Stack Cubes only improves from \(0.02\) to \(0.50\) at pass@\(32\). The practical value of action evaluation concentrates in the intermediate-difficulty regime, where the evaluator can efficiently translate latent competence into reliable execution: Turn Switch/Rotate QRCode start from \(0.31/0.35\) to near \(1\) at pass@32.

Observation 3: Diminishing Marginal Returns Suggest the Importance of Smarter Scaling Strategies. Although pass@\(k\) improves monotonically with \(k\), the marginal gain shrinks rapidly as the sampling budget grows: \(\Delta_{1\rightarrow2}\!=\!0.13,\Delta_{2\rightarrow4}\!=\!0.13,\Delta_{4\rightarrow8}\!=\!0.12,\Delta_{8\rightarrow16}\!=\!0.10,\Delta_{16\rightarrow32}\!=\!0.08\). This diminishing-return pattern exposes inefficiency of naive independent sampling: while drawing more rollouts can surface better trajectories, the informational gain per sample shrinks rapidly even as environment-interaction cost grows linearly. A practical scaling strategy should therefore allocate additional computation more efficiently, such as evaluating promising actions before execution.

0.0.4 Look Before You Leap: Addressing the Evaluation Bottleneck with SVA↩︎

We consider an embodied task as a language-conditioned MDP \(\langle \mathcal{S}, \mathcal{A}, T, R, \gamma, \chi \rangle\), where \(\mathcal{S}/\mathcal{A}\) is the state/action space, \(T(s'|s,a)\) is the state transition function, \(R(s,a)\) is the reward function, \(\gamma\) is the discount factor, and \(\chi\) is the language space. At each step \(t\), the agent receives a state \(s_t\in\mathcal{S}\) that may comprise an egocentric image, robot proprioception, or both, and selects action \(a_t\in\mathcal{A}\) according to a policy \(\pi(\cdot|s_t;l)\) conditioned on the instruction \(l\in\chi\) that describes the task (e.g., open the door).

We adopt an inference strategy to maximize the expected return of a frozen VLA policy \(\pi_\theta\): at each step \(t\), we draw a set of \(N\) candidate actions \(\{a^{(1)}, \ldots, a^{(N)}\}\) from \(\pi_\theta(\cdot | s_t; l)\), and select the action that maximizes a learned Q-function that predicts the expected consequence of candidate actions: \(a_t \!=\! \mathop{\mathrm{argmax}}_i\hat{Q}_\phi(s_t, a^{(i)}; l)\). This formulation treats \(\pi_\theta\) as a fixed proposal distribution and redirects selected actions toward higher-return regions via \(\hat{Q}_\phi\), enabling policy improvement by scaling test-time compute. Our setting is agnostic to the action granularity: \(a\) can denote an action chunk, a high-level skill (e.g., find the apple), or a low-level control primitive (e.g., a continuous end-effector displacement). Below, we sometimes omit the notation \(l\) for ease of reading.

0.0.4.1 Search: Mining Evaluation Signals via MCTS

Recent work on RL for foundation models reinforces the view of the Bitter Lesson: the true value of RL lies not in parameter updates themselves, but in the search process – rolling out policies, comparing outcomes, and assigning credit [51]. We employ MCTS [52] to fully explore the VLA’s policy distribution, performing structured look-ahead search to discover high-informative trajectories.

Each edge \((s,a)\) of the search tree stores an action-value \(Q(s, a)\), visit count \(n(s, a)\), and prior probability \(P(s, a)\). The tree is traversed starting from the root state with the following procedure:

Selection. Action \(a_t\) is selected using the PUCT (Predictor Upper Confidence bound for Trees) rule: \[a_t = \mathop{\mathrm{argmax}}_a \left(Q(s_t,a) + c_{\text{puct}} \, P(s_t, a) \frac{\sqrt{n(s_t)}}{1 + n(s_t, a)} \right),\] so as to maximize action value plus a bonus that is proportional to the prior probability but decays with repeated visits to encourage exploration, and \(c_{\text{puct}}\) controls the exploration degree.

Expansion. When the traversal reaches a leaf node \(s_L\) at step \(L\), the leaf node may be expanded. The policy \(\pi_\theta\) processes \(s_L\) just once to sample \(N\) candidate actions \(\{a^{(1)}, \ldots, a^{(N)}\}\); each candidate is added to the tree as a new edge, with the successor state as the corresponding new leaf node. The output probabilities are stored as prior probabilities \(P\) for each action \(a\), \(P(s_L, a) = \pi_\theta(a|s_L)\).

Evaluation. From the newly expanded leaf node \(s_L\), we estimate its value by performing a simulation rollout. Specifically, we clone the simulator state at \(s_L\) and execute the policy \(\pi_\theta\) forward for up to \(D\) steps (or until task termination), accumulating the discounted return as \(G(s_L) = \sum\nolimits_{i=0}^{D-1} \gamma^i r_{L+i}\). This rollout provides a Monte-Carlo estimate of the long-term consequence of reaching \(s_L\) under \(\pi_\theta\).

Backup. After obtaining the rollout return \(G(s_L)\), we propagate it back along the traversal path from the leaf to the root. For every edge \((s, a)\) visited during the selection phase, the statistics are updated: \[n(s, a) \leftarrow n(s, a) + 1, \qquad Q(s, a) \leftarrow Q(s, a) + \frac{G(s_L) - Q(s, a)}{n(s, a)},\] i.e., the visit count is incremented and the action value is updated as a running mean of all rollout returns that have passed through that edge. These updated statistics refine the PUCT scores in subsequent iterations, progressively biasing the search toward higher-return branches of the tree.

After several iterations of the selection-expansion-evaluation-backup loop under a finite budget, for each task we collect a few trajectories from diverse search episodes, including both successful and failed rollouts, providing valuable contrastive signals for learning relative action quality in Sec. 0.0.4.2.

0.0.4.2 Value: Learning a Deployable Consequence Evaluator

Having used MCTS to search for informative trajectories, we now turn to the complementary pillar of the Bitter Lesson: learning. Rather than relying on expensive tree search at every deployment step, we distill the knowledge discovered by MCTS into a lightweight model \(\hat{Q}_\phi\) that predicts the expected consequence of candidate actions. This amortization serves two purposes: i) it compresses the rich contrastive signals produced by search into a compact, fast-to-evaluate function, enabling real-time action selection without simulation; and ii) because neural networks generalize across states, the learned \(\hat{Q}_\phi\) can transfer the credit-assignment insights obtained from searched trajectories to unseen situations/tasks, effectively extending the reach of search beyond its original computational budget.

The consequence evaluator \(\hat{Q}_\phi(s,a;l)\) is built on a lightweight VLM backbone (e.g., Qwen3.5-0.8B), augmented with LoRA adapters and an ensemble of small MLP value heads. We append a special <VALUE> token to a prompt template containing the textual instruction \(l\), visual/proprioceptive inputs \(s\), and the candidate action \(a\). The hidden state positioned at this token after self-attention is fed into the ensemble of value heads. The ensemble mean is used as the predicted Q-value, while the standard deviation provides an uncertainty estimate. The model is optimized using a Smooth-\(L1\) loss that provides stable gradients on large errors (like \(L1\)) and smooth optimization on small errors (like \(L2\)): \[\mathcal{L}(\phi) = \begin{cases} \frac{1}{2}\epsilon^2 & \text{if} |\epsilon| < 1 \\ |\epsilon| - \frac{1}{2} & \text{otherwise} \end{cases} \quad\quad \epsilon=\frac{1}{B} \sum_{i=1}^B\hat{Q}_\phi^{(i)}(s,a;l)-Q(s,a;l),\] where \(B\) is the number of value heads, and target values \(Q(s,a; l)\) are provided by the collected data in Sec. 0.0.4.1 and normalized by dataset statistics. Only value heads and LoRA adapters are fine-tuned.

0.0.4.3 Act: Evaluation-Guided Action Selection at Test Time

At test time, we use the VLA policy \(\pi_\theta\) as a proposal distribution and the learned Q-model \(\hat{Q}_\phi\) as a verifier. The frozen \(\pi_\theta(\cdot | s_t; l)\) generates \(N\) candidate actions \(\{a^{(1)}, \ldots, a^{(N)}\}\), where the empirical prior of each candidate is estimated by its sampling frequency: \[p\left(a\mid s_t~;~l\right) = \frac{1}{N}\sum\nolimits_{i=1}^{N}\mathbb{I}\left[a^{(i)} = a\right].\] Finally, we select the candidate with the highest uncertainty-regularized Q-value: \[a_t=\mathop{\mathrm{argmax}}_{i\in\{1,\ldots,N\}} \left[\mu_\phi\left(s_t,a^{(i)}; l\right)-\lambda_1 \sigma_\phi\left(s_t,a^{(i)}; l\right)+\lambda_2 \log p\left(a^{(i)}\mid s_t;l\right)\right],\] where \(\mu_\phi(\cdot)/\sigma_\phi(\cdot)\) is the mean/standard deviation of the Q-ensemble, and (\(\lambda_1\), \(\lambda_2\)) are regularization coefficients. Action \(a_t\) is executed until completion, task termination, or invalid feedback.

Inference Latency. The best-of-N strategy nominally requires \(N\) VLA forward passes for action candidates and \(N\) Q-model passes for scoring. Since the lightweight Q-model (e.g., 0.8B) is far smaller than the VLA backbone (e.g., 7B), the scoring overhead is minimal. Generation cost is well below the naive \(N\times\) estimate for modern VLAs that decouple a heavy VLM backbone from a lightweight action expert (e.g., \(\pi_{0.5}\)): the VLM runs once per observation and only the small action expert is invoked \(N\) times, adding little overhead over single-shot inference (see Sec. 0.0.5.3).

Scaling Behavior. The best-of-N strategy provides a natural knob for test-time compute scaling: increasing \(N\) improves the probability of selecting a high-quality action, which naturally supports adaptive compute allocation across tasks of varying difficulty. Experiments suggest that scaling test-time evaluation in SVA can be more cost-effective than scaling model size (see Sec. 0.0.5.3).

0.0.5 Experiments↩︎

Benchmarks. We use three benchmarks spanning two categories: 1) embodied reasoning, including EB-Habitat and EB-Navigation from EmbodiedBench [53]; 2) robot manipulation, including SimplerEnv [7] on the WidowX platform and RoboTwin 2.0 [54]. See Appendix 0.0.8 for benchmark details and Appendix 0.0.9 for their real-robot relevance. For each task, episodes are split into training and evaluation sets at a 3:2 ratio, and success rate is reported. See Appendix 0.0.10 for experiment details.

Baselines. For EmbodiedBench, we adopt five backbones as the base policy spanning proprietary (GPT-4o), open-source (Qwen3.5-4B/9B/27B), and lightweight (Gemma-4-E4B-it) families, probing SVA’s model-agnostic property across scales and architectures. For SimplerEnv and RoboTwin, we compare against OpenVLA as a reference generalist, \(\boldsymbol{\pi_0/\pi_{0.5}}\) as SOTA VLAs trained on large-scale real-robot data (ruling out the concern that gains merely compensate for a weak base), and \(\boldsymbol{\pi_0/\pi_{0.5}}\)+RoboMonkey as the most directly comparable test-time selection method, which reranks candidates via a preference model trained on synthetic data. Comparing against RoboMonkey under identical proposal distribution and candidate budget directly tests whether SVA yields stronger action evaluation than single-step preferences. See Appendix 0.0.11 for details.

0.0.5.1 Main Results

0.0.5.1.0.1 EmbodiedBench.

Table ¿tbl:tab:embodiedbench-main? summarizes results on EB-Habitat and EB-Navigation. SVA consistently improves success rates across all five backbones, yielding average gains of +15.4 on EB-Habitat and +13.2 points on EB-Navigation. These improvements demonstrate that SVA serves as a model-agnostic test-time enhancement strategy that generalizes across scales and architectures. The most pronounced gains emerge on categories requiring long-horizon planning or visual grounding: Long Horizon (+23.34 on Qwen3.5-27B) and Common Sense (+28.33 on Qwen3.5-27B), where the base policy tends to commit prematurely to locally plausible but globally suboptimal actions. By scoring candidate actions with predicted long-term consequences, the Q-model effectively corrects these myopic preferences. The few minor regressions are confined to categories where the base policy already performs near ceiling, leaving little headroom for reranking.

2.5pt

0.0.5.1.0.2 Manipulation.

Fig. 4 examines whether SVA generalizes to manipulation policies requiring precise object interaction. On SimplerEnv, \(\pi_0\)+SVA achieves 50.7% average success rate, outperforming \(\pi_0\)+RoboMonkey without requiring any curated preference annotations. The largest gains concentrate on contact-rich multi-step tasks: Stack Cubes (63.9 vs., +26.4) and Eggplant in Basket (83.3 vs., +13.9), where selecting the correct grasp-then-place sequence is critical and the Q-model provides a clear discriminative signal among candidates. On RoboTwin, \(\pi_{0.5}\)+SVA improves the average from 36.0% to 43.5% (+7.5), with gains spanning diverse manipulation primitives: Turn Switch (+20.0), Press Stapler (+16.7), and Click Bell (+15.0). Also, SVA surpasses competitive RoboMonkey by +5.2 points, verifying the superiority of its action verifier. These tasks span distinct motor skills (rotation, pressing, tapping), indicating that the Q-evaluator transfers across manipulation modalities rather than overfitting to a single skill. These results confirm that SVA extends beyond embodied reasoning to fine-grained robot manipulation: the Q-model acts as a general-purpose action verifier whenever the base policy produces diverse candidates.

0.0.5.2 Ablation Study

To isolate each component’s contribution, we evaluate three ablations: 1) w/o MCTS, training the Q-model with only policy-rollout returns; 2) w/o Q-model, replacing value evaluation with majority voting; and 3) w/o multi-cand., querying a single candidate (pass-through). Results in Fig. 5 show that: i) Dropping MCTS (SVA vs.w/o MCTS) causes a moderate decline on EB-Navigation (56.11 vs.); ii) Removing the Q-model yields the largest drop (56.11\(\to\)​43.33 on EB-Navigation), confirming that an explicit value estimator is essential to distinguish among plausible but unequally effective candidates; iii) Eliminating multi-candidate selection reduces performance to the single-shot backbone (56.11\(\to\)​39.17). In summary, SVA outperforms all ablations by at least +5.28 avg., confirming that each component is necessary to the overall performance.

0.0.5.3 Scaling SVA at Test Time: Performance, Latency, and Cost-Effectiveness

SVA exhibits strong test-time scaling. We analyze SVA’s scaling behavior by sweeping the number of candidate actions \(N\) at deployment and measuring both success rate and inference latency. As shown in Fig. 6, increasing \(N\) yields monotonic gains in success rate for SVA (9B), confirming our pass@k observation: the base policy already contains competent behaviors but lacks a reliable mechanism to identify them. SVA closes this gap via scaling action evaluation, turning otherwise wasted samples into consistent performance gains.

SVA’s inference latency grows sub-linearly in \(N\). The evaluation latency is negligible compared to generation (\(\leq 1.33\)s vs.up to \(11.2\)s at \(N{=}32\)), thanks to the lightweight Q-model design. The generation latency scales sub-linearly with \(N\): each doubling of the candidate set (\(1{\to}2, 2{\to}4, \ldots, 16{\to}32\)) incurs only a \(10.63\%\)–\(36.58\%\) increase in latency (\(26\%\) avg.), reflecting effective decoding within the VLA backbone. This sub-linear growth is what makes test-time scaling practical for robotics, where real-time execution is non-negotiable.

Scaling test-time evaluation is more cost-effective than scaling model size. Single-shot Qwen3.5-27B reaches \(46.7\%\) success at \(10.2\)s of inference latency. In contrast, Qwen3.5-9B + SVA reaches 51.1% at 5.87s with best-of-4 and 53.6% at 7.43s with best-of-8: a 7-point gain over the 27B baseline at 27% lower latency, using a \(3\times\) smaller backbone. This result reframes action evaluation at test time as a first-class lever for VLA, orthogonal to (and substantially cheaper than) data scaling and policy post-training. This is also consistent with test-time scaling findings in LLMs [55].

0.0.5.4 Case Study

We provide a qualitative case study with the instruction: On the sofa there’s an apple, but instead find a plate and move it to the brown table. This task is challenging because the first clause introduces a salient but irrelevant object-location pair (apple-sofa), distracting from the true goal of moving the plate to the brown table. Full results are provided in Appendix 0.0.15. As shown in Fig. 7, the base policy is misled by the distractor, navigating to the sofa and then issuing repeated invalid recovery actions.

r0.42

2pt

In contrast, SVA selects a plate-centric plan and reaches the target table in just 4 steps with no invalid actions. The candidate scoring in Table 1 explains this success: although the base policy assigns high likelihood to the distractor-driven sofa-first plan, the Q-network ranks the lower-prior plate-centric plan highest. The value model can override myopic proposal preferences via long-horizon consequence evaluation, improving robustness and reducing invalid behaviors.

0.0.6 Discussion and Limitations↩︎

We revisited VLA’s policy improvement from an under-explored angle: reframing VLA’s failure mode as an evaluation bottleneck rather than purely a generation one. Across embodied benchmarks, our method delivers consistent gains over diverse backbones, showing that scaling test-time evaluation can be more cost-effective than scaling model size. We hope these results encourage the community to view evaluation as a first-class lever for VLA improvement, orthogonal to data scaling and policy fine-tuning. Also, our method reveals several limitations, including decoupled tree search and value learning, reliance on resettable simulators, and sim-only evaluation (test on physical robots is the most pressing next step). See Appendix 0.0.7 for details of limitations that we leave for future work.

Appendix↩︎

0.0.7 Limitations and Future Work↩︎

While SVA delivers consistent gains across embodied reasoning and manipulation benchmarks, several limitations remain that point to promising directions for future research.

First, our pipeline is deliberately staged with decoupled tree search and value learning, so search is blind to the evaluator being learned and the policy never benefits from improved values beyond test-time re-ranking, exploiting only a slice of what RL has to offer. A natural next step is to unify the stages into an online search-and-learning loop, where an up-to-date Q-model guides MCTS and on-policy MCTS rollouts continually refine the Q-model, unlocking more of RL’s potential while still touching the generalist backbone as gently as possible.

Second, the Search stage relies on a resettable simulator with a task-success signal; while this is standard for benchmarks such as EmbodiedBench and RoboTwin, it limits direct applicability to settings where high-fidelity simulation or reward functions are unavailable. A promising path forward is to replace, or supplement, simulator rollouts with learned world models or sparse human/auto-labeled outcomes, so that the same Search–Value–Act recipe can mine evaluation signals with improved sample efficiency in domains where ground-truth simulation is impractical.

Third, our evaluation is conducted entirely in simulation, and the calibration of the learned Q-model on physical robots remains untested; validating SVA on real hardware, possibly via sim-to-real co-training or lightweight online residual calibration of the evaluator, is the most pressing next step.

Taken together, these limitations chart a coherent research agenda: tightening the search–learning loop, relaxing the simulator assumption, and bridging the sim-to-real gap. We view SVA as an initial step toward scalable test-time reasoning for embodied agents, and we hope the Search–Value–Act paradigm provides a foundation upon which the community can build more general, scalable, deployable, and self-improving embodied systems.

0.0.8 Benchmarks↩︎

We evaluate SVA on EmbodiedBench and two VLA manipulation benchmarks. EmbodiedBench is used for the main experiments, while SimplerEnv and RoboTwin 2.0 are used to evaluate whether value-guided reranking generalizes to continuous-control manipulation settings.

0.0.8.0.0.1 EmbodiedBench.

We use two EmbodiedBench suites. EB-Habitat is a household rearrangement benchmark built on Habitat. The agent receives a first-person RGB observation and a natural-language instruction, and executes high-level discrete skills including navigation, pick, place, open, and close. We evaluate on the base, common_sense, complex_instruction, spatial_relationship, visual_appearance, and long_horizon splits, with a maximum episode length of 30 environment steps. EB-Navigation is an object-goal navigation benchmark built on AI2-THOR. The action space contains eight primitive actions: move forward, move backward, move right, move left, rotate right, rotate left, look up, and look down. We evaluate on the base, common_sense, complex_instruction, visual_appearance, and long_horizon splits, with a maximum episode length of 20 environment steps.

0.0.8.0.0.2 VLA manipulation benchmarks.

We additionally evaluate on SimplerEnv and RoboTwin 2.0. SimplerEnv uses ManiSkill2-based real-to-sim WidowX manipulation tasks. We evaluate four tasks: widowx_carrot_on_plate, widowx_stack_cube, widowx_spoon_on_towel, and widowx_put_eggplant_in_basket. Observations include the front RGB image, the language instruction, and a 7-D Bridge end-effector proprioceptive state. RoboTwin 2.0 evaluates bimanual manipulation in a SAPIEN-based simulator. We use 10 selected tasks with the aloha-agilex embodiment. Observations include head and wrist RGB images, joint states, and end-effector proprioception. Success is determined by the simulator-provided success signal.

3pt

0.0.9 On the Real-Robot Relevance of Our Simulation Study↩︎

All our experiments are conducted in simulation, but several deliberate choices make the conclusions directly informative for physical-robot deployment. We summarize the evidence below.

0.0.9.0.0.1 Our benchmarks are explicitly designed for real-robot predictivity.

SimplerEnv [7] is constructed and quantitatively validated as a real-world proxy for VLAs: its authors show that policy rankings on SimplerEnv match rankings on the corresponding physical Google Robot and WidowX setups across RT-1, RT-1-X, Octo, and OpenVLA, highlighting the potential of simulation-based approaches for evaluating generalist real-world manipulation policies in a scalable and reliable way. RoboTwin 2.0 [54] is a scalable simulation framework supporting five robotic arms (Franka, Piper, UR5, ARX-X5, and AlohaAgileX), specifically optimized for sim-to-real transfer via structured domain randomization along five axes (clutter, lighting, background, tabletop height, and language instructions) with its authors reporting promising results on real hardware. SVA’s gains are consistent across both benchmarks, making the conclusions directly informative for physical-robot deployment.

0.0.9.0.0.2 Our backbones are real-robot policies, not simulation-only ones.

\(\pi_0\) and \(\pi_{0.5}\) are state-of-the-art generalist VLAs trained on large-scale real-robot demonstrations and deployed on physical platforms by their original authors [4], [15]. SVA keeps these backbones strictly frozen, so the action distribution we re-rank is exactly the one a real robot would produce. By construction, the evaluation bottleneck exposed by our pass@k analysis is a property of this real-data-grounded distribution, and the Q-model’s role in selecting among already-feasible candidates is identical on hardware.

0.0.9.0.0.3 SVA is simulator-free at deployment.

Unlike methods that require online tree search, environment resets, or dense reward signals at inference time, SVA invokes only the frozen VLA and the lightweight Q-model. Both consume the same RGB + proprioception + language inputs a real robot produces and emit actions in the robot’s native action space. The simulator is used solely during training of the Q-model, as a source of task-success labels for MCTS rollouts, and never as a source of privileged features that the deployment-time evaluator depends on. Crossing the sim-to-real boundary therefore reduces to the standard VLA observation-distribution shift, a challenge that is orthogonal to our contribution and is already targeted by the chosen backbones’ real-data pretraining.

0.0.9.0.0.4 Transferring SVA to a real robot requires no algorithmic change.

Because the Q-model is a feed-forward network over standard observations, deploying SVA on a physical robot requires either (a) using the simulator-trained Q-model zero-shot, or (b) lightly fine-tuning it on a small set of real-robot rollouts with the same MCTS-style return targets. We highlight a physical-robot replication of our SimplerEnv/RoboTwin results as the most pressing next step.

0.0.10 Experimental Details of SVA↩︎

0.0.10.1 Q-Model Architecture

The Q-model is initialized from Qwen/Qwen3.5-0.8B. We convert the backbone into a multimodal action-value estimator by adding a special <|VALUE|> token to the tokenizer. For each candidate action sequence, the input contains the current observation, language instruction, recent interaction history, and candidate action sequence. The hidden state at <|VALUE|> is used as the state-action representation.

For EmbodiedBench, candidate actions are discrete high-level skills and are serialized directly into the textual prompt. For SimplerEnv and RoboTwin, candidate actions are continuous robot control vectors. To represent these continuous action chunks in the language-model input space, we introduce the FAST [12] action tokenizer from physical-intelligence/fast. Given an action chunk, FAST converts the continuous action vectors into a sequence of discrete action tokens. We then map these tokens to newly added Qwen special tokens, bracketed by action boundary tokens, and append them to the prompt before <|VALUE|>. This design allows the same Qwen backbone to score both symbolic action sequences and continuous robot action chunks.

The backbone hidden dimension is 1024. We attach five bootstrapped scalar Q-heads. Each head is implemented as \[\texttt{Dropout}(0.1) \rightarrow \texttt{Linear}(1024,512) \rightarrow \texttt{GELU} \rightarrow \texttt{Linear}(512,1).\] The final Q-value is the ensemble mean, and the ensemble standard deviation is used as an uncertainty estimate during reranking. The five Q-heads contain 2,626,565 trainable parameters in total.

We fine-tune the backbone with LoRA and train the Q-heads jointly. LoRA is applied to q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, and down_proj. We use rank 16, \(\alpha=32\), dropout 0.05, and do not train bias terms.

0.0.10.2 Search-Stage Data Collection

Offline supervision is collected using simulator-backed MCTS. Each sample corresponds to a visited tree edge and stores the instruction, observation, interaction history, candidate action sequence, MCTS-backed return target, visit count, and metadata. MCTS is used only on the training split and is not used during act-time evaluation. For the VLA manipulation benchmarks, we use open-source policy checkpoints as action proposers: the \(\pi_0\) proposer in SimplerEnv is initialized from , and the \(\pi_{0.5}\) proposer in RoboTwin 2.0 is initialized from .

3pt

0.0.10.3 Value-Stage Training

The Q-model is trained by supervised regression to MCTS-backed returns. Target values are normalized using the mean and standard deviation of each training set. We train separate Q-models for EB-Habitat, EB-Navigation, SimplerEnv and RoboTwin 2.0. Both models are trained for 5 epochs with AdamW, bfloat16 mixed precision, batch size 8, learning rate \(1\times10^{-4}\), and gradient clipping of 1.0. All training and evaluation experiments are conducted on NVIDIA RTX PRO 6000 GPUs.

0.0.10.4 Act-Stage Evaluation

At each decision step, the base policy samples multiple candidate action sequences or continuous action chunks. The Q-model scores all candidates in a batch, and the candidate with the highest reranking score is executed. The agent then replans until task success, invalid execution, or the episode budget is reached.

3pt

0.0.11 Baselines↩︎

We also evaluate a RoboMonkey-style candidate reranking baseline for comparison. The baseline follows the same candidate selection formulation but uses benchmark-specific verifiers.

For RoboTwin 2.0, we collect MCTS supervision in simulation. At each decision state, we sample a set of candidate action chunks and derive preference supervision from MCTS over these candidates. A Qwen3.5-0.8B-based scorer is then trained with a pairwise preference objective. During evaluation, the policy samples 16 candidate chunks, the learned scorer ranks them conditioned on the multi-view observation and language instruction, and the top-ranked chunk is executed. For scoring, 5 action proposals are generated by \(\pi_{0.5}\), and the remaining proposals are sampled from a Gaussian distribution fitted to these policy outputs. The grasp state is handled separately and determined by majority voting.

For SimplerEnv, we use the released RoboMonkey verifier, , as a frozen verifier. The policy samples multiple 7-DoF WidowX action chunks. For each chunk, we query the verifier on each action and use the mean verifier reward as the chunk score. For scoring, 5 action proposals are generated by \(\pi_0\), and the remaining proposals are sampled from a Gaussian distribution fitted to the policy outputs, while the grasp state is determined separately by majority voting. The candidate with the highest mean score is selected for execution.

0.0.12 Detailed Pass@k Results↩︎

Table ¿tbl:tab:passk95task95level? reports task-level Pass@\(k\) results on representative manipulation benchmarks. The results show a consistent gap between Pass@1 and larger \(k\) across all benchmarks. This suggests that base VLA models frequently sample successful or near-successful candidates, but their default likelihood ranking does not always select them.

0.0.13 Detailed VLA Manipulation Results↩︎

Table 5 and Table 6 provide task-level results on SimplerEnv and RoboTwin 2.0. On SimplerEnv, SVA improves over the \(\pi_0\) baseline on manipulation tasks that require temporally extended object interaction, such as stacking cubes and placing the eggplant into the basket. On RoboTwin 2.0, SVA improves the average success rate and outperforms the base \(\pi_{0.5}\) policy on most tasks.

4pt

5pt

0.0.14 Detailed Results on Ablation Study↩︎

Table ¿tbl:tab:ablation95full? provides the complete ablation results on EB-Navigation. The results further support the necessity of the three-stage SVA design.

2.5pt

0.0.15 More Results on Case Study↩︎

We provide additional qualitative results for two representative cases, each comparing the base policy with a successful SVA rollout.

0.0.15.0.0.1 Case 1: distractor-aware instruction following.

The instruction is: On the sofa there’s an apple, but instead find a plate and move it to the brown table. As shown in Fig. 7, the base policy is distracted by the sofa clause and fails after repeated invalid actions, while SVA follows the plate-centric plan and completes the task.

0.0.15.0.0.2 Case 2: spatial-relation grounding.

We further examine the instruction: Find a wrench and move it to the right of the sink. As shown in Fig. 8, the base policy repeatedly navigates to the sofa and attempts to pick up the wrench, producing no task progress. In contrast, SVA selects the spatially correct sequence: navigate to the sink, pick up the wrench, move to the right counter, and place it there.

Across both cases, SVA improves robustness by ranking candidate action sequences according to predicted long-horizon return rather than proposal likelihood alone. This enables the policy to reject distractor-driven or spatially incorrect plans and choose actions that better satisfy the task constraints.

References↩︎