1 Introduction↩︎

Recent advances in robot learning have positioned Vision-Language-Action (VLA) models [1]–[7] and World-Action-Model (WAM) policies [8]–[13] at the forefront of manipulation research. By unifying vision, language, and action within a single policy interface, these models have demonstrated strong performance across tabletop and mobile manipulation, as well as long-horizon instruction following. As capabilities scale, however, evaluation has emerged as a key bottleneck. A policy may appear general under one robot, scene, or task distribution yet fail under another; conversely, aggregate metrics may obscure competent subskills masked by a few physical bottlenecks. Progress therefore requires evaluation protocols that are not only realistic, but also sufficiently standardized and diagnostic to enable interpretable performance assessment.

The dominant evaluation paradigm in robot learning reflects a persistent tradeoff between reproducibility and physical validity. Existing benchmarks typically optimize for one at the expense of the other: they either provide controlled, large-scale evaluation in simulation, or capture real-world complexity with limited standardization. Simulator-centric benchmarks (e.g., RLBench [14], CALVIN [15], LIBERO [16], ManiSkill [17], VLABench [18]) offer reproducible protocols but rely on simplified dynamics and weakly controlled distribution shifts, making it unclear whether performance reflects genuine reasoning or dataset regularities. In contrast, real-robot evaluations capture realistic factors (e.g.., perception noise, contact dynamics, latency), yet are often tied to specific setups, limiting comparability across studies. More broadly, current protocols either focus on short-horizon tasks or consider complex settings without systematic control; they either report aggregate metrics or provide fine-grained diagnostics at the cost of scalability; and they either enable scalable simulation or faithfully capture real-world physics, but rarely both. Consequently, attributing performance to reasoning, data, or task design remains challenging.

To address these challenges, we introduce ManipArena, a real-robot evaluation framework for standardized and attribution-aware assessment. It is built upon five core principles:

• Reasoning-Oriented: ManipArena prioritizes reasoning-oriented manipulation by incorporating complex spatial constraints, multi-stage bimanual operations, and semantic understanding that go beyond simple motor skills, requiring models to demonstrate genuine execution reasoning or semantic reasoning rather than trajectory memorization.

• Multi-Level Generalization: It facilitates controlled multi-level generalization through a green-screen enclosed environment with fixed lighting, systematic diversity guides for data collection, and layered out-of-distribution testing across materials, appearances, and spatial configurations.

• Mobile Manipulation: Beyond tabletop tasks, ManipArena incorporates long-horizon mobile manipulation requiring navigation, spatial memory, and sustained whole-body control across an extended workspace, providing comprehensive coverage of real-world operational scenarios.

• Rich Sensory Diagnostics: The framework provides rich and open sensory signals, including low-level motor currents and joint velocities beyond standard joint states, allowing developers to perform granular force-aware action modeling and diagnostic analysis.

• Real-to-Sim Synchronization: By utilizing high-quality 3D scanning, we provide synchronized real and simulated environments, constructing physically consistent simulation counterparts to enable scalable evaluation and sim-to-real diagnostics without sacrificing physical realism.

Specifically, ManipArena comprises 20 real-robot tasks, 10,812 trajectories, and approximately 188 hours of demonstrations. The benchmark includes 15 tabletop tasks and 5 mobile-manipulation tasks. Tabletop tasks span both execution-heavy manipulation (e.g.., insertion, pouring, bimanual placement, rearrangement) and semantic manipulation (e.g.., classification, ordering, matching, category-based selection), while mobile tasks extend to navigation-conditioned, long-horizon scenarios. Each trajectory is annotated at three language levels: L1 provides abstract task instructions, L2 specifies spatially grounded actions, and L3 incorporates scene-level context. This design treats language granularity as a controllable experimental variable rather than a fixed dataset attribute.

Using the ManipArena protocol, we evaluate seven tabletop baseline configurations spanning VLA and WAM policy families, including \(\pi_{0.5}\), WALL-OSS-0.5, X-VLA, DreamZero, and Fast-WAM, with both task-specific and multi-task variants. Key findings are as follows:

• Unsaturated benchmark: The best aggregate tabletop score (WALL-OSS-0.5-Specialist) reaches 951/1500 (63.4%), and no policy consistently dominates across tasks.

• Nontrivial effects of task sharing: Multi-task training improves \(\pi_{0.5}\) (626 \(\rightarrow\) 703) but degrades WALL-OSS-0.5 (951 \(\rightarrow\) 865), indicating sensitivity to model design and task structure.

• Balance vs.scale: A balanced setting (200 trajectories per task) outperforms a larger-scale regime (\(\sim\)​500 per task) for \(\pi_{0.5}\).

• Task-dependent language granularity: L2-level tasks yield the strongest execution performance, whereas L3-level tasks are more effective for semantic reasoning.

• Challenges in mobile manipulation: ManipArena includes five navigation-conditioned mobile tasks that extend evaluation beyond tabletop settings. Current policy evaluation is limited to a scoped task-specific \(\pi_{0.5}\) configuration, with per-task results reported in Appendix 10.

Our contributions are threefold:

• We design a green-screen-based evaluation setup with fixed hardware and environment configurations, together with schema-driven task construction and stratified protocols, enabling systematic control over task variation and distribution shifts.

• ManipArena comprises 20 real-robot tasks, 10,812 trajectories, and 188 hours of demonstrations, spanning both tabletop and mobile manipulation. Each trajectory is annotated at three language levels (L1–L3), treating language granularity as a controllable experimental variable.

• We benchmark representative VLA and WAM policies under a unified protocol and provide detailed insights into task sharing, data scaling, language granularity, and long-horizon manipulation, revealing key limitations of current approaches.

2 Related Work↩︎

2.1 Benchmarking for Robot Manipulation↩︎

A large body of work has focused on developing benchmarks for robot manipulation to evaluate learning-based policies. Early simulation-centric benchmarks, such as RLBench [14], provide a diverse set of tasks with scalable demonstration generation and multi-modal observations, enabling reproducible evaluation across a wide range of learning paradigms. Subsequent efforts extend this direction toward more complex manipulation. For example, CALVIN [15] introduces compositional long-horizon tasks grounded in language instructions, emphasizing sequential decision-making and cross-task generalization. LIBERO [16] further explores lifelong learning and knowledge transfer, providing structured task suites for evaluating continual learning in embodied agents. More recent benchmarks aim to improve diversity and generalization. ManiSkill2 [19] introduces a unified simulation platform with large-scale object diversity, dynamic physics, and multi-task support, significantly improving coverage of manipulation scenarios. VLABench [18] further incorporates reasoning-intensive tasks, requiring models to integrate language understanding, spatial reasoning, and world knowledge for long-horizon manipulation.

Despite these advances, most existing benchmarks remain predominantly simulation-based. While simulators offer scalability and controllability, they fail to capture real-world complexities such as perception noise, contact dynamics, and system latency. Although some recent efforts [23]–[26] have introduced richer object diversity, dynamic physics they still cannot replicate the full visual, physical, and temporal variation of uncontrolled deployment environments. As a result, performance in simulation often does not reliably translate to real-world deployment, limiting their effectiveness for evaluating general-purpose embodied intelligence.

2.2 Real-World Evaluation for Robot Learning↩︎

To overcome the limitations of simulation, a growing body of work has explored real-world benchmarks and evaluation frameworks for robot manipulation. Early efforts such as FurnitureBench [27] introduces a standardized and reproducible platform for long-horizon manipulation through furniture assembly tasks, supported by large-scale teleoperation data and unified system configurations. SceneReplica [28] proposes replicable real-world scenes using standardized objects and layouts, enabling consistent evaluation across different laboratories. Platforms such as the Real Robot Challenge [29] provide remote access to standardized robotic systems, allowing policies to be evaluated under consistent hardware and protocols at scale. Recent frameworks including RoboArena [20], RoboChallenge [21], and ManipulationNet [22] further extend this direction by enabling distributed real-world benchmarking and attempting to standardize evaluation pipelines across different settings.

Despite these advances, existing real-world evaluation approaches still exhibit several limitations. First, many benchmarks are task-specific or domain-specific, such as furniture assembly or warehouse picking, limiting their generality. Second, most setups lack systematic control over generalization factors, making it difficult to disentangle the effects of appearance, spatial configuration, and task variation. Third, current benchmarks rarely emphasize reasoning-oriented manipulation, instead focusing primarily on execution-level skills such as grasping or pick-and-place. As a result, there remains a critical need for a unified, controlled, and reasoning-oriented real-world evaluation framework that enables fair comparison, interpretable diagnostics, and systematic analysis of generalization in embodied AI systems.

3 ManipArena Benchmark↩︎

ManipArena is a schema-driven real-robot benchmark for evaluating manipulation generalization under controlled physical conditions. It contains two robot platforms, 20 tasks, 10,812 demonstrations, a stratified 10-trial evaluation protocol, partial-credit scoring, three-level language annotations, and a paired simulation extension. We describe the benchmark components needed for the main experiments; per-task statistics, observation/action fields, sampling procedures, rubrics, and OOD assignments are deferred to Appendix 14–19.

3.1 Controlled setup and benchmark scale↩︎

ManipArena uses physical execution as the final measurement. The tabletop platform is a fixed bimanual system with a head camera and two wrist cameras; the mobile platform adds a lifting column and omnidirectional base inside a \(3\,\mathrm{m}\times3\,\mathrm{m}\) reconfigurable enclosure (Figure 2). Both platforms run at 20 Hz in green-screen booths with fixed lighting, camera placement, and workspace geometry, and release proprioceptive state/action vectors (Appendix 15). This control is used to reduce incidental physical confounds during evaluation, not to claim that deployment scenes are visually simple.

The benchmark contains \(20\) real-robot tasks, \(10{,}812\) demonstrations, \(13.5\)M frames, and approximately \(188\) hours of data. The 15 tabletop tasks support the cross-model studies in Section 4; the 5 mobile tasks provide a scoped long-horizon manipulation assessment. Table 2 summarizes the task families.

5pt

3.2 Schema-driven task design↩︎

The central unit of ManipArena is a task schema. A schema specifies the task stages, object and layout variables, distractor condition, and held-out condition reserved for evaluation. Demonstrations and evaluation trials are therefore sampled from declared regions of the same task space, making train/evaluation overlap and held-out variation explicit.

Formally, for task \(i\), a sampled configuration \(c=(d_{i,1},\ldots,d_{i,M_i},\delta)\) contains the values of its diversity dimensions and distractor condition. The training collection draws \[\label{eq:schema} c \sim P_i^{\mathrm{train}},\tag{1}\] where \(P_i^{\mathrm{train}}\) is specified by the task schema. The task families separate execution-heavy manipulation, semantic object reasoning, and navigation-conditioned mobile manipulation (Table 2); full schema definitions are given in Appendix 16.

3.3 Stratified real-robot evaluation↩︎

For each tabletop task, ManipArena uses a fixed 10-trial evaluation protocol. Trials 1–4 sample in-domain configurations, trials 5–8 introduce controlled visual or configuration shifts, and trials 9–10 use the strongest held-out condition available for the task: \[\label{eq:strat} c_t \sim \begin{cases} P_i^{\mathrm{ID}}, & t \in \{1,\ldots,4\}, \\ P_i^{\mathrm{shift}}, & t \in \{5,\ldots,8\}, \\ P_i^{\mathrm{held}}, & t \in \{9,10\}, \end{cases}\tag{2}\] where all three distributions are induced by the same task schema. The final band is task-dependent: 7 tabletop tasks use object-level OOD conditions, while the remaining 8 test spatial, ordering, or procedural held-out conditions. The per-task audit is provided in Appendix 19.

3.4 Partial-credit scoring↩︎

ManipArena scores each trial with task-specific subgoals rather than a binary success label. Each trial is worth 10 points, distributed over stages such as grasping, transport, alignment, insertion, pouring, semantic selection, and final retraction: \[\label{eq:score} S_i^{(t)} = \sum_{k=1}^{K_i} w_{i,k}\, \mathbf{1}\!\left[g_{i,k}\text{ achieved on trial }t\right], \qquad \sum_{k=1}^{K_i} w_{i,k}=10.\tag{3}\] For a task subset \(\mathcal{I}\), we report aggregate partial-credit score and success rate, where success requires at least \(9/10\) points: \[\label{eq:agg95sr} S(\mathcal{I})=\sum_{i\in\mathcal{I}}\sum_{t=1}^{10} S_i^{(t)}, \qquad \mathrm{SR}(\mathcal{I}) = \frac{1}{|\mathcal{I}|}\sum_{i\in\mathcal{I}}\frac{1}{10} \sum_{t=1}^{10}\mathbf{1}\!\left[S_i^{(t)}\geq 9\right].\tag{4}\]

Partial credit exposes whether a policy fails at perception, grasping, alignment, semantic selection, or final completion, which binary success would collapse. Per-band aggregation and evaluation algorithms are provided in Appendix 16; per-task rubrics are summarized in Appendix 17.

3.5 Language annotations and simulation extension↩︎

Each trajectory includes three language fields: L1 is a task-level instruction, L2 adds episode-specific object and spatial information, and L3 adds a generated scene description. This makes language granularity a controlled training variable in Section 4.2.

ManipArena also includes paired simulation environments for three tabletop tasks. They share task schemas and observation/action conventions with the real setup and are used only as a scoped screening diagnostic in Section 4.5, before real-robot validation.

4 Experiments↩︎

We use ManipArena as a controlled diagnostic for real-robot generalization, not only as a leaderboard. Across the 15 tabletop tasks, we evaluate seven baseline configurations spanning VLA and world-action-model (WAM) policies, task-specific and unified fine-tuning, and distinct pretraining provenance. Mobile manipulation is included as a task-suite extension, with scoped task-specific \(\pi_{0.5}\) results on the five mobile tasks reported in Appendix 10. The studies below vary one evaluative axis at a time—data quantity, language annotation, training regime, model provenance, or real/sim source—while using the stratified trial protocol and partial-credit scoring of Sections 3.3–3.4. Within each model family and controlled study, we keep the implementation family, observation/action interface, test prompt convention, and evaluation protocol fixed unless that factor is explicitly varied; details are summarized in Appendix 8.

4.1 Data scaling and sampling effects↩︎

We first test whether increasing demonstration count produces monotonic real-robot gains. Holding the architecture, unified training regime, language field, robot platform, and scoring rule fixed, we fine-tune \(\pi_{0.5}\) on 50, 100, 200, 300, and 400 trajectories per task, and on the full dataset. All runs use 60k steps except a full-data 120k-step control. Full per-task results are given in Appendix 12.

The aggregate trend is non-monotonic (Figure 3). The 200/task recipe obtains the best five-task score (312/500, 42% SR), exceeding the full-data 60k recipe (277/500, 34% SR) and the full-data 120k control (248/500, 26% SR). The same ordering appears in the full 15-task evaluation: 200/task reaches 853/1500 and 35.3% SR, while full-data 120k reaches 727.3/1500 and 23.3% SR. This result should not be read as a universal optimum at 200 demonstrations per task; rather, it shows that raw demonstration count is an incomplete proxy for policy quality, and that sample balance and task composition can materially affect real-robot conclusions.

4.2 Language annotation granularity↩︎

r0.45 5pt

We next vary the language field paired with each demonstration while evaluating all policies with the same L1 task-level prompt. The three unified \(\pi_{0.5}\) policies differ only in training-time annotation: L1 is a task goal, L2 adds episode-specific object and spatial information, and L3 further adds a generated scene description.

12pt

Scene-specific language improves performance despite being absent at test time (Table ¿tbl:tab:language?). L2 raises total score from 703 to 867 and SR from 20% to 32.7%, while L3 obtains the highest SR (35.3%) and strongest semantic score (370/500). The category split is informative: L2 is best on execution tasks (535/1000), whereas L3 shifts performance toward semantic reasoning while reducing execution score to 432/1000. Figure 4 illustrates the same distinction qualitatively. Annotation granularity should therefore be treated as part of the training distribution: action-centered grounding benefits spatial manipulation, whereas broader scene context is more useful for semantic disambiguation. Per-task results are given in Appendix [app:language95detail].

4.3 Task-level trade-offs in multi-task training↩︎

We next isolate the fine-tuning regime by comparing task-specific specialists with a unified multi-task checkpoint for both \(\pi_{0.5}\) and the WALL-OSS-0.5 reference model. The robot platform, task set, trial protocol, and scoring rule are fixed.

The effect has opposite signs across architectures (Table 3). Unified training improves \(\pi_{0.5}\) by 76.3 points, but reduces WALL-OSS-0.5 by 86.0 points. Per-task results in Appendix 10 show that these shifts are competence redistributions rather than uniform gains or losses. The pattern is consistent with a provenance-based interpretation: WALL-OSS-0.5 is pretrained on data from a closely related embodiment, whereas \(\pi_{0.5}\) uses broader cross-platform pretraining. The evidence is correlational, but it shows that benchmark rankings depend on the interaction between architecture, pretraining distribution, and fine-tuning regime.

4.4 Cross-architecture evaluation↩︎

We then place these controlled studies in a broader model landscape. Under the same tabletop protocol, we compare seven configurations spanning VLA and WAM policies: task-specific and unified variants of \(\pi_{0.5}\) [1] and the embodiment-aligned WALL-OSS-0.5 reference model, together with X-VLA, DreamZero [8], and Fast-WAM. WALL-OSS-0.5 naming and release scope are clarified in Appendix 7. Figure 5 includes the \(\pi_{0.5}\) diagnostic variants from Sections 4.1–4.2 to expose recipe effects alongside model-family differences.

8pt

The task-specific WALL-OSS-0.5 achieves the highest aggregate score (950.8/1500, 38% SR), with gains primarily concentrated in execution tasks (664.3 vs. for \(\pi_{0.5}\) Unified), while semantic performance is comparable (286.5 vs.). This suggests that the advantage is better attributed to embodiment-aligned pretraining than to architecture alone. Diagnostic variants of \(\pi_{0.5}\) further show that training recipes can rival model-family differences: without architectural changes, performance increases from 702.5 (full-data L1) to 853.0 (200/task) and 867.3 (L2 annotations). Despite these gains, the benchmark remains unsaturated, with the best model reaching 63.4% of the maximum tabletop score. Overall, real-robot performance reflects a combination of architecture, pretraining, fine-tuning, data composition, and annotation design. ManipArena makes these factors sufficiently disentangled for analysis, rather than collapsing them into a single leaderboard.

4.5 Paired simulation diagnostics↩︎

As a final diagnostic axis, we evaluate the paired simulation extension of Section [sec:paired95sim]. The claim is scoped: simulation is used to compare data-source and mixture recipes, while real-robot trials remain the endpoint for capability assessment. We evaluate \(\pi_{0.5}\) on three paired tasks with real-data, simulated-data, and mixed real/sim checkpoints.

Real-robot transfer results are reported in Appendix 13. The effect is task-dependent: simulated-data training improves press_button, cotraining does not uniformly dominate, and real-data training remains strongest on pick_fruits. We therefore treat simulation as a screening instrument, not a substitute benchmark: simulated gains require physical validation before being interpreted as real-robot capability gains.

5 Conclusion↩︎

We presented ManipArena, a standardized real-robot evaluation framework for attribution-aware assessment of manipulation policies. By integrating controlled physical environments, schema-driven task design, and stratified protocols, ManipArena enables systematic analysis of generalization, reasoning, and execution under real-world conditions. The benchmark spans diverse tabletop and mobile tasks, with multi-level language annotations that treat language granularity as a controllable experimental variable. Our results show that current VLA and WAM policies remain far from saturated and are highly sensitive to task sharing, data balance, and language specification. These findings expose limitations of existing evaluation practices and motivate more controlled and diagnostic real-world benchmarks. We expect ManipArena to support more rigorous and interpretable evaluation, and to advance research in generalization, long-horizon reasoning, and real-world deployment.

5.0.0.1 Limitations.

The present conclusions have bounded scope. Real-robot execution is stochastic, and stronger confidence would require more repeated trials. Coverage is constrained by cost: the tabletop study includes seven models, mobile policy evaluation is limited to a scoped task-specific \(\pi_{0.5}\) configuration, and the paired simulation extension covers three tasks. Results are also limited to the tested embodiments, controlled environments, and shared L1 instruction interface. Additional details are provided in Appendix 18. These limits define natural extension axes: because ManipArena specifies task schemas, protocols, scoring, language annotations, and simulation assets, future work can vary scenes, embodiments, horizons, and instruction interfaces without redefining the benchmark. ManipArena is therefore intended as a controlled testbed, not a fixed leaderboard.

6 Broader Impacts↩︎

ManipArena can accelerate progress toward more reliable embodied AI agents by making real-robot generalization measurable and interpretable. However, strong performance on the benchmark alone should not be assumed sufficient for real-world deployment. The controlled green-screen environment, fixed lighting, and standardized object sets do not capture the full visual diversity, clutter, and dynamic human presence of uncontrolled homes, offices, or factories. Care should always be exercised when transferring policies evaluated under these conditions to safety-critical applications involving physical human-robot interaction or high-stakes manipulation tasks.

7 Baseline model naming and scope↩︎

For all model-zoo comparisons, baselines are fine-tuned using the ManipArena training demonstrations specified for the corresponding setting (task-specific specialist or unified multi-task) and evaluated with the same 10-trial protocol. No baseline receives additional ManipArena task demonstrations, evaluation examples, or task-specific real-robot data beyond the stated training split. This does not make upstream pretraining identical; differences in pretraining provenance are treated as part of the evaluation.

8 Baseline implementation consistency↩︎

For each model family, we use a single implementation and configuration family throughout the corresponding comparisons. The purpose is to ensure that differences reported in the main text are attributable to the intended experimental variable rather than to incidental changes in preprocessing, action representation, prompting, or evaluation. Within each controlled study, we keep the robot platform, observation streams, action interface, normalization assets, evaluation prompts, trial protocol, and partial-credit rubric fixed unless that factor is explicitly the subject of the study.

8.0.0.1 Controlled changes.

Task-specific versus unified training changes only the training data grouping while keeping the model family, action interface, test prompts, and evaluation protocol fixed. The data-scaling study changes only the number of sampled demonstrations per task. The language study changes only the training-time language field, while all policies are evaluated with the same L1 task-level instruction. The Real2Sim diagnostic changes only the data source or mixture used for fine-tuning, followed by the same real-robot evaluation endpoint.

4pt

9 Experimental resources↩︎

We report the physical and compute resources used for the ManipArena study to make the evaluation cost and reproducibility assumptions explicit. These numbers describe data collection, real-robot evaluation, and fine-tuning for this paper; they exclude upstream pretraining of external foundation models. The released dataset contains \(10{,}812\) trajectories, \(13.5\)M frames, and approximately \(188\) robot hours collected on the tabletop and mobile robot platforms in controlled green-screen environments. Beyond robot execution time, data collection and evaluation require teleoperation, scene reset, calibration checks, failure recovery, safety supervision, and rubric-based scoring.

Each tabletop evaluation uses 10 real-robot trials per task. The model-zoo study alone covers seven configurations across 15 tabletop tasks, corresponding to at least \(7 \times 15 \times 10 = 1{,}050\) tabletop robot trials, before including the data-scaling, language, Real2Sim, and scoped mobile diagnostic run. Table 6 summarizes the approximate fine-tuning compute. GPU-hours are computed as the number of GPUs multiplied by wall-clock training time.

4pt

For reference, 15 tabletop \(\pi_{0.5}\) task-specific specialists correspond to approximately \(15 \times 48 = 720\) GPU-hours. Each 60k-step unified \(\pi_{0.5}\) diagnostic run costs approximately 160 GPU-hours, and a 120k-step control run is approximately 320 GPU-hours under linear scaling. The total training cost of the paper is therefore larger than a single model-zoo row, because the data-scaling and language studies train multiple additional diagnostic checkpoints.

10 Per-task model zoo results↩︎

Table 7 reports the complete per-task scores and success rates for all seven baseline configurations.

3pt

11 Language Experiment Details↩︎

Table 8 reports the per-task breakdown of the language annotation experiment. L2 dominates execution-heavy tasks such as put_blocks and pick_items, while L3 achieves the strongest scores on semantic tasks such as classify_items and pick_fruits.

4pt

12 Data scaling details↩︎

Table 9 reports the per-task score and success rate for each data recipe.

3pt

Full-data model trained for 120k steps; all other rows use 60k steps.

13 Real2Sim transfer details↩︎

Table 10 reports the real-robot evaluation of checkpoints trained with real demonstrations, simulated demonstrations, and a 50% mixed real/sim recipe. The paired simulation extension is used as a diagnostic data source; real-robot trials remain the endpoint for capability assessment.

4pt

14 Per-task dataset statistics↩︎

Table 11 summarizes the collected demonstrations for each task, including trajectory count, average duration, and the primary manipulation capability being evaluated.

4pt

15 Data format and observation/action layout↩︎

ManipArena trajectories are stored in LeRobot v2.1 format. Tabletop tasks use 56-dimensional state and action vectors. Mobile manipulation tasks add 6 dimensions for head, lifting column, and chassis, giving 62D in total. The end-effector frame uses \(+x\) forward, \(+y\) left, \(+z\) up. The last element of each 7D joint group (indices 20, 27, 34, 41, 48, 55) is the gripper channel. Table 12 lists the full per-dimension layout.

4pt

ManipArena additionally releases joint velocities and joint currents per timestep (indices 21–34, 42–55), providing additional signals related to contact and actuation dynamics that are useful for compliant- or impedance-based control policies. For mobile trials, the head, lifting-column, and chassis channels are released alongside the arm channels so that navigation-conditioned manipulation policies can be trained and evaluated on a uniform observation/action interface.

16 Evaluation protocol details↩︎

This appendix collects the formal procedures referenced from Section 3.

16.0.0.1 Schema specification and sampling.

The training-time distribution in Equation 1 is given by the per-task schema \(\mathcal{S}_i = \{(D_{i,j}, p_{i,j})\}_{j=1}^{M_i}\), where \(D_{i,j}\) is the \(j\)-th diversity dimension and \(p_{i,j}\) is a categorical distribution over its admissible values. In the common case, \(P_i^{\mathrm{train}}(c) = \prod_{j=1}^{M_i} p_{i,j}(d_{i,j})\,p_i^{\delta}(\delta)\). Other tasks use joint distributions over subsets of dimensions (e.g., enumerated 4-item combinations in pick_items_basket, fixed orderings in press_button) or refresh certain dimensions only at batch boundaries (the change_frequency field). The full per-task schema (diversity dimensions, distributions, and constraints) is released alongside the dataset. Algorithm 6 summarizes the data-collection loop.

16.0.0.2 Per-band aggregation.

The stratified distributions in Equation 2 , the per-trial score in Equation 3 , and the aggregate metrics in Equation 4 are used for all real-robot results in Section 4. Per-band score means provide a finer-grained breakdown than the cross-task aggregate: \[\label{eq:band} \bar S_i^{B} \;=\; \frac{1}{|B|}\sum_{t \in B} S_i^{(t)}, \qquad B \in \{T_{1\text{-}4},\, T_{5\text{-}8},\, T_{9\text{-}10}\}.\tag{5}\] The single-task success rate is the special case \(\mathrm{SR}_i = \mathrm{SR}(\{i\})\) of Equation 4 and counts a trial with score \(\geq 9/10\) as a success, providing a near-completion signal alongside the partial-credit total.

17 Scoring rubric summary↩︎

4pt

18 Detailed limitations↩︎

18.0.0.1 Real-robot stochasticity.

ManipArena evaluates policies on physical robots, where repeated executions of the same checkpoint on nominally identical trials can still vary due to contact conditions, object pose perturbations, sensing noise, latency, controller variation, and reset differences. The stratified 10-trial protocol and partial-credit scoring reduce the effect of a single binary outcome, but the reported scores should still be interpreted as empirical estimates under a fixed protocol rather than deterministic checkpoint properties.

18.0.0.2 Cost and coverage.

Real-robot evaluation requires robot time, scene reset, failure recovery, and hardware supervision; the controlled studies also require training multiple checkpoints across data recipes, language fields, and model families. This cost limits the breadth of the current sweep. The tabletop model zoo includes seven configurations, mobile policy evaluation is limited to one task-specific \(\pi_{0.5}\) configuration on the five mobile tasks, and the paired simulation extension covers three tabletop tasks. These choices provide controlled evidence for the factors studied in the main paper, but do not exhaust all robot foundation models, training schedules, prompt formats, data mixtures, or real/sim recipes.

18.0.0.3 Platform and environment scope.

The controlled green-screen environments are designed to isolate evaluation variables, but they do not cover the full visual and physical complexity of open homes, factories, or outdoor environments. The current results are also tied to the tested robot embodiments, object sets, reset procedures, and scoring rubrics. Future instantiations can reuse the same task schemas and partial-credit framework while varying scene, lighting, camera viewpoint, object set, or robot embodiment.

18.0.0.4 Instruction interface.

The cross-model evaluation uses a common high-level L1 prompt to keep the test-time instruction interface fixed across VLA and WAM policies. This supports controlled comparison, but it does not characterize every prompting regime. Some policy families may benefit from more explicit scene-grounded or action-level instructions. Systematically varying test-time prompt granularity and model-specific instruction interfaces is therefore left for future work.

19 OOD authenticity audit↩︎

The 10-trial stratification (Section 3.3) reserves trials T9–T10 for the strongest held-out condition available per task. Whether that condition introduces an unseen object depends on whether object identity is part of the task semantics. Table 14 records the per-task assignment used in this paper.

6pt

For the 8 Configuration OOD tasks, T9–T10 stresses spatial, ordering, or procedural generalization rather than object semantics. As discussed in Section 3.3, this separation is intentional. Future versions of the benchmark can introduce object OOD for the remaining 8 tasks (e.g., triangular rings for put_ring, orange or purple buttons for press_button) without altering the partial-credit framework or the controlled environment.

References↩︎