1 Introduction↩︎

The ability to creatively solve problems and adapt to unexpected challenges is fundamental to intelligent behavior. In everyday environments, humans and animals routinely encounter situations where initial strategies fail, tools behave differently than expected, or misleading alternatives are present. Success in such settings relies not only on executing motor skills, but also on reasoning about object geometry, material properties, and physical constraints, and flexibly revising plans when assumptions are violated. For robots to operate reliably in the real world, they must demonstrate similar cognitive problem-solving capabilities.

Despite significant progress in robotic manipulation, current robotic benchmarks largely focus on skill-level execution under well-specified conditions [1]–[4], such as grasping, pushing, or predefined tool use [5]. As shown in Figure 1, VLAs mastering skill-level execution only struggle with small changes in the scene, while VLAs thinking out of the box could reason about the constraints implied in the scene and creatively adapt strategies. While recent VLAs showcase impressive tool-use and failure-recovery demonstrations, such as cardboard box assembly [6] and small-object collection with a paper plate [7], there is no unified benchmark that systematically evaluates a robot’s ability to reason, adapt, and handle unexpected challenges. As a result, the reasoning capabilities of current VLA models are not well understood, and the absence of systematic evaluation makes it difficult to diagnose their failure modes or identify the specific reasoning components that must be improved to advance toward more capable, reasoning-driven VLAs.

To address this gap, we introduce RoboWits, a new bi-manual robotic benchmark that explicitly targets evaluating cognitive reasoning, creative tool use, and robustness to unexpected challenges. The benchmark consists of manipulation tasks that require geometric, material, and assembly-based reasoning, where naïve solutions are often invalidated or inefficient. Tasks are organized across multiple difficulty levels, enabling fine-grained evaluation of how robotic models scale with increasing reasoning complexity.

Designing such tasks manually is time-consuming and extremely challenging for even human designers. To produce diverse, high-quality tasks at scale, we propose an automated task generation pipeline formulated as a multi-agent cooperative framework. The pipeline decomposes task construction into specialized agents with different roles: a seed task generator that proposes cognitively challenging task specifications, a task mutation generator that expands each seed into diverse task mutations where original solution are blocked by small scene changes, a task metric generator that produces executable evaluation criteria, a task verifier that ensures feasibility, simulatability, and necessity of reasoning, and a scene generator that constructs realistic and physically consistent environments. Through structured cooperation among these agents, the pipeline automatically produces a diverse and scalable set of cognitively challenging tasks that would be difficult to design by hand. We construct 30 seed tasks spanning geometry, material, and assembly-based reasoning, and expand them into 208 tasks with graded difficulty and full evaluation code. We also collect 50 human-teleoperated demonstrations for 10 of the seed tasks to facilitate benchmarking.

While cognitive reasoning is arguably learnable by imitation, existing robotic policy models remain unable to reliably perform unseen tasks in a zero-shot manner, even after large-scale pre-training. We evaluate a representative suite of models on RoboWits, including imitation learning baselines, pre-trained Vision–Language–Action (VLA) models, and Vision–Language Model (VLM)-based planners with oracle access to object states, under both single-task fine-tuning and multi-task learning regimes. Our empirical results show that pre-trained VLAs leverage their prior knowledge to outperform from-scratch models in low-data settings (50 demonstrations), yet still exhibit substantial difficulty with tasks requiring complex material interactions and assembly-level reasoning. Even with oracle state access, modular planners driven by VLMs achieve reasonable performance on seed tasks but fail to generalize effectively to mutated task variants. Overall, RoboWits offers a rigorous framework for quantifying the gap between low-level manipulation proficiency and high-level cognitive adaptation.

In summary, our contributions are as follows:

• We introduce RoboWits, a new robotic benchmark that evaluates cognitive reasoning, creative tool use, and robustness to unexpected challenges in bi-manual manipulation.

• We propose an automated, multi-agent task generation pipeline that enables scalable construction of diverse, reasoning-centric manipulation tasks.

• We benchmark existing robotic policy methods, revealing achievements and limitations in reasoning and adaptation beyond low-level skill execution.

2 Related Works↩︎

2.1 Robotic Manipulation and Benchmarks↩︎

Robot policy model has seen rapid progress recently due to increased data available and better architectures [11]–[23]. With modern physics simulators such as Genesis [24], SAPIEN [25], MuJoCo [26], IsaacGym [27], and other specialized simulators [28]–[30], many robotic benchmarks have been developed and have played an important role in evaluating and improving robot policy models [31]–[36]. RLBench [1] includes 100 diverse human-designed manipulation tasks, LIBERO [2] features task suites across four dimensions, RoboTwin 2.0 [4] provides 50 bi-manual manipulation tasks, RoboEval [8] incorporates structured evaluation, and VLABench [9] provides manipulation tasks requiring language-level reasoning. Other benchmarks such as HomeRobot [37], RoboCasa [38], and RoboSuite [39] further extend toward mobile manipulation. Despite their success, most existing benchmarks mainly emphasize skill execution under well-specified settings. Our benchmark and automated pipeline take a step further by systematically generating and evaluating bi-manual tasks that target explicit reasoning, diverse materials, creative tool use, and robust adaptation under unexpected constraints.

2.2 Creative Problem Solving↩︎

Creative problem solving has been widely studied in domains such as NLP and cognitive science, where benchmarks evaluate multi-step planning, strategy revision, and compositional generalization [10], [40]–[43]. However, these benchmarks are often limited to specific settings like text-only and require extensive human evaluation. In robotics, physics simulators provide a natural testbed to study creative problem solving through grounded interaction and automatic success metrics. Some robotic benchmarks begin to touch on related aspects, often through multi-task instruction following or language-conditioned manipulation [3], [9], [35], [44], but they do not systematically isolate and evaluate creative tool use and adaptation under unexpected constraints. In contrast, RoboWits targets these capabilities with reasoning-centric bi-manual tasks and metric-based evaluation at scale.

2.3 Automated Task Generations↩︎

Automatically generating tasks is essential for scaling benchmarks while reducing the cost of manual design and human labor. Prior work explores task diversification through domain randomization [1], [4], which mainly varies superficial factors such as instructions, textures, and object placements. Several lines of research explore scene generation via procedural generation with hand-crafted rules [2], [45], image-driven pipelines that leverage 2D priors to construct 3D layouts [46], [47], and, more recently, LLM-based methods that either generate symbolic constraints for external solvers or directly predict object placements through prompting [48], [49]. Other works focus on metric generation, where task success or even rewards are defined through programmatic checks over simulator states and physical conditions [50]–[52]. Building on these directions, we propose a unified multi-agent pipeline that jointly performs task creation, diversification, verification, scene instantiation, and metric synthesis. This enables scalable generation of reasoning-centric bi-manual tasks with diverse solution spaces, varying cognitive difficulty, and ground-truth evaluation.

3 RoboWits Benchmark↩︎

We first introduce our task settings in Sec. 3.1, then our automatic agentic task generation pipeline in Sec. 3.2, and finally present the benchmark in Sec. 3.3.

3.1 Task Settings↩︎

To evaluate robotic creative problem-solving and strategy adaptation capabilities under unexpected conditions, RoboWits tasks are designed to tightly couple cognitive reasoning with physical interactions. All tasks are set in a tabletop environment with a dual-arm robot equipped with parallel grippers. The workspace contains everyday objects with diverse shapes, sizes, and material properties. Each task \(T\) is defined by a natural language instruction \(\mathcal{I}\), an object list \(\mathcal{O}\), an initial 3D scene configuration \(\mathcal{C}\), an evaluation metric \(\mathcal{M}\), and a difficulty score \(s \in \{1,2,3,4,5\}\). \[T=(\mathcal{I, O, C, M}, s)\] The instruction \(\mathcal{I}\) specifies only the goal and intentionally avoids hinting at the solution. For example, an instruction may state “pinch the bank card" rather than “push the bank card to the table edge and pick it up". The reasoning part to discover an appropriate solution is left entirely to the robot. Each object in the object list \(\mathcal{O}\) contains a natural language description of the desired appearance and functional attributes, while \(\mathcal{C}\) instantiates all the objects with a specified 3D model, parameterized physical material, and group relationships. The evaluation metric \(\mathcal{M}\) includes both a binary success indicator and a continuous progress score function. The difficulty score \(s\) reflects both physical execution difficulty and reasoning complexity.

3.2 Task Generation↩︎

To generate diverse and cognitively challenging tasks at scale, we propose a multi-agent task generation pipeline as illustrated in Figure 2. The system consists of 5 distinct components: a Seed Task Generator, a Task Verifier, a Metric Generator, a Task Mutator, and a Scene Generator. Each component acts as an independent agent powered by a foundation model, collaborating to complete the task generation process. We describe these components in detail below and provide an illustrative example in Appendix 7.2.

3.2.1 Seed Task Generation↩︎

We define three task taxonomies: geometry-based reasoning, material-based reasoning, and assembly reasoning. These categories require robots to reason about how geometric or material properties relate to object functionality, as well as how multiple objects can be composed into a functional tool or system to accomplish a goal. To balance structure and diversity, we represent all three categories in a unified task schema, which we provided an example of in Appendix 10.6. The schema standardizes task specification and evaluation-relevant fields, while still allowing substantial variation in task content, scene composition, and difficulty. Each seed task consists of: (i) a natural-language instruction, (ii) a structured object list, (iii) a potential solution (used only for verification and not exposed to the robot), and (iv) a short description explaining why the task requires reasoning.

As shown in Fig 3, each item in the object list is defined by its appearance attributes, functional attributes, and a set of candidate instances. This abstraction encourages the Seed Task Generator to focus on an object’s underlying role in the task, rather than overfitting to a specific instance. For example, an object may be designated by the abstract name container, with appearance attributes indicating a narrow open, and functional attributes specifying a rigid material and stable placement on the table. The candidate instances then help concretize the abstraction by mapping it to everyday objects. This representation supports generating diverse task instances that share the same reasoning pattern but differ in concrete object realizations.

3.2.2 Task Verification↩︎

The Task Verifier is a tool-augmented and simulator-aware agent. It determines task validity along three dimensions: simulatability, solution feasibility, and solution efficiency.

Simulatability. The verifier first checks whether the task can be instantiated and executed within the simulator. Tasks that require physical properties or interactions beyond the simulator’s capabilities are immediately rejected. To ensure that all referenced objects can be created in the simulator, the verifier can invoke an asset retrieval tool to search for suitable 3D assets from both a local asset library and online resources³, and a 3D asset generation tool to try to create the assets. If an object cannot be matched to an appropriate 3D asset and fails to be generated, the task cannot be instantiated and is therefore rejected.

Solution feasibility and efficiency. Given the potential solution, the verifier examines whether the required operations are realistically executable, rejecting solutions that rely on extreme precision control, highly dynamic motions, or operations beyond the capability of a bi-arm robot with parallel grippers. In addition, the verifier checks whether the task admits significantly easier alternative solutions. Tasks with such shortcut solutions are rejected to ensure that success requires non-trivial reasoning.

Difficulty score. For tasks that pass the above checks, the verifier assigns an operational difficulty score \(s_o\) based on execution-related factors, including the number of required steps, the complexity of the manipulation primitives (e.g., simple pick-and-place versus precise control), and whether bi-manual coordination is required.

3.2.3 Task Mutation↩︎

Given a seed task, the Task Mutator agent collaborates with the Task Verifier agent to generate a set of diverse and valid task mutations that require a novel solution by adding minimal constraints. The mutator proposes candidate mutations by modifying the task specification (primarily the object list and object attributes), while preserving the original task goal and evaluation metric. The verifier then checks their validity and provides feedback when revisions are needed. Through this iterative process, we obtain a family of task instances with diverse challenges and graded difficulty. To systematically increase task diversity and difficulty, we design three mutation strategies.

Pivot strategy blocks the current solution by removing or modifying key objects in the object list, and introduces new objects that enable alternative valid solutions. This mutation promotes diversity in feasible solutions, rather than generating superficial variations.

Trap strategy increases both reasoning and execution difficulty by injecting "trap objects" into the scene. Trap objects are designed to appear useful but are functionally ineffective due to subtle constraints such as mismatched resizing, being fixed to the table, being placed out of reach, or having unexpected material properties. For example, a board that visually resembles a rigid lever can be assigned a soft material, making it unsuitable for applying torque. These traps substantially increase reasoning difficulty by introducing deceptive failure modes and test whether a policy can recognize failure and recover or adapt its strategy accordingly.

Add strategy increases scene complexity by adding useless objects as distractors. Although these objects don’t affect task solutions, they introduce clutter and perceptual noise, making tool selection and reasoning more challenging.

Mutation-Verification Workflow The overall mutation-verification workflow is summarized as Algorithm 9 in the Appendix 7.1. We maintain a valid task pool, initialized with the seed task. At each mutation step, we randomly sample a task from the pool and select a mutation strategy from {pivot, trap, add}. The mutator then generates a candidate task variant for the verifier to evaluate. If the candidate is deemed invalid, the verifier provides structured feedback, and the mutator revises the task accordingly. This back-and-forth refinement is allowed for up to 3 rounds. Verified task variants are added to the pool, and others are discarded. To avoid repetitive variants, we prevent applying the same mutation strategy to a task more than twice.

In practice, we prioritize the pivot strategy in the early stages to maximally explore the solution space while keeping tasks minimal (i.e., avoiding redundant or distracting objects). This step is critical for discovering fundamentally different solution strategies rather than superficial variations. After establishing diverse solution families, the trap strategy introduces deceptive but inefficient alternatives that increase cognitive difficulty. And the add strategy is applied to increase scene clutter, primarily raising execution and control difficulty without changing the core reasoning structure. We show example mutation-verification trees in Figure 10.

3.2.4 Task Scene Generation↩︎

Given a task specification, the Scene Generator agent is invoked to produce the initial 3D scene configuration, following prior work [53]. This includes automatically retrieving new 3D asset models if necessary, assigning physical properties (e.g., material types, density, and friction coefficients), and determining the initial positions and orientations of all objects. The agent is equipped with several tools to interact with the physics simulator, iteratively place objects, assign physical parameters, and render four views of the current scene and the scene after some physics steps to verify that the generated scene code is plausible and that the scene is stable for robots to execute tasks. The Scene Generator agent enables scalable, diverse scene instantiation while maintaining both physical and visual realism.

3.2.5 Task Metric Generation↩︎

The final stage of the pipeline generates task evaluation metrics, including a binary success indicator and a detailed progress score function. Following prior work [51], [52], we predefine a set of simulator APIs for querying object states and physical conditions, which the Metric Generator agent uses to produce executable evaluation code. Crucially, all mutated task variations share the same goal and thus the same evaluation metric of their seed task. This means metric generation occurs only once per seed task. While manual verification remains a standard necessity across benchmarks to mitigate edge cases, our design drastically reduces this scaling bottleneck. By amortizing evaluation logic across mutation task families, humans verify just 30 unique scripts to guarantee reliable high-fidelity evaluation for the entire 208-task benchmark with minimal labor.

3.3 Benchmark Overview↩︎

RoboWits contains \(208\) diverse bi-manual manipulation tasks generated by our automated agentic pipeline introduced in Sec. 3.2, featuring \(30\) seed tasks and \(3\sim9\) mutations each requiring novel solutions, spanning three categories: geometry-based reasoning, material-based reasoning, and assembly reasoning, as shown in Fig. 5. Each task is assigned a difficulty score ranging from 1 to 5, reflecting increasing levels of reasoning and execution complexity, as shown in Fig. 6. Table [tab:delta95summary] further demonstrates the mutation strategies successfully adjust task difficulty as intended. Each task captures a distinct problem-solving pattern and is paired with a fixed evaluation metric that precisely defines task success. We instantiate all the tasks in Genesis [24], a unified physics simulator that supports diverse materials and robot control.

To support training and fine-tuning, we collect and release human teleoperation demonstrations for RoboWits. For the 30 seed tasks, we provide demonstrations from simulated teleoperation.

4 Experiments↩︎

We conduct a series of experiments on RoboWits to answer three core research questions.

• RQ1: Can Vision-Language-Action models control robots to creatively solve manipulation problems?

• RQ2: Can Vision-Language-Action models perform robustly in the face of unexpected challenges?

• RQ3: Can Vision Language Model-driven modular planners think out of the box?

4.1 Experimental Setup↩︎

While the capability to creatively solve manipulation problems is arguably learnable by imitation, and our benchmark demands zero-shot evaluation ideally, no existing robotic models have demonstrated such generalization. We follow similar principles in robotics manipulation benchmarks [1], [2] to fine-tune/train robotic models in a multi-task setting with 50 collected human demonstrations on 10 seed tasks, and test them on the seed tasks for 50 trials, and each of the mutation tasks for 5 trials respectively. All trials have randomized object position and rotations.

Robot system details. The robot system setup for RoboWits uses two 7-DoF Marvin arms with three cameras (two wrist and one base) and a 16-dimensional configuration and action space.

Baselines. We evaluate pre-trained VLAs, robot policy models, and VLM-based modular planners.

• ACT [54], a transformer-based policy that learns action chunking from demonstrations

• \(\pi_0\) [55], a pre-trained 3B VLA that predicts continuous action with flow matching.

• \(\pi_{0.5}\) [56], a 3B VLA pre-trained with internet-scale data and robotics data that can generalize better.

• VLM Planner\(^{**}\) A modular planner with oracle access to object states and perfect action primitives. It employs a VLM (gpt-4o) for high-level planning and assumes error-free execution.

• VLM Controller\(^*\) A modular planner similar to the above, but without perfect action primitives. It uses the same VLM for planning but relies on a human-scripted low-level controller for execution.

4.2 Experimental Results↩︎

Current VLAs struggle on RoboWits** even with single-task fine-tuning.** As detailed in Table 2, even after fine-tuning on 50 human demonstrations, pre-trained models like \(\pi_0\) exhibit only preliminary success on seed tasks. In contrast, imitation learning baselines trained from scratch, such as ACT, fail significantly; this highlights the difficulty of our tasks and the insufficiency of limited demonstrations for models lacking strong vision-language pre-training.

VLAs lack robustness under unexpected challenges. A comparison between seed tasks and their corresponding mutations reveals a drastic performance collapse across all VLAs, with progress scores frequently halved. For instance, in the Align Blocks task, \(\pi_0\) achieves a \(14\%\) success rate on the seed version but fails \(90\%\) of the time when faced with constraints like immovable blocks or occluded tools. This suggests that current VLAs rely on superficial correlations rather than a generalized understanding of task physics.

Vision-language planners show promise through closed-loop reasoning. Given oracle object states and action feedback, VLM-based planners achieve reasonable performance on seed tasks and demonstrate superior robustness to mutations. Our ablation study comparing open-loop and closed-loop variants underscores the necessity of reactivity: closed-loop planning (re-planning after each action) outperforms the open-loop baseline by 27.4 percentage points on seed tasks and 29.3 points on mutations. This gap confirms that real-time feedback is critical for adapting strategies to the unexpected challenges inherent in our benchmark.

Further findings, along with comprehensive supporting results, are detailed in Appendix 9.

5 Conclusion↩︎

We present RoboWits, a new bi-manual manipulation benchmark designed to evaluate robotic creative problem solving, tool use, and robustness to unexpected challenges. To scale task construction beyond manual design, we propose an automated multi-agent pipeline that jointly performs seed task creation, task diversification, verification, scene instantiation, and metric synthesis, enabling diverse reasoning-centric tasks with ground-truth evaluation. Using this pipeline, we build 30 seed tasks and expand them into 208 tasks with graded difficulty across geometry, material, and assembly reasoning. Comprehensive experiments on popular robot policy models in both zero-shot and fine-tuned settings show that while fine-tuning improves basic manipulation skills, substantial gaps remain in reasoning, strategy adaptation, and robustness under deceptive or constrained conditions. We hope RoboWits will serve as a scalable testbed for developing and diagnosing next-generation reasoning-driven VLA models and robust robotic problem-solving systems.

6 Task Details↩︎

The details of 30 seed tasks are as follows. The ego-centric view of the seed task gallery is in Figure 8.

1. align blocks: Align the three cubes perfectly in a straight line.
   The task is to arrange multiple scattered cubes into a precise linear sequence. Attempting this cube-by-cube with a standard gripper is inefficient and prone to misalignment. The solution exploits the one-dimensional geometric property of the ruler’s edge to act as a physical constraint, ensuring that all objects sharing contact with that edge are perfectly aligned.

2. retrieve cube: Retrieve the cube from the container and place it on the green target area.
   The challenge involves retrieving a target object that is not directly reachable by the robot’s end-effector due to geometric constraints. The task requires the robot to first pour the cube out of the deep narrow container and then move it to the target area.

3. gap retrieve: Place the lemon on the green target area.
   The task is to place a lemon on a target area. The lemon is located in a narrow channel. The channel is formed by the left and right boundary blocks, creating a space too narrow for a standard robotic gripper to enter. The solution requires first moving one boundary block to form a larger gap, then grasping the lemon and placing the lemon on the target area.

4. pinch card: Pick the bank card up from the ‘small table’ surface.
   The ‘bank card’ is too thin and slippery for a standard gripper to push directly on a flat surface. The robot must utilize the friction and deformability of the ‘eraser’ to create enough lateral force to move the card. This material property is essential because the high-friction interface allows the robot to translate the card to a position (the small table’s edge) where its geometric constraints no longer prevent a successful grasp.

5. roll up box: Move the box into the basket.
   The task is to move a box vertically onto a basket. Since the box is too heavy to lift, the robot must use the inclined plane (a simple machine formed by the board and the basket edge) to reduce the required force. This utilizes the box’s sliding property to smoothly ascend the ramp surface.

6. dominos: Push the red block onto the green target area.
   The ‘red block’ and ‘white block 1,2,3’ are out of reachability, but happen to form a domino structure when they fall. The robot must use the ‘white block 4’ within the reachability to reach ‘white block 3’. To make a perfect chain of falling, the robot must push the upper part of ‘white block 3’ and finally, the ‘red block’ is pushed onto the ‘target area’.

7. stand pages: Assemble the two pages using the stabilizing bar to create a standing hinge structure.
   Individually, the pages lack the base width to overcome tipping forces. The robot must transform these 2D objects into a stable 3D structure by utilizing a pin-joint mechanism. By inserting the bar through the holes, the robot creates a mechanical constraint that links the two objects, allowing them to lean against each other and support their combined weight against gravity.

8. round dough sheet: Flattern the dough ball and make a perfect round sheet
   The task requires creating a precise round shape. It’s difficult to flatten the dough evenly with a parallel gripper, so the robot uses the planar surface of the board to distribute pressure. Achieving a perfect round shape requires utilizing the circular symmetry of the separate cutting-edge object.

9. hold cup: Put the cup on the green target area on the slope without sliding down.
   The ‘cup’ and ‘slope’ lack sufficient friction to remain stationary. The robot must utilize the surface friction and thin-film flexibility of the cloth to stabilize the cup. This material property is essential because the coaster acts as a high-friction interface that compensates for the low-friction property of the standard rigid materials (the cup and the table).

10. collect screws: Collect the screw into the container
    The screw is too small to be easily grasped directly. The robot must use the dustpan and spatula to collect the screw first and then put it into the container.

11. place tall box: Move the rectangular box onto the green target area.
    The obstacle enforces a geometric clearance constraint. The beam cannot be moved or raised. The solution relies on changing orientation to reduce the effective height, exploiting the different cross-sectional dimensions of the box to fit under the low beam. Tipping the box to a diagonal lean to and pushing it under the beam is not feasible, because the gripper cannot keep the balance of the box.

12. ball into bottle: Transfer the ball from the blue container into the red bottle.
    The ‘funnel’ provides a wide capture entrance. Gravity guides the ball through the spout into the bottle. The ‘ball container’ is graspable and can be poured into the funnel opening. Otherwise, without the funnel, directly pouring the ball into the bottle’s mouth is very difficult and requires high precision.

13. cover with lid: Cover the pot precisely using one lid.
    This requires reasoning about cross-sectional fit. Only the correctly sized circular lid will cover the opening with minimal excess. The circular geometry and ring reference enable precise centering without measurements.

14. stack cubes: Build a stable two-layer stack so the red cube’s center is above the green colored dot.
    A wide base increases stability by lowering the combined center of mass and increasing the support polygon. The geometric arrangement is necessary to keep the top cube centered over the marker.

15. separate marbles and sand: Remove sand from the jar and keep the marbles in the jar.
    Individual grains are too small to pick efficiently. By assembling a colander over a bowl, the robot creates a gravity-powered separator that splits marbles from sand and enables clean transfer into the jar.

16. stand bulb: Stand the tall object with a curved bottom stably within the green target area.
    A tall object with a curved bottom is prone to tipping due to a curved base. A surrounding ring provides lateral support and increases the effective base, using geometry for stability.

17. ball onto tower: Place the ball stably on top of the tower.
    The tower pole alone cannot support the ball as it would roll off. The base can only fit 2.5 rings inside its height, so stacking all three rings causes the top ring to protrude above the pole, creating a hollow cup-like surface where the ball can rest stably in the center.

18. cylinder through hole: Insert the long rod through the opening so it rests over the green colored patch.
    A circular peg requires coaxial alignment to pass a circular hole. This uses cross-section geometry and orientation matching.

19. stack bowls: Nest the smaller bowl concentrically inside the larger one.
    Concentric alignment leverages circular geometry; matching rims provide a strong geometric signal for proper nesting.

20. ball into jar: Put the foam ball fully into the jar.
    The ‘foam ball’ is larger than the ‘glass jar’ mouth when undeformed. The task leverages the elasticity and compressibility of the foam to change its shape and pass through the restriction, which is not possible with a rigid sphere.

21. seal colander: Make the container hold water for a short time.
    The perforations of the ‘perforated container’ prevent it from holding liquid. By placing the ‘curved holder’ onto the ‘perforated container’, the robot can pour the ‘water’ from the ‘pitcher’ into the ‘curved holder’ onto the’perforated container’ and verify it holds.

22. stabilize bottle: Make the tube stand upright in the bowl without falling for a short time.
    The narrow base of the ‘tube’ makes it unstable. Granular packing of the ‘dry sand’ around the base provides distributed support and friction, stabilizing the tube without direct clamping by the gripper.

23. place book: Move the ‘thin book’ onto the ‘blue target mat’ without moving the heavy block from its place.
    This is a practical tabletop retrieval scenario: a book is pinned under a heavy object (like a paperweight or block) and must be pulled out without disturbing the weight. The pry board acts as a shim to get under the book edge, locally reducing contact pressure and friction so the book can be slid out while the block is stabilized from above.

24. raise platform: Create an elevated platform and place the book on it.
    The goal requires supporting a load above the table. Assembling two equal-height supports and a rigid deck creates a stable platform that can hold the book without touching the table.

25. water into mug: Collect water in the mug without moving the pitcher.
    The ‘pitcher’ is fixed and cannot be tilted to pour water directly. By dropping the ‘heavy large object’ into the pitcher, the robot exploits Archimedes’ principle of water displacement to raise the water level and cause overflow into the ‘mug’, demonstrating understanding of fluid displacement physics.

26. align chopsticks: Make all six chopsticks point in the same direction, with their small thin ends all facing the same way.
    The task is to align six chopsticks so they all point in the same direction. Initially, the chopsticks lean on a board with mixed orientations, some with the heavy end on the board, others with the thin end. By pushing a ruler to slide the chopsticks further onto the board, the asymmetric mass distribution causes chopsticks of different orientations to behave differently: those with the heavy end on the board will settle flat, while others remain tilted. This physical cue reveals which chopsticks need to be flipped, enabling efficient sorting and alignment.

27. retrieve roll: Use the long rod to retrieve the out-of-reach roll and move it to the green target area.
    The roll is placed beyond the robot’s direct reach, making it impossible to grasp directly. By using the rod as a tool extension, the robot can extend its effective reach. Inserting the rod into the hollow part of the roll creates a mechanical coupling through friction and geometric constraint, allowing the robot to lift and transport the roll as a single rigid-body system. This task demonstrates tool use for reach extension and object manipulation through insertion.

28. move cube: Move any of the three cubes to the left of the goal line.
    This task requires the robot to understand and exploit the friction properties of different surfaces. The smooth slope has very low friction, allowing objects to slide freely, while the rough slope has very high friction that would stop the cube. The robot must select the correct slope to enable the cube to gain enough momentum to slide past the goal line. This tests the robot’s ability to reason about material properties and their effect on object dynamics.

29. balance board: Place the board onto the wall and keep it balanced.
    The goal requires balancing a board on a narrow support. The key insight is to recognize that the board must be placed with its center of mass directly above the wall to maintain equilibrium.

30. differentiate cubes: Place the wooden cube to the green target area. Keep the metal cube inside the mug.
    The challenge involves differentiating two objects that appear similar but have different material properties. The robot must use physical reasoning to identify which cube is wooden by exploiting the buoyancy property - wood floats in water while metal sinks. This requires the robot to perform an intermediate action (pouring water) to reveal the hidden material property before completing the placement task.

7 Method Details↩︎

7.1 Mutation and Verification Process↩︎

The mutation and verification process can be summarized as Algorithm 9.

7.2 An Illustrative Case: Retrieve Cube↩︎

As shown in Figure 4, we use the retrieve cube task as a running example to illustrate how different mutation strategies generate diverse task variants. The seed task retrieve cube requires the robot to retrieve a cube from a narrow-opening container and place it onto a green target area (Initial). Since the opening is too small for the gripper to enter, a natural potential solution is to lift the container, pour the cube out, and then place the cube onto the target area.

During the pivot phase, one effective way to block the original solution is to fix the narrow-open container to the table, making it impossible to lift and tilt anymore. Under this constraint, one pivot introduces a long spoon with a small head, enabling the robot to scoop the cube out (Pivot1). Another pivot adds a cup of water and specifies the cube has low density, allowing it to float upward until it becomes reachable (Pivot2). Building on (Pivot2), the water cup is replaced with a water tank, making direct pouring infeasible. The task is then pivoted again by adding a kitchen ladle, which allows the robot to transfer water from the tank into the container (Pivot3).

As examples of trap mutations, based on (Pivot1), a large spoon is added as a trap (Trap1), since it appears useful for scooping but it can’t fit in the narrow opening. Another trap adds a cup of water (Trap2) to Initial, which may look helpful but leads to a less efficient and more difficult strategy compared to scooping. Also based on (Initial), a wooden skewer is added to mislead an implausible strategy of stabbing the cube and pulling it out (Trap3); however, the cube is rigid and can’t be penetrated, making the skewer inefficient.

Finally, for add mutations, tasks sampled from the task pool (Initial, Pivot1-3, Trap1-3) are augmented with random distractor objects from a predefined list. In this example, distractors include an IKEA plate, a TV remote controller, a copper kettle, a lemon, a paper cup, a water bottle with handle and pump, and a wrench, which form additional cluttered variants (Add1-3).

7.3 Example Mutation↩︎

Fig 10 depicts three examples of mutation trees.

8 Baseline Details↩︎

VLM Controller and VLM Planner are agents based on gpt-4o, who has access to different levels of oracles. The prompt templates can be found in Appendix 10.8 and  10.9.

ACT is trained for 100,000 steps with a chunk size of 50, batch size of \(256\).

\(\pi_0\) is fine-tuned for 100,000 steps with an effective batch size of \(64\).

\(\pi_{0.5}\) is fine-tuned for 100,000 steps with an effective batch size of \(64\).

8.1 Compute↩︎

We trained baseline models on 8 RTX 6000 for two days on average.

9 Additional Results↩︎

9.1 Benchmark Coverage Statistics↩︎

To better illustrate the coverage of the tasks in our benchmark, we compiled the following statistics.

9.2 VLM Baselines with Gemini 3 Pro↩︎

We conducted additional evaluations of the VLM Planner using Gemini 3 Pro. As shown in Table 4, the overall performance is comparable to GPT-4o, although we observed slightly higher API latency with Gemini.

For the results reported in the paper, we keep a minimalist prompt design, focusing on standardizing the I/O format and defining available action primitives. We tried prompt tuning and did not observe obvious differences. From our observation, the planner is not sensitive to specific prompt engineering, and the bottleneck lies in the model’s core reasoning ability.

9.3 Failure Analysis across Mutation Types↩︎

To clarify whether failures stem from reasoning, execution, or other factors, we analyzed success rates (SR) across mutation types. Here, Add(n) denotes introducing \(n\) irrelevant distractor objects during the Add mutation.

Video observations reveal that the specific physical errors directly correlate with the mutation type.

For Pivot, which has the lowest SR, models show no strategy transfer. Physically, they may fail by stubbornly attempting to manipulate immovable objects or continuing original trajectories even when objects are accidentally displaced, indicating a true lack of physical reasoning.

For Trap and Add, SR drops as more objects are introduced, with Add(1/2) SR \(>\) Add(3) SR. The physical errors here are primarily driven by distraction, where models may physically freeze due to out-of-distribution visual data or incorrectly reach distractor objects.

9.4 Demonstration Scaling↩︎

To investigate whether more demonstration data could help models perform better on RoboWits, we train \(\pi_0\) with 50 and 200 demonstrations, respectively, for two representative tasks.

Dominos relies on a single optimal strategy, i.e., using a pusher to topple the dominos. Scaling the demonstration data provides a clear, consistent signal to resolve the motor-control bottleneck, improving Seed performance from 70% to 94%. In contrast, Align Blocks requires closed-loop adjustments across multiple valid strategies, such as gathering blocks before aligning them, or aligning two blocks first and then placing the third. This multi-modal nature explains its marginal improvement from data scaling and its greater robustness.

Most importantly, despite improvements on the Seed tasks, performance on the Mutation tasks remains low. This confirms our main conclusion: current VLAs cannot perform robustly when faced with unexpected reasoning-level challenges. The primary bottleneck is indeed poor reasoning rather than control limitations.

9.5 Limitations↩︎

As a simulation-based benchmark, this work is limited by the sim-to-real gap. Although we use high-fidelity rendering and advanced physics engines to narrow this divide, perfectly replicating complex real-world contact dynamics remains an open challenge. Additionally, the reliance on highly realistic physical interactions yields a computational trade-off: the non-linear dynamics of soft-body simulations are computationally expensive, which limits the rapid execution of large-scale evaluations compared to rigid-body counterparts.

10 Schemas and Prompt Templates↩︎

10.1 Prompt Template for the Seed Task Generator↩︎

10.2 Prompt Template for the Task Verifier↩︎

10.3 Prompt Template for the Task Mutator↩︎

10.4 Prompt Template of Metric Generator↩︎

10.5 Prompt Template for the Scene Generator↩︎

10.6 Task Schema↩︎

An example of task representation.

10.7 Verification Result Format↩︎

An example of a task verification result.

10.8 Prompt Template for the VLM Controller↩︎

10.9 Prompt Template for the VLM Planner↩︎

11 Potential Social Impacts↩︎

Positive Impact. This work advances the study of reasoning-driven robotic manipulation by introducing RoboWits, a benchmark that evaluates creative problem solving, tool use, and robustness to unexpected challenges. By enabling more systematic diagnosis of reasoning failures and guiding the development of more capable robot policies, our benchmark may contribute to safer and more reliable robotic systems for real-world applications such as household assistance, elder care, warehouse logistics, and disaster response. In addition, our automated task generation pipeline reduces the cost of manual benchmark design and supports scalable, reproducible evaluation, which can accelerate progress in robotics research and education.

Negative Impacts. As with many advances in robotics and automation, improved manipulation capabilities may contribute to labor displacement in domains that rely on repetitive manual work. There is also potential for misuse of stronger robotic tool-use and problem-solving abilities in adversarial settings. Finally, components of our pipeline may rely on foundation models that can produce incorrect or biased outputs, which could affect task generation quality. Careful deployment, human oversight, and continuous auditing are necessary to mitigate these risks and ensure that such systems are used for socially beneficial purposes.

References↩︎