Abstract

Bridging the sim-to-real gap is a core challenge in deploying learned manipulation policies. Sim-to-real learning is attractive because it can replace expensive real robot demonstrations with scalable synthetic data, yet world-action models have not previously been shown to transfer from simulation to real robotic manipulation. We study whether a world-action model can be trained from synthetic priors and deployed zero-shot in the real world. To this end, we build upon Cosmos Policy, a video diffusion model adapted for visuomotor control. We construct simulation environments with extensive domain randomization and generate demonstrations using the AnyTask motion planning pipeline. We evaluate our approach across object lifting, drawer opening, and pick-and-place tasks using \({\sim}800\) synthetic demonstrations per task and no real demonstrations. When deployed zero-shot on a Franka Robot, our policy attains a 35% average success rate. To our knowledge, this represents the first successful sim-to-real transfer of a world-action model for robotic manipulation.

Note: This paper represent a part of early result of our official world-action model zero-shot sim-to-real transfer work, which will be released soon.

Figure 1: Qualitative comparison between model-predicted and synchronized live camera observations during put strawberry into bowl. The three groups correspond to the approach, grasp, and lift stages. In each group, the overlays “Pred Cam” and “Live Cam” denote the predicted future RGB view from the world-action model and the real camera frame observed during execution, respectively. The side-by-side alignment illustrates that the policy’s predicted frames preserve the object location, robot pose, and task stage despite training only in simulation. — Figure 1: Qualitative comparison between model-predicted and synchronized live camera observations during *put strawberry into bowl*. The three groups correspond to the approach, grasp, and lift stages. In each group, the overlays “Pred Cam” and “Live Cam” denote the predicted future RGB view from the world-action model and the real camera frame observed during execution, respectively. The side-by-side alignment illustrates that the policy’s predicted frames preserve the object location, robot pose, and task stage despite training only in simulation.

1 Motivation↩︎

Training manipulation policies in the real world is prohibitively expensive. While simulation offers a scalable alternative through parallelized data collection [1], [2], inherent visual and physical discrepancies, known as the sim-to-real gap, often lead to transfer failures [3], [4]. A successful sim-to-real system can therefore reduce or eliminate the need for costly real-world teleoperation data.

We focus on world-action models: unified generative models that predict future visual observations together with robot actions [5]–[7]. Prior work has demonstrated the promise of such models in sim-to-sim or real-to-real settings, but it remains unclear whether they can cross the simulation-to-real gap for robotic manipulation. This gap is important because a positive result would extend the benefits of sim-to-real learning—cheap, scalable, automatically generated data—to a new class of policy models.

Our contribution is therefore not an ablation proving that joint video-action prediction is necessary for sim-to-real transfer. Instead, we ask a simpler empirical question: can a world-action model trained entirely from synthetic demonstrations perform zero-shot real-world manipulation? To answer this question, we combine Cosmos Policy with extensive domain randomization and AnyTask’s [8] automated demonstration generation, and present, to our knowledge, the first successful sim-to-real transfer of a world-action model.

2 Methodologies & Experiments↩︎

2.1 Cosmos Policy↩︎

Cosmos Policy [7] adapts a pretrained video diffusion model (Cosmos-Predict2) into a robot policy through single-stage post-training. In this framework, actions, future observations, and value estimates are encoded as latent frames within a unified diffusion process.

2.2 Simulation and Demonstration Generation↩︎

We construct training environments using GPU-accelerated simulation [1] with comprehensive domain randomization: (1) textures: object surfaces, table materials, and backgrounds drawn from procedural and photographic libraries; (2) camera poses: perturbed translations and rotations of wrist and third-person cameras; (3) lighting: varied number, position, intensity, and color temperature of scene lights; (4) object placements: uniformly sampled within the reachable workspace. These variations are designed to cover a broad range of visual appearances so that the real workspace is more likely to fall within the randomized training distribution, without requiring us to manually match the real test scene [3].

We leverage AnyTask [8] to generate synthetic priors, i.e., expert demonstrations, via foundation-model-guided motion planning (ViPR) without human teleoperation across four tasks (lift banana, lift brick, open drawer, put strawberry into bowl), as shown in Figs. [fig:title] and 1. This process produces \({\sim}\)800 demonstrations per task (\({\sim}3{,}200\) total) of RGB observations paired with end-effector action trajectories.

2.3 World-Action Model Training↩︎

The training of Cosmos Policy runs for 32 epochs on 40 H100 GPUs (\({\sim}\)72 hours). Figure 1 provides a qualitative forward-prediction check on real executions: the model-predicted frames are visually consistent with the synchronized real camera observations at the approach, grasp, and lift stages. This visualization suggests that the model has learned real-scene dynamics useful for execution, even though all policy training data come from simulation.

2.4 Zero-Shot Sim-to-Real Transfer Results↩︎

We deploy the trained policy directly on a real Franka Research 3 without collecting real demonstrations or fine-tuning the policy on real data. The only real-world setup is the standard camera configuration used at inference time: fixed RGB views from the wrist and third-person cameras are mounted and streamed to the policy. Table 1 summarizes the success rates across the four tasks.

Table 1: Real-robot success rate over 10 trials per task. Our world-action model is trained with \({\sim}800\) synthetic demonstrations per task and *zero* real demonstrations. Diffusion Policy (DP) baselines are trained with 10 or 50 real demonstrations per task, providing a real-data reference rather than an architecture-matched ablation.
Method	Banana	Brick	Drawer	Strawb.	Avg.
DP w/ 10 real demos	0/10	0/10	2/10	0/10	5%
DP w/ 50 real demos	4/10	3/10	3/10	0/10	25%
Ours (800 sim)	5/10	5/10	2/10	2/10	35%

2.5pt

Figure 2: Out-of-distribution generalization on a real robot: although the bottle object is not part of the training task set, the policy approaches, grasps, and lifts it using the same zero-shot sim-to-real policy.

Our trained model achieves a 35% average success rate using \({\sim}800\) fully synthesized simulation demonstrations per task and zero real-world demonstrations. The DP baselines in Table 1 should not be interpreted as a controlled proof that world-action models are categorically better than Diffusion Policy; they use real demonstrations and therefore answer a different question. Instead, they provide a practical reference for the cost that sim-to-real methods aim to avoid. The key result is that a world-action model trained entirely from automatically generated synthetic data can achieve nonzero zero-shot real-world manipulation performance, reaching a higher average success rate than these real-demo DP references in our setup.

We attribute this transfer ability to the combination of pretrained video priors, broad domain randomization, and the joint action-video prediction objective, while leaving a controlled ablation of each factor to future work. Qualitatively, Fig. 1 further shows that, even without manually reproducing the real test scene in simulation, the model’s predicted future frames remain aligned with the live robot rollout.

Out-of-distribution generalization. We further evaluate lifting a novel bottle unseen during training (Fig. 2). The policy successfully approaches, grasps, and lifts it, suggesting that the model acquires generalizable visuomotor primitives rather than only memorizing the original task objects.

3 Conclusion↩︎

We presented, to our knowledge, the first zero-shot sim-to-real transfer of a world-action model for robotic manipulation. Across four real-world tasks, the policy achieves 35% average success using \({\sim}800\) fully automated simulation demonstrations per task and zero real robot demonstrations. This result shows that world-action models can inherit the central benefit of sim-to-real learning—reducing dependence on expensive real data—and opens a data-efficient direction for transferring generative robot policies from synthetic priors to the real world.

References↩︎

[1]

V. Makoviychuk et al., “Isaac Gym: High performance GPU-based physics simulation for robot learning,” arXiv preprint arXiv:2108.10470, 2021.

[2]

W. Zhao, J. P. Queralta, and T. Westerlund, “Sim-to-real transfer in deep reinforcement learning for robotics: A survey,” arXiv preprint arXiv:2009.13303, 2020.

[3]

J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel, “Domain randomization for transferring deep neural networks from simulation to the real world,” in IEEE/RSJ international conference on intelligent robots and systems (IROS), 2017.

[4]

OpenAI et al., “Solving Rubik’s cube with a robot hand,” arXiv preprint arXiv:1910.07113, 2019.

[5]

D. Ha and J. Schmidhuber, “World models,” arXiv preprint arXiv:1803.10122, 2018.

[6]

Y. Du et al., “Learning universal policies via text-guided video generation,” Advances in Neural Information Processing Systems, vol. 36, 2024.

[7]

M. J. Kim et al., “Cosmos policy: Fine-tuning video models for visuomotor control and planning,” in International conference on learning representations, 2026.

[8]

R. Gong et al., “AnyTask: An automated task and data generation framework for advancing sim-to-real policy learning,” arXiv preprint arXiv:2512.17853, 2025.

Efficient Sim-to-Real Transfer of World-Action Models from Synthetic Priors