1 Introduction↩︎

We have witnessed impressive generalization capabilities of robotic models [1]–[3] in certain tabletop or kitchen scenarios, achieved through high-quality teleoperation data. However, collecting real-world expert data [4], [5] remains a time-consuming and costly process. Despite advancements in teleoperation systems [6]–[8], the labor involved is still intensive, making this a key challenge in robotics research. Recent efforts [9], [10] have explored inter-institutional collaboration to address this issue. In contrast, simulation data offers the advantage of exponential scalability with computational resources, making it an appealing and renewable alternative for training robotic policies. Research on sim-to-real transfer [11], [12] has increasingly focused on generating high-quality simulated or synthetic data to train real-world robotic policies. Unfortunately, simulation data often exhibits significant sim-to-real gaps, making it necessary to collect target domain data for effective policy fine-tuning.

In real-to-sim-real scenarios, consistency with the real world in appearance and geometry is essential. High-quality RGB rendering ensures visual consistency between real and simulated domains, while accurate mesh reconstruction captures scene geometry for realistic interactions. To achieve this, we propose a 3D-photorealistic real-to-sim-to-real system, Re³Sim, short for reconstruction-rendering-based real-to-sim. Re³Sim faithfully replicates real-world scenarios by integrating photorealistic RGB rendering with precise 3D geometry reconstruction. The visual rendering is realized via Gaussian rasterization [13], while the 3D geometry is reconstructed using multi-view stereo (MVS) techniques [14], [15]. This dual-stage pipeline enables efficient, high-fidelity simulation, where physical dynamics are handled by physics engine backends [16]–[18], and real-time rendering is achieved through a dedicated hybrid rendering engine.

Specifically, Re³Sim adopts a sequential real-to-sim pipeline comprising three key steps: (a) mesh recovery for reconstructing scene and object geometry; (b) hybrid visual rendering for compositing foreground and background elements, and (c) real-world alignment to synchronize world coordinates between the real and simulated environments. Human involvement is minimal in the real-to-sim process and limited to: (a) Placing ArUco markers for real-sim alignment. (b) Capturing images/videos of the scene and objects, including an extra step for robot base alignment.

After importing all assets and robots into the simulator, Re³Sim utilizes a privileged policy to collect high-fidelity expert data. It incorporates the following features to enhance sim-to-real transfer:

• High-fidelity geometry and vision: Accurately reconstructs geometry and generates photorealistic views, closely matching real-world appearances to reduce sim-to-real gaps.

• Rapid scene reconstruction: Reconstructs novel scenes with under three minutes of manual effort.

• Efficient rendering: Achieves 24 FPS at 480p across two independent camera views.

To validate the effectiveness of the pipeline, we design several tabletop tasks demonstrating the generality and applicability of the real-to-sim-to-real process, as shown in Fig. [fig:teaser] The zero-shot sim-to-real policies achieve an average success rate of 58% with only about 10 minutes of data collection in simulation, showcasing the effectiveness of our approach. We expect these experiments to highlight the potential of the real-to-sim-to-real pipeline in reducing sim-to-real gaps, paving the way for future advancements in scalable, efficient, and generalizable robotic policy.

2 Background↩︎

2.1 3D Reconstruction↩︎

3D reconstruction recovers scene geometry from inputs like images, point clouds and etc. Camera poses and sparse point clouds can be obtained from structure-from-motion approaches (e.g., Colmap) and be used to reconstruct dense point clouds via either traditional methods [19] or network-based approaches [20]. Besides, commercial software solutions, such as ARCode and PolyCam, leverage RGB-D SLAM methods to estimate camera poses and depth information, and often include some post-processing techniques (e.g., smoothing, inpainting) for enhancing the usability and quality of the reconstructed 3D models.

2.2 Rendering↩︎

Rendering involves generating photorealistic images from specified camera views based on either explicit or implicit 3D representations. Implicit methods, like Neural Radiance Fields (NeRF [21]), represent the 3D space using a density field encoded in a neural network and render images through volumetric rendering techniques. Explicit representations, such as meshes or point clouds, employ texture mapping blinn1976texture?. 3D Gaussian Splatting (3DGS) [13] uses Gaussian primitives and adopts Gaussian rasterization for high-fidelity rendering. Compared with other methods, 3DGS achieves more realistic rendering results and offers higher rendering efficiency. In detail, each Gaussian primitive is modeled as an ellipse with a full 3D covariance matrix \(\Sigma\) defined in world space, centered at point \(\mu\): \(G(x) = \exp({-\frac{1}{2} x^T \Sigma^{-1} x}),~\) where the covariance matrix \(\Sigma = R S S^T R^T~\) of a Gaussian primitive is derived from its rotation \(R\) and scaling factor \(S\). Additionally, each Gaussian is assigned a color \(c_i\) and depth-ordered using the Z-buffer. Using the alpha-blending formula, the rendered color \(C\) for a view is calculated as: \[C = \sum_{i=1}^N \prod_{j<i} \alpha_j (1 - \alpha_i) c_i~.\]

3 Re³Sim↩︎

Robot simulators [16], [18], [22] offer high-accurate simulation results based on underlying physics engines [22]–[24]. However, it’s hard to directly obtain an accurate 3D model of the real-world scene, and the simulators are not good at rendering high-fidelity images, leaving a large gap from real-world images. We introduce Re³Sim, a 3D reconstruction-based system, to bridge the visual gap and recover realistic 3D meshes for sim-to-real transfer.

A straightforward approach is to reconstruct objects and the background together. However, using Multi-View Stereo (MVS) methods for background reconstruction often leads to suboptimal visual quality, while techniques like SuGaR  [25] struggle with accurately reconstructing flat planes, causing protrusions and indentations that reduce realism in object movement. Additionally, segmenting foreground objects for manipulation is labor-intensive. To address these issues, we propose separately reconstructing background meshes for collision estimation and leveraging 3DGS to improve rendering realism, aligning the two. Since object rendering constitutes a small portion of the robot’s visual observations, we opt not to use 3DGS to render objects.

3.1 System Overview↩︎

Re³Sim is a real-to-sim-real system that reconstructs meshes for collision estimation, utilizes 3D Gaussian scene representations for rendering, and generates simulation data within a physics-based simulator². It is worth noting that we only consider reconstructing the background and foreground objects, and robots are not involved in our system since their accurate 3D assets are usually easily accessible. Further, we take this work as an initial study of Re³Sim, validated on a table-top robot arm for rigid body tasks. As shown in Fig. 1, Re³Sim are composed of four components: mesh recovery for object collision computation, hybrid visual rendering combining 3DGS for the background and mesh-based rendering for the foreground, real-world alignment of the reconstructed scene and robot positions, and large-scale data generation for manipulation tasks.

3.2 Mesh Recovery↩︎

This step reconstructs objects and background geometries for accurate collision detection.

Mesh reconstruction. The meshes of backgrounds and objects are reconstructed separately for flexibility and are represented in USD for extensibility. For the background, we first employ SfM techniques (COLMAP) to estimate the pose for each image and get the sparse point cloud based on a set of images, and then utilize OpenMVS [15] for detailed mesh reconstruction. PolyCam [26] is also able to reconstruct the mesh of the background, but empirically, the geometric reconstruction results of OpenMVS exhibit fewer minor protrusions and depressions. Subsequently, we simplify this mesh to create collision bodies. This approach effectively captures intricate details while maintaining accurate planar reconstruction. For foreground objects, we employ ARCode [27], which can automatically segment the object, balancing usability and quality.

Post-processing. Reconstructed collision bodies often have voids or sharp edges due to imperfect image quality, causing erratic behavior in simulation. For example, objects may fail to rest stably on surfaces. Void filling and surface smoothing help improve stability and realism in simulation. As acquiring physical parameters like mass and friction is challenging with only initial static visual data, we use default values to simplify the process without sacrificing performance, and find it works well for many manipulation tasks.

3.3 Hybrid Visual Rendering↩︎

Color images, as a key type of perception signal, often yield a large visual gap between simulation and the real world. To close such a gap, we render the foregrounds and backgrounds with hybrid real-to-sim rendering techniques. For the foreground objects, object renderings are applied from the mesh with ray tracing support. Background reconstruction contains most parts of the scenes and applies more realistic rendering using 3DGS. Z-buffer rendering is applied to mix objects according to ground-truth depth. Notably, hybrid visual rendering supports multi-view rendering according to the perspective views. The rendering time consumption of each component is shown in 4.

3.4 Real-World Alignment↩︎

Despite the prior alignments between rendering and meshes, another key step is to align with real-world scenarios, specifically, the positions of the background/foreground relative to the robots. For foreground objects, we define a range of possible placements in the simulation. Regarding the background, we leverage the ArUco marker on the table to align the 3DGS coordinates with the marker coordinates. Then the ICP  [28] post-processing is applied to optimize the coordinate alignment gap. Specifically, ICP optimizes the relative transformation between the partial point cloud obtained from the depth camera and the complete point cloud sampled from the reconstructed mesh.

3.5 Expert Data Collection↩︎

To verify the effectiveness of the real-to-sim-to-real system, following the designed task scenarios such as pick-and-place, we collect large-scale demonstrations using the script policy. During each rollout, we introduced domain randomization, including source/target object randomization, and robot arm’s base position randomization. Define an expert policy \(\pi_\text{priv}(a_t | o_{\text{priv}, t})\) based on privileged information \(o_{\text{priv}, t}\), such as the exact pose of target objects. We can then make use of \(\pi_\text{priv}\) to interact with the simulator and generate observation-action pairs \((o_t, a_t)\). Here, \(o_t\) represents observations that are accessible from the real world, including images and proprioception data. Specifically, we compute the 6D pose of the gripper at some key steps and use the motion planner based on the RRTConnect [29] algorithm to plan the path between the key steps in the joint space. During data generation, reject sampling is applied to filter out failed rollouts, improving the dataset quality and ensuring reliability. We can continue to generate such data with the amount according to our needs, obtaining the simulated datasets \(\mathcal{D}\).

4 Experiment↩︎

We mainly investigate the following three research questions based on the proposed Re³Sim system:

1. Can Re³Sim generate high-fidelity simulation data with small 3D sim-to-real gaps to benefit real-world manipulation problems?

2. How does large-scale simulation data help in more challenging real-world tasks?

3. Can Re³Sim rapidly construct simulated scenes and synthesize data at low cost?

4.1 Experimental Setups↩︎

Hardware platform. In this paper, we evaluate Re³Sim with a Franka Research 3 robot equipped with a parallel gripper. Two Realsense D435i depth cameras capture visual observations—one mounted on the end-effector and another positioned beside the robot for a third-person view.

Policy model and training details.

During our sim-to-real experiments, we adopt an ACT-structured policy [30] with DINOV2 [31]. The images in the dataset are compressed using JPEG format. During training, we apply visual augmentations, and for evaluation, the temporal ensemble [30] technique is used to smooth the final action for rollouts.

4.2 Zero-Shot Sim-to-Real Experiments↩︎

[ques:2] is the main question that motivates us to build Re³Sim, thus we design three table-top manipulation tasks to investigate the usage of Re³Sim:
1.Pick and drop a bottle into the basket. The robot must pick an orange bottle randomly placed within a \(25\text{cm} \times 35\text{cm}\) area on a table and place it into a basket.
2.Place a vegetable on the board. A cucumber model is placed on the table in an area of \(35\text{cm} \times 50\text{cm}\), and the cutting board’s position is varied within a \(35\text{cm} \times 80\text{cm}\) area. The robot must grasp the vegetable and accurately place it on the cutting board.
3.Stack blocks. The robot is tasked with stacking three black cubes on the table. Each cube is placed randomly within a \(30\text{cm} \times 10\text{cm}\) area separately, and the robot must grasp and stack them.

Tasks are rebuilt in simulation using Re³Sim, with 100 trajectories collected to train independent policies. As a baseline, we also adopt a rule-based policy that prioritizes picking objects closer to the robot base to reduce action variability.

Visual quality evaluation. An open-loop visual comparison is conducted by selecting a trajectory from the simulation dataset, replayed in the real world to capture ground-truth images with calibrated cameras. Visual similarity is evaluated from wrist and third-person perspectives, with qualitative results in Figure 2. These confirm that Re³Sim achieves high-quality visuals through accurate reconstruction and precise alignment.

We also compare with other reconstruction methods. Visual comparisons of reconstruction results from more methods are shown in Figure  3. PolyCam tends to blur details in certain areas, OpenMVS’s reconstructed mesh texture shows cracks, while 3DGS produces high-quality renderings. Table 1 reports PSNR and SSIM for the visual sim-to-real gap, comparing Re³Sim with OpenMVS and PolyCam [32]. Alignment error still causes minor pixel-level discrepancies, leading to a absolute lower PSNR and SSIM value in texture-rich scenes. 3DGS in Re³Sim surpasses PolyCam in both metrics and achieves higher SSIM than OpenMVS, despite comparable PSNR. OpenMVS’s PSNR is slightly higher than 3DGS’s, likely because the background and crack colors are similar, making PSNR less sensitive. However, the cracks cause increased variance within patches, leading to a lower SSIM compared to 3DGS. These results validate Re³Sim’s rendering design for minimizing visual gaps.

Sim-to-real results.The real-world success rates of policies trained on simulation data from Re³Sim are evaluated, compared with a real-to-sim baseline, RialTo [32], and policies trained on 50 real-world trajectories. Doubling the simulation data yields performance comparable to real-world data, indicating a minimal sim-to-real gap arising from differences in scene initialization and trajectory collection. For comparison, AnyGrasp [33], a state-of-the-art grasping method, is implemented with scripted primitives for the pick and drop a bottle into a basket task. AnyGrasp predicts only grasp poses, limiting its applicability to other tasks with non-fixed placement locations.

The numerical results listed in Table 2 demonstrate that the data generated by Re³Sim help the policy achieve zero-shot sim-to-real transfer, showing strong performance even compared with the existing real-to-sim and state-of-the-art grasping method. Furthermore, policies trained on our extensive synthetic dataset can achieve comparable or even slightly better performance than those only trained on real-world data, indicating the effectiveness and huge potential of high-fidelity simulation data.

4.3 Large-Scale Sim-to-Real Experiments↩︎

To push the limit of utilizing synthetic data for real-world manipulation problems by answering [ques:3], we choose a clear objects on the table task: the robot has to clean up the table by picking all items that are randomly placed on the table in a \(40\text{cm} \times 50\text{cm}\) area, and put them into the basket. This is a long-horizon task requiring the policy to sequentially pick both seen and unseen objects one by one. To provide a sufficient evaluation, we design four different setups:

• Seen: four seen objects (the bottle, cucumber, corn, and eggplant) included in the data for training.

• Unseen: four objects (the green pepper, banana, red bowl, and momordica charantia) that are unseen during training.

• Cluttered: all eight objects from seen and unseen.

• Darkness: the testing brightness is significantly lower than that of the simulation data.

All models are evaluated on four seen objects. The results are presented in Fig. 4 (a). Each setup was tested 10 times, with two additional automatic grasp attempts per object in case of failure.

Result analysis. Despite training on only five objects, the policy generalizes effectively to grasping novel objects of similar size, regardless of shape or color variations. We hypothesize that the robot exploits the static scene, detecting objects via background color contrasts. Additionally, objects with varying shapes often have similar grasping positions, enabling the robot to execute successful grasps. Larger datasets with multiple objects enhance the policy’s success rate, whereas smaller datasets limit its performance. Compared with RialTo-IL [32], policies trained on data from Re³Sim exhibit superior performance across all settings. Moreover, the policy shows robust performance under varying lighting conditions.

\begin{table}[tb]

\centering
\begin{tabular}{lccc}
\toprule
    Tasks & Time Cost (minutes) \\
\midrule 
Pick and drop a bottle into the basket  & 12.35 \\
Place a vegetable on the board & 13.78\\
Stack blocks & 6.45 \\
\bottomrule 
\end{tabular}

\vskip

-0.1in \end{table}

We further explored how the success rate in simulation and real-world settings changes with varying amounts of data. As shown in Fig. 4 (b), increasing the size of the synthetic dataset generated by Re³Sim leads to a significant improvement in imitation learning performance. When the amount of simulation data is limited, the real-world performance is almost negligible. Doubling the data size often results in a significant improvement in success rate until convergence at a high performance.

4.4 Sim-Real Consistency↩︎

We evaluated the consistency of policy performance between simulated and real-world environments using policy ACT. In this experiment, we evaluate policies trained at different stages (from 0 to 120000 training steps) and with 3 different seeds on the pick and drop a bottle into a basket task.

As illustrated in Fig. [fig:teaser] (d), the Pearson correlation coefficient of Re³Sim is 0.924, while that of RialTo is 0.710, indicating the correlation between performance in simulation and the real world of Re³Sim is better. Specifically, models that achieve higher success rates in simulation tend to exhibit similar success rates in real-world testing. When the success rate in simulation is low, it is often difficult for the model to perform well in real-world scenarios. This could be due to alignment errors, leading to a small sim-to-real gap, or the fact that the number of real-world test instances is much lower than in simulation, potentially introducing bias. This consistency suggests that policies trained with Re³Sim are well-suited for real-world deployment, reflecting a minimal sim-to-real gap.

4.5 System Efficiency↩︎

To validate [ques:4], we record and compute the time of Re³Sim to reconstruct both the background and objects, along with the time of generating demonstrations.

Human effort during reconstruction. In Re³Sim, background reconstruction only requires a set of images or a short video, plus a single depth-aligned image. While the process is straightforward, it still involves some human effort. We report average reconstruction times for both foreground and background, based on five users with basic knowledge of modern phones or cameras. The workflow is simple: (1) capture photos or video using a phone or camera, and (2) follow ARCode’s guided interface. Users typically need just 1–3 practice runs to become proficient. As shown in Table 3, video input saves time but may slightly reduce quality due to motion blur, while images yield better results with slightly more effort.

Data collection cost. Table ¿tbl:tab:data-collection-time? shows the time required to collect 100 simulation episodes for each task in Section 4.2, using 8 RTX 4090 GPUs. The time required for data collection is much lower than the time needed for teleoperation in real-world scenarios, especially considering Re³Sim only involves machine runtime, and the latter mostly requires the time of data collection experts. This demonstrates our capability to efficiently generate large-scale simulation data at minimal cost. The breakdown of the time consumption of each step is detailed in Tab. 4.

Time consumption breakdown. We measured the average time spent on each phase during the data collection of the pick and drop a bottle task in the simulator over 60 steps, as shown in Tab. 4. The reconstructed mesh, with numerous vertices and faces, demands more time for physics simulation. Rendering 480p images from two camera views using 3DGS adds 12.93 milliseconds, representing one-third of the total time per step. Re³Sim operates at approximately 24 frames per second (FPS) with two cameras, ensuring real-time performance and visual fidelity.

4.6 Impact of Mesh-Based vs. 3DGS Rendering Methods↩︎

We explore using 3DGS for foreground rendering as an alternative to ARCode-reconstructed meshes. Leveraging the GSDF [34] framework, we generate aligned mesh–3DGS pairs for integration into our hybrid system. As reported in Table 5, substituting ARCode with 3DGS offers no appreciable benefit in task success. In light of this, we continue to use ARCode-based meshes as the default in our main experiments.

5 Related Work↩︎

Sim-to-real. Sim-to-real transfer requires techniques to enable policies to successfully adapt from simulation to the real world. The most direct approach is improving simulators [16], [18], which reduces the sim-to-real gap. Other methods, such as domain randomization [35], [36] and system identification [37], [38], also aim to bridge this gap.

Real-to-sim-to-real. Many recent works leverage real-world data to enhance simulation models. Reconstruction methods integrated with grasping techniques, such as Evo-NeRF [39] and LERF-TOGO [40], enable the grasping of objects using only RGB images. Meanwhile, GaussianGrasper [41] and GraspSplats [42], use 3D Gaussian splatting [13] for fast rendering and explicit representation in robotics tasks. Additionally, methods like URDFormer [43] and Articulate Anything [44] use a single image to reconstruct the environment directly, allowing for collecting large amounts of data for imitation learning or reinforcement learning. These approaches enhance data by varying articulations and leveraging various articulations from simulation datasets to train models deployable in the real world.

With the help of 3DGS works like RoboStudio [45], SplatSim [46] and RoboGSim [47] use multi-view images or video for world reconstruction, improving multi-view rendering quality with minimal cost. These methods are especially effective in manipulation tasks involving multiple objects or occlusions. 1) RoboStudio focuses on reconstructing the URDF of a robot, offering a photorealistic rendering result and an accurate collision mesh. 2) SplatSim utilizes pre-obtained 3D models of objects and backgrounds to collect trajectories in the physics simulator and then re-render them with 3DGS to reduce the visual gap between simulated and real-world environments, but acquiring the 3D models is hard for many tasks. 3) RoboGSim compares rendering quality, validates high-quality rendering results for novel pose synthesis, and shows the potential for evaluating various manipulation algorithms. However, its sim-to-real validation remains limited. Different from them, Re³Sim reconstructs both geometric and visual aspects with small gaps, and validates robot policies trained on simulated data through extensive experiments in the real world.

6 Conclusion↩︎

We introduced Re³Sim, a novel Real-to-Sim-to-Real system that integrates Gaussian splatting with NVIDIA Isaac Sim’s PhysX engine, improving scene reconstruction and sim-to-real transfer for robotic manipulation tasks. Extensive experiments demonstrate its efficacy and scalability, showing that Re³Sim significantly reduces the sim-to-real gap and enables reliable real-world task performance. Models trained on large simulated datasets achieved competitive success rates compared with those trained on real-world data, highlighting the potential Re³Sim to generate diverse, high-quality data for pre-training large-scale robot models.

Limitations. Currently, Re³Sim does not support the reconstruction of deformable objects or liquids. The system also relies on manually defined physics parameters rather than automated system identification. Furthermore, rule-based policies for data collection become increasingly impractical as task complexity grows. Addressing these limitations is left to future work.

Acknowledgment↩︎

The SJTU team is supported by National Natural Science Foundation of China (62322603).

References↩︎

---

1. \(^{1}\)Shanghai Jiao Tong University, \(^{2}\)Shanghai AI Laboratory, \(^{3}\)The University of Hong Kong. * denotes equal contribution. \(\dagger\) denotes corresponding authors.↩︎

2. Isaac Sim [16] is used by default, but other ray tracing options are also feasible.↩︎