1 Introduction↩︎

Autonomous driving systems (ADS) must be evaluated before deployment because failures can have safety-critical consequences [1]. Modern ADS rely heavily on machine learning and deep learning components for perception and decision-making [2], whose data-driven nature introduces failure modes that differ fundamentally from those of traditional software, motivating specialized testing techniques [3]. Real-world testing provides the strongest evidence of system performance under realistic operating conditions, but it is expensive, difficult to reproduce, time-consuming, and unsafe for many corner cases [4].

Dataset-based evaluation improves reproducibility, yet it replays fixed observations and therefore cannot capture the closed-loop interaction between the ADS and its environment [5]. As a result, failures emerging from the interaction between perception, planning, and control may remain undetected.

Simulation-based testing addresses this limitation by enabling the ADS to interact with a controllable and reproducible virtual environment. However, its effectiveness is fundamentally limited by the sim-to-real gap [4]. Differences in textures, illumination, scene appearance, and other modeling inaccuracies can alter the input distribution perceived by the ADS, leading to behaviors that deviate from those observed in the real world [6], [7]. Consequently, improving the realism and trustworthiness of simulation environments has become a central objective of recent research in ADS testing [8], [9].

Recent advances in neural rendering and scene reconstruction, including GANs, diffusion models, Neural Radiance Fields, and Gaussian Splatting, have shown promise in generating visually realistic scenes from real-world recordings [7], [10], [11]. However, existing approaches such as DriveRecon [12] mainly focus on rendering viewpoints or image sequences rather than executable, closed-loop simulations [13]. Conversely, prior work on scenario reconstruction from police reports and structured accident descriptions focuses on rebuilding executable scenarios [14], but does not address the perception-level sim-to-real gap.

To address this gap, we present Cam2Sim, the first framework, to the best of our knowledge, that transforms real-world 2D driving recordings and camera datasets into 3D playable CARLA scenarios for closed-loop ADS testing. Starting from front-facing camera images and poses, Cam2Sim reconstructs road geometry, ego trajectories, parked vehicles, and simulation assets inside CARLA [15]. The framework then augments the reconstructed environment with Gaussian Splatting [16] to render camera observations that resemble the original recording. The resulting scenarios support both trajectory replay and closed-loop ADS testing, where the ADS receives the GS-rendered camera observations during execution. Cam2Sim supports ROS-based extraction, parked-vehicle detection from camera or LiDAR data, OpenStreetMap-based map generation, CARLA scenario construction, local GS training, trajectory replay, and closed-loop execution with a camera-based ADS. It outputs CARLA-ready maps, trajectories, parked-vehicle placements, GS models, replay videos, steering logs, and execution traces for downstream analysis.

We evaluate Cam2Sim on a real-world urban-driving scenario using a camera-based end-to-end driving model. Our evaluation assesses reconstruction quality and closed-loop behavioral fidelity under simulator-rendered and GS-rendered observations. Results show that GS-rendered observations reduce the visual discrepancy with respect to standard simulator rendering while enabling behaviors that more closely resemble the real-world reference runs.

This tool paper makes the following contributions:

Tool. We present Cam2Sim, a framework that transforms real-world recordings into executable CARLA scenarios for closed-loop ADS testing through reconstruction and Gaussian-Splatting-based rendering in CARLA.

Evaluation in a real-world urban driving scenario. We demonstrate improvements in reconstruction quality and behavioral similarity compared to a simulation-only baseline. In our experiments, the GS-augmented simulation successfully completes all runs, while the simulation-only baseline consistently fails the scenario.

2 Cam2Sim Framework↩︎

1 shows the Cam2Sim workflow, which consists of five components: (1) data extraction, (2) dataset processing, (3) simulation-asset generation, (4) Gaussian Splatting preparation, and (5) driving simulation. Together, these components transform raw recordings or prepared camera datasets into CARLA-ready scenarios with map data, trajectories, parked-vehicle positions, GS-training inputs, and executable replay or closed-loop simulations.

2.1 Data Extraction↩︎

The data extraction component converts ROS-based recordings into the structured dataset format used by Cam2Sim. It can be skipped when front-facing camera images and corresponding camera poses are already available. The minimum required input is a sequence of camera images and poses, which define the camera trajectory and are sufficient for scenario reconstruction, GS training, and replay.

When ROS bags are used, Cam2Sim extracts camera frames and associates each frame with the ego pose at the corresponding timestamp. It can also extract optional LiDAR, odometry, steering angles, and model-prediction streams. These signals improve reconstruction and support validation: LiDAR enables more accurate parked-vehicle detection, while steering and model-prediction logs support comparison with the ADS. Sensor streams with different frequencies are synchronized with ego poses through timestamp-based interpolation. The output is a structured dataset containing camera frames, poses, and point clouds, and trajectories.

2.2 Dataset Processing↩︎

The dataset processing component extracts the information required for scenario reconstruction, map generation, GS training, and simulation. The camera trajectory is used as the reference path for reconstructing and replaying the scenario.

Cam2Sim detects parked vehicles and provides a graphical interface for manual refinement. From camera data, it applies monocular 3D object detection with FCOS3D [17], transforms detections into world coordinates using camera-to-ego calibration and timestamp-aligned ego poses, and clusters repeated detections across frames. Final positions are computed as confidence-weighted averages [18], while lane-side and orientation labels are obtained through majority voting. When LiDAR is available, Cam2Sim can instead detect parked vehicles with PointPillars [19]; users can then move, rotate, insert, or delete boxes to produce refined parked-vehicle files.

This component also prepares map and GS inputs. From the trajectory or a manually specified address, Cam2Sim retrieves OpenStreetMap data through the Overpass API [20]. For GS training, it selects frames at a configurable interval, removes the visible ego-vehicle hood from the selected frames, generates sky masks using SegFormer trained on Cityscapes [21], [22], and splits the route into overlapping chunks containing images, masks, and pose metadata.

2.3 Simulation Data Generation↩︎

The simulation-data generation component converts the processed scenario into CARLA-ready assets: the OpenDRIVE map, replay trajectory, initial hero-vehicle pose, and parked-vehicle spawn positions. Cam2Sim exports both center-position and rear-axle trajectory variants to account for the difference between recorded sensor poses and CARLA vehicle transforms.

Coordinate conversion maps dataset positions to CARLA coordinates through the OpenDRIVE map. When geographic coordinates are used, Cam2Sim converts WGS84 coordinates into the OpenDRIVE reference frame and adapts the axis convention for CARLA. The tool also provides utilities to load the generated map, inspect trajectory alignment and parked-vehicle placement, and prepare the final CARLA scenario.

2.4 Gaussian Splatting Preparation↩︎

Using the processed frames and reconstructed trajectories, the GS preparation component trains the models used to render camera observations that resemble the original recording. Cam2Sim trains local GS models to improve scalability and specialization to local scene appearance.

Cam2Sim runs COLMAP [23] with the selected images, calibrated camera model, and optional sky masks to estimate camera poses and a sparse 3D point cloud. It then trains GS models with Nerfstudio [24], using splatfacto or splatfacto-big; sky masks exclude sky regions so the model focuses on static scene content. After training, Cam2Sim aligns the Nerfstudio and dataset coordinate systems by matching reconstructed camera poses with the original pose annotations and estimating the corresponding alignment parameters. These models are later used to render GS images from CARLA camera poses.

Because the deployed ADS directly consumes the rendered camera stream, successful closed-loop execution also provides an operational assessment of the generated observations: severe rendering artifacts or missing scene information would directly affect steering behavior and trajectory stability.

2.5 Driving Simulation↩︎

The driving simulation component executes the reconstructed scenario in CARLA using the generated map, trajectories, parked vehicles, camera configuration, and, when enabled, the trained GS models and alignment files. Cam2Sim supports both trajectory replay and closed-loop execution.

In trajectory-replay mode, the hero vehicle follows the recorded trajectory frame by frame. This mode is used to inspect the reconstructed scenario and compare observations from the same viewpoints as the original recording. Cam2Sim can save CARLA-rendered images, GS-rendered images, and side-by-side outputs.

In closed-loop mode, the ADS controls the ego vehicle. Cam2Sim currently supports a camera-based DAVE-2 model [25], served by a separate inference process that predicts steering commands from RGB images. The ADS can receive either raw CARLA images, forming the simulation-only baseline, or GS-rendered images to evaluate whether GS-based rendering reduces the gap to the original driving scenario. The output includes replayed camera streams, trajectories, steering commands, and execution logs for evaluating reconstruction quality, visual fidelity, and behavior similarity.

3 Preliminary Validation↩︎

We evaluate whether Cam2Sim can transform a real-world driving recording into a reconstructed simulation that is geometrically consistent with the original scenario, visually closer to the original camera stream, and suitable for closed-loop ADS testing. The validation follows the structure of the pipeline. First, we assess whether the reconstruction and simulation-generation components correctly reproduce the road geometry, ego trajectory, and parked vehicles of the target scenario. Second, we evaluate whether Gaussian Splatting preserves visually relevant scene information required for closed-loop ADS execution.

To this aim, we conduct a real-world data-collection campaign using an ADS-equipped research vehicle on a public urban road and use the resulting recording as the target scenario for Cam2Sim. The evaluation compares three domains: (i) the real-world reference execution, (ii) a simulation-only baseline using the reconstructed CARLA scenario, and (iii) a GS-augmented simulation in which camera observations are rendered using Gaussian Splatting.

System Under Test. We use a camera-based end-to-end DAVE-2 model [25] that predicts continuous steering directly from RGB images captured by the front-facing camera. During execution, longitudinal control is handled independently by a PID controller that maintains a fixed speed of 20 km/h (2 (A)).

Autonomous Driving Platform. We use fortuna [26], a modified Volkswagen Passat Variant GTE research vehicle equipped with cameras, Velodyne LiDARs, and GNSS/INS localization. The front-facing camera is used both for scenario reconstruction and as input to the ADS, while the roof-mounted LiDAR supports parked-vehicle detection during real-world data collection (2 (B)).

Collection Scenario. Experiments are conducted in Munich on a 450 m two-way residential street with a 30 km/h speed limit. The scenario includes an unmarked roadway, sidewalks, three bends, and parked vehicles partially narrowing the driving lane, providing a realistic yet controlled urban evaluation environment (2 (C)).

3.1 Scenario Reconstruction Quality↩︎

We first evaluate whether Cam2Sim reconstructs the target scenario with sufficient geometric fidelity for simulation-based testing. We apply the reconstruction pipeline to the collected recording and generate a CARLA scenario containing the reconstructed road map, replay trajectory, and parked-vehicle placements. We then execute trajectory replay and compare the rendered semantic maps against the real-world semantic observations from the same viewpoints.

We assess reconstruction quality using Intersection over Union (IoU) over the road, car, and background classes. Across 3,144 replayed frames, Cam2Sim achieves a mean IoU (mIoU) of \(0.774\) (\(\sigma=0.071\)). The reconstructed road layout achieves the highest agreement with the real-world reference, with a mean IoU of \(0.847\) (\(\sigma=0.056\)), while background regions achieve \(0.889\) (\(\sigma=0.037\)). Vehicle reconstruction is more challenging due to localization inaccuracies and partial occlusions, yet the car class still achieves a mean IoU of \(0.577\) (\(\sigma=0.206\)). An example semantic comparison is shown in 2 (D). Overall, these results suggest that Cam2Sim reconstructs urban-driving scenarios with sufficient geometric fidelity to support replay and closed-loop ADS testing.

3.2 Image Generation Quality↩︎

We next evaluate whether GS reduces the visual gap between the reconstructed CARLA scenario and the real recording. Starting from the reconstructed scenario, we train the GS model using Step 4 on the prepared camera frames, poses, sky masks, and local training splits. We then execute Step 5 again in trajectory-replay mode and render the camera stream with GS from the replayed viewpoints.

We compare three image streams: the real-world camera frames, the simulation-only CARLA frames, and the GS-rendered frames. The goal is to verify whether GS produces observations closer to the real camera stream than the raw simulator output. We quantify image quality using three full-reference metrics: , , and . Results are shown in .

3.3 Behavior Fidelity↩︎

Finally, we evaluate whether the GS-augmented simulation improves system-level testing realism. We execute the driving-simulation component in closed-loop mode with the deployed ADS in three domains: the real-world setting, a simulation-only baseline, and a GS-augmented simulation. In the simulation-only baseline, the ADS receives raw CARLA camera images; in the GS-augmented setting, it receives camera observations rendered by the GS model.

To account for variability and the non-determinism of the end-to-end driving model, we execute each domain three times. We compare the generated trajectories against the real-world reference executions using failure and completion rates, as well as trajectory-similarity and steering-smoothness metrics, including Fréchet distance, corridor violations, lateral excess, and steering jitter.

Results are shown in 2 (E) and Table 1. The simulation-only baseline consistently fails to complete the scenario, while the GS-augmented simulation successfully completes all runs, matching the real-world executions. Moreover, the GS-based simulation produces trajectories substantially closer to the real-world reference and produces stable steering behavior across all runs. Overall, these preliminary results suggest that GS-rendered observations improve the robustness and behavioral fidelity of closed-loop ADS testing compared to standard simulator rendering.

4 Conclusions↩︎

We presented Cam2Sim, a tool that transforms real-world recordings or prepared camera datasets into playable CARLA scenarios augmented with Gaussian Splatting. Starting from camera images and poses, the tool reconstructs scenario geometry, prepares simulation-ready assets, trains local GS models, and supports both trajectory replay and ADS closed-loop execution.

As future work, we plan to extend Cam2Sim with dynamic-object reconstruction, support for additional neural-rendering backends and sensor modalities, and larger-scale evaluations across diverse driving environments and ADS architectures.

5 Data Availability Statement↩︎

Cam2Sim is released as a GitHub repository under the Apache-2.0 license [27]. The artifact includes the tool implementation, the ROS bag used in our validation, the camera-based ADS, example outputs, and scripts for reproducing the GS-rendering and closed-loop evaluations. The repository also documents the required dependencies, including CARLA, ROS, COLMAP, Nerfstudio, and a CUDA-capable GPU. Our validation setup uses an Intel Core Ultra 9 machine with 32 GB RAM and an NVIDIA RTX 4090 GPU.

The review dataset is provided for reproducibility purposes only; an anonymized version with blurred faces and license plates will be publicly released under a CC BY-NC 4.0 license. A public DOI is not released at submission time to avoid premature dissemination; the artifact is available as a GitHub repository for review and will be archived with a DOI upon acceptance.

References↩︎