1 Introduction↩︎

Ensuring the adversarial robustness of vision systems is important, as computer vision is applied in safety-critical domains like autonomous vehicles. For example, adversarial perturbations when put on stop signs can deceive object detection systems, potentially causing them to misclassify or fail to recognize traffic signs altogether [1]. Adversarial perturbations are typically generated through an iterative optimization process without ever inserting the perturbations into the simulator itself. This begins with an initial pattern, commonly referred to as a “patch,” alongside a collection of environmental images in which the patch is to be evaluated. This patch is optimized for the desired adversarial task. The optimization process does not account for complex interactions between the patch and the environment, such as lighting variations, or material properties. As a result, when the optimized patch is eventually placed into the simulation environment, its visual appearance often changes dramatically, due to these unmodeled physical factors, leading to reduced adversarial effectiveness, as shown in 2. This observation aligns with findings from prior work, which reports similar degradation in performance under such conditions [2]. This issue is particularly concerning, given that simulations are intended as a proxy for real- world situations [1]. Consequently, the failure of the attack to remain effective under realistic lighting and material conditions raises significant concerns about its transferability from digital to physical environments [3], [4].

Why do researchers continue to use this sub-optimal pipeline? A primary reason is that the photorealistic simulation platforms such as Unreal Engine [5] that allow for complex human and object movements, and realistic physics modules for collision and light scattering [1], are all non-differentiable (1). Researchers have tried inserting patches into the simulation itself and estimating gradients using PyTorch transformations [2]. However, such approaches restrict the patch to shapes and transformations that can be approximated in two dimensions, limiting their applicability in creating complex 3D adversarial objects [6]. While differentiable renderers are often used in the creation of adversarial textures they lack other attributes to act as a satisfactory substitute, such as simulating movement of objects and an easy-to-use interface [7]. This fundamental gap explains why recent adversarial methods use this optimization pipeline or other similar approximation techniques; direct access to gradients within realistic simulation environments is unavailable, preventing end-to-end optimization of textures [2]. As a result, there is not a unified approach that enables differentiable optimization within 3D simulation, a critical gap that creates a barrier to research progress [1].

We propose UNDREAM: Unified Natural Differentiable Rendering for End-to-end Adversarial Modeling, a novel practical framework that bridges photorealistic 3D simulation and differentiable optimization, offering the best of both worlds. UNDREAM converts 3D objects and textures from Unreal Engine into a fully differentiable system, enabling end-to-end optimization of adversarial patterns on arbitrarily-shaped 3D objects within photorealistic environments. We make the following key contributions:

1. UNDREAM: First Software Framework Bridging Differentiable Rendering and Photorealistic Simulation for End-to-End Adversarial Attacks. Unlike existing approaches that optimize textures in isolation and later superimpose them (2), UNDREAM optimizes textures as they appear natively in simulation, preserving lighting, perspective, and physical material interactions. (3.2)

2. Automatic, Precise 3D Transformation Enabling Faithful Threat Modeling. By embedding adversarial textures directly into the simulation environment, UNDREAMeliminates the need for calculating the bounds of the adversarial object, leading to better alignment with realistic threat models and provides a more faithful evaluation of adversarial attacks. (3.1.2)

3. Open-Source Implementation Enabling New Research Directions. UNDREAM offers a flexible and scalable interface that allows researchers to optimize adversarial textures on 3D objects of arbitrary shape by changing only one line of code. This flexibility, combined with the realism of high-fidelity simulation, opens the door to novel research directions in physical adversarial attacks (4). UNDREAM’s code is available at https://github.com/poloclub/undream and is ready for public release.

2 Related Work↩︎

2.1 Simulation in Adversarial Machine Learning↩︎

High-fidelity simulators have become a cornerstone for developing and testing autonomous systems [3], [14], [15] as they bridge adversarial optimization and physical-world evaluation by offering photorealistic graphics, dynamic environments, and configurable sensor suites. Simulators such as CARLA [8] and Airsim [9], both of which are built on Unreal Engine [5], excel at modeling real-world conditions including variable weather, lighting, and camera perspectives making them ideal for generating diverse and scalable datasets [11]–[13], [16]–[18] that offer enhanced controllability and scalability compared to real-world benchmarks [19]–[21], enabling more systematic evaluation of perception models. However, all of them are non-differentiable. 1 shows a detailed comparison of UNDREAM and existing frameworks, highlighting it as the only framework that simultaneously allows environment customization and differentiation of textures on the 3D objects in realistic simulation. While platforms facilitate dataset generation they provide no support for differentiable textures or gradient-based editing in the simulation. Thus, they lack the ability to seamlessly fit in the adversarial optimization pipeline. As a result, adversarial studies using these frameworks or datasets typically resort to post-hoc superimposition of adversarial textures onto rendered images. This approach breaks the rendering consistency, causing adversarial objects to appear disharmonious with the scene by failing to account for lighting, shadows, and occlusion [2]. This limitation not only reduces visual realism, but also severely undermines the transferability and effectiveness of such attacks in real-world scenarios [3].

2.2 Differentiable Rendering↩︎

Differentiable rendering bridges the gap between traditional computer graphics and gradient-based optimization by enabling gradients to propagate from an image-based loss function back to underlying scene parameters, such as geometry, lighting, and textures [22]–[24]. This capability has proven especially valuable in adversarial machine learning, as it allows adversarial textures to be optimized directly on 3D objects under varying viewpoints and illumination conditions [25]–[28]. However, a significant limitation persists: differentiable renderers like Mitsuba and Pytorch3D do not support simulation features or complex weather and often require specialized knowledge and manual scene configuration. Thus, adversarial researchers prefer using non-differentiable platforms that offer simulation capabilities like Unreal Engine or CARLA instead [7].

These limitations have created a long-standing divide: researchers must choose either high-fidelity, non-differentiable simulation that requires gradient approximation techniques [2], or differentiable renderers with restricted scene dynamics that constrain the creation of photorealistic simulations and animations and demand specialized expertise and manual scene configuration [22]–[24]. Our work closes this gap by offering a framework that bridges differentiable optimization to high-fidelity 3D environments.

3 Method↩︎

One of the core contributions of UNDREAM is its ability to bridge a longstanding gap between high-fidelity simulation and differentiable rendering — two critical components required for end-to-end optimization in simulation-based environments. Simulation platforms commonly used in adversarial machine learning research provide highly realistic visual outputs and complex scene dynamics, but lack native support for gradient-based optimization. In contrast, differentiable rendering frameworks enable precise, gradient-based optimization but are limited in their ability to generate complex, photorealistic environments with dynamic elements.

Selection of simulation platform. While there is a range of platforms to choose from when it comes to picking the simulations, most of the widely used simulation platforms are built on Unreal Engine [8], [9], [18] making it a natural choice. We implement UNDREAM using Unreal Engine 5 — instead of the Unreal Engine 4 used in the previously mentioned platforms — as it introduces new light scattering techniques such as Nanite and Lumen that improve photorealism in rendering [29].

Selection of differentiable renderer. Mitsuba is a popular differentiable renderer used in adversarial attacks [7]. We use Mitsuba in UNDREAM implementation as it a has Python API which allows for integration with existing adversarial ML implementations. Mitsuba, due to its differentiable ray tracing method, has increased photorealism compared to alternatives such as PyTorch3D [30].

UNDREAM unifies the strengths of these two paradigms by combining the visual realism and scene richness of Unreal Engine with the differentiability and optimization capabilities of Mitsuba. This enables gradient-based adversarial optimization directly within photorealistic simulations. Moreover, UNDREAM is implemented entirely in Python, ensuring seamless compatibility with widely adopted adversarial machine learning frameworks, particularly those built on PyTorch.

3.1 Setup↩︎

3.1.1 Forward Rendering: Sequences in Unreal Engine↩︎

A LevelSequence is Unreal Engine’s native tool for constructing cinematics and is used to capture dynamic scenes with precise temporal and spatial coherence. Users may either import an existing LevelSequence into their Unreal Engine project or create a new sequence from scratch. When creating a new sequence, converting key scene elements such as cameras and dynamic objects into spawnable assets ensures accurate tracking of object motion and reliable reproduction of scenes across frames.

3.1.2 Inverse Rendering: Scene Transformation.↩︎

Transforming scenes from Unreal Engine to Mitsuba involves reconciling differences across several axes. UNDREAM converts left-handed Z-up coordinate system, with Y-facing cameras, Euler rotations, and meter units from Unreal Engine into right-handed Y-up coordinate system, with Z-facing cameras, axis-based rotations, and centimeter units in Mitsuba, preserving consistent object–camera geometry across renderings. This transformation process is fully automated and can be executed via the “initialize.py” script provided in the project repository. The transformation pipeline involves the following key steps:

Extract camera and object poses. Retrieve information from the LevelSequence including the 3D positions and orientations of adversarial object, camera, and any other object we would like to track across the frames.

Apply scale and coordinate conversion. Transform retrieved location and rotation from the LevelSequence from the Unreal Engine coordinate system to the Mitsuba coordinate system and convert the relevant units. This includes adjustments to axes, handedness, and rotation conventions.

Generate Mitsuba XML scenes. For each frame in the sequence, using the transformed object positions, orientations, and camera parameters create XML files that replicate the relationship between the camera and adversarial object such that the object appears exactly as it does in the simulation.

Add adversarial texture to XML. Restore the desired adversarial texture in the XML files in preparation for gradient-based optimization of adversarial texture.

Another intermediate step is replacing the object’s texture with a plain white texture in the XML scenes, and save the resulting rendering. This can help us verify XML rendering. Further details about the transformation can be found in [label:32appendix-unreal-mitsuba32transfer]

3.2 Run adversarial attack↩︎

While existing attack libraries support a range of gradient-based attacks, their implementations present several limitations in the context of differentiable simulation [31]: (1) they do not support integration with differentiable renderers, (2) they lack mechanisms for inserting updated textures into a simulation at each optimization step, and (3) they do not allow dynamically updating input images based on the output of new renderings after each iteration. UNDREAM enables smooth adaptation of attacks as shown in 3 with the Algorithm 4.

The attack implementation in UNDREAM builds upon existing adversarial attacks available in widely used attack libraries such as ART [31], with modifications to enable integration with the differentiable rendering process, as detailed in the bridging step of Algorithm 4. Our pipeline iteratively renders scenes, computes gradients for differentiable rendering, and updates textures within simulation. Integrating new attacks into the UNDREAM pipeline requires modifying only a single line of code — the line responsible for updating the adversarial texture using the computed gradient in the differentiable renderer as shown in Line [algo-step:diff32renderer]. By altering this line, users can easily switch between different attack methods supported by existing attack libraries while retaining full compatibility with the differentiable simulation workflow.

4 Experiments↩︎

We evaluate the capabilities of UNDREAM along two primary axes: the level of control afforded within the simulation environment, and the flexibility and effectiveness of adversarial attacks implemented through the framework. These two dimensions highlight UNDREAM’s utility both as a research tool for simulation-based adversarial machine learning. An overview of the available controls spanning both simulation and optimization components is presented in ¿tbl:tab:tool?. This includes parameters such as scenes, objects, lighting, models, tasks and algorithms.

4.1 Object↩︎

One of the main contributions of UNDREAM is it enables researchers to optimize adversarial textures for any arbitrary 3D object in high-fidelity simulations. This will open many new research directions as the current optimization pipeline uses 2D approximation techniques when dealing with photorealistic simulations [2], thus limiting the adversarial objects possible to only 2D rectangular shapes. Objects in the simulation utilize a technique known as UV mapping to project a 2D texture onto their 3D surface. This mapping allows us to determine how each pixel in the texture corresponds to a specific point on the object’s surface, facilitating precise gradient-based updates during optimization. An example of the UV map editor in Unreal Engine is shown in  5 (a). Upon completion of the optimization process, we observe that only the regions of the texture directly visible to the camera are significantly modified, as illustrated in  5 (b).  6 presents several examples of objects used in our experiments, along with their optimized adversarial textures.

4.2 Environment and Lighting↩︎

Environmental and lighting conditions play a critical role in adversarial machine learning, particularly in the context of physically realizable attacks. Variations in lighting, shadows, time of day, or weather can significantly influence the effectiveness and transferability of adversarial attacks [1]. However, existing datasets and simulation pipelines typically offer limited support for systematically varying environmental parameters. Even when present, they are often not available as controllable inputs for experimentation, but preset variables [6]. In contrast, UNDREAM provides fine-grained control over lighting and environmental conditions within the simulation, enabling researchers to evaluate the robustness of adversarial attacks across diverse settings.  7 demonstrates examples of the same scene rendered under different lighting and weather conditions using UNDREAM.

4.3 Attacks and Tasks↩︎

UNDREAM enables seamless experimentation with a wide range of adversarial attacks and tasks within a high-fidelity simulation environment. As described in Algorithm 4, the structure of the attack implementation in UNDREAM mirrors that of popular adversarial attack libraries. This ensures that attacks can be easily adapted and integrated into the simulation with minimal code modifications. As examples to demonstrate successful attack execution, 2 shows the results for the PGD and Auto-PGD attacks for the tasks of classification and detection on detr-resnet-50 for the sequences provided in the code repository; reducing the classification accuracy from 100% to 9.5%, and detection mAP from 100% to 6.9% a significant drop commonly seen in unbounded attacks [2]. Our code repository includes implementations for attacking a variety of models (e.g., detr-resnet-50, YOLOv8, YOLOv11).

5 Conclusions and Future Work↩︎

We presented UNDREAM, the first software framework bridging differentiable rendering and photorealistic simulation for end-to-end adversarial attacks. UNDREAM offers automatic, precise 3D transformation enabling faithful threat modeling and surpasses existing simulation tools, uniquely standing as the only framework that offers differentiable rendering while simultaneously supporting customizable asset editing in realistic environments. Its open-source implementation enables new research directions. UNDREAM currently supports a wide variety of scenes, sequences, camera angles, object trajectories, 3D models, and lighting configurations. With UNDREAM’s open-source implementation, researchers and developers may easily extend UNDREAM to support more simulation assets, attacks (3.2), and even defenses, and test them all under realistic, high-fidelity conditions. We are excited by the possibilities enabled by UNDREAM in bridging the gap between photorealistic evaluations and physical-world deployment, further strengthening research on adversarial robustness.

6 Transformation details↩︎

Transformation of a scene from Unreal Engine to Mitsuba is the first fundamental step in bridging simulation with a differentiable renderer. The difference between these platforms are detailed in 3. While scale conversion is relatively easy, the steps to tackle coordinate and rotation conversion are expanded on here:

6.0.0.1 World coordinate:

To convert Left handed Z up to a Right handed Y up system, we must swap values on the Y and Z axes. Keep in mind this also affects rotation, as the rotation for Mitsuba is defined by coordinate axes.

6.0.0.2 Rotation:

Unless the pitch (rotation around Y) in Unreal Engine is a \(\pm\) 90 degrees, we can simply rotate the camera and object at the origin before translation. However, if the Y rotation is \(\pm\) 90 degrees, the object is subject to Gimbal Lock, a phenomenon in which the object loses one of its degrees of freedom, thus rotations around Z axis, may not actually apply.

10pt

References↩︎