1 Introduction↩︎

Vision-language-action (VLA) models extend large-scale vision-language learning to robot control, offering a promising route toward general-purpose robot manipulation [1]–[4]. This integration empowers robotic agents to generalize a wide variety of dynamic tasks in diverse domains, such as household manipulation [5], autonomous driving [6], and industrial automation [7]. However, as VLAs increasingly transition from controlled laboratory environments to safety-critical, real-world deployments, their tightly coupled multimodal architectures expose them to novel security vulnerabilities [8], [9]. While recent research has highlighted the vulnerability of VLAs to advanced threats, such as physical sensor attacks [10], adversarial attacks [11], [12], and training-time backdoor injections [13], we find that the underlying vulnerability often stems from poor generalizability rather than security flaws requiring advanced attacker knowledge. To highlight this fundamental weakness, we systematically demonstrate that state-of-the-art VLA models fail on standard benchmarks when subjected to minor physical environmental variations, such as a slight shift in lighting conditions caused by a targeted physical spotlight.

Adversarial training is widely recognized as an effective countermeasure for improving the generalizability of VLA models in prior literature [10]. However, we find that naive adversarial training does not consistently yield genuine robustness; rather, it often conditions VLA models to entirely ignore the visual features that were perturbed during the training process. For example, when applying broad HSV-space data augmentation to the training data, the resulting VLA model behaves similarly to one trained exclusively on grayscale images. Consequently, the model loses the capacity to solve semantic tasks where fine-grained color information plays a critical role.

To resolve this defense dilemma, we introduce ChromaGuard, as depicted in Fig. 1, a chroma-preserving adversarial training methodology. We demonstrate that our approach not only provides robust defensive capabilities against severe lighting attacks but also maintains high performance in benign, color-dependent operations. Our findings highlight a critical lesson for embodied AI: standard handcrafted countermeasures can introduce biases that override fundamental semantic signals, particularly in VLAs, where specific elements of the visual observation dictate safety-critical actions. The major contributions of this study are as follows:

• We systematically expose a generalization vulnerability in state-of-the-art VLAs by proposing FLARE. This optimized physical spotlight attack framework degrades task success to 0.0% while inducing trajectory deviations of up to 115.5 cm in simulation.

• We identify a critical defensive trap in standard adversarial training methods, such as naive HSV transformation. Through diagnostic validation, we demonstrate that naive augmentations inadvertently condition the model to discard color cues. While the model retains up to 90.5% success on purely grayscale inputs, this conditioning translates to a steep performance drop to no more than 47.5% success in real-world semantic tasks requiring color differentiation.

• We propose ChromaGuard, a novel augmentation architecture designed to preserve texture and hue. We empirically prove that this method achieves state-of-the-art resilience against dynamic lighting attacks, reaching 70.0% success in color-invariant tasks. Furthermore, it strictly retains the perceptual capabilities necessary for complex physical manipulation by yielding success rates of 97.5% and 92.5% in benign and attacked color-dependent scenarios, respectively.

2 Background↩︎

2.1 Security and Robustness of VLA Models against Physical Attacks↩︎

The multimodal nature of VLAs exposes a wide variety of attack channels [8], [9] such as training-time backdoor attacks [13], [14], inference-time adversarial attacks [11], [12], [15]. Meanwhile, these advanced attacks naturally extend the prior attacks effective against general DNN models. The major contributions of these previous works are in overcoming the major challenges in triggering the vulnerabilities in their VLA domains. In this work, we highlight that VLA models are indeed vulnerable to these kinds of attacks, but the current state-of-the-art VLA models are poorly generalized and are just vulnerable to physical environmental variations, rather than particularly vulnerable to these advanced attacks. LIBERO-PRO [16] also reports that four major state-of-the-art VLA models easily collapse under small task and position perturbations on the LIBERO [17], one of the most popular VLA benchmarks. All models achieve \(\geq\)​90% success rates on the original LIBERO tasks but fall to nearly 0% under the perturbations, showing poor generalizability. Under these circumstances, any attacks may succeed. This motivates us to start by investigating the impact of lighting conditions, which can be perturbed in normal benign scenarios.

2.2 Defenses for VLA Models↩︎

To handle the vulnerability of VLA systems, recent research has explored both runtime guardrails [18], [19] and training-time regularization [10]. To directly address the generalizability of DNN models, adversarial training [10], [20] has been widely adopted not only in VLA [10] but also in many other application domains [21]–[23]. However, we find that the naive use of adversarial training may expose a critical backfire on the model’s generalizability. To be robust against lighting conditions changes, a naive approach is to apply color jittering for input images, also known as one of the data augmentation techniques [24]. As our study demonstrates, this inadvertently conditions the VLA to discard essential color and texture cues, effectively degrading its visual encoder into a purely shape-biased, grayscale processor. In complex manipulation tasks where color differentiation is critical to object interaction, this loss of semantic fidelity translates to tangible task failures. Consequently, there is an urgent need for a defensive framework that simultaneously achieves robustness against dynamic illumination while strictly preserving the integrity of critical chroma features.

3 Methodology↩︎

3.1 Threat Model↩︎

We follow the common threat model adopted in prior work [10], which assumes a strict black-box setup, i.e., the adversary has no access to the target model’s internal architecture, weights, gradient information, or training datasets. As depicted in Fig. 1, the adversary can deploy physical lighting fixtures within the robot’s workspace. When they place the light, they can control the spatial placement, illumination footprint, intensity, and the specific chromatic properties (hue, saturation, and lightness) of the emitted light. However, they cannot dynamically change the light once deployed; the parameters remain fixed across all evaluation episodes.

3.2 FLARE Framework: Optimized Physical Spotlight Attack↩︎

We propose the Framework for Lighting Adversarial Robustness Evaluation (FLARE), illustrated in Fig. 2, to generate worst-case lighting scenarios. We consider a black-box adversary that can place and adjust a spotlight in the workspace. The attack is physically plausible: it does not modify the robot, camera, or target objects, and operates exclusively through illumination. In simulation, the spotlight is rendered directly to optimize these parameters; on hardware, the same parameterization corresponds to a physical lamp whose pose and output can be adjusted within safe bounds.

We parameterize the spotlight by: \[\theta = [h_{\text{light}}, s_{\text{light}}, v_{\text{light}}, I, z_{\text{light}}, \alpha_{\text{cutoff}}],\] where \(h_{\text{light}}, s_{\text{light}}, v_{\text{light}}\) denote the hue, saturation, and value of the light source, \(I\) is the light intensity, \(z_{\text{light}}\) is the spatial height (z-coordinate), and \(\alpha_{\text{cutoff}}\) represents the spotlight’s cutoff angle.

Given a clean observation \(o_t\), the attacked observation is produced by a lighting operator \(o_t^{\text{adv}} = \mathcal{A}(o_t; \theta)\), which alters illumination while preserving physical scene geometry and object layout. The attack objective follows the intuition that dangerous failures are not only unsuccessful but also catastrophically distant from the intended trajectory. For episode \(i\), we score a spotlight parameter by combining trajectory deviation and task failure: \[L_i(\theta) = \frac{1}{S}\sum_{t=1}^{S}\lVert \mathbf{P}_t^{(i)} - \mathbf{P}_t^{\prime (i)} \rVert_2 + \lambda_1 \lVert \mathbf{P}_T^{(i)} - \mathbf{P}_T^{\prime (i)} \rVert_2 + \lambda_2 \mathbb{I}_f^{(i)}, \label{eq:objective}\tag{1}\] where \(S\) is the maximum number of steps, \(\mathbf{P}_t^{(i)}\) and \(\mathbf{P}_t^{\prime (i)}\) are the clean and attacked end-effector positions respectively, \(\mathbb{I}_f^{(i)} \in \{0,1\}\) indicates a binary task failure, and \(\lambda_1, \lambda_2\) are weighting hyperparameters. We aim to maximize the average objective score over \(N\) episodes: \(J(\theta) = \frac{1}{N}\sum_{i=1}^{N} L_i(\theta)\).

Because physical and simulated rollouts are expensive and internal model gradients are unavailable, we employ Bayesian Optimization to efficiently search the continuous spotlight parameter space for values that maximize \(J(\theta)\). Specifically, we utilize the Optuna [25] with a Tree-structured Parzen Estimator (TPE) sampler. While conventional Gaussian Process-based Bayesian Optimization (GP-BO) is widely used, its computational overhead increases significantly with the number of trials. In our context demanding hundreds of iterations of high-cost robot rollouts involving multi-modal VLA inference and physical simulation, this overhead presents a severe bottleneck. To avoid this problem, we select the TPE sampler because its computational cost is substantially lower than that of GP-BO, making it highly suitable for high-dimensional continuous search spaces under a limited time budget. Furthermore, evaluating all \(10\) tasks within a single LIBERO task suite at every iteration is computationally impractical in terms of total execution time. To balance computational efficiency with the discovery of truly universal adversarial parameters, we evaluate the tasks in alternating subsets. At each optimization iteration, the task suite is partitioned into two distinct subsets of \(5\) tasks, which are alternately evaluated to compute the trial score. This evaluation protocol prevents the optimization from overfitting to specific scenarios within a constrained computational budget.

3.3 ChromaGuard: Hue-Preserving Adversarial Training↩︎

Naive lighting augmentation typically perturbs all color channels indiscriminately, often through aggressive hue, saturation, or texture-distorting transforms. We first introduce a typical naive data augmentation (we call this baseline Naive-Aug) which is a standard combination of color and sharpness jittering. We denote the augmentation parameters as \(\phi = [h, s, v, c, \gamma]^\top\), representing perturbations on hue, saturation, brightness, contrast, and sharpness, respectively. These parameters are randomly sampled from uniform distributions. While this approach improves general visual robustness, unconstrained hue variations (\(h \neq 0\)) can alter intrinsic object colors, inadvertently confounding performance on strictly color-dependent tasks.

To isolate illumination effects, ChromaGuard restricts the augmentation parameter space. Specifically, it allows bounded perturbations to saturation, brightness, contrast, and sharpness, but explicitly fixes the hue perturbation to zero (\(h = 0\)). We use this constrained parameter space, denoted as \(\Phi_{\text{CG}}\), to simulate challenging lighting conditions via a standard adversarial objective. Given a task loss function \(\mathcal{L}\), an input image \(x\) with label \(y\), and a model \(f_{w}\) subject to transformations \(T\), we optimize the network against the worst-case illumination within this hue-constrained space, i.e., \(\min_{w} \mathbb{E}_{x, y} \left[ \max_{\phi \in \Phi_{\text{CG}}} \mathcal{L}(f_{w}(T(x; \phi)), y) \right]\).

4 Simulator Experiments: Exposing Vulnerability against Spotlight Attack↩︎

We evaluate the performance of our VLA model against spotlight attacks within a simulated environment. In this section, we demonstrate that (1) even state-of-the-art VLA models can be easily compromised by simple spotlight illumination, and (2) a high degree of robustness against this attack can be seemingly achieved through naive data augmentation, to motivate the real-world experiments (§5), which reveal that naive augmentation merely causes the model to ignore color information.

4.1 Experimental Setup↩︎

We conduct experiments within the LeRobot framework [26], utilizing the widely adopted LIBERO multi-task manipulation benchmark [17]. We train the SmolVLA architecture [3] for 200,000 steps. While the vision and language backbones utilize standard pre-trained weights, the action expert is trained from scratch. In this configuration, we train a Baseline model with no data augmentation and a Naive-Aug model with data augmentation that perturbs input images across multiple visual properties: specifically, \(\pm 180^\circ\) in hue, \([0.0, 4.0]\) in saturation, \([0.2, 3.0]\) in value, \([0.8, 1.2]\) in contrast, and \([0.5, 1.5]\) in sharpness. We then evaluate these models across three diverse task suites: LIBERO-Spatial, LIBERO-Object, and LIBERO-10, conducting 500 evaluation episodes for each suite. The spotlight attack is optimized in a continuous parameter space spanning spatial height (\(z_{\text{light}} \in [0.5, 3.0]\)), full HSV color bounds, illumination intensity, and cutoff angles, with shadows simulated by the MuJoCo physics engine [27]. For the random attack, parameters are sampled uniformly at random for each of the 500 evaluation episodes per suite.

Evaluation Metric. We quantify model robustness using two metrics. The Task Success Rate (SR) measures the proportion of successfully completed episodes, serving as the baseline for operational reliability. To capture the severity of adversarial deviations, we measure the Trajectory Error (TE) defined as the Euclidean distance between the nominal execution path (benign) and the perturbed path under attack. We report both the Average Trajectory Error (TE-Avg) across the entire episode and the Maximum Trajectory Error (TE-Max), which serves as a critical safety proxy for worst-case physical collisions. To ensure that the trajectory metrics reflect pure adversarial influence rather than inherent model incompetence, TE is strictly computed over episodes where the baseline model succeeds but fails under attack.

4.2 Results and Implications↩︎

Table ¿tbl:tab:attack95results? shows the results of the spotlight attack against the baseline and the naively data-augmented model. As shown, the spotlight attack significantly degrades the SR of the baseline model to entirely zero. Even random spot lighting can drop the SR by less than half of the standard laboratory lighting (Benign) scenarios, where the SmolVLA-Baseline model achieves a high 83.0%/89.4%/58.4% success rates for each task suite. Beyond mere task failure, the attack induces severe physical hazards: the maximum trajectory error (TE-Max) reaches \(\leq\)​115.5 cm, forcing the robotic arm far outside its intended operational envelope. We also conduct an ablation study of our object function design. Details are in the supplementary material.

Based on the results, the naive data augmentation looks seemingly sufficient to defend against the spotlight attack. In the next section, we will demonstrate that this numerical resilience is highly deceptive, i.e., the VLA effectively degrades into a shape-biased, grayscale processor. By indiscriminately perturbing color channels during training, the augmentation forces the model to bypass the illumination challenge by discarding essential color cues.

5 Real-World Experiments↩︎

To validate the practical implications of our simulated findings, we deploy the VLA models onto a physical robotic platform. In this section, we first provide diagnostic evidence of the semantic degradation caused by naive data augmentation. We then evaluate the models across both color-invariant and color-dependent physical manipulation tasks to demonstrate that our ChromaGuard can effectively address the limitation.

5.1 Experimental Setup↩︎

As illustrated in Fig. 3, our experimental setup employs a 6-DoF SO-101 Arm Pro manipulator [28] equipped with a dual-camera perception system, comprising a wrist-mounted camera and a static third-person camera. To synthesize the spotlight attack, an external, programmable RGB spotlight is positioned outside the camera’s field of view. We define two task configurations: a color-invariant task (e.g., picking a red ball and placing it into a black bin) and a highly color-dependent task (e.g., selecting between structurally identical red and blue balls based on language commands). Detailed task descriptions are in the supplementary material.

To perform these tasks, we train both the SmolVLA [3] and \(\pi_{0.5}\)  [29] architectures by fine-tuning the lerobot/smolvla_base and lerobot/pi05_base checkpoints, respectively. Both models are fine-tuned for 100,000 steps using our collected real-world demonstrations, comprising 50 episodes for the color-invariant task and 240 episodes for the color-dependent task. As our adversarial training approach, ChromaGuard fixes hue and perturbs \([0.0, 4.0]\) in saturation, \([0.2, 3.0]\) in value, \([0.8, 1.2]\) in contrast, and \([0.5, 1.5]\) in sharpness during the model training. We use the same parameters for the Baseline and Naive-Aug as those used in §4.1. For the spotlight attack generation, we first generate several attack candidates in the simulator with our FLARE optimization while limiting our available colors (red, blue, and green), and then we select the most effective attack in the real-world. We evaluate the trained models by conducting 20 trials for the color-invariant task. For the color-dependent task, we conduct a total of 40 trials, consisting of 20 trials for each language instruction.

5.2 Evaluation of Color Reliance via Grayscale Inference↩︎

To systematically verify whether the augmented model loses its reliance on color information, we first conduct a grayscale diagnostic evaluation. We convert the input images to grayscale during inference across three different LIBERO task suites. As shown in Table 1, the baseline model, which naturally learned color features during training, suffers a significant drop, its success rates falling to near zero across the suites. In contrast, the Naive-Aug model maintains almost identical performance on grayscale inputs as on standard RGB, e.g., 90.5% vs. 89.8% SR on LIBERO-Object.

This phenomenon exposes a critical flaw in naive data augmentation. Because tasks rarely require disambiguating identically shaped objects solely by color in standard benchmarks, the Naive-Aug model learns to discard color channels entirely as noisy variables. It bypasses the illumination challenge not by learning robust color representations, but by collapsing its perception into a purely shape- and position-biased state, effectively ignoring semantic color cues.

5.3 Evaluation of Color-Dependent and Color-Invariant Tasks↩︎

To assess the practical impact of augmentation-induced perceptual degradation, we evaluated the models on physical hardware across invariant and color-dependent tasks. Table 2 shows the success rates of the color-invariant and color-dependent tasks for each model. As shown, in the color-invariant scenario, our ChromaGuard and Naive-Aug model defended against the spotlight attack as expected based on the results in §4. The success rates of the SmolVLA and \(\pi_{0.5}\) improve from 0% to 70% and from 30% to 75% under attack, respectively.

However, in the color-dependent scenario, where resolving the semantic instruction relies entirely on visual color cues, the Naive-Aug model generalized poorly. Even under benign laboratory lighting, its success rate dropped to 47.5% compared to the Baseline’s 77.5%, as the robot frequently struggled to distinguish between the identical red and blue objects. This confirms that defense techniques relying on indiscriminate color perturbation can severely degrade the semantic grounding of color information required for complex real-world tasks.

ChromaGuard resolves this trade-off by decoupling illumination variations from chroma features during training. As shown in Table 2, ChromaGuard matches the robustness of Naive-Aug in the color-invariant setting (\(\geq\)​70% SR under attack) while demonstrating substantial improvements in the color-dependent setting. Specifically, it successfully preserves semantic fidelity: SmolVLA achieves a 97.5% SR under benign conditions and 92.5% under attack, while \(\pi_{0.5}\) yields 55.0% and 70.0% SR, respectively. We consider that the limited color discrimination of \(\pi_{0.5}\), even under benign conditions, stems from the pre-training of its foundation model rather than the augmentation strategy. As evidence, \(\pi_{0.5}\) frequently grasps objects of the incorrect color even in benign scenarios. This factor accounts for 66.7% of all failure cases, which is a rate six times higher than the corresponding failure rate (11.1%) observed in SmolVLA; furthermore, ChromaGuard improves the benign success rate to 55.0% and achieves 70.0% under attack, surpassing its performance in benign scenarios.

9pt

6 Limitations↩︎

Although our FLARE and ChromaGuard demonstrate substantial attack and mitigation capabilities on spotlight attacks, we do not intend to serve them as definitive attack and defense methods. Instead, they serve as analytical tools to expose the acute vulnerability of VLA models to subtle environmental perturbations. Our analysis reveals that naive data augmentation fails to resolve these vulnerabilities, merely discarding color features rather than learning robust representations. Furthermore, these tools do not yet explore temporal dynamics, such as time-varying lighting or closed-loop adaptive attacks. Our primary objective is to expose the critical robustness crisis facing current VLA models, rather than to engineer an exhaustive adversarial attack or an impenetrable defense.

7 Conclusion↩︎

We investigated the robustness of Vision-Language-Action (VLA) models under targeted physical illumination variations. Utilizing our proposed FLARE evaluation framework, we demonstrated that optimized spotlight perturbations can reliably degrade baseline VLA performance to a 0.0% success rate while inducing substantial trajectory deviations. Furthermore, we identified a practical limitation in standard color data augmentations: while they successfully improve robustness to lighting changes, they inadvertently bias the model to discard semantic color cues, leading to performance degradation in physical tasks that rely on color discrimination. To address this limitation, we introduced ChromaGuard, a hue- and texture-preserving augmentation strategy. Our physical evaluations on a 6-DoF manipulator indicate that ChromaGuard effectively balances robustness and perceptual fidelity, maintaining reliable success rates under targeted lighting attacks without compromising the visual acuity necessary for semantic manipulation. These findings emphasize the urgent necessity of achieving a certain level of generalizability of the current state-of-the-art VLA models before diving too deeply into their security.

References↩︎