1 Introduction↩︎

Autonomous robots deployed in open-world environments face embodiment-specific adversities that cannot be fully anticipated beforehand. These embodiment-specific adversities have effects that are relative to each robot. A wet floor, for instance, poses a danger to a wheeled robot but not to a micro-aerial vehicle. However, between two given wheeled robots, the adversity may manifest differently due to differences in their tractive systems. Currently, mobile robots use different frameworks for localization and odometry estimation; some use simple wheel odometry with IMU based fusion, while others fuse stereo-inertial visual odometry or even other sensors. A mobile robot can use any of the multitude of available SOTA frameworks. However, each of them fails under different edge cases: a Visual Odometry (VO) pipeline might suffer near-overexposure on its cameras or near textureless surfaces as we see with [1] [2], and a LiDAR Inertial Odometry (LIO) system can fail near reflective surfaces [3] [4]. Inside any mobile robot, a series of such complex algorithms accomplishing tasks ranging from perception, localization, mapping and control, operate in tandem, and each of these has different modes of adversity that are undocumented or highly unintuitive. As the world moves towards more end-to-end learning based systems for each of these tasks, it’s imperative to also consider the edge cases where such trained models could fail. While large vision-language models (VLMs) offer a shortcut [[5]][6][[7]][8] by enumerating hazards from visual scenes, they lack awareness of embodiment-specific interactions, yielding over-conservative planners that may detour around objects posing no real threat to the specific robot. This raises the key question: how can an embodied agent autonomously discover, characterize, and adapt to adversities through its own operational experience?

We present Don’t Fool Me Twice (DFM2), a framework that builds a library of semantically associated adversities through operational experience. We define an adversity as any disturbance that produces a measurable deviation in a robot’s operational signals from nominal behavior. A key insight of DFM2 is that interaction-driven disturbances often exhibit structured spatial behavior tied to semantic geometry. By associating anomalous operational deviations with semantic features, DFM2 constructs a retrievable library of adversity experiences grounded in operational interaction. Each adversity is further characterized through semantic voxel-centric disturbance modeling, enabling the robot to adapt when similar hazards reappear. Our main contributions are:

• Experience-Driven Semantic Association: We discover and characterize long-tail hazards by attributing anomalous operational signal deviations to dense visual semantics, building a personalized “danger library” through post-hoc reasoning.

• Decoupled Few-Shot Interaction Modeling: We constrain spatial geometry via semantic voxel priors and structured spatial disturbance models, reducing adaptation to a constrained optimization requiring only sparse interaction data.

• Uncertainty-Aware Predictive Adaptation: We estimate epistemic uncertainty via Bayesian Linear Regression over a fixed shape template, enabling proactive, conservative planning near novel hazards.

2 Related Work↩︎

2.1 Semantic Hazard Anticipation with Vision–Language Models↩︎

Recent research has increasingly leveraged Vision–Language Models (VLMs) to provide semantic priors for proactive hazard assessment in unstructured environments. For example, Language as Cost (LaC) [7] utilizes zero-shot VLM prompting to interpret visual scenes and map textual hazard cues into spatial costmaps via Gaussian cost propagation. Similarly, V-RoAst [8] introduces a VLM-based visual question-answering (VQA) framework to classify infrastructure safety attributes at regular road segments. Chakraborty et al. [9] further addressed the perceptual limitations of VLMs in raw image recognition by proposing a “scene-graph–guided alignment" that structures environmental cues into relational object graphs. While these efforts demonstrate that foundation models encode rich hazard concepts, they predominantly rely on a proactive”query at regular intervals" strategy. This approach typically casts a wide net by cataloging all potential adversities based on internet-scale common-sense priors, which introduces significant computational latency and over-conservatism often forcing detours around objects that look dangerous but pose no actual threat to a specific robot embodiment.

2.2 Few-Shot Interaction Modeling↩︎

Modeling robot–environment interactions has traditionally been studied through online system identification [10], which assumes a fixed parametric structure and sufficient training points, assumptions that often break down during transient events. To relax these constraints, kernel-based regression and Gaussian Processes (GPs) have been widely used to model unknown dynamics and disturbance fields [11], [12]. A common failure mode across kernel and GP methods in low-data regimes is joint hyperparameter optimization: when kernel length-scales are optimized alongside amplitude from N \(<\) 50 observations, the marginal likelihood landscape is non-convex and poorly conditioned, often yielding degenerate solutions [[13]][14]. Related approaches learn residual dynamics or disturbance fields on top of nominal models [15] or estimate spatial fields using basis functions or grids [16], [17], but typically treat residuals as unstructured functions or require dense exploration. Meta-learning and fast adaptation methods [18], [19] mitigate data scarcity via task-level pre-training, yet offer limited robustness to genuinely novel, long-tail hazards.

2.3 Continual Learning and Runtime Monitoring↩︎

Modern frameworks have moved toward hierarchical monitoring loops and retrieval-augmented memory to prevent catastrophic forgetting. Recent developments have solidified the utility of slow-fast loops. The Monitor-Analyse-Plan-Execute-Knowledge (MAPE-K) architecture has evolved into variants like MAPLE-K. MAPLE-K enhances this structure by embedding a continuous learning (L) step, which integrates semantic grounding in our case. Our framework, DFM2, synthesizes strengths from several contemporary systems: AESOP[6]: Provides high-rate anomaly detection via embedding similarity but lacks semantic causality. FORTRESS[5]: Uses VLMs to identify safe fallback goals but suffers from over-conservativeness due to binary hazard labels. DRAE[20]: Utilizes a non-parametric Bayesian memory to store specialized “experts" without corruption.

3 Preliminaries↩︎

3.1 System Definitions↩︎

We consider a mobile robot operating in a 3D environment with state \(\mathbf{x} \in \mathbb{R}^3\) representing its Cartesian position. The robot is tasked with following a reference trajectory \(\tau_{\text{ref}}\) generated by a high-level planner. We assume the robot is equipped with a reliable low-level controller (e.g., a PID or MPC-based position controller) that maps control inputs \(\mathbf{u} \in \mathcal{U}\) to velocities, maintaining a bounded tracking error in nominal, disturbance-free conditions. We define an adversity as any disturbance that degrades a robot’s ability to execute its intended task, manifesting as measurable deviations in operational signals such as, but not limited to, trajectory tracking error or state estimation reliability. To capture these varied failure modes across heterogeneous robotic platforms and sensory configurations, we introduce a generalized, time-varying operational monitoring signal \(\nu(t)\). In this work, the signal \(\nu(t)\) is dynamically mapped based on the active embodiment and context: \[\nu(t) = \begin{cases} \lVert \mathbf{x}_{\text{obs}}(t) - \mathbf{x}_{\text{ref}}(t) \rVert, & \text{(Trajectory Tracking Error)} \\ \text{Tr}\left(\mathbf{\Sigma}_{\text{pose}}(t)\right), & \text{(State Estimation Uncertainty)} \end{cases}\] where \(\mathbf{x}_{\text{ref}}(t)\) and \(\mathbf{x}_{\text{obs}}(t)\) represent the reference and executed coordinates respectively, and \(\text{Tr}\left(\mathbf{\Sigma}_{\text{pose}}(t)\right)\) denotes the trace of the state estimator’s pose covariance matrix. For most of the work, we will use the definition of tracking error for defining an adversity.

An adversity event is then parameterized via a binary operational indicator function \(a(t) \in \{0, 1\}\) defined as: \[a(t) = \mathbb{I}\left(\nu(t) > \theta\right)\] where \(\mathbb{I}(\cdot)\) is the indicator function and \(\theta\) is a context-specific detection threshold. To precisely calibrate \(\theta\) and characterize baseline sensor and actuator noise floor variations, the robot executes a prior calibration run path within a nominal, anomaly-free space.

3.2 Problem Statement↩︎

Given an active reference trajectory \(\tau_{\text{ref}}\) and a continuous stream of online operational measurements \(\nu(t)\), the objective of this work is to build an experience-driven reasoning architecture that can:

1. Detect anomalous operational adversities in real-time when the indicator function turns active (\(a(t) = 1\)).

2. Infer the latent semantic cause \(c \in \mathcal{C}\) of the localized disturbance post-hoc by leveraging a Vision-Language Model (VLM).

3. Learn a highly sample-efficient predictive model \(f: \mathbb{R}^3 \to \mathbb{R}\) of the localized disturbance field magnitude alongside its epistemic uncertainty.

4 Methodology↩︎

After calibration, the robot follows its planned path \(\tau_{\text{ref}}\) while continuously monitoring for any perceived disturbances. The architecture is organized into two complementary pathways: a fast loop and a slow loop.

4.1 3D Semantic Search for Adversity↩︎

At the core of the fast loop is a high-frequency perception module that detects risk-relevant objects using the dynamic danger library \(\Omega_t\), which stores embeddings of previously observed adversity causes. Detection is performed via embedding similarity \(\texttt{sim}(\cdot,\cdot)\) search between encoded image features and the danger library, offering spatial granularity and richer generalizability over text-based matching.

We build on the NARadio encoder from RayFronts[21] with openVDB [22] based mapping backbone. NARadio leverages Radio vision foundation model encoder[23] which distills features from diverse vision encoders (SAM for segmentation, DINOv2 for context, and SigLIP for language alignment). RADIO’s vision embeddings are projected into the SigLIP[24] language space through adapters trained during distillation. A simple patch level projection through the adapter yields dense language aligned visual features Incoming RGB images are thus transformed by \(\texttt{Embed}_{vision}\) into spatially consistent feature maps at the pixel level in the SigLIP space (\(\texttt{Embed}_{lang}\)). By performing \(\texttt{sim}(\cdot,\cdot)\), we extract hotspot masks through simple thresholding. These pixels are projected, voxelized and integrated into the underlying openVDB generated voxel map. The resulting masks are an approximation of the segmentation mask of the object, since we work with a resized similarity map. Spatio-temporal consistency filtering across frames is further done to ensure sharper 3D segmentation.

4.2 Anomaly Detection and Reasoning↩︎

The system continuously evaluates trajectory tracking error \(\Delta \tau = |\tau_{\text{obs}} - \tau_{\text{ref}}|\) against the calibrated threshold \(\theta\) derived during risk-free calibration. When \(\Delta \tau > \theta\), an anomaly event is flagged. The anomaly detection module records the onset and offset of each breach and buffers multimodal data (RGB, depth, pose throughout the breach event). Upon anomaly detection, the system transitions into a narrative construction and reasoning stage. It synthesizes the intended (\(\tau_{\text{ref}}\)) and actual (\(\tau_{\text{obs}}\)) behaviors and analyzes relevant metrics (e.g., \(\Delta \tau\), rate of change, curvature). Instead of querying the VLM blindly we provide a multimodal grounding.

The system generates a multi-modal experience log, similar to [5] but with a linguistically-grounded narrative about the experience and a set of visual anchor images taken before the anomalous encounter. Anomaly detection acts as a better event-driven key-frame extraction method to feed the VLM compared to most methods that simply query a VLM at regular intervals. Event-driven querying reduces conservatism, filters candidates by spatial proximity, and isolates the single causal object, directly reducing false positives. We test our narration framework with different models and reasoning techniques. The full prompting process is shown in Fig. 2

4.3 Cause Localization↩︎

The VLM outputs a textual description of the object, the VLM might simply output a visual anchor, For example instead of naming the cause as a vent opening on the floor, it outputs the cause as a silver rectangular object on the floor. We need to transfer this visual anchor into a meaningful semantic embedding. We encode the VLM’s generalized textual description into its embedding set \(\mathcal{E}_t = \texttt{Embed}_{lang}(\text{text})\) and extract dense per-pixel embeddings \(\mathcal{E}_p = \texttt{Embed}_{vision}(\text{image})\) from the narration image. We compute similarity scores \(s_p = \texttt{sim}(\mathcal{E}_t, \mathcal{E}_p)\) between the VLM text embedding and each pixel embedding, identifying hotspots corresponding to visually grounded concepts. Using these similarity scores as weights, we compute a weighted average over all pixel embeddings to obtain a richer, contextually grounded embedding. This embedding is stored and then used for re-identification.

\[s_p = \texttt{sim}(\mathcal{E}_t, \mathcal{E}_p), \quad \quad \mathbf{e} = \frac{\sum_p s_p \mathcal{E}_p}{\sum_p s_p}.\]

Upon estimating this embedding, we use it to localize the cause object of the anomaly corresponding to the narration as shown in Fig. 3. We extract the segmentation masks and segment these in our voxel map, getting accurate 3D voxel-level localization for the perceived cause of danger. If we see the cause again in future observations, identified by high \(\texttt{sim}(\mathcal{E}_t, .)\) with the embeddings in \(\Omega_t\), masks are extracted to localize these as well.

4.4 Semantic Adversity Association and Characterization↩︎

We model the predictive effect of a hazard on the agent by learning how different adversities manifest in downstream tracking error (or other adversity signals) as discussed before. Kernel Regression is a natural choice for characterizing these behaviours [25]. By grounding adversity causes in voxel-centric semantic maps and deferring model fitting to the slow loop, DFM2 reframes adversities as valuable signals: disturbances that can be captured, generalized, and reused to improve future resilience.

4.4.1 Voxel-Centric Kernel Regression↩︎

We model adversity at the voxel level, where each semantic voxel encodes the local effect on the robot’s pose. The deviation at a query position \(x\) is predicted as a weighted sum of voxel contributions: let \(\mathcal{V}\) be the set of voxels whose cosine similarity to the cause embedding exceeds \(0.5\). The overall disturbance model is obtained by optimizing the superposed contribution of all voxel-level kernels in \(\mathcal{V}\) to match the observed tracking error. By enforcing identical kernels across voxels, we frame this as a constrained optimization problem that treats adversities as transferable behaviors. We employ anisotropic RBF kernels: their locality confines each voxel’s influence spatially, while the anisotropic extension captures directional robot-environment interactions.

Let \(\mathcal{V}\) denote the set of semantic voxels associated with a hazard, with centers \(c_j \in \mathbb{R}^3\). For a query position \(x \in \mathbb{R}^3\), the disturbance in space is modeled as \[f(x) = A \sum_{j \in \mathcal{V}} \exp\!\left(-\tfrac{1}{2} Q_\ell(x - c_j)\right) + b,\] where \(A, b \in \mathbb{R}\) are regression parameters, and \(Q_\ell(\cdot)\) is an anisotropic quadratic form \[Q_\ell(x - c_j) = \frac{(x - c_j)_x^2 + (x - c_j)_y^2}{\ell_{xy}^2} + \frac{(x - c_j)_z^2}{\ell_{z}^2}.\]

Here, \(\ell_{xy}, \ell_z > 0\) are horizontal and vertical length-scales that govern disturbance spread. Given trajectory tracking errors \(d(x_i)\) along the planned path, we fit \((A, b, \ell_{xy}, \ell_z)\) by minimizing \[\min\text{J} = \min_{\ell_{xy}, \ell_z} \frac{1}{N} \sum_{i=1}^N \big(d(x_i) - f(x_i)\big)^2\]

The non-convex optimization of Equation 6 is solved using the L-BFGS-B algorithm [26] with a multi-start strategy to mitigate local minima. This voxel-centric kernel regression using a multi start L-BFGS-B solver yields fine-grained, geometry-aware fields that generalize across hazards of varying size and pose (e.g., large versus small fans), while retaining computational efficiency. Parameters learned for one hazard instance \((\mathcal{V}, A, b, l_{xy}, l_{z})\) are stored in the danger library with the semantic embedding of the cause. The characterization for some of the causes is shown in Fig. 3

4.4.2 Epistemic Uncertainty Estimation↩︎

While the voxel-centric kernel regression defined in Equation 4-5 provides a robust mean approximation of a disturbance, it is completely deterministic. For a robot traversing a hazard at \(1.0\) m/s with a \(15\) Hz pose estimator, the resulting \(15\)–\(20\) samples are insufficient for full Gaussian Process posterior propagation or heteroscedastic modelling with full covariance modelling. To ensure safety, we must quantify the epistemic uncertainty loosely defined as the uncertainty due to lack of data.We keep the uncertainty estimation tractable by leveraging our semantic prior: we assume the spatial shape of the disturbance (governed by length-scales \(\ell_{xy}, \ell_z\)) is fixed by the object class, while its intensity (amplitude \(A\) and bias \(b\)) is treated as uncertain. This allows us to reformulate the problem as Bayesian Linear Regression (BLR) over a fixed basis. We define a feature vector \(\boldsymbol{\phi}(\mathbf{x})\) consisting of the superposed “shape template" \(\phi_{\text{ker}}\) and a bias term:

\[\begin{align} \boldsymbol{\phi}(\mathbf{x}) &= \begin{bmatrix} \phi_{\text{ker}}(\mathbf{x}) \\ 1 \end{bmatrix}, \\ \phi_{\text{ker}}(\mathbf{x}) &= \sum_{j \in V} \exp\left( -\frac{1}{2} Q_{\ell}(\mathbf{x}-\mathbf{c}_j) \right) \end{align}\]

The unknown parameters are the weights \(\mathbf{w} = [A, \; b]^\top\). Instead of finding a single point estimate for \(\mathbf{w}\), we compute the posterior covariance \(\boldsymbol{\Sigma}_w\). By assuming an uninformative prior (\(\boldsymbol{\Sigma}_p \to \infty\))—reflecting the robot’s lack of a priori knowledge regarding the magnitude of a novel hazard—the weight covariance reduces to:

\[\boldsymbol{\Sigma}_w = \sigma_n^2 (\boldsymbol{\Phi}^\top \boldsymbol{\Phi})^{-1}\]

where \(\boldsymbol{\Phi}\) is the design matrix of collected observations and \(\sigma_n^2\) is the aleatoric noise variance, estimated online from the residuals of the least-squares fit. The predictive variance at any query state \(\mathbf{x}_*\) is then the sum of irreducible noise and the parameter uncertainty projected into the feature space:

\[\sigma^2(\mathbf{x}_*) = \underbrace{\sigma_n^2}_{\text{Aleatoric}} + \underbrace{\boldsymbol{\phi}(\mathbf{x}_*)^\top \boldsymbol{\Sigma}_w \boldsymbol{\phi}(\mathbf{x}_*)}_{\text{Epistemic}}\]

In regions where the shape template predicts a strong disturbance (near the hazard) but data is scarce, the uncertainty in \(A\) is amplified, causing the safety margin to expand conservatively. Conversely, where the template is zero (far from the hazard), the uncertainty collapses to the ambient noise level, preventing false positives.

5 Evaluation↩︎

5.1 Hypotheses↩︎

Hypothesis 1: An experience-driven disturbance library captures edge-case disturbances that may be otherwise missed, leading to higher survival rate.

Hypothesis 2: An event-driven VLM querying mechanism mitigates over-conservativeness of preemptive libraries, leading to faster arrival times and shorter path lengths.

Hypothesis 3: Characterizing the disturbance’s spatial impact, rather than just treating semantics as binary obstacles, leads to more effective downstream planning.

Hypothesis 4: DFM2’s flexible definition of an adversity signal demonstrates applicability across qualitatively distinct failure modes and across multiple robot embodiments and navigation stacks.

5.2 Metrics↩︎

To comprehensively evaluate the performance of DFM2 and the baselines across our hypotheses, we define the following metrics:

• Arrival Time (\(T_{\text{arr}}\))

• Total Path Length (\(L_{\text{path}}\))

• Survival Rate (\(R_{\text{surv}}\))

• Semantic Precision (Prec.)

• Mean Reciprocal Rank (MRR): A measure of the ranking quality of the identified causes, computed as \[\text{MRR} = \frac{1}{M} \sum_{j=1}^{M} \frac{1}{\text{rank}_j},\] where rank_j is the position of the first ground-truth cause in the predicted list for trial j. This metric rewards the system for prioritizing the true cause of a disturbance at the top of its hypothesis list.

• Normalized Cumulative Disturbance (\(\bar{D}_{\text{cum}}\)): To quantify the total tracking error experienced during a mission, we compute the cumulative deviation between the observed trajectory \(\boldsymbol{\tau}_{\text{obs}}\) and the reference trajectory \(\boldsymbol{\tau}_{\text{ref}}\), normalized by the nominal path length: \[\bar{D}_{\text{cum}} = \frac{1}{L_{\text{nom}}} \sum_{i=1}^{N} \|\boldsymbol{\tau}_{\text{obs}}(t_i) - \boldsymbol{\tau}_{\text{ref}}(t_i)\|_2 \cdot \Delta t,\] where \(L_{\text{nom}}\) is the length of the nominal (disturbance-free) reference trajectory and \(\Delta t\) is the sampling interval.

5.3 Baselines↩︎

We compare against 3 baselines that ablate key design choices: semantic grounding, library construction strategy (preemptive vs.experience-driven), and disturbance characterization.

DROAN-GL (Pure Geometric): A pure geometric planner that performs obstacle avoidance using voxel-projected occupancy from depth sensors, with no semantic querying or disturbance characterization. This is a GPU accelerated version based on [27]. All baselines leverage DROAN-GL for path tracking and obstacle avoidance. The DROAN optimization objective can easily be modified to achieve the desired effect with different cost functions.

Pro-Active Hazard Reasoning: Following LaC [7] and Ganai et al. [5], we implement a preemptive baseline using NARadio-based language-embedding similarity lifted to a 3D navigation costmap. We also include Pro-Active-Avoid, which queries the VLM for required clearance and constructs spherical avoidance regions around grounded semantics.

5.4 Ablations↩︎

DFM2-Avoid Fixed Radius: Uses the experience-driven danger library but replaces disturbance characterization with a fixed spherical avoidance radius. Tested at 1.5m (DFM2-Avoid-1.5) and 2.5m (DFM2-Avoid-2.5) to isolate the contribution of adaptive spatial modeling (H3).

DFM2 (Full Method): Combines VLM-based grounding, experience-driven library construction, and voxel-centric disturbance characterization.

To ensure consistency, all methods share the same low-level controller and collision-avoidance stack.

6 Simulation Experiments↩︎

6.1 E1: Cause Reasoning↩︎

We evaluate event-driven causal attribution against traditional anticipative querying (every 4s). We curated a dataset of aerial and wheeled robotic failures across cluttered and sparse environments. In a 40s episode, anticipative querying generates 30–40 potential hazards; with a singular root cause, this yields precision \(<5\%\). In contrast, the DFM2 mechanism isolates the disturbance event before querying the VLM. Among the models tested in Table [tab:vlm95comparison], o4-mini (CoT) achieved the highest performance (0.96 MRR), demonstrating that “Chain-of-Thought" reasoning is beneficial for pruning irrelevant visual artifacts. Notably, Gemini 2.5 Flash emerged as the most efficient non-reasoning model, providing a precision of \(82.9\%\) while maintaining a low output count \((O_p=1.9)\). These results suggest that current VLMs can provide reliable semantic attribution when the search space is constrained to a localized temporal failure window.

6.2 E2: Simulation↩︎

We implement our simulation using NVIDIA Isaac Sim [28], which provides realistic PhysX-based physics and photorealism.

Robot Dynamics, Control, and Perception: We simulate a flying quadrotor. To simulate the dynamics of a quadrotor robot, we use the Isaac Pegasus extension [29]. Pegasus simulates forces and torques from each propeller to drive the body mass while outputting sensor data to PX4 [30]. For quadrotor control, we implement our own software stack, AirStack, that sends velocity and yaw control commands (\(v_x, v_y, v_z, \omega\)) to the PX4 controller. The robot uses RGB-D camera sensors (360\(\times\)​480). We simulate physical disturbances (fans, vents, windows) using Isaac Sim’s force-field APIs.

Experimental Setup: Experiments run in a modular warehouse (IsaacSim assets) with seven aisles and an open area (Fig. 5). We spawn 2–3 disturbance sources per aisle with randomized orientations. The drone must traverse an aisle to reach a goal, with racks of dense semantic clutter flanking each aisle to challenge causal attribution. We run 25 randomized navigation trials each lasting \(>\)​30s.

6.3 Results↩︎

We show the quantitative results of randomized control trials in Table 4, Table 2 and Table 3. DFM2 consistently outperforms all baselines in survival rate and cumulative disturbance, validating H1–H3. DROAN-GL crashes most frequently without semantic awareness; preemptive methods improve safety but suffer over-conservatism (H2). Against binary ablations (DFM2-Avoid-1.5/2.5), DFM2’s continuous disturbance fields (H3) enable emergent behaviors—stopping, reversing, and rerouting around hazards—rather than blind persistence, as shown in Fig. 4.

7 Hardware Experiments↩︎

The simulation experiments validate DFM2 against external physical disturbances. We now present a different kind of adversity: perception-induced localization degradation, wherein the robot’s internal state estimator is compromised by environmental visual properties rather than external forces. This failure mode is common in the real world and interesting because (i) it produces no external observable drift until catastrophic drift occurs, and (ii) it is highly embodiment-specific, the same surface that destroys RGBD odometry is irrelevant to a LiDAR-inertial system. We validate Hypothesis 4 (H4) by deploying DFM2 on a wheeled mobile robot under two such failure modes to evaluate whether DFM2 generalizes across robotic embodiments, navigation stack and disturbance modality. In contrast to wind fields, terrain interaction, or contact forces, the disturbances studied in this section originate from degradation within the perception and localization pipeline itself wherein the robot gets “fooled" into believing it has moved.

To evaluate this setting, we deploy DFM2 on a wheeled mobile robot equipped with an Intel RealSense D455 RGBD camera, a Velodyne VLP32 LiDAR, and an Xsens IMU. The platform performs local navigation using the Dynamic Window Approach (DWA) [31] with laser scan based collision avoidance augmented with DFM2-derived disturbance costs, used through a simple artificial potential field (APF) [32] formulation for adaptation. We evaluate DFM2 using the RTAB-Map RGBD-Inertial Odometry pipeline [1], which combines feature-based visual correspondence estimation, RGBD geometric constraints, IMU-assisted motion prediction, and local graph optimization for pose tracking. Because pose estimation depends directly on the quality and spatial distribution of tracked visual correspondences, the system is particularly sensitive to perceptual degradation such as textureless surfaces and infrared overexposure, which reduce feature observability, destabilize frame-to-frame matching.

We study two commonly observed but often under-characterized real-world perception failures:

• Textureless Surface Degeneration:

  

  Large textureless surfaces reduce the number of reliable visual correspondences available to RGBD odometry systems, producing unstable tracking and increased estimator uncertainty. We induce this failure mode using large feature-deprived sheets positioned near turning regions in narrow corridors, where localization behavior depends strongly on viewpoint and field-of-view occupancy rather than obstacle distance alone.

  Table 5: Transfer axes validated across DFM2 experiments.

  Adversity Mode	Quadrotor (AirStack)	Wheeled Robot (DWA)
  External Physical	✔ Sim + HW	—
  (wind, fans, vents)	(H1–H3)
  Perception Pipeline Failure	—	✔ Hardware
  (textureless, IR overexposure)		(H4)
  Danger Library Transfer	✔ Sim	✔ Hardware
  (re-encounter, new instance)	(H1)	(H4)

• Infrared Sensor Overexposure:

  

  Active infrared RGBD sensors are susceptible to strong illumination and IR interference, resulting in degraded depth observations and unstable odometry estimation. We reproduce this phenomenon by exposing the RealSense D455 sensor to high-intensity illumination from a pocket torch during operation.

We monitor the trace of the pose covariance matrix \(\text{Tr}(\Sigma)\) as the adversity signal in place of position tracking error, since perception-induced failures manifest in estimator uncertainty before producing observable trajectory drift. The disturbance model \(f(\mathbf{x})\) characterizes the spatial distribution of this covariance inflation. The detection threshold \(\theta\) is calibrated identically via disturbance-free runs, preserving the same pipeline structure as the simulation setting. Across both scenarios, DFM2 detects and spatially characterizes localized perceptual degradation before substantial trajectory drift becomes externally observable, giving us a rich characterization where avoidance is not just simple hardcoded clearances, capturing richer dependencies that are a function of distances, adversity size and intensity. The planner adapts its motion behavior to reduce exposure to perceptually degraded regions, indirectly improving feature observability and estimator stability during navigation. We compare DFM2 against the DWA planner for the best qualitative analysis as the avoid mechanisms almost always miss the cause or produce infeasible trajectories within the narrow corridor which is just 2.5m wide, while the robot track width is 0.4m. Pro-Active-Avoid constructs spherical avoidance regions of radius 0.75-2.5m around detected semantics making navigation geometrically infeasible without manual radius tuning specific to this environment which defeats the purpose of this work. All onboard computation was performed on a consumer-grade laptop equipped with an octa-core Ryzen 7 5600X CPU, 16GB RAM, and an RTX 3060 GPU, running the complete perception, planning, and control stack in real-time with all the sensors.

7.1 Evaluation Metrics↩︎

Unlike the simulation experiments, perception-driven failures in real-world systems often emerge before significant trajectory drift becomes externally observable. We therefore evaluate localization health using internal estimator consistency metrics in addition to qualitative navigation behavior.

• Inlier Ratio \((R_{inlier})\): Measures the fraction of geometrically consistent feature correspondences used by the odometry backend: \[R_{inlier} = \frac{N_{inlier}}{N_{matches}}\] Lower values indicate degraded visual observability and unstable tracking.

• Pose Covariance Trace \((\mathrm{Tr}(\Sigma))\): Measures estimator uncertainty using the trace of the pose covariance matrix reported by the odometry system: \[\mathrm{Tr}(\Sigma)=\sum_i \Sigma_{ii}\] Higher covariance indicates reduced estimator confidence and localization instability.

• Degradation Duration \((T_{deg})\): Measures cumulative time spent in degraded localization states, determined using covariance thresholding and tracking instability indicators.

7.2 Results↩︎

The results in Table 6 demonstrate that DFM2 improves localization robustness under perception degradation caused by large textureless regions validating H4. This hypothesis is the most architecturally significant of the four. H1–H3 establish that DFM2 works; H4 establishes why it works across contexts. A wheeled robot experiencing localization drift from a textureless surface and a quadrotor experiencing trajectory deviation from a wind field are, from DFM2’s perspective, identical events. Compared to baseline DWA navigation, DFM2 maintains a substantially higher feature correspondence inlier ratio while simultaneously reducing pose covariance inflation and cumulative localization failure duration. Qualitatively, the baseline planner frequently approaches feature-deprived regions too aggressively during turning maneuvers, resulting in unstable tracking and intermittent odometry degradation. In contrast, DFM2 adapts robot motion to reduce exposure to perceptually degraded viewpoints, indirectly improving visual observability and estimator stability before catastrophic localization failure occurs.

8 limitations and Future Work↩︎

One of the main concerns with DFM2 remains its reliance on the visual observability of a failure cause. While there are a handful of scenarios where DFM2 could identify the closest visual indicator, for example a wet floor where the wet patch is not visible the characterization can be learnt for the “do not slip" sign kept at the center of the puddle, this reliance on vision is a limiting factor. The reliance on a single VLM for cause association is a limiting factor, although the current models show great promise at identifying these hazards, fallbacks and causality based verification could be a next step. Future advances in multimodal reasoning models may further improve attribution robustness. Additionally, attributing adversities from a single encounter is brittle in cluttered scenes; accumulating evidence across multiple interactions and applying causal reasoning to arbitrate among competing hypotheses would improve attribution reliability. DFM2’s engineered, disjoint modules may introduce compounding failure modes at module boundaries and limit joint optimization; end-to-end learning approaches that tightly couple perception, causal reasoning, and disturbance modeling within a unified differentiable framework represent a natural next step. Several directions remain open for future work including experimenting with different indicators for adversity.

9 Conclusion↩︎

We present Don’t Fool Me Twice (DFM2), a framework for continual, experience-driven adaptation to embodiment-specific adversities in unstructured environments. By coupling event-driven kinematic narration with semantic voxel-centric kernel regression, DFM2 attributes online disturbances to causal semantics, characterizes their spatial extent, and proactively plans around them upon re-encounter. Across 25 randomized simulation trials, DFM2 achieves an 81.8% survival rate and reduces cumulative disturbance by more than 3x relative to the next best method, validating all four core hypotheses. Our framework has been validated across diverse adversity modes and an interesting application is traversability estimation where costmaps can be learnt with our characterization.

10 Acknowledgements↩︎

We would like to thank the folks at the AirLab at Carnegie Mellon University and the Autonomous Systems Lab at IIT Madras for their contributions to the ideation and development of this framework. We would also like to thank Dr John Dolan and Ms Rachel Burcin for facilitating the RISS program and their continued support. We greatly appreciate helpful discussions with Jay Patrikar, Jay Karhade, Andrea Bajcsy and the Intent Lab. Finally we thank Sayooj Raveendran, Satyam Singh, Dipshikha Hazari, John Keller, Seungjae Baek, and AirLab’s small drone team for help with systems deployment. This work at CMU was supported by the Defense Science and Technology Agency (DSTA) #DST000EC124000205 contract.

References↩︎