1 Introduction↩︎

Learning visuomotor policies from demonstrations has enabled impressive manipulation behaviors [1]–[3]. However, these policies often remain brittle when deployed outside the training visual context [4], [5]. In particular, task-irrelevant appearance changes—such as tabletop color shifts, background clutter, or object recoloring—can induce severe performance drops even when task semantics and dynamics remain unchanged.

Recent progress has primarily focused on scaling model capacity, including generative policies (diffusion/flow) [1], [6], [7] and vision–language(-action) systems [3], [8]–[10]. While these approaches improve capability and task diversity, robustness to visual distribution shifts is typically addressed through data augmentation, broader pretraining, or additional sensing. In most cases, the policy continues to consume raw RGB observations, where nuisance appearance factors dominate the input distribution [2], [11]–[13].

In this work, we instead revisit the observation interface. Rather than modifying the policy architecture or scaling data, we explicitly control what the policy sees. We leverage promptable segmentation foundation models [14], [15] to construct a task-aware visual abstraction that suppresses appearance variation while preserving action-relevant structure.

Concretely, we introduce a two-level observation design. L0 (Seg-repaint) segments task-relevant entities (target object and robot/gripper) and repaints the scene into a canonical label-colored image with a constant background. L1 (Seg+Depth) further injects masked monocular depth to retain geometric cues when finer spatial structure is important. Both representations remain standard 3-channel, image-like inputs and can be paired with existing policy backbones without architectural modification.

We evaluate the proposed interface under controlled appearance shifts on RoboMimic, ManiSkill, and RLBench, as well as on a real Franka robot. Across both flow-matching and VLA-style policies, our method preserves in-distribution performance while substantially improving robustness under out-of-distribution visual changes.

Contributions: (1) A task-aware semantic–geometric observation interface that canonicalizes appearance via segmentation-based repainting with optional depth injection, while keeping a standard image input; (2) A systematic robustness evaluation under controlled appearance shifts across simulation and real-world benchmarks, demonstrating consistent OOD gains without additional policy fine-tuning.

2 Related Work↩︎

In this section, we review recent progress in (i) visuomotor policy learning, with an emphasis on common visual observation setups and action parameterizations, (ii) modern foundation models for segmentation, and how segmentation/semantic priors are integrated into robot policies to improve robustness and generalization.

2.1 Visuomotor Policy Learning↩︎

Recent visuomotor policies are often built by choosing (i) an observation-conditioning interface—what sensory and task signals the policy conditions on (e.g., multi-view RGB/RGB-D, proprioception, language, or goal images) [16]–[19]—and (ii) an action parameterization. Actions are either decoded directly (continuous end-effector deltas or discretized symbols) [3], [10] or generated via implicit refinement, where the network predicts an auxiliary quantity (e.g., diffusion denoiser/score or a flow-matching velocity field) and iterative refinement yields action chunks [1], [7], [20]. Consistent with this view, VLA systems mainly differ in the decoder attached to a VLM backbone: token autoregression [3], [8], [9], continuous regression heads [10], or diffusion/flow-based decoders for continuous action chunks [21]–[23].

Across simulation benchmarks, the prevailing camera setup uses one or more third-person RGB views and, when available, an eye-in-hand/wrist view; depth is optional and benchmark-dependent [12], [13]. When wrist observations are unavailable, recent work explores “virtual cameras” that synthesize in-hand views from external observations at test time [24]. Large-scale real-robot datasets further reinforce this multi-view pattern: Open X-Embodiment aggregates multi-robot trajectories at scale [11], and models such as Octo and OpenVLA are trained on this corpus with third-person and optional wrist observations [2], [9].

2.2 Modern Segmentation Models in Robot Learning↩︎

Modern segmentation has shifted toward promptable foundation models that make masks transferable across objects, scenes, and prompts. Segment Anything (SAM) introduced scalable interactive segmentation [25], SAM 2 extends promptable segmentation to images and videos via memory-based tracking [14], and SAM 3 further targets open-vocabulary concept segmentation [15]. These advances enable segmentation to serve as a controllable perception interface for policy learning rather than merely a standalone vision output.

In robotics, segmentation and object-centric priors have been used to improve robustness and generalization in several ways. Shadow leverages masks for cross-embodiment policy transfer [26], object-centric grounding reduces clutter sensitivity in VLA systems [27], and ARRO filters task-irrelevant regions online to improve robustness under visual appearance shifts [28]. Related directions study category-level generalization with spatial–semantic representations [29], minimal intermediate visual representations [30], semantic augmentation through synthetic variation [31], and foundation-model priors for distraction-robust imitation [32]. Recent systems also integrate promptable segmentation directly into policy architectures [33], [34].

Unlike these approaches, our focus is controlled observation-level canonicalization. Rather than filtering image regions, augmenting the training distribution, replacing the visual encoder with latent foundation-model features, or modifying the policy architecture, we construct a standard 3-channel semantic–geometric observation that explicitly canonicalizes task-relevant entities under object and background appearance shifts.

3 Methodology↩︎

As shown in Fig. 1, we construct a semantic–geometric observation interface using open-vocabulary segmentation and optional depth cues, and train a visuomotor policy on the resulting canonical observations.

3.1 Problem Setup↩︎

We study visuomotor policy learning from demonstrations. At each time step \(t\), the robot observes an RGB image \(I_t \in \mathbb{R}^{H \times W \times 3}\) and executes an action \(a_t \in \mathbb{R}^{d_a}\). Each episode yields a trajectory \(\tau=\{(I_t,a_t)\}_{t=1}^{T}\), and we are given a dataset of expert demonstrations \(\mathcal{D}=\{\tau^{(i)}\}_{i=1}^{N}\).

Our key design choice is to replace raw RGB with a task-aware extracted observation \[\tilde{o}_t = g(I_t, c), \label{eq:obs95extraction}\tag{1}\] where \(c\) is an open-vocabulary task specification (e.g., an object name), and \(g(\cdot)\) suppresses task-irrelevant appearance while preserving action-relevant structure. We instantiate \(\pi_\theta\) with a Flow Matching Policy (FMP) [35], [36], while the observation extraction \(g(\cdot)\) is policy-agnostic and can be paired with other imitation learning backbones.

Algorithm 2 summarizes the overall pipeline: at each time step we (i) use SAM3 with text prompts to segment the robot/gripper and task object(s), (ii) render a canonical label-colored observation by repainting the image based on the resulting masks (L0), and (iii) optionally inject geometric structure by overwriting the segmented regions with normalized monocular depth (L1). In the following subsections, we detail the construction of L0 and L1 and then describe policy learning with FMP on the resulting observations.

3.2 Task-Aware Observation Extraction (L0)↩︎

Given a raw RGB observation \(I_t \in \mathbb{R}^{H \times W \times 3}\), we construct a task-aware observation that canonicalizes appearance by rendering the scene into a label-colored canvas. We assume an open-vocabulary task specification \(c\) and form text prompts \(p_t\) using a fixed template per benchmark (e.g., robot gripper and the target object name). We apply SAM3 with these prompts to obtain binary masks for the robot/gripper and the target object: \[\begin{align} M^{\mathrm{r}}_t &= \mathrm{SAM3}(I_t, p^{\mathrm{r}}), \\ M^{\mathrm{o}}_t &= \mathrm{SAM3}(I_t, p^{\mathrm{o}}), \end{align} \qquad M^{\mathrm{r}}_t, M^{\mathrm{o}}_t \in \{0,1\}^{H \times W}. \label{eq:sam395masks}\tag{2}\]

When multiple instances are predicted for a prompt, we take their union. We denote the union mask as \(M_t = M^{\mathrm{r}}_t \lor M^{\mathrm{o}}_t\). We then render a canonical, 3-channel label-colored image by assigning fixed colors to each entity and a constant color to the background: \[\begin{align} I^{\mathrm{L0}}_t &= \kappa \cdot \Bigl(1-\mathrm{broadcast}(M_t)\Bigr) \\ &\quad + \mathbf{c}_{\mathrm{r}} \odot \mathrm{broadcast}(M^{\mathrm{r}}_t) + \mathbf{c}_{\mathrm{o}} \odot \mathrm{broadcast}(M^{\mathrm{o}}_t), \end{align} \label{eq:l095repaint}\tag{3}\] where \(\kappa\) is a constant background color (we use \(\kappa=\mathbf{0}\)), \(\mathbf{c}_{\mathrm{r}}\) and \(\mathbf{c}_{\mathrm{o}}\) are fixed RGB colors for the robot/gripper and target object, respectively, and \(\mathrm{broadcast}(\cdot)\) expands a 2D mask to three channels.

This L0 observation removes most nuisance appearance variation (e.g., textures and colors) while preserving the spatial layout of action-relevant entities.

3.3 Geometry Injection via Depth Anything (L1)↩︎

While L0 suppresses nuisance appearance by mapping the scene to a canonical, entity-colored canvas, some tasks still require geometric cues such as shape and fine spatial structure. We therefore construct L1 by estimating monocular depth with Depth Anything 3 [37] and injecting masked, normalized depth into the target-object region, yielding a single 3-channel, image-like observation (no channel concatenation).

3.3.0.1 Monocular depth estimation

Given the raw RGB frame, we predict a dense depth map \[\hat{D}_t = \mathrm{DepthAnything}(I_t), \qquad \hat{D}_t \in \mathbb{R}^{H \times W}.\] Since monocular depth is scale-ambiguous, we use relative depth normalized within the target-object mask \(M_t^{\mathrm{o}}\).

3.3.0.2 Masked depth normalization

Let \(\Omega_t^{\mathrm{o}} = \{(u,v)\mid M_t^{\mathrm{o}}(u,v)=1\}\) denote target-object pixels. We compute \[D^{\mathrm{norm}}_t(u,v) = \frac{\hat{D}_t(u,v)-d_{\min}}{d_{\max}-d_{\min}+\epsilon}, \quad (u,v)\in\Omega_t^{\mathrm{o}}, \label{eq:depth95norm}\tag{4}\] where \(d_{\min}=\min_{(u,v)\in\Omega_t^{\mathrm{o}}}\hat{D}_t(u,v)\) and \(d_{\max}=\max_{(u,v)\in\Omega_t^{\mathrm{o}}}\hat{D}_t(u,v).\) We set \(D^{\mathrm{norm}}_t(u,v)=0\) for \((u,v)\notin\Omega_t^{\mathrm{o}}\).

3.3.0.3 Depth overwrite fusion

We tile the normalized depth map to three channels, \(I_t^{\mathrm{dep}}=\mathrm{tile}(D_t^{\mathrm{norm}},3)\), and overwrite only the target-object region on the L0 canvas: \[I^{\mathrm{L1}}_t = I^{\mathrm{L0}}_t \odot \bigl(1-\mathrm{broadcast}(M_t^{\mathrm{o}})\bigr) + I^{\mathrm{dep}}_t \odot \mathrm{broadcast}(M_t^{\mathrm{o}}). \label{eq:depth95overwrite}\tag{5}\] The resulting observation remains a 3-channel image that can be processed by standard vision encoders without modifying the policy architecture.

3.4 Policy Learning with Flow Matching↩︎

We instantiate \(\pi_\theta\) as a Flow Matching Policy (FMP) [35], [36]. Given the extracted observation \(\tilde{o}_t\) (L0/L1), FMP learns a time-dependent vector field \(v_\theta(x,s;\tilde{o}_t)\) that transports samples from a simple base distribution to the expert action distribution conditioned on \(\tilde{o}_t\). Training minimizes the flow-matching objective: \[\mathcal{L}_{\mathrm{FM}}(\theta) = \mathbb{E}_{(\tilde{o}_t,a_t)\sim\mathcal{D}, x_0 \sim p_0,\; s\sim \mathcal{U}(0,1)} \; \left[ \left\| v_\theta(x_s,s;\tilde{o}_t) - \dot{x}_s \right\|_2^2 \right], \label{eq:fm95loss}\tag{6}\] where \(x_0 \sim p_0\) (e.g., Gaussian distribution), \(x_s\) lies on a smooth flow from \(x_0\) to the expert action \(a_t\), and \(\dot{x}_s\) is the corresponding target velocity. At inference, we integrate the learned vector field from \(s=0\) to \(1\) with a lightweight ODE solver to obtain an action sample conditioned on \(\tilde{o}_t\).

4 Experiments↩︎

We evaluate whether our task-aware semantic–geometric observation interface improves robustness to appearance shifts—novel object textures/colors and background changes—while preserving in-distribution performance. To isolate the effect of the observation, we keep the policy training pipeline fixed (same backbone, vision encoder, and optimization settings) and vary only the input representation. We report results on two complementary policy families: a flow-matching policy and a vision–language–action model (SmolVLA [38]), demonstrating that the proposed abstraction transfers across backbones rather than being tailored to a single policy.

4.1 Experimental Setup and Evaluation Protocol↩︎

4.1.0.1 Benchmarks and tasks

We conduct experiments on RoboMimic (Lift) [12], ManiSkill (YCB object grasp) [39], and RLBench (CloseGrill, ToiletSeatDown, CloseMicrowave and OpenBox) [13]. We further validate on real Franka robot arm setup, described in Sec. 4.4.

4.1.0.2 Methods compared

We compare three observation variants: (i) ORG: raw images; (ii) L0: SAM3-driven canonical label-colored rendering (constant background with fixed colors for robot/gripper and target object; Sec. 3.2); and (iii) L1: L0 with optional geometry injection by overwriting masked regions with normalized monocular depth from Depth Anything 3 (DA3) (Sec. [sec:depth95injection]). All variants share the same policy backbone and training recipe (FMP or SmolVLA); only the input observation is changed. Our goal is to isolate the effect of the observation interface: all policy backbones, encoders, optimization settings, and demonstrations are kept fixed, and only the input representation is changed. Training-time augmentation/domain randomization and pretrained latent visual encoders are complementary robustness mechanisms and are therefore not the focus of this controlled comparison.

4.1.0.3 Appearance-shift (OOD) protocol

We report both in-distribution (ID) and out-of-distribution (OOD) performance. ID uses the standard benchmark visual settings. For OOD, we introduce controlled appearance shifts that are held out from training:

• OOD-Obj: novel object appearances via unseen colors.

• OOD-Bg: novel scene appearance via background color changes and/or additional clutter distractors.

We keep task success definitions unchanged, isolating appearance as the primary source of distribution shift.

4.1.0.4 Implementation details

We train each method with the same network architecture, optimizer, and training epochs and report mean over \(3\) random seeds. For our observation extractor, SAM3 is prompted by the open-vocabulary task specification (text prompt) (Sec. 3.2). Depth is estimated from RGB using DA3 and normalized within the segmented region before overwrite fusion (Sec. [sec:depth95injection]). SAM3/DA3 are adapted on the ID training split only; no OOD data is used for perception adaptation.

4.2 Foundation Models Adaptation and Usage↩︎

We adapt the foundation models used by our observation interface (SAM3 for segmentation and DA3 for monocular depth) with lightweight LoRA adapters [40] to match the visual statistics of each benchmark. Importantly, we fine-tune these models using only the in-distribution (ID) training data used to train the policy, and do not use any OOD data for perception adaptation.

4.2.0.1 SAM3 LoRA fine-tuning

To improve segmentation reliability, we fine-tune SAM3 using LoRA adapters. Each RGB frame yields two training samples: (i) a robot/gripper union mask and (ii) a task-object union mask, paired with fixed text prompts (e.g., robot gripper, target object). We freeze the SAM3 backbone and insert LoRA modules into the text encoder self-attention projections, targeting q_proj and v_proj in the last \(4\) transformer blocks. We use LoRA rank \(r{=}8\), \(\alpha{=}16\), dropout \(0.05\), and train for 50 epochs with AdamW (lr \(5{\times}10^{-4}\), weight decay \(10^{-2}\)) under mixed precision. We optimize a standard mask supervision loss, \(\mathcal{L}=\mathcal{L}_{\mathrm{BCE}}+0.5\,\mathcal{L}_{\mathrm{Dice}}\), computed between the predicted mask logits and ground-truth masks. The same LoRA-adapted SAM3 checkpoint is used for all runs within a benchmark and applied at test time without further adaptation.

3.5pt

4.2.0.2 DA3 LoRA fine-tuning

To better preserve task-relevant geometry under appearance variation, we adapt Depth Anything 3 (DA3) with lightweight LoRA while keeping the pretrained backbone frozen. Specifically, we insert LoRA modules into selected nn.Linear layers of the DA3 encoder, targeting attention and MLP projection layers (e.g., qkv/q_proj/k_proj/v_proj/out_proj and fc1/fc2), and optimize only the LoRA parameters. Training uses paired RGB–depth supervision with DA3-style preprocessing: images are resized with an upper-bound strategy to process_res\(=504\), aligned to patch_size\(=14\), and ImageNet-normalized. We minimize the scale-invariant SILog loss (\(\lambda{=}0.85\)). We use LoRA rank \(r{=}8\), \(\alpha{=}16\), dropout \(0.05\), and AdamW with lr \(10^{-4}\) (weight decay \(0\)) for \(100\) epochs. (For real-robot experiments, we use the pretrained DA3 model as-is; DA3 LoRA adaptation is only applied in simulation benchmarks.)

5pt

3.0pt

4.3 Simulation Benchmarks↩︎

We evaluate robustness to appearance shifts on three simulation benchmarks: RoboMimic, ManiSkill, and RLBench. For each benchmark, we report in-distribution (ID) performance under the default visual configuration and out-of-distribution (OOD) performance under controlled appearance changes. All methods use the same policy backbone (FMP) and training budget; only the input observation differs (Sec. 4.1).

4.3.0.1 RoboMimic (Lift): object & support-surface color shifts

On RoboMimic Lift, we test robustness to changes in the cube appearance (OOD-Obj) and the table/support-surface appearance (OOD-Bg). OOD-Obj varies the cube color relative to training, while OOD-Bg changes the dominant tabletop color without altering task dynamics. Tab. 1 summarizes the results.

Under both cube-color and tabletop-color shifts, raw RGB exhibits severe brittleness: the mean success drops from \(98.7\%\) (ID) to \(62.0\%\) under OOD-Obj and further to \(18.4\%\) under OOD-Bg (Tab. 1), with some tabletop colors nearly collapsing performance (e.g., \(1.3\%\) on OOD3). In contrast, segmentation-based repainting (L0) substantially stabilizes performance across appearances, improving mean OOD success to \(89.6\%\) (OOD-Obj) and \(90.7\%\) (OOD-Bg).

Adding depth-guided geometry preservation (L1) yields only marginal changes on Lift: L1 is comparable to L0 for OOD-Obj (\(89.8\%\) vs.\(89.6\%\) mean) and OOD-Bg (\(91.1\%\) vs.\(90.7\%\) mean), without a consistent advantage across individual OOD settings. This is expected because Lift is largely governed by coarse object localization and reaching, where segmentation already removes most nuisance appearance and the remaining geometric ambiguity is limited; consequently, explicit depth cues provide little additional benefit beyond semantic filtering in this task.

4.3.0.2 ManiSkill (YCB Object grasp): unclutter-to-clutter generalization

We study a YCB-style grasping task (green lego). Policies are trained in a non-cluttered environment (ID) and evaluated in visually distinct cluttered environments (OOD-Bg) containing distractor objects and different scene layouts. This isolates robustness to background clutter and distractors. Tab. 2 reports success rates under increasing clutter.

The raw RGB baseline is highly sensitive to clutter: although it achieves \(98.0\%\) success in ID, performance drops sharply in cluttered scenes (OOD mean \(15.0\%\)), indicating that distractors and background complexity substantially disrupt visuomotor inference. In contrast, both segmentation-based observation variants are robust across clutter levels. L0 (Seg-repaint) maintains high success under OOD (mean \(93.3\%\)), demonstrating that suppressing task-irrelevant regions is sufficient to recover most of the lost performance. Adding depth-guided geometry preservation (L1) provides only a small additional gain over L0 in this grasping setup (OOD mean \(94.0\%\) vs.\(93.3\%\)), suggesting that once the target is isolated from distractors, monocular-depth cues offer limited incremental benefit for this relatively local grasping task.

4.3.0.3 RLBench: tabletop color shifts

We evaluate on RLBench tasks CloseGrill, ToiletSeatDown, CloseMicrowave, and OpenBox. ID uses the default training appearance, while OOD-Bg changes the dominant tabletop color (OOD1–3) without altering task dynamics. Table 3 summarizes success rates under these controlled background shifts.

Across all four tasks, the raw-RGB baseline degrades sharply under OOD-Bg: for example, CloseMicrowave drops from \(80.7\%\) (ID) to \(7.3\)–\(11.3\%\) (OOD), and OpenBox drops from \(60.0\%\) (ID) to near-zero success (\(1.3\)–\(4.7\%\)). In contrast, segmentation-based repainting (L0) substantially stabilizes performance under background changes, recovering strong OOD success on CloseGrill (\(73.3\)–\(74.0\%\)), ToiletSeatDown (\(80.7\)–\(83.3\%\)), and CloseMicrowave (\(82.7\)–\(86.7\%\)), and improving OpenBox to \(52.0\)–\(54.7\%\).

Adding the geometry cue (L1) further improves or maintains robustness, with the clearest gains on CloseMicrowave (ID \(80.7\%\rightarrow96.7\%\); OOD \(89.3\)–\(92.0\%\)) and moderate improvements on CloseGrill and ToiletSeatDown. For OpenBox, L1 helps in one OOD setting (OOD1 \(62.0\%\)) but is not uniformly better than L0 (e.g., OOD3 \(51.3\%\) vs.\(52.0\%\)), suggesting the depth-guided overwrite is most beneficial on tasks where finer spatial structure is the limiting factor rather than distractor suppression alone.

4.4 Real Robot Experiments↩︎

4.0pt

We further validate the proposed observation interface on a real Franka robot under controlled appearance shifts (Fig. 4). We consider two manipulation tasks.

4.4.0.1 Task 1: ReachX

The robot moves the end-effector to touch the marker labeled “X” among three planar markers. Marker layout and support-surface appearance vary across ID and OOD settings. As this task primarily requires planar localization, we compare Org (RGB) and L0.

4.4.0.2 Task 2: CloseCabinet

The robot closes a partially open cabinet. This task requires contact and spatial alignment; accordingly, we compare Org (RGB) and L1.

4.4.0.3 Deployment protocol

We deploy the Flow Matching Policy (FMP) to predict a target joint state conditioned on the extracted observation (L0 or L1) from the current and previous frames. The predicted state is executed via franky control library [41], which tracks the target using low-level joint control. No additional policy fine-tuning or test-time adaptation is performed under OOD appearances.

For each task, we collect 25 demonstrations using GELLO [42]. In each run, 20 demonstrations are used for training and 5 are held out; training is repeated with three random splits.

For L0, segmentation masks used to construct the training set are obtained using SAM3 with text prompts; when text prompting fails, a minimal click prompt is provided to correct the mask, and the resulting masks are used to fine-tune SAM3 via LoRA on the training data only. At inference time, segmentation relies solely on text prompts.

For L1, monocular depth is estimated using the pretrained Depth Anything 3 model without fine-tuning on real-robot data. No OOD data is used for perception adaptation, and both perception models remain fixed during policy training and evaluation.

At test time, we evaluate 20 rollouts per trained model under each ID/OOD condition, varying object/cabinet pose and support-surface appearance. Success rates averaged over three seeds are reported in Table 4. On an RTX 4060 Ti, SAM3 takes \(\approx\)​0.015 s per prompt (robot+object \(\approx\)​0.03 s/frame), and Depth Anything 3 takes \(\approx\)​0.052 s/frame. This latency is compatible with our target-joint tracking setup and task dynamics, which do not demand high-frequency visual updates.

4.4.0.4 Results

On ReachX, the RGB baseline drops from 61.7% (ID) to 21.7% / 25.0% (OOD1/2). In contrast, L0 maintains strong performance across conditions (88.3% ID; 83.3% / 81.7% OOD), substantially reducing the robustness gap.

On CloseCabinet, RGB degrades from 85.0% (ID) to 45.0% / 48.3% (OOD1/2), whereas L1 improves OOD robustness to 78.3% / 75.0% while maintaining comparable ID performance (81.7%). These results are consistent with simulation: L0 performs well on marker reaching, and L1 maintains robust performance on the cabinet-closing task under appearance shifts.

4.5 Ablations↩︎

4.5.0.1 Robot/gripper mask ablation

Beyond segmenting the task-relevant object, we examine whether explicitly including the robot/gripper in the SAM3 vocabulary is necessary. As shown in Table 5, retaining only the target object (Target-only) causes performance to collapse on both RoboMimic Lift (7.3%) and RLBench CloseGrill (5.3%). This suggests that masking out the robot removes action-critical visual state—e.g., gripper pose, approach direction, and contact configuration—which is essential for mapping observations to actions. In contrast, adding the robot/gripper mask (Target+Robot) restores strong performance (98.7% on Lift, 74.7% on CloseGrill), indicating that the robot itself should be treated as a first-class entity in the task-aware observation interface.

6pt

4.5.0.2 SAM3 fine-tuning ablation

We evaluate the impact of LoRA fine-tuning on SAM3 segmentation quality under ID and held-out tabletop appearances. Table 6 reports mask IoU on ToiletSeatDown for the robot/gripper (R-IoU) and the target object (O-IoU).

The pretrained SAM3 fails to recover the robot/gripper mask \((0.0\%\) R-IoU) and yields low-quality object masks (14.0–32.9% O-IoU). LoRA fine-tuning substantially improves segmentation, achieving 98.9–99.3% R-IoU and 99.2–99.5% O-IoU consistently across ID and OOD settings. Figure 5 shows an OOD example: the pretrained model misses the gripper and produces incomplete object masks, while the LoRA-finetuned model segments both reliably.

4.0pt

4.5.0.3 Generalization to VLA backbones (SmolVLA)

To verify that our observation interface is policy-agnostic and not tailored to Flow Matching Policy (FMP), we additionally fine-tune and evaluate a representative vision–language–action model, SmolVLA [38], on RLBench. Table 7 reports success rates under the training tabletop appearance (ID) and two held-out tabletop colors (OOD1/2). While raw RGB already attains reasonable ID performance, it degrades substantially under appearance shifts—most notably on CloseMicrowave, where success drops from 64.0% (ID) to 4.0%/8.0% (OOD1/2). In contrast, our semantic repainting (L0) markedly improves robustness on both tasks, increasing OOD success to 64.0%/60.0% on CloseMicrowave and 64.0%/66.0% on CloseGrill. Injecting depth geometry (L1) further boosts performance on the more geometry-sensitive CloseMicrowave, improving both ID (84.0%) and OOD (78.0%/76.0%) while maintaining strong robustness on CloseGrill. These results indicate that the proposed observation mapping consistently stabilizes policy rollouts across visual shifts beyond diffusion/flow-style policies, and transfers to VLA-style controllers without any backbone-specific modification.

4.5.0.4 Compare with S2Diffusion

S2Diffusion [29] is closely related to our setting: it combines segmentation and monocular depth cues by channel-wise concatenation to form a spatial–semantic observation. Since the paper does not specify how this multi-channel tensor is fed into a standard 3-channel ResNet encoder, we implement an S2-style baseline in our FMP framework by constructing a 2-channel input with channel-0 = binary mask and channel-1 = normalized depth (matching our dataset), and adding a lightweight \(1{\times}1\) conv adapter (\(2{\rightarrow}3\)) before the visual encoder to avoid modifying the ResNet backbone. As shown in Table 8, this S2-style representation underperforms our L1 repaint/overwrite observation on both CloseGrill and CloseMicrowave, in-distribution and under background shifts (OOD1–3).

5pt

5 Conclusion↩︎

We presented a task-aware semantic–geometric observation interface that leverages vision foundation models for segmentation and monocular depth to canonicalize visual input via segmentation-based repainting (L0) and optional depth injection (L1), improving robustness to appearance shifts without modifying the policy backbone.

Across RoboMimic, ManiSkill, and RLBench, the proposed interface preserves in-distribution performance while substantially improving robustness under object and background appearance changes. The same abstraction benefits both Flow Matching Policies and SmolVLA, indicating backbone-agnostic gains. Real-robot experiments on a Franka arm further confirm that the interface significantly reduces performance degradation under support-surface shifts without additional policy fine-tuning or test-time adaptation.

A limitation is that canonicalization may discard cues that are task-relevant, such as color, texture, or fine surface details; L1 also depends on reliable segmentation and monocular depth under occlusion. Combining the interface with data augmentation or pretrained latent visual features is a promising direction for future work.

These results highlight the importance of revisiting the observation interface as a complementary approach to scaling policy capacity for robust visuomotor learning.

References↩︎

---

1. \(^{1}\)Authors are at Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), Abu Dhabi, UAE. Email: {firstname.lastname}@mbzuai.ac.ae↩︎