1 Introduction↩︎

Vision-and-Language Navigation (VLN) [1] models have demonstrated superior performance across a variety of tasks. Moreover, the emergence of Multimodal Large Models (MLMs) [2] has injected new momentum into the VLN field. By grounding natural language instructions into visual observations, state-of-the-art agents [1], [3]–[9] demonstrate remarkable proficiency in goal-oriented planning. However, as embodied AI transitions from simulated testbeds to real-world deployment, the definition of navigation success must evolve beyond mere reachability to encompass social compliance and safety. In complex urban scenarios, valid trajectories are governed strictly by semantic rules (i.e., may I go?) rather than just physical geometry (i.e., can I go?) [10], [11]. Although physically traversable, a road may be semantically forbidden due to regulatory signage (e.g., “No Entry”); disregarding such constraints can lead to critical safety hazards. Therefore, enabling agents to perceive and follow these rule-based constraints is not just an improvement but a critical requirement for safe urban navigation and human-computer interaction [12].

Despite obvious progress in visual reasoning [13], current VLN models exhibit a critical compliance deficit. Existing works primarily focus on optimizing shortest-path efficiency or maximizing exploration coverage [14], [15], often neglecting the integration of high-level regulatory signals. As shown in Figure 1, standard agents tend to fall into a “Goal-driven trap”, where they follow clear paths but ignore critical semantic restrictions. Existing models over-rely on salient geometry and fail to ground small-scale semantic cues effectively, preventing them from overriding geometric priors [16], [17]. Furthermore, the scarcity of diverse, safety-critical training data in existing datasets [4], [18], [19] exacerbates this alignment problem.

To bridge the gap between idealized paths and real-world constraints, we build upon Touchdown [4] to introduce Rule-VLN. Compared to the existing VLN task, our dataset explicitly incorporates traffic signs, which serve as important navigational cues despite their small visual footprint. Spanning a massive 29,000-node urban environment, Rule-VLN injects 177 diverse regulatory categories into 8,180 specific nodes systematically organized across four curriculum levels. To ensure these generated constraints are visually and geometrically realistic, we employ dynamic graph modification alongside a Mask-Prioritized Semantic Injection (MPSI) pipeline. Evaluations demonstrate that these non-physical obstacles significantly impede the navigation performance of current SOTA models. To achieve the ability of rule navigation, we propose the Semantic Navigation Rectification Module (SNRM), a plug-and-play, zero-shot module that integrates macro-to-micro visual reasoning with local dynamic mapping navigation. This training-free approach enables generic VLN models to adhere to rules, maximally increasing Task Completion (TC) by up to 5.97% and reducing Constraint Violation Rates (CVR) by 19.26%.

Our contributions are summarized as follows: (1) We propose Rule-VLN, the first large-scale benchmark for rule-based urban navigation, presenting a rigorous challenge through high-quality semantic injection and dynamic graph modification. (2) We introduce SNRM, a universal, zero-shot module that equips pre-trained agents with rule compliance by effectively bridging visual perception with topological planning. (3) Extensive experiments validate the significant difficulties posed by Rule-VLN and demonstrate that SNRM effectively restores navigation capabilities and safety in constrained environments.

2 Related Work↩︎

2.1 Vision-Language Navigation (VLN)↩︎

VLN requires agents to ground natural language into long-horizon actions. In outdoor settings, benchmarks like Touchdown [4] and map2seq [18] introduce complex graph-structured environments. Recently, foundation models have reshaped the field. NaviLLM [20] introduces schema-based instruction tuning to enhance human-like reasoning, while MapGPT [7] employs map-guided prompting for adaptive global planning. NavGPT-2 [21] enables large language models to have visual navigation capabilities. VELMA [22] verbalizes observations for reasoning, VLN-Video [23] exploits temporal video cues, and FLAME [24] leverages synthetic data for MLLM adaptation. NavAgent [25] performs navigation tasks by integrating multi-scale environmental contexts, while MMCNav [26] accomplishes its goals via a multi-agent collaborative framework. Despite these advances, general VLMs prioritize navigational efficiency over constraint adherence, creating a granularity gap by favoring salient geometry over subtle rule signals. While retraining [27] can address this, it requires high training costs and lowers overall navigation performance. Consequently, developing a plug-and-play module has become crucial.

2.2 Rule-Compliant and Safe Navigation↩︎

As VLN advances toward real-world deployment, safety has transitioned from an implicit metric to an explicit objective. Existing approaches primarily address safety through two lenses: physical traversability and social compliance. Methods like Safe-VLN [28] and others [29], [30] focus on collision avoidance by predicting navigable areas, while VLM-Social-Nav [31] incorporates social norms by treating them as soft cost scores during planning. In terms of constraint modeling, recent works like CA-Nav [32] and GC-VLN [33] decompose instructions into subgoals to guide heuristic search. However, a critical limitation remains: these systems typically model constraints as optimization objectives (soft rewards) rather than strict prohibitions. They struggle to handle semantic safety scenarios where a path is geometrically traversable but logically forbidden (e.g., “No Entry”). To address this, our SNRM framework introduces an Epistemic Mental Map that treats semantic rules as hard constraints. Unlike soft-penalty approaches, SNRM enables zero-shot, plug-and-play topological rectification, strictly enforcing rule adherence without altering the navigation backbone.

2.3 Data Synthesis for Embodied AI↩︎

To address the high annotation costs and long-tail disturbances in embodied AI, mechanism-oriented data synthesis has become a pivotal strategy. In VLN, specialized benchmarks like R2R-UNO [34], VLN-ChEnv [35], and RAM [36] target specific failure modes ranging from path obstructions to environmental changes. General-purpose 2D editing has progressed with AnyDoor [37] and SmartEdit [38]. However, applying generic synthesizers to Rule-VLN presents two distinct challenges. First, standard models operating in perspective space typically ignore the equirectangular distortion inherent in panoramic views. Second, they often fail to generate verifiable regulatory signals. This lack of reliability stems from weak spatial-relation consistency [39], [40], severe text unreadability [41], [42], and attribute-binding violations [43]. Our MPSI pipeline addresses these limitations via a dual-mask conditioning strategy with explicit panoramic projection, ensuring injected rules are geometrically rectified and semantically legible.

3 Rule-VLN Benchmark↩︎

3.1 Task Formulation↩︎

In Rule-VLN, the agent is tasked with navigating a discrete environment \(G = (V, E_{\text{geo}})\) following a natural language instruction \(X = (x_1, \dots, x_L)\), where \(V\) denotes navigable nodes and \(E_{\text{geo}}\) represents intrinsic geometric connectivity. At each step \(t\), the agent at node \(v_t\) perceives a panoramic observation \(O_t = \{ o_{t,k} \}_{k=1}^K\), consisting of \(K\) discrete visual slices. The navigation policy selects a neighbor \(v_{next} \in \mathcal{N}(v_t)\) by identifying the slice \(o_{t,k}\) that visually aligns with the target direction, governed by a geometric projection mapping \(k = \Pi(v_t, v_{next})\).

Semantic Connectivity vs. Geometric Connectivity. Standard VLN assumes an identity mapping between geometry and traversability, i.e., any \(e \in E_{\text{geo}}\) is navigable. Rule-VLN challenges this by introducing a Dynamic Semantic Constraint. We posit that edge traversability is conditional on rule-compliance, modeled by a binary validity mask \(\mathcal{M}\): \[\mathcal{M}(v_t, v_{next} \mid X, \mathcal{R}) = \mathbb{I}\left[ \mathcal{C}(o_{t, \Pi(v_t, v_{next})} \mid \mathcal{R}) = 1 \right]\] where \(\mathbb{I}[\cdot]\) is the indicator function and \(\mathcal{C}\) evaluates whether the visual slice corresponding to the edge \(e=(v_t, v_{next})\) contains regulatory prohibitions (e.g., “Do not enter” signs) defined in the rule set \(\mathcal{R}\).

Consequently, the agent operates on a Semantically Pruned Graph \(G' = (V, E_{\text{sem}})\), where \(E_{\text{sem}} = \{ e \in E_{\text{geo}} \mid \mathcal{M}(e) = 1 \}\). The objective is to find a trajectory \(\tau = \langle v_0, \dots, v_n \rangle\) that reaches the target defined by \(X\), subject to the strict constraint that \(\tau \subseteq E_{\text{sem}}\). When the intended geometric path is obstructed by \(\mathcal{M}(e)=0\), the agent must suppress geometric shortest-path heuristics and infer a latent feasible detour compliant with \(\mathcal{R}\).

3.2 City Navigation Rules Dataset↩︎

To bridge the perception-compliance gap, we introduce CityNav-Rules-73K (Figure 2a), the first large-scale rule-semantic dataset explicitly coupling visual signals with fine-grained actionable constraints. Comprising 73,937 samples across 177 categories from different regions around the world, the dataset features two core innovations designed for navigation logic rather than mere classification:

LLM-Driven Discrete Action Mapping. To translate abstract rules into rigorous control constraints, we employ GPT-5 to map each visual category to a Permissible Action Subspace \(\mathcal{A}_{valid}\) derived from the global discrete action set (Straight, Left, Right, U-turn). This process converts semantic classifications into precise geometric lookup tables (e.g., “no-right-turn” yields \(\mathcal{A}_{valid} = \{ \text{Straight, Left, U-turn} \}\)), directly linking perception to graph traversal.

Visual-Semantic Hybrid Representation. We construct a Hybrid Representation interleaving Visual Descriptors (e.g., “Red Circle”) and Semantic Imperatives (e.g., “No Entry”). This challenges agents to decouple appearance from intent—for instance, learning that a “Red Circle” implies prohibition while a “Blue Circle” implies mandatory action. This design penalizes superficial pattern matching and enforces robust causal links between visual cues and geometric navigability.

3.3 Touchdown-Semantic Constraint Benchmark Construction↩︎

To strictly evaluate rule compliance, we construct Rule-VLN by injecting semantic constraints into the Touchdown graph (Fig. 2b). We identify strategic nodes via a criticality score aggregating four metrics: (1) Degree Centrality [44] for local intersection complexity; (2) Betweenness Centrality [45] for global bottlenecks; (3) Path Dependence [46] to assess detour robustness; and (4) Path Frequency [47] to prevent shortcut memorization. Based on this score, we filter impactful nodes to establish four curriculum difficulty levels, injected via MPSI (Sec. 4.1). As shown in Tables 1 and ¿tbl:tab:rule95vln95stats?, constraints escalate from 31.44% (Level-1) to 91.13% (Level-4). Unlike limited indoor or synthetic datasets, Rule-VLN provides a real-world urban environment with 177 explicit rule types, of which a subset is injected into the node network following a 60% action-impacting rule distribution, enforcing compliant detours through progressive difficulty levels rather than simple shortest-path heuristics.

4pt

4 Semantic Injection and Navigation Rectification Module↩︎

4.1 Mask-Prioritized Semantic Injection (MPSI)↩︎

To mitigate diffusion hallucinations, we propose MPSI (Fig. 3), a pipeline that injects node-action-compliant signs into panoramas.

Spatial Grounding and Rule Decoupling. We decouple regulatory signals into geometric shape \(S\) and semantic rule \(R\). Given a target node \(v\) with permissible action subspace \(\mathcal{A}_{valid}(v)\), we retrieve a semantically aligned instance \((S, R) \sim \mathcal{D}_{insert}\). To ensure global quality, we apply a cropping operator \(\mathcal{C}\) to the panoramic observation \(O \in \mathbb{R}^{H \times W \times 3}\), yielding \(O_{crop} = \mathcal{C}(O) \in \mathbb{R}^{H_c \times W_c \times 3}\). Simultaneously, we employ a segmentation prior \(\mathcal{F}_{seg}\) to extract a binary road mask \(M_{road} = \mathcal{F}_{seg}(O_{crop}) \in \{0, 1\}^{H_c \times W_c}\), physically anchoring the insertion region for \(S\) before semantic injection to avoid the position offset.

Mask-Guided DiT Synthesis. Inspired by Insert Anything [50], we employ a mask prompt Diffusion Transformer (DiT) [51] \(\mathcal{G}_{\theta}\) with dual-mask conditioning to prevent feature bleeding. First, the shape prior \(S\) is projected into \(O_{crop}\) via \(M_{road}\) to form \(V_S\), which is optically rectified into \(V'_S\) to yield a precise boundary mask \(M_S\). Unlike text-driven generation, we utilize a reference rule image \(I_{ref}\) for visual-semantic guidance. We concatenate the binary masks (\(M_S\) and rule-specific \(M_R\)) and spatial features \(V'_S\) with the noisy latent \(z_t\), while injecting semantic features \(\mathcal{E}_{ref} = \text{CLIP}_{\text{img}}(I_{ref})\): \[\hat{O}_{crop} = \mathcal{G}_{\theta}\left(z_t \oplus V'_S \oplus M_S \oplus M_R, t \mid \mathcal{E}_{ref}\right)\] where \(\oplus\) denotes channel-wise concatenation and \(t\) is the diffusion timestep. This strictly confines the latent trajectory within explicit boundaries, effectively eliminating semantic hallucinations.

GMM-Based Quality Filtering and Stitching. To ensure semantic legibility, we evaluate the CLIP alignment \(s_{align} = \text{cos}(E_{\text{img}}(\hat{O}_{crop}), E_{\text{text}}(\mathcal{P}_{text}))\) between the synthesized crop and the rule category \(\mathcal{P}_{text}\). Instead of rigid heuristics, we model the distribution of \(s_{align}\) using a bimodal Gaussian Mixture Model (GMM) (\(K=2\)) to adaptively prune low-confidence outliers: \[p(s_{align}) = \sum_{k=1}^2 \pi_k \mathcal{N}(s_{align} \mid \mu_k, \sigma_k^2)\] Candidates belonging to the high-confidence mode are accepted and seamlessly blended back into the original panorama via a stitching operator \(\hat{O} = \mathcal{S}(\hat{O}_{crop}, O)\), ensuring full-scale visual fidelity and geometric consistency.

4.2 Semantic Navigation Rectification Module (SNRM)↩︎

To bridge the granularity gap, we propose SNRM, a plug-and-play, model-agnostic module that acts as a semantic reflection module for any frozen VLN policy \(\pi_{base}\). SNRM decomposes compliance into a dual-stage perception framework (Fig. 4).

Dual-Stage Coarse-to-Fine Perception Framework. To reliably extract subtle regulatory cues from complex observations, SNRM employs a coarse-to-fine pipeline (Figure 4 a). It can be divided into two main stages. Stage 1: Macro-Micro Visual Prompting. Motivated by visual prompting [52], we design a human-like attention method that follows a “global-to-local” perception paradigm. To mitigate computational costs, a lightweight detector (e.g., DINO) first scans the panorama \(O_t\) for potential cues using generic prompts \(P_{detect}\). Upon detection, we generate a Macro-Micro Visual Prompt: a global view \(V_{macro}\) with highlighted bounding boxes to preserve topological context, and a magnified crop \(V_{micro}\) to enhance fine-grained symbol resolution. By incorporating explicit visual prompts, this approach enhances the model’s perception of fine-grained rule signals while maintaining robust global awareness. This ensures that the generated actions are both rule-compliant and consistent with environmental connectivity. Stage 2: Knowledge-Driven Rule Grounding (KDRG). To prevent open-ended hallucinations, we anchor VLM inference to a predefined regulatory knowledge base \(\mathcal{K}\), a lightweight textual rule-name bank containing the 177 normalized CityNav-Rules categories. We employ SigLIP to compute the zero-shot similarity between \(V_{micro}\) and rule descriptions in \(\mathcal{K}\), retrieving the top-ranked text prior \(T_{rule} = \arg\max_{t \in \mathcal{K}} \text{Sim}(V_{micro}, t)\). Finally, the tuple \((V_{macro}, V_{micro}, T_{rule})\) and the intended action \(a_{base} \sim \pi_{base}\) are fed into a VLM (e.g., Qwen-3VL). Through structured Chain-of-Thought (CoT) reasoning, the VLM outputs a safety token \(s \in \{\text{Safe, <Correct Action>}\}\), strictly gating the execution of \(a_{base}\).

Epistemic Mental Map and Trajectory Correction. Upon detecting a conflict, SNRM overrides the base policy via a dynamic 2D virtual topology (Fig. 4 b-c). Defining the conflict origin as \(P_{start}=(0,0)\), we anchor the intended target at \(P_{target\_v} = (\sin(\Delta \theta_{exp}), \cos(\Delta \theta_{exp}))\) based on the expected heading \(\Delta \theta_{exp}\). As the agent executes forced compliant actions to a deviation point \(P_{dev}\) (updated via dead reckoning), SNRM replans by evaluating candidates \(\mathcal{C} = \{C_1, \dots, C_n\}\) through a penalized greedy heuristic: \[C_{best} = \arg\min_{C_i \in \mathcal{C}} \Big( \| P(C_i) - P_{target\_v} \|_2 + \lambda \cdot \mathbb{I}_{backtrack}(P(C_i)) \Big)\] where \(P(C_i)\) is the predicted coordinate, and \(\mathbb{I}_{backtrack}\) applies a severe penalty \(\lambda\) if \(P(C_i)\) falls within a critical radius of visited nodes. To further robustify against spatial drift, we employ a visual memory buffer: if the cosine similarity between current and start node features \(\text{cos}(f_{curr}, f_{start}) > \tau_{sim}\), a closed-loop trap is detected, triggering a forced detour correction.

5 Experiment↩︎

5.1 Experimental Setup↩︎

Evaluation Metrics. We evaluate navigation via Task Completion (TC) [4], Shortest-Path Distance (SPD) [4] and Success weighted by Path Length (SPL) [53]. To rigorously quantify rule adherence, we introduce the Constraint Violation Rate (CVR), which normalizes the number of violations \(V_i\) by the agent’s actual exposure to active constraints (\(\mathcal{R}_{neighbor}\)): \[\text{CVR} = \frac{1}{N} \sum_{i=1}^{N} \frac{V_i}{\sum_{t=1}^{L_i} \mathbb{I}(\text{Node}_t \in \mathcal{R}_{neighbor})}\] where \(L_i\) is path length and the indicator function \(\mathbb{I}[\cdot]\) identifies steps with proximal regulatory signals. Synthesis quality is measured via PSNR [54], SSIM [54], LPIPS [55], and FID [56].

Implementation Details. We benchmark against ORAR [57], Loc4plan [58], VELMA [22], and FLAME [24] using official hyperparameters. To ensure rule visibility, we retrained models that take partial views as input, such as FLAME, by turning the FOV from \(60^{\circ}\) to \(120^{\circ}\) and fine-tuning. The MPSI was performed on a single NVIDIA RTX 4090 GPU. Training model is conducted on 4\(\times\) NVIDIA H100 GPUs, while inference and SNRM deployment utilize a single H100. Notably, our proposed SNRM is training-free.

5.2 Evaluating the Robustness of Current VLN Methods in Rule-VLN↩︎

Analysis of Evaluation Results. The algorithmic injection of regulatory symbols establishes a rigorous four-level curriculum (Table ¿tbl:tab:rule95vln95stats?). To establish baselines, we evaluate state-of-the-art (SOTA) VLN architectures. As illustrated in Fig. 5, the introduction of explicit semantic constraints induces a catastrophic performance collapse across all paradigms, compared to an environment without constraints (Level 0). Even recent foundation models like FLAME exhibit a sharp degradation in Task Completion (TC) alongside alarmingly high Constraint Violation Rates (CVR, often exceeding 30-40%).

This phenomenon highlights a severe granularity gap: current models overfit to idealized topologies and remain entirely blind to fine-grained semantic imperatives (e.g., driving into “No Entry” zones). These baseline results validate Rule-VLN as a necessary testbed and forcefully underscore the need for our proposed SNRM module, which we analyze quantitatively in Sec. 5.3.

Trajectory Visualization Analysis. Figure 6 visualizes a scenario where a prohibitory sign obstructs the path at \(t=4\). While baselines (Loc4plan, FLAME) exhibit semantic blindness by executing illegal forward actions, SNRM successfully grounds the visual cue and triggers a preemptive U-turn. Leveraging its epistemic mental map, the agent executes a legal detour to bypass the restricted zone, ultimately realigning with the global instruction (“the first left”) at \(t=33\). This visually corroborates SNRM’s ability to transform blind geometric traversal into robust, rule-compliant navigation.

5.3 Evaluation of the SNRM↩︎

Table 2 demonstrates SNRM’s zero-shot integration with Loc4plan and FLAME across four difficulty levels. Table 3 compares the real-time inference performance after incorporating the SNRM module.

Universal Plug-and-Play Efficacy. SNRM bridges the granularity gap across diverse architectures, yielding significant gains for both traditional (Loc4plan) and MLLM-based (FLAME) agents. For Loc4plan (Level-2), SNRM not only reduces CVR by 8.70% but also improves TC by 1.53%, proving its effectiveness even without heavy reasoning priors. This impact is amplified in FLAME (Level-3), where SNRM slashes CVR by 19.26% while boosting TC by 5.97%, validating its capability to transform blind traversal into semantically constrained navigation regardless of the backbone. Although SNRM inevitably leads to more detours during navigation, it still improves SPL in most experimental settings.

Robustness and Compliance. In Level-1, despite identical TC (24.36%), SNRM improves efficiency (SPD: 16.58 vs. 17.12) and strictly enforces compliance (CVR: 12.47% vs. 21.17%). As constraints intensify in Level-3, the baseline collapses (TC 11.43%, CVR 41.79%), whereas SNRM exhibits superior robustness, recovering TC to 17.40% while suppressing violations. This advantage persists in Level-4, validating SNRM as a reliable safety envelope for complex navigation.

Inference latency and navigation performance. As shown in Table 3, incorporating the VLM introduces additional inference latency, but the event-triggered design of SNRM limits this overhead by relying on the base navigation model in most cases. For SNRM-8B, the trigger rate is only about 26%; taking FLAME as an example, the average per-step latency increases from 0.095 s to 1.158 s, while the CVR decreases by 18.25 percentage points. To further improve practical efficiency, we design a 4B SNRM variant without CoT, which adds only 0.224 s latency per step while still reducing CVR by 10.55 percentage points.

3.5pt

5.4 Ablation experiments and analysis of SNRM↩︎

To dissect the contribution of each module within SNRM, we conduct a comprehensive ablation study on the most challenging Level-4 configuration (Table [tbl:tab:ablation95SNRM]). The baseline setup (#1), devoid of SNRM’s components, exhibits poor navigation success (TC: 8.95) and a high rule violation rate (CVR: 33.89%), acting as a naive geometric navigator.

Impact of the Dual-Stage Perception Framework. The removal of either Macro-Micro Visual Prompting(MMVP) (Setup #3) or Knowledge-Driven Rule Grounding(KDRG) (Setup #2) explicitly degrades performance. Without MMVP (#3), the VLM struggles to simultaneously focus on subtle regulatory symbols and the global topological context, leading to an increased CVR (26.86%). Similarly, removing the KDRG module (#2) deprives the VLM of deterministic text priors, causing it to occasionally hallucinate or misclassify long-tail signs, which marginally drops the TC. Setup #5 demonstrates that these two perception modules act synergistically to reliably translate raw visual cues into actionable semantic constraints.

Impact of Topological Recovery. Setup #4 (w/o Mental Map) achieves the lowest CVR (21.28%) but compromises TC (6.53%), as the agent stops at constraints without a path-generation mechanism. Conversely, the full framework (#5) improves TC to 9.94% and SPD to 25.61, demonstrating that the Epistemic Mental Map is essential for translating static rule recognition into active navigation and detour computing.

Impact of VLM Backbones. We instantiate SNRM with Qwen3-Vl [59], InternVL-3.5 [60], LLaVA-1.6 [61], and Pixtral [62] to explore the influence of different VLM models on the perception of internal rules. Intra-family scaling (Qwen-VL) confirms that parameter capacity enhances spatial CoT execution. Qwen-VL and InternVL achieve the optimal balance via robust grounding. In contrast, LLaVA-1.6 is overly conservative; its low CVR (21.17%) leads to excessive detouring and navigation failure (low TC). Finally, Pixtral 12B’s underperformance against 8B models confirms that for embodied tasks, visual-spatial grounding is more critical than parameter scale.

5.5 High-Fidelity Semantic Injection via MPSI↩︎

Visual Realism and Fidelity. As shown in Table [tbl:tab:image95quality], MPSI significantly outperforms the FLUX.1-Fill [63] baseline. High pixel-level fidelity (SSIM 0.9280, PSNR 32.41) confirms that our dual-mask conditioning strictly confines the latent space and prevents artifact bleeding into the native panorama. Furthermore, MPSI drastically improves perceptual realism (FID 7.72, LPIPS 0.0503), effectively mitigating the geometric distortions typical of generic diffusion models in localized synthesis. Due to the masking design, MPSI achieves a 58.81% reduction in GPU memory footprint compared to FLUX.1-Fill.

Semantic Alignment and Filtering. To ensure the injected signs serve as actionable constraints, we evaluate cross-modal semantic legibility via CLIP scores (Fig. 7). MPSI achieves a higher overall mean (16.42 vs. 15.20). Crucially, the raincloud plot (Fig. 7a) reveals that while FLUX exhibits a long tail of low-scoring outliers (indicating hallucinations), MPSI’s distribution is sharply truncated at the lower end. This explicitly validates our GMM-Based Quality Filtering in actively pruning corrupted generations. Moreover, MPSI demonstrates consistent superiority across diverse rule categories (Fig. 7b), guaranteeing unambiguous visual cues for downstream tasks.

Qualitative Comparison. Fig. 8 compares MPSI, FLUX.1-Fill, and Google Nano Banana 2. FLUX suffers from severe semantic hallucinations, and Nano Banana 2 yields spatially misplaced injections due to a lack of positional conditioning. MPSI consistently achieves superior visual fidelity and precise geometric grounding.

6 Limitations and Future Work↩︎

Despite its effectiveness, our framework has several limitations. On the data synthesis side, although MPSI injects rules under dual-mask constraints, the quality of the synthesized data remains affected by the complexity of the environment and the quality of rule-related images. Moreover, following the discrete-node task formulation of Touchdown may lead to viewpoint-consistency issues when the same rule is observed from adjacent nodes. For SNRM, we design its components, such as QwenVL and SigLIP, in a plug-and-play manner to enhance generalization to diverse real-world regulations. This enables SNRM to benefit from advances in foundation models without retraining the entire module, but it also introduces additional inference latency. Future work will explore stronger generative vision models and 3D geometric constraints to narrow the gap between synthesized and real-world scenes. We will also investigate model distillation, feature caching, and quantization to reduce computational overhead while preserving generalization.

7 Conclusion↩︎

In this work, we introduce semantic rule constraints into Vision-and-Language Navigation (VLN) to address the issue where existing agents prioritize geometric accessibility over social compliance in real-world environments. We propose Rule-VLN, the first benchmark incorporating such rule compliance challenges into VLN. This is achieved via a novel Mask-Prioritized Semantic Injection (MPSI) pipeline that precisely integrates fine-grained traffic rule signals into urban topology and visual observations. Leveraging Rule-VLN, we demonstrate the limitations of current VLN methods and further introduce the Semantic Navigation Rectification Module (SNRM). This module enables agents to effectively adapt to rule-constrained environments through macro-micro visual prompting and an epistemic mental map method. Our approach achieves state-of-the-art results on Rule-VLN without requiring alterations to the original model architectures. We believe that addressing the shift from physical accessibility to social compliance is critical for the practical deployment of VLN agents and for evaluating their capacity for safe navigation under complex rule constraints.

Acknowledgements↩︎

We thank the reviewers for their valuable comments and constructive suggestions. We also gratefully acknowledge the computing resources provided by The Hong Kong University of Science and Technology (Guangzhou) through the Red Bird MPhil Program.

References↩︎