1 Introduction↩︎

Vision-and-Language Navigation (VLN) represents a fundamental capability for developing autonomous embodied agents, requiring them navigate physical environments by following natural language instructions. This task poses significant challenges in modeling cross-modal alignment between textual and visual cues, processing temporal visual history, and inferring future actions. To address these issues, recent studies [1]–[5] have adapted Vision-Language Models (VLM) into navigation decision-making, leveraging their rich inherent priors in language understanding and visual perception.

Prior research [2]–[5] typically leverages imitation learning to train agents on navigation data from diverse sources. Despite substantial advancements, this paradigm suffers from covariate shift, where cumulative errors arise as the agent encounters out-of-distribution states not seen during training, thereby incurring poor generalization. To mitigate this, the Dataset Aggregation (DAgger) algorithm [6] has been applied as an effective IL strategy to augment the training set with oracle-labeled off-policy trajectories. Nevertheless, they remain constrained by a heavy reliance on expert demonstrations, limiting their generalization to unseen environments.

Inspired by the success of Reinforcement Learning from Verifiable Rewards (RLVR) [7]–[10], several studies focus on applying reinforcement techniques, such as GRPO [7], following the IL phase. For instance, VLN-R1 [11] and OctoNav [12] employ GRPO training by leveraging existing trajectories. Although effective in improving action prediction, the use of in-distribution data fails to adequately address covariate shift during inference. This naturally motivates us to integrate the strengths of on-policy GRPO and off-policy DAgger training.

To this end, we introduce JOP-VLN, a novel VLN model that leverages a joint on-and-off policy learning paradigm via three progressive training stages. In the initial stage, the model is trained on diverse tasks, including navigation data and trajectory summarization, to acquire fundamental navigation skills. Subsequently, the model is fine-tuned using off-policy trajectories collected by the learned policy via the DAgger algorithm to improve its error-recovery capabilities. To further enhance generalization, the final stage integrates off-policy imitation learning with on-policy reinforcement learning, drawing inspiration from CHORD [13]. This involves jointly optimizing the on-policy GRPO objective and the IL objective on off-policy trajectories collected via the DAgger algorithm. Our method distinguishes itself from CHORD [13] through several VLN-specific techniques. First, we introduce a trajectory sampling strategy to exclude low-entropy trajectories, preventing the model from becoming overconfident and degrading model performance. Second, a trajectory sorting strategy is employed to prioritize error-prone samples, allowing the model to focus on its frequent failures and facilitating a seamless transition from IL to RL.

Extensive experiments demonstrate the superiority of JOP-VLN, which achieves a new state-of-the-art on the Val-Unseen split of R2R [14] at 69.9% SR. It also obtains 68.0% SR and 59.3% SPL on RxR [15] Val-Unseen split, demonstrating its excellent long-horizon decision-making capability. Additionally, we conduct real-world experiments in challenging indoor and outdoor scenarios under a few-shot adaptation setting, and the results validate the effectiveness of our proposed method in real-world deployment.

In summary, our contributions are as follows:

• We introduce JOP-VLN, a novel VLN framework that leverages a joint on-and-off policy learning paradigm via three progressive stages.

• We devise two techniques to adapt the CHORD framework to the VLN setting, i.e., a trajectory sampling strategy that retains high-entropy trajectories and a trajectory sorting strategy that prioritizes error-prone samples.

• We conduct extensive experiments in both simulation and real-world environments. Our method achieves a new state-of-the-art of 69.9% SR on the R2R Val-Unseen split, demonstrating the superiority of our training framework.

2 Related Work↩︎

2.1 Vision-and-Language Navigation (VLN)↩︎

As a fundamental task in embodied AI, VLN [16]–[20] has spurred extensive interest into various learning techniques, ranging from multi-task learning [1], [2], [4], [5] and curriculum learning [21], [22] to reinforcement learning [11], [17], [23]–[26]. Our work is closely related to studies of reinforcement learning for VLN. Early efforts focus on designing heuristic rewards, including arrival signal [17], instruction-landmark alignment [23], and navigation progress [24]. More recently, there has been an incresing emphasis on leveraging RL to optimize Vision-Language Models (VLMs) for VLN. Specifically, VLN-R1 [11] and OctoNav [12] incorporates Group Relative Policy Optimization (GRPO) by utilizing a matching score between the ground truth and prediction as the reward signal to bolster navigation performance. ActiveVLN [25] extends the GRPO process to multi-turn active exploration. In contrast with these work, JOP-VLN structurally fuses on-policy RL exploration with off-policy IL supervision. This allows the agent to learn simultaneously from expert demonstrations and environmental exploration signals.

2.2 Navigation with Vision-Language Models↩︎

Motivated by the remarkable success of Vision-Language Models (VLMs), recent research [1]–[5], [27]–[29] has adapted VLMs to embodied navigation for improved generalization to unseen scenarios. Pioneering works such as NaVid [3] and Uni-NaVid [4] leverage multi-task learning across diverse data sources to enhance VLM-based navigation models. Subsequently, NaVILA [2] introduces a heuristic pipeline to automatically annotate human videos, facilitating effective sim-to-real transfer. Furthermore, StreamVLN [5] and CorrectNav [30] leverage the pre-collected trajectories for enhancing the error-recovery capabilities during the navigation process. Despite these advancements, imitation learning from expert demonstrations may restrict generalization capabilities to mere memorization, thus limiting robustness. Incorporating on-policy RL within the off-policy IL process aids JOP-VLN in enhancing exploration while mitigating the risk of overfitting on off-policy expert data.

3 Framework↩︎

3.1 Task Formulation↩︎

The Vision-and-Language Navigation (VLN) task requires an agent to navigate to a target location based on a language instruction \(I\). Specifically, at each step \(t\), the agent first perceives the current visual observation \(o_t\), and then predicts the next action chunk \(a_t\) based on the entire observation and action histroy \(\{o_{1:t},a_{1:t-1}\}\), denoted by \(a_t \sim \pi_{\theta}(\cdot \vert I, \{o_{1:t},a_{1:t-1}\})\). Following previous work [5], each chunk consists of four consecutive actions within a discrete space encompassing forward, left, right, and stop.

3.2 Model Architecture↩︎

Our model is built upon a large vision-language model, i.e., Qwen3-VL [31]. As shown in Figure 1, our model adopts a multi-turn dialogue paradigm following previous work [5], where the model processes the language instruction \(I\), a long-term memory of \(n\) frames, and a short-term memory of \(m\) frames. Specifically, at step \(t\), the long-term memory consists of frames uniformly sampled from the distant history \(\{o_{1:t-k-1}\}\), and the short-term memory maintains the most recent interleaved sequences \(\{o_{t-k:t}, a_{t-k:t-1}\}\), where \(k=t\%m\). Upon generating an action chunk, the model receives a new observation and appends both to the current sequence. This design facilitates efficient KV cache reuse, enabling the sequential prediction of action chunks within a near-linear inference budget. Upon the short-term buffer reaching its capacity \(m\), the long-term memory is updated via uniform resampling, while the short-term memory is flushed and reset to the current observation.

4 On-and-Off Policy Training↩︎

We devise an on-and-off policy training paradigm consisting of three progressive stages. In the initial stage, the model is trained in a multi-task manner to acquire the fundamental navigation skills. Then, we generate DAgger trajectories using a heuristic strategy, which are subsequently used for imitation training, thereby enhancing its error-recovery capabilities. In the final stage, we perform on-and-off policy training following the CHORD [13] framework. This involves on-policy GRPO training and off-policy imitation learning on the DAgger trajectories.

4.1 Stage 1. Multi-task Training↩︎

As shown in Figure 2, in this stage, we co-train our model on the tasks of action prediction and trajectory summarization. Specifically, for action prediction, the model is tasked with predicting action chunks within the multi-turn dialogue paradigm detailed in Section 3.2. For trajectory summarization, the model is required to synthesize textual instructions conditioned on sampled frames from navigation videos.

4.2 Stage 2. Imitation Learning with DAgger Trajectories.↩︎

The Dataset Aggregation (DAgger) algorithm [6] is widely adopted for generating trajectories through a mixed policy that combines learned and oracle strategies. The implementation of StreamVLN [5] adapts the original framework by introducing a heuristic strategy. Specifically, this strategy prioritizes the learned policy and only triggers oracle intervention when the agent’s deviation from the ground-truth trajectory exceeds a predefined threshold. StreamVLN’s action labels may contain erroneous actions generated by the model, which can hinder the fine-tuning process. To mitigate this issue, following [32], we replay all visited states to obtain their corresponding oracle action chunks, which are subsequently utilized for imitation learning as shown in Figure 3.

4.3 Stage 3. Joint On-and-Off Policy Learning with High-entropy and Error-correction-prioritised Trajectories↩︎

The DAgger algorithm is reapplied, during which the entropies of the model-predicted tokens are recorded. The CHORD [13] framework is employed on these DAgger collected trajectories with high-entropy trajectory sampling for GRPO and error-correction-prioritised trajectory sorting for imitation learning, as shown in Fiture 4.

4.3.1 On-policy exploration with GRPO on high-entropy trajectories↩︎

Given the input comprising language instructions, long-term and short-term memory, the policy \(\pi_{\theta}\) generates \(G\) responses \(\{a_1^*, a_2^*, \ldots, a_G^*\}\) for each sample. The reward for each response is calculated using the following four reward functions.

Format reward \(R_{format}\) checks whether the response adheres to the specified format, i.e., containing at most four consecutive discrete actions. \[ R_{format} = \begin{cases} 1, & \text{if the response adheres to the specified format} \\ 0, & \text{otherwise} \end{cases}\]

Exact match reward \(R_{em}\) evaluates whether the response \(a^*\) precisely matches the oracle action chunk \(a\). \[ R_{em} = \begin{cases} 1, & \text{if } a^* = a \\ 0, & \text{otherwise} \end{cases}\]

Position reward \(R_{pos}\) assesses the cosine similarity between \(pos^*\) and \(pos\), where \(pos^* = (\triangle x^*, \triangle y^*, \triangle yaw^*)\) and \(pos = (\triangle x, \triangle y, \triangle yaw)\) denote the change in the robot’s pose after executing the predicted action chunk \(a^*\) and the oracle action chunk \(a\), respectively. \[R_{pos} = \frac{pos^\top pos^*}{\lVert pos \rVert_2 \, \lVert pos^* \rVert_2}\]

Distance reward \(R_{dis}\) quantifies the Euclidean distance between the predicted displacement \((\triangle x^*, \triangle y^*)\) and the oracle displacement \((\triangle x, \triangle y)\). This distance is then normalized by dividing it by the maximum possible distance, calculated as \(2 \times c \times s\), where \(c\) is action chunk size and \(s\) is the forward step size. \[R_{dis} = \frac{\sqrt{(\triangle x^* - \triangle x)^2 + (\triangle y^* - \triangle y)^2}}{2 \times c \times s}\]

The advantages \(\hat{A}_i\) of rewards \(r = \{r_1, r_2, \ldots, r_G\}\) for the \(G\) responses sampled from the current policy \(\pi_{\theta}\) is calculated as: \[\hat{A}_i = \frac{r_i - \text{mean}(\{r_i\}_{i=1}^{G})}{\text{std}(\{r_i\}_{i=1}^{G})}\]

The policy is optimized via the GRPO objective: \[{!}{ \begin{align} & L_{\mathrm{GRPO}}(\theta) = \mathbb{E}_{i=1}^G \Biggl[ \min \Biggl( \frac{\pi_\theta(a_i^* \mid I)}{\pi_{\theta_{\mathrm{old}}}(a_i^* \mid I)} \hat{A}_i, \\ & \operatorname{clip}\!\left( \frac{\pi_\theta(a_i^* \mid I)}{\pi_{\theta_{\mathrm{old}}}(a_i^* \mid I)}, 1 - \epsilon,\; 1 + \epsilon \right) \hat{A}_i \Biggr) - \beta \cdot D_{\mathrm{KL}}\!\left(\pi_\theta \,\|\, \pi_{\text{ref}}\right) \Biggr] \end{align} } \label{eq:grpo95loss}\tag{1}\]

where \(I\) is the input to the model, \(a_i^*\) is the \(i\)-th response, and \(\pi_{\text{ref}}\) is the reference model. In our implementation, the token-level policy gradient loss is applied for enhancing training stability [8].

High-entropy trajectory sampling. Previous studies [33], [34] have identified the critical role of high-entropy minority tokens in reinforcement learning training. Remarkably, training with only 20% of these high-entropy tokens can achieve performance that is comparable to or even surpasses that obtained with 100% of the tokens. Inspired by these findings, we adopt a high-entropy trajectory sampling strategy. During the DAgger trajectory collection process, we record the entropy of each model-predicted action token and conduct GRPO training using only the top 20% of samples with the highest entropy values. Employing the high-entropy trajectory sampling strategy fosters exploration and prevents the model from entering a “zero-gradient" state caused by low-entropy samples, where the advantage collapses to zero, eliminating training signals and rendering rollouts uninformative.

4.3.2 Off-policy imitation learning↩︎

Let \(\mathcal{D}=\{(I,a)\}\) denote a corpus of expert demonstrations collected using the DAgger algorithm, where \(a\) denotes the oracle actions corresponding to the input \(I\), organized within the multi-turn dialogue paradigm as described in Section 3.2. Imitation learning aims to minimize the sentence-level cross-entropy:

\[ \mathcal{L}_{\mathrm{IL}}(\theta) \;=\; \mathbb{E}_{(I, a)\sim\mathcal{D}} \bigl[-\log \pi_\theta\bigl(a \mid I\bigr)\bigr]. \label{eq:il95loss}\tag{2}\]

4.3.3 Joint on-and-off policy learning with CHORD framework prioritizing error-correction trajectories↩︎

Following CHORD [13], imitation learning is integrated as a dynamically weighted auxiliary objective within the on-policy reinforcement learning process. This is achieved through a combined loss function that minimizes a weighted sum of both RL and IL losses:

\[\mathcal{L}_{\text{CHORD}}(\theta) = (1-\mu) \mathcal{L}_{\text{GRPO}}(\theta) + \mu \mathcal{L}_{\text{IL}}(\theta) \label{eq:chord95loss}\tag{3}\]

where \(\mathcal{L}_{\text{GRPO}}(\theta)\) is the GRPO loss defined in 1 , \(\mathcal{L}_{\text{IL}}(\theta)\) denotes the IL loss defined in 2 . The hyperparameter \(\mu \in [0, 1]\) regulates the trade-off between imitation learning and reinforcement learning.

A decay schedule for \(\mu\) is implemented to facilitate a seamless transition from a balanced imitation-reinforcement combination to on-policy dominated optimization, as illustrated in Figure 5. The training process initiates with \(\mu = 0.5\), assigning equal weight to off-policy imitation and on-policy exploration so that the model can benefit from expert demonstrations while stabilizing the early RL updates. As training advances, \(\mu\) is progressively reduced to \(0.05\), thereby shifting the emphasis towards on-policy exploration while reducing the risk of overfitting on off-policy expert data.

Error-correction-prioritised trajectory sorting. The dynamic scheduling strategy for \(\mu\) prioritizes imitation learning during the early stage of training. To improve the model’s error recovery capabilities, we devise an error-correction-prioritised trajectory sorting strategy. Specifically, for a multi-turn dialogue trajectory sample as described in Section 3.2, we identify which actions within the trajectory are corrected by the oracle while the model made mistakes. The learning value of a trajectory is determined by the proportion of these corrected action tokens. These expert demonstrations are then sorted in descending order of learning value for the model to perform imitation learning. This ensures that the model more frequently learns from trajectories where it previously made mistakes during the DAgger trajectory collection process.

5 Experiment↩︎

5.1 Experimental Setup↩︎

Simulation benchmarks and evaluation metrics. We evaluate JOP-VLN on two widely used VLN-CE [17] benchmarks, including R2R [14] and RxR [15]. Both datasets are built upon Matterport3D [35] scenes within the Habitat simulator [36]. Following prior work [2]–[4], we report standard VLN metrics, including navigation error (NE), success rate (SR), oracle success rate (OS), success weighted by path length (SPL), and normalized dynamic time warping (nDTW). All evaluations are conducted on the Val-Unseen split where the scenes are not seen during training.

Real-World Experiment Configuration. Our real-world experiments are conducted on a Unitree Go2 quadruped robot, equipped with an Intel RealSense D435i RGB-D camera. JOP-VLN operates on a remote workstation utilizing an L40S GPU. During the navigation process, the robot transmits the captured RGB frames to the server over a WiFi connection. Subsequently, the server processes the received data and sends the predicted action commands back to the robot for execution. The system achieves an average inference time of 0.3s per action chunk, with a communication delay of 1.2s. Since each chunk contains up to 4 sequential discrete actions, the per-action execution rate reaches up to 2.67Hz. The communication delay dominates the end-to-end latency and can be further reduced through on-device deployment.

5.2 Implementation Details↩︎

Training Data. During the first stage, we aggregate 793.4k samples for action prediction from R2R [14], RxR [15], EnvDrop [37], and ScaleVLN’s subset [38], and 30.8k video-instruction pairs from R2R and RxR for trajectory summarization. In the second stage, we synthesize 11.3k trajectories based on the R2R and RxR annotations. By combining them with the trajectories from the R2R, RxR, and ScaleVLN’s subset, we obtain 453.3k training samples. In the last stage, we collect an additional 7.4k DAgger trajectories. From this, 33.3k high-entropy samples are employed for GRPO training, while 30.0k samples are used for imitation learning.

Training Details. JOP-VLN is built upon the Qwen3-VL-8B-Instruct [31] backbone and optimized using AdamW. During the first two stages, the peak learning rates for the language model, projector, and vision encoder are set to 2e-5, 1e-5, and 5e-6, respectively. In the third stage, these rates are adjusted to 1e-6, 5e-7, and 5e-8, respectively. For the GRPO objective, we generate eight rollouts per sample, with the KL coefficient \(\beta\) and clipping parameter \(\epsilon\) configured at \(0.1\) and \(0.2\), respectively. The parameter \(\mu\) follows a dynamic schedule, decaying from 0.5 to 0.05 over the first 200 training steps and remaining constant thereafter.

3mm

5.3 Main Results↩︎

5.3.1 Comparisons with state-of-the-arts on VLN-CE benchmarks↩︎

Table 1 presents a comparison between JOP-VLN and state-of-the-art methods on the VLN-CE R2R and RxR Val-Unseen splits. Within the setting of using only single-view RGB observations, JOP-VLN achieves state-of-the-art performance on the R2R benchmark, outperforming the previous state-of-the-art method, CorrectNav [30], by 4.8%, 8.9%, and 2.6% in terms of SR, OS, and SPL, respectively. Furthermore, JOP-VLN even outperforms methods employing panoramic views, such as OmniNav [50] and NavFoM [49]. In addition, our method obtains 68.0% SR and 59.3% SPL on RxR Val-Unseen split, demonstrating its excellent long-horizon decision-making capability. Figure 7 shows qualitative examples of JOP-VLN on the VLN-CE benchmarks.

5.3.2 Real-world qualitative results↩︎

We evaluate the performance of JOP-VLN in physical environments with various characteristics, including an outdoor garden and an indoor office area. To bridge the significant domain shift between simulation and reality, we employ a few-shot adaptation strategy. Specifically, for each environment, we collect 10 navigation trajectories to perform 10-shot fine-tuning before testing. Qualitative examples of the model’s trajectories are provided in Figure 6. The results demonstrate that JOP-VLN consistently follows instructions accurately across both settings, underscoring its robustness and adaptability to real-world conditions. Failure arises primarily from ambiguous referents in long instructions - when several visually similar landmarks (e.g., multiple doors along a corridor) co-exist, the agent occasionally enters the wrong one, leading to early termination or detours.

5.4 Ablation Studies↩︎

Table ¿tbl:tab:ablation? presents the ablation studies evaluating the effectiveness of our three-stage training pipeline. TS denotes trajectory summarization, while CHORD-ec refers to the use of the error-correction-prioritized trajectory sorting strategy for imitation learning within the CHORD framework.

5.4.1 Effectiveness of trajectory summarization (TS)↩︎

Comparing rows 1 and 2, the integration of trajectory summarization during the first stage leads to improved performance on the R2R benchmark. This demonstrates that navigational capabilities benefit from enhanced visual understanding.

5.4.2 Effectiveness of imitation learning on DAgger trajectories↩︎

Comparing rows 2 and 3, finetuning on the DAgger trajectories in the second stage yields a substantial improvement, with SR rising from 49.9% to 64.4%. Furthermore, the integration of newly collected DAgger trajectories in Stage 3 further increases the SR to \(68.1\%\). These results underscore the critical role of off-policy DAgger trajectories in mitigating covariate shift and enhancing error-recovery capabilities.

5.4.3 Effectiveness of CHORD with high-entropy sampling and error-correction-prioritised trajectory sorting (CHORD-ec)↩︎

In row 5, the application of reinforcement learning using the GRPO objective on high-entropy samples surpasses the post-training methods of pure imitation learning in terms of SPL and NE metrics, demonstrating the effectiveness of on-policy exploration. In row 6, compared to pure RL in row 5, CHORD integrates the dual advantages of on-policy exploration and error recovery capabilities from off-policy expert demonstrations, outperforming both pure IL and pure RL methods. Comparing row 7 to row 6, the inclusion of error-correction-prioritized trajectory sorting allows the model to more frequently encounter trajectories where it previously made mistakes, thereby enhancing its performance.

6 Conclusion↩︎

In this paper, we introduce JOP-VLN, a novel vision-and-language navigation framework that synergistically integrates off-policy imitation learning and on-policy reinforcement learning. This joint learning approach harnesses the dual benefits of error recovery from expert demonstrations and trajectory exploration driven by reward signals. The high-entropy trajectory sampling strategy promotes exploration and prevents the model from entering a “zero-gradient” state caused by low-entropy samples, thereby enhancing the efficiency of RL training. Additionally, the error-correction-prioritized trajectory sorting strategy ensures that the model more frequently learns from trajectories where it previously made mistakes. Extensive experiments demonstrate the superiority of JOP-VLN, achieving new state-of-the-art performance on the VLN-CE R2R benchmark.

References↩︎