1 Introduction↩︎

The increasing democratization of mobile robots beyond research and industrial environments is creating a need for intuitive human–robot interaction mechanisms. As robotic platforms become accessible to non-expert users, natural language provides an intuitive and user-friendly communication modality, allowing users to specify tasks without requiring knowledge of robot programming, coordinates, or navigation systems.

Traditional navigation systems typically rely on explicit goal definitions, including metric coordinates, manually selected waypoints, or predefined semantic maps. Although these approaches provide reliable and accurate navigation, they are limited to expert users. A fundamental challenge in this context is language-driven navigation. In real-world scenarios, users typically express navigation goals through high-level instructions such as “go to the fridge” or “move to the door”, rather than explicit spatial coordinates. Executing such requests requires the robot to relate linguistic concepts to elements in the environment and transform them into actionable navigation goals.

Recent advances in Vision-Language Models (VLMs) provide a promising solution to this problem. By processing visual and textual information, VLMs can associate linguistic concepts with visual elements in a scene, enabling robots to identify objects and regions referenced in natural language commands. Combined with RGB-D perception, these models can be used not only to identify the target object referenced by the user but also to estimate its position in the physical environment.

In this work, we present a language-driven navigation framework that combines VLM-based semantic perception, RGB-D sensing, and autonomous navigation. Given a natural language request, the system identifies the referenced target, estimates its three-dimensional position through geometric projection, and generates a navigation goal that can be executed by a standard robotic navigation stack. The framework is implemented in ROS 2 and leverages the Navigation2 (Nav2) stack for autonomous navigation, enabling deployment on different robotic platforms equipped with an RGB-D camera.

The main contributions of this paper are:

• A language-driven navigation framework that transforms natural language requests into executable navigation goals through the integration of semantic perception and autonomous navigation.

• A method for grounding VLM detections in metric space using RGB-D data, enabling the generation of physically meaningful navigation objectives.

• The implementation and validation of a complete ROS 2-based pipeline capable of interpreting both direct and contextual navigation requests without requiring explicit coordinates or predefined waypoints.

To promote reproducibility and facilitate further research in language-driven robotics, the source code associated with this work will be released as open source upon acceptance.

The remainder of this paper is organized as follows. Section 2 reviews related work, Section 3 presents the proposed framework, Section 4 describes the experimental evaluation, and Section 5 concludes the paper.

2 Related Work↩︎

Recent advances in Vision-Language Models (VLMs) have significantly improved the ability to associate natural language descriptions with visual observations. Models such as CLIP [1] and BLIP-2 [2] learn joint visual-textual representations that support semantic understanding, visual grounding, and multimodal reasoning. These capabilities have motivated their adoption in robotics, where language and visual perception must be combined to understand human intentions.

Several works have explored the integration of foundation models into robotic systems. SayCan [3] combines large language models with robot affordances to generate executable action plans, while RT-2 [4] and PaLM-E [5] demonstrate the potential of unified vision-language-control architectures for embodied tasks. Similarly, \(\pi_{0.5}\) [6] extends vision-language-action models with improved open-world generalization, enabling robots to perform complex tasks in previously unseen environments. These approaches highlight the growing role of VLMs as an interface between human language and robot behavior.

Natural language interaction has been extensively studied in Vision-and-Language Navigation (VLN), where agents must follow navigation instructions using visual observations [7]. The introduction of the Room-to-Room benchmark established a standard framework for evaluating navigation from natural language descriptions and inspired a large body of subsequent research. More recently, surveys such as [8] have identified language grounding, generalization, and robustness as the primary challenges in VLN systems. Although these approaches have demonstrated promising results in benchmark environments, many of them rely on task-specific datasets, end-to-end learned policies, or simulation-oriented settings that are difficult to integrate into conventional robotic navigation frameworks.

A key challenge in language-driven navigation is converting semantic references into physically meaningful navigation goals. Early works explored the mapping of language instructions into actions and visual goals [9], while more recent approaches have leveraged vision-language representations to construct semantic maps and support object-centric navigation. For example, VLMaps [10] uses VLM features to enable robots to localize semantically described objects within an environment. Complementing semantic understanding, RGB-D sensing remains one of the most widely adopted perception modalities in mobile robotics due to its ability to provide geometric information for localization and navigation [11]. Combined with classical geometric projection methods [12], RGB-D data enable visual detections to be transformed into metric coordinates that can be processed by navigation systems such as Nav2 [13].

In contrast to existing VLN and embodied AI approaches, this work focuses on a lightweight and portable ROS 2 framework that directly transforms natural language requests into executable navigation goals. Rather than learning complete navigation policies, the proposed approach combines VLM-based semantic understanding with RGB-D geometric localization to estimate target positions in metric space and generate goals compatible with standard navigation systems. This design enables deployment across different robotic platforms while preserving compatibility with existing ROS 2 and Nav2 infrastructures.

3 System Architecture↩︎

Figure 1 presents the architecture of the proposed language-driven navigation framework, organized around two main entities: the user and the robot. The user provides navigation requests through natural language, while the robot perceives the environment and executes the requested actions. To bridge both entities, the system is composed of four main modules: the User Module, the Communication Module, the Semantic Perception Module, and the Movement Module. Implemented in ROS 2, these modules cooperate to transform natural language requests into navigation actions through semantic perception, goal generation, and autonomous navigation, while also providing natural-language feedback to the user. The ROS 2-based architecture facilitates deployment across different robotic platforms equipped with RGB-D sensors.

3.1 User Module↩︎

The User Module provides the interface between the user and the robotic system, enabling navigation goals to be specified through natural language rather than coordinates or predefined waypoints. User commands are forwarded to the Communication Module for processing.

Typical requests include commands such as “go to the refrigerator”, “move to the door”, or “navigate to the chair”. After processing the instruction, the system generates a natural-language response confirming the identified destination, for example, “Okay, I will go to the chair”. This feedback allows the user to verify that the request has been correctly interpreted before navigation begins.

By abstracting low-level navigation details, the User Module provides an intuitive interaction mechanism that facilitates the use of mobile robots by non-expert users.

3.2 Communication Module↩︎

The Communication Module acts as the central coordinator of the architecture and implements the execution logic of the system. All interactions between the different modules are managed through this component, which is responsible for controlling the information flow and determining when data must be acquired, processed, and transmitted throughout the navigation pipeline.

This module also provides an abstraction layer between the software architecture and the underlying robotic platform. This capability is enabled by ROS 2, which serves as the communication backbone of the system through topics, services, and actions. By translating robot-specific interfaces into system-level interfaces, the Communication Module allows the remaining modules to operate independently of the particular hardware configuration. As a result, the proposed framework can be deployed on different robotic platforms equipped with RGB-D sensors by adapting only the corresponding ROS 2 topic and action interfaces.

When a navigation command is received from the User Module, the Communication Module acquires the RGB and depth information provided by the robot sensors and forwards the corresponding request to the VLM Module. Once the target has been identified and localized, the resulting navigation objective is delivered to the Movement Module for execution. In addition, the module collects feedback from both the VLM and navigation processes.

3.3 Semantic Perception Module↩︎

The Semantic Perception Module connects natural language understanding and robot perception. Its primary objective is to establish communication with the server hosting the Vision-Language Model (VLM), process the user’s navigation request, identify the visual target referenced in the command within the RGB image acquired by the robot, and transform the detected target into a three-dimensional navigation objective expressed in the global map reference frame.

Upon receiving a request from the Communication Module, the Semantic Perception Module sends both the user’s natural language instruction and the RGB image captured by the robot to the remote VLM server. By jointly analyzing the visual and textual information, the model identifies the object or scene element associated with the user’s request and returns its location in the image as a bounding box. The detected bounding box is defined by the image points \(\mathbf{p}_{tl}\in\mathbb{R}^{2}\) and \(\mathbf{p}_{br}\in\mathbb{R}^{2}\), corresponding to its top-left and bottom-right corners, respectively. Additionally, the VLM generates a natural language response, such as “Okay, I will go to the chair”, which is forwarded to the user to confirm that the instruction has been correctly interpreted.

Figure 2 illustrates an example of the semantic perception process for the command “go to the mailbox”. The VLM identifies the mailbox in the RGB image and returns the corresponding bounding box. The resulting detection is then combined with the depth image provided by the RGB-D sensor to estimate the three-dimensional position of the target.

Since the subsequent geometric processing requires a single image point, the center of the bounding box is selected as a representative target location:

\[\mathbf{p}=(p_x,p_y)=\frac{\mathbf{p}_{tl}+\mathbf{p}_{br}}{2}.\]

From this point onward, \((p_x,p_y)\) denotes the image coordinates associated with the detected target.

Once the target has been identified, the module combines the selected image coordinates with the corresponding depth information provided by the RGB-D sensor to estimate the target position in three-dimensional space. Since depth measurements may be noisy or affected by missing values, the depth assigned to the target is computed as the minimum valid depth value within a square neighborhood of size \((2r+1)\times(2r+1)\) centered at \((p_x,p_y)\). Invalid measurements, such as zero or infinite depth values, are discarded during this process to improve the robustness of the estimation, \[d= \min_{\substack{ u \in [p_x-r,p_x+r]\\ v \in [p_y-r,p_y+r]\\ 0 < D(u,v) < \infty }} D(u,v),\]

where \(D(u,v)\) denotes the depth image and \(r\) defines the size of the neighborhood considered around the center pixel. Selecting the minimum valid depth value increases the probability of associating the measurement with the detected object rather than with background elements.

Since the objective is to approach the target rather than reach its geometric center, an offset \(\delta\) can be applied to the estimated depth. The corrected depth is computed as \(d' = d - \delta\), where \(d\) is the measured depth and \(d'\) is the adjusted depth used for goal generation. The value of \(\delta\) depends on the target object; for example, a larger structure such as a bus stop may require a greater offset than a person or a mailbox. This adjustment allows the robot to stop at a suitable distance from the target instead of navigating towards its center.

Given the target image point \(\mathbf{p}=(p_x,p_y)\) and its associated adjusted depth \(d'\), the corresponding 3D position in the camera reference frame can be recovered using the pinhole camera model [12]: \[X = \frac{(p_x-c_x)d'}{f_x}, \quad Y = \frac{(p_y-c_y)d'}{f_y}, \quad Z = d'.\] where \((c_x,c_y)\) represent the coordinates of the principal point of the camera, corresponding to the optical center of the image, and \((f_x,f_y)\) are the focal lengths expressed in pixels, obtained from the intrinsic calibration parameters of the camera. The resulting point \[\mathbf{P}_c=(X,Y,Z)\in\mathbb{R}^{3}\] defines the local position of the detected target with respect to the camera reference frame. Using the robot localization information together with the coordinate transformations available through the ROS 2 TF framework, this point is transformed into the global map frame according to \[\mathbf{P}_m = {}^{m}\mathbf{T}_{c}\mathbf{P}_c,\] where \({}^{m}\mathbf{T}_{c}\in SE(3)\) denotes the rigid-body transformation between the camera and map reference frames obtained from the ROS 2 TF tree. The resulting point \(\mathbf{P}_m\in\mathbb{R}^{3}\) defines the target position in the global map reference frame and constitutes the navigation goal subsequently forwarded to the Movement Module for execution.

3.4 Movement Module↩︎

The Movement Module acts as the interface between the high-level navigation goals generated by the Semantic Perception Module and the robot navigation stack. Its primary responsibility is to transform the navigation objective received from the Communication Module into an executable navigation action and monitor its execution.

Upon receiving a goal position expressed in the global map frame, the Movement Module sends the corresponding navigation request to the ROS 2 Navigation Stack (Nav2) [13]. Nav2 computes a feasible path to the target location and generates the control commands required to execute the navigation task. Throughout the execution process, the module continuously receives feedback from the navigation action, including task progress, goal completion, cancellations, and execution failures.

The proposed framework assumes that a functional low-level controller is available on the robotic platform. Consequently, the Movement Module operates independently of the specific locomotion system of the robot. Through Nav2, navigation commands are translated into velocity commands published through the standard cmd_vel interface, which can then be executed by the robot-specific control layer.

Path planning and obstacle avoidance are delegated to Nav2. A global planner computes a collision-free path using the global costmap, while a local planner continuously adapts the robot motion using the local costmap to account for static and dynamic obstacles. These costmaps are generated from the robot’s onboard sensors, such as LiDAR and depth cameras. By leveraging these capabilities, the Movement Module focuses on connecting the semantic navigation objective generated by the proposed system with the robot navigation infrastructure, enabling the robot to safely reach the goal position estimated by the Semantic Perception Module.

4 Experiments↩︎

The proposed framework was evaluated through three complementary experiments. The first experiment assesses the accuracy of the Semantic Perception Module in simulation by comparing the generated navigation goals against ground-truth target locations. The second experiment evaluates the performance of the complete language-driven navigation pipeline, including semantic perception, goal generation, and autonomous navigation. Finally, the third experiment validates the Semantic Perception Module on a real robotic platform, demonstrating the transferability of the proposed approach to real-world environments.

The simulation experiments were conducted in Gazebo using a TurtleBot3 Waffle integrated with ROS 2 and Nav2. The real-world experiment was performed using a Unitree Go2 equipped with an Intel RealSense RGB-D camera. For all experiments, different natural language prompts and robot initial poses were considered to evaluate the robustness of the proposed approach under varying operating conditions. Figure 3 provides an overview of the experimental scenarios. To facilitate reproducibility and further research, the implementation of the proposed framework will be released as open source upon acceptance.

4.1 Experiment 1: Semantic Perception Evaluation↩︎

The objective of this experiment is to evaluate the ability of the Semantic Perception Module to transform natural language instructions into accurate navigation goals. To isolate the performance of the perception pipeline, a single target object, a bus stop, is considered throughout the experiment. During this evaluation, no navigation action is executed, allowing the perception stage to be assessed independently of the robot navigation stack.

To evaluate the robustness of the proposed approach, four trials were conducted using different natural language formulations referring to the same target object and different initial robot poses were considered, resulting in varying viewing angles and distances to the target. This configuration allows the robustness of the Semantic Perception Module to be evaluated against both linguistic variations and changes in the robot viewpoint while maintaining the same semantic target.

The evaluated prompts were:

1. “Can you go to the bus stop, please?”

2. “Find a bus stop”

3. “Reach the bus stop in front of you”

4. “Would you be able to reach the bus stop?”

For each trial, the system receives a natural language command and estimates a navigation goal using the procedure described in Section 3.3. The generated goal incorporates the depth offset parameter \(\delta\), which is used to position the robot at an appropriate distance from the target according to its characteristics. For the bus stop considered in this experiment, the offset was set to \(\delta = 0.6\) m. The estimated goal position is then compared with a ground-truth navigation goal extracted from the simulation environment.

The localization error is defined as the Euclidean distance between the ground-truth navigation goal in the map frame, \((x_{gt},y_{gt})\), and the goal estimated by the Semantic Perception Module, \((x_{est},y_{est})\):

\[\label{e195error} e_{goal} = \sqrt{ (x_{gt}-x_{est})^{2} + (y_{gt}-y_{est})^{2} }.\tag{1}\] For all experiments, the ground-truth navigation goal was fixed at \((18.0,0.0)\) m. Table 1 summarizes the results obtained for the different prompts and initial robot poses. Since the target object remains unchanged across all trials, the resulting localization error provides a direct measure of the robustness of the semantic perception pipeline. Tests 1–4 correspond to prompts 1–4 listed above, respectively.

Figure 4 presents qualitative results corresponding to the experiments reported in Table 1. Although all prompts refer to the same bus stop, the robot starts from different initial poses, resulting in varying viewpoints and scene compositions. Despite these changes, the system identifies the target and generates a bounding box around it. The center of the bounding box is then used by the Semantic Perception Module to estimate the target’s three-dimensional position and generate the corresponding navigation goal.

The results show that the proposed Semantic Perception Module can reliably identify the target despite variations in viewpoint and language formulation. An average localization error of \(0.68\) m was obtained across all experiments. It should be noted that this error is measured with respect to the ground-truth target position, whereas the generated goal is intentionally shifted using an offset of \(\delta=0.6\) m to place the robot at a practical navigation distance from the bus stop. Consequently, a non-zero localization error is expected even when the target is correctly detected and localized.

3pt

4.2 Experiment 2: End-to-End Navigation Evaluation↩︎

The objective of this experiment is to evaluate the complete language-driven navigation framework. In contrast to Experiment 1, the robot executes the navigation goals generated by the Semantic Perception Module using the Movement Module and the Nav2 navigation stack.

To assess the robustness of the proposed approach, six experiments were conducted considering two semantic targets (a person and a mailbox), different natural language formulations, and different initial robot poses. For each trial, the system receives a natural language command, identifies the referenced target, generates a navigation goal, and autonomously navigates towards it.

For each trial, the system receives a natural language command, generates a navigation goal using the Semantic Perception Module, and executes the navigation task through Nav2. Once the navigation process is completed, the final robot pose is compared with the ground-truth target position.

Let \(I=(x_i,y_i)\) denote the initial robot pose, \(R=(x_r,y_r)\) the final robot pose after navigation, and \(G=(x_{gt},y_{gt})\) the ground-truth position of the target object. The navigation error, \(e_{nav}\), is defined as the Euclidean distance between \(R\) and \(G\):

\[\label{e295error} e_{nav} = \sqrt{ (x_{gt}-x_r)^2 + (y_{gt}-y_r)^2 }.\tag{2}\]

In addition to the navigation error, the execution time and traveled distance were recorded. Furthermore, the natural-language response generated by the VLM was included in the evaluation, as it provides feedback in natural language to the user regarding the identified target and allows the interpreted destination to be verified before navigation begins.

Table 2 summarizes the obtained results. Besides the navigation metrics, the table reports the original user prompt, the response generated by the VLM, the target object, the initial robot pose \(I\), the final robot pose \(R\), and the ground-truth target position \(G\). For this experiment, the depth offset parameter was set to \(\delta=0.3\) m. The navigation error \(e_{nav}\) is computed according to Eq. (2 ), allowing a direct comparison between the desired destination and the position ultimately reached by the robot.

It should be noted that the objective is not to reach the exact target coordinates, which may correspond to the geometric center of the object, but rather to safely approach the desired destination. Consequently, non-zero values of \(e_{nav}\) are expected due to the applied depth offset, the dimensions of the target object, and the obstacle avoidance behavior of Nav2. This effect is particularly noticeable for the mailbox target, which errors are larger than the person target due to its larger size.

In addition to the quantitative evaluation, qualitative results were collected to illustrate the behavior of the proposed framework during navigation. Figure 2 in Sec.3.3 shows the first-person view acquired by the robot during target identification, while Fig. 5 presents a third-person view of a representative navigation trial. Together, these figures illustrate the complete pipeline, from semantic target detection to the execution of the generated navigation goal.

The results demonstrate that the proposed framework successfully combines semantic perception and autonomous navigation. Across all experiments, the robot correctly identified the intended target, generated meaningful natural-language feedback, and reached the desired destination with an average navigation error of \(0.70\) m. Furthermore, the robot traveled an average distance of \(6.22\) m and required an average execution time of \(26\) s, confirming the feasibility of the proposed language-driven navigation framework.

4.3 Experiment 3: Real-World Validation↩︎

The objective of this experiment is to validate the proposed Semantic Perception Module on a real robotic platform and assess its transferability from simulation to real-world environments. The evaluation was conducted using a Unitree Go2 equipped with an Intel RealSense RGB-D camera. Different target objects, natural language prompts, and robot viewpoints were considered to evaluate the robustness of the system under realistic sensing conditions.

Figure 6 presents the two experimental layouts used during the evaluation. The considered targets were a microwave and a chair, while both direct and contextual natural language prompts were employed.

The evaluated prompts were:

1. “Reach the microwave”

2. “I need something to heat up my food”

3. “Reach the chair”

4. “I want to sit”

For each trial, the robot receives a natural language request, identifies the referenced object using the Semantic Perception Module, and estimates its position from RGB-D data. Ground-truth target positions were obtained through direct measurements in the environment and used to compute the localization error according to Eq. (1 ). Since the evaluation is performed in the robot’s local reference frame, both the estimated and ground-truth coordinates are expressed relative to the robot position.

Table 3 summarizes the obtained results. Tests 1–4 correspond to prompts 1–4 listed above, respectively.

Figure 7 presents representative qualitative results obtained with the Unitree Go2 platform. Four first-person observations are shown, corresponding to the two target categories viewed from different robot poses and scene configurations. Despite variations in viewpoint, distance, and scene composition, the VLM consistently identifies the requested target and generates accurate bounding boxes, which are subsequently used by the Semantic Perception Module for geometric localization.

The results demonstrate that the proposed Semantic Perception Module generalizes successfully to real-world environments. Across all evaluated scenarios, the system correctly identified the requested objects and obtained an average localization error of \(0.51\) m. Since this value is comparable to the applied offset (\(\delta=0.5\) m), most of the measured error can be attributed to the intentional displacement used to position the robot at a practical distance from the target. Despite variations in object category, viewpoint, scene layout, and language formulation, the VLM consistently identified the intended target, confirming the transferability of the proposed approach from simulation to a physical robotic platform.

5 Conclusions↩︎

This paper presented a language-driven navigation framework that combines Vision-Language Models (VLMs), RGB-D perception, and autonomous navigation to enable mobile robots to interpret natural language requests and navigate towards the referenced targets. By integrating semantic understanding with geometric localization, the proposed system transforms high-level instructions into executable navigation goals without requiring explicit coordinates, enabling its use to non-expert users.

The framework was validated through three complementary experiments, including two simulation-based evaluations and a real-world deployment on a real platform. Across all experiments, the system successfully identified the intended targets, generated meaningful natural-language feedback, and produced reliable navigation goals. Furthermore, the real-world validation confirmed the transferability of the proposed approach beyond simulation environments.

Overall, the results demonstrate the potential of VLMs to bridge natural language interaction, semantic perception, and autonomous navigation. Future work will focus on extending the framework to support more complex multi-step tasks, improving goal generation through adaptive offset estimation, and evaluating the system with a broader range of objects and in more dynamic environments.

References↩︎

---

1. This work has been supported by MCIN/AEI/10.13039/501100011033, under project AI-FUSE-ROBOT (SAIA202500X163851SV0) and by CIN/AEI/10.13039/501100011033 and by ERDF "A way of making Europe" under project COBUILD: Collaborative team of robots and humans for advanced digital building (PID2024-161069OB-I00).↩︎