1 Introduction↩︎

Multi-robot systems demonstrate significant potential in diverse environments, from domestic service to industrial manufacturing [1]–[3], by expanding manipulation capabilities and enhancing efficiency to offer great system extensibility. However, achieving such systems imposes a substantial burden on manipulation complexity, necessitating coordination between heterogeneous robots and effective emergency response to unexpected failures during execution.

Recently, the integration of Large Language Models (LLMs) has offered new avenues for robotic manipulation by leveraging their capabilities in semantic understanding and logical reasoning [4]–[6]. Existing LLM-based approaches fall broadly into two categories. The first uses LLMs to synthesize single-robot manipulation policies, code, grasping guidance, or constraints [7]–[12]. While these methods demonstrate promising physical grounding, they lack the coordination mechanisms required for tasks that span multiple workspaces. The second paradigm targets multi-robot systems but focuses primarily on high-level task allocation and symbolic planning [13]–[17], typically validating plans prior to execution while treating manipulation as an idealized primitive. This abstraction assumes deterministic success and neglects real-world execution uncertainties such as grasp failures, contact variability, external disturbances, or unreachable workspace.

To achieve robust and self-correction capabilities in real-world tasks, we propose a closed-loop agentic workflow for robust multi-robot manipulation. Our core insight is that tackling real-world uncertainties requires a shift from static execution pipelines to an iterative agentic process that incorporates a hierarchical recovery mechanism. Unlike previous methods that restrict validation to the pre-execution phase and subsequently rely on idealized actions, an agentic workflow empowers the system to actively reason, act, and reflect on its performance. By integrating semantic planning, physical grounding, and outcome verification, our architecture enables robots to perceive environmental feedback and adaptively recover from failures, achieving robust and verifiable multi-robot manipulation and ensuring reliability by allowing agents to actively address execution uncertainties in real-world environments.

Specifically, we instantiate this workflow with three specialized agents that operate in a closed-loop cycle of reasoning, acting, and reflecting, enabling refinement of both high-level collaboration decisions and low-level execution. The information flow begins with the Planning Agent, which serves as the reasoning core to handle task decomposition and allocation while balancing semantic instruction logic with the specific capabilities of heterogeneous robots. To ensure accurate semantic grounding, it employs a bidirectional interaction mechanism to query bottom-up perception from the Manipulation Agent whenever instructions are ambiguous. Following this, the Manipulation Agent takes charge of the acting phase. It translates sub-tasks into fine-grained operations by adaptively leveraging tools such as keypoint localization and grasp generation models to ground semantic plans into reality. Finally, to tackle inevitable execution uncertainties, the verification functions as a reflecting mechanism. It monitors the status of each sub-operation, and upon detecting a failure, it utilizes historical interaction data and current observations to diagnose the cause of error and trigger hierarchical recovery strategies. For execution-level errors, it initiates local re-planning, whereas for capability constraints, it escalates to global re-planning to seek collaboration from other robots. This multi-level mechanism effectively resolves errors at the appropriate layer and prevents failures from disrupting the global multi-robot workflow. Together, these agents form a collaborative system that ensures reliable deployment for complex multi-robot manipulation.

Extensive real-world experiments on six tasks validate our framework in three aspects: (1) Versatility: outperforming both learning-based and LLM-based baselines in diverse skills ranging from pick-and-place to articulated object interactions; (2) Robustness: evidenced by a multi-level recovery mechanism that handles physical disturbances via local re-planning and resolves capability constraints through inter-robot collaborative recovery; (3) Adaptability: demonstrated by the seamless transition by our framework from tabletop setups to cross-workspace collaborations involving heterogeneous robots.

2 Related works↩︎

2.1 LLMs for Robotics↩︎

While conventional robotic manipulation models [18]–[25] demonstrate promising performance, their understanding of tasks and environments remains constrained. To address this limitation, Large Language Models (LLMs) are increasingly integrated into robotic systems to leverage their strengths in semantic reasoning and high-level planning [26]–[28]. Early works, such as [29]–[32], demonstrate the efficacy of LLMs in decomposing semantic instructions into sequential sub-tasks. Leveraging the robust code generation capabilities of LLMs [33]–[35], a specific stream of research focuses on synthesizing executable policies directly. [7], [36] utilize LLMs to generate Pythonic control logic that invokes predefined action primitives. To improve physical grounding, methods like [9], [37]–[39] formulate manipulation as constraint optimization problems, using the geometric information of the scene, translating natural language into cost maps or spatial constraints to guide motion planning. While these approaches achieve promising manipulation capabilities in single-robot settings, they are inherently limited to individual workspaces. They lack the requisite coordination mechanisms to manage spatio-temporal constraints across multiple robots, making them insufficient for collaborative tasks that demand synchronized execution and long-horizon cooperative behaviors.

In contrast to these single-robot-centric approaches, our framework extends LLMs manipulation paradigm to the multi-robot domain. We introduce a collaborative planner through an agentic workflow that enables heterogeneous robots to achieve parallel execution and cross-workspace collaboration.

2.2 Multi-Robot Planning and Collaboration↩︎

Prior to the advent of LLMs, multi-robot planning focuses on classical symbolic planning and optimization methods. Traditional multi-robot Task and Motion Planning frameworks typically utilize Planning Domain Description Language (PDDL) or temporal logic to solve long-horizon collaboration problems under geometric constraints [40]–[42]. While these methods ensure logical correctness and collision-free execution, they rely on rigid, predefined domain rules and struggle to generalize to open-ended semantic instructions.

Recognizing the limitations of traditional approaches in semantic understanding and adaptability, recent research has extended LLMs capabilities to multi-robot systems [43]–[49]. A prominent line of work treats multi-robot collaboration as a communication and consensus problem mimicking human teams. For instance, [50], [51] and [52] introduce multi-turn dialog mechanisms where robots discuss strategies to reach a consensus on task distribution. Other frameworks explore centralized paradigms. [16] utilizes a single centralized LLM to decompose instructions into multi-stage task planning. In a more structured approach, [17] adopts a centralized hierarchical pipeline that generates sub-task proposals, allocating specific actions to heterogeneous robots based on their capabilities.

Despite their success in high-level planning, most existing LLM-based frameworks limit their validation before execution, which implicitly assume that the scheduled tasks will be executed successfully. They tend to treat the interaction process as idealized abstract actions, neglecting the execution uncertainties[53], [54] inherent in the physical world. Consequently, these planners lack the requisite dynamic verification and feedback mechanisms during deployment, often leading to cascading system-wide failures when local errors occur.

In contrast to the aforementioned approaches, our work bridges the gap between single-robot manipulation capabilities and multi-robot collaboration. We propose a closed-loop multi-agent architecture that bridges low-level physical manipulation with high-level collaborative task planning. By incorporating a Verification Agent, our framework moves beyond open-loop execution, enabling active outcome monitoring and adaptive failure recovery to ensure robust real-world collaboration.

3 Method↩︎

3.1 System Overview↩︎

In this section, we present a closed-loop multi-agent framework designed for reliable multi-robot manipulation in real-world environments. As shown in Fig. 1, our framework is structured as a collaborative agentic workflow. It takes language instructions and a coarse-grained scene description, which include the robot’s position and the general location of objects as input. It integrates semantic planning, physical manipulation, and verification into an iterative cycle coordinated by three specialized agents: the Planning Agent handles high-level reasoning and task decomposition, the Manipulation Agent translates the sub-task into the real world and the Verification Agent assesses action feasibility and execution outcomes. Crucially, these agents dynamically interact to translate abstract instructions into real-world primitives, while robustly adapting to execution-time uncertainties. Detailed designs of each agent and their collaborative mechanisms will be elaborated in the following sections. Specific model selections are detailed in Appendix.@.

3.2 Planning Agent: Semantic Reasoning and Task Allocation↩︎

The Planning Agent serves as the semantic reasoning core of the framework. As shown in Fig. 1 (a), its primary role is to interpret user instructions and generate a high-level task graph that bridges the gap between abstract semantic goals and robot-specific capabilities through two key mechanisms.

Dependency-Aware Task Decomposition. Given a natural language instruction and coarse-grained scene descriptors, the Planning Agent decomposes the global objective into a directed acyclic graph (DAG) of logical sub-tasks. Unlike generating action sequences, this agent focuses on identifying task dependencies, determining which sub-tasks must be executed sequentially and which can be performed simultaneously. It outputs a schedule of sub-tasks augmented with a parallel flag, allowing robots to perform operations in parallel during this stage which enables concurrent execution by heterogeneous robots to optimize operational efficiency.

Capability-Based Allocation & Interactive Refinement. The Planning Agent assigns each sub-task to the most suitable robot by matching the sub-task requirements (e.g., transport) with the heterogeneous robots’ capabilities (e.g., mobile base and fixed base). Furthermore, to handle potential uncertainties, the agent is equipped with an interactive refinement capability. In scenarios where the initial instruction or scene information is insufficient (e.g., an ambiguous “Tidy up” command or the object to manipulate is unknown), the Planning Agent can conditionally send a further_perception flag. This triggers a targeted query to the corresponding Manipulation Agent to obtain fine-grained observations, allowing the Planning Agent to refine the task graph based on the updated semantic context.

3.3 Manipulation Agent: Physical Action Grounding↩︎

The Manipulation Agent functions as the embodied executor within our agentic workflow. As shown in Fig. 1 (b), upon receiving a logical sub-task from the Planning Agent, its objective is to translate this semantic instruction into action primitives that can control the robot directly. To achieve this, the agent leverages a flexible mechanism of adaptive tool invocation. Rather than following a fixed pipeline, it dynamically calls upon a suite of specialized modules, ranging from VLM-based decomposition to visual prompting tools to decompose a sub-operation, perceive targets, and get action parameters for the specific physical context. Specifically, the agent re-perceives the environment between sub-operations in response to potential changes occurring after an action (e.g., opening a drawer).

Role-Based Operation Resolution. Unlike Planning Agent that decomposes tasks at a high-level, Manipulation Agent focuses on breaking down sub-tasks into executable fine-grained actions. Before execution, the agent first divides sub-task into manipulation steps through VLM-based reasoning. Based on the interaction logic, it not only categorizes entities into active and passive roles but also determines their semantic part descriptions (e.g., “handle”, “surface”) for each object. Simultaneously, the agent evaluates object attributes to determine whether a dual-arm action is required at this stage. This structured resolution ensures that subsequent tools focus on the correct entities.

Tool-Augmented Visual Perception. To bridge the gap between semantic roles and action parameters, the agent executes a comprehensive visual grounding pipeline utilizing the part descriptions extracted in the previous stage. Object Detection & Segmentation: The agent first invokes a VLM-based detector to identify bounding boxes for active and passive objects, followed by the Segment Anything Model (SAM) [55] to generate precise pixel-level masks. Affordance-Aware Keypoint Selection: The agent employs a visual prompting tool [38] to localize optimal interaction points, such as the upper part of a cup handle. For dual-arm grasping, the agent applies a geometric heuristic to identify the key points for center symmetry of objects to ensure stability. Crucially, these 2D keypoints are projected into 3D space using depth information, establishing the spatial anchors required for manipulation.

Action Primitives Generation with Adaptive Tool Invocation. Once the spatial keypoints are established, the agent synthesizes the operation by querying an action primitive library containing skills in Tab ¿tbl:tab:primitives?. During this generation process, the agent adaptively invokes specialized tools if a primitive requires physical grounding. For instance, when instantiating a grasp primitive, semantic keypoints alone may be insufficient for physical stability. Current 6-DoF grasp generation methods are diverse [56]–[59], the agent triggers AnyGrasp [58] to generate candidate 6-DoF grasping poses (details are provided in Appendix.@.C), which are then matched against the extracted 3D semantic keypoints to select the optimal pose that minimizes spatial alignment error. In addition to spatial precision provided by the keypoints selection, many manipulation tasks require appropriate object orientation to satisfy specific semantic requirements. To addressing this, the agent can activate the rotation perception tool. Leveraging rigid-body assumption, this tool projects the robot’s end-effector frame onto the object’s center and a VLM then performs spatial reasoning to analyze the object’s current state and the task requirements, predicting the necessary rotation axis, direction, and angle. This allows the agent to execute intuitive reorientation actions.

Memory-Augmented Reasoning. To enhance the manipulation efficiency and reduce the ambiguity in sub-tasks description, we implemented a dual-layer memory mechanism. First, for the short-term memory, the agent maintains an interaction history of previous manipulation steps. It will be used in the sub-task decomposition phase which resolves ambiguous sub-task instructions and verification phase. For example, when receiving “Pour water", it can determine from historical manipulations whether the grasping action has already been performed or if only the pouring action needs to be executed. Second, for manipulation efficiency, we implement an experience pool for the sub-task decomposition and action primitives generation. For a sub-task, the agent first identifies the actions required for the task and the objects involved in the interaction and records them as a template signature (e.g., <Action> Active\(\to\)Passive). After this sub-task is successfully executed, the agent will record the results of sub-task decomposition and the corresponding action primitives. Next time the agent encounters a sub-task exactly matching the signature, it triggers a shortcut, using the validated sub-task decomposition and the primitives directly. The experience pool reduces the number of VLM API calls which accelerates execution time, enabling manipulation to proceed directly into the perception phase.

3.4 Verification Agent: Local Monitoring and Hierarchical Recovery↩︎

The Verification Agent acts as the reflection component in the agentic workflow to ensure system reliability. As shown in Fig. 1 (c), it transforms the linear execution into a robust closed-loop system by monitoring the task states and leveraging a hierarchical recovery mechanism. Each Manipulation Agent is equipped with a Verification Agent which obtains the execution status and triggers different strategies based on the types of failures: executing a local self-correction when execution fails, while interacting with the Planning Agent for global re-planning when encountering feasibility errors.

Discrete Visual Outcome Validation. To verify each sub-operation acted by the Manipulation Agent, Verification Agent employs a visual feedback tool to perform a discrete and semantic assessment. Upon the completion of a sub-operation, the agent analyzes pre-action and post-action images relative to the instructions, performing logic validation to generate a binary success prediction.

Hierarchical Error Recovery. When a failure is detected, the Verification Agent diagnoses the error type and triggers recovery at the appropriate system level. For execution failures, where the task remains feasible (e.g., a grasping slip or misalignment), the agent initiates a local recovery loop without disturbing the global plan. By referencing the interaction history and the visual states, the agent autonomously constructs a corrective sequence to re-establish the necessary states. These recovery steps are dispatched directly to the Manipulation Agent, forming an efficient inner loop for rapid self-correction.

Conversely, if an action is identified as physically infeasible, typically due to workspace constraints or missing objects , the Verification Agent will send the failure to the Planning Agent, triggering a global recovery loop. It will submit a failure feedback containing the specific error context and state, which activates re-allocation rules in the Planning Agent. Consequently, the Planning Agent generates a new task graph based on the cause of the error and the current state of robots, while continuing to pursue the original task objectives. For example, when a fixed-based robot arm can’t pick up an object out of its working space, Planning Agent will dispatch a mobile robot to solve the problem. This hierarchical mechanism ensures task continuity through agentic collaboration, preventing single-robot failures from stopping the entire workflow.

4 Experiments↩︎

In this section, we evaluate the effectiveness of our proposed framework through extensive real-world experiments. We designed 6 distinct tasks of varying complexity, ranging from single-robot manipulation to heterogeneous multi-robot collaboration. The task execution processes are shown in Fig.2.

4.1 Experimental Setup↩︎

Hardware Configuration: Our system comprises three types of robots: (1) an Agilex Cobot Magic dual-arm mobile robot, (2) an ABB-YuMi dual-arm desktop robot, and (3) an Agilex Piper single-arm desktop robot. Based on the number of robots, we have categorized 6 tasks into two categories:

4.1.0.1 Table Top Dual-Arm Tasks

(1) Block Stacking: A precision task requiring the robot to place a rectangular block across two cylinders (bridge) and stack a cube on the top. (2) Table Organization: The robot needs to tidy up a workspace by multi-round pick & place 5 odds and ends into a basket sequentially under fuzzy instructions without knowing the objects on the table. (3) Object Stowing: An articulated object manipulation task which contains multiple stages involving opening a drawer, placing an object inside, and closing drawer.

4.1.0.2 Multi-Robot Collaboration

(1) Basket Shelving: A Piper robot places an object into a basket. The Cobot Magic robot lifts the basket with dual-arm and transports it to a shelf in a different workspace. (2) Collaborative Pouring: Receiving an instruction to pour a bowl of water, Cobot Magic first needs to move to a table which has a bowl on it, transport it to ABB’s table, then place it in an appropriate place for ABB to pour water into it. (3) Food Preparation: A three-robot task. The ABB robot pours water into an instant noodle cup and places the cup back. Simultaneously, the Piper places two food objects on the plate on a different table. Finally, the Cobot Magic transports the prepared food plate from Piper’s table to the ABB’s table.

4.2 Dual-arm Table Top Task↩︎

We validate our framework’s manipulation ability against (1) OpenVLA-OFT [21] and (2) \(\pi_{0}\) [20], representing mainstream learning-based end-to-end policies. (3) ReKep [9], a learning-free method that employs VLMs to optimize relational keypoint constraints for robot manipulation. We report success trials and task completion (Comp. %) over 20 trials for each task.

Experimental Setup. Both learning-based baselines (OpenVLA-OFT and \(\pi_{0}\)) are fine-tuned in a single-task setting using 50 demonstrations per task. To isolate perception errors and strictly evaluate manipulation capabilities, we provide ReKep with ground-truth keypoint annotations manually during execution.

2pt

As summarized in Tab. 1, our method significantly outperforms all baselines on three tasks. The learning-based methods struggled in contact-rich tasks like Block Stacking and Object Stowing. Lacking precise geometric grounding, minor deviations in the predicted end-effector’s pose frequently led to hard collisions with blocks and drawer, causing immediate failures. In long-horizon task Table Organization, baselines often fail to grasp the target object yet continue to execute the subsequent trajectory, letting these execution errors remain undetected and unchecked throughout the sequence. The learning-free method ReKep completely failed in Object Stowing because their original method could not define the end key points when operating drawers. In Block Stacking, the optimization solver often returns sub-optimal solutions that failed to strictly avoid collisions while attempting to reach the goal. In contrast, our framework achieved the highest success rates across all tasks, demonstrating that our tool-augmented perception ensures the precision required for stacking, the diverse combination action primitives provide the necessary versatility, and the Verification Agent provides the closed-loop robustness necessary to detect and try to recover from the execution errors.

5.5pt

4.3 Robustness to External Disturbances↩︎

To evaluate the robustness of our framework against real-world uncertainties, we designed three distinct disturbances ranging from local interference to global failure. We bring these disturbances at random intervals during the execution of both desktop and multi-robot tasks.

4.3.0.1 Action Level

Human operators remove or displace the target object while the robot is attempting to interact, simulating environmental changes or interaction failure.

4.3.0.2 Sub-task Level

After a sub-task is completed, the operator resets the environment state, specifically testing its ability to execute multi-step recovery sequences.

4.3.0.3 Global Level

In multi-robot task, a failure may occur if the target is placed outside the workspace of a fixed-base robot. The system must invoke a collaborative recovery and dispatch other robots to assist in the manipulation.

As shown in Tab. 2, learning-based baselines struggled significantly under these perturbations, resulting in low average success rates. Due to the absence of a closed-loop feedback mechanism, these policies often continued to execute subsequent trajectories despite environmental change. While they may attempt a simple re-grasp if a failure occurs during the initial phase, they fail to recover from complex disturbances that require multi-step reasoning. In contrast, our method maintained a robust average success rate of 63%. Compared to the undisturbed condition, the success rate shows only a slight decrease, demonstrating the critical necessity of our verification loop.

4pt

4.4 Verification Agent Analysis↩︎

To demonstrate the reliability of our Verification Agent, we tracked four metrics under the aforementioned disturbance conditions: Failure Identification Rate (ID), Re-planning Success Rate (Re-Plan), Physical Recovery Success Rate (Phy-Rec) which quantifies the robot’s ability to physically resolve the disturbance conditioned on a valid plan and Final Task Success Rate (Task SR). The results are shown in Tab. 3. The Verification Agent shows robust failure identification capabilities around 90–100% failure identification rate across all tasks. This confirms that our VLM-based verification module effectively distinguishes between successful outcomes and varying types of errors.

In single-robot tasks, the system primarily relies on an internal recovery loop. The Re-planning Success Rate was consistently 100%. However, there is a gap between detection and final success in execution, as irreversible states may occur during task execution, such as collapse of the building blocks, which require more meticulous operational skills to recover. In contrast, Object Stowing achieved a high 80% Final SR, proving the system’s effectiveness in handling reversible state regressions. In the multi-robot task, we validate the global recovery capability of our framework in the Basket Shelving task. We placed an object outside the fixed-base robot’s workspace, making it unreachable for the required operation. The system successfully identified the error caused by exceeding the workspace and performed global re-planning with 100% re-planning success. The only exception occurred when the object moved out of the field of view after a failed placement operation.

2.5pt

4.5 Ablation Study↩︎

To analyze the contributions of each module within our framework, we conducted an ablation study across 6 tasks. We evaluated the system by systematically removing: (1) the visual perception tool, (2) the Verification Agent, (3) short-term memory (interaction history recorded by the Manipulation Agent), and (4) long-term memory (experience pool). Quantitative results are summarized in Tab. 4, and Fig. 3 details module contributions to task failures.

Impact of visual perception tool. We employ the same visual perception method from ReKep[9] to replace our process, which uses position and feature clustering to obtain representative key points throughout the image. As shown in Fig. 3, the absence of the object-specific keypoint selection process resulted in a catastrophic performance drop. The results reveal the problems of previous multi-robots planning methods, they treat execution as idealized process, ignoring that physical manipulation is coupled with actual perceptual process. For LLM-based manipulation approaches, the precise keypoint selection serves as the essential bridge connecting the high-level plan to the real-world environment. It is not merely an auxiliary step but essential for any successful manipulation.

Impact of Verification Agent. The ablation of the Verification Agent led to a decline particularly in long-horizon tasks like Food Preparation and Table Organization. In these tasks, every individual operation carries a non-zero failure probability. Without Verification Agent, these minor execution errors accumulate, eventually becoming fatal to the overall task. This validates that a closed-loop mechanism for error detection and recovery after each execution is essential for the successful completion of complex multi-robot tasks.

Impact of Short-Term Memory. Removing short-term memory caused a drop in sequential manipulation tasks, most notably in Table Organization and Food Preparation. To address changes in the environment following an operation, we decompose the sub-operation into action primitives. Therefore, we observed that without historical context, some sub-task instructions might be ambiguous, such as “place” command could mean that an object can be placed directly, or needs to be grasped before being placed. This was particularly evident in pouring sequences. Short-term memory eliminates these redundancies by recording the interaction history and state, helping Manipulation Agent generate sub-operations that are logically consistent with the robot’s current status.

Impact of Long-Term Memory. For the long-term memory module, our primary objective is to enhance manipulation efficiency and reduce the frequency of LLM invocations. Experimental results indicate that removing this module only results in a slight decline and contributes to zero failures in method performance. This demonstrates the robustness of our Manipulation Agent in decomposing sub-tasks and generating action primitives, proving the framework does not strictly rely on memorization. Second, it demonstrates that the design of the experience pool enhances stability. By retrieving validated actions for repetitive sub-tasks, the system avoids the variance introduced by VLM generation, ensuring higher consistency and efficiency in execution.

4.6 System Error Breakdown↩︎

To identify the limitations of our current framework, we record the failure distribution across individual modules based on the results from the full model experiments in Tab. 4. As illustrated in Fig. 4, keypoint selection constitutes the dominant failure mode. This bottleneck arises because the VLM struggles to localize keypoints that precisely satisfy the semantic constraints, resulting in a mismatch between the VLM’s reasoning and the actual environment. Action primitives generation failures followed, sometimes the robot failed to find a kinematic solution or the VLM failed to correctly reason the relation between keypoints, resulting in incorrect parameter input. Beyond these two modules, failures such as Verification Agent errors and grasp failures account for a low percentage. Additionally, irrecoverable failures occasionally arise due to physical constraints, such as robotic arm collisions, noisy sensor data from depth cameras, and occasional navigation drift. However, each of these failures represents a relatively minor portion of the total cases.

5 Conclusions↩︎

In this paper, we propose a closed-loop multi-agent framework to enable robust and collaborative manipulation for a multi-robot system. By structuring the system into three specialized agents-Planner, Manipulation, and Verification, we bridge the gap between high-level reasoning and low-level physical execution. Extensive experiments across fine-grained desktop tasks to multi-robot collaborative tasks validate our framework possesses high versatility in executing diverse skills, ranging from precision pick-and-place to articulated object interactions. Furthermore, our framework demonstrates robustness by maintaining continuous multi-robot workflows under disturbances through hierarchical recovery. Finally, transition from desktop setups to cross-workspace collaborations confirms the framework’s adaptability in scaling from single-robot to heterogeneous multi-robot systems.

Limitations and Future Work. Currently, our system relies on a library of action primitives, which limits its ability to handle more complex manipulation tasks. Additionally, the current workflow is constrained by the execution speeds of heterogeneous robots. The system primarily employs a dependency check, lacking a dynamic scheduling mechanism to optimize the global efficiency. In future work, we aim to integrate learning-based primitive discovery and explore concurrent task allocation methods to minimize idle time and enhance the efficiency of multi-robot collaboration.

Acknowledgments↩︎

This work was supported partially by NSFC (92470202), Guangdong NSF Project (No.2023B1515040025), Guangdong Key Research and Development Program (No.2024B0101040004, No.2025B0909020002).

6 Hardware Setup↩︎

We employ three distinct robotic platforms to validate our framework (shown in Fig. 5). To ensure a unified action space, all action primitives operate via the End-Effector (EEF) pose commands. These pose targets are translated into joint angles using inverse kinematics (IK) solvers for execution. The detailed configurations are as follows:

6.0.0.1 Mobile Dual-Arm Platform (Agilex Cobot Magic)

This platform features two 6-DoF arms on a two-wheeled differential chassis equipped with a Livox Mid-360 LiDAR. Leveraging the LiDAR data, we employ the FAST-LIO[60] algorithm for mapping and localization, while the ROS1 navigation stack is utilized for global path planning and navigation. Visual perception is provided by an Intel RealSense D435 camera. For low-level control, we utilize the PyRoki[61] library for inverse kinematics solving.

6.0.0.2 Single-Arm Platform (Agilex Piper)

The Agilex Piper is a 6-DoF arm with a 1.5 kg payload. Visual perception is provided by an Intel RealSense D435 camera. Similar to the mobile platform, its IK solving is handled by PyRoki[61].

6.0.0.3 Dual-Arm Platform (ABB YuMi)

We use the ABB YuMi (IRB 14000) for high-precision tasks, with an overhead Intel RealSense L515 camera. The robot is interfaced using YumiPy, which handles communication and IK solving.

7 Perception Tools Details↩︎

7.1 VLM-based Object Detection and Segmentation↩︎

To achieve precise object manipulation, our system employs a coarse-to-fine perception strategy. We first utilize a Vision-Language Model (VLM) to obtain initial bounding boxes, which subsequently serve as spatial prompts for the Segment Anything Model (SAM) to generate pixel-wise masks. The object detection module bridges the high-level task planner and the low-level segmentation module through two key stages.

First, in the Detection via VLM stage, we employ Qwen3-VL-Plus to detect the identified targets within the current RGB observation. The model is queried with a structured instruction to identify the specified objects and return their 2D bounding boxes (BBoxes) in JSON format. Subsequently, during Segmentation, the system performs a post-processing step to associate the detected bounding boxes with their semantic roles. These structured BBoxes and their associated roles are then passed to the SAM module as prompt inputs for fine-grained segmentation.

7.2 Fine-grained Keypoint Selection↩︎

Following object detection and segmentation, the system determines the precise 3D manipulation coordinates. We utilize a Grid-based Visual Prompting method [38] (as shown in Fig. 6) that enables the VLM to perform pixel-level semantic localization without requiring pixel-coordinate output capabilities.

To bridge the gap between semantic instructions (e.g., “grasp the handle”) and spatial coordinates, we generate a visual prompt for each detected object. This process involves three steps:

7.2.0.1 ROI Normalization and Enhancement

To emphasize the object’s geometry, we apply Canny edge detection to the segmented region and highlight the contours in blue. The processed image is then cropped and resized to a standardized resolution to ensure the visibility of the small features and structural details.

7.2.0.2 Grid Overlay

A labeled grid (e.g., \(10 \times 10\)) is overlaid on the object image. To ensure the keypoints fall on the object surface, the grid labels are positioned based on the foreground pixel density within each cell.

7.2.0.3 Semantic Selection

The VLM receives this grid-annotated image along with the interaction description of the sub-stage. Instead of predicting continuous coordinates, the model performs a classification task by selecting the grid indices that correspond to the desired functional parts.

Once the VLM selects the target grid indices, the system maps the 2D grid centers back to the original image coordinate system \((u, v)\). To obtain the corresponding 3D world coordinates \((x, y, z)\), we employ a 3D Coordinate Extraction strategy. This mitigates noise in the raw point cloud through Neighborhood Sampling, where we sample a local \(N \times N\) neighborhood (e.g., \(7 \times 7\)) around the target pixel, and Outlier Rejection & Median Filtering, which filters out invalid depth values based on spatial continuity constraints. The final 3D coordinate is computed as the spatial median of the valid neighborhood points, ensuring geometric stability for the robotic manipulation.

7.3 Grasp Pose Generation↩︎

To translate the semantic keypoints into executable 6-DoF grasp poses, we employ a hybrid strategy that adapts to object characteristics and task requirements.

For Geometric Heuristic Generation, utilized primarily for small objects (height \(\le\) 0.05m), articulated objects and bimanual tasks involving dual keypoints, the system derives grasp poses directly from the geometric properties of the detected bounding boxes (BBoxes). To bridge the perception and manipulation spaces, the four vertices of the BBox’s are projected into the robot’s base coordinate system. The algorithm determines the grasp orientation by aligning the gripper’s approach vector with the BBox surface normal facing the robot, while aligning the gripper closing direction with the shortest edge of BBox or specific functional edges. Finally, the grasp center is anchored at the task-relevant keypoints which are selected in keypoint selection module.

For general objects, we integrate the AnyGrasp[58] with our semantic guidance. The process proceeds in two steps:

7.3.0.1 Kinematic Filtering

Upon retrieving dense grasp candidates, we transform them back to the world frame and apply kinematic constraints. We filter out poses with invalid approaching vectors (e.g., collision-prone orientations) to ensure the reachability of the robotic arm.

7.3.0.2 Keypoint-guided Selection

Given the set of candidate grasps \(\mathcal{G} = \{g_i\}_{i=1}^N\) generated by the AnyGrasp, where each grasp \(g_i\) is parameterized by its rotation \(\mathbf{R}_i\), translation \(\mathbf{t}_i\), and confidence score \(s_i\). Let \(\mathbf{p}_{key} \in \mathbb{R}^3\) denote the 3D semantic keypoint generated from keypoint selection module.

We formulate the selection of the optimal grasp \(g^*\) as a radius-constrained optimization problem. The goal is to maximize the grasp quality score \(s_i\) within a dynamically expanding neighborhood of \(\mathbf{p}_{key}\). The optimal grasp is defined as:

\[g^* = \underset{g_i \in \mathcal{S}_k}{\arg\max} \;s_i\]

subject to the spatial feasibility set \(\mathcal{S}_k\) at iteration \(k\):

\[\mathcal{S}_k = \{g_i \in \mathcal{G} \mid \|\mathbf{t}_i - \mathbf{p}_{key}\|_2 \le r_k \}\]

where the search radius \(r_k\) is updated iteratively according to \(r_k = r_{init} + k \cdot \Delta r\). The search starts with \(k=0\) and increments until \(\mathcal{S}_k \neq \emptyset\) or \(r_k > r_{max}\). This iterative expansion prioritizes semantic proximity, ensuring that the robot acts upon the specific functional part (e.g., the handle), falling back to a wider search area only if no valid high-quality grasps are available in the immediate vicinity.

7.4 Rotation Tool↩︎

While translation tasks primarily rely on spatial coordinate perception, many tasks necessitate object rotation to satisfy specific instructions (e.g., “pour water” and “insert a pen”). We propose a VLM-guided framework to determine the rotation parameters. The process consists of two stages:

7.4.0.1 Reference Frame Visualization

To ground the rotation axes in the physical world, the system first computes the geometric median of the target object’s point cloud. It then projects the end-effector’s local coordinate axes (X, Y, Z) onto the object’s image(As shown in Fig. 7).

7.4.0.2 Parameter Inference

The VLM receives the current task instruction, the end-effector’s rotation matrix and the annotated images from the last step. Then the model predicts rotation parameters: the Rotation Axis (local X/Y/Z), Direction (clockwise/counter-clockwise), and Angle (degrees). This semantic parameterization bridges the gap between high-level language commands and low-level control.

8 Model Specifications and Prompts Details↩︎

Model Specifications In our framework, we primarily utilize the Qwen3 series models[4]. Specifically, the Planner Agent utilizes Qwen3-Max (snapshot qwen3-max--01-23), while the Manipulation and Verification Agents operate on Qwen3-VL-Plus (snapshot qwen3-vl-plus-2025-12-19). Both models are accessed via the Alibaba Cloud DashScope API.

System Prompts We list the system prompts employed in our framework, categorized by their functional roles. Please note that for the sake of brevity and clarity, the prompts presented below are simplified versions. We focus on displaying the core roles, goals, and reasoning constraints.

a) Planning Agent:

b) Manipulation Agent:

b) Verification Agent:

References↩︎