1 Introduction↩︎

Imagine a robot tasked with spraying a plant: it must grasp the bottle stably, aim it toward the leaves, and squeeze the trigger to release the spray, all in a coordinated sequence. This type of task is a form of Functional Object Manipulation (FOM) [1]–[5], where the robot actuates an object’s internal degrees of freedom (e.g., trigger, button, lever) while controlling its external DoF (the object pose) to apply the function to a target region [6], [7]. We refer to these problems as Compositional Dexterous Functional Object Manipulation (CD-FOM). Because they require coordinating internal actuation with precise object motion, CD-FOM tasks remain a major challenge in robotics.

CD-FOM tasks require bridging semantic understanding and physical dexterity. The robot must interpret the task context—what the object is for, where it should be actuated, and where its effect should be applied—while executing precise physical interactions such as stable functional grasps, coordinated arm–hand motion, and controlled force application. This calls for guiding physical skill learning with semantic reasoning.

General object manipulation methods fall short for CD-FOM. Learning from Demonstration methods acquire dexterity from expert teachers, while semantic understanding is implicitly encoded in their behavior [8]–[11]. However, learning the correlation between semantics and dexterity from demonstrations is difficult because it requires large amounts of data collected through teleoperation of complex multi-fingered hands to actuate objects with internal mechanisms [10]–[13]. Recent imitation-from-human-videos methods remove the need for labor-intensive teleoperation, but instead require overcoming human–robot morphological differences with limited object-specific strategies [7], [14]–[16]. Alternatively, optimization-based approaches such as reinforcement learning [17], [18] and analytical grasp synthesis [19]–[23] achieve the physical dexterity required for CD-FOM without demonstrations. However, their lack of semantic understanding requires external object-specific guidance in the form of reward design and optimization objectives, limiting their applicability and autonomy [2], [5], [24]–[26]. General semantic understanding can instead be obtained from large-scale pre-trained models such as Vision-Language Models (VLMs) [27]–[31]. However, early integrations of VLMs into robotic systems [30], [31] revealed limitations in geometric and embodied understanding, restricting their guidance to coarse, abstract levels that are insufficient for the intricate, coordinated hand–arm motions required in CD-FOM [27]–[29].

In this work we introduce CoDex, a framework that bridges semantic understanding and physical dexterity for CD-FOM through the use of semantic constraints. We define semantic constraints as a set of geometric and spatial conditions derived from an object’s function and the overall task goal. CoDex integrates a VLM into an iterative refinement procedure to achieve zero-demonstration semantic understanding, interpreting the task and generating two types of semantic constraints: local (e.g., where to press a trigger and in which direction) and global (e.g., where to aim a nozzle). These constraints guide a two-stage learning pipeline. First, analytic constrained optimization generates functionally valid grasp candidates. Second, reinforcement learning refines these candidates into a full grasp–move–actuate policy.

We demonstrate the capabilities of CoDex to operate six previously unseen objects for different CD-FOM tasks, controlling a 7-DoF robot arm equipped with a 16-DoF multi-fingered hand and achieving 73% average combined success rate. Our experiments validate that the VLM-generated semantic constraints are crucial to this performance. A human participant study demonstrates that our method for determining global constraints produces significantly more appropriate poses than prior VLM-based approaches. Furthermore, we show that our final policy learning stage is critical for achieving physical dexterity, improving functional success by over 40% when compared to using analytical grasps combined with the same VLM-generated constraints.

2 Related Work↩︎

CoDex bridges semantic understanding and physical dexterity for compositional dexterous functional object manipulation (CD-FOM). We position our work relative to three key research areas.

Semantic Understanding via Vision-Language Models. Vision–Language Models (VLMs) provide strong zero-shot semantic understanding for robotic tasks, allowing robots to interpret goals from language and visual context [27]–[31]. However, VLM outputs are typically abstract and lack the geometric precision required for dexterous manipulation [27]–[29]. Recent VLM-based manipulation systems [28], [29], [32] show promise but either require extensive training data or remain limited to coarse manipulations. ReKep [31] uses VLMs to generate keypoint constraints for manipulation, while PIVOT [30] employs iterative visual prompting to refine robot actions. CoDex leverages VLMs to generate concrete semantic constraints, both local (actuation and function points) and global (target poses), that directly inform constrained optimization and policy learning, effectively translating abstract VLM knowledge into CD-FOM policies without requiring task-specific training data.

Functional Object Grasping and Physical Dexterity. In task-oriented grasping, the aim is to select a grasp that not only stabilizes the object but also facilitates the intended function [4], [5], [25], [26], [33]–[35]. Several methods focus on optimizing contacts to achieve stable grasps, often predicting force-closure metrics like the Ferrari-Canny [25], [34]–[36], but cannot be applied to grasps that enable the actuation of objects’ internal degrees of freedom. Recent analytical strategies [4], [26], [33] consider internal degrees of freedom but they do not compose them with post-grasping trajectories to actuate the object at the right location to achieve a task. All these methods aim to provide a grasp synthesis solution that generates a successful grasp from images to be executed by a predefined controller, which can lead to failures. Recently, some methods have integrated a simulator with a model of the specific object into the loop for online improvement of grasping strategies using reinforcement learning or exploiting the simulator’s differentiability for optimization [2], [25], [37]–[39]. While demonstrating better performance, this strategy requires manual annotation of the rewards and has yet to be extended to complex objects with internal degrees of freedom (DoF) and post-grasping motion. Moreover, all previous methods improve grasp stability and/or functionality, but treat grasping as an isolated problem, missing the opportunity to reason about it in conjunction with subsequent motion to enhance dynamics and stability during actuation.

Compositional Dexterous Functional Object Manipulation. Often, successful tool use demands composing the control of both in-hand adjustments and whole-arm extrinsic motions [6], [7], [24], [40], but most existing works on dexterous manipulation focus on one or the other. For instance, significant research addresses in-hand manipulation, focusing on fine finger coordination for tasks like object reorientation [5], [13], [24], [39], or rotating caps [2], [41], [42], without consideration for full arm motion to control the object’s overall trajectory for a task. Other works tackle extrinsic manipulation, where a grasped tool interacts with the environment, such as hammering or shoveling [5], [43], [44] on objects without internal degrees of freedom. Closer to ours, [5] provides a composed solution for arm and multi-fingered hand motion, but that method requires human demonstrations and focuses on optimizing grasp stability to resist forces from subsequent arm motion, failing to actuate the object’s internal degrees of freedom. In contrast, CoDex holistically addresses the entire grasp-move-and-actuate problem, generating a composed solution that actuates both the object’s internal and external degrees of freedom.

3 CoDex: Compositional Dexterous Functional Object Manipulation↩︎

CoDex bridges semantic understanding and physical dexterity by translating VLM reasoning into explicit geometric task constraints that can be enforced by optimization and reinforcement learning. As illustrated in Fig. 2, given a language task description \(\mathcal{L}\) and an RGB-D scene observation \(\mathit{I}\), our pipeline sequentially executes two stages: VLM-Generated Semantic Constraints and Constraint-Guided Policy Training. Constraint-guided Policy Training can be further divided into two sub-stages: Constrained Optimization and Constraint-Guided RL. Below, we detail each stage.

3.1 VLM-Generated Semantic Constraints↩︎

Given the input pair \((\mathcal{L}, I)\), this stage uses VLMs to generate semantic task constraints that guide policy learning. These constraints include: 1) local constraints (the actuation point \(p_{\text{act}}\) and function point \(p_{\text{fnc}}\) on the object, and the target point \(p_{\text{tgt}}\) on the environment) and 2) global constraints (the functional object’s target 6D pose \(T^*\)).

The process starts with Visual Task Parsing. The functional object is identified from \((\mathcal{L}, I)\) using open-vocabulary segmentation (LangSAM). The segmented functional object’s 3D mesh \(\mathcal{M}\) is then constructed using a shape reconstruction and completion method (Tripo [45]). The reconstructed meshes are scaled to match the heights measured in the scene Point Clouds un-projected from the RGB-D images.

3.1.1 Local Semantic Constraints↩︎

Next, to derive the local semantic constraints, CoDex queries a VLM to generate text descriptions of key task-relevant interaction points (e.g., “trigger”, “nozzle”) that are then linked to 2D image pixels using MoLMo [46] and unprojected into 3D by aligning the image and the reconstructed mesh using an image-based tracker (FoundationPose [47]) and matching 2D to 3D mesh points. This generates the Actuation Point \(p_{\text{act}}\) and Function Point \(p_{\text{fnc}}\). CoDex assumes that the actuation direction, \(d_{\text{act}}\), is parallel to the surface normal at \(p_{\text{act}}\), pointing into the object. Fig. 3 depicts example results of this process.

3.1.2 Global Semantic Constraints↩︎

Finally, CoDex derives global semantic constraints (the goal pose \(T^*\)) through VLM-Guided Cross-Entropy Method (VLM-CEM), an algorithm inspired by [30]. VLM-CEM leverages the VLM’s reasoning to drive an iterative pose search: at each round, the VLM is prompted with the history \(\mathcal{H}\) of previously scored candidate poses and proposes \(K\) new candidate poses expected to score higher. We render these candidates (see Fig. 10), score them with the VLM, append them to \(\mathcal{H}\), keep only an elite subset, and repeat for \(N\) rounds. As detailed in Algorithm 4, this iterative proposal-and-evaluation loop allows the VLM to perform an implicit optimization, converging on a pose that is both functionally correct and physically grounded.

3.2 Constraint-Guided Policy Training↩︎

Using the semantic constraints generated in the previous stage, this stage learns a policy \(\pi\) for the complete grasp-move-actuate sequence. Rather than learning dexterous manipulation from scratch, CoDex first uses analytic constrained optimization to synthesize a diverse set of functionally valid grasp candidates. These candidates provide structured initialization for reinforcement learning, dramatically reducing the exploration required to learn the full grasp–move–actuate behavior.

3.2.1 Analytic Constrained Optimization↩︎

This phase translates the VLM-generated local semantic constraints into concrete mathematical objectives for grasp synthesis. We first sample initial hand configurations \(q_0\) from \(\mathcal{Q}\)—the valid joint space—using inverse kinematics, biasing a finger to be near the actuation point \(p_{\text{act}}\). These samples are then refined via constrained optimization. The local constraints (\(p_{\text{act}}, d_{\text{act}}\)) directly inform the functional terms in Eq. 1 : the Actuation Pt Proximity term ensures a fingertip is within a distance \(\delta_{\text{dist}}\) of \(p_{\text{act}}\), while the Actuation Alignment term ensures the fingerpad normal aligns with \(-d_{\text{act}}\). These are optimized alongside physical constraints (stability, collision avoidance) and an objective to maximize grasp robustness, measured by the min-weight force closure metric \(l^*(q)\) [23], [36]. \[\label{eq:grasp95opt95appendix95full} {\small \!\!\begin{align} \max_{q \in \mathcal{Q}} \quad & l^*(q) \\ \text{s.t.} \quad & l^*(q) \ge l_{\min} && \text{(Min F. Closure)}\\ & s(FK_i(q)) = 0,\;\forall i \in \{1, ..., n_c\} && \text{(Surface Contact)}\\ & \sigma_j(q) \ge d_j,\;\forall \text{ collision pairs } j && \text{(Coll. Avoidance)}\\ & \exists i_{act} \in F_{act} \text{ s.t.} \\ & \quad \|FK_{tip}(q, i_{act}) - p_{act}\|_2 \le \delta_{dist} && \text{(Act Pt Proximity)}\\ & \quad n_{pad}(q, i_{act}) \cdot (-d_{act}) \ge \cos(\delta_{angle}) && \text{(Act Alignment)} \end{align}}\tag{1}\] If the optimization fails, we resample \(q_0\) and restart. This process yields a diverse set of feasible, function-aligned candidates (Fig. 5) for initializing the RL policy.

3.2.2 Constraint-Guided RL↩︎

While the analytic candidates provide functionally-aligned and statically stable starting points, they do not account for the dynamics of the full manipulation task. The goal of this stage is therefore to learn a policy \(\pi\) that learns a dynamically robust motion sequence for the entire grasp-move-actuate task. By initializing the RL policy with the optimized candidates, we significantly constrain the exploration problem, enabling the agent to focus on learning the complex dynamics of contact, movement, and actuation.

Policy Parametrization The learned policy \(\pi\) takes an observation \(o\) consisting of the relative hand pose, candidate finger joints, and the designated actuation finger index. It outputs an action vector \(a \in \mathbb{R}^{38}\) that parameterizes a motion primitive by defining targets relative to the input candidate \(q_{\text{cand}}\):

• Target Grasp Joint Offsets: \(\Delta j_{\text{grasp}} \in \mathbb{R}^{16}\).

• Target Pre-Grasp Joint Percentages: \(p_{\text{pregrasp}} \in \mathbb{R}^{16}\).

• Residual Hand Pose Offset: \(\Delta T_{\text{hand}} \in \mathbb{R}^{6}\).

Motion Primitive. These action parameters guide a multi-stage motion primitive, visualized in Fig. 7. The primitive breaks down the complex task into a sequence of simpler steps: (1) the hand first moves to a pre-contact pose near the object, (2) it then transitions to the final grasp pose, (3) the fingers close to secure the object, (4) finally, it moves the object towards the global goal \(T^*\) while simultaneously performing the required actuation. This structured, parameterized primitive makes the high-dimensional control problem tractable for RL.

Reward Function The policy is trained with PPO to maximize a unified reward function \(R\). To avoid the need for task-specific reward engineering [26], \(R\) is formulated as a normalized weighted sum of continuous shaped rewards (\(R_k\)), binary stage-completion bonuses (\(S_k\)), and a final task success bonus (\(S_{\text{success}}\)). Specifically, the shaped rewards (\(R_k\)) provide dense feedback for universally required physical behaviors: (1) grasp stability (penalizing in-hand object shifting), (2) lifting (raising the object while keeping it upright), (3) actuation alignment (minimizing the distance between the acting finger and the semantic targets \(p_{\text{act}}, d_{\text{act}}\)), and (4) directional force (applying sufficient force along \(d_{\text{act}}\)). To heavily discourage dropping the object, a penalty resets the reward to zero if significant grasp displacement is detected. The entire online training process converges in approximately one hour in ManiSkill3 simulation [48].

The result of the RL is a full policy that generates a compositional grasp-move-actuate motion for the CD-FOM task. In the final step, the RL policy is executed on the real robot, using a Franka arm with a LEAP hand.

4 Experimental Evaluation↩︎

In our experiments, we evaluate whether CoDex successfully bridges high-level vision–language understanding and low-level, physics-grounded execution. To this end, we investigate the following research questions:

Experimental Setup: We evaluate CoDex on a 7‑DoF Franka Emika Panda arm with a 16‑DoF LEAP Hand end-effector, as shown in Fig. 6. Policies are trained in ManiSkill3 with 2,048 parallel environments and then directly deployed on the real robot. We evaluate six functional manipulation tasks introduced in §3. At the start of each episode, the object is placed at a random pose on the table. We evaluate each trial with two binary criteria: (i) correct object movement (the object reaches the task-required pose), and (ii) correct actuation (the mechanism is triggered). A trial is counted as successful only when both criteria are satisfied and no additional failure occurs. For Q3, we also record grasp stability (whether the object slips from the hand during execution).

Q1 — Performance on Compositional Dexterous Functional Manipulation Tasks

In real world complete CD-FOM tasks, we compare CoDex against two baselines: Analytical+VLM-CEM and RL+PIVOT. The Analytical+VLM-CEM baseline combines the best-performing initial grasp candidate from Li et al. [23] (selected via an oracle for maximum stability, see Q3) with the global constraint from our VLM-CEM. The RL+PIVOT baseline uses the same policy training procedure as ours but uses PIVOT [30] (SE3 variant) to generate global constraints.

Figure 8 summarizes our real-world results. Our method achieves an overall task success rate of 73%, significantly outperforming the baseline’s 33% (Analytical+VLM-CEM) and 0% (RL+PIVOT). The performance improvement of our method over Analytical+VLM-CEM comes from an increased robustness in the actuation of the dexterous functional manipulation. In the analytical baseline, grasps are often sufficient for lifting and coarse transport, but small in-hand shifts during motion frequently break the precise contact needed for mechanism triggering. With our method, thanks to the robustifying effect of policy learning with RL, the robot is able to actuate the internal degrees of freedom after transport and achieves success. We quantify further this reliability gap in Q3.

The baseline RL+PIVOT achieves overall 0% success rate on entire CD-FOM tasks (although it obtains partial success) because none of PIVOT’s generated global constraints meets the task requirements (e.g., the spray bottle is not correctly aimed at the plant or the glue gun is not aimed at the wooden block). Although RL+PIVOT can often trigger the object’s internal mechanisms, it consistently fails to execute correctly the required object movement. We further analyze this global-constraint mismatch in Q2.

Failures of CoDex primarily arise in two stages of the pipeline: 1) semantic constraint generation and 2) execution of dexterous functional manipulation.

Failures of semantic constraint generation. Local constraints are generated reliably and did not cause failures in our experiments. However, failures can occur when predicting the global constraint. In some cases, the predicted goal pose is spatially incorrect despite appearing plausible in 2D renderings, e.g., in the illuminate toy task the flashlight may appear to aim at the toy in image space while being misaligned in 3D. In other cases, the predicted goal pose and the grasp are individually reasonable but jointly infeasible due to joint limits or collisions (e.g., table collision near the target pose in the illuminate toy task). Overall, these spatial mismatches and collisions resulted in incorrect object motion in 23.3% of all trials (87.5% of all failed trials).

Failures of dexterous functional manipulation. Successful actuation requires grasps that remain both stable and contact-precise throughout the grasp–move–actuate sequence. However, failures arise from sim-to-real discrepancies between training and execution, including geometry reconstructed from 2D images and differences in friction, density, and deformability. Across all experiments, actuation failed in 23.3% of trials (note that some trials include both semantic constraint failures and actuation failures). The most common failure mode is actuation inaccuracy on high-precision objects. This occurs most prominently with the flashlight used in the illuminate toy task, which requires pressing a small deformable button (radius \(<0.5\,\mathrm{cm}\)) exactly at its center using the robot’s relatively large fingers. Dexterous functional manipulation of this object failed in all trials. Less frequently, sim-to-real discrepancies cause grasp instability, such as occasional salt-grinder toppling or keyboard button slippage. Stronger domain randomization and closed-loop policies that can reactively correct contact errors are promising directions to mitigate these failures.

Q2 — Quality of VLM‑CEM Generated Constraints

While we use binary criteria to assess whether an object’s final pose is correct in Q1, some final poses are better than others (e.g., a spray bottle aimed at the edge of the plant versus its center). To evaluate the quality of our VLM-CEM procedure for generating global constraints, we asked twenty participants to rate rendered images generated from the resulting goal poses of VLM-CEM and three additional baselines:

• VLM-CEM: our keypoint-anchored sampler that generates candidate goal poses around detected interaction points on the object.

• VLM-CEM (Dir.): a variant that restricts translations sampling to be along the object’s functional axis (e.g., nozzle direction).

• PIVOT (SE3) [30]: an adaptation of PIVOT that perturbs full 6-DoF poses in image-space without explicit keypoint anchoring, often resulting in misalignment in depth or lateral offset.

• PIVOT (Trans.) [30]: akin to the original PIVOT method, searching only in 2D image-space translations based on coarse visual alignment.

We generate three multi-view images per task per method and request human ratings on a five-point scale (1 = unreasonable, 3 = acceptable, 5 = perfect).

Fig. 9 shows the human-rating results. Across all tasks, both VLM-CEM variants are ranked above the PIVOT baselines. We verify this trend with a Wilcoxon signed-rank test, obtaining \(p<0.02\) for each comparison between the two PIVOT variants and the two VLM-CEM variants.

Surprisingly, for the spray plant task, our VLM-CEM with directional exploration ablation scores significantly higher than the VLM-CEM with full translational exploration. We hypothesize this is because the directional constraint, while reducing the exploration space, effectively filters out candidate poses that might appear plausible in the 2D rendered images used for VLM scoring but are functionally misaligned in 3D (e.g., aiming near the plant but slightly off-axis). By enforcing alignment along the functionally critical nozzle direction, the directional variant ensures better geometric task relevance for the highest-scoring poses in this specific spraying task.

Manual inspection of low-scoring examples supports this interpretation: most come from PIVOT, whose image-space perturbations often introduce lateral or depth offsets relative to the interaction line. In contrast, VLM-CEM samples around detected interaction keypoints, producing goal poses that are both semantically appropriate and geometrically consistent with the task (see visual comparisons in Fig. 10).

Q3 — Benefits of Constraint-Guided RL Policy Learning

In this experiment, we isolate the contribution of compositional policy learning by evaluating its improvements in grasp stability and actuation robustness compared to the direct execution of analytical grasps generated via constrained optimization (adapted from [23]). For each object, we evaluate three analytical grasp candidates, each repeated five times with randomized object initialization. We report both the average performance across candidates and the oracle best performance, i.e., the single best candidate selected post-hoc after evaluating all candidates. Fig. 11 summarizes the results.

We observe that our constraint-guided policy learning significantly improves grasp stability, exceeding the average performance of the initial grasp candidates by over 36% and surpassing the oracle best performance (the maximum potential success achievable without refinement) by over 12%. The benefit of policy learning is even more pronounced for functional actuation: the learned policy achieves 60% higher actuation success than the average grasp candidate and crucially, over 26% higher success than the best possible outcome using only the initial grasp candidates (oracle).

Interestingly, none of the initial grasp candidates achieved stable grasping success (let alone actuation) for the challenging salt grinder, likely due to its slippery surface and geometry. However, CoDex’s policy learning stage successfully discovers a stable grasp, highlighting the method’s ability to improve even on difficult cases. This demonstrates the importance of the holistic, simulation-based policy learning stage. It allows CoDex to refine statically plausible grasp candidates into dynamically robust policies that significantly enhance functional viability compared to executing the initial grasp candidates directly, even when considering the best possible initial grasp candidate.

This stage-wise diagnosis in Q1 is consistent with Q2 and Q3: Q2 isolates the quality of global constraints, while Q3 quantifies how policy learning improves contact maintenance and actuation robustness over direct analytical execution.

5 Limitations↩︎

While CoDex demonstrates success across multiple compositional FOM tasks, it also reveals several limitations that we plan to address in future work. First, tasks requiring pinpoint contact, such as pressing tiny push-buttons, are sensitive to finger size and actuator tolerances, demanding higher accuracy in control and possibly different hand morphology to ensure reliable execution. Second, many tasks go beyond reaching a single goal pose and instead require sustained, coupled arm–hand motion—for example, actuating scissors while sliding along paper—calling for extensions toward trajectory-level constraints and closed-loop feedback. Finally, the current policy assumes a single actuation point, leaving out objects that need alternating or multi-point actuation; extending CoDex to these tasks will open new robot capabilities.

6 Conclusion↩︎

We addressed the challenging problem of zero-demonstration functional object manipulation by introducing CoDex, a framework that translates abstract VLM guidance into concrete semantic constraints. Our method enforces these VLM-generated local (e.g., actuation points) and global (e.g., target poses) constraints through a two-phase process: first, analytic constrained optimization efficiently generates a set of stable, function-aligned grasp candidates, and second, constraint-guided reinforcement learning initializes from these candidates to discover the complete, dynamically robust grasp-move-actuate policy. Our experiments on a physical robot demonstrated that this tight integration of semantic reasoning with a physics-grounded, constraint-enforcing pipeline is crucial. CoDex autonomously discovered and executed complex strategies for diverse tasks with unseen objects, validating our approach. This work represents a key step toward versatile and autonomous tool manipulation. Future directions include extending the framework to more complex object mechanisms, incorporating tactile feedback for finer control, and exploring richer, interactive VLM dialogues.

References↩︎

---

1. All authors are with Robot Interactive Intelligence Lab (RobIn), University of Texas at Austin↩︎

2. This work was supported in part by a Meta research gift. Any opinions, findings, and conclusions expressed in this material are those of the authors and do not necessarily reflect the views of the sponsors.↩︎