1 Introduction↩︎

Retargeting human-object interaction demonstrations to physics-based simulation is a long-standing problem in robotics and computer vision [1]–[6]. The goal of such retargeting extends beyond replicating body poses. Successful manipulation requires recovering correct object kinematics and contact dynamics in the target environment [7]. A common approach tracks kinematic trajectories from human demonstrations and replays them in simulation, which is effective for body parts that do not directly interact with the manipulated object, such as the torso, arms during reaching, and the wrist. However, for the hands, where most object interaction occurs, simply replaying kinematic trajectories is insufficient (Fig.1, left). For hands, the objective is not merely to reproduce human finger poses, but to reproduce the object motion and contacts that realize manipulation. However, position-based kinematic data alone cannot encode the contact forces that govern such manipulation. This observation raises a natural question: rather than pursuing increasingly precise finger tracking, can we let the hands be guided by object and contact dynamics instead?

We address this question by decoupling whole-body control into two regimes: kinematic pose-guided control for contact-free body parts and object- and contact-guided control for contact-rich hand manipulation. This decoupling reflects a fundamental asymmetry in physics-based interaction. Body motion without direct object contact is often well-described in position space, while finger motion during interaction is dominated by contact force dynamics that kinematic trajectories cannot capture. Instead of relying on finger kinematic references, we use object kinematics as an outcome-level cue of successful control, together with contact objectives to shape physical interaction.

The key component that makes this decoupling possible is the wrist. The wrist is largely free from object contact, making it amenable to kinematic guidance. At the same time, it determines the global pose of the fingers and plays a key role in both affordance and grasp success. Consider grasping a cup by its handle versus its top (Fig.1, bottom right): the wrist position and orientation differ significantly between these two affordances, while finger configurations can be learned from object-pose objectives and contact dynamics once the wrist is appropriately positioned.

Building on this wrist-centric insight, we introduce WristMimic, a whole-body control framework for affordance-aware manipulation that requires only body and wrist supervision. WristMimic positions the wrist in affordance-aware poses through wrist-specific reset boundaries and reward design, while allowing reinforcement learning to discover finger configurations through physics-based exploration, object-pose objectives, and contact dynamics without explicit finger supervision. As a consequence, this approach avoids relying on explicit finger pose supervision or complex finger-level capture data [8]–[19]. Through extensive experiments, we demonstrate that precise wrist control achieves performance comparable to existing methods without relying on finger kinematic references, while supporting finger-agnostic retargeting across different hand embodiments. Our main contributions are:

• We propose a decoupled whole-body control framework that separates kinematic pose-guided body control from object- and contact-guided finger manipulation for human-object interaction retargeting.

• We identify the wrist as a critical bridge between control of contact-free body parts and contact-rich hand manipulation, and introduce wrist-specific reset boundaries and reward design to enable affordance-aware manipulation.

• We demonstrate that precise wrist control without finger kinematics achieves performance comparable to methods using full finger pose supervision, while supporting finger-agnostic retargeting across different hand embodiments.

2 Related Work↩︎

2.1 Physics-based Motion Imitation↩︎

Physics-based motion imitation has progressed from single-clip tracking [20] to large-scale whole-body control. AMP [21] and ASE [22] introduced adversarial motion priors for stylized control without explicit reference tracking. Scaling to large motion datasets, ScaDiver [23] employed mixture of experts, while PHC [24] achieved high-fidelity imitation on AMASS [25] dataset without external forces. PULSE [26] further distilled these motor skills into a universal latent representation reusable across downstream tasks. Recent universal trackers [27]–[30] further improved generalization to unseen motions, and SONIC [31] demonstrated favorable scaling properties across data, model capacity, and compute. These works show that learning from kinematic trajectory is increasingly effective for contact-free body motion.

While locomotion imitation has been increasingly successful, extending physics-based imitation to manipulation remains challenging due to contact-rich dynamics and the difficulty of recovering interaction forces from kinematic data. Early approaches such as InterPhys [32] required task-specific reward design, limiting scalability. PhysHOI [33] introduced contact graphs for task-agnostic HOI imitation, and SkillMimic [34], [35] proposed skill-level imitation robust to noisy demonstrations. Scaling further, InterMimic [36] achieved universal whole-body HOI control through teacher-student distillation with contact-guided rewards, while OmniGrasp [37] leveraged a universal dexterous motion representation to grasp over 1200 objects without paired full-body MoCap data. TokenHSI [38] and CLoSD [39] unified diverse interactions through task tokenization and diffusion-physics coupling respectively, and Wu . [40] proposed a complete pipeline from language instructions to physically plausible finger-object interactions. At the hand level, Chen . [7] propose a hierarchical wrist-guided framework for object-centric dexterous manipulation from human motion data. GraspXL [41] further studies large-scale grasp motion generation by learning policies that produce object-conditioned grasping trajectories across diverse objects and hand configurations. In contrast, we address whole-body HOI retargeting with decoupled supervision in a single policy. DexMachina [42], ManipTrans [43], and SPIDER [44] further advance dexterous retargeting through object-state tracking, residual transfer learning, and physics-informed embodiment bridging, respectively. Despite these advances, existing approaches either rely on dense kinematic supervision of detailed finger motion or focus on hand-level manipulation without integrating full-body control.

2.2 Dexterous Manipulation with Multi-Fingered Hands↩︎

Early learning-based dexterous grasp generation benefited from differentiable contact modeling. Grasp’d [45] synthesizes multi-finger grasps by optimizing through differentiable contact-rich simulation. The availability of large-scale synthetic grasp datasets such as DexGraspNet [46] shifted the field toward data-driven prediction of stable grasp poses. Subsequent works improved coverage and multimodality with proposal-and-policy formulations [47] and generative modeling, including diffusion and transformer-based grasp synthesis [48]–[50]. Beyond purely geometric conditioning, recent methods incorporate task intent via language. DexGYS [51] enables language-guided dexterous grasp generation, while SemGrasp [52] and DexVLG [53] further align grasp representations with vision-language priors. Dexonomy [54] organizes grasps into a taxonomy and AffordDexGrasp [55] targets open-set language-guided affordances. DextER [56] introduces contact-based embodied reasoning by predicting which finger links should contact where on the object surface before generating the final grasp, improving intention alignment and enabling steerable grasp synthesis. While these methods generate text and affordance-aligned grasp poses, they focus primarily on the final grasp configuration, rather than learning physics-aware trajectories or accurate manipulation sequences.

Beyond static grasp pose generation, dexterous manipulation learning targets temporally extended, closed-loop control for contact-rich multi-fingered hands. Early successes with large-scale reinforcement learning demonstrated agile in-hand manipulation via massive simulation and domain randomization [57], [58]. Imitation learning offers an alternative supervision source: Qin . [59] transfers single-camera teleoperation to multiple hand embodiments, while EgoMimic [60] scales policy learning from egocentric human videos. Subsequent work improved robustness and horizon length, including DeXtreme [61] for sim-to-real agile in-hand skills and Sequential Dexterity [62] for chaining dexterous policies over long-horizon tasks. To better leverage contact information, RotateIt [63] integrates vision and tactile for general in-hand rotation. Recently, foundation-model-driven policies have begun to couple task-level reasoning with low-level dexterity: DexGraspVLA [64] combines a VLM-based planner with a diffusion action controller trained by imitation for robust, instruction-following dexterous grasping, and DexUMI [65] explores using human hands as a universal manipulation interface for scalable demonstration collection. EgoScale [66] shows that scaling egocentric human data can serve as predictable supervision for dexterous manipulation by pretraining on large-scale wrist and retargeted hand actions and transferring to real robots with minimal robot data. These hand-centric advances motivate our study of how to integrate dexterous manipulation into whole-body retargeting while reducing reliance on finger kinematic references.

3 Method↩︎

We train a policy network that outputs actions in a physics-based simulator to reproduce a reference human-object interaction sequence. The reproduced motion must (i) track the reference body trajectory and (ii) manipulate the object so that its motion follows the reference kinematic trajectory. For the first one, standard imitation learning is often sufficient as the objective primarily involves joint movement. However, accurate object manipulation cannot be achieved by body pose imitation alone. Motion capture is imperfect, and position-based trajectories do not encode interaction forces; for example, grasping an object with weak or strong force may correspond to identical finger positions. As a result, correct pose imitation does not necessarily imply correct manipulation. Since most manipulation-related contacts occur at the hands, we decouple supervision by contact regime: the policy tracks kinematic pose targets only for object-contact-free joints, including the body and wrists, while finger joints receive no kinematic targets and are instead learned through object motion and contact outcomes, which reflect successful physical interaction. To make this decoupled, object-driven learning formulation reliable, we treat the wrist as the bridge between contact-free body control and contact-rich manipulation. We constrain wrist behavior around contact using a contact-window with time-varying reward weights and phase-specific reset thresholds. Our pipeline is illustrated in Fig.[fig:overall_architecture] and detailed in Alg.[alg:wristmimic].

3.1 Problem Formulation.↩︎

Given reference motion data \(\{\hat{\mathbf{q}}_t, \hat{\mathbf{q}}^{\text{obj}}_t, \hat{\mathbf{c}}_t\}_{t=1}^T\) including body poses \(\hat{\mathbf{q}}_t\), object trajectories \(\hat{\mathbf{q}}^{\text{obj}}_t\), and contact labels \(\hat{\mathbf{c}}_t\), we learn a physics-based policy \(\pi\) which outputs action \(\boldsymbol{a}_t\) to generate simulated states \(\{\mathbf{q}_t, \mathbf{q}^{\text{obj}}_t, \mathbf{c}_t\}\). We formulate this as a MDP \(\mathcal{M}=\langle\boldsymbol{S}, \boldsymbol{A}, \boldsymbol{T}, \boldsymbol{\mathcal{R}}, \gamma \rangle\) with state space \(\boldsymbol{S}\), action space \(\boldsymbol{A}\), simulator transition dynamics \(\boldsymbol{T}\), reward \(\boldsymbol{\mathcal{R}}\), and discount factor \(\gamma\).

The state \(\boldsymbol{s}_t = (\boldsymbol{s}_t^p, \boldsymbol{s}_t^g)\) consists of a proprioceptive state \(\boldsymbol{s}_t^p\) from simulator and a goal state \(\boldsymbol{s}_t^g\) from the reference. The policy \(\pi(\boldsymbol{a}_t | \boldsymbol{s}_t)\) predicts actions for all 51 joints, with actions sampled from Gaussian distributions \(\mathcal{N}(\boldsymbol{\mu}_t, \sigma^2 \mathbf{I})\). Here \(\boldsymbol{\mu}_t \in \mathbb{R}^{51 \times 3}\) are predicted means and \(\sigma\) is fixed standard deviation. We optimize \(\pi\) with PPO [67] to maximize reward \(r_t = \boldsymbol{\mathcal{R}}(\boldsymbol{s}_t^p, \boldsymbol{s}_t^g)\) measuring alignment between the simulated state and the reference. Detailed policy representation is in §3.2.

3.2 Policy Representation for Decoupled Manipulation↩︎

Proprioceptive State. The humanoid follows the SMPL-X [68] configuration with 52 rigid bodies connected by 51 actuated joints, each with 3 DoFs. The 51 actuated joints are partitioned into two sets: \(\mathcal{J}_{\text{cf}}\) (object-contact-free joints) consisting of 19 body joints and 2 wrists, and \(\mathcal{J}_{\text{cr}}\) (contact-rich joints) consisting of 30 finger joints. We denote the root pose as \(\mathbf{q}^{\text{root}}_t = (\mathbf{p}^{\text{root}}_t, \boldsymbol{\theta}^{\text{root}}_t) \in \mathbb{R}^{3} \times \mathbb{R}^{6}\) and velocity as \(\dot{\mathbf{q}}^{\text{root}}_t = (\mathbf{v}^{\text{root}}_t, \boldsymbol{\omega}^{\text{root}}_t) \in \mathbb{R}^{3} \times \mathbb{R}^{3}\), where \(\mathbf{p}\), \(\boldsymbol{\theta}\), \(\mathbf{v}\), and \(\boldsymbol{\omega}\) represent position, orientation, linear velocity, and angular velocity, respectively. The 51 actuated joints have poses \(\mathbf{q}_t \in \mathbb{R}^{51 \times 3} \times \mathbb{R}^{51 \times 6}\) and velocities \(\dot{\mathbf{q}}_t \in \mathbb{R}^{51 \times 3} \times \mathbb{R}^{51 \times 3}\) with the same representation. The 52 rigid bodies are used to compute body-level contacts and interaction graph features. Object state is represented by \(\mathbf{q}^{\text{obj}}_t\) with velocities \(\dot{\mathbf{q}}^{\text{obj}}_t\). Binary contact markers \(\mathbf{c}_t \in \mathbb{R}^{52}\) indicate contact for each rigid body, and interaction graph vectors \(\mathbf{i}_t \in \mathbb{R}^{52\times 3}\) encode the spatial relationship between each humanoid rigid body and its nearest object surface point [69]–[72]. The proprioceptive state is: \[\boldsymbol{s}_t^p = \{\mathbf{q}^{\text{root}}_t, \dot{\mathbf{q}}^{\text{root}}_t, \mathbf{q}_t, \dot{\mathbf{q}}_t, \mathbf{q}^{\text{obj}}_t, \dot{\mathbf{q}}^{\text{obj}}_t, \mathbf{c}_t, \mathbf{i}_t\}.\]

Goal State. The goal state \(\boldsymbol{s}_t^g = \{g_{t,t+k}\}_{k \in K}\) encodes differences between the current state and future reference states at time horizons \(K\): \[g_{t,t+k} = \{\Delta\boldsymbol{\theta}_{t+k}, \Delta\mathbf{p}_{t+k}, \Delta\boldsymbol{\theta}^{\text{obj}}_{t+k}, \Delta\mathbf{p}^{\text{obj}}_{t+k}, \Delta\mathbf{i}_{t+k}, \Delta\mathbf{c}_{t+k}\},\] where \(\Delta\) denotes the difference between the reference state at \(k\) frames ahead and the current state, using \(\ominus\) for rotation differences and \(-\) for others. For decoupled supervision, we assign kinematic pose targets only to joints in \(\mathcal{J}_{\text{cf}}\) and exclude all joints in \(\mathcal{J}_{\text{cr}}\). No kinematic reference trajectories are provided for finger joints. As a result, finger behavior is not supervised through pose imitation, but is instead optimized indirectly through object motion and contact objectives, i.e., via the object pose targets \(\Delta\boldsymbol{\theta}^{\text{obj}}_{t+k}\) and \(\Delta\mathbf{p}^{\text{obj}}_{t+k}\), together with contact alignment rewards. The contact target \(\Delta\mathbf{c}_{t+k} \in \mathbb{R}^{23}\) consists of 21 explicit contact labels for the non-finger rigid bodies and 2 binary hand-level contact indicators (one per hand). A hand-level contact indicator is set to 1 if any rigid body belonging to that hand is in contact with the object. All goal elements are normalized relative to the root position and facing direction [24], [26], [34]–[37], [73], [74].

Reward. Our reward function follows a multiplicative structure: \[r_t = r^p_t \cdot r^{\theta}_t \cdot r^{p,\text{obj}}_t \cdot r^{\theta,\text{obj}}_t \cdot r^c_t,\] where each component takes the form \(r = \exp(-\lambda E)\), penalizing error \(E\) between simulated and reference states with component-specific weights \(\lambda\). The five components measure: joint position tracking (\(r^p_t\)), joint rotation tracking (\(r^{\theta}_t\)), object position tracking (\(r^{p,\text{obj}}_t\)), object rotation tracking (\(r^{\theta,\text{obj}}_t\)), and contact alignment (\(r^c_t\)). Following InterMimic [36], \(E^p\) and \(E^\theta\) prioritize rotation when joints are far from the object and position when joints are near.

To implement decoupled supervision, we compute pose-tracking errors (\(r^p_t\), \(r^{\theta}_t\)) only over joints in \(\mathcal{J}_{\text{cf}}\), and exclude all 30 finger joints from direct kinematic tracking. For contact alignment, we compute error over the 23 contact elements defined in the goal state (21 non-finger bodies + 2 hand-level indicators). As a result, contact-free body parts receive reward signals from kinematic references, while finger joints are influenced by object pose tracking and contact alignment. This reward structure aligns with our decoupled formulation, where finger joints are not directly supervised by pose targets but are shaped by object motion and contact alignment.

Action. Actions \(\boldsymbol{a}_t \in \mathbb{R}^{51 \times 3}\) specify target rotations for all 51 actuated joints in exponential map coordinates [36], which are converted to torques via PD controllers. Although finger joints receive no kinematic supervision in the goal state, they remain fully actuated in the action space. Their behavior is learned implicitly through object pose rewards, contact alignment, and interaction dynamics.

3.3 Learning Wrist-Guided Manipulation↩︎

After decoupling kinematic supervision for contact-free body parts from object- and contact-guided finger behavior, we identify the wrist as the structural bridge between these two regimes. While finger behavior is learned through object motion and contact outcomes, effective exploration requires the hand to be placed in an affordance-consistent and reachable configuration. Since the wrist determines the global pose of the fingers and the reachable contact set, a misaligned wrist may prevent access to the intended contact region. Moreover, as the wrist is typically free from direct object contact, it remains suitable for kinematic guidance using reference trajectories. We therefore prioritize wrist placement through selective relaxation of proximal arm supervision and wrist-specific phase-dependent reset thresholds. Finger configurations are then allowed to emerge from interaction dynamics.

Prioritized Alignment Strategy. To ensure feasible wrist placement that enables finger behavior to emerge through interaction dynamics, we adopt a prioritized alignment strategy. Motion capture data is inherently imperfect when executed in physics-based simulation, making it difficult to simultaneously align all body joints with the reference motion. For example, motion capture sequences are often defined with a floating root frame that may violate physical constraints in simulation. As a result, some joints must be allowed to deviate to achieve accurate wrist alignment. We define a contact window \([t_c - t_b, t_c + t_a]\) centered on the first contact frame \(t_c\), where \(t_b\) and \(t_a\) denote frames before and after contact. Within this window, we enforce tight wrist constraints while relaxing upper arm supervision, allowing the policy to adjust upper arm configurations freely to position the wrist optimally. This strategy is implemented through two complementary mechanisms: reward weight modulation and phase-specific reset boundaries.

Reward Weight Modulation. To prioritize wrist alignment while relaxing upper arm supervision, we modulate joint reward weights over time. Specifically, the joint rewards are formulated to exclude all 30 finger joints: \[r^\alpha_t = \exp\left(-\lambda^\alpha \sum_{j \in \mathcal{J}_{\text{cf}}} w_j(t) E_j^\alpha\right) \quad \text{where } \alpha \in \{p, \theta\},\] where \(\mathcal{J}_{\text{cf}}\) denotes the 19 body joints and 2 wrist joints, and \(w_j(t)\) are time-varying weight factors defined as: \[w_j(t) = \begin{cases} 0, & \text{if } j \in \mathcal{J}_{\text{arm}},\;t \in [t_c - t_b, t_c + t_a], \\ 1, & \text{if } j \in \mathcal{J}_{\text{wrist}},\;t \in [t_c - t_b, t_c + t_a], \\ w_{\text{red}}, & \text{if } j \in \mathcal{J}_{\text{cf}} \setminus (\mathcal{J}_{\text{arm}} \cup \mathcal{J}_{\text{wrist}}),\; t \in [t_c - t_b, t_c + t_a], \\ 1, & \text{otherwise}. \end{cases} \label{eq:reward95weight95modulation}\tag{1}\] where \(\mathcal{J}_{\text{arm}}\) denotes the shoulder and elbow joints of the grasping hand(s), \(\mathcal{J}_{\text{wrist}}\) denotes the wrist joints, and \(w_{\text{red}} < 1\) is the reduced weight factor. This selective relaxation removes proximal arm constraints (\(w_j=0\)), keeps wrist supervision unchanged (\(w_j=1\)), and reduces the weights of the remaining contact-free body joints (\(w_j=w_{\text{red}}\)) during the contact window. Moreover, reducing other body joint weights (\(w_j=w_{\text{red}}\)) allows the policy to focus on hand-object contact and object dynamics in contact window. Specific weight values are detailed in the supplementary material.

Phase-Specific Reset Thresholds. We employ early termination to prevent inefficient exploration when episodes diverge from feasible trajectories. Specifically, episodes terminate if: (i) body pose deviates severely from reference, (ii) root height drops below a threshold (indicating fall), (iii) object pose drifts too far from reference, or (iv) contact error persists for multiple frames [36], [37]. In addition to this base reset strategy, we apply phase-specific reset thresholds to the wrist of grasping hand(s) within the contact window. We subdivide the contact window into three phases centered on \(t_c\): approach \([t_c - t_b, t_c + \tau_1]\), grasping \([t_c + \tau_1, t_c + \tau_2]\), and stabilization \([t_c + \tau_2, t_c + t_a]\), where \(0 \leq \tau_1 < \tau_2 \leq t_a\) define phase transitions after first contact. During the grasping phase, we apply tight thresholds of 7cm in position and 0.2 radians in orientation, terminating if wrist deviates beyond these bounds from reference (Fig.1, top right). The approach and stabilization phases use looser thresholds of 15cm and 0.5 radians, allowing trajectory exploration while maintaining approximate alignment. This design—using reduced reward weights with strict reset boundaries rather than high reward weights—enables the policy to discover optimal wrist poses within a feasible search space, accommodating mocap inaccuracies while ensuring natural grasp configurations.

4 Experiments↩︎

4.1 Experimental Setup↩︎

Datasets and Evaluation Metrics. We evaluate WristMimic on two human–object interaction datasets: ParaHome [76] and OMOMO [75]. These datasets provide object and subject geometries, human motion trajectories represented via SMPL-X [68] or bone vectors, and corresponding object trajectories. Each episode spans approximately 100–250 frames. We train per-scene policies on 20 representative sequences from ParaHome and OMOMO respectively. This setup allows us to evaluate the robustness of our wrist-guided, object-kinematics-driven approach across diverse manipulation scenarios. We assess performance using three primary metrics across 10,000 rollout trials per scene:

• Success rate (%): A trial is successful if the sequence reaches completion without early termination, the average object position error remains below 10 cm, and the robot maintains contact for at least 80% of the reference contact duration.

• Object position/rotation error (cm, \(°\)): Aligning with our principle that manipulation imitation should prioritize object trajectory over exact finger replay, we measure the average position/rotation deviation of the object from its reference trajectory throughout the sequence.

Implementation Details. We implement WristMimic in Isaac Gym [77] and train all policies using Proximal Policy Optimization (PPO) [67]. Training uses 2048 parallel environments on a single NVIDIA RTX 3090 GPU with 30 Hz control frequency. The policy network consists of separate actor and critic MLPs with SiLU [78] activations. Contact window parameters are set to \(t_b = 10\), \(t_a = 15\) frames, with phase transitions at \(\tau_1 = -2\) and \(\tau_2 = 12\) frames relative to the first contact frame \(t_c\). Complete hyperparameters are provided in the supplementary material.

4.2 Quantitative Results↩︎

We compare WristMimic against InterMimic [36] and SkillMimicV2 [35]. For fair comparison, we train per-scene policies for all methods. Evaluation is conducted on 40 sequences in total, consisting of 20 sequences from ParaHome [76] and 20 from OMOMO [75]. For SkillMimicV2, evaluation is performed only on ParaHome, since SkillMimicV2 requires bone-vector representations, which are available only in ParaHome.

Quantitative results are shown in Tab.[tab:sota_comparison]. WristMimic achieves the best performance across all evaluation metrics despite not relying on explicit finger pose supervision. In particular, on the OMOMO dataset [75], WristMimic reduces both object position and rotation errors by nearly half compared to recent methods. These results indicate that guiding manipulation through wrist placement and object kinematics provides a strong structural signal for object-trajectory–aware control.

On the ParaHome dataset [76], both InterMimic and SkillMimicV2 exhibit significantly lower success rates. Notably, while InterMimic shows a clear performance gap between OMOMO and ParaHome, WristMimic maintains relatively consistent performance across the two datasets. Since ParaHome contains more dexterous interaction scenarios such as grasping small handles or objects with narrow contact regions, these results suggest that wrist-guided constraints provide robust structural guidance for delicate grasping.

4.3 Qualitative Results↩︎

We present representative qualitative results comparing our method with InterMimic [36] on six scenes from the ParaHome dataset (Fig.[fig:qualitative_results]). As discussed in §4.2, previous methods often struggle in these dexterous manipulation scenarios. In particular, as illustrated in Fig.[fig:wrong_wrist_correct_finger], even when SkillMimicV2 produces visually plausible finger configurations, incorrect wrist placement prevents stable object grasping. This example highlights that correct wrist placement is a prerequisite for establishing feasible contact configurations for grasping. In contrast, our approach first establishes a feasible wrist configuration, allowing finger poses to naturally adapt to the object and enabling both stable grasping and accurate tracking of the reference object trajectory. These examples demonstrate that guiding manipulation through wrist placement enables affordance-consistent grasping in dexterous interaction scenarios.

Fig.[fig:seq_qual] further shows sequential frames sampled at 50-frame intervals from two representative manipulation sequences. WristMimic maintains stable object contact while producing natural hand poses throughout the sequence, demonstrating consistent manipulation behavior over time.

In the Move Book scenes in Fig.[fig:qualitative_results] and Fig.[fig:seq_qual], a slight deviation from the ground truth motion can be observed. The human demonstration initially lifts the book using both hands before releasing one hand, whereas our policy continues to support the book with both hands due to grasping stability. This behavior reflects our method’s design choice to prioritize stable object manipulation and trajectory tracking over strict imitation of the human body pose.

4.4 Ablation Study↩︎

Ablation on Wrist Constraints. We evaluate the effect of wrist-specific constraints on 8 representative ParaHome [76] sequences. As shown in Tab.[tab:wrist_ablation], the decoupled formulation without wrist constraints performs significantly worse, indicating that decoupling alone is not sufficient for reliable manipulation. We further decompose our wrist-centric design into two components: reward weight modulation within the contact window and phase-specific wrist reset thresholds. Phase-specific resets provide the main improvement by keeping exploration near feasible wrist poses, while reward weight modulation alone gives only marginal gains. However, combining reward modulation with phase-specific resets yields the best overall performance, substantially improving success rate and object pose accuracy. These results show that WristMimic’s gains come from explicitly guiding wrist placement during contact-rich phases, which is critical for stabilizing hand–object interactions and achieving affordance-consistent grasps.

Ablation on Reset Threshold. We further ablate the grasping-phase wrist reset thresholds on four representative ParaHome [76] scenes. As shown in Tab.[tab:reset_thr_ablation], 3.5 cm/0.1 rad (tight) is too restrictive for exploration and pre-grasp recovery, while 15 cm/0.5 rad (loose) admits implausible wrist states that destabilize contact. Our default 7 cm/0.2 rad achieves the best trade-off between feasible exploration and reliable wrist placement.

Ablation on sequences of non-hand-contact bodies. Fig.[fig:non_hand_body_contact] and Tab.[tab:non_hand_body_contact] present ablation studies on non-hand body contact scenarios, where body parts other than the hands serve as the primary contact regions. Unlike hand-object manipulation, these interactions mainly involve coarse body-level support or displacement, and thus can be handled by the standard whole-body mimicking objective without additional task-specific design. The results show that such non-hand contacts do not require any wrist-like design, allowing our framework to extend naturally to scenarios such as sitting on a chair and displacing a small table with the foot.

Hand Morphology Generalization. Tab.[tab:ablation_hand_morphology] demonstrates that our finger-agnostic formulation enables learning across different hand morphologies without requiring finger-specific motion data. We retrain policies using three hand configurations: the default scene-specific SMPL-X hand, the hand model from InterMimic [36], and the OmniGrasp [37] hand, evaluated on 10 sequences from ParaHome [76]. These configurations differ substantially in hand size, joint lengths, and initial wrist and finger poses. Despite these differences, grasping remains successful across all configurations, suggesting that wrist guidance provides a strong structural signal for affordance-aware grasping.

Fig.[fig:hand_generalization] visualizes representative successful manipulations with the three hand types. All policies are trained independently using the same wrist-guided strategy without explicit finger pose supervision. As shown in Tab.[tab:ablation_hand_morphology], performance remains comparable across different hand morphologies, indicating that the policy adapts to varying hand geometries through interaction dynamics rather than kinematic supervision. Among the three configurations, the InterMimic [36] hand exhibits the most realistic finger poses and highest performance due to its constrained joint ranges that prevent physically implausible poses while other hands set joint range as \([-\pi, \pi]\). Since the joint range-constrained setting is more realistic in the real world, this shows that the wrist-only approach can make natural grasping poses emerge while maintaining grasping quality.

4.5 Limitations↩︎

Our method focuses on affordance-aware grasping and object manipulation rather than precise in-hand manipulation that requires detailed finger pose control. While wrist-guided learning enables the policy to discover natural grasp configurations through interaction dynamics, tasks involving fine-grained finger repositioning or object reorientation within the hand remain beyond the current scope. Extending the framework to support such in-hand manipulation without explicit finger supervision is an interesting direction for future work.

Another limitation is that our current framework trains scene-specific policies, requiring separate training for each scene. This contrasts with methods such as OmniGrasp [37], InterMimic [36], and MaskedManipulator [73], which aim to generalize across scenes. Nevertheless, our experiments show that the wrist-guided formulation generalizes across different hand morphologies within the same scene, suggesting that the learned interaction structure is not tied to a specific hand model. Developing scene-general policies that retain this wrist-centric interaction structure remains an important direction for future research.

5 Conclusion↩︎

We have presented WristMimic, a wrist-guided framework for physics-based retargeting of human–object interaction demonstrations. Our approach decouples supervision between contact-free body motion and contact-rich hand manipulation, and highlights the wrist as a key bridge between these two regimes. Because the wrist largely remains contact-free while determining the global configuration of the fingers, guiding the wrist provides a useful structural signal for learning manipulation behaviors without explicit finger pose supervision. Through experiments across multiple datasets and hand morphologies, we show that wrist-guided learning can achieve grasping and object manipulation performance comparable to methods that rely on detailed finger supervision. These results suggest that emphasizing wrist guidance may offer a practical direction for reducing reliance on dense finger motion capture in physics-based manipulation learning. More broadly, this perspective suggests that manipulation policies can be learned by focusing on structurally important joints rather than densely supervising all degrees of freedom.

Acknowledgements↩︎

This work was supported by Samsung Electronics (IO201208-07822-01) and also by IITP grants (RS-2022-II220113: Developing a Sustainable Collaborative Multi-modal Lifelong Learning Framework (50%), RS-2022-II220290: Visual Intelligence for Space-Time Understanding & Generation (25%), RS-2024-00457882: National AI Research Lab Project (20%), RS-2019-II191906: AI Graduate School Program at POSTECH (5%)) funded by Ministry of Science and ICT, Korea.

6 Additional Results↩︎

6.1 Ablation on Finger Guidance↩︎

4pt

Table [tab:supp_parahome_quantitative] reports results on the extended 20-sequence ParaHome split, complementing the 8-sequence ablation in Tab. 2 of the main paper and adding a variant with explicit finger-tracking guidance. WristMimic achieves superior performance across all three metrics compared to the decoupled baseline without wrist constraints (Decoupled). Interestingly, WristMimic which does not use explicit finger kinematic trajectories in either the state or reward formulation, also outperforms the variant with additional finger tracking guidance. This observation suggests that prioritizing object kinematics and contact outcomes can provide a stronger learning signal than directly supervising high-dimensional finger poses. One possible explanation is the extremely high degrees of freedom of the hands (45 DoFs per hand), where enforcing finger pose tracking may introduce large pose deviations or unintended wrist misalignment while satisfying finger-related rewards. As a result, focusing on wrist placement together with object-centric objectives leads to more stable and effective manipulation behavior.

6.2 Ablation on Embodiment Generalization↩︎

Figure [fig:supp_hand_embodiment] presents additional experimental results showing robots with different hand morphologies grasping a variety of objects, supplementing Fig. 6 in the main paper. WristMimic consistently produces stable grasp configurations across diverse hand embodiments, demonstrating that the learned manipulation strategy remains robust despite variations in hand geometry.

6.3 Qualitative Results on OMOMO dataset↩︎

Figure [fig:supp_omomo_qual] demonstrates qualitative results of our method on OMOMO [75] dataset. In contrast to the reference motion exhibiting unstable grasping and floating hand artifacts, our method produces accurate grasping poses that are faithful to object dynamics.

7 Simulation Setup↩︎

7.1 Physical Assets↩︎

Humanoid. We use scene-specific SMPL-X [68] to generate humanoid bodies following OMOMO [75] and InterMimic [36]. Physical parameters in  Tab.[tab:humanoid_params] use higher stiffness for torso and spine (1000) for stability and lower stiffness for fingers (100) for compliant grasping.

Objects. Static objects in ParaHome(table, desk, gasstove, bookshelf, sink) have fixed base links. Dynamic objects use provided meshes as visual geometry with collision meshes generated by uniformly sampling 20% vertices. Parameters in  Tab.[tab:object_params] show that we set max_convex_hulls to 80 for hollow-handled objects (kettle, cup), 50 for structurally complex furniture (chair), and 10 for simpler objects.

7.2 Simulator Configuration↩︎

We use Isaac Gym [77] with PhysX backend. Key parameters in  Tab.[tab:simulator_config] include num_position_iterations set to 10 (vs. default 4) to prevent object penetration during grasping, which is critical for stable contact-rich manipulation.

8 Training Details↩︎

Training configuration. We train with 2048 parallel environments using minibatch size 8192 and horizon length 32. Details are in  Tab.[tab:training_config].

Reward weights. Our reward function follows the multiplicative form \(r = \exp(-\lambda E)\), where \(E\) is the tracking error and \(\lambda\) is the reward weight. See Tab.[tab:reward_weights] for detailed reward weights.

Design rationale. During the contact window, we reduce body weights (gbp: 30\(\rightarrow\)​10, gbr: 2.5\(\rightarrow\)​1) and zero upper arm joints to focus on contact dynamics. Wrist weights stay constant (gwp = 10, gwr = 1) across all phases, with stage-specific reset boundaries providing the primary guidance. The grasping phase uses strict thresholds (7cm position, 0.2 rad orientation), while approach and stabilization phases allow more exploration (15cm, 0.5 rad).

References↩︎