1 Introduction↩︎

Hand-object interaction (HOI) lies at the core of embodied intelligence, enabling humans to manipulate tools, operate daily objects, and perform complex dexterous skills. A defining characteristic of human dexterity is the continuous regulation of contact through tactile feedback, which governs contact formation, force modulation, and manipulation stability. While vision provides spatial and geometric awareness [1]–[3], tactile perception determines how physical interaction is established and maintained at the hand-object interface. Neuroscience and robotics research consistently show that successful manipulation relies heavily on force modulation rather than mere spatial positioning [4]–[7]. Consequently, human dexterity is not solely a function of kinematic trajectories, but fundamentally grounded in contact-level physical interaction.

Transferring such human dexterity to robotic systems remains a longstanding challenge. Recent advances in dexterous manipulation have explored human-to-robot skill transfer through imitation learning [8]–[12], motion retargeting [13]–[16], and reinforcement learning frameworks [17], [18]. These approaches typically align human hand kinematics with robotic embodiments, leveraging visual observations and joint-level motion supervision to reproduce task behaviors. Despite impressive progress, this trajectory-centric paradigm inherently overlooks the tactile dynamics essential for regulating contact formation and force distribution. Consequently, robotic policies often successfully replicate superficial movement patterns, but deviate significantly in contact behavior, interaction stability, and force modulation.

A key factor underlying this gap is the absence of tactile-rich benchmarks and standardized evaluation protocols. Without strictly synchronized tactile-kinematic demonstrations, learning objectives are implicitly forced to prioritize motion matching over physically grounded interaction, leaving a substantial gap in achieving true human-like robotic dexterity.

To address this gap, we introduce TactiDex, a real-world tactile-rich HOI dataset for contact-aware human-to-robot dexterous transfer. TactiDex provides synchronized whole-hand tactile sensing, fine-grained hand kinematics, wrist 6D poses, and object 6D trajectories, forming a temporally aligned and contact-rich representation of human manipulation dynamics. Unlike prior datasets that focus primarily on visual observations or joint trajectories [19]–[24], TactiDex captures contact interactions between the hand and object through spatially distributed pressure measurements. The dataset spans diverse single-hand and bimanual manipulation tasks, covering varied object geometries and interaction patterns, thereby providing a broad testbed for studying tactile-aware transfer. Beyond data collection, we establish standardized contact-aware evaluation protocols that quantify contact fidelity, force alignment, and interaction stability. By integrating synchronized multi-granularity tactile and spatial states within a unified framework, TactiDex establishes a benchmark tailored to studying tactile-aware human-to-robot transfer.

Building upon this data paradigm, we propose TactiSkill, built on the Tactile-Guided Tri-Component Transfer Framework that leverages synchronized contact signals to guide robotic policy learning. Directly incorporating raw pressure measurements into reinforcement learning objectives, however, presents non-trivial challenges. Naïve formulations based solely on contact force matching often lead to unstable optimization, reward exploitation, or physically unrealistic behaviors in simulation [25]–[29]. To address this, we introduce a tri-component tactile objective conceptualizing tactile signals as structured supervision, which integrates (1) contact guidance to encourage meaningful and timely contact formation, (2) human-like alignment to match force distribution patterns observed in human demonstrations, and (3) safety constraints to regularize excessive force and promote stable interaction. This structured formulation enables stable optimization while preserving contact-level fidelity, effectively moving dexterous transfer beyond kinematic mimicry toward physically grounded human-likeness.

In summary, our contributions are as follows:

(1) We introduce TactiDex, the first high-precision, long-horizon tactile-rich hand–object interaction dataset for dexterous bimanual manipulation, providing temporally synchronized whole-hand tactile signals, fine-grained hand kinematics, and object 6D trajectories, together with tactile-aware evaluation metrics for human-to-robot transfer.

(2) We propose TactiSkill, a tactile-guided human-to-robot transfer framework that leverages tactile sensing as a core supervisory signal for dexterous policy learning. To stabilize contact-rich interaction, we design a tri-component tactile reward framework that (i) encourages proactive contact formation, (ii) aligns tactile distributions with human demonstrations to achieve force-level human-likeness, and (iii) constrains excessive contact forces to ensure physically stable manipulation.

(3) Extensive experiments demonstrate that tactile-guided transfer significantly improves geometric robustness and physical fidelity across diverse manipulation tasks, and we further validate the effectiveness of TactiSkill through real-world dexterous hand deployment, showing that policies derived from our dataset can reproduce stable contact-rich manipulation on physical hardware.

2 Related Work↩︎

2.1 Hand-Object Interaction Dataset↩︎

Numerous datasets facilitate hand-object interaction (HOI) modeling [19]–[24], [33], [34]. Motion-capture datasets [19], [20], [22], [31] provide high-precision 3D trajectories for detailed manipulation analysis, while egocentric datasets [23], [35] expand semantic diversity in open-world and instruction-conditioned settings. However, both paradigms primarily infer contact from geometric proximity or visual observation, leaving actual force modulation physically unmeasured. Synthetic benchmarks [30] offer scalable contact signals but rely on simulated geometry, lacking real-world tactile fidelity. Most recently, OPENTOUCH [32] introduced real-world whole-hand tactile sensing with text annotations; yet, it predominantly focuses on egocentric perception tasks (e.g., cross-modal retrieval and grasp classification) rather than dynamic manipulation control. Consequently, as summarized in Table 1, existing HOI datasets largely lack the tactile grounding necessary for physically interactive policy learning. To bridge this gap, we introduce TactiDex, a tactile-grounded HOI benchmark that synchronizes whole-hand tactile sensing with multi-granularity kinematic states and establishes standardized, contact-aware evaluation protocols, enabling physically grounded modeling of human dexterity beyond geometric imitation.

2.2 Dexterous Manipulation & Human-to-Robot Transfer↩︎

Dexterous manipulation has witnessed substantial progress through model-based control, reinforcement learning, and large-scale policy learning [17], [18], [36]–[38]. Concurrently, learning from human demonstrations offers an intuitive pathway for transferring dexterity to robotic hands [8], [9]. Methods such as imitation learning and motion retargeting align trajectories under morphological constraints [13], [15], while recent RL frameworks refine these transferred policies using reference motions and task-specific rewards [10], [39]–[42]. Although these methods significantly improve trajectory reproduction, contact-level discrepancies frequently persist. Geometric alignment does not guarantee consistency in force modulation or pressure distribution, often resulting in unstable grasps, excessive force, or physically implausible interactions during delicate, contact-rich tasks. Therefore, while trajectory imitation is a strong starting point, it fails to fully resolve the physical interaction gap. In contrast, our approach introduces tactile-guided supervision that directly constrains contact formation and force distribution, shifting the human-to-robot transfer paradigm from purely geometric trajectory alignment to physically grounded interaction alignment.

2.3 Tactile-Guided Policy Learning↩︎

Tactile sensing is increasingly recognized as a critical modality for robust manipulation. Early tactile-guided reinforcement learning (RL) applications predominantly utilize parallel-jaw grippers to solve foundational tasks like peg-in-hole insertions and terminal assemblies [43]–[46]. To improve generalizability, subsequent studies extensively explore visuo-tactile fusion and sim-to-real transfer paradigms to address the reality gap during unknown object manipulation [47]–[54]. More recently, tactile RL has scaled to highly articulated multi-fingered dexterous hands [26], [55], accompanied by the development of multimodal pretraining benchmarks [25] and attempts to leverage human tactile demonstrations for imitation learning [56]. Despite this rapid progress, existing frameworks largely treat tactile signals merely as auxiliary observation vectors for unconstrained trial-and-error optimization, suffering fundamentally from the absence of synchronized, whole-hand hand-object interaction (HOI) physical datasets. To bridge this gap, our work introduces a true human-to-robot tactile transfer paradigm by leveraging the real-world TactiDex dataset. Instead of learning contact heuristics from scratch, TactiSkill formulates high-fidelity human force distributions not as passive observations, but as active, structural physical priors. By directly supervising the force modulation of the robotic hand, our framework ensures the resulting policies transcend geometric mimicry to achieve physically grounded, human-like dexterity.

3 TactiDex Dataset↩︎

TactiDex, as overviewed in Fig. 1, is a large-scale, real-world tactile-grounded dataset designed to capture the physical dynamics of human dexterous manipulation. It synchronizes whole-hand tactile sensing, articulated hand kinematics, wrist 6D poses, and object 6D trajectories within a unified motion-capture environment. The dataset comprises 49 calibrated objects and 757 interaction sequences, spanning from fundamental pick-and-place behaviors to complex functional operations and object-object bimanual interactions. By providing structured interaction annotations and task-level textual descriptions alongside multimodal sensory streams, TactiDex establishes a physically measurable and semantically grounded benchmark for contact-aware human-to-robot transfer.

3.1 Hardware Setup↩︎

To develop the TactiDex dataset, we built a multimodal data collection system for tactile-grounded hand–object interaction that synchronously records hand contact information, hand kinematic states, and accurate object 6D poses in a global coordinate frame. The system consists of three hardware components: a full-hand tactile glove that captures pressure distributions, a hand-tracking glove that acquires articulated joint states, and an OptiTrack motion capture system that provides global 6-DoF poses for both the object and the hand.

3.1.1 Tactile Measurement.↩︎

The tactile gloves measure pressure via piezoresistive changes in flexible sensors. The tactile array comprises 162 sensing elements densely distributed across the fingertips and palm, with a force resolution of up to 0.01 N and a sampling rate of 17 Hz. We further redesigned the structure of tactile gloves and wearing configuration so that it can be worn externally over the motion capture glove. This modification enables stable acquisition of tactile signals from the whole hand without degrading the kinematic tracking quality of the mocap glove, while also minimizing interference with natural manipulation, improving reliability and consistency for long-duration data collection.

3.1.2 Motion Capture.↩︎

Hand kinematics are captured using a motion-tracking glove, where IMU measurements are fused with optical tracking data to estimate high-fidelity articulated joint states. Global 6-DoF poses for both the wrist and the objects are provided by an OptiTrack system equipped with 8 PX13W cameras operating at 120 fps within a 1.2 m \(\times\) 1.8 m workspace. As both hand and object poses are processed within the same motion-capture environment, they are inherently aligned within a unified spatial coordinate system and temporal base.

3.2 Data Collection and Processing↩︎

Synchronized Multimodal Capture. To ensure high-fidelity data acquisition, we established a strict collection protocol. Participants perform manipulation tasks within the calibrated multi-camera optical workspace. The hardware-synchronized pipelines ensure frame-level temporal alignment between the 120 Hz kinematic/pose streams and the tactile pressure data, enabling precise temporal modeling of contact formation and evolution.

Object Modeling and Calibration. The 49 objects in our dataset were meticulously selected to cover diverse geometries, scales, and physical properties. Object meshes are reconstructed using a hybrid pipeline that integrates high-resolution mobile 3D scanning applications with SAM3D-based reconstruction methods [57]. To overcome the inherent feature-matching failures caused by reflective, transparent, or highly symmetric surfaces, we applied specialized non-reflective textured tapes and temporary fiducial markers during the scanning phase. This rigorous treatment ensured robust visual feature extraction, yielding high-fidelity triangular meshes regardless of challenging material properties. Following reconstruction, each object is dynamically calibrated within the OptiTrack system to ensure its 6D trajectory is spatially consistent with the hand’s global coordinate frame.

Task Coverage and Data Scope. We recorded 757 interaction sequences encompassing single-hand and bimanual manipulation. Task complexity ranges from basic physical interactions (e.g., pipkplace, inspect) to tool use and functional operations (e.g., click, cut, shake). Each sequence pairs the synchronized multimodal sensory data with structured annotations, including interaction-phase timestamps, task identifiers, and natural-language descriptions, providing a comprehensive foundation for multimodal policy learning.

Optimization and Processing. Following OMOMO [58], we adopt the MANO model [59] for kinematic representation. We independently fit MANO parameters to the raw 3D motion-capture keypoints via L-BFGS optimization. To ensure anatomical and temporal consistency, we share shape parameters across all frames for each subject, and apply both an \(\ell_2\) pose regularizer and a second-order temporal smoothness penalty to suppress high-frequency jitter.

While kinematic fitting captures accurate free-space trajectories, residual errors often cause physically implausible “floating” or penetration during object interaction. To resolve this, we introduce a two-stage tactile-constrained post-optimization. In the first stage, we identify genuine contact intervals by gating geometric proximity with measured tactile pressure, effectively filtering out visually deceptive near-contacts. Within each interval, we select a stable reference grasp frame and optimize its spatial alignment using tactile-modulated per-finger weights: active fingers (exceeding the pressure threshold) are strongly attracted to the object surface, while inactive fingers are down-weighted to prevent phantom contacts. In the second stage, this refined grasp is propagated across the contact interval via temporal inverse kinematics, preserving original motions in non-contact regions. Finally, a collision-aware adjustment optimizes finger articulation against the object’s Signed Distance Field (SDF) [60] to eliminate residual penetrations. This pipeline yields the final TactiDex annotations, ensuring that interactions are both kinematically precise and physically grounded by real-world tactile evidence.

4 Methodology↩︎

As illustrated in Fig. 2, the core objective of our framework is to leverage the TactiDex dataset to train dexterous hands in simulation, enabling them to replicate both the spatiotemporal kinematics and the dynamic contact patterns of human demonstrations.

4.1 Overview & Problem Formulation↩︎

To achieve physically grounded transfer, we model the contact-aware dexterous manipulation as a constrained Markov Decision Process (MDP), denoted by the tuple \(\mathcal{M} = \langle \mathcal{S}, \mathcal{A}, \mathcal{P}, \mathcal{R}, \gamma \rangle\). Unlike traditional kinematic-centric setups, we explicitly incorporate high-fidelity tactile signals as Target Contact Features into the state space to enforce physical realism:

• State Space \(\mathcal{S}\): At timestep \(t\), the state \(\mathbf{s}_t \in \mathcal{S}\) includes standard proprioception (joint positions \(\mathbf{q}_t\), velocities \(\dot{\mathbf{q}}_t\)) and object states (pose \(\mathbf{p}_t\)). Crucially, we introduce a Target Tactile Reference denoted as \(\mathbf{F}_t^{\text{human}}\) derived from the TactiDex, which guides the robot on the exact normal force each finger should apply at the current moment.

• Action Space \(\mathcal{A}\): The action \(\mathbf{a}_t = \Delta \mathbf{q}_t\) is defined as the residual joint position commands, serving to fine-tune the base kinematic trajectory.

The optimization objective is to find an optimal policy \(\pi^*(\mathbf{a}_t | \mathbf{s}_t)\) that minimizes kinematic tracking errors while maximizing force consistency with the human demonstration.

4.2 Tactile-Guided Tri-Component Human-to-Robot Transfer↩︎

To effectively transfer human demonstrations to morphologically distinct robotic hands, we propose TactiSkill, a novel policy learning method built upon the TactiDex dataset. As depicted in the central block of Fig. 2, TactiSkill operates through our proposed Tactile-Guided Tri-Component Transfer Framework.

Building upon recent residual reinforcement learning architectures designed for morphological retargeting [10], our final policy \(\pi^*\) is composed of a frozen kinematic imitation policy \(\pi_{\text{base}}\) and a learnable residual policy \(\pi_{\text{res}}\). However, instead of acting merely as a geometric compensator to fix trajectory errors, TactiSkill fundamentally re-purposes \(\pi_{\text{res}}\) as a Force Modulator. By actively inducing and regulating simulated contact forces, \(\pi_{\text{res}}\) bridges the gap between geometric motion mimicry and true physical interaction through a tightly integrated three-step pipeline.

(1) Finger-wise Sensor-to-Sim Data Alignment.
The TactiDex dataset captures raw sensor signals (high-frequency ADC readings), whereas the physics simulation engine computes contact forces in Newtons based on rigid-body dynamics. To establish a unified physical baseline for supervision, we construct a finger-wise, non-linear calibration mapping function \(\mathcal{M}: \mathbb{R}^{\text{raw}} \rightarrow \mathbb{R}^N\). This mapping precisely converts the raw human demonstration signals into aligned normal forces consistent with the simulation environment.

(2) Asymmetric Observation with Privileged Tactile Information.
To enhance training stability and sample efficiency, TactiSkill employs an asymmetric Actor-Critic architecture. The Actor only receives the target tactile reference \(\mathbf{F}_t^{\text{human}}\), mimicking the real-world deployment constraint where the robot possesses a desired force reference but cannot directly foresee the resulting contact dynamics. Conversely, the Critic receives privileged information—specifically, the real-time simulated fingertip contact force \(\mathbf{F}_t^{\text{sim}}\). This enables the Critic to accurately evaluate the value gradient in the latent space, efficiently guiding the Actor to adjust joint poses to approximate the target force distribution.

(3) Physics-Aware Reward: The Tri-Component Supervision.
To enforce force-level human-likeness, the composite tactile reward \(R_{\text{tactile}}\) regulates the normal contact force \(F_i\) of the \(i\)-th finger (out of \(N_f\) fingers). It is formulated as a sum of three distinct components, Tactile Guidance \(r_{\text{guide}}\), Human-Like Alignment \(r_{\text{align}}\) and Contact Constraints \(r_{\text{safe}}\) balanced by weights \(w_g, w_a, w_s\). \[R_{\text{tactile}} = w_g \cdot r_{\text{guide}} + w_a \cdot r_{\text{align}} + w_s \cdot r_{\text{safe}}. \label{eq:tactile95total}\tag{1}\]

i. Tactile Guidance. Overcomes the common “air grasping” issue by providing a binary contact bonus. It acts as a “contact trigger,” rewarding the agent only when it successfully replicates an active contact event initiated by the human demonstrator, where \(\tau\) is a minimal contact threshold: \[r_{\text{guide}} = \frac{1}{N_f} \sum_{i=1}^{N_f} \mathbb{I}\left(F_{i}^{\text{sim}} > \tau\right) \cdot \mathbb{I}\left(F_{i}^{\text{human}} > \tau\right). \label{eq:r95guide}\tag{2}\]

ii. Human-Like Alignment. Merely touching the object is insufficient; the robot must replicate the force modulation patterns. We employ a robust Tanh-based distance metric to minimize the discrepancy between simulated and human forces. Unlike Mean Squared Error, this formulation provides normalized gradients and preserves the force distribution across fingers while resisting sensor outliers (\(\sigma\) is a scaling factor): \[r_{\text{align}} = \frac{1}{N_f} \sum_{i=1}^{N_f} \left( 1 - \tanh\left(\frac{|F_{i}^{\text{sim}} - F_{i}^{\text{human}}|}{\sigma}\right) \right). \label{eq:r95align}\tag{3}\]

iii. Contact Constraints. Prevents the policy from exploiting the physics engine to generate unrealistically high forces. We define a dynamic safety limit \(F_{\text{limit}} = F_{i}^{\text{human}} + \delta\) (where \(\delta\) is a safety margin). An exponential penalty is applied only when the simulated force exceeds this limit, ensuring a physically stable and safe Sim-to-Real transfer: \[r_{\text{safe}} = \exp\left( - \lambda \sum_{i=1}^{N_f} \max\left(0, F_{i}^{\text{sim}} - F_{\text{limit}}\right)^2 \right). \label{eq:r95safe}\tag{4}\]

Total Reward Formulation. To ensure the policy converges to a kinematically accurate and physically plausible solution, the final reward function at each timestep \(t\) combines our novel tactile supervision with standard kinematic constraints:

\[R_{total} = \underbrace{\lambda_{im} r_{im} + \lambda_{task} r_{task} + \lambda_{reg} r_{reg}}_{\text{Kinematic \& Task Constraints}} + \underbrace{\lambda_{tactile} R_{tactile}}_{\text{Physical Realism}}, \label{eq:total95reward}\tag{5}\]

While \(r_{im}\) (pose tracking), \(r_{task}\) (object trajectory), and \(r_{reg}\) (energy efficiency) guide the robot to the correct spatiotemporal configuration, the \(R_{\text{tactile}}\) term acts as a critical dynamic regularizer, guaranteeing that the interaction forces remain human-like and physically stable.

5 Experiments↩︎

5.1 Data Split and Evaluation Metrics↩︎

To rigorously evaluate the effectiveness of our tactile-guided transfer framework, we select 73 representative interaction sequences from the TactiDex dataset. This evaluation split encompasses both single-hand and bimanual scenarios, covering diverse object geometries, materials, and varying task complexities. Based on the evaluation dimensions shown in Table 2, we comprehensively assess the transferred policies across four distinct aspects:

1) Kinematic and Object Tracking. Following established practices [10], we evaluate geometric alignment using Object Tracking Error in Translation (\(\boldsymbol{OTE}_\boldsymbol{t}\)) and Rotation (\(\boldsymbol{OTE}_\boldsymbol{r}\)). To evaluate how closely the robotic hand replicates human kinematics, we compute the Mean Per-Joint Position Error (\(\boldsymbol{MPJPE}\)) and specifically the Mean Per-Fingertip Position Error (\(\boldsymbol{TipErr}\)), as fingertips are the primary medium for interaction.

2) Tactile and Physical Fidelity. To benchmark physical contact-level human-likeness—a core contribution of TactiSkill—we introduce a suite of fine-grained tactile metrics. Let \(f_{t,i}^{\text{sim}}\) and \(f_{t,i}^{\text{gt}}\) denote the normal force magnitude on the \(i\)-th finger (out of \(N_f\) total fingers) at timestep \(t\) (out of \(T\) total frames).

• Mean Tactile Force Error (MTFE). Computes the mean absolute difference between simulated forces and the aligned human ground truth across all fingers and temporal frames: \[\text{MTFE} = \frac{1}{T \cdot N_f} \sum_{t=1}^{T} \sum_{i=1}^{N_f} \left| f_{t,i}^{\text{sim}} - f_{t,i}^{\text{gt}}. \right|\]

• Contact F1-Score: We binarize the continuous force signals into contact states \(C_{t,i} = \mathbb{I}(f_{t,i} > \tau_c)\) using a minimal contact threshold \(\tau_c\). The F1-Score evaluates the spatial-temporal precision and recall of the simulated contact states against the ground truth, measuring whether the robot engages the correct fingers at the optimal time.

3) Safety and Reliability. To explicitly penalize unstable physical spikes and momentary destructive penetrations, we monitor the peak transient force discrepancy \(\text{MaxErr} = \max_{t, i} | f_{t,i}^{\text{sim}} - f_{t,i}^{\text{gt}} |\) and introduce two episode-level safety metrics:

• PeakSafe@3N: Calculates the fraction of episodes where the maximum instantaneous force error stays strictly below a safety threshold of 3N (\(\text{MaxErr} < 3\) N), indicating stable and safe execution without destructive force spikes.

• SafeTac@3N: A strictly combined metric requiring the episode to simultaneously achieve tactile-aware success and satisfy the peak safety constraint (\(\text{MaxErr} < 3\) N).

4) Task Success Rate (SR). We evaluate task completion at two distinct levels of strictness. First, the Kinematic Success Rate (\(\text{SR}_{\text{kin}}\)) adopts the criteria from [10]: a trial is successful if \(\text{OTE}_\text{r} \le 30^\circ\), \(\text{OTE}_\text{t} \le 3\) cm, \(\text{MPJPE} \le 8\) cm, and \(\text{TipErr} \le 6\) cm. For bimanual tasks, both hands must simultaneously meet these spatial conditions. Second, to explicitly emphasize physical realism, we employ a stricter Tactile-Aware Success Rate (\(\text{SR}_{\text{tac}}\)). A trial achieves tactile success only if it satisfies the kinematic criteria and demonstrates high physical fidelity, specifically requiring \(\text{MTFE} \le 3.0\) N and \(\text{Contact F1} \ge 0.3\).

5.2 Main Results and Ablative Analysis↩︎

To isolate the contributions of our tactile-guided supervision, we evaluate our TactiSkill framework against a kinematic baseline (ManipTrans [10]) and conduct an ablation study removing core tactile components: (1) w/o Bonus, (2) w/o Align, and (3) w/o Safety (Table 2).

The Insufficiency of Pure Kinematic Imitation. The purely kinematic baseline (Base) overfits to geometric alignment at the expense of physical realism. While achieving a seemingly adequate \(\text{SR}_{\text{kin}}\) (72.91%) and \(\text{OTE}_\text{t}\) (1.1947 cm), this performance is deceptive. By ignoring contact mechanics, the baseline frequently accomplishes goals through “air grasping” or mesh penetration. Consequently, its tactile-aware success rate (\(\text{SR}_{\text{tac}}\)) drops precipitously to 39.35%, and its safety metrics remain low (Contact F1 = 0.5569, \(\text{SafeTac@3N}\) = 35.84%). This unequivocally indicates that kinematic imitation alone is insufficient for robust tactile manipulation.

The Synergistic Effect of Tactile Guidance. In contrast, the full TactiSkill framework demonstrates that tactile supervision synergizes with kinematic tracking to act as a powerful physical regularizer. By enforcing valid contacts, TactiSkill prevents collapsing into invalid geometric optima, attaining the highest \(\text{SR}_{\text{kin}}\) (81.95%) alongside the lowest tracking errors (\(\text{OTE}_\text{t}\) = 0.9577 cm, \(\text{OTE}_\text{r}\) = \(6.1974^\circ\)). Most importantly, TactiSkill vastly outperforms all baselines in physical fidelity. It secures the highest Contact F1-score (0.7384) and boosts \(\text{SR}_{\text{tac}}\) to 64.64%. The superior episode-level safety metrics, particularly \(\text{PeakSafe@3N}\) (72.80%) and \(\text{SafeTac@3N}\) (53.57%), validate that our method yields highly reliable and human-like manipulation behavior without destructive force spikes. Specifically, this leap in tactile fidelity highlights a critical shift from passive trajectory following to active force regulation.

Ablation on Contact Initiation and Alignment. The ablation variants validate our tri-component design. Removing the contact bonus (w/o Bonus) cripples the incentive to initiate stable grasps, causing the Contact F1-score to plummet to 0.5425, and \(\text{PeakSafe@3N}\) to drop to 57.00%. Omitting force alignment (w/o Align) leads to spatially awkward grasps. Failing to match the ground-truth force distribution means the policy cannot utilize anatomically optimal contact patches. This misaligned application severely degrades dynamic stability, reducing Contact F1-score (0.5134) and resulting in a subpar tactile-aware success rate (\(\text{SR}_{\text{tac}}\) = 42.20%).

The Paradox of the Safety Penalty. Interestingly, removing the safety penalty (w/o Safety) yields the lowest mean errors (\(\text{MPJPE}\) = 5.0327 cm, \(\text{MTFE}\) = 0.0921 N). However, these seemingly superior average metrics are highly deceptive. Without an explicit penalty for transient force spikes, the unconstrained policy learns to game the average metric, permitting occasional, highly destructive force impulses. This is evidenced by the degradation in peak safety: compared to the full model, \(\text{PeakSafe@3N}\) drops to 68.70%, and functional safe success (\(\text{SafeTac@3N}\)) falls significantly to 38.72%. This highlights our core philosophy: the safety term is crucial not for minimizing isolated local average errors, but for suppressing momentary destructive behaviors, thereby achieving genuinely reliable, safety-aware manipulation at the task level.

5.3 Qualitative Analysis↩︎

Figure 3 showcases the qualitative transfer results of our TactiSkill framework evaluated on four representative sequences from the TactiDex dataset. For each single-hand and bimanual task, we present the hand-object interaction (HOI) reference alongside a five-frame rollout of the simulated execution.

Guided by our tactile-aware supervision, the dexterous hands successfully establish flush and physically stable contact, effectively avoiding unnatural penetrations or hovering. Specifically, in dynamic single-hand tasks (Rows 1-2), our policy seamlessly adapts its multi-fingered posture to diverse object geometries, maintaining secure grasps during complex translations and simulated actions. Furthermore, in demanding bimanual scenarios (Rows 3-4), TactiSkill exhibits exceptional coordination. Whether performing asymmetrical manipulation or dynamic spatial handovers, the framework accurately aligns respective contact patches and regulates interactive forces without dropping the object. Ultimately, this high-fidelity sequential execution underscores the efficacy of our physics-aware tri-component reward in long-horizon environments.

5.4 Real-World Deployment↩︎

We deploy our policy on a bimanual platform with two 7-DoF Franka arms and two Inspire dexterous hands. Since the Inspire hand exposes only 6 independent actuators, we retarget the simulated trajectory \(\mathbf{q}^{\text{sim}}_{1:T}\) to an actuation trajectory \(\tilde{\mathbf{q}}_{1:T}\in\mathbb{R}^6\).

A coarse anchor mapping \(\mathbf{q}^{\text{anchor}}_t=\Psi(\mathbf{q}^{\text{sim}}_t)\) provides initialization, and the actuator configuration is expanded to the full kinematic chain via \(\Gamma(\cdot)\). The retargeting objective is \[\mathcal{L} = \mathbb{E}_t \left[ \sum_f \|\mathbf{x}^{\text{sim}}_{t,f}-\Phi_f(\Gamma(\tilde{\mathbf{q}}_t))\|_2 + \lambda_a \|\tilde{\mathbf{q}}_t-\mathbf{q}^{\text{anchor}}_t\|_2^2 + \lambda_s \|\tilde{\mathbf{q}}_{t+1}-\tilde{\mathbf{q}}_t\|_2^2 \right].\]

Arm motions are executed by solving inverse kinematics to track the retargeted wrist pose. Tactile sensing is available on the Inspire hands but is not used as feedback in the current deployment.

6 Conclusion and Discussion↩︎

Conclusion In this work, we introduce TactiDex, a tactile-rich benchmark designed to advance dexterous manipulation from purely kinematic imitation toward physically grounded, human-like interaction. TactiDex provides synchronized object trajectories, articulated hand motion, and temporally aligned tactile signals, enabling contact-aware evaluation and learning. Building upon this benchmark, we propose TactiSkill under the Tactile-Guided Tri-Component Transfer Framework, which explicitly regulates contact formation, force magnitude, and temporal alignment. Through comprehensive quantitative and qualitative experiments, we demonstrate that incorporating structured tactile supervision significantly improves object stability, contact fidelity, and manipulation success over purely kinematic transfer. Overall, our results show that tactile guidance is not merely auxiliary, but essential for bridging the gap between motion imitation and physically realistic dexterous manipulation.

Limitations & Future Work Discrepancies in sensor noise, resolution, and physical compliance between simulation and real hardware can degrade the accuracy of learned force interactions when deploying policies directly. To mitigate this, future work will explore on-robot fine-tuning with real tactile feedback, enabling closed-loop adaptation. More broadly, integrating vision, proprioception, and whole-hand tactile sensing within a unified, contact-centric learning framework represents a promising direction toward achieving robust, real-world dexterous manipulation.

7 TactiDex Details↩︎

7.1 Data Scale and Inventory↩︎

The TactiDex dataset features over 700 meticulously recorded interaction sequences, specifically targeting human-to-robot dexterous manipulation. To ensure comprehensive coverage of contact-rich scenarios, we curated a diverse set of 49 everyday objects spanning multiple categories.

As illustrated in Fig. 5, these objects encompass various geometries, sizes, and physical properties. Rather than being limited to simple rigid bodies, our inventory ranges from delicate items (e.g., eggs) and everyday containers to complex tools and matching multi-part accessories (e.g., a grinder with a rod, or a wok with a lid). This strategic selection explicitly supports the execution of intricate, long-horizon object-object interactions.

Furthermore, to provide a holistic view of the interaction diversity, Fig. 6 visualizes the macroscopic distribution of our recorded sequences. Following previous large-scale interaction datasets, we employ a Sankey diagram to map the explicit hierarchical relationships between task divisions (i.e., single-hand versus bimanual tasks), fine-grained action types (e.g., functional simulation, bimanual handovers, object-object interactions), and specific object categories.

7.2 Hardware Setup and Capture System↩︎

To capture high-fidelity, synchronized human demonstrations, we constructed a custom multi-modal capture arena that seamlessly integrates global motion capture, fine-grained hand kinematics, and whole-hand tactile sensing.

Optical Motion Capture System. The global tracking arena is powered by an OptiTrack system equipped with 8 PrimeX 13W (PX13W) cameras, operating at a high sampling rate of 120 fps. Prior to data collection, the capture volume is meticulously calibrated using a specialized vendor-provided active wand. This rigorous calibration process achieves an exceptional mean ray error of exactly 0.2 mm, ensuring sub-millimeter precision for recording both hand trajectories and object 6D poses.

Kinematic and Tactile Dual-Glove Setup. For fine-grained finger kinematics, human demonstrators wear MANUS Metagloves Pro, renowned for their high-precision and low-latency joint tracking. To guarantee anatomical accuracy across different subjects, a standardized calibration routine is strictly performed before every single recording session.

Simultaneously, the user wears our high-resolution tactile glove on the exterior to capture continuous pressure maps. Integrating these two state-of-the-art systems presented a unique hardware challenge: tracking the Metagloves within the OptiTrack ecosystem requires a rigid marker backplate. To accommodate the outer tactile glove without damaging its delicate internal wiring or compromising sensor fidelity, we engineered a custom, non-destructive mechanical modification for the backplate integration (as illustrated in Fig. 7).

Object Registration and Migration. Beyond capturing raw trajectories, we ensure all object interactions are grounded in a unified physical space. Once the 3D meshes of the 49 objects are generated, we manually align their scale, translation, and rotation within the multi-view OptiTrack coordinate system. This standardized spatial registration process guarantees that the recorded 6D object poses are inherently aligned with the mesh geometry, allowing the entire dataset to be easily migrated and correctly rendered on any new computational platform or simulation environment.

7.3 Standard Operating Procedure (SOP) and Collection Pipeline↩︎

Given the unprecedented scale of the TactiDex dataset, relying on manual recording protocols for 700+ multi-modal sequences would be highly susceptible to inconsistencies and human error.

To systematically standardize our collection process, we designed and deployed a dedicated web-based data collection system. This integrated web platform serves as the central command hub during demonstrations. It dynamically displays the required task instructions to the human subject, tracks the real-time progress of the task inventory, and logs metadata for each recorded trial. By formalizing the entire workflow through this automated SOP interface, we significantly reduced overhead and ensured that task executions remain highly consistent across the entire dataset.

8 TactiSkill Implementation↩︎

8.1 Network Architecture and State-Action Space↩︎

Our TactiSkill framework is implemented using Proximal Policy Optimization (PPO) with an asymmetric Actor-Critic architecture. Both networks are modeled as multi-layer perceptrons (MLPs) featuring hidden dimensions of \([512, 256, 128]\) with ELU activations.

To bridge the sim-to-real gap, the state space is rigorously decoupled. Taking the 12-DoF Inspire hand as our primary single-hand example, the observation space consists of three distinct components (Note: for bimanual tasks, all dimensions are exactly doubled):

1. Proprioceptive State (\(S_{prop} \in \mathbb{R}^{46}\)): Contains the internal kinematic status of the robotic hand.

• Joint States (36-dim): Current joint positions \(\mathbf{q} \in \mathbb{R}^{12}\), and their trigonometric encodings \(\sin(\mathbf{q}), \cos(\mathbf{q}) \in \mathbb{R}^{24}\) to ensure continuity in rotational space.

• Wrist Base State (10-dim): The 6D pose (quaternion) and linear/angular velocities of the wrist base, excluding absolute global positioning to promote spatial invariance.

2. Target Reference (\(S_{target} \in \mathbb{R}^{330}\)): Provides dense spatial and temporal cues from the human demonstration to guide the imitation process.

• Tactile & Geometry Prior (133-dim): A Basis Point Set (BPS) encoding of the object’s point cloud (\(128\)-dim) coupled with the ground-truth target tactile distances for the fingertips (\(5\)-dim).

• Future Kinematic Trajectory (197-dim): The delta kinematics (positions, rotations, linear/angular velocities) over the future horizon for the wrist (\(23\)-dim), finger joints (\(135\)-dim), manipulated object (\(23\)-dim), and the relative object-to-joint transformations (\(16\)-dim).

3. Privileged Information (\(S_{priv} \in \mathbb{R}^{49}\)): Accessible exclusively to the Critic during simulation training to accurately estimate the value function.

• Exact Object Dynamics (14-dim): Ground-truth Cartesian position (\(3\)), quaternion (\(4\)), velocities (\(6\)), and object weight (\(1\)).

• High-Fidelity Contact Force (20-dim): Simulated fingertip contact forces \(\mathbf{F}^{sim} \in \mathbb{R}^{20}\), explicitly capturing both the 3D directional force vectors and their scalar magnitudes across all \(5\) fingertips.

• Internal Physics (15-dim): Joint velocities \(\dot{\mathbf{q}} \in \mathbb{R}^{12}\) and the object’s exact Center of Mass (CoM) coordinates (\(3\)-dim).

Network Inputs & Action Space: During training, the Actor network observes only the deployable real-world states: \(\mathcal{S}_{actor} = S_{prop} \oplus S_{target} \in \mathbb{R}^{376}\) (or \(752\) for bimanual). The Critic network receives the full state including privileged dynamics: \(\mathcal{S}_{critic} = S_{prop} \oplus S_{target} \oplus S_{priv} \in \mathbb{R}^{425}\) (or \(850\) for bimanual). The action space \(\mathcal{A} \in \mathbb{R}^{12}\) outputs the residual target joint positions, which are executed via a low-level PD controller.

8.2 Hyperparameters and Reward Weights↩︎

Table 3 comprehensively lists the hyperparameters used for training the residual policy, including the specific weights for our tactile-guided tri-component reward.

The simulation environment is built upon NVIDIA Isaac Gym with a PhysX backend.

• Physics Parameters: The simulation operates at \(60\) Hz with \(2\) sub-steps per frame. To ensure stable grasping and realistic interaction dynamics, we set the lateral friction coefficient of both the fingertips and objects to \(1.0\), and the joint damping to \(0.5\) Ns/m.

• Tactile Signal Mapping: To bridge the gap between human demonstrations and the simulation engine, raw ADC signals from the physical tactile sensors are mapped to simulated normal forces using a linear calibration model: \(F_{sim} = k \cdot (ADC_{raw} - ADC_{offset})\), where \(k\) is determined via a calibrated force gauge. Within the PhysX engine, this ground-truth force corresponds to the rigid-body contact impulse divided by the simulation time step.

9 Real-World Hardware Deployment Details↩︎

While Section 5 in the main text outlines our retargeting objective for zero-shot sim-to-real transfer, this section provides the low-level engineering and communication details required to reproduce our bimanual hardware experiments.

9.1 Bimanual System Architecture and Arm Control↩︎

Our physical deployment platform consists of two 7-DoF Franka Emika Panda arms and two Inspire dexterous hands. The entire dual-arm control framework is orchestrated using the Deoxys control interface.

To mitigate the sim-to-real global coordinate gap, the Franka arms track relative delta-poses rather than absolute Cartesian coordinates. Before execution, the initial pose of the simulated trajectory is aligned with the physical robot’s starting state. At each timestep, the delta position and relative rotation (computed via quaternion transformations) are applied to the physical anchor pose. The arms are then driven by an Operational Space Controller (OSC) operating at a \(100\) Hz control loop (\(\Delta t = 0.01\) s), which converts Cartesian tracking errors into executable joint torques.

9.2 Kinematic Optimization for the Inspire Hand↩︎

The simulated hand possesses 12 DoF, whereas the physical Inspire hand is underactuated, driven by only 6 independent linear actuators: Index flexion, Middle flexion, Ring flexion, Pinky flexion, Thumb pitch, and Thumb yaw.

To execute the simulated trajectory \(\mathbf{q}^{\text{sim}}_{1:T}\) on the physical hand, we employ an offline optimization-based retargeting pipeline. We utilize PyTorch Kinematics to construct the differentiable forward kinematics chain of the Inspire hand. For each trajectory, we initialize a 6-DoF command sequence \(\tilde{\mathbf{q}}_{1:T}\) using a heuristic anchor mapping. We then utilize the Adam optimizer (learning rate \(= 0.01\), \(100\) iterations) to minimize the following composite loss: \[\mathcal{L} = W_{\text{pos}} \mathcal{L}_{\text{pos}} + W_{\text{anchor}} \mathcal{L}_{\text{anchor}} + W_{\text{smooth}} \mathcal{L}_{\text{smooth}}\] Here, \(\mathcal{L}_{\text{pos}}\) penalizes the Cartesian distance between the 12-DoF simulated fingertips and the 6-DoF retargeted fingertips. \(\mathcal{L}_{\text{anchor}}\) prevents the solution from drifting too far from the anchor prior, and \(\mathcal{L}_{\text{smooth}}\) penalizes temporal jitter. Based on empirical tuning, the weights are set to \(W_{\text{pos}} = 50.0\), \(W_{\text{anchor}} = 20.0\), and \(W_{\text{smooth}} = 10.0\).

9.3 Command Mapping and Open-Loop Execution↩︎

Once optimized, the 6-DoF joint radians are calibrated and linearly mapped to discrete actuation commands \(\in [0, 1000]\). These commands are continuously dispatched to drive the Inspire hand via a \(115200\) baud serial interface.

Crucially, while the physical Inspire hands are equipped with tactile sensors, our current deployment relies exclusively on open-loop execution (i.e., the tactile signals are not fed back into the policy online). The successful sim-to-real transfer demonstrates a core advantage of our TactiSkill framework: by actively regulating simulated contact forces via the tri-component reward during training, the policy intrinsically learns robust, physics-aware force distributions. Consequently, the resulting zero-shot kinematic trajectories naturally establish stable, physically grounded grasps on the real hardware, successfully overcoming typical geometric domain gaps without requiring fragile online tactile feedback loops.

10 Extended Qualitative Results and Video Guide↩︎

10.1 Visualization of the TactiDex Dataset↩︎

Before evaluating the learned policies, we first visualize the rich, multi-modal human demonstrations captured in our TactiDex dataset. As the foundation of our tactile-guided reinforcement learning framework, the dataset goes beyond standard kinematic tracking by synchronously recording high-resolution contact forces.

As illustrated in Fig. 8, we present representative sequences across diverse manipulation scenarios, including left-hand (LH), right-hand (RH), and long-horizon bimanual (BiH) tasks. Alongside the 3D kinematic hand-object spatial relationships, we visualize the corresponding human contact heat maps at each timestep (mapped to a purple-to-yellow color scale). These dynamic pressure distributions explicitly reveal the underlying physical intent of the demonstrator. For instance, the heat maps clearly distinguish between the localized fingertip forces required to pinch an apple or hold a telephone, versus the broader palm-level pressure required to firmly press a blackboard eraser. By capturing these nuanced, physics-grounded interactions, the TactiDex dataset provides the essential dense tactile priors that our TactiSkill framework leverages during policy training.

10.2 Qualitative Comparison of Interaction Quality↩︎

To visually evaluate the impact of our proposed tri-component tactile reward, we provide a localized, frame-by-frame comparison between our TactiSkill policy and a standard kinematic-only imitation baseline. As illustrated in Fig. 9, we examine representative left-hand (LH), right-hand (RH), and bimanual (BiH) tasks along their execution timestamps.

Without the explicit guidance of tactile priors, the baseline policy relies entirely on minimizing joint-space discrepancies. This purely kinematic approach frequently suffers from critical physical inconsistencies during complex interactions, as highlighted in the magnified red regions:

• Kinematic-only Baseline: The policy fails to respect physical contact boundaries. This results in obvious unnatural hovering (e.g., fingers leaving large gaps around the telephone handle or paddle) or severe mesh penetration (e.g., fingers aggressively clipping through the kettle handle to artificially secure a grasp).

• TactiSkill (Ours): By actively regularizing simulated contact forces during the RL process, our policy ensures the robotic fingertips establish flush, physically grounded contacts (highlighted in the magnified green regions). Our tactile-guided policy gracefully conforms to the object’s surface boundaries, maintaining stable and realistic grasps across all manipulation stages.

10.3 Supplementary Video Guide↩︎

The dynamic performance, temporal stability, and zero-shot real-world deployment of our framework are best appreciated in motion. We strongly encourage the reviewers to refer to the accompanying Supplementary Video for a comprehensive evaluation of the uncurated policy rollouts and hardware experiments.

References↩︎