1 Introduction↩︎

Hand-object pose estimation from a single RGB image is a fundamental task in multimedia and computer vision, and plays a critical role in applications such as virtual reality [4], [5], augmented reality [6], robotics [7], [8], and human-robot interaction [9], [10], [11].

Hand-object pose estimation has recently attracted considerable attention and achieved significant progress [12], [13], [14], [15]. However, robust estimation remains highly challenging, primarily due to severe self-occlusions of the hand and mutual occlusions during hand-object interactions. We argue that a key limitation of existing approaches is due to their inability to model occlusions which hinders accurate perception and reasoning of occluded hand-object relationships, thereby limiting their ability to perform robust pose estimation.

Existing hand-object pose estimation methods can be broadly categorized into keypoint-based methods [16],[17],[18] and implicit 3D representation-based methods [19],[20],[3]. The former first detect 2D keypoints in RGB images and then utilize predefined 3D keypoints of objects, combined with the Perspective-n-Point (PnP) algorithm to estimate object poses. For hand pose estimation, keypoint-based methods predict 2D joint locations and shape parameters based on the MANO parameters model [21]. However, under severe occlusions during hand-object interactions, these methods often struggle to accurately locate the keypoints which result in incorrect feature associations, ultimately reducing the stability and accuracy of pose estimation. Implicit 3D representation-based methods utilize continuous functions such as the signed distance field (SDF) to encode the geometric structure of the hand and object. Although these methods provide improved geometric modeling capability, they heavily rely on visible regions. Under high occlusions, the lack of sufficient context leads to inaccurate SDF prediction, which hampers the perception of complete hand-object interaction structures, thereby limiting the accuracy of pose estimation.

To address the above challenges, we propose an occlusion-aware hand-object pose estimation framework that accurately reasons hand-object interaction relationship under severe occlusions, enabling precise pose estimation. A comparison of previous methods to our occlusion-awareness method is shown in Fig. 1. We propose an occlusion-aware hand-object pose estimation method based on masked autoencoders (MAE), termed HOMAE. First, we introduce a target-focused masking strategy that imposes structured occlusion on hand-object interaction regions, encouraging the masked autoencoder to learn occlusion-aware features and infer the occluded structures. Specifically, we identify hand-object interaction regions by leveraging object bounding boxes, and apply masking selectively within these regions to simulate realistic occlusions. This compels the model to focus on the spatial relationships between the hand and object, thereby enhancing its ability to reason about interaction under occlusion. To address the problem of inaccurate SDF prediction under occlusion, we integrate hierarchical features from multiple decoding stages of the MAE to predict the SDF. This enables the model to capture both global structural consistency and fine-grained geometric details. Our design improves the accuracy of the SDF prediction by preserving high-frequency geometric information while incorporating essential contextual cues. To further enhance geometric perception, we combine the implicit SDF with an explicit point cloud representation derived from the SDF, leveraging the complementary strengths of both representations. While the SDF encodes the global structure in a continuous manner, the explicit point cloud emphasizes structurally salient regions, providing precise local geometric cues. This fusion enables hand-object interaction modeling that is robust to occlusions by jointly capturing global context and local details.

Our contributions are summarized as follows:

• We propose an occlusion-aware hand-object pose estimation method based on masked autoencoders to perceive and infer occluded regions in hand-object interactions by reconstructing masked input images, improving occlusion reasoning and interaction understanding ability.

• We introduce a multi-scale feature aggregation strategy, integrating hierarchical features from multiple decoding stages of the MAE to predict the SDF, capturing both global structure and fine-grained details for more accurate SDF prediction under occlusion.

• To further improve occlusion-aware reasoning, we integrate both implicit and explicit representations by combining the predicted SDF with a point cloud derived from it. While the SDF encodes implicit global context, the derived point cloud provides explicit and localized geometric cues that are crucial for capturing fine-grained surface details. This complementary fusion enhances robust and accurate hand-object pose estimation.

2 Related Work↩︎

Since our method leverages the occlusion-awareness capabilities of MAE to address the occlusion challenge in 3D hand-object pose estimation, we categorize related work into two aspects: 3D hand-object pose estimation and masked autoencoders.

2.1 3D Hand-Object Pose Estimation↩︎

Previous researches have primarily focused on hand [6], [22], [23] or object pose estimation [24], [25], [26]. With the development of large-scale hand-object interaction datasets[27], [28], [29], [14], increasing numbers of researchers have begun exploring hand-object interaction pose estimation [1], [30], [31], [32]. This has led to more accurate modeling of hand-object dynamics, driving advancements in robotic manipulation and human-robot interaction. Existing hand-object pose estimation methods can be broadly categorized into two main approaches: keypoint-based methods [16],[17],[18],[33], [34] and implicit 3D representation-based methods [19],[20],[35],[3],[36].

For keypoint-based methods, Doosti et al.[16] proposed a lightweight deep learning framework to accurately predict hand-object poses from a single RGB image. Their framework employs a hand decoder to predict 2D joints and a 3D mesh parameterized by the MANO [21] model, while an object decoder estimates the 2D locations of predefined 3D corner points. The object pose is then recovered using the PnP algorithm. To address the challenge of obtaining ground-truth annotations in real-world scenarios, Liu et al.[17] introduced a joint learning framework that leverages spatiotemporal consistency in large-scale hand-object videos as constraints for generating pseudo-labels in a semi-supervised learning paradigm. They further utilized a transformer-based [37] approach to perform explicit contextual reasoning between hand and object representations, thereby enhancing hand-object pose estimation. Hampali et al.[18] utilized a cross-attention mechanism to model the correlation between 2D keypoints and 3D hand-object poses. Lin et al.[33] introduced a dual-stream backbone strategy that enables the hand and object to be extracted as distinct entities in intermediate layers, preventing feature competition during learning. The shared higher-level representations enforce feature harmonization between the hand and object, facilitating mutual feature enhancement. To better exploit contextual information, Kuang et al. [34] proposed a context-aware transformer framework that jointly models global context and local hand–object features through shared decoder layers. By imposing mutual contextual priors between the hand and object, their method enhances robustness under occlusions.

For the implicit 3D representation-based methods, Chen et al. [19] proposed a joint learning framework for 3D hand-object reconstruction, integrating the advantages of parametric mesh models and SDF. Their approach estimates hand and object poses using a parametric model while leveraging an SDF network to learn hand and object shapes in a pose-normalized coordinate space. To better model the 3D geometry of hand-object interactions, Chen et al.[20] further predicted kinematic chains for pose transformations and aligned SDF representations with highly articulated hand poses. By enforcing geometric alignment, their method improves the visual features of 3D points and enhances robustness against motion blur by incorporating temporal information. Qi et al. [3] introduced an SDF-guided hand-object pose estimation network that jointly utilizes hand and object SDFs to provide a global implicit representation over the complete reconstructed volume. Zhang et al. [35] employed a deep distance field as an implicit shape representation. They proposed a 2D ray-based feature aggregation scheme and a 3D intersection-aware hand pose embedding to extract local features, effectively capturing hand-object interactions. Unlike above methods that solely rely on implicit 3D features, we further incorporate explicit 3D geometric features to enhance the representation of hand-object interactions.

2.2 Masked Autoencoders↩︎

MAE [39] learn robust visual representations by randomly masking patches of the input image and training the model to reconstruct the masking patches. This mask-and-reconstruct mechanism enables the model to capture contextual dependencies and infer occluded structures, thereby enhancing its ability to reason under occlusions. Owing to its strong generalization and occlusion-aware capabilities, MAE has been successfully applied to a wide range of vision tasks. In 2D vision tasks [40] [41], [42], Hu et al. [43] reconstructed the masked input image, forcing the model to capture all relevant features, thereby enhancing the reasoning ability of human images and better achieving the pedestrian re-identification task. Bar et al.[44] leveraged MAE as a self-supervised learning paradigm to construct hand-object interaction datasets. This global feature learning has shown to significantly enhance performance across various 2D vision downstream applications. MAE has also been increasingly adopted in 3D vision tasks [45], [46], [47]. By extending the masked autoencoding paradigm to 3D data, recent studies have explored various strategies for learning geometric representation. For instance, Mo et al. [48] proposed a novel voxel-aware masking strategy that adaptively aggregates background/foreground information from voxelized point clouds, resulting in better point cloud generation. Xu et al. introduce a masking mechanism over partial body joint coordinates and leverage spatiotemporal dependencies to recover the masking joints, thereby capturing richer relational cues for enhanced feature learning. These works demonstrate that MAE can effectively reconstruct 3D structures and learn geometry-aware representations, which are beneficial for downstream tasks in 3D vision.

We are the first to introduce MAE into hand-object pose estimation. This paper presents HOMAE, which explores the potential of MAE in this task and achieves state-of-the-art performance.

3 Method↩︎

We propose an occlusion-aware framework for hand-object pose estimation, as illustrated in Fig. 2, which aims to jointly estimate the hand-object pose from a single RGB image. Specifically, the regression targets include joint rotations \(\theta \in \mathbb{R}^{3 \times 16}\) and a shape vector \(\alpha \in \mathbb{R}^{10}\), as defined by the MANO model [21], along with the 6-degree-of-freedom (6D) object pose, which includes a 3D rotation vector \(r \in \mathbb{R}^3\) and a 3D translation vector \(t \in \mathbb{R}^3\). Our approach consists of four key components: occlusion-aware masked autoencoders (Section 3.1), multi-scale feature-guided field regression (Section 3.2), implicit-explicit geometric aggregation (Section 3.3), and hand-object pose regression (Section 3.4).

3.1 Occlusion-Aware Masked Autoencoders↩︎

In monocular RGB-based hand-object interaction pose estimation, accurately extracting features of the hand and object remains challenging, especially under severe occlusions. To address this, we introduce an occlusion-aware learning mechanism to enhance feature reasoning capabilities in occlusion regions. Specifically, we propose a target-focused masking strategy that imposes structured occlusions on hand-object interaction regions. This design adaptively suppresses irrelevant background information while highlighting critical interaction regions, thereby guiding the model to focus on informative features during training and enhancing its ability to reason under occlusion. Furthermore, we incorporate a masked autoencoder for feature learning, where the encoder extracts semantically rich representations of hand-object interactions, and the decoder reconstructs occluded regions to reinforce the occlusion reasoning capabilities of the model. This joint encoding-decoding mechanism enables the model to focus on contextually relevant structures, thereby enhancing the quality of features in hand-object interaction and ultimately improving the accuracy of hand-object pose estimation.

Target-Focused Masking: Given an input image \(I \in \mathbb{R}^{ H \times W \times 3}\), we divide it into patches of size \(P \times P\). We randomly mask the \(\rho\) patches. \[B = (x_{\min}, y_{\min}, x_{\max}, y_{\max}),\] where \(B\) denotes the object bounding box. In the grid coordinate system, the patch range covering the object is: \[(x'_{\min}, y'_{\min}, x'_{\max}, y'_{\max}) = \frac{B}{P},\] and the number of patches covering the object region is given by: \[N_o = (x'_{\max} - x'_{\min} + 1) \cdot (y'_{\max} - y'_{\min} + 1).\]

Randomly masking \(\mu\) patches within the object region. The number of masked patches in the object region is given by: \[N_m^o = \mu \rho,\] the remaining patches are then randomly selected from the entire image, excluding those already chosen in the object region: \[N_m^b = \rho - N_m^o,\] where the \(N_m^b\) patches are randomly sampled from the entire image, including both the object region and the background. The final mask matrix \(M\) has the following shape: \[M_{i,j} = \begin{cases} 1, & \text{if masked} \\ 0, & \text{otherwise} \end{cases}, \quad M \in \{0,1\}^{\frac{H}{P} \times \frac{W}{P}}.\]

After determining the mask matrix \(M\), we generate the masked image \(I_m \in \mathbb{R}^{ H \times W \times 3}\) by replacing each masked patch \((i, j)\) in \(I\) with a Gaussian noise block sampled randomly: \[I_m[:, i P: (i+1) P, j P: (j+1) P] = \mathcal{N}(0,1),\] where \(\mathcal{N}(0,1)\) represents Gaussian noise sampled independently for each masked patch.

Autoencoder: After obtaining the masking image \(I_m\), we employ DINOv2 [49] as the encoder \(\mathcal{E}\) to fully leverage its powerful feature extraction capability for capturing global semantic information. Pre-trained through self-supervised learning, DINOv2 effectively extracts structured features from images, enabling it to focus on key regions even in complex backgrounds and occluded scenarios. Formally, the extracted feature representation is given by: \[F = \mathcal{E}(I_m),\] where \(F \in \mathbb{R}^{\frac{H}{14} \times \frac{W}{14} \times 256}\) represents the high-level semantic features extracted by the encoder \(\mathcal{E}\). To enhance the model robustness to occlusion and preserve fine-grained interaction cues, we employ a decoder \(\mathcal{D}\) composed of a stack of MLP layers, which progressively upsamples \(F\) through \(L\) decoding layers to generate multi-scale features for reconstructing the masked image: \[\hat{I} = \mathcal{D}(F) = \prod_{l=1}^{L} \mathcal{D}_l (F),\] where \(\prod\) denotes the sequential decoding operations that generate multi-scale feature maps and reconstruct the complete image.

To enforce accurate reconstruction, we employ the mean squared error loss to supervise the predicted image \(\hat{I}\) against the ground-truth image \(I\): \[L_{\text{rec}} = \frac{1}{\mathbb{P}} \sum_{i=1}^{\mathbb{P}} \| \hat{I}_i - I_i \|^2,\] where \(\mathbb{P}\) denotes the total number of pixels. This loss ensures that the reconstructed image preserves structural consistency with the ground truth. This encoder-decoder framework not only enhances the understanding of hand-object interaction regions but also significantly improves occlusion awareness. The visualization results of the masking and reconstructed images are shown in Fig. 3. Our target-focused masking strategy effectively covers a wide range of occlusion types commonly encountered in real-world hand-object interactions, encompassing both fine-grained occlusions (e.g., fingertips) and large-area occlusions (e.g., palms or objects). Meanwhile, our method demonstrates strong reconstruction robustness under severe occlusions, effectively preserving the spatial relationships between the hand and object while recovering interaction regions with clear structural consistency.

3.2 Multi-Scale Feature-Guided Field Regression↩︎

To address the challenge of inaccurate SDF prediction under occlusions, we leverage multi-scale image features to enhance the implicit representation of hand-object interactions, thereby enabling more accurate and robust SDF estimation. The SDF encodes the distance of each point to the hand and object nearest surface, providing a continuous and differentiable representation of object and hand geometry. By integrating hierarchical image features, our method improves the robustness of SDF estimation, effectively capturing local texture details and global semantic cues.

Feature Alignment: To enhance SDF estimation, we exploit hierarchical features by aggregating intermediate outputs from multiple stages of the decoder. This allows the model to capture both fine-grained local textures and high-level semantic information. Formally, given the encoder output feature \(F\), the decoder produces a multi-scale feature representation \(\tilde{F} \in \mathbb{R}^{H \times W \times C}\), which integrates information from different decoding levels. \[\tilde{F} = \text{MLP} \left( \bigoplus_{l \in L} \mathcal{D}^{(l)}(F) \right),\] where \(\bigoplus\) denotes the channel-wise concatenation of features from different decoding stages. This hierarchical feature representation captures comprehensive multi-scale information beneficial for predicting SDF.

Given the 3D surface points \(p\) of the hand and object, we project them onto the 2D image plane using the camera intrinsic matrix \(K\). During training, the surface points are directly sampled from the ground-truth hand and object mesh. For inference, we sample potential surface points within the voxel space following HOISDF [3] without requiring ground-truth meshes. To bridge the distribution gap between training and inference, clean mesh-sampled surface points are used for SDF prediction to ensure accurate signed distance learning, while during SDF feature learning, Gaussian-perturbed surface points are fed into the network to simulate the voxel-sampled distribution encountered at inference, thereby improving robustness. \[F_{\text{img}}^{x} = \tilde{F} \left( \pi(K, p_{x}) \right), \quad x \in \{h, o\},\] where \(\pi(K, p)\) represents the projection of the 3D point \(p\) onto the 2D image plane. This process ensures that the 3D geometric structure is effectively aligned with image semantic information, facilitating more accurate SDF estimation in the subsequent regression module.

SDF Regression: For hand SDF estimation, in order to preserve the geometric properties of the original 3D structure, we apply Fourier Positional Encoding [50] to the hand surface points \(p_h \in \mathbb{R}^{600 \times 3}\), obtaining a high-dimensional representation \(\gamma(p_h) \in \mathbb{R}^{600 \times 30}\). The final feature representation for each surface point \(p_h\) is constructed by fusing the fourier encoded, pixel-aligned image features \(F^h_{\text{img}} \in \mathbb{R}^{600 \times C}\), and the original 3D hand points. This fused representation is then fed into an MLP to regress the SDF for the hand surface: \[SDF_h = \text{MLP} \left( \gamma(p_h) \oplus F^h_{\text{img}}\oplus p_h \right).\] The process for estimating the \(SDF_o\) follows the same steps as the \(SDF_h\), with \(p_o \in \mathbb{R}^{200 \times 3}\), \(\gamma(p_o) \in \mathbb{R}^{200 \times 30}\), and \(F^o_{\text{img}} \in \mathbb{R}^{200 \times C}\) represent the object surface points and features. By leveraging hierarchical multi-scale image features and geometric priors, our method enables accurate SDF estimation for both the hand and object surfaces, even under occlusions and complex hand-object interactions.

3.3 Implicit and Explicit Geometric Aggregation↩︎

To achieve robust hand-object pose estimation under occlusion, we integrate implicit and explicit geometric representations. Implicit representations provide continuous, differentiable surface encoding, while explicit 3D point clouds offer structured spatial cues. Combined with multi-scale image features from the masked autoencoder, our model learns both local geometric structures and rich contextual information, enhancing its ability to infer accurate, occlusion-aware hand-object poses.

To extract point-wise geometric features, the sampled hand point clouds is fed into a PointNet [38] backbone, producing global feature \(F_{\text{3D}}^{\text{h}} \in \mathbb{R} ^{600 \times C}\) that encodes the structural characteristics of each point. Simultaneously, multi-scale image features \(F^h_{\text{img}}\) corresponding to the 2D projection of \(p_h\) are obtained from the decoder layers of the masked autoencoder. The final aggregated feature \(F_{\text{agg}}^{\text{h}} \in \mathbb{R}^{600 \times C}\) for each hand point is obtained by concatenating its corresponding image feature and explicit 3D geometric feature, and then further fused with the implicit representation derived from the predicted SDF, enabling a richer encoding of both visual, geometric, and implicit spatial information. \[F_{\text{agg}}^{\text{h}} = \left( F^h_{\text{img}} \oplus F_{\text{3D}}^{\text{h}} \right) \cdot \frac{1}{\beta_h} \cdot \sigma_h\left(\frac{SDF_{\text{h}}(p_h)}{\beta_h}\right),\] Where \(\sigma_h(\cdot)\) is the sigmoid function, \(\beta_h\) is the learnable scale parameter. The object aggregated feature \(F_{\text{agg}}^{\text{o}} \in \mathbb{R}^{200 \times C}\) is obtained using the same strategy as for the hand. By unifying implicit SDF encoding, explicit point cloud representations, and dense image features within a multi-modal fusion framework, our approach leverages the implicit geometric cues learned from the SDF to adaptively modulate and enhance explicit spatial features. This modulation mechanism strengthens structural reasoning under occlusion, leading to more robust and accurate hand-object pose estimation.

3.4 Hand-Object Pose Estimation↩︎

To estimate the hand and object poses, we utilize their respective aggregated features, \(F_{\text{agg}}^{\text{h}}\) and \(F_{\text{agg}}^{\text{o}}\), which fuse implicit and explicit geometric information with multi-scale semantics cues. These representations capture rich spatial context, enabling robust pose estimation under occlusions and complex hand-object interactions. For hand pose estimation, \(F_{\text{agg}}^{\text{h}}\) is fed into a transformer block [37]. The refined features are then passed through an MLP to predict the MANO parameters [21], including joint rotations and shape vector: \[(\{\theta_i \}_{i=0}^{16}, \alpha) = \text{MLP}_{\text{h}}(\text{Transformer}_{\text{h}}(F_{\text{agg}}^{\text{h}})),\] where \(\theta\in \mathbb{R}^{3 \times 16}\) denote joint rotations and \(\alpha \in \mathbb{R}^{10}\) is the shape vector. For object pose estimation, the aggregated object feature \(F_{\text{agg}}^{\text{o}}\) is processed by a transformer block, followed by an MLP that outputs the object 6D pose parameters: \[(r, t) = \text{MLP}_{\text{o}}(\text{Transformer}_{\text{o}}(F_{\text{agg}}^{\text{o}})),\] where \(r \in \mathbb{R}^3\) and \(t \in \mathbb{R}^3\) represent rotation and translation vector, respectively.

Loss Function: Our overall training objective combines multiple loss functions to jointly optimize image reconstruction quality and pose estimation accuracy. Specifically, a reconstruction loss \(L_{\text{rec}}\) is employed to supervise masked image reconstruction. The hand pose loss \(L_{\text{mano}}\) and the object pose loss \(L_{\text{obj}}\), both implemented as smooth L1 losses, are used to constrain the predicted MANO parameters and object pose, respectively. Additionally, the SDF loss \(L_{\text{SDF}}\) supervises the predicted implicit surface representation. The auxiliary loss terms \(L_{\text{others}}\) follow HOISDF [3], providing additional regularization or task-specific constraints. The total loss is defined as: \[L_{\text{total}} = \lambda_{\text{1}} L_{\text{rec}} + \lambda_{\text{2}} L_{\text{mano}} + \lambda_{\text{3}} L_{\text{obj}} + \lambda_{\text{4}} L_{\text{SDF}} + \lambda_{\text{5}} L_{\text{others}}.\]

Each term is weighted by its corresponding coefficient \(\lambda\), to balance the contributions to the overall loss.

4.1pt

4 Experiments↩︎

To demonstrate the effectiveness of our approach, we compare with state-of-the-art methods on two benchmark datasets with occlusion challenges, DexYCB [28] and HO3Dv2 [27]. Furthermore, we conduct ablation studies to evaluate the advantages of each proposed innovation.

4.1 Datasets and Evaluation Metrics↩︎

DexYCB dataset is a large-scale benchmark for hand-object interaction pose estimation. The dataset captures 10 subjects manipulating 20 YCB objects [57]. Following the standard evaluation protocol, we adopt the S0 split, using sequences from 8 subjects for training and the remaining 2 subjects for testing to ensure fair comparison with existing methods. To comprehensively evaluate both hand and object pose estimation performance, we adopt Mean Joint Error (MJE) and Procrustes-Aligned MJE (PA-MJE) [58] for assessing hand pose accuracy, and Object Center Error (OCE), Mean Corner Error (MCE), and the average closest point distance (ADD-S) for evaluating object pose accuracy. Additionally, we report Mean Mesh Error (MME), the area under the curve of the percentage of correct vertices (V-AUC), and F-scores at 5mm and 15 mm thresholds (F@5mm, F@15mm), together with their Procrustes-aligned versions following H2ONet [59] to assess the quality of hand mesh reconstruction.

HO3Dv2 dataset comprises 77K images from 68 video sequences, covering 10 different subjects interacting with 10 objects from the YCB dataset [57]. We follow the official data split protocol for training and testing and submit our test results to the official evaluation server. Performance is assessed using five key metrics: Mean Joint Error (MJE), Scale-Translation aligned MJE (STMJE) [60], and Procrustes-Aligned MJE (PA-MJE) [58] for evaluating hand pose estimation, while Object Mesh Error (OME) and ADD-S are used to assess object pose estimation accuracy.

4.2 Implementation Details↩︎

HOMAE is implemented in PyTorch and trained and inferred on a single NVIDIA RTX 3090 GPU. We adopt the Adam optimizer with a batch size of 24. The initial learning rate is set to 1e-4 and decayed by a factor of 0.7 every 5 epochs. The model is trained for 60 epochs on both the DexYCB [28] and HO3Dv2 [27] datasets, which is sufficient to achieve satisfactory performance. The input images are cropped around the object and resized to 224×224. In our masking strategy, the patch size \(P\) is set to 28, the total number of masks \(\rho\) is 12, and the probability \(\mu\) of a mask falling within the object bounding box is set to 50%. The feature dimension C is set to 223. Through extensive experiments, we found that setting \(\lambda_1\), \(\lambda_2\), \(\lambda_3\), and \(\lambda_4\) at 1 yields the best performance, while \(\lambda_5\) is set according to the configuration in HOISDF [3].

4.3 Comparison with State-of-the-Art Methods↩︎

Quantitative comparisons on DexYCB Dataset: Table 1 presents a comparison of our method with other approaches on the DexYCB dataset. The results demonstrate that our method achieves state-of-the-art performance across nearly all evaluation metrics. While achieving a competitive MJE of 10.6 mm, slightly behind Qi et al. [3], our method achieves the best PAMJE of 5.08 mm, indicating more accurate joint localization after alignment. More importantly, our method significantly outperforms all previous methods on object mesh accuracy, obtaining the lowest OCE of 17.1 mm and MCE of 25.2 mm. Furthermore, we achieve the highest accuracy object pose estimation, with an ADD-S of 11.8 mm. These results highlight the strength of our approach in modeling fine-grained hand-object interactions, ensuring precise articulation and detailed geometric reconstruction, especially in challenging contact and occlusion scenarios. The qualitative comparison between our method and HOISDF [3] on the DexYCB dataset is shown in Fig. 4. The visualization results demonstrate that our method achieves more accurate pose estimation under occluded scenarios, leading to more realistic and plausible hand-object contact. In contrast, HOISDF often fails to handle occlusions effectively, resulting in incorrect hand-object interactions. In addition, since our method incorporates MANO-based hand reconstruction, we compare our method with recent state-of-the-art hand mesh reconstruction methods in Table 2. Our method consistently achieves the best results across both procrustes-aligned and non-aligned metrics. Specifically, it achieves the lowest PA-V-PE of 4.9 mm, and the highest PA-J-AUC of 89.8%, PA-V-AUC of 90.0%, and PA-F@5 of 81.7%, demonstrating precise articulation and high-fidelity mesh prediction. In the non-aligned setting, our method also outperforms previous approaches, reaching an F@15 of 94.4%, significantly surpassing methods such as HandOccNet [52] and H2ONet [54]. These results confirm that our approach not only ensures joint-level accuracy, but also achieves detailed mesh reconstruction under complex occlusions and hand-object interactions.

Quantitative comparisons on HO3Dv2 Dataset: A comparison with existing methods on the HO3Dv2 dataset is provided in Table 3. Our method achieves the lowest MJE of 21.8 mm, STMJE of 20.5 mm, demonstrating its superior capability in recovering fine-grained hand articulations even under severe occlusions. Compared to the previous state-of-the-art Qi et al. [3], our approach further improves both MJE and STMJE, while maintaining competitive performance in PAMJE. Notably, our method also surpasses previous works in object pose estimation, achieving an OME of 39.3 mm and the ADD-S of 14.2 mm. These results highlight the effectiveness of our framework in learning occlusion-aware representations that jointly enhance hand and object pose estimation. In addition, Table 4 further demonstrates the robustness of our method across diverse object categories in the HO3Dv2 dataset. The qualitative comparison between our method and HOISDF [3] on the HO3Dv2 dataset is illustrated in Fig. 5. The results show that our framework exhibits robustness and effectively handles severe occlusions, whereas HOISDF struggles to maintain accurate pose estimation. Overall, our method remains stable and reliable under challenging occlusion conditions.

4.4 Ablation Studies↩︎

To analyze the contributions of different components in our framework, we performed several ablation experiments on the HO3Dv2 dataset [27].

Effect of Target-Focused Masking and Image Reconstruction: To isolate the effect of our occlusion-aware masking strategy, we retain the encoder and decoder framework, but remove the masking mechanism. As shown in the first row of Table 5, this leads to a noticeable performance drop across all metrics. In particular, PAMJE increases from 9.8 mm to 10.5 mm, and OME increases from 39.3 mm to 43.8 mm. By focusing the masking strategy on object-centric regions and reconstructing the masked images, this approach enhances the model occlusion-awareness and its structural understanding of hand-object interactions.

Effect of Multi-Scale Feature Fusion for SDF Regression: To evaluate the effectiveness of our multi-scale image feature fusion, we ablate the hierarchical aggregation mechanism and instead directly regress the SDF using only the final-layer decoder features. As shown in the second row of Table 5, this simplification weakens the model capacity to capture fine-grained local geometry and spatial variations. These results demonstrate that leveraging multi-scale features provides prediction.

Effect of Implicit and Explicit Geometric Aggregation: We assess the contribution of the implicit–explicit geometric aggregation module by removing it and retaining only the implicit SDF-based representation. As shown in the third row of Table 5, this leads to a slight performance degradation across all evaluation metrics. The results highlight the importance of fusing implicit and explicit geometric cues, as this fusion effectively captures both fine-grained surface details and the global spatial structure, which is especially advantageous for accurate and occlusion-aware hand-object pose estimation.

Effect of Noise Perturbation on Mesh-Sampled Surface Points: Removing the Gaussian perturbation applied to mesh-sampled surface points leads to a moderate drop in all metrics, indicating that this perturbation effectively bridges the distribution gap between training and voxel-sampled inference, allowing the network to learn smoother implicit fields.

Effect of Backbone Substitution (DINOv2 → DINOv3) : We evaluate the impact of the visual backbone by substituting DINOv2 with DINOv3 [61] under the same settings. As shown in Table 5, performance remains largely comparable, with a slight decline due to DINOv3 producing lower-resolution feature maps, which marginally reduces fine-grained spatial cues. This confirms the robustness and adaptability of our framework across different backbones.

Effect on Loss Functions: The ablation results of different loss components in our framework are summarized in Table 6. Removing the reconstruction loss \(L_{\text{rec}}\), which belongs to our occlusion-aware MAE, leads to a moderate performance drop across all metrics, indicating that this term helps maintain feature consistency under occlusions. In contrast, removing the SDF supervision loss \(L_{\text{SDF}}\) results in a drastic degradation across all metrics, confirming that the signed distance field provides an essential implicit geometric constraint for learning the hand–object interaction space. When excluding the MANO parameter regression loss \(L_{\text{mano}}\), the model completely fails to estimate accurate hand poses, as reflected by extremely high errors in MJE, STMJE, and PAMJE. Similarly, removing the object pose loss \(L_{\text{obj}}\) leads to an almost complete collapse in object localization (OME and ADD-S), demonstrating that this loss is indispensable for stabilizing the object pose branch. Overall, these results highlight that both \(L_{\text{mano}}\) and \(L_{\text{obj}}\) are fundamental for their respective pose estimation tasks, while \(L_{\text{SDF}}\) and \(L_{\text{rec}}\) provide crucial geometric and feature-level consistency to achieve optimal performance.

Effect of Masking Ratio and Patch Number: We further study the influence of the masking ratio within the object region and the total number of masking patches. As shown in Table 7, the best performance is achieved when half of the object region is masked and twelve patches are sampled in total. A lower masking ratio provides insufficient supervision for occluded regions, while overly large masking ratios cause information loss and degrade reconstruction quality. Likewise, using either too few or too many patches slightly reduces performance, suggesting that a balanced masking strategy is crucial for effective occlusion reasoning and hand–object pose estimation. As shown in the last row of Table 7, when the entire object bounding box is masked, most object-related and some hand-related visual cues are occluded, leading to a severe drop in object pose accuracy and a slight decrease in hand pose performance.

Effect of Masking Types: We further conduct an ablation study on different masking strategies, including zero-masking, mean-masking, and Gaussian noise masking. As shown in Table 8, Gaussian noise masking achieves the best performance in all metrics. This suggests that during training, Gaussian noise introduces a more challenging reconstruction task, enabling the model to learn occluded region features more effectively and perform more robust reasoning. In contrast, zero-masking causes severe information loss in occluded areas, leading to suboptimal learning. Mean-masking retains some contextual information but lacks the variability needed for effective occlusion reasoning, making it less effective than Gaussian noise.

4.5 Computational Efficiency Analysis↩︎

To assess the computational efficiency of our proposed framework, we evaluate the inference speed, parameter count, and memory consumption. The model achieves a real-time inference speed of 20 FPS, demonstrating its practicality for real-world applications. Despite incorporating the DINOv2 backbone and Transformer-based fusion modules, the total parameter count remains moderate at 206.9 MB, and the memory footprint is limited to approximately 1668 MB. These results indicate that our framework achieves a good balance between efficiency and accuracy, making it suitable for deployment in scenarios requiring real-time or near real-time hand–object pose estimation.

4.6 Failure Cases and Limitations↩︎

Although our method is robust in occlusion scenarios of hand-object interaction, it still has limitations under extreme occlusion conditions. Failure cases in extreme-occlusion interaction scenarios are shown in Fig. 6. Specifically, when the object is completely occluded, the model faces challenges in accurately inferring its spatial relationships, often leading to imprecise pose estimations. Likewise, in cases of severe hand occlusion, incomplete feature perception may cause the model to misinterpret hand articulation. These limitations suggest that future research could further enhance the model robustness and spatial reasoning capability under extreme occlusion conditions through the integration of temporal cues or multi-view fusion strategies.

5 Conclusion↩︎

We introduced HOMAE, an occlusion-aware framework for 3D hand-object pose estimation from a single-view RGB image. By leveraging a target-focused masking strategy within masked autoencoders, our method enables context-aware feature learning and structural reasoning under occlusions. The integration of multi-scale SDF predictions with explicit point cloud representations further enhances geometric understanding, facilitating accurate and robust hand-object pose estimation. Our approach demonstrates strong generalization across complex interaction scenarios, as evidenced by state-of-the-art results on DexYCB and HO3Dv2.

Limitations and Future Work. Although HOMAE demonstrates strong performance on benchmark datasets, its robustness decreases under extreme occlusion conditions. When both the hand and object are completely obscured, the model lacks sufficient contextual cues to support accurate pose estimation. Future work could investigate the temporal cues or multi-view fusion techniques to further enhance the robustness and accuracy in extreme occlusion scenes. In addition, we plan to investigate lightweight model architectures and deployment strategies for mobile or AR/VR devices, which would enable real-time inference and broaden practical applicability.

References↩︎

---

1. This work is supported by the National Natural Science Foundation of China under Grants 62473141 and U22A2059, National Science and Technology Major Project of China under Grants 2026ZD1610900, Natural Science Foundation of Hunan Province under Grant 2024JJ5098, and the Open Foundation of the State Key Laboratory of Advanced Design and Manufacturing for Vehicle Body. Ajmal Mian was supported by the Australian Research Council Future Fellowship Award funded by the Australian Government under Project FT210100268. (Corresponding authors: Wei Sun; Jian Liu.)↩︎

2. Hui Yang, Wei Sun, Jian Liu, and Jian Xiao are with the National Engineering Research Center for Robot Visual Perception and Control Technology, School of Artificial Intelligence and Robotics, Hunan University, Changsha, 410082, China. (e-mail: {huiyang, wei_sun, jianliu, xiaojian2002}hnu.edu.cn?)↩︎

3. Jin Zheng is with the School of Architecture and Art, Central South University, Changsha, 410082, China. (e-mail: zheng.jin@csu.edu.cn)↩︎

4. Ajmal Mian is with the Department of Computer Science and Software Engineering, The University of Western Australia, WA 6009, Australia. (e-mail: ajmal.mian@uwa.edu.au)↩︎