1 Introduction↩︎

Motion capture from monocular videos is an essential technology for various applications such as gaming, VR/AR, sports analysis, . One ultimate goal is to achieve real-time capture while being accurate and physically plausible in world space. Despite the recent advancements, this task is still far from being solved, especially under the settings of in-the-wild captures with hand-held moving cameras.

Compared to optimization-based methods [1]–[7], learning-based approaches [8]–[11] can directly regress human poses from images, significantly enhancing computational efficiency while addressing the inherent issues in optimization-based methods of initialization sensitivity and local optima entrapment. As data-driven solutions, the performance and generalization capabilities of learning-based methodologies are heavily bounded by the accuracy and diversity of the training data. Unfortunately, existing datasets are unable to simultaneously meet these requirements. On the one hand, datasets with sequential ground truth 3D pose annotations  [12]–[15] are mostly captured by marker-based or multi-view systems, which makes it hard to scale up to a satisfactory level of diversity in human appearances and backgrounds. On the other hand, numerous in-the-wild datasets [16], [17] excel in the richness of human and scenario diversity but they lack real-world 3D motions and most of them only provide individual images instead of videos. Recently, researchers tried to create synthetic data [18]–[20] by rendering human avatars with controllable cameras, but it remains difficult to bridge domain gaps between the real-world images and the rendered ones, and is too expensive to scale up.

In this paper, we follow the spirit of creating synthetic data, but turn to render 2D proxy representations instead of person images. By using proxy representations (, silhouettes [21], [22], segmentations [23], [23]–[25], IUV [26], [27] and 2D skeletons [21], [28]–[34]), the whole motion capture pipeline can be divided into two steps: image-to-proxy extraction and proxy-to-motion lifting. In the divided pipeline, the image-to-proxy extraction is pretty accurate and robust as there are plenty of annotated 2D ground truths in real-world datasets, while the proxy-to-motion step can leverage more diverse training data to mitigate the generalization issue and reduce the domain gap. Here we adopt the 2D skeleton sequences as the proxy representation for its simplicity and high correlation with 3D motion. Meanwhile, we combine random virtual camera trajectories upon the existing large-scale motion sequence database, , AMASS [35], to synthesize nearly infinite data for learning proxy-to-motion lifting.

Though the proposed proxy dataset can be generated at large scales, there remain two challenges for regression-based networks to learn physically plausible motions from proxy data: how to recover i) world-space human motion under moving cameras, and ii) physically plausible motion with steady body-ground contact. For a video captured with moving cameras, the trajectories/rotations of humans and cameras are coupled with each other, making the recovery of world-space human motion extremely difficult. To address this issue, the latest solutions [36], [37] estimate the camera poses from the background using SfM [38], [39], and then estimate the human motion from a camera-centric perspective. However, the SfM requires texture-rich backgrounds and may fail when the foreground moving character dominates the image. Their post-processing optimization pipelines are also not suitable for real-time applications due to expensive computational costs. Besides, these previous solutions learn human motion in a camera-centric perspective like [40], which is actually ambiguous for the regression network.

In this paper, we would point out that one of the main challenges arises from the camera-centric settings in previous solutions. In such a setting, the same motion sequence captured under different cameras will be represented as multiple human motion trajectories, making it difficult for the network to understand the inherent motion prior. In contrast, we propose to learn human-centric motions to ensure consistent human motion outputs under different camera trajectories in synthetic data. Specifically, our network learns the local translations and poses in a human coordinate system, together with the relative camera extrinsic parameters in this space. After that, we accumulate the local translations of humans in each frame to obtain the global camera trajectories. Benefiting from the proposed proxy-to-motion dataset, we are able to synthesize different camera trajectories upon the same motion sequence to learn the human motion consistency. In this way, our network can disentangle the human poses from the moving camera more easily via the strong motion prior without SfM.

On top of human-centric motion regression, we further enhance the physical plausibility by introducing a contact-aware neural motion descent module. Specifically, our network first predicts coarse motions and then refines them iteratively based on foot-ground contact and motion misalignment with the proxy observations. Compared with the global post-processing optimization used in previous work [36], [37], [41], our method learns the descent direction and step instead of explicit gradient back-propagation. We demonstrate that our method, termed ProxyCap, is more robust and significantly faster to support real-time applications. To sum up, our contributions can be listed as follows:

• To tackle the data scarcity issue, we adopt 2D skeleton sequences as proxy representations and generate proxy data in world space with random virtual camera trajectories.

• We design a network to learn motions from a human-centric perceptive, which enables our regressor to understand the consistency of human motions under different camera trajectories.

• We further propose a contact-aware neural descent module for more accurate and physically plausible predictions. Our network can be aware of foot-ground contact and motion misalignment with the proxy observations.

• We demonstrate a real-time full-body capture system with plausible body-ground contact in world space under moving cameras.

2 Related Work↩︎

Monocular motion capture has been an active research field recently. We give a brief review of the works related to ours and refer readers to [42] for a more comprehensive survey.

Motion Capture Datasets. Existing motion capture datasets are either captured with marker-based [12], [14] or marker-less [43]–[46] systems. Due to the requirement of markers or multi-view settings, the diversity of these datasets is limited in comparison with in-the-wild datasets. To enrich the motion datasets, numerous efforts [9], [47]–[50] have been dedicated to generating pseudo-ground truth labels with better alignment in the image space but do not consider the motion in world space. On the other hand, researchers have also resorted to using synthetic data [18], [51], [52] by rendering human models with controllable viewpoints and backgrounds. However, such synthetic datasets are either too expensive to create or have large domain gaps with real-world images.

Proxy Representations for Human Mesh Recovery. Due to the lack of annotated data and the diversity of human appearances and backgrounds, learning accurate 3D motions from raw RGB images is challenging even for deep neural networks. To alleviate this issue, previous approaches have exploited the different proxy representations, including silhouettes [21], [22], 2D/3D landmarks [21], [28]–[34], segmentation [23], [23]–[25], and IUV [26], [27]. These proxy representations can provide guidance for the neural network and hence make the learning process easier. However, the proxy representations simplify the observation and introduce additional ambiguity in depth and scale, especially when using proxy representations in a single frame [26], [27], [32]. In this work, we alleviate this issue by adopting 2D skeleton sequences as proxy representations and propose to generate proxy data with accurate motions in world space.

Full-body Motion Capture. Recent state-of-the-art approaches [10], [23] have achieved promising results for the estimation of body-only [10], [23], hand-only [53], and face-only [54] models. By combining the efforts together, these regression-based approaches have been exploited for monocular full-body motion capture. These approaches [11], [55]–[59] typically regress the body, hands, and face models by three expert networks and integrate them together with different strategies. For instance, PIXIE [58] learns the integration by collaborative regression, while PyMAF-X [11] adopts an adaptive integration strategy with elbow-twist compensation to avoid unnatural wrist poses. Despite the progress, it remains difficult for existing solutions to run at real-time while being accurate in world space. In this work, we achieve real-time full-body capture with plausible foot-ground contact by introducing new data generation strategies and novel network architectures.

Neural Decent for Motion Capture. Traditional optimization-based approaches [3] typically fit 3D parametric models to the 2D evidence but suffer from initialization sensitivity and the failure to handle challenging poses. To achieve more efficient and robust motion prediction, there are several attempts to leverage the learning power of neural networks for iterative refinement. HUND [60] proposes a learning-to-learn approach based on recurrent networks to regress the updates of the model parameters. Song  [32] propose the learned gradient descent to refine the poses of the predicted body model. Similar refinement strategies are also exploited in PyMAF [10] and LVD [61] by leveraging image features as inputs. In our work, we propose a contact-aware neural decent module and exploit the usage for more effective motion updates.

Plausible Motion Capture in World Space. Though existing monocular motion capture methods can produce well-aligned results, they may still suffer from artifacts such as ground penetration and foot skating in world space. For more physically plausible reconstruction, previous works [40], [41] have made attempts to leverage more accurate camera models during the learning process. To encourage proper contact of human meshes, Rempe  [62] propose a physics-based trajectory optimization to learn the body contact information explicitly. HuMoR [63] introduces a conditional VAE to learn a distribution of pose changes in the motion sequence, providing a motion prior for more plausible human pose prediction. LEMO [64] learns the proposed motion smoothness prior and optimizes with the physics-inspired contact friction term. Despite their plausible results, these methods typically require high computation costs and are unsuitable for real-time applications. For more effective learning of the physical constraints, there are several attempts [65], [66] to incorporate the physics simulation in the learning process via reinforcement learning. However, these methods [65], [66] typically depend on 3D scene modeling due to the physics-based formulation. Recently, there are also attempts to recover camera motion via SLAM techniques [36], [37] or regress the human motion trajectory [67], [68]. Despite the progress, it remains challenging for these methods to run in real-time or produce physically plausible in world space. In our work, we achieve real-time capture with plausible foot-ground contact in world space by designing novel networks to learn human-centric motion.

3 Proxy Data Generation↩︎

To tackle the data issue, we synthesize sequential proxy-to-motion data based on 2D skeletons and their corresponding 3D rotational motions in world space. In the following, we describe the synthesis and integration of different types of labels in our proxy data, including body motions, hand gestures, and contact labels.

Body proxy data. We adopt the motion sequences from the AMASS dataset [35] to generate proxy-to-motion pairs for the body part. The AMASS dataset is a large-scale body motion sequence dataset that comprises 3,772 minutes of motion sequence data, featuring diverse and complex body poses. We downsample the motion data to 60 frames per second, resulting in 9407K frames.

Integration with hand gestures. Since the hand poses in the AMASS dataset are predominantly static, we augment the proxy data with hand poses from the InterHand [69] dataset, which contains 1361K frames of gesture data captured at 30 frames per second. We employ Spherical Linear Interpolation (Slerp) to upsample the hand pose data to 40, 50, and 60 fps and randomly integrate them with the body poses in the AMASS motion sequences.

Integration with contact labels. We calculate the continuous contact indicators \(ind\) for 3D skeletons as follows: \[\label{equ:contact95indicator} \widehat{ind}_i = Sigmoid(\frac{v_{max}-v_i}{k_v})\cdot Sigmoid(\frac{z_{max}-z_i}{k_z})\tag{1}\] where \(v_i\) and \(z_i\) denote the velocity and the height to the xz-plane of the given joint. \(v_{max}\) and \(z_{max}\) is set to \(0.2m/s\) and \(0.08m\), and \(k_v\) and \(k_z\) is set to \(0.04\) and \(0.008\).

Camera setting. For each 3D motion sequence, we generate 2D proxy data under different virtual camera trajectories (four cameras in practice). Such proxy data enhances the learning of the inherent relationship between 2D proxy and 3D motions and the consistency across different camera viewpoints.

Specifically, we uniformly sample the field of view (FOV) from \(30^{\circ}\) to \(90^{\circ}\) for the intrinsic parameters of cameras. When setting the extrinsic parameters for camera trajectories, we position the cameras at distances ranging from \(1\) meter to \(5\) meters around the human, and at heights varying from \(0.5\) meters to \(2\) meters to the ground. Finally, we generate pseudo 2D skeleton annotations by projecting 3D joints into the 2D pixel-plane using these virtual cameras. Moreover, to simulate the jitter of 2D detectors, we add Gaussian noise \(\Delta X \sim \mathcal{N}(0, 0.01)\) to 3D joints before projections.

4 Method↩︎

As illustrated in Fig. 1, we first detect the 2D skeletons from images and input them into our proxy-to-motion network to recover 3D local motions in human space. These relative local motions are transformed into a world coordinate system and accumulated along the sliding window. Additionally, we leverage a neural descent module to refine the accuracy and physical plausibility. In this section, we start with introducing the human-centric setting of our motion prediction.

4.1 Human-Centric Motion Modeling↩︎

For more accurate camera modeling, we adopt the classical pinhole camera model instead of using the simplified orthogonal or weak perspective projection [8]–[10], [70]–[72]. As shown in Fig. 2, we transform the global human motion and the camera trajectories into local human space from two adjacent frames, where we adopt \(\{\beta\in\mathbf{R^{10}}, \theta \in \mathbf{R^{22\times3}}, t \in \mathbf{R^{3}}\}_{t}\) to denote the parameters of shape, pose, and translation at frame \(t\), and \(\{R \in \mathbf{R^{3\times 3}}, T \in \mathbf{R^{3}} \}_t\) to denote the camera extrinsic parameters. Given the pose and shape parameters, the joints and vertices can be obtained within the SMPL Layer: \(\{J, V\} =\mathcal{X}\left(\beta, t, \theta, g\right)\). In the following, we use the subscript and to distinguish between the human coordinate system and the world coordinate system.

During the learning of human-centric motions, we adopt a similar setting used in previous works on temporal human motion priors [41], [63], [73]. Specifically, each motion sequence will be normalized by a rigid transformation to remove the initial translation in x-z plane and rotate itself with the root orientation heading to the z-axis direction. With this setting, our network can learn the relative movements of humans, which are independent of observation viewpoints. The detailed implementation of the human-to-world coordinate transformation and the global motion accumulation in world space can be found in our supplementary material.

4.2 Sequential Full-body Motion Initialization↩︎

As shown in Fig. 1, at the first stage of our method, the skeleton sequences are processed by temporal encoders and then fed into decoders for the predictions of the initial motion \(\{\mathbf{\beta}^0,\mathbf{\theta}^0,\mathbf{t}^0\}\), initial camera \(\{\mathbf{R}^0_{\mathbf{H}},\mathbf{T}^0_{\mathbf{H}}\}\), the contact indicator \(\mathbf{ind}\), and the hand poses \(\mathbf{g}\). Following previous baseline [70], we build these encoders upon temporal dilated convolution networks.

For better body-hand compatibility, we exploit the cross-attention mechanism to facilitate the motion context sharing during the body and hand recovery. Specifically, we first obtain the initial body features \(F_{body}\) and hand features \(F_{hand}\) from the temporal encoders and map them as Query, Key, Value matrices in forms of \(Q_{body/hand}\), \(K_{body/hand}\), and \(V_{body/hand}\), respectively. Then we update the body features \(F'_{body}\) and hand features \(F’_{hand}\) as follows: \[\label{equ:body95hand} \begin{align} F'_{body} &= V_{body} + Softmax(\frac{Q_{hand} K_{body}^\top}{\sqrt{d_k}}) V_{body},\\ F'_{hand} &=V_{hand} + Softmax(\frac{Q_{body} K_{hand}^\top}{\sqrt{d_k}}) V_{hand}. \end{align}\tag{2}\]

The updated features \(\{F'_{body}, F'_{hand}\}\) can be further utilized in the contact indicators \(ind\) and serve as the temporal context in the Neural Descent module, as will be described shortly. In our experiments, we demonstrate that the feature fusion in Eq. 2 can make two tasks benefit each other to produce more comparable wrist poses in the full-body model.

4.3 Contact-aware Neural Motion Descent↩︎

At the second stage of our method, the initial motion predictions will be refined to be more accurate and physically plausible with the proposed contact-aware neural motion descent module. As shown in Fig. 1, this module takes the 2D misalignment and body-ground contact status as input and updates motion parameters during iterations.

4.3.0.1 Misalignment and Contact Calculation.

At the iteration of \(i\in \{0, 1, ..., N\}\), we calculate the 2D misalignment status by projecting the 3D joints on the image plane and calculate the differences between the re-projected 2D joints and the proxy observations: \(\mathcal{S}_{proj} = \Pi(J_i, \{K, R_H^i, T_H^i\}) -\widehat{J}_{2D}\). Here, \(\Pi(\cdot)\) denotes the perspective projection function, and \(K\) denotes the intrinsic parameter.

For the contact status, we calculate the velocity of 3D joints \(v_{xz}^i\) in xz-plane and the distance to the ground as \(d_{y}^i\). Moreover, we also leverage the temporal features from inputs 2D skeletons to predict the contact labels \(ind\), which will be used as an indicator to mask the body-ground contact. Then, the contact status of the current predictions can be obtained as \(\mathcal{S}_{contact} = ind \odot (v_{xz}^i, d^i)\), where \(\odot\) denotes the Hadamard product operation.

4.3.0.2 Motion Update.

After obtaining the contact and misalignment status, we feed them into the neural motion descent module for motion updates. As shown in Fig. 3, the descent module takes the two groups of tensors as input: i) the state group includes the current SMPL parameters in the human coordinate system \(\beta^i, t^i_H, \theta^i_H\), camera pose \(R^i_H, T^i_H\) and the sequential motion context \(F_{seq}=\{F'_{body}, F'_{hand}\}\); ii) the deviation group includes the current misalignment status \(\mathcal{S}_{proj}\) and contact status \(\mathcal{S}_{contact}\).

A straightforward solution would be using an MLP to process these two groups of tensors. However, the values of these two groups exhibit significant differences. For instance, the values of the state tensors change smoothly while the values of the deviation tensors may change rapidly along with the misalignment and contact status. Simply concatenating them as inputs introduces difficulty in the learning process. Note that the magnitude of the deviation tensors is highly correlated with the parameter updates. When the body model is well-aligned without foot skating or ground penetration, the values of the deviation tensors are almost zero, indicating that the refiner should output zeros to prevent further changes in the pose parameters. Otherwise, the refiner should output larger update values for motion adjustments. To leverage such a correlation property, we exploit a cross-attention module to build a more effective architecture.

As shown in Fig. 3, two fully connected layers are leveraged to process the tensors of the state and deviation groups and produce the Query, Key, and Value for the cross-attention module. In this way, our contact-aware neural motion descent module can effectively learn the relationship between the state and deviation groups and hence produce more accurate motion updates. Moreover, the sequential motion context \(F_{seq}\) is also leveraged in our neural descent module to mitigate the depth uncertainty and improve the motion predictions.

Compared with previous work [32], [60], [63], [64], the proposed contact-aware neural motion descent module offers the advantage of freeing us from the heavy cost of explicit gradient calculations or the manual tuning of hyper-parameters during testing. Furthermore, the module is capable of learning human motion priors with contact information from our synthetic dataset, which provides a more suitable descent direction and steps to escape the local minima and achieve faster convergence.

4.4 Loss Function↩︎

In our solution, the full-body motion recovery module and the contact-aware neural motion descent module are trained sequentially. Benefiting from the proxy-to-motion learning, the ground-truth full-body pose \(\theta, g\), and human body shape \(\beta_b\) can be obtained for supervision from our synthetic dataset. Overall, the objective function of motion recovery can be written as follows: \[\label{equ:loss95rec} \mathcal{L}_{rec} = \mathcal{L}_{3D} + \mathcal{L}_{2D} + \mathcal{L}_{\theta} + \mathcal{L}_{\beta}+ \mathcal{L}_{cam} + \mathcal{L}_{consist} + \mathcal{L}_{smooth}\tag{3}\]

Specifically, \(\mathcal{L}_{3D}\) involves 3D MPJPE loss and 3D trajectory L1 loss while \(\mathcal{L}_{2D}\) is the projected 2D MPJPE loss. \(\mathcal{L}_{\theta}, \mathcal{L}_{\beta}, \mathcal{L}_{cam}\) represents L1 loss between the estimated human pose, shape and camera pose to our synthetic ground truth. \(\mathcal{L}_{consist}\) is a L1 loss to restrict the consistency of the local motion outputs \(\theta_{H}, t_{H}\) of the same 3D motion sequence via different observations from virtual cameras. \(\mathcal{L}_{smooth}\) is adopted from  [74] by penalizing the velocity and acceleration between the estimation and the ground truth. For the neural descent module, the objective loss can be written as: \[\label{equ:loss95descent} \begin{cases} \mathcal{L}_{desc} = \sum_k {u^{N-k} (\mathcal{L}_{rec} + \mathcal{L}_{contact})} \\ \mathcal{L}_{contact} =\sum_i{ind^{gt}\odot(||v_{xz}||_2 + ||d_y||_2)} \\ \mathcal{L}_{ind} = \sum_i{Entropy(ind^{gt}, ind^{est})} \end{cases}\tag{4}\] where \(k=1,2,..., N\) is the iteration time and \(u\) is the decay ratio to emphasize the last iteration. We set \(K=3, u=0.8\) in practice. \(\mathcal{L}_{contact}\) involves the error of trajectory drifting, foot floating, or ground penetration. \(\mathcal{L}_{ind}\) refers to the loss between the predicted contact label to the ground truth.

5 Experiments↩︎

In this Section, we validate the efficacy of our method and demonstrate accurate human motion capture results with physically plausible foot-ground contact in world space.

Dataset. The RICH dataset [15] is collected with a multi-view static camera system and one moving camera that the ground truth 3D human motions can be recovered using spatial stereo geometry. The EgoBody [75] is captured by a multi-camera rig and a head-mounted device, focusing on the social interactions in 3D scenes. Dynamic Human3.6M is a benchmark proposed in  [41] to simulate the moving cameras on Human3.6M [12] by randomly cropping with a small view window around the person.

Metrics. In our experiments, we follow previous work [41] to report various metrics, primarily focusing on the evaluation of the world coordinate system. The WA-MPJPE metric reports the MPJPE after aligning the entire trajectory of both the predicted and GT through Procrustes Alignment. The W-MPJPE metric reports the MPJPE after aligning the first frame of sequences. The PA-MPJPE metric reports the MPJPE error after applying the ground truth trajectories to each frame. The ACCEL metric is used to evaluate the joint acceleration.

5.1 Comparison with the State of the Art↩︎

We compare our approach with the state-of-the-art approaches to human motion recovery under dynamic cameras, including GLAMR [41], SLAMHR [37] and PACE [36]. Both the SLAMHR and PACE require a pre-processing SLAM to reconstruct the scene to solve the camera trajectories (refer to Tab. 3). Such a process is time consuming and requires texture-rich backgrounds, which narrows their applications. To validate the effectiveness of the proposed solution, we primarily compare our method with GLAMR, as it also runs without SLAM. We also conduct comparison experiments on the RICH and EgoBody datasets, as shown in Tab. 1. As shown in the table, our method achieves significant improvements over previous solutions in all metrics.

Visual comparisons with previous different solutions are also depicted in Fig. 5 and the video in our supplementary materials, where our method again shows superior results in terms of model-image alignment and foot-ground contact in world space.

5.2 Ablation Studies↩︎

We conduct ablation studies to validate the effectiveness of the proposed neural descent method on the Dynamic Human3.6M dataset following the setup of  [41]. As shown in Tab 2, the Neural Descent module can significantly reduce the motion errors in world space.

We also report the metric of ground penetration [65] (GP) and foot floating (FF) in the Human3.6M [12] dataset. The GP is defined as the percentage of the frames that penetrate to the ground. The FF is defined as the percentage of frames with foot-ground distances far from the given threshold. We report the curves of GP and FF with respect to the distance to ground in Fig. 6 with a logarithmic scale, where we can conclude that the neural descent algorithm can significantly improve the ground contact plausibility.

5.3 Runtime↩︎

It is also worth noting that the proposed method has improved the speed by an order of magnitude compared to the previous methods. The speeds of different methods are reported in Tab. 3. Our method can reach real-time performance at 30 FPS in a laptop with RTX4060, which is very promising to enable various applications related to virtual humans.

6 Conclusion↩︎

In this paper, we present ProxyCap, a real-time monocular full-body motion capture approach with physically plausible foot-ground contact in world space. We leverage a proxy dataset based on 2D skeleton sequences with accurate 3D rotational motions in world space. Based on the proxy data, our network learns motions from a human-centric perceptive, which enhances the consistency of human motion predictions under different camera trajectories. For more accurate and physically plausible motion capture, we further propose a contact-aware neural motion descent module so that our network can be aware of foot-ground contact and motion misalignment. Based on the proposed solution, we demonstrate a real-time monocular full-body capture system under moving cameras.

6.0.0.1 Limitations.

As our method recovers 3D motion from 2D joint observations, the depth ambiguity issue remains especially when the person is captured in nearly static poses.

Supplementary Material

7 Implementation Details↩︎

7.1 Human-to-World Coordinate Transformation↩︎

We estimate the local human pose of each frame in our proxy-to-motion network, then we transform it into the global world space. In the first frame, the accumulated translation of human in x-z plane \(t_{xz}\) is set to zero. For the later human-space estimations, we firstly rotate the front axis of camera in human space to align the y-z plane by \(R_{front}\). Denote the target parameters of camera in world space as \(R_W, T_W\) and the predicted parameters in human space as \(R_H, T_H\). We have \(R_W = R_H \cdot R_{front}\), \(T_W = -R_W \cdot (R_{front}^T \cdot (-R_H^T \cdot T_H) + t_{xz})\). And the human orientation should also be rotate in the same time to maintain relative stillness: \(\theta_W(root) = R_{front}^T \cdot \theta_H(root)\). The world translation can be calculated by \(t_W = R_{front}^T \cdot (t_H+J_{root})-J_{root} + t_{xz}\). Here \(J_{root}\) is the root joint of SMPL model in T-pose to eliminate the error resulted from the root-misalignment with the original point of SMPL model. Finally, the accumulated translation of human in x-z plane \(t_{xz}\) is updated by \(t_{xz}^{t+1} = t_{xz}^{t} + R_{front}^T \cdot t_H\).

7.2 Partial Dilated Convolution↩︎

It should be noted that the hand area, being an extremity, is more prone to being affected by heavy motion blur and severe occlusions, resulting in missing detections. Simply setting the corrupted data to zero is not a viable solution as the original convolution kernel is unable to distinguish between normal data and corrupted data, leading to a significant reduction in performance as noise is propagated through the network layers.

To overcome this challenge, we employ partial convolution [77] to enhance our 1D dilated convolution framework. As illustrated in Fig. 7, rather than indiscriminately processing the input signals as in the original convolution operator, we utilize a mask-weighted partial convolution to selectively exclude corrupted data from the inputs. This enhances the robustness of hand recovery in scenarios involving fast movement and heavy occlusion. Specifically, the latent code \(X_0\) is initially set as the concatenation of the \((x, y)\) coordinates of the \(J\) joints for each frame \(f\in1, 2, ..., L\), while the mask \(M_0\) is initialized as a binary variable with values of 0 or 1, corresponding to the detection confidence. Then we integrated the 2D partial convolution operation and mask update function from  [77] into our 1D dilated convolution framework: \[\label{equ:partial95conv} \begin{cases} M_{k+1} = I\left(sum(M_k)>0\right) \\ X_{k+1} = M_{k+1} \left( W_k^\top \left(X_k \odot M_k \right) \frac{size(M_k)}{sum(M_k)} + b_k \right) \end{cases}\tag{5}\] where \(W\) and \(b\) denotes the convolution filter weights and bias at layer \(k\), and \(\odot\) denotes the element-wise multiplication. Furthermore, in the training procedure, half of the sequential inputs are randomly masked to simulate detection failures that may occur in a deployment environment.

7.3 Training↩︎

We train our network using the Adam optimizer with a learning rate of 1e-4 and a batch size of 1024 in NVIDIA RTX 4090. We adopt a two-stage training scheme. Firstly, we train our proxy-to-motion initialization network (Sec. 4.2) for 50 epochs. Subsequently, we fix the weights of the motion recovery network and train the neural descent network (Sec. 4.3) for another 50 epochs.

7.4 Proxy Dataset↩︎

We conduct the training process on our synthetic proxy dataset. The 3D body rotational motions of the training set are sampled from AMASS [35]: [ACCAD, BMLmovi, BMLrub, CMU, CNRS, DFaust, EKUT, Eyes Japan Dataset, GRAB, HDM05, HumanEva, KIT, MoSh, PosePrior, SFU, SOMA, TCDHands, TotalCapture, WEIZMANN] and [SSM, Transitions] are for testing. Otherwise, for the generation of hand motions, we adopt the same dataset division of InterHand [69]. Then We animate the SMPL-X mesh and generate virtual cameras to obtain the pseudo 2D labels.

8 Computational Complexity↩︎

In this section, we compare the inference speed of our method. Our real-time monocular full-body capture system can be implemented on a single Laptop (NVIDIA RTX 4060 GPU). Specifically, for the 2D pose estimator, we leverage the off-the-shelf Mediapipe [78] and MMPose and re-implement it on the NVIDIA TensorRT platform. We report the inference time of each module in Table. 3.

References↩︎