1 Introduction↩︎

In this work, we focus on monocular 3D human pose estimation (HPE), where the goal is to estimate 3D human skeletal joints from a single 2D image. This problem is inherently ill-posed due to the ambiguity introduced by single-view (monocular) observations.

While there exist methods that solve this task in an end-to-end manner, a common approach is two-stage: 1) detect 2D keypoints from the input image, by using a pre-trained detector such as HRNet [2], and 2) lift the detected 2D keypoints to 3D by means of another trained model. A geometrically consistent 2D-to-3D lifting model must respect rotation equivariance, as the input 2D keypoints rotate in-plane (about the image centre), the output 3D pose must rotate accordingly about the optical axis. Lifting models, however, typically do not exhibit this property, since it is insufficiently represented in the data they were trained on, and their design does not account for it; see the predictions of such a model without equivariance in Figure 1.

To address this, we investigate whether rotation equivariance should be enforced by architectural design or can be learned effectively by data augmentation. We evaluate these approaches on the common human pose estimation benchmarks Human3.6M [3] and MPII-INF-3DHP [4], as well as EMDB [1] and SportsCap [5].

The scope of our work and the assumptions we make are thus as follows: (i) the input is a single 2D image, a single frame; (ii) the 2D keypoints are detected by an off-the-shelf keypoint detector, equivariance is only investigated in lifting models; (iii) no additional features are provided in the input to the lifting model, besides 2D keypoints.

Our contributions are that we quantitatively and qualitatively demonstrate that for monocular 3D HPE,

• Typical lifting models lack geometric consistency and fail on rotated or rotation-like out-of-distribution poses,

• We address this by incorporating rotation equivariance as an inductive bias, showing that the way it is incorporated plays a crucial role in performance (see Figure 1),

• Rotation equivariance can be effectively learned via rotation-based data augmentation, outperforming fully equivariant methods in this context, while maintaining a lower computational cost.

2 Background↩︎

2.1 Human Pose Estimation↩︎

Monocular 3D human pose estimation is a challenging but vast field of research. Methods can be split into two categories: two-stage and end-to-end approaches.

2.1.1 Two-Stage Approaches↩︎

Two-stage approaches decompose monocular 3D human pose estimation into two stages: 2D keypoint detection and 3D pose lifting. This allows the lifting module to focus completely on geometric reasoning.

A well-established baseline in this category is the work by Martinez  [6], which uses a lightweight fully connected network to regress 3D joint coordinates from 2D inputs. Despite its simplicity, this method achieves strong performance on Human3.6M [3] and clearly demonstrates that accurate 3D pose estimation is feasible given reliable 2D keypoints. Due to its effectiveness and minimal architectural assumptions, it remains a standard baseline for evaluating lifting-based methods, including in our work.

Subsequent research extends this paradigm by incorporating additional constraints and refinement strategies. Li  [7] propose a cascaded lifting framework with iterative refinement and data evolution to improve robustness to noisy 2D detections. Xu  [8] integrate kinematic priors directly into the lifting network, enforcing physically plausible joint relationships to reduce depth ambiguity and implausible poses. Extending lifting methods to more challenging viewpoints, Wang  [9] augment 2D keypoints with scene geometry cues, addressing severe occlusions and missing joints common in egocentric settings.

More recently, transformer-based architectures have been adopted to enhance lifting performance by modelling global spatial and temporal dependencies. PoseFormer [10] achieves improved robustness to occlusion and temporal inconsistency compared to convolutional temporal models.

2.1.2 End-To-End Approaches↩︎

End-to-end methods aim to predict 3D human pose directly from RGB images, eliminating the explicit 2D lifting stage. Wang  [11] introduce a unified framework that jointly performs detection and 3D pose estimation while modelling uncertainty through distribution-aware representations, enabling robust multi-person inference in crowded scenes. Earlier work from Moon  [12] incorporates camera distance estimation into a top-down pipeline to alleviate scale and depth ambiguities inherent in monocular images. Complementary to fully supervised end-to-end approaches, Wandt and Rosenhahn [13] propose RepNet, a weakly supervised adversarial reprojection framework that learns image-to-3D pose mappings using only 2D keypoint supervision.

For further reading and comprehensive overviews, we refer to these two survey papers [14], [15].

2.2 Rotation Equivariance↩︎

Rotations in \(n\)D are the actions of the special orthogonal group \(\text{SO}(n)\), and can be represented by \(n \times n\) matrices \(R\) such that \(R^\top R = R R^\top = \mathrm{I}_n\), with \(\mathrm{I}_n\) being the identity matrix, and \(\det{R} = 1\).

A function \(f : \mathcal{X} \rightarrow \mathcal{Y}\) is said to be \(\text{SO}(n)\)-equivariant, equivariant under \(n\)D rotations, if for every \(R \in \text{SO}(n)\), in the function output space, \(\mathcal{Y}\), there exists a transformation representation, \(\rho(R)\), such that \[\label{eq:equivariance} \rho(R) \, f(X) = f(R X) \text{\quad for all~} R \in \text{SO}(n), \; X \in \mathcal{X} \subseteq \mathbb{R}^n.\tag{1}\] If the transformation representation in the output space is identity, if for every \(R \in \text{SO}(n)\), \(\rho(R) = \mathrm{I}_n\), we call a function \(f: \mathcal{X} \rightarrow \mathcal{Y}\)  \(\text{SO}(n)\)-invariant: \[\label{eq:invariance} f(X) = f(R X) \text{\quad for all~} R \in \text{SO}(n), \; X \in \mathcal{X} \subseteq \mathbb{R}^n.\tag{2}\]

Rotation equivariance is a powerful inductive bias, and when incorporated appropriately, leads to improved and robust performance, in point-cloud classification [16], trajectory prediction [17], molecular properties prediction [18], and energy and force prediction for materials science applications [19].

In this work, we focus specifically on equivariance under 2D rotations, \(\text{SO}(2)\) actions, as well as their embedding into \(\text{SO}(3)\), in the context of 2D-to-3D lifting for monocular 3D HPE. This is unlike the DECA method [20], which is an end-to-end HPE method working on depth images, equivariant to camera viewpoint changes (3D rotations of the camera around the human).

In addition, the work by Howell et al. [21] examines the theoretical basis for incorporating rotation equivariance constraints when learning 3D representations from 2D images. Namely, it shows that only restricted symmetry actions can be consistently defined between 2D input images and 3D outputs. While we do not build our work on this framework directly, our observations made in Section 4 are consistent with their analysis.

3 Methodology↩︎

3.1 Geometrically Consistent Lifting Model↩︎

We call a lifting model, \(f(X): \mathbb{R}^{N\times2} \rightarrow \mathbb{R}^{N\times3}\), geometrically consistent if the following equivariance property holds for any \(X \in \mathbb{R}^{N\times2}\) and an in-plane rotation, \(R_\theta \in \text{SO}(2)\), about the angle \(\theta \in [0, 2\pi)\): \[\label{eq:geometrical95consistency} f(X\,R^\top_\theta) = f(X) \begin{bmatrix} R^\top_\theta & 0 \\ 0 & 1 \end{bmatrix} ,\tag{3}\] which means that any (global) in-plane rotation of the input 2D keypoints (about the image centre) corresponds to a rotation about the optical axis of the output 3D point set (see Figure 2), leaving the depth-coordinate invariant. At the same time, enforcing exact equivariance given by Eq. 3 may be too restrictive since the true 2D-to-3D mapping is often only approximately rotation-equivariant in practice. The exact equivariance can thus prevent the model from producing an optimal prediction.

In Section 4, we investigate how geometrically consistent common 2D-to-3D lifting models [6], [10] are and how useful and limiting the constraint in Eq. 3 becomes in practice by employing the following types of models:

• Fully equivariant: Models that satisfy Eq. 3 exactly by design

• Hybrid: Models that only produce rotation-equivariant \(xy\)-coordinates by design, with a non-invariant output depth (hybrid) and trained to produce approximately invariant depth (hybrid + aug)

• Vanilla: Models that are non-equivariant (vanilla) and trained to approximately satisfy Eq. 3 by augmentation (vanilla + aug).

3.2 Models↩︎

In this section, we outline the models we build according to the types described in Section [sec:method]. We strive to conduct a fair comparison and therefore construct the models with a similar number of parameters (\({\sim}3.7-4.4\)M) determined by, when applicable, a matching number of layers and feature dimensions. Our choice of models is motivated by their relevance to the scope of our work (see Section 1), reported performance on related tasks, and reproducibility.

3.2.1 Fully equivariant↩︎

We utilise VN-Transformer [17], a transformer architecture based on the vector neurons framework [16] for rotation-equivariant point cloud processing. In addition, we employ GotenNet [18], a recent efficient rotation-equivariant model for molecular analysis. We use their open-source implementations.

In order to utilise these models, we need to ensure that the input joints (2D) are in the same dimension as the output (3D). While we can use a constant value as the third coordinate for all input points for VN-Transformer (\(1\)), this does not work for GotenNet due to the joint-wise differences used in the model, which leads to the third coordinate in the output always being zero. Thus, for both models, we append each 2D \(xy\)-vector in the \(N\times 2\) input with its norm (which is rotation-invariant), \(z=\lVert xy \rVert\), preserving rotation equivariance. We present an ablation study on this in Section 4.3.2.

3.2.2 Vanilla↩︎

We use the simple ResNet-based baseline by Martinez et al. [6], as well as a transformer-based baseline, PoseFormer, by Zheng et al. [10]. These vanilla models do not consider rotation equivariance in their 3D human pose estimates, thus failing when encountering a rotated out-of-distribution pose, as seen in the example in Figure 1.

The architecture from Martinez et al. [6] consists of two sequential blocks of ResNet, taking the flattened 2D pose as input vector, and outputs the corresponding 3D pose vector. The PoseFormer [10] models the spatial and temporal correlation between human joints via the transformer architecture. Originally, PoseFormer is designed to take in a sequence of 2D poses as input. To maintain fair comparison, we modify the architecture to work with single-frame inputs by reducing the length of the input sequence to \(1\) and decreasing the depth of the temporal transformer module to \(2\), resulting in a comparable number of learnable parameters.

3.2.2.1 Reference method (FMPose)

For reference and a broader perspective, we also include FMPose [22], a state-of-the-art flow-matching-based method for monocular 3D HPE, with \(4.5\)M learnable parameters in total. However, unlike the other models within the scope of our work, FMPose operates on multiple 2D poses (sets of keypoints), obtained from HRNet-predicted [2] keypoint heatmaps, and produces multiple 3D hypotheses. Later in Section 4.3, we therefore report both its deterministic (single hypothesis) and probabilistic (best of multiple hypotheses) variants.

3.2.3 Hybrid↩︎

In order to analyse the potential limitation of the expressiveness of fully equivariant models mentioned in Section [sec:method], we combine an equivariant model, VN-Transformer or GotenNet, used for \(xy\)-prediction only, with the ResNet baseline [6] predicting the depth coordinate, as shown in Figure 3. Given the VN-Transformer implementation, we straightforwardly instantiate it for 2D, whereas the GotenNet implementation is optimised for 3D, and we run it as-is by initialising the third coordinate to \(0\) and extracting the first two coordinates for each joint in the output pose. In both cases, we reduce the number of parameters of the equivariant models to keep the same non-equivariant baseline ResNet, so that the total number of parameters falls within the range of the other models in comparison.

4 Experiments and Results↩︎

4.1 Datasets↩︎

We first train the models described in Section 3.2 on the Human3.6M dataset [3], which contains diverse motion capture data in a laboratory setup. Following the standard training protocol [6], [10], we use the first five subjects (S1, S5, S6, S7, S8) for training and the remaining two (S9, S11) for testing. To examine the generalisation of the models, we additionally evaluate them on the test set of the MPII-INF-3DHP dataset [4] without any fine-tuning. Besides these two common datasets that mostly contain simple standing, sitting, and walking poses, we evaluate the models on the more challenging EMDB [1] dataset (specifically, the two test sequences 68_outdoor_handstand and 69_outdoor_cartwheel; see Figures 1 and 6) and qualitatively on representative samples from the SportsCap dataset [5] (without 3D ground truth), containing complex poses with realistic full-body rotations (see Figure 4).

4.1.1 Metrics↩︎

We use the mean per-joint position error (MPJPE) as the main evaluation metric for comparisons. The MPJPE measures the average L2 distance between the estimated 3D human poses \(\hat{\mathbf{X}}\in\mathbb{R}^{N\times 3}\) and ground truth \(\mathbf{X}\in\mathbb{R}^{N\times 3}\) in millimetre (mm), computed as \[\mathrm{MPJPE}(\mathbf{X}, \hat{\mathbf{X}}) = \frac{1}{N} \sum_{n=1}^{N} \left\|\mathbf{X}_n-\hat{\mathbf{X}}_n\right\|_{2}~. \label{eq:mpjpe}\tag{4}\] In Table 1, we refer to the MPJPE metric as Protocol 1. We also report the Procrustes-aligned mean per-joint position error (PA-MPJPE), commonly known as Protocol 2, in Table 2. The PA-MPJPE eliminates global rotation, translation, and scaling errors, focusing instead on the local joint pose error between the estimations and the ground truth.

4.2 Implementation Details↩︎

4.2.1 Training details↩︎

We use PyTorch [23] in our experiments. The inputs to the models are the 2D poses collected from an off-the-shelf estimator HRNet [2]. The input 2D poses are standardised to a zero mean and a standard deviation of one. The ground-truth 3D poses are in camera coordinates and root-aligned at the origin. All models are trained to minimise the mean squared error (MSE) between the ground truth and estimated human poses, for a total of \(100\) epochs on the training set of the Human3.6M dataset, with a batch size of \(1024\) (reduced to \(512\) for GotenNet due to computational constraints), Adam optimiser, an initial learning rate of \(10^{-3}\), and the ExponentialLR scheduler with \(\gamma = 0.96\).

For FMPose, included as a state-of-the-art reference, we adopt the preprocessing and training setting from the official implementation [22].

4.2.2 Data augmentation↩︎

When \(+\)aug is specified, we perform data augmentation by randomly transforming the input \(X\in \mathbb{R}^{N\times 2}\) and the \(xy\)-coordinates of the corresponding target pose \(\mathbf{X}\in \mathbb{R}^{N\times 3}\) with the same rotation \(R_\theta\). The rotation angle \(\theta\) is sampled from a uniform distribution, \(\theta \sim \mathcal{U}[0, 2\pi)\), during training. To evaluate the equivariance property, geometric consistency, we also apply random rotations to the test data in the same way, when Rotated is specified.

4.2.3 Computational resources↩︎

All experiments are conducted on the NVIDIA A100 devices with 40GB of memory. To demonstrate the reproducibility, each model is evaluated on three different random seeds. The mean error and standard deviation across three runs are reported for the quantitative results. The training and inference times are presented in Table 3.

4.3 Results and Discussion↩︎

The results presented in Tables 1 and 2 consistently demonstrate that the vanilla lifting models do not learn the equivariance property from the data (see the performance on the Original vs. Rotated). However, we address this effectively via augmentation, resulting in vanilla+aug and hybrid+aug models outperforming their fully equivariant and non-augmented counterparts, across all the Rotated datasets. Notably, the performance of the +aug models, in most cases, is slightly reduced on the Original datasets, since poses resembling full-body rotations are underrepresented in them. Furthermore, the domain generalisation of the vanilla models (see Original MPII-INF-3DHP results) and vanilla+aug models (see Rotated MPII-INF-3DHP results) is better than that of other models, suggesting that the augmentation does not incur overfitting to a specific dataset.

Overall, augmentation-based models (vanilla+aug and hybrid+aug) consistently achieve the best performance, followed by fully equivariant and non-augmented (hybrid and vanilla) models, confirming our intuition that strictly imposing the equivariance constraint Eq. 3 on a lifting model harms the task performance while guaranteeing robustness to rotations by design. These observations are supported by the qualitative examples presented in Figures 5, 6, and 7, and more in the Supplementary Material.

Remarkably, the best-performing models overall, vanilla+aug, are also among the fastest to train and have the fastest inference, as presented in Table 3. While the augmentation adds to the training time, in total it constitutes only an insignificant fraction of the training time required by fully equivariant models.

Evaluated on the subset of EMDB 2D pose sequences containing natural, realistic full-body rotations (see the rightmost column in Tables 1 and  2 and Figures 1 and 6), we observe that the models endowed with learned equivariance (vanilla+aug and hybrid+aug) outperform their non-equivariant and fully equivariant counterparts.

All in all, the baseline ResNet [6] endowed with equivariance learned via augmentation clearly exhibits a favourable performance/training/inference trade-off, followed closely by PoseFormer+aug and hybrid+aug GotenNet. Notably, despite its strong performance on the original test sets, FMPose exhibits significant degradation under in-plane rotations (see Original vs. Rotated in Tables 1 and 2), consistent with the behaviour of vanilla lifting models. This indicates that rotation equivariance is not inherently captured even by modern generative pose models. When trained with rotation-based augmentation, however, its performance becomes stable across rotations, both for the deterministic and probabilistic (multi-hypothesis) variants, and improves on the natural rotations (the EMDB sequences); see FMPose vs. FMPose+aug.

This further supports our findings, demonstrating that the benefits of learning rotation equivariance via augmentation extend beyond simple lifting architectures to more complex probabilistic models. As a trade-off, however, FMPose has considerably higher inference cost than the simpler models (see Table 3).

4.3.1 Rotation equivariance diagnostics↩︎

In addition, we sample 1024 human pose images from the Human3.6M test set and run rotation diagnostics on their HRNet-detected 2D keypoints. For each resulting 2D pose, \(X \in \mathbb{R}^{N \times 2}\), we apply 24 uniformly spaced rotations \(\theta \in [0, 2\pi)\), and evaluate each model, \(f(X): \mathbb{R}^{N\times2} \rightarrow \mathbb{R}^{N\times3}\), using the two metrics: MPJPE 4 for consistency with the previous evaluation and the normalised equivariance error \[\label{eq:equivariance95error} \epsilon_\text{eq}(X, \theta) = \frac{\sum_{n=1}^N \left\| f(X\,R^\top_\theta)_n - \left( f(X) \begin{bmatrix} R^\top_\theta & 0 \\ 0 & 1 \end{bmatrix} \right)_n \right\|_{2}}{\sum_{n=1}^N \left\| f(X)_n \right\|_{2}}~.\tag{5}\] The results in Figure 8 indicate that common lifting models (vanilla) exhibit strong performance degradation under rotations, while vanilla+aug models maintain stable high accuracy and close to constant equivariance error. In contrast, equivariant-by-design models preserve equivariance but at the cost of reduced accuracy. These trends are further illustrated qualitatively in the Supplementary Material (see Figure 11).

4.3.2 Ablation studies↩︎

4.3.2.1 Input depth initialisation for equivariant models

In Table 4, we present an ablation study on the input depth initialisation for the fully equivariant models. Here, we compare the selected initialisation with the norm of the 2D coordinate of the joint with other approaches. Namely, one alternative is when we append every \(N\times 2\) input with a vector of size \(N\) sampled randomly only once when the model is instantiated (therefore, having the same specific depth per joint for all of the inputs), which we call fixed rand init \(z\). Its modification is when this vector becomes learnable (learn init \(z\)). As we see from the results, the learnable initial depth for the input does not improve results beyond reducing the variance compared to the fixed, randomly initialised depth. Similarly, the homogeneous input strategy (applicable only for the VN-Transformer) does not improve the results. Hence, we use the norm depth initialisation (\(z = \lVert xy\rVert\)) for both models.

4.3.2.2 Hybrid model ablation

We also conduct an ablation study for the hybrid models by using the first layer equivariant model features for the depth prediction (see Figure 3). The results presented in Table 5 suggest that no consistent improvement is gained compared to the original hybrid models, despite the parameter count increase caused by the growth of the dimensionality in the input to the non-equivariant baseline for depth prediction.

5 Conclusions↩︎

In this work, we show that typical 2D-to-3D lifting models, including current state-of-the-art methods for monocular HPE, exhibit significant performance degradation on poses resembling in-plane or full-body rotations, and address this by incorporating rotation equivariance as an inductive bias. We compare learning equivariance via data augmentation with enforcing it via architectural design, and show that it can be effectively learned via rotation-based augmentation applied jointly to input and output poses, outperforming methods that are fully equivariant by design. This suggests that strictly enforcing equivariance constraints can be misaligned with the monocular lifting problem, whereas learning equivariance from data via augmentation provides a more flexible and effective alternative.

5.0.0.1 Limitations

The scope of our work, as outlined in Section 1, per se, constitutes the main limitations of our work. The selection of the models designed specifically for human pose estimation was conducted using our work scope as the main criterion, as well as the reported performance of the models and their official implementation availability (which affects reproducibility). The selection of the equivariant-by-design models was done based on their strong reported performance on point set/point cloud tasks.

5.0.0.2 Future work

The methodology of this work can be applied to analysing temporal and probabilistic lifting models, as well as lifting models incorporating more features beyond 2D keypoints, and end-to-end methods. The application scope can also be extended beyond HPE and rotation symmetries.

Acknowledgments↩︎

This work was supported by the Wallenberg AI, Autonomous Systems and Software Program (WASP), by the Swedish Research Council (VR) through a grant for the projects Uncertainty-Aware Transformers for Regression Tasks in Computer Vision (2022-04266) and Dorsal Stream Robot Vision (2022-04206), and the strategic research environment ELLIIT. The computations were enabled by resources provided by the National Academic Infrastructure for Supercomputing in Sweden (NAISS), partially funded by the Swedish Research Council through grant agreement no. 2022-06725.

6 Additional qualitative results↩︎

We present additional qualitative results on the gymnastic and pole vault poses from the SportsCap dataset, in Figures 9 and 10 respectively. We additionally illustrate the rotation diagnostics from the main paper with a qualitative example in Figure 11.

References↩︎