1 Introduction↩︎

Wearable sensors like smart glasses [1], rings [2], and wristbands [3] are becoming ubiquitous. Beyond enabling XR experiences, they serve as everyday companions that continuously monitor user activities. Consequently, developing algorithms to robustly perceive human-object interactions from such sparse signals is a key research challenge with far-reaching impact, unlocking applications in healthcare, personal assistants, entertainment, robotics, and spatial AI.

Recent work [4] demonstrates that it is possible to reconstruct users’ motion in-the-wild from sparse sensors. Despite these advances, egocentric human-object interaction (HOI) remains underexplored. Existing methods [5], [6] typically require RGB images, pre-scanned scenes, or specialized capturing suits, and rely on hand-crafted constraints, restricting scalability and generalization. While learning-based approaches offer a promising alternative, current HOI datasets remain limited in scale and diversity, especially when compared to large human motion collections such as AMASS [7]. Hence, what is missing is a unified egocentric HOI method capable of learning from multiple modalities jointly.

We address this gap with ECHO (Fig. 1), the first method to model full-body HOI from head and hand tracking over arbitrary-length sequences. ECHO simultaneously predicts human pose, object trajectory, and contact dynamics. While anchored in 3-point tracking, it can additionally leverage sparse observations from any of these modalities (e.g., partial object tracking) to further constrain its predictions. The model is also robust to intermittent hand tracking, which is common in real-world scenarios, and has no constraints on the sequence length.

We achieve this through a unified framework with several innovations. The core of the method is a tri-variate diffusion formulation that effectively learns the inter-modal relationships between human, object, and contacts. This design enables training on a combination of datasets, including human-only motion collections such as AMASS [7] and HOI datasets like BEHAVE [8] and OMOMO [9]. By using these data sources within one generative framework, the model learns a strong human motion prior while simultaneously learning interaction-specific dynamics, something existing HOI modeling methods cannot achieve.

Crucially, our approach supports flexible conditioning, leveraging partial observations of any modality to ensure robustness against sensor noise and intermittent tracking. By explicitly modeling both human-environment and human-object contacts, we further enable self-supervised inference guidance to ensure physically plausible interactions. Finally, we extend the inference approach of HMD² [4] with a novel smooth inpainting that blends past and current predictions, enabling seamless, real-time processing of arbitrarily long sequences.

ECHO relies on minimal assumptions, making it flexible and adaptable to diverse settings. Its tri-variate diffusion formulation naturally accommodates additional modalities (e.g., human tracking from IMUs or object tracking from RGB), making ECHO a universal approach for egocentric HOI modeling. ECHO achieves state-of-the-art performance, surpassing competitors that lack the same flexibility. Through detailed evaluation, we demonstrate the effectiveness of our design choices, which we believe will be instrumental for further research in egocentric perception and interaction modeling.

Our key contributions are:

• We present the first method to reconstruct HOI from wearable 3-point tracking, jointly recovering human motion, object trajectory, and contact.

• We introduce a unified tri-variate diffusion model that supports flexible cross-modal conditioning and partial observations, is robust to sensor noise, and performs inference for arbitrary-length sequences.

• We achieve state-of-the-art performance and validate our design through extensive ablations. The code and trained model are available at https://github.com/ptrvilya/echo.

2 Related Work↩︎

2.1 Egocentric Motion Reconstruction↩︎

Using body-worn and head-mounted sensors for human motion reconstruction is an emerging research area. Early methods used body-worn cameras to recover hand positions [10]–[15] or full-body motion [16]–[23]. These approaches focus on joint angles, ignoring the user-scene relation.

Other works utilize body-worn sensors like EMs [24], EMGs [25], and most popularly, IMUs [26]–[34]; however, double-integrating acceleration for position induces drift.

HPS [35] advanced the field by fusing IMU tracking with camera localization for global pose estimation in large scenes. Follow-up works enabled scene scanning [36], [37], scaled datasets [38]–[42], reduced sensor counts [43], [44], [44]–[48], conditioned on past observations [49], and integrated biomechanical constraints [50]. Generative diffusion models enable realistic motion synthesis from underconstrained inputs [51]–[53]. For instance, EgoEgo [54] generates full-body motion from head trajectories, while LookOut [55] recovers head trajectories from in-the-wild video.

Recent works [4], [56]–[61] utilize modalities like RGB, point clouds, and hand detections, exploring appearance modeling [62] and multi-modal fusion [63].

We similarly use sparse head and hand conditioning. However, unlike these methods, our approach models dynamic interactions with objects, enabling a more complete reconstruction of human activity in real-world settings.

2.2 Exocentric HOI Modeling↩︎

Early HOI reconstruction focused on human-centric modeling in static environments [64]–[70], including recent work on reconstructing and placing humans within scenes [71]–[73]. These methods do not model object displacement in the scenes, limiting their applicability in real-world scenarios.

More relevant are methods modeling dynamic HOI. Many condition on text or action labels [74]–[79], which is flexible but lacks fine-grained control and detailed interactions. Such flexible conditioning is explored for human motion generation [80]–[82], but without modeling HOI. Others, like TRUMANS [83], condition on the scene, improving realism but requiring prior scene knowledge. Visual conditioning (RGB [84]–[93], multi-view [94], [95], RGBD [8], [96], or text and images [97]) boosts fidelity but relies on cameras, limiting scalability. I’M HOI [98] uses an object-mounted IMU, enhancing dynamics but requiring the instrumented object. Several methods use an AMASS motion prior externally to the HOI model via retrieval-and-optimization (InterDreamer [99]) or fine-tuning an AMASS-pretrained MDM (HOI-Diff [100]), rather than training jointly on motion-only and HOI data.

Another line of research generates interactions from partial information, such as past observations [101], [102], object positions [9], [103], [104], or human pose [105], [106]. Among these, TriDi [107] models the joint human-object-interaction distribution, but operates on static poses and trains on HOI-only data. ECHO extends the tri-variate formulation to arbitrary-length sequences and, crucially, trains from both motion-only (AMASS) and HOI data in a single model, building the robust motion prior that sparse egocentric conditioning requires.

2.3 Egocentric HOI Reconstruction Methods↩︎

Most methods that model human-scene interaction from egocentric data assume static environments [35]–[37], [46], and are therefore less relevant to our focus on dynamic interaction.

iReplica [5] is the only other method considering both humans and dynamic objects in egocentric scenarios. However, iReplica has significant limitations: it requires full-body IMUs, a pre-defined 3D scene, and a complex initialization process. Furthermore, it relies on hand-crafted heuristics for interaction modeling, whereas ECHO learns HOI dynamics from data. Recent methods like EgoGrasp [108] and WHOLE [109] recover interactions from egocentric video but focus solely on hands; Glove2Hand [110] synthesizes hand-object interaction from multi-modal sensing gloves, likewise modeling only the hands and requiring specialized worn hardware. Concurrent work IMU-HOI [111] reconstructs HOI from six body-worn IMUs plus an object-mounted IMU. Unlike ECHO, it requires a dense IMU setup and an IMU attached to the object, whereas ECHO operates from sparse three-point head-and-hand tracking and uninstrumented objects. ECHO is the first method to model full-body HOI under sparse egocentric conditioning and over arbitrary-length sequences, handling objects of various types and sizes from wearable 3-point trackers, and remaining robust to intermittent hand tracking.

3 Method↩︎

In this section, we present ECHO, the first approach for joint human-object interaction modeling from head and wrist tracking. At the core of our method is a transformer-based diffusion model that predicts human motion \(\mathcal{H}\), object motion \(\mathcal{O}\), and contact sequence \(\mathcal{I}\) from three-point conditioning. Subsequent sections introduce the representations for all modalities (Sec. 3.1) and describe our formulation of the diffusion process, architecture, and training (Sec. 3.2). Finally, we present the smooth inpainting that we adopt for online inference with arbitrary sequence lengths (Sec. 3.3).

3.1 Representation for Human, Object and Contact↩︎

3.1.1 Head-centric modeling.↩︎

One of the key design choices for our method is the representation for the network’s input and output modalities. Unlike sequence-level canonicalization [4], [54], we build on the per-frame variant of EgoAllo [56] by extending the representation to hands and objects. We choose to use the canonicalized head position on each frame as an anchor for the object’s position. This per-frame definition maintains invariance to the global configuration of the sequence and allows the use of arbitrary chunks of the sequence for inference without explicitly canonicalizing each window. We illustrate this representation in Fig. 2 and discuss details in the following paragraphs.

3.1.2 Object.↩︎

For every object, we assume that its canonical mesh is given to the model as input. Hence, we represent the object as a sequence of its \(SE(3)\) transformations in a head-centric coordinate frame, consisting of rotation and translation pairs w.r.t. the head at each frame \(\boldsymbol{T_\mathcal{O} } =(R_{\mathcal{O} },t_{\mathcal{O} })\). Following [112], we convert all rotations to \(\mathbb{R}^6\).

Thus a sequence of \(N\) object poses is denoted as: \[\mathcal{O} =\{ \boldsymbol{T_\mathcal{O} } ^{1..N} \}\]

To represent the object’s class and geometry, we encode the class label in a one-hot encoded vector \(\boldsymbol{y}_\mathcal{O}\), and extract a feature vector \(\boldsymbol{f}_\mathcal{O} \in \mathbb{R}^{1024}\) from the canonicalized object vertices \(\boldsymbol{V}_\mathcal{O}\) using PointNext [113]. The resulting pair \(\mathcal{C}_{\mathcal{O} } = (\boldsymbol{y}_\mathcal{O} , \boldsymbol{f}_\mathcal{O} )\) is used as a global conditioning for ECHO.

3.1.3 Human.↩︎

To represent the human body, SMPL-X [114] is a natural choice. SMPL-X is a parametric body model that can be seen as a function \(SMPL(\boldsymbol{T}_\mathcal{H} , \boldsymbol{\theta} , \boldsymbol{\beta}, \boldsymbol{\psi})\) of root position and orientation \(\boldsymbol{T}_\mathcal{H} \in SE(3)\), pose \(\boldsymbol{\theta}\), shape \(\boldsymbol{\beta}\), and facial parameters \(\boldsymbol{\psi}\). The function \(SMPL\) maps parameters to posed vertices \(\boldsymbol{V}_\mathcal{H} \in \mathbb{R}^{10475 \times 3}\) of the predefined template mesh. The pose vector \(\boldsymbol{\theta}\) is a concatenation of parameters for body, hands, eyes, and jaw poses in axis-angle format. As our method focuses on realistic full-body interactions, we model only the body pose \(\boldsymbol{\theta}_\mathcal{H} \in \mathbb{R}^{21 \times 6}\) part of the pose vector and assume known shape parameters \(\boldsymbol{\beta}\); for the rest of the manuscript, we simplify the notation to \(SMPL(\boldsymbol{T}_\mathcal{H} , \boldsymbol{\theta}_\mathcal{H} )\). We follow standard practice and infer \(\boldsymbol{T}_\mathcal{H}\) via aligning the head joint of the SMPL-X model with the head position from three-point tracking.

Hence, we define a sequence of human motion with \(N\) frames as: \[\mathcal{H} = \{ \boldsymbol{\theta}_\mathcal{H} ^{1..N} \}\]

3.1.4 Contacts.↩︎

Explicit contact modeling is crucial for robust interaction reconstruction, providing strong training signals, enabling self-supervised guidance, and allowing flexible control via sparse conditioning. We model contact as a continuous modality \(\mathcal{I}\), encompassing both human-object and human-ground interactions. For human-object contact, we compute the shortest distance \(d(p, \boldsymbol{V}_\mathcal{O} )\) from sampled SMPL-X surface points \(\boldsymbol{P}_{c}\) to the object. To fit the diffusion framework and avoid instability, we map distances to \([0, 1]\) using sigmoid: \[\boldsymbol{c}_{\mathcal{I} }^{\text{HOI}} = \left\{ \sigma \left( \alpha \cdot (\tau_c - d(p, \boldsymbol{V}_\mathcal{O} )) \right) \mid p \in \boldsymbol{P}_{c}\subset \boldsymbol{V}_\mathcal{H} \right\}\] where \(\sigma\) is the sigmoid function, \(\tau_c\) is the distance threshold, and \(\alpha\) controls decay sharpness. Similarly, we compute human-environment contact \(\boldsymbol{c}_{\mathcal{I} }^{\text{Env}}\) for lower body joints based on velocity and ground proximity [56], [115]. The final contact vector is \(\boldsymbol{c}_{\mathcal{I} }= \{\boldsymbol{c}_{\mathcal{I} }^{\text{HOI}}, \boldsymbol{c}_{\mathcal{I} }^{\text{Env}}\}\), and the sequence is defined as: \[\mathcal{I} =\{ \boldsymbol{c}_{\mathcal{I} }^{1..N} \}\]

3.1.5 Egocentric conditioning.↩︎

ECHO extends the conditioning formulation of EgoAllo [56] by incorporating relative hand transformations at each frame into the method’s conditioning. The method is conditioned on canonicalized head and both hands orientations \(\boldsymbol{R}^{t}_{\text{can,head}} , \boldsymbol{R}^{t}_{\text{can,hands}}\), head-to-floor distance \(h^{t}_{\text{head}}\), and relative head and hand transformations. The relative head transformation between the current and previous frame \(\Delta \boldsymbol{T}^{t-1, t}_{\text{head}} \in SE(3)\) is computed as: \[\Delta \boldsymbol{T}^{t-1, t}_{\text{head}} = (\boldsymbol{T}^{t-1}_{\text{world, head}})^{-1} \cdot \boldsymbol{T}^{t}_{\text{world, head}}\] where \(\boldsymbol{T}^{t}_{\text{world, head}}\) is the transformation of the head at time \(t\). Similarly, we compute the relative transformation for the hands \(\Delta \boldsymbol{T}^{t-1, t}_{\text{hands}}\). We represent the transformation as \(\mathbb{R}^6\) rotation and \(\mathbb{R}^3\) translation before passing it to the network.

Thus, egocentric conditioning for a sequence \(\mathcal{E}\) is defined as: \[\mathcal{E} = \{ [\Delta \boldsymbol{T}^{t-1, t}_{\text{head}} , \boldsymbol{R}^{t}_{\text{can,head}} , h^{t}_{\text{head}} , \Delta \boldsymbol{T}^{t-1, t}_{\text{hands}} , \boldsymbol{R}^{t}_{\text{can,hands}} ]^{1..N} \}\]

3.2 ECHO model↩︎

ECHO models the joint distribution of human motion \(\mathcal{H}\), object motion \(\mathcal{O}\), and contact sequence \(\mathcal{I}\), conditioned on the three-point tracking \(\mathcal{E}\) and object features \(\mathcal{C}_{\mathcal{O} }\). Below, we formulate a three-variate diffusion process that is used to model the three modalities within one network and provide details on the underlying network’s architecture.

3.2.1 Background↩︎

A diffusion process in the context of generative neural networks is divided into two phases. The forward phase progressively adds noise to an original data sample, while the backward phase uses a learned model to recover the sample from noise. We adopt the formulation of Denoising Diffusion Probabilistic Model (DDPM) [116] in our work with a modification following [117], predicting the original sample with the neural network instead of predicting the added noise. To achieve this, we parametrize the reverse process by a denoising neural network \(\mathcal{D}_\psi\) that is trained to recover the original sample \(\mathbf{z}^0\) from the noisy sample \(\mathbf{z}^\mathcal{T}\) at denoising step \(\mathcal{T}\) given the condition \(c\). Defining for brevity \(\mathbb{E}_p \equiv \mathbb{E}_{\mathbf{z}^0 \sim p_{data}}\), \(\mathbb{E}_\mathcal{T} \equiv \mathbb{E}_{\mathcal{T} \sim \mathcal{U}\{0,...,T\}}\), and \(\mathbb{E}_q \equiv \mathbb{E}_{\mathbf{z}^\mathcal{T} \sim q(\mathbf{z}^\mathcal{T} | \mathbf{z}^0)}\) we obtain the training objective (the full definition of forward and backward processes is included in the Sup. Mat.): \[\begin{align} \min_\psi \mathbb{E}_p\, \mathbb{E}_\mathcal{T}\, \mathbb{E}_q\, \|\mathcal{D}_\psi(\mathbf{z}^\mathcal{T}; c, \mathcal{T}) - \mathbf{z}^0\|_2. \end{align} \label{eqn:objective}\tag{1}\]

The original formulation of the diffusion model [116], [118] focuses on generating single-modality data, e.g., images. Inspired by TriDi [107], we formulate a tri-variate diffusion process for HOI modeling, proposing a new formulation that diffuses motion sequences.

3.2.2 Tri-variate diffusion with independent schedules.↩︎

We formulate a three-variate diffusion process for human motion \(\mathcal{H}\), object trajectory \(\mathcal{O}\), and sequence of contacts \(\mathcal{I}\), denoting the corresponding sets of denoising steps as \(\mathcal{T}_\mathcal{H} , \mathcal{T}_\mathcal{O} , \mathcal{T}_\mathcal{I}\). We define: \[\begin{align} \mathbb{E}_p &\equiv \mathbb{E}_{(\mathcal{H} ^0, \mathcal{O} ^0, \mathcal{I} ^0) \sim p(\mathcal{H} ,\mathcal{O} , \mathcal{I} | \mathcal{E} )},\\ \mathbb{E}_\mathcal{T} &\equiv \mathbb{E}_{(\mathcal{T}_\mathcal{H} , \mathcal{T}_\mathcal{O} , \mathcal{T}_\mathcal{I} ) \sim \mathcal{U}\{0,...,T\}^{N\times3}},\\ \mathbb{E}_q &\equiv \mathbb{E}_{\mathcal{H} ^{\mathcal{T}_\mathcal{H} } \sim q(\mathcal{H} ^{\mathcal{T}_\mathcal{H} } | \mathcal{H} ^0), \mathcal{O} ^{\mathcal{T}_\mathcal{O} } \sim q(\mathcal{O} ^{\mathcal{T}_\mathcal{O} } | \mathcal{O} ^0), \mathcal{I} ^{\mathcal{T}_\mathcal{I} } \sim q(\mathcal{I} ^{\mathcal{T}_\mathcal{I} } | \mathcal{I} ^0)} \end{align}\]

The key feature of this formulation is that ECHO diffuses the three modalities following three independent time schedules, allowing them to vary (e.g., one can provide tracking information for the human in addition to three-point conditioning and predict the object motion and contact sequence corresponding to it). The main minimization objective for parameters \(\psi\) of a model \(\mathrm{ECHO}_\psi\) is: \[\begin{align} \mathbb{E}_p \mathbb{E}_\mathcal{T} \mathbb{E}_q \| & \mathrm{ECHO}_\psi( \mathcal{H} ^{\mathcal{T}_\mathcal{H} }, \mathcal{O} ^{\mathcal{T}_\mathcal{O} }, \mathcal{I} ^{\mathcal{T}_\mathcal{I} } ; \mathcal{T}_\mathcal{H} , \mathcal{T}_\mathcal{O} , \mathcal{T}_\mathcal{I} ; \mathcal{C}_{\mathcal{O} } , \mathcal{E} ) - (\mathcal{H} ^0, \mathcal{O} ^0, \mathcal{I} ^0) \|_2 \label{eqn:diffusion95objective} \end{align}\tag{2}\]

3.2.3 Universal multi-modal architecture.↩︎

We build the ECHO denoising network on the Diffusion Transformer (DiT) [119] with rotary positional embeddings [120], adapting it to our multi-modal setting (Fig. 3). The model takes as input the sequences \(\mathcal{H}\), \(\mathcal{O}\), and \(\mathcal{I}\) with varying noise levels, along with egocentric \(\mathcal{E}\) and object \(\mathcal{C}_{\mathcal{O} }\) conditioning, and the denoising step for each modality \(\{ \mathcal{T}_\mathcal{H} , \mathcal{T}_\mathcal{O} , \mathcal{T}_\mathcal{I} \}\). A key advantage of our design is the use of independent noise schedules for each modality, unlike standard approaches that use a single schedule. This enables flexible conditioning: by setting the noise level to zero for known modalities (e.g., observed human motion), the model effectively predicts the remaining components (e.g., object motion and contacts) consistent with the input. When no interaction data is available, it generates realistic motion based solely on egocentric conditioning. Furthermore, this formulation naturally handles partial observations, allowing ECHO to leverage sparse signals – such as intermittent object tracking or partial human pose from IMUs – to constrain the generation. Ultimately, the method outputs the full human-object interaction sequence \(\{ \hat{\mathcal{H} },\hat{\mathcal{O} },\hat{\mathcal{I} } \}\).

3.2.4 Training.↩︎

During training, each modality (\(\mathcal{H}\), \(\mathcal{O}\), \(\mathcal{I}\)) is either diffused or provided as clean condition. To ensure stability, we sample from the \(2^3\) combinations of noisy/denoised states and synchronize the noise level across all diffused modalities, instead of sampling independent noise levels for each modality. This effectively trains the model to reconstruct missing modalities from the observed ones. To improve generalization, we simulate intermittent tracking by randomly dropping hand and object conditioning. Experiments confirm that ECHO remains robust to moderate tracking degradation.

The objective function is a weighted sum of six terms: reconstruction losses for each diffused modality, object trajectory smoothness, human joint error, and a foot skating penalty (details in Sup. Mat.). Given the limited scale of HOI datasets (OMOMO, BEHAVE), we augment training with the large-scale AMASS [7] dataset to learn a robust human motion prior. For AMASS samples, we replace object conditioning \(\mathcal{C}_{\mathcal{O} }\) with learnable tokens, signaling the model to ignore object interactions; ablations confirm the benefits of this joint training.

3.3 Inference↩︎

3.3.1 Smooth inpainting↩︎

To generate temporal sequences, recent methods [56] employ a sliding window approach. Unlike offline methods [52], [121], this approach doesn’t require the full sequence to be present at the start, enabling online inference. However, sliding windows require post-processing to stitch results. HMD² [4] addresses this via inpainting-based inference, conditioning new windows on past predictions. However, it discards new predictions in the overlapping region, relying solely on past data. Our idea is to take the inpainting-based inference one step further by incorporating smooth blending between past and new predictions, which we call smooth inpainting (Fig. 4). It replaces predictions in the overlapping region with a weighted average of past and current predictions, allowing for a gradual transition between windows. This enables ECHO to autoregressively generate arbitrarily long sequences with smooth transitions. Additionally, the overlap size can be tuned to balance context length and inference latency for real-time applications.

3.3.2 Inference guidance↩︎

To further enhance the quality of generated interactions, we adopt a classifier-based guidance [122] approach at inference time. Based on the guidance function’s value, the network’s prediction is updated at each step of the denoising process. We formulate the guidance loss to ensure that the predicted human and object meshes align with the predicted contact vector \(\widehat{\boldsymbol{c}}_\mathcal{I}\). The function, therefore, includes two terms: one for human-object contact and one for foot-floor contact.

4 Experiments↩︎

4.0.1 Implementation details.↩︎

The ECHO model comprises 57.7M parameters and is trained using the AdamW optimizer [124] for 300k steps (approximately 30 hours) with a batch size of 256 and a learning rate of \(5e^{-4}\) on a single NVIDIA RTX 5090 GPU. For inference, we use 100 DDPM steps to balance generation quality and latency. With a 30-frame overlap, ECHO processes a 60-frame window in 640 ms without guidance and 980 ms with guidance on an RTX 5090 GPU. This corresponds to a throughput of approximately 46 FPS and 30 FPS, respectively, enabling operation at low latency of 640–980 ms per window, , \(\sim\)​21–33 ms amortized per frame. Visualizations are generated using aitviewer [125] and Blendify [126].

4.0.2 Datasets.↩︎

We train and evaluate our method on the union of the BEHAVE [8], OMOMO [9], and AMASS [7] datasets. We follow the official train-test splits for BEHAVE and OMOMO, and the EgoAllo [56] split for AMASS. To use the SMPL-X body model in ECHO, we convert BEHAVE sequences from the SMPL+H [127] format using code from [128]. We downsample all sequences to 30 fps for consistency, as the BEHAVE data is limited to this frame rate. During training, we sample windows of size \(W=60\). For evaluation, we perform continuous inference on full sequences.

4.0.3 Metrics.↩︎

To evaluate human motion reconstruction, we use the Mean Per-Joint Position Error (MPJPE, in cm), Mean Per-Joint Velocity Error (MPJVE [44]), and Foot Contact (FC) score [54], [56]. For object reconstruction, we compute the vertex-to-vertex error \(E_{v2v}\) [102], [105] (cm), center error \(E_c\) [105] (cm), and contact accuracy \(\text{Acc}_{\mathcal{I} }\) [107]. Detailed metric definitions are provided in the Supp. Mat.

4.1 Comparison with baselines↩︎

2pt

4.1.1 Baselines.↩︎

ECHO is the first approach for the end-to-end modeling of egocentric human-object interactions. To evaluate its performance, we select several existing end-to-end egocentric motion modeling methods as baselines and extend them for HOI modeling. While some recent methods employ multi-stage training and inference, adapting them to HOI modeling would require significant architectural modifications. We compare against BoDiffusion [52], which builds on DiT [119] to process concatenated input tracking and noisy motion. During training, we canonicalize each window to a head-centric space to ensure a fair comparison. To evaluate HOI prediction, we extend BoDiffusion to include object modeling, referring to this baseline as BoDiffusion+\(\mathsf{O}\). We modify the network to process object poses and conditioning by concatenating object transformations to the human motion and egocentric tokens, and appending object shape embeddings and class information to the global conditioning. We also construct an HOI baseline on top of EgoAllo [56] by integrating object pose prediction. Since EgoAllo is originally conditioned only on head tracking, we extend it to support hand conditioning, following the official implementation. We refer to this method as EgoAllo+\(\mathsf{H}\)+\(\mathsf{O}\). For fair comparison, we train ECHO and all baselines on the union of AMASS, BEHAVE, and OMOMO. Additionally, both baselines predict object pose in a head-centric space, matching ECHO, to ensure a consistent coordinate system.

3pt

4.1.2 HOI generation.↩︎

We evaluate HOI reconstruction quality on the BEHAVE and OMOMO test sets, reporting mean and variance across three runs in Tab. ¿tbl:tab:baselines?. ECHO significantly outperforms BoDiffusion+\(\mathsf{O}\) in both human and object predictions, and performs on par with EgoAllo+\(\mathsf{H}\)+\(\mathsf{O}\) in human motion while surpassing it in object prediction. This superior object tracking demonstrates the efficacy of our tri-variate modeling. Qualitative results (Fig. 5) confirm this advantage: while the baselines often fail to maintain realistic contact, resulting in artifacts like penetration or floating, ECHO generates physically plausible interactions. We further demonstrate generalization on an Aria Digital Twin [123] sequence in Fig. 6; see the supplementary video for dynamic visualizations.

4.1.3 Motion generation.↩︎

While ECHO is designed for human-object interactions, robust human motion reconstruction is also essential. We evaluate motion quality by comparing ECHO against the baselines on the AMASS dataset. Results are reported in Tab. 1 (mean and variance across three runs). Our method outperforms both baselines, demonstrating the effectiveness of our joint modeling formulation. Notably, performance significantly degrades when ECHO is trained without the AMASS dataset (NoAMASS), highlighting the importance of large-scale motion data for learning a strong human motion prior. For fair comparison, all models (except (NoAMASS)) were trained on the union of the three datasets.

4pt

4.2 Noisy conditioning and sparse input↩︎

Egocentric human-object interactions are captured using diverse technologies and settings, often involving data streams (e.g., IMUs, head-mounted cameras) that provide additional, albeit often sparse and noisy, information. To study the robustness and versatility of ECHO under such conditions, we simulate two scenarios: noisy hand tracking and access to additional sparse tracking information. To simulate intermittent hand tracking, we randomly drop a percentage of hand tracking data from the input while keeping head tracking intact. Tab. 2 demonstrates that ECHO is robust to missing hand tracking, showing significant performance degradation only when 75% or more of the data is missing.

While tracking objects from an egocentric camera remains challenging [129], [130], the object’s location and orientation can often be determined for a few frames. For human motion, it is also possible to obtain partial tracking from RGB-based pose estimation or IMU suits. Such information provides important cues to reduce uncertainty, and ECHO is designed to leverage these opportunities. We test this capability by simulating a scenario where either the human or object modality is only partially observed. Tab. ¿tbl:tab:sparse? shows that additional constraints significantly improve the accuracy of the partially observed modality, while predictions for the unobserved modality improve slightly.

4pt

4.3 Ablation↩︎

We conduct an ablation study to analyze the effectiveness of our method’s components. All models are trained in the same setting, varying only the ablated component. First, we evaluate the model without inference-time guidance (NoGuide), observing that it primarily affects object prediction quality on noisier data (i.e., BEHAVE). Results on BEHAVE are presented in Tab. 3; OMOMO results are in the Supplementary Material. Inpaint w/o smooth performs inference using standard inpainting, without smooth blending between past and current predictions. The performance drop without smooth inpainting highlights its importance for generating temporally consistent interactions. The model without the interaction modality (i.e., using only \(\boldsymbol{(\mathcal{H} , \mathcal{O} )}\)) exhibits substantially worse quality for both human and object motion, suggesting that contacts play a crucial role in linking these modalities. Training without the AMASS dataset (NoAMASS) leads to a significant decrease in human motion modeling quality, once again highlighting the importance of large-scale datasets for learning human motion priors.

3pt

5 Conclusions↩︎

In this work, we introduced ECHO, the first unified framework for modeling human-object interactions solely from sparse egocentric tracking. Central to our approach is a novel tri-variate diffusion formulation with independent noise schedules that jointly models human pose, object motion, and contact dynamics. This design not only captures the complex interdependencies between the user and the object but also enables flexible inference, allowing the model to adapt to intermittent tracking and partial observations. Crucially, it facilitates training on a combination of large-scale human motion datasets and smaller HOI collections, learning strong priors while capturing interaction nuances. Additionally, our smooth inpainting inference mechanism ensures temporally consistent generation for sequences of arbitrary length. Extensive evaluations demonstrate that ECHO significantly outperforms state-of-the-art methods, offering a robust and versatile solution for egocentric HOI reconstruction.

5.0.1 Limitations and future work.↩︎

Despite its strong performance, ECHO has limitations that open exciting directions for future research. First, our current model focuses on object interactions and environmental contacts with the ground. Incorporating sophisticated constraints (, those coming from dynamic surroundings) would be crucial for consistent long-term motion in complex scenes. Second, although robust for various object types, the absence of fine-grained finger tracking limits the reconstruction of dexterous interactions with small objects (e.g., pens, scissors). Integrating additional modalities, such as egocentric RGB video, could help resolve these fine motor details. Finally, extending our framework to support visual conditioning, alongside the collection of currently absent RGB-based egocentric HOI datasets, represents a promising avenue for achieving comprehensive egocentric perception.

5.0.2 Acknowledgments↩︎

Special thanks to Nikita Kister and Berna Kabadayi for the helpful discussions. This work is funded by the Deutsche Forschungsgemeinschaft - 409792180 (EmmyNoether Programme, project: Real Virtual Humans). G. Pons-Moll is a member of the Machine Learning Cluster of Excellence, EXC number 2064/1 – Project number 390727645. The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting I. A. Petrov. R. Marin has been supported by the European Union’s Horizon 2020 research and innovation program under the Marie Skłodowska-Curie grant agreement No 101109330. The project was made possible by funding from the Carl Zeiss Foundation. The computational resources for this project were provided by the Google Cloud grant. This work was supported by the European Research Council (ERC) Advanced Grant SIMULACRON and by the GNI Project “AI4Twinning”.

Abstract. This supplementary material provides a discussion on the broader impacts of our work in 6. In [sec:a95background], we provide background on the diffusion process and a summary of the notation used in the text. We present additional experimental results, including evaluations with contact conditioning, ablation studies on the OMOMO dataset in 8. We report further implementation details, including the training objective, inference guidance, and smooth inpainting formulation in 9. Finally, in 10, we provide full definitions of the evaluation metrics. In the supplementary video, we show dynamic visualizations of the generated results.

6 Broader impacts↩︎

The ability of our model to capture and generate continuous human-object interactions offers significant value for fields such as digital content creation and ergonomics. This research direction could enable new applications for studying human behavior and developing realistic virtual experiences. However, this technology can also be misused. Tracking detailed human actions could lead to unauthorized surveillance, creating privacy issues. We recognize that future advances might make this technology easier to abuse. Therefore, we believe that its responsible development must be an ongoing priority.

7 Background and Notation↩︎

3pt

7.0.1 Background.↩︎

The forward diffusion process can be formulated as a Markov chain with \(T\) steps. Starting from a clean sample \(\mathbf{z}^0\), it produces a series of distributions \(q(\mathbf{z}^\mathcal{T} | \mathbf{z}^{\mathcal{T}-1})\): \(q(\mathbf{z}^{1:T} | \mathbf{z}^0) = \prod_{\mathcal{T}=1}^{T}{q(\mathbf{z}^{\mathcal{T}} | \mathbf{z}^{\mathcal{T}-1})}\). We add noise to the distribution for \(T\) steps, until finally \(\mathbf{z}^\mathcal{T}\) becomes a sample from \(\mathcal{N}(\mathbf{0},\mathbf{I})\). Defining \(\beta_0=0\), and \(\beta_\mathcal{T} \in (0,1)\) we obtain: \[\begin{align} q(\mathbf{z}^\mathcal{T} | \mathbf{z}^{\mathcal{T}-1}) & = \mathcal{N}( \mathbf{z}^\mathcal{T}; \sqrt{1 - \beta_\mathcal{T}} \mathbf{z}^{\mathcal{T}-1}, \beta_\mathcal{T} \mathbf{I} ). \end{align}\]

Formulation of DDPM [116] allows us to obtain a closed-form expression for \(\mathbf{z}^\mathcal{T}\). Let \(\alpha_i=1 - \beta_i\), \(\bar{\alpha}_\mathcal{T}=\prod_{i=1}^{\mathcal{T}}{\alpha_i}\), and \(\epsilon \sim \mathcal{N}(0, \mathbf{I})\): \[\begin{align} q(\mathbf{z}^\mathcal{T} | \mathbf{z}^0) & = \mathcal{N}( \mathbf{z}^\mathcal{T}; \sqrt{\bar{\alpha}_\mathcal{T}}\mathbf{z}^0, (1 - \bar{\alpha}_\mathcal{T}) \mathbf{I} ), \\ \mathbf{z}^\mathcal{T} & = \sqrt{\bar{\alpha}_\mathcal{T}} \mathbf{z}^0 + \sqrt{1 - \bar{\alpha}_\mathcal{T}} \epsilon. \label{eqn:sup95ddpm} \end{align}\tag{3}\]

Reversing the process, we obtain a formulation for the inference. Concretely, starting from \(\mathbf{z}^T \sim \mathcal{N}(0, \mathbf{I})\), we can step-by-step recover the sample from the original distribution. We train our network to recover the original sample \(\mathbf{z}^0\) directly as in [117] (instead of the traditional formulation, in which the added noise \(\epsilon\) is recovered). To achieve this, we parametrize the reverse process by a denoising neural network \(\mathcal{D}_\psi\) that is trained to recover the original sample \(\mathbf{z}^0\) from the noised sample \(\mathbf{z}^\mathcal{T}\) at denoising step \(\mathcal{T}\) given the condition \(c\). Defining for brevity \(\mathbb{E}_p \equiv \mathbb{E}_{\mathbf{z}^0 \sim p_{data}}\), \(\mathbb{E}_\mathcal{T} \equiv \mathbb{E}_{\mathcal{T} \sim \mathcal{U}\{0,...,T\}}\), and \(\mathbb{E}_q \equiv \mathbb{E}_{\mathbf{z}^\mathcal{T} \sim q(\mathbf{z}^\mathcal{T} | \mathbf{z}^0)}\) we obtain the training objective: \[\begin{align} \min_\psi \mathbb{E}_p\, \mathbb{E}_\mathcal{T}\, \mathbb{E}_q\, \|\mathcal{D}_\psi(\mathbf{z}^\mathcal{T}; c, \mathcal{T}) - \mathbf{z}^0\|_2. \end{align} \label{eqn:sup95dif95objective}\tag{4}\]

An iterative denoising process with denoising network \(\mathcal{D}_\psi\) is defined by the following:

\[\mathbf{z}^{\mathcal{T}-1} = \sqrt{\bar{\alpha}_{\mathcal{T} - 1}} \mathcal{D}_\psi(\mathbf{z}^\mathcal{T}; c, \mathcal{T}) + \sqrt{1 - \bar{\alpha}_{\mathcal{T}-1}} \epsilon, \label{eqn:diffusion95inference}\tag{5}\]

where \(\hat{\mathbf{z}}^0 = \mathcal{D}_\psi(\mathbf{z}^\mathcal{T}; c, \mathcal{T})\).

7.0.2 Notation.↩︎

Tab. 4 defines symbols used in our work.

8 Additional evaluation↩︎

1.6pt

3pt

4pt

4pt

3pt

8.0.1 Two-stage AMASS training.↩︎

ECHO uniquely trains jointly on motion-only (AMASS) and HOI data in a single model. To isolate the contribution of this formulation, we train an identical model in two stages (\(\mathsf{A\text{-}FT}\): AMASS pre-train \(\rightarrow\) OMOMO+BEHAVE fine-tune; Tab. 5); it degrades substantially, especially on AMASS, confirming the gain comes from the tri-variate formulation, not data access.

8.0.2 Physical plausibility.↩︎

We evaluate physical realism in Tab. ¿tbl:tab:sup95phys? using established metrics: min. H-O distance (Min. D. \([cm]\)), contact % and pen. score (\(C_{\%}, P_{sc}\) \([cm]\), CHOIS [74]), pen. depth (\(PD\) \([cm]\), DiffH₂O [131]). ECHO’s results are close to GT, outperforming baselines that produce floaters and penetrations. , BoDiffusion+\(\mathsf{O}\)’s slightly better \(P_{sc}\) stems from many floaters (only \(54\%\) \(C_{\%}\)), while ECHO balances contact and penetration with \(79.5\%\) \(C_{\%}\) at \(0.037\,cm\) \(P_{sc}\).

8.0.3 Evaluating the model with contact conditioning.↩︎

Following the evaluation of ECHO with sparse \(\mathcal{H}\) or \(\mathcal{O}\) tracking, we test the model’s performance with \(\mathcal{I}\) data provided as additional conditioning. We report the results in Tab. ¿tbl:tab:sup95contacts?. Providing contact information allows for the greatest performance improvement, compared to other modalities. This highlights that contact information plays an essential role in modeling human-object interactions.

8.0.4 Qualitative results.↩︎

More qualitative results of ECHO and comparison with baselines are provided in Fig. 7.

8.0.5 Noise simulation and ablation on OMOMO.↩︎

We complement the evaluation of ECHO with noise simulation in the main paper with results on OMOMO [9] in Tab. 6. ECHO demonstrates robustness to noise, maintaining stable performance even with significant missing hand tracking data. Furthermore, we provide ablation results on OMOMO in Tab. 7. We observe similar trends to the ablation results on BEHAVE [8].

9 Implementation details↩︎

9.0.1 Losses.↩︎

The objective function used to train our network is the weighted combination of the following losses: \[\begin{align} L^{\mathcal{H} }_{n} &= \|\boldsymbol{\theta}_\mathcal{H} - \widehat{\boldsymbol{\theta}}_\mathcal{H} \|_2 \\ L^{\mathcal{O} }_{n} &= \|\boldsymbol{T_\mathcal{O} } - \widehat{\boldsymbol{T}}_\mathcal{O} \|_2 \\ L^{\mathcal{I} }_{n} &= \|\boldsymbol{c}_{\mathcal{I} }- \widehat{\boldsymbol{c}}_\mathcal{I} \|_2 \\ L^{\mathcal{O} }_{s} &= \|\widehat{\boldsymbol{U}}_\mathcal{O} - \tau_{vel} \|_2 + \|\widehat{\boldsymbol{\omega}}_\mathcal{O} - \tau_{ang} \|_2 \\ L^{\mathcal{H} }_{j} &= \|\boldsymbol{J}_\mathcal{H} - \widehat{\boldsymbol{J}}_\mathcal{H} \|_2 \\ L^{\mathcal{H} }_{s} &= \|\boldsymbol{c}_{\mathcal{I} }^{feet} * \widehat{\boldsymbol{U}}_\mathcal{H} ^{feet}\|_2 \end{align}\] where \(\widehat{\boldsymbol{U}}_\mathcal{O}\) is the velocity of predicted object keypoints, \(\widehat{\boldsymbol{\omega}}_\mathcal{O}\) is the angular velocity of predicted object keypoints, \(\widehat{\boldsymbol{J}}_\mathcal{H}\) are predicted human joints inferred from SMPL, \(\widehat{\boldsymbol{U}}_\mathcal{H} ^{feet}\) is the velocity of predicted feet joints, and \(\boldsymbol{c}_{\mathcal{I} }^{feet}\) is the ground-truth binary contact labels for feet. The resulting loss function is: \[\begin{align} L_{{ECHO}} = & \lambda^{\mathcal{H} }_{n} L^{\mathcal{H} }_{n} + \lambda^{\mathcal{O} }_{n} L^{\mathcal{O} }_{n} + \lambda^{\mathcal{I} }_{n} L^{\mathcal{I} }_{n} + \\ & \lambda^{\mathcal{O} }_{s} L^{\mathcal{O} }_{s} + \lambda^{\mathcal{H} }_{j} L^{\mathcal{H} }_{j} + \lambda^{\mathcal{H} }_{s} L^{\mathcal{H} }_{s} \end{align}\] with weighting coefficients set to: \(\lambda^{\mathcal{H} }_{n}=\lambda^{\mathcal{O} }_{n}=5.0, \lambda^{\mathcal{I} }_{n}=1, \lambda^{\mathcal{O} }_{s}=\lambda^{\mathcal{H} }_{j}=\lambda^{\mathcal{H} }_{s}=0.01\).

9.0.2 Inference guidance.↩︎

We adopt a classifier-based guidance [122] approach at inference time. We formulate the guidance loss to ensure that the predicted human and object meshes align with the predicted contact. The function therefore includes two terms: one for human-object contact and one for foot-floor contact. The human-object term forces the contacts inferred from predicted human and object meshes to align with the contacts diffused by the network: \[\mathcal{F}_\text{HOI}(\hat{\mathcal{H} }, \hat{\mathcal{O} }, \hat{\mathcal{I} }) = \frac{1}{|\boldsymbol{P}_{c}|} \left< \boldsymbol{\hat{d}}, \widehat{\boldsymbol{c}}_\mathcal{I} ^{\text{HOI}} \right>\] where \(\widehat{\boldsymbol{c}}_\mathcal{I} ^{\text{HOI}}\) is a vector of predicted human-object contacts, \(\boldsymbol{\hat{d}} \in \mathbb{R}^{64}\) is a vector of distances between the subset \(\widehat{\boldsymbol{P}_{c}}\) of predicted human mesh vertices \(\widehat{\boldsymbol{V}}_\mathcal{H}\) and object mesh vertices \(\widehat{\boldsymbol{V}}_\mathcal{O}\): \[\mathbf{\hat{d}}_j= \min_{i = 1, \dots, |\widehat{\boldsymbol{V}}_\mathcal{O} |} \left\| \widehat{\boldsymbol{P}_{c}}^j - \widehat{\boldsymbol{V}}_\mathcal{O} ^i \right\|_2, \quad j = 1, \dots, |\widehat{\boldsymbol{P}_{c}}|.\]

For human-floor interaction, we penalize excessive foot skating based on the dynamics of predicted contacts. Similarly to [56], we define: \[\mathcal{F}_\text{skate}(\hat{\mathcal{H} }, \hat{\mathcal{O} }, \hat{\mathcal{I} }) = \sum_{t, j}{\left\| \frac{1}{2} \left( \left(\widehat{\boldsymbol{c}}_\mathcal{I} ^{\text{Env}}\right)^t_j + \left(\widehat{\boldsymbol{c}}_\mathcal{I} ^{\text{Env}}\right)^{t-1}_j \right) \left( \left(\widehat{\boldsymbol{J}}_\mathcal{H} \right)^t_j + \left(\widehat{\boldsymbol{J}}_\mathcal{H} \right)^{t-1}_j \right) \right\|_2}\] where \(\widehat{\boldsymbol{c}}_\mathcal{I} ^{\text{Env}}\) is a vector of predicted human-environment contacts, \(\widehat{\boldsymbol{J}}_\mathcal{H}\) are predicted human body joints of the SMPL model, the summation is done over time \(t\) within window length \(W\) and selected joints \(j\) (i.e., ankles and toes).

The final loss function for the guidance is a weighted sum of the above terms, with \(\lambda_\text{HOI}=150, \lambda_\text{skate}=0.25\): \[\mathcal{F}(\hat{\mathcal{H} }, \hat{\mathcal{O} }, \hat{\mathcal{I} }) = \lambda_\text{HOI}\mathcal{F}_\text{HOI}(\hat{\mathcal{H} }, \hat{\mathcal{O} }, \hat{\mathcal{I} }) + \lambda_\text{skate}\mathcal{F}_\text{skate}(\hat{\mathcal{H} }, \hat{\mathcal{O} }, \hat{\mathcal{I} })\]

We adopt the reconstruction guidance formulation of [132], where the predicted sample \((\hat{\mathcal{H} }, \hat{\mathcal{O} }, \hat{\mathcal{I} }) = \mathrm{ECHO}_\psi(\mathcal{H} , \mathcal{O} , \mathcal{I} ; \mathcal{T}_\mathcal{H} , \mathcal{T}_\mathcal{O} , \mathcal{T}_\mathcal{I} , \mathcal{C}_{\mathcal{O} } , \mathcal{E} )\) is directly modified on each denoising step. The reconstruction guidance with scale \(\lambda=0.1\) is thus formulated as: \[(\hat{\mathcal{H} }, \hat{\mathcal{O} }, \hat{\mathcal{I} }) \coloneq (\hat{\mathcal{H} }, \hat{\mathcal{O} }, \hat{\mathcal{I} }) - \lambda \nabla_{\mathcal{H} ^{\mathcal{T}_\mathcal{H} }, \mathcal{O} ^{\mathcal{T}_\mathcal{O} }, \mathcal{I} ^{\mathcal{T}_\mathcal{I} }} \mathcal{F}(\hat{\mathcal{H} }, \hat{\mathcal{O} }, \hat{\mathcal{I} }).\]

9.0.3 Smooth inpainting.↩︎

To ensure smooth transitions during inference on long sequences, we introduce smooth inpainting. This method extends standard inpainting inference [4] by blending new predictions with past ones in the overlapping window region, rather than discarding them. The blending is performed at each diffusion step according to the following formula: \[\begin{align} \hat{\mathcal{H} }^{\mathcal{T}_\mathcal{H} }_\mathcal{W} &\coloneq \alpha\hat{\mathcal{H} }^{\mathcal{T}_\mathcal{H} }_\mathcal{W} + (1-\alpha)\hat{\mathcal{H} }_{\mathcal{W}-1}, \\ \hat{\mathcal{O} }^{\mathcal{T}_\mathcal{O} }_\mathcal{W} &\coloneq \alpha\hat{\mathcal{O} }^{\mathcal{T}_\mathcal{O} }_\mathcal{W} + (1-\alpha)\hat{\mathcal{O} }_{\mathcal{W}-1}, \\ \hat{\mathcal{I} }^{\mathcal{T}_\mathcal{I} }_\mathcal{W} &\coloneq \alpha\hat{\mathcal{I} }^{\mathcal{T}_\mathcal{I} }_\mathcal{W} + (1-\alpha)\hat{\mathcal{I} }_{\mathcal{W}-1}, \end{align}\] where \(\hat{\mathcal{H} }^{\mathcal{T}_\mathcal{H} }_\mathcal{W}, \hat{\mathcal{O} }^{\mathcal{T}_\mathcal{O} }_\mathcal{W}, \hat{\mathcal{I} }^{\mathcal{T}_\mathcal{I} }_\mathcal{W}\) are the predictions for the set of denoising steps \(\mathcal{T}_\mathcal{H} , \mathcal{T}_\mathcal{O} , \mathcal{T}_\mathcal{I}\) for the current window \(\mathcal{W}\), \(\hat{\mathcal{H} }_{\mathcal{W}-1}, \hat{\mathcal{O} }_{\mathcal{W}-1}, \hat{\mathcal{I} }_{\mathcal{W}-1}\) are the predictions for the previous window \(\mathcal{W}-1\), and \(\alpha=0.4\) is a blending factor.

10 Metrics↩︎

10.0.1 Evaluating human prediction.↩︎

MPJPE measures the average \(L_2\) distance between predicted \(\widehat{\boldsymbol{J}}_\mathcal{H}\) and ground-truth body joints \(\boldsymbol{J}_\mathcal{H}\): \[\text{MPJPE}(\boldsymbol{J}_\mathcal{H} , \widehat{\boldsymbol{J}}_\mathcal{H} ) = \frac{1}{|\boldsymbol{J}_\mathcal{H} |}\sum\limits_{i \in |\boldsymbol{J}_\mathcal{H} |}{\|\boldsymbol{J}_\mathcal{H} ^i - \widehat{\boldsymbol{J}}_\mathcal{H} ^i\|_2}\]

MPJVE measures the average velocity error for predicted \(\widehat{\boldsymbol{J}}_\mathcal{H}\) and ground-truth body joints \(\boldsymbol{J}_\mathcal{H}\). Velocity at step \(i\) is computed as: \(\boldsymbol{U}_\mathcal{H} ^i = \boldsymbol{J}_\mathcal{H} ^i - \boldsymbol{J}_\mathcal{H} ^{i-1}\), thus we define: \[\begin{align} & \text{MPJVE}(\boldsymbol{J}_\mathcal{H} , \widehat{\boldsymbol{J}}_\mathcal{H} ) = \frac{1}{|\boldsymbol{J}_\mathcal{H} | - 1} \sum\limits_{i \in \{1..|\boldsymbol{J}_\mathcal{H} |\}} {\|\boldsymbol{U}_\mathcal{H} ^i - \widehat{\boldsymbol{U}}_\mathcal{H} ^i\|_2} \end{align}\]

Foot Contact (FC) measures the fraction of frames with any of 4 feet joints (ankle and foot for both legs) located closer to the ground than a pre-defined threshold (10 and 5 cm, respectively).

10.0.2 Evaluating object prediction.↩︎

\(E_{v2v}\) measures the average \(L_2\) distance between the positions of the predicted object vertices and the ground-truth ones: \[E_{v2v}(\boldsymbol{V}_\mathcal{O} , \widehat{\boldsymbol{V}}_\mathcal{O} ) = \frac{1}{|\boldsymbol{V}_\mathcal{O} |}\sum\limits_{i \in \{0..|\boldsymbol{V}_\mathcal{O} |\}}{\|\boldsymbol{V}_\mathcal{O} ^i - \widehat{\boldsymbol{V}}_\mathcal{O} ^i\|_2}\]

\(E_{c}\) measures the average \(L_2\) distance between the position of the predicted object center and the ground-truth one: \[\begin{align} & E_{c}(\boldsymbol{V}_\mathcal{O} , \widehat{\boldsymbol{V}}_\mathcal{O} ) = \left\Vert{ \frac{1}{|\boldsymbol{V}_\mathcal{O} |} \sum\limits_{i \in \{0..|\boldsymbol{V}_\mathcal{O} |\}}{\boldsymbol{V}_\mathcal{O} ^i} - \frac{1}{|\widehat{\boldsymbol{V}}_\mathcal{O} |} \sum\limits_{i \in \{0..|\widehat{\boldsymbol{V}}_\mathcal{O} |\}}\widehat{\boldsymbol{V}}_\mathcal{O} ^i }\right\Vert_2 \end{align}\]

Rotation Difference (Rot. Diff.) measures the average angular difference between predicted global rotation for the object and the ground-truth one.

The contact accuracy metric \(\text{Acc}_{\mathcal{I} }\) is defined as the accuracy between the ground-truth binary contact vector and a binary contact vector inferred by thresholding a vector of distances \(\boldsymbol{\hat{d}}\) between predicted human and object meshes.

References↩︎