1 Introduction↩︎

Automatically assessing how well a person performs a skilled activity could enable personalized, real-time coaching through Augmented Reality (AR) glasses or fixed camera setups in domains such as sports, medical training, and music [1], [2]. This requires detecting when skilled actions occur and determining how well they are performed. Temporal action detection (TAD) methods localize when actions occur in untrimmed recordings by predicting temporal segments with start and end times [3]–[7], but do not assess execution quality. Action quality assessment (AQA) methods score how well a person performs an action from clips pre-cut around a single execution [2], [8], [9]. However, neither of these tasks jointly detects and grades individual skill moments at the timestamp level in continuous recordings—a prerequisite for moment-by-moment feedback.

Ego-Exo4D [10] introduced this task in the proficiency demonstration benchmark: detecting timestamps of skilled actions in ego-exo videos and additionally classifying each timestamp as good execution or needing improvement (see 1). Only a minimal baseline exists for this task so far [10]. We adapt seven state-of-the-art TAD architectures to establish a comprehensive baseline. To separately evaluate detection and grading, we extend the evaluation protocol with class-agnostic mAP and balanced accuracy. This reveals that current TAD architectures achieve low detection and grading quality near the random baseline.

In this paper, we introduce SkillSpotter, a pose-aware multi-view framework comprising three components that each address specific challenges of timestamp-level skill detection and grading. Adaptive temporal suppression addresses the large variation in action density across activities—from sub-second spacing in basketball to several seconds in music—by learning activity-specific suppression radii instead of using a fixed threshold. Gated 3D body pose fusion adds body kinematics to better assess execution quality. Bidirectional cross-view attention combines ego and exo views to prevent grading accuracy loss observed when naively concatenating these views. We show that our modules transfer to other TAD architectures, and that SkillSpotter generalizes beyond Ego-Exo4D to HoloAssist.

We summarize our key contributions as follows:

1. SkillSpotter, a pose-aware multi-view framework for timestamp-level skill detection and grading on the Ego-Exo4D proficiency benchmark, introducing three modules: adaptive temporal suppression, gated 3D body pose fusion, and bidirectional cross-view attention. Our method improves class-specific mAP from 12.40 to 21.82 (+76%) and balanced accuracy from 55.99% to 60.40% over the strongest baseline, with consistent gains across all view settings and activities, generalizing beyond Ego-Exo4D to HoloAssist.

2. A comprehensive evaluation of seven state-of-the-art TAD architectures on this task with an extended protocol that separately measures detection and quality grading, revealing that current architectures achieve low detection and near-random grading quality. Applying our proposed modules to these TAD architectures substantially improves their performance as well.

3. The first empirical inter-annotator agreement ceiling for the Ego-Exo4D proficiency demonstration benchmark (BA: 64.6, Cohen’s \(\kappa\)=0.29), based on the benchmark’s multi-annotator data aligned with the evaluation protocol. The ceiling provides a calibration point for future work and shows that SkillSpotter reaches 94% of it, suggesting that further BA gains are increasingly limited by label subjectivity rather than model capacity.

2 Related Work↩︎

2.0.1 Skill Assessment in Ego-Exo Video.↩︎

The Ego-Exo4D dataset [10] provides synchronized egocentric and exocentric video of skilled activities across eight scenarios with two proficiency benchmarks: demonstrator proficiency (classifying a person’s skill level) and demonstration proficiency (timestamp-level detection and grading). Recent works have focused on demonstrator proficiency estimation [11]–[13] and language feedback generation [14], [15]. Beyond visual features, egoPPG [16] uses heart rate as a complementary cue for proficiency estimation. The EgoExo-Fitness dataset [17] addresses a related problem in the fitness domain, providing segment-level proficiency annotations. Unlike previous work that classifies demonstrator proficiency or generates language feedback, SkillSpotter jointly localizes and grades individual skilled actions by combining adaptive temporal suppression, 3D body pose, and ego-exo cross-view attention.

2.0.2 Action Quality Assessment (AQA)↩︎

evaluates how well a person performs an action, typically by regressing a quality score from a pre-segmented clip [1], [2], [8], [9], [18], [19]. These methods assume known temporal boundaries, whereas Ego-Exo4D demonstration proficiency requires untrimmed timestamp localization and per-timestamp good-versus-tip classification.

2.0.3 Temporal Action Detection and Action Spotting.↩︎

Temporal action detection (TAD) localizes action segments in untrimmed video, evolving from two-stage proposal pipelines [20]–[23] to one-stage anchor-free architectures [3], [4], [24]–[26] and DETR-style set prediction [5] or hybrid Mamba–attention backbones [6], [27], unified under the OpenTAD toolbox [7]. These methods target segment localization with intersection-over-union (IoU)-based evaluation. Action spotting also predicts single timestamps with tolerance-based matching [28], [29]. Recent methods use dense per-token classification with displacement heads [30], [31]. The Ego-Exo4D demonstration proficiency benchmark shares this single-timestamp formulation but differs in three key aspects: it requires quality grading of each detected moment, operates on multi-view ego-exo input, and exhibits dense co-occurring events of different classes.

2.0.4 Suppression and Fusion for Dense Events.↩︎

Standard TAD pipelines apply Soft-NMS [32] to remove duplicate detections, with learned alternatives including density-aware thresholds [33], re-scoring networks [34], and DETR-style set prediction [35] that removes NMS entirely. None of these address timestamp-level, class-aware suppression for co-occurring feedback types. For pose fusion, two-stream RGB–pose architectures are effective for action recognition [36], with 3D pose recoverable from ego-view head tracking [37], [38] or multi-view triangulation [39], [40]; gated fusion [41] and feature-wise modulation [42] offer lightweight integration. Cross-attention across parallel view streams has been explored in representation learning [43], [44] but not for ego-exo skill assessment.

3 Method↩︎

3.1 Problem Formulation↩︎

Given an untrimmed egocentric and exocentric video of a person performing a skilled activity, such as basketball, cooking, or music, the goal is to detect individual moments of correct or incorrect execution and classify each accordingly. We formalize this following the demonstration proficiency estimation task in Ego-Exo4D [10], which jointly requires (i) temporally localizing feedback timestamps and (ii) classifying each as good execution or tip for improvement.

3.1.1 Input.↩︎

The input consists of synchronized egocentric and exocentric video features for a single task demonstration: \(\mathbf{F}_{\text{ego}} \in \mathbb{R}^{C \times T}\) and \(\{\mathbf{F}_{\text{exo}}^{m}\}_{m=1}^{M}\) with \(\mathbf{F}_{\text{exo}}^{m} \in \mathbb{R}^{C \times T}\), where \(C\) is the feature dimension and \(T\) the number of temporal tokens. Optionally, 3D body pose sequences \(\mathbf{P} \in \mathbb{R}^{D_p \times T}\), where \(D_p\) is the pose feature dimension, provide complementary skeletal information.

3.1.2 Output.↩︎

The model predicts a set of detections \(\hat{\mathcal{D}} = \{(\hat{t}_k, \hat{y}_k, \hat{s}_k)\}_{k=1}^{K}\), where \(\hat{t}_k \in \mathbb{R}^{+}\) is a predicted timestamp in seconds, \(\hat{y}_k \in \{\texttt{good}, \texttt{tip}\}\) is the quality, and \(\hat{s}_k \in [0, 1]\) is the confidence score. Ground-truth annotations \(\mathcal{G} = \{(t_j, y_j)\}_{j=1}^{J}\) consist of expert-annotated timestamps each labeled as a good execution or a tip for improvement. Multiple annotations can share the same timestamp.

3.1.3 Evaluation.↩︎

For each radius \(\delta \in \{0.25, 0.5, 1.0\}\) s, the evaluator sorts predictions by confidence and greedily matches each prediction to the closest unmatched ground-truth timestamp if \(|\hat{t}_k - t_j| \le \delta\). The original benchmark reports class-specific mAP, which requires both correct localization and correct label assignment. To decompose detection from classification, we extend the evaluation protocol with three complementary metrics: (i) class-agnostic mAP (mAP_A), which ignores the quality label and measures pure timestamp detection; (ii) class-specific mAP (mAP_S), equivalent to the original metric; and (iii) balanced accuracy (BA) and macro-F1 (F1), computed on matched prediction–ground-truth pairs, which partially decouple classification quality from localization but still depend on matching coverage.

3.2 Overall Architecture↩︎

SkillSpotter extracts Omnivore [45] video features from synchronized ego and exo videos, processes them with an ActionFormer [3] temporal backbone, and finally, predicts per-timestamp quality logits (see 2). We introduce three modules that each target a specific challenge of timestamp-level skill detection. Bidirectional cross-view attention (3.2.1) allows information exchange between ego and exo features. Gated pose fusion (3.2.2) incorporates 3D body kinematics as a complementary signal for assessing execution quality. Adaptive temporal suppression (3.2.3) replaces fixed-radius non-maximum suppression (NMS) with learned, activity-specific suppression radii to address the large variation in skilled action density across activities. Formally, the ActionFormer backbone processes the video features \(\mathbf{F} \in \mathbb{R}^{C \times T}\) into a multi-scale temporal pyramid of \(L{=}8\) levels \(\{\mathbf{f}_l \in \mathbb{R}^{D \times T_l}\}_{l=0}^{L-1}\) with feature dimension \(D{=}512\) and \(T_l = T / 2^l\). For single-view settings (Ego and Exos), cross-view attention is removed.

3.2.1 Bidirectional Cross-View Attention.↩︎

Naively concatenating ego and exo features degrades grading performance for ActionFormer (see 4). We introduce bidirectional cross-view attention to enable information exchange between views. The ego and exo feature streams are each processed through a shared-weight backbone into separate pyramids \(\{\mathbf{f}_{\text{ego},l}\}\) and \(\{\mathbf{f}_{\text{exo},l}\}\). At each pyramid level \(l\), both streams attend to each other via multi-head cross-attention (\(N_h{=}4\) heads) with separate query, key, and value projections per direction: \[\begin{align} \mathbf{f}_{\text{ego},l}' &= \mathbf{f}_{\text{ego},l} + \sigma(g_l^{\text{e}}) \cdot \text{MHA}(\mathbf{f}_{\text{ego},l},\, \mathbf{f}_{\text{exo},l}), \tag{1} \\ \mathbf{f}_{\text{exo},l}' &= \mathbf{f}_{\text{exo},l} + \sigma(g_l^{\text{x}}) \cdot \text{MHA}(\mathbf{f}_{\text{exo},l},\, \mathbf{f}_{\text{ego},l}), \tag{2} \end{align}\] where \(\text{MHA}(\mathbf{Q}, \mathbf{KV})\) denotes multi-head cross-attention with the first argument as queries and the second as keys and values. Cross-view attention operates over the full temporal extent, unlike the backbone’s windowed self-attention. The enhanced features are merged into a unified representation: \[\mathbf{f}_l = \mathbf{f}_{\text{ego},l}' + \sigma(g_l^{f}) \cdot \text{Proj}([\mathbf{f}_{\text{ego},l}';\, \mathbf{f}_{\text{exo},l}']), \label{eq:cross95fusion}\tag{3}\] where the fusion gate \(g_l^{f}\) is initialized to \(-1.0\) and the residual path is from Ego, making the default model ego-centric. Pose fusion (4 ) is applied after cross-view fusion to integrate skeletal information into the unified representation.

3.2.2 Gated Pose Fusion.↩︎

Skill assessment is inherently tied to body mechanics. We incorporate 3D body pose as a complementary signal via gated late fusion at each pyramid level. Our default pose input \(\mathbf{P} \in \mathbb{R}^{D_p \times T}\) with \(D_p{=}118\) consists of 51 raw 3D keypoint coordinates (17 COCO joints \(\times\) 3) and 67 kinematic features (joint angles, pairwise distances, and inter-frame velocities). We train with the manually annotated GT Ego-Exo4D body pose [10] and evaluate with the pose that we predict ourselves, ensuring no ground-truth leakage at test time. We report the GT-vs-predicted pose comparison in 12 (suppl.). For the egocentric view, we predict body pose from the Aria camera trajectory using the official Ego-Exo4D baseline [10], [37], [38], [46] (PA-MPJPE: 10.70 cm). For exocentric views, we detect the actor, estimate 2D keypoints with ViTPose [39], and recover 3D pose via calibrated multi-view triangulation with RANSAC-DLT (PA-MPJPE: 17.98 cm). We encode pose \(\mathbf{P}\) with a convolutional backbone to 256 channels and map it to a temporal feature pyramid with output dimension \(D{=}512\) per level. At each level \(l\), we fuse video and pose features via a gated residual: \[\mathbf{f}_l' = \mathbf{f}_l + \sigma(g_l^{p}) \cdot \text{Proj}([\mathbf{f}_l;\, \mathbf{p}_l]) \label{eq:pose95fusion}\tag{4}\] where \(\mathrm{Proj}:\mathbb{R}^{2D}\!\to\!\mathbb{R}^{D}\) is a \(1{\times}1\) masked convolution followed by group normalization, ReLU, and dropout. \(g_l^p\) is a learnable scalar gate initialized to \(-2.0\).

3.2.3 Adaptive Temporal Suppression.↩︎

Standard post-processing in TAD applies fixed-radius NMS to remove duplicate detections. However, in skill assessment, events of different classes can co-occur at near or identical timestamps, and event density varies dramatically across scenarios: in Basketball, over 80% of events have a neighbor within 0.5 s, whereas in Music fewer than 20% do (see 3). A single suppression threshold either erases valid co-occurring detections in dense scenarios or under-suppresses in sparse ones.

We replace fixed NMS with a learnable suppression mechanism that predicts per-detection suppression radii conditioned on the detection’s features and scenario. Unlike Adaptive NMS [33], our mechanism operates in the temporal domain with continuous exponential decay and class-aware constraints that allow co-occurring event types to coexist. For each candidate detection \(\mathbf{h}_k \in \mathbb{R}^{D}\) from the classification head, a two-layer MLP predicts a suppression radius: \[r_k = \text{Softplus}(\text{MLP}(\mathbf{h}_k)) + \mathbf{e}_{s}, \label{eq:radius}\tag{5}\] where MLP : \(\mathbb{R}^D \to \mathbb{R}^{128} \to \mathbb{R}^1\) with ReLU activation, and \(\mathbf{e}_s \in \mathbb{R}\) is a per-scenario offset from a learnable embedding table (\(S{=}8\) scenarios, indexed by the scenario label from the dataset metadata).

Given \(K\) candidate detections sorted by confidence score \(s_k\), each higher-scoring candidate \(i\) suppresses lower-scoring candidate \(j\) via exponential decay: \[w_{ij} = \exp\!\Big(\!-\frac{|t_i - t_j|}{r_i}\Big) \cdot \mathbb{1}[s_i > s_j] \cdot \mathbb{1}[y_i = y_j], \label{eq:suppression95weight}\tag{6}\] where the last indicator enforces class-aware suppression: detections of different quality classes are not directly suppressed by this class-aware term, preserving co-occurring good/tip feedback at the same timestamp. The adjusted score is \(\tilde{s}_j = s_j \cdot \exp\!\big(\!-\sum_i w_{ij}\,/\,\tau\big)\), where \(\tau\) is a learnable temperature initialized to 1.0.

To train the suppression module end-to-end, we introduce an auxiliary loss \(\mathcal{L}_\text{sup}\) as the suppression module only operates at inference time and receives no gradient from the detection objective. For each training video, we apply the suppressor to the classification scores to obtain adjusted scores \(\tilde{s}_k\). Gaussian targets centered on ground-truth timestamps define the supervision signal: \(\hat{y}_k = \max_j \exp\!\big(\!-(t_k - t_j^*)^2 / 2\sigma_s^2\big)\) with \(\sigma_s{=}1.0\) grid units. The auxiliary loss is the binary cross-entropy between the adjusted scores and the Gaussian targets: \(\mathcal{L}_{\text{sup}} = \text{BCE}(\text{logit}(\tilde{s}_k),\, \hat{y}_k)\). The indicator \([\hat{s}_i > \hat{s}_j]\) and candidate selection act as fixed masks that determine suppression topology. Gradients flow through the smooth exponential decay \(\exp(-d_{ij} / r_i)\) to the radius predictor and temperature.

3.2.4 Training Objective↩︎

For classification, we use sigmoid focal loss [3]. For temporal score alignment, we adopt the cumulative loss formulation of Kwak et al. [47] used by the Ego-Exo4D baseline [10]: \[\mathcal{L}_{\text{reg}} = \frac{1}{N_{\text{pos}}} \sum_{c=1}^{N_c} \sum_{t} \Big(\mathrm{CDF}_{\text{pred}}^{c}(t) - \mathrm{CDF}_{\text{gt}}^{c}(t)\Big)^2, \label{eq:reg95loss}\tag{7}\] where \(N_{\text{pos}}\) is the loss normalizer, \(N_c\) is the number of classes, and \(\mathrm{CDF}_{\text{pred}}^{c}(t)\) and \(\mathrm{CDF}_{\text{gt}}^{c}(t)\) denote cumulative class-wise temporal score distributions. The final loss combines all terms with \(\lambda_{\text{reg}} = \lambda_{\text{sup}} = 1.0\): \[\mathcal{L} = \mathcal{L}_{\text{cls}} + \lambda_{\text{reg}}\,\mathcal{L}_{\text{reg}} + \lambda_{\text{sup}}\,\mathcal{L}_{\text{sup}}. \label{eq:total95loss}\tag{8}\]

3.3 Training↩︎

We train all models for 15 epochs (5 warmup epochs) with AdamW [48] (learning rate of \(1 \times 10^{-3}\), weight decay of \(0.05\)) and a batch size of 8. Training takes less than 15 minutes on one NVIDIA H200 GPU for all view settings. Our full Ego+Exos model has 67.37M parameters and runs at 19.70 FPS on one H200 GPU. See 6 (suppl.) for a full performance analysis.

4 Experiments↩︎

4.1 Dataset and Evaluation Protocol↩︎

Following egoPPG [16], we use 90% of the official training set of the proficiency demonstration task for training, 10% for validation, and the official validation set for testing. As input, we use frozen Omnivore features (\(C{=}1536\)) to ensure fair comparison across all architectures and avoid overfitting the video backbone on the moderately sized training set (556 takes). Takes without the required pre-extracted Omnivore features are excluded (see 17 16 for statistics, suppl.). The scenario sizes are highly imbalanced with 3–46 takes but classes near-balanced (49.5% good).

We evaluate three view settings: Ego (egocentric only), Exos (four exocentric views), and Ego+Exos (all five synchronized). Following Ego-Exo4D [10], we concatenate all feature streams for the Exos and Ego+Exos settings. For our cross-view attention model (see 3.2.1), we keep ego and mean-pooled exo features as separate streams. Single-view results without concatenation are in 8 (suppl.). All metrics are averaged over timestamp matching radii \(\{0.25, 0.5, 1.0\}\) s (see 3.1.3). For fair comparison, all methods share the same Omnivore features and training protocol, and each baseline uses its default Soft NMS configuration as implemented in OpenTAD [7].

4.2 Evaluation Results↩︎

In 1, we compare SkillSpotter to seven state-of-the-art TAD architectures [3]–[6], [25]–[27]. All re-implemented baselines substantially outperform the original benchmark result [10] (mAP\(_\text{S}\): 3.27 vs.–12.40) but remain below 12.40 class-specific mAP, with detection limited to 8.65–17.11 mAP\(_\text{A}\) and quality classification within 6 percentage points of the random baseline (BA 49.51–55.99% vs.% random). The naive baselines follow the Ego-Exo4D protocol [10] (see 1). F1 tracks BA closely throughout all experiments; we focus on BA hereafter. To reduce the chance that weak baseline performance is caused by under-tuning, we sweep TadTR learning rates/query counts (see 7, suppl.) and compare single-view vs.concatenated inputs for four of the implemented baselines (see 8, suppl.). These sweeps do not close the gap to SkillSpotter. We additionally evaluate if AQA heads are better suited for this task than our classification head by substituting three AQA heads into our pipeline (see 11, suppl.). Our default head remains strongest in grading quality, while detection stays comparable across heads.

SkillSpotter improves class-specific mAP from 12.40 to 21.82 (+76%) and balanced accuracy from 55.99% to 60.40% on Ego (see 1). These gains generalize across all view settings: Ego (21.82 mAP\(_\text{S}\), 60.40 BA), Exos (21.12, 60.59), and Ego+Exos (21.34, 60.39), outperforming all baselines by a large margin. Notably, some baselines degrade under naive Ego+Exos concatenation (e.g., ActionFormer BA 55.99\(\to\)​50.34) compared to only using a single view as input. We analyze this instability in 4.3.

4.3 Ablation Study↩︎

4.3.1 Suppression Strategy.↩︎

Removing Soft NMS entirely (No NMS) from the ActionFormer baseline [3] increases mAP\(_\text{A}\) substantially (Ego: 16.85\(\to\)​22.50) but mAP\(_\text{S}\) only modestly (12.12\(\to\)​13.96), while BA decreases slightly (55.34\(\to\)​54.81). This means that the additional detections recovered without suppression are poorly classified, indicating a trade-off between detection and grading precision.

Our introduced temporal adaptive suppression outperforms both fixed strategies (No NMS and Soft NMS) by a large margin: mAP\(_\text{S}\) rises from 13.96 to 18.82, mAP\(_\text{A}\) from 22.50 to 25.87, and BA from 54.81 to 59.74—jointly improving detection and classification. Class-aware decay restricts suppression to detections of the same class (see 6 ), allowing temporally overlapping good and tip events to coexist. We sweep Soft NMS \(\sigma\) in 9 (suppl.), showing that no single fixed \(\sigma\) jointly optimizes detection and classification in our setting, and that the best fixed setting remains below adaptive suppression.

4 shows the ground-truth event density (left) and the median learned suppression radii (right) per scenario of our temporal suppression module. The median learned radius per scenario correlates with the ground-truth event spacing (Spearman \(\rho = 0.83\), \(p = 0.010\)): without any explicit density supervision, the model learns smaller radii for densely labeled scenarios such as Basketball (median nearest-neighbor gap 0.05 s, radius 0.23 grid units) and larger radii for sparser ones such as Music (7.05 s, 0.32 grid units). Per-scenario radius distributions are provided in 13 (suppl.).

4.3.2 Pose Fusion.↩︎

Adding 3D body pose further improves both detection and classification. On Ego, mAP\(_\text{S}\) increases from 18.82 to 21.82 (+3.00), mAP\(_\text{A}\) from 25.87 to 27.89 (+2.02), and BA from 59.74 to 60.40. On Exos, the BA gain is even larger (58.29\(\to\)​60.59, +2.30), suggesting that pose is particularly complementary to exo-view appearance features. We also evaluate whether predicted pose compared to ground truth during inference affects performance and find only a marginal decrease when using our predicted pose from the ego and exo views (see 12, suppl.). Adding hand-pose (using Ego-Exo4D’s POTTER baseline) to our gated mechanism gives mAP\(_\text{S}\) \(21.82{\rightarrow}20.92\), BA \(60.40{\rightarrow}59.56\) on Ego. We therefore did not include hand pose further.

4.3.3 Cross-View Attention.↩︎

While adaptive suppression and pose fusion improve all metrics in the single-view settings, the Ego+Exos setting exhibits a classification collapse: BA drops from 51.79 to 45.77 with adaptive suppression and pose fusion.

Cross-view attention addresses this by maintaining view-specific feature streams with explicit bidirectional information exchange before gated fusion, rather than relying on the backbone to separate concatenated ego-exo representations. BA recovers to 60.39 (+14.62 over the concatenated), which is close to the Ego/Exos single-stream results (60.40/60.59 in 2) and higher than the Ego+Exos per-camera single-view variant (55.93 in 7, suppl.). We observe a slight mAP\(_\text{S}\) decrease (21.99\(\to\)​21.34) relative to the pose-only concatenated row, indicating a minor detection-grading tradeoff. These results are consistent with the benefit of view-specific processing with explicit bidirectional exchange before fusion.

4.3.4 Cross-Backbone Evaluation.↩︎

We also apply adaptive suppression and pose fusion to all implement backbones, except TadTR [5] (as it lacks a suppression stage), to test whether our modules transfer beyond ActionFormer (see 3 and 10, suppl.). Adaptive suppression and pose fusion improve detection (mAP\(_\text{S}\), mAP\(_\text{A}\)) across all backbones except VideoMambaSuite, for which only Adaptive Suppression improves performance. Pose fusion’s benefit to grading (BA, F1) is less consistent: it improves BA across all view settings only for TriDet and CausalTAD, while the other three each show a BA regression in at least one view setting.

4.4 Analysis↩︎

4.4.1 Per-Scenario Breakdown.↩︎

¿tbl:tab:analysis? (top) reports Ego results per scenario for our method and ActionFormer. We compare against ActionFormer because it provides the strongest baseline trade-off in the Ego setting. Per-scenario trends are consistent across view settings. Our method improves mAP\(_\text{S}\) and mAP\(_\text{A}\) across all eight scenarios, with the largest mAP\(_\text{S}\) gains on Basketball (+17.89) and Rock Climbing (+10.02). BA improves by up to +9.68 (Dance), with the exception of Cooking, Health, and Soccer where ActionFormer retains a higher BA. The largest gains appear in Basketball, Rock Climbing, and Dance, which are also among the denser scenarios (see 4). We also observe that scenarios with coarse full-body motion benefit more than scenarios dominated by subtle manipulation cues. This pattern is consistent with adaptive suppression helping dense-event regimes and with the current feature set (appearance + body pose) capturing gross kinematics better than fine hand-object interactions. Cooking and Music remain challenging for detection (mAP\(_\text{S}\): 3.08 and 1.57 for our method), indicating that current appearance-plus-body-pose cues still miss part of the quality signal. In particular, fine-grained hand-object interactions likely require stronger hand-level or object-state representations than we currently use. Soccer (BA: 49.48) and Cooking (F1: 43.22) remain difficult grading regimes. We interpret Soccer cautiously because the evaluation split contains only three videos (see 17, suppl.). We additionally report detection recall (R), defined as the fraction of ground-truth events matched by any prediction within the matching radius. Our method consistently achieves higher recall than ActionFormer across all scenarios (e.g., 72.89% vs.% on Basketball), confirming that the BA and F1 improvements are not artifacts of selective matching but reflect genuinely better coverage of annotated events.

4.4.2 Per-Radius Breakdown.↩︎

¿tbl:tab:analysis? (bottom) reports results at individual matching radii for SkillSpotter and ActionFormer on Ego. SkillSpotter improves mAP\(_\text{S}\) over ActionFormer at every radius, with the largest relative gain at the strictest threshold (13.49 vs. at 0.25 s). For both methods, BA decreases at looser radii while mAP\(_\text{S}\) increases: stricter matching retains temporally precise detections, while looser matching admits predictions farther from annotated timestamps. Detection recall (R) confirms that this BA drop is not an artifact of selective matching: at 0.25 s only 38.92% of GT events are matched, rising to 63.09% at 1.0 s, so the lower BA at loose radii reflects genuinely harder cases entering the evaluation pool. SkillSpotter consistently matches more GT events than ActionFormer at every radius (e.g., 38.92% vs.% at 0.25 s), indicating that its mAP and BA gains are not due to cherry-picking easy timestamps. The mAP\(_\text{A}\)–mAP\(_\text{S}\) gap widens at looser radii for both methods (e.g., Ours: 3.57 at 0.25 s vs. at 1.0 s).

4.4.3 Inter-Annotator Agreement.↩︎

Ego-Exo4D provides 2–5 expert annotators per take, giving a first empirical ceiling for grading quality on this benchmark. For each pair of experts annotating the same take, we match their timestamps using the same matching radii as our evaluation protocol (\(\delta \in \{0.25, 0.5, 1.0\}\) s). Only 13–27% of expert annotations have an inter-expert counterpart across these radii. On the co-located subset, averaged across radii, inter-expert balanced accuracy is 64.6 with a Cohen’s \(\kappa\) of 0.29 (fair agreement). Our method’s BA of 60.40 reaches 94% of this empirical ceiling, suggesting that further gains in balanced accuracy on this benchmark are increasingly limited by label subjectivity rather than model capacity, and that mAP\(_\text{S}\) remains the more discriminative metric for measuring future progress.

4.5 Generalization to HoloAssist↩︎

To test generalization beyond Ego-Exo4D, we evaluate SkillSpotter on HoloAssist [49], an egocentric mistake-detection benchmark. While HoloAssist’s official protocol performs classification on pre-segmented clips, SkillSpotter extends this protocol by jointly performing action detection and classification on untrimmed video. On HoloAssist’s official classification metric, SkillSpotter reaches F1=53.3, close to the SOTA (DR-MoE [50], F1=57.0; 4). We additionally extend HoloAssist to action localization, which has not been previously evaluated on this dataset: our modules more than double ActionFormer’s detection (mAP\(_\text{S}\): 3.33\(\to\)​7.24), with absolute performance consistent with the hand-manipulation scenarios (Cooking, Music) in Ego-Exo4D.

4.6 Discussion and Limitations↩︎

Performance varies substantially across scenarios: mAP\(_\text{S}\) ranges from 1.57 (Music) to 40.76 (Basketball), indicating that the current method is not uniformly robust across activity types. We interpret this variation together with strong scenario-size imbalance (3–46 videos and 191–2,409 annotations in our filtered evaluation split; see 17, suppl.). Accordingly, we treat Soccer (3 videos) as a low-sample regime with high metric variance and avoid strong scenario-specific conclusions, similar to the findings in egoPPG [16]. At the same time, persistent errors in high-volume scenarios such as Cooking and Music indicate that data scarcity alone does not explain the remaining failure cases. We observe larger gains in scenarios with coarse full-body motion (e.g., Basketball and Rock Climbing) than in scenarios dominated by subtle manipulation cues (e.g., Cooking). Cross-view attention recovers Ego+Exos BA to single-view levels (60.39) but does not surpass them, indicating that effective multi-view fusion for skill grading remains an open challenge [51]. However, resolving the classification collapse is a necessary prerequisite for future multi-view improvements. Our HoloAssist results confirm that SkillSpotter and its modules generalize beyond Ego-Exo4D, though broader cross-dataset evaluation remains an open direction. Furthermore, our inter-annotator agreement analysis yields an empirical BA ceiling of 64.6 for this benchmark, which our method’s 60.40 BA approaches (94%). This indicates that skill assessment is inherently subjective and that mAP\(_\text{S}\) remains the more discriminative metric for measuring future progress on EgoExo4D. Promising directions for future work include incorporating physiological signals from egocentric vision, such as heart rate [16], [52], heart-rate variability [53], [54], electrodermal activity [55], [56], emotional states [57], multi-view recordings of motions and activities [58], [59], and pose-object interaction features to address the remaining failure scenarios. End-to-end training from raw video could further jointly optimize feature extraction and temporal modeling.

5 Conclusion↩︎

We have introduced SkillSpotter, a pose-aware multi-view framework that jointly localizes and grades skilled actions in untrimmed ego-exo video through three components specifically designed for this task: adaptive temporal suppression, gated 3D body pose fusion, and bidirectional cross-view attention. On the Ego-Exo4D demonstration proficiency benchmark, SkillSpotter improves class-specific mAP from 12.40 to 21.82 and balanced accuracy from 55.99% to 60.40% compared to the best baseline, with consistent gains across all viewpoints. Our adaptive suppression and pose fusion modules transfer to other backbones with similarly substantial improvements and SkillSpotter itself generalizes beyond Ego-Exo4D to HoloAssist. The gains are thus not architecture- or dataset-specific.

Performance remains uneven across scenarios, with the strongest results in activities dominated by coarse full-body motion and weaker results in scenarios driven by subtle manipulation cues (Cooking and Music), indicating that the current appearance-plus-body-pose representation is insufficient for some tasks. We additionally establish the first empirical inter-annotator agreement ceiling for this benchmark (BA: 64.6, Cohen’s \(\kappa\)=0.29). SkillSpotter reaches 94% of this ceiling, which suggests that further BA gains are increasingly limited by label subjectivity rather than model capacity. Incorporating pose-object interaction features and extending the framework toward natural-language feedback are promising directions for future work.

6 Computational Cost↩︎

5 reports model size, FLOPs, throughput, and latency for each ablation configuration. Adaptive suppression adds negligible overhead (+0.07M parameters, \(<\)​1 GFLOPs). Pose fusion introduces a separate pose backbone, increasing parameters by \(\sim\)​13M and reducing FPS from 46.80 to 36.90 (Ego). Cross-view attention is the most expensive addition, bringing the full Ego+Exos model to 67.37M parameters and 19.70 FPS. FPS/latency reflect model inference only. They do not include video decoding, Omnivore feature extraction, and pose estimation.

7 TadTR Learning Rate and Query Count↩︎

In contrast to feature pyramid methods (ActionFormer, TriDet, CausalTAD, SkillSpotter) where every temporal position serves as a candidate detection, DETR-based detectors use a fixed set of learned queries, each predicting one event, imposing a hard upper bound on the number of detections. Ego-Exo4D takes contain 42.3 annotations on average (median: 38, max: 266). The default query count of TadTR [5] is 40, which covers only 54.1% of takes. 100 queries cover 96.1% and 200 queries 99.7%. 6 ablates both the query count (40, 100, 200) and the learning rate (\(10^{-3}\) as used by all anchor-free models in our benchmark, and \(10^{-4}\) following the standard DETR protocol).

The best detection result (\(\text{lr}{=}10^{-3}\), 200 queries, Ego mAP\(_\text{S}\): 7.79) remains far below ActionFormer (12.40) and our method (21.82), reported in 1. At \(\text{lr}{=}10^{-4}\), performance degrades across all query counts, with 200 queries collapsing entirely (Ego mAP\(_\text{A}\): 0.66).

8 Single-View vs.Concatenated Features↩︎

The Ego-Exo4D baseline implementation [10] concatenates all available camera features into a single representation for the Exo and Ego+Exos views. For evaluation completeness, 7 compares this concatenation strategy against single-view processing, where each camera view is treated as an independent sample. Concatenation improves detection mAP for most baselines but does not consistently improve classification. For our method, concatenation yields the highest Ego+Exos mAP\(_\text{S}\) (21.99) but BA drops (45.77), similar to TriDet [4] and ActionFormer [3]. Our introduced cross-view attention recovers BA to 60.39 with minimal detection loss.

9 Soft NMS Sigma Sweep↩︎

8 reports ActionFormer [3] with Soft NMS at varying \(\sigma\) values. Detection metrics (mAP\(_\text{A}\), mAP\(_\text{S}\)) decrease monotonically with increasing \(\sigma\). No fixed \(\sigma\) value can reach the performance when not using any NMS (No NMS). BA peaks at \(\sigma{=}0.5\) (Ego: 55.34) but the margin over No NMS (54.81) is small.

10 Cross-Backbone Evaluation↩︎

We apply adaptive temporal suppression and gated pose fusion to all evaluated architectures except TadTR [5] to evaluate if our modules are specific to the ActionFormer backbone (see 3). We exclude TadTR because its DETR-style set prediction removes the suppression stage entirely, making adaptive temporal suppression inapplicable. All models use the same pre-extracted Omnivore features, training protocol, and evaluation settings as in the main paper.

10.0.1 Adaptive Suppression Consistently Improves Detection.↩︎

Across all backbones and all view settings, adaptive suppression substantially improves mAP\(_\text{S}\) over both the Soft NMS and No NMS baselines. These gains are consistent with those observed on ActionFormer (see 2, main paper).

10.0.2 Pose Fusion’s Effect on Grading Is Backbone- and View-Dependent.↩︎

Unlike adaptive suppression, gated pose fusion does not uniformly improve balanced accuracy (BA) across all backbones. On CausalTAD and TriDet, pose fusion improves BA in every view setting (e.g., CausalTAD Ego: 54.01\(\to\)​59.34, +5.33; TriDet Exos: 57.54\(\to\)​60.56, +3.02). On the remaining three backbones, pose fusion slightly degrades single-view grading: VideoMambaSuite (Ego: 56.89\(\to\)​55.01; Exos: 57.59\(\to\)​54.46) and TemporalMaxer (Ego: 59.95\(\to\)​58.66; Exos: 61.27\(\to\)​59.66) both lose BA when pose is added on Ego and Exos, while gaining mAP\(_\text{S}\)/mAP\(_\text{A}\). In Ego+Exos settings, however, pose fusion consistently recovers or improves grading even where it hurts single-view BA (e.g., TemporalMaxer Ego+Exos: 46.86\(\to\)​59.42).

10.0.3 Cross-View Attention Is Architecture-Dependent.↩︎

Unlike ActionFormer, which suffers severe BA collapse under naive Ego+Exos concatenation (see 2, main paper), all of the evaluated backbones achieve reasonable Ego+Exos BA with adaptive suppression and pose fusion alone. Adding cross-view attention does not improve and can degrade performance in these cases. We attribute this to the fact that these architectures do not exhibit the classification collapse under multi-view concatenation that we observe with ActionFormer (see 2, Ego+Exos BA: 45.77 vs. and 60.01). Since cross-view attention specifically addresses this collapse, we apply it only to ActionFormer, where the failure mode is present.

11 AQA Drop-In Classification Heads↩︎

Action quality assessment (AQA) methods regress a scalar quality score from pre-segmented clips but provide no localization. To test whether AQA heads perform better on skill action grading than our classification head, we replace SkillSpotter’s classification head with three AQA heads, adapting only the components required by our binary good/tip classification task: TechCoach [60], GDLT [61], and USDL [19]. Our default head wins on grading quality (BA, F1) in all three view settings by a wide margin, while detection quality is slightly better using USDL in most settings. TechCoach’s performance on grading is closest to ours on Ego (BA: 58.87), but is unstable across views, collapsing on Exos (BA: 43.11).

12 Predicted vs.Ground Truth Pose↩︎

15 compares three pose configurations: training and testing on predicted pose, training on ground truth with predicted pose at test time (our default), and training and testing on ground truth pose (oracle). Training on ground truth pose outperforms training on predicted pose, even when both test on predicted input (e.g., Ego mAP\(_\text{S}\): 21.82 vs.), indicating that the model learns cleaner biomechanical representations from ground truth annotations. The gap between our default setting and the oracle is small across all configurations (e.g., Ego BA: 60.40 vs., \(\Delta{=}0.99\)), confirming that predicted pose quality is sufficient for effective skill grading during inference.

13 Per-Scenario Radius Distributions↩︎

5 shows the distribution of learned suppression radii across all eight scenarios. The suppression head learns scenario-specific radii without explicit supervision on the radius values—only the auxiliary suppression loss (3.2.3) provides a training signal. Scenarios with sparse, well-separated events (Music, Health) produce larger median radii (0.32 and 0.27 grid units, respectively), allowing broader suppression of isolated false positives. Conversely, dense-event scenarios (Rock Climbing, Basketball) yield smaller median radii (0.21 and 0.23) but exhibit wider spread, reflecting the need for context-dependent radii: detections near temporal clusters require tight suppression to preserve neighbors, while isolated detections permit larger radii. This adaptive behavior explains the failure mode of fixed-radius NMS: a single radius cannot simultaneously avoid over-suppression in dense scenarios and under-suppression in sparse ones (see 3).

14 Dataset Statistics↩︎

We report annotation statistics at two levels: the official Ego-Exo4D proficiency annotation files and the filtered evaluation database that our models are actually trained and evaluated on.

14.0.1 Official Annotations.↩︎

16 shows the per-scenario annotation counts directly from the official annotation files released by the Ego-Exo4D benchmark. The training split contains 556 takes with 11,942 annotations, and the test split contains 177 takes with 3,707 annotations. Good and bad labels are near-perfectly balanced across all scenarios (approximately 50/50).

14.0.2 Evaluation Database.↩︎

Our preprocessing pipeline converts the official annotations into a unified database while applying several necessary filters: (1) takes whose take_uid is absent from the Ego-Exo4D metadata are dropped; (2) takes with anomalous frame rates (outside \([10, 100]\) fps) are excluded; and (3) views for which the pre-extracted Omnivore feature files are missing are skipped. These filters reduce the training set from 556 to 492 takes and the test set from 177 to 176 takes (17). The small reduction confirms that the vast majority of officially annotated takes have complete metadata, valid frame rates, and available features. The dropped takes are predominantly missing pre-extracted Omnivore features for one or more views.

4pt

4pt

References↩︎