1 Introduction↩︎

Understanding human activities from video is a cornerstone of computer vision, with applications ranging from robotics and human-computer interaction to video content analysis [1]–[3]. While significant progress has been made in action recognition and temporal action segmentation [4]–[7], the vast majority of methods model human activity as a single stream of actions. This paradigm falls short in capturing the intricate dynamics of bimanual activities, which are ubiquitous in real-world tasks such as assembly, cooking, and surgery [8]–[10].

Dense, frame-wise segmentation of dual-hand actions presents a set of unique and critical challenges that are not addressed by conventional methods: (1) Complex Inter-Hand Dependencies: The two hands exhibit complex temporal relationships, switching between coordinated, cooperative, and independent action patterns; (2) Visual Asymmetry: The same action can manifest with distinct appearance and motion patterns for the left versus right hand; (3) Representation Conflict: In a joint model, the dominant hand may dominate gradient updates, leading to suboptimal feature learning for the non-dominant hand; (4) Semantic Ambiguity: Relying solely on visual features can fail to capture fine-grained semantic distinctions between similar actions (e.g., “screw nut onto bolt" vs. ”screw nut onto shaft").

To tackle these challenges, we propose Polyphony, a three-stage method for dual-hand action segmentation. Inspired by polyphonic music, where independent melodies harmonize simultaneously, Polyphony models dual-hand actions as interdependent streams requiring both individual understanding and collective coordination. To address representation conflict and visual asymmetry, we introduce an Alternating Dual-Hand Vision Transformer (ADH-ViT) that alternates training between left- and right-hand mini-batches, preventing gradient dominance while maintaining a shared encoder for efficient learning. To combat semantic ambiguity, we develop Semantic Feature Conditioning that enriches visual features by aligning them with structured, compositional language descriptions (e.g., verb-object-tool decomposition). To model complex inter-hand dependencies, we design a Diffusion-Based Segmentation model with cross-hand feature fusion for coherent dual-hand predictions and adaptive loss weighting for balanced training. The shared encoder design reflects the holistic perception of bimanual activities by humans [11], while diffusion-based iterative refinement mirrors humans’ progressive action understanding through repeated predictions and corrections [12]. Polyphony’s modular and versatile design (i) permits independent component improvement and (ii) allows ADH-ViT to operate as a standalone action-recognition model for broader downstream tasks.

Extensive experiments on three challenging benchmarks validate our approach. On the dual-hand datasets HA-ViD and ATTACH, our method sets a new state-of-the-art, significantly outperforming previous work. Our model additionally outperforms others on the Breakfast dataset for single-stream action segmentation, demonstrating generalizability beyond the bimanual domain. Comprehensive ablation studies confirm the efficacy of each core component, while qualitative analyses highlight Polyphony’s capacity to address the core challenges of complex inter-hand dependencies, visual asymmetry, and semantic ambiguity in dual-hand action segmentation.

2 Related Work↩︎

Temporal Action Segmentation. Temporal action segmentation (TAS) aims to predict frame-wise action labels in untrimmed videos, a fundamental task for understanding procedural activities. Early approaches established Temporal Convolutional Network (TCN) as the foundation for modeling long-range temporal dependencies [13]. Building on this foundation, subsequent work introduced multi-stage refinement strategies [5], boundary-aware losses [14], and graph-based modeling [15] to address persistent over-segmentation errors. The field evolved further with transformer-based architectures [7], [16], which leverage self-attention mechanisms to capture long-range dependencies more effectively. Most recently, diffusion models have emerged as a powerful paradigm, formulating action segmentation as conditional generation through iterative denoising processes [6]. However, existing TAS methods predominantly focus on single-stream action sequences and do not address the unique challenges of dual-hand scenarios, where two interdependent action streams must be modeled simultaneously while attending to their complex coordination patterns. Zheng et al. proposed two dual-hand action segmentation methods, DuHa [17] and DuCAS [18], specifically for bimanual assembly scenarios, but both methods require ground-truth object bounding boxes as input, limiting their general applicability. In this paper, we present a diffusion-based dual-hand action segmentation model that simultaneously predicts actions for both hands without requiring object annotations.

Vision-Language Alignment and Semantic Conditioning. Integrating semantic knowledge with visual understanding has shown significant benefits across various vision tasks [19]. Vision-language models such as CLIP [20] and ALIGN [21] align visual and textual embeddings through contrastive learning, enabling zero-shot transfer. These models have been successfully adapted to video understanding through temporal attention [22] and video-specific prompting [23], achieving strong performance on action recognition. In temporal action segmentation, FACT [7] leverages video transcripts, parsed from frame-wise action labels, to construct action-level tokens that can enhance frame-wise action predictions through cross-attention. However, relying on simple action labels limits the semantic richness needed to disambiguate actions with fine-grained semantic distinctions. Compositional action understanding [24] addresses this by explicitly modeling verb-object structure, with recent approaches exploiting knowledge bases [25], semantic disentanglement [26], and grammar-based decomposition [27] for distinguishing semantically nuanced actions. We extend this paradigm by incorporating structured compositional descriptions into our dual-hand segmentation framework, decomposing frame-wise action labels into verb, object, and tool components [28]. A TCN is applied to align visual features with these language-model–encoded descriptions, enabling semantic conditioning that addresses the semantic ambiguity.

3 Methodology↩︎

3.1 Overview and Problem Formulation↩︎

Given an untrimmed video, our goal is to perform dense frame-level action segmentation for both the left hand (LH) and right hand (RH) simultaneously through one model. Let \(\mathcal{V} = \{v_\textit{t}\}_{t=1}^T\) denote a video sequence of \(T\) frames. We aim to predict frame-wise action label sequences \(Y^h = \{y_t^h\}_\textit{t=1}^T\) , \(h\in\{LH,RH\}\), where \(y_t \in \{1, 2, ..., C\}\) represents one of \(C\) action classes. To address the dual-hand action segmentation challenges, we propose Polyphony (Figure 1), which has three stages: (1) Dual-Hand Feature Extraction, (2) Semantic Feature Conditioning, and (3) Diffusion-Based Dual-Hand Action Segmentation.

3.2 Stage 1: Dual-Hand Feature Extraction↩︎

We propose ADH-ViT to extract hand-specific features for the subsequent action segmentation stage.

3.2.1 Data Sampling↩︎

To adapt untrimmed videos for clip-based ADH-ViT training, complementary sampling strategies are applied independently to each hand.

Segment-Based Sampling. From frame-wise annotations \(\{y_t^h\}_{t=1}^T\), we extract coherent action segments. Formally, for action class \(c_i\), the set of segments is defined as: \[S_{seg}^h = \left\{(t_{\text{start}}, t_{\text{end}}, c_i) \mid y_t^h = c_i \;\forall t \in [t_{\text{start}}, t_{\text{end}}]\right\}\] Each segment \((t_{\text{start}}, t_{\text{end}}, c_i)\) yields a training clip \(x_{seg}\) labeled with action class \(c_i\).

Random Clip Sampling. To enhance temporal diversity and augment training data, we randomly sample \(n_{clip}\) clips of \(l_{clip}\) consecutive frames throughout the video. The label for each random clip \(x_{clip}\) is determined by the action class of its middle frame.

The complete samples \(\mathcal{X}^h = \{x_{seg}^h\} \cup \{x_{clip}^h\}\). The trimmed video clip dataset for each hand is obtained, \(\mathcal{D}_{trim}^h = \{(x_i^h,y_i^h)\}_{i=1}^{N_h}\), where \(N_h\) denotes the total number of video clips for hand \(h\).

3.2.2 Alternating Dual-Hand Vision Transformer↩︎

The architecture of ADH-ViT consists of:

Shared Spatio-Temporal Encoder. A ViT [29] backbone \(\mathcal{E}_\phi\) processes video clips via tubelet embeddings. Given \(x\in\mathbb{R}^{3\times L \times H \times W} \;(x \in \mathcal{D}_{trim})\) , it is partitioned into non-overlapping tubelet patches of size \(l \times p \times p\). \[z^h = \mathrm{Tubelet}(x^h; l, p) + \mathbf{E}_{\text{pos}} \in \mathbb{R}^{N \times D}\] where \(N=\frac{L}{l} \cdot \frac{H}{p} \cdot \frac{W}{p}\) is the number of patches, \(D\) is the embedding dimension, and \(\mathbf{E}_{\text{pos}}\in \mathbb{R}^{N \times D}\) are sinusoidal positional embeddings. The \(\mathrm{Tubelet}\) embedding is implemented as a 3D convolution. Then, the shared spatio-temporal feature is encoded as: \[e^h = \mathcal{E}_\phi(z^h) \in \mathbb{R}^D\] Dual Classification Heads. Two independent linear classifiers map shared spatio-temporal features to hand-specific predictions: \[\hat{y}^{h} = \text{softmax}(\mathbf{W}_{h} e^{h} + b_{h}) \in \mathbb{R}^{C}, \quad h \in \{LH, RH\}\] where \(\boldsymbol{W}_{h} \in \mathbb{R}^{C \times D}\) and \(b_{h} \in \mathbb{R}^{C}\) are learnable parameters.

3.2.3 Alternating Training Strategy↩︎

To ensure balanced multi-task optimization, we alternate between LH and RH mini-batches every \(\Delta\) steps. Let \(\tau(j)\in \{LH,RH\}\) denote the active task at training step \(j\): \[\tau(j) = \begin{cases} \text{LH} & \text{if } \lfloor j/\Delta \rfloor \mod 2 = 0 \\ \text{RH} & \text{otherwise} \end{cases}\] At each step, we sample a mini-batch exclusively from the dataset corresponding to \(\tau(j)\): \[\mathcal{B}^{(j)} = \begin{cases} \text{sample}(\mathcal{D}_{trim}^{LH},B) & \text{if } \tau(j)=LH \\ \text{sample}(\mathcal{D}_{trim}^{RH},B) & \text{if } \tau(j)=RH \end{cases}\] where \(B\) is the batch size. The loss at step \(k\) is computed using only the active head: \[\mathcal{L}^{(j)} = \frac{1}{B} \sum_{i=1}^{B} \mathcal{L}_{CE}(\hat{y}_{i}^{\tau(j)}, y_{i}^{\tau(j)})\] where \(\mathcal{L}_{CE}\) is cross-entropy loss. The gradient update modifies both the shared backbone and the active head. The advantage of alternating training over naive joint training is that it prevents tasks by one hand from dominating the gradients updates.

3.2.4 Dense Frame-Wise Feature Extraction↩︎

To perform action segmentation on untrimmed videos, we extract dense frame-wise features using the trained ADH-ViT. During feature extraction, we apply a \(l_{clip}\)-long sliding window with symmetric padding over the entire video \(\mathcal{V} = \{v_\textit{t}\}_{t=1}^T\) to get \(T\) video clips. Each clip \(x_t=\{v_i\}_{i=t-l_{clip}/{2}+1}^{t+l_{clip}/{2}}\) is processed through the trained encoder to extract both shared and hand-specific features: \[\begin{align} e_t^{\text{shared}} = \mathcal{E}_{\phi}(\mathrm{Tubelet}(x_t; l, p) + \mathbf{E}_{\text{pos}}) \end{align}\] \[\begin{align} e_t^{h, \text{class}} = \mathbf{W}_h e_t^{\text{shared}}+b_{h} \end{align}\] where the \(e_t^{\text{shared}}\) and \(e_t^{h, \text{class}}\) are the shared backbone feature and hand-specific classification logit for video clip \(x_t\).

3.3 Stage 2: Semantic Feature Conditioning↩︎

Visual features alone may conflate semantically distinct but visually similar actions. To enhance semantic discrimination, a TCN is applied to align visual features with semantic embeddings.

3.3.1 Structured Action Description↩︎

We represent each action class \(c\) as a structured, compositional semantic description (parsed from the original label), following the methodology of HR-SAT [28]: \[\begin{align} D_c = \text{``Action verb is } av_c; \text{manipulated object is } mo_c; \\ \text{target object is } to_c; \text{tool is } tl_c\text{.''} \end{align}\] For example, “screw nut onto bolt" becomes: “Action verb is screw; manipulated object is nut; target object is bolt; tool is null". This structured format provides explicit semantic attributes that distinguish similar actions.

Each description \(D_c\) is encoded using a pre-trained language model: \[e_c = \text{LM}(D_c) \in \mathbb{R}^{D_{sem}}\] where \(D_{sem}\) is the semantic embedding dimension. These embeddings serve as supervision signals for semantic conditioning. These descriptions are used only for training, not inference, preventing label semantics leakage.

3.3.2 TCN for Semantic Alignment↩︎

We employ a multi-layer TCN to model temporal context and align visual features with semantic embeddings. The TCN consists of \(M\) residual blocks with exponentially increasing dilation: \[h_m = \text{TCNBlock}(h_{m-1}; d_m = 2^m), m=1,\dots, M\] where \(h_0\) is shared backbone feature \(e_t^{\text{shared}}\). The TCN output is projected into the semantic space: \[\begin{align} e_t^{h,\text{sem}} &= \text{LN}(\mathbf{W}_{sem} h_M + b_{sem}) \in \mathbb{R}^{D_{\text{sem}}} \end{align}\] where \(\mathbf{W}_{sem}\) and \(b_{sem}\) are learnable projection matrices.

3.3.3 Adaptive Alignment Loss↩︎

We train the TCN using an adaptive multi-objective loss that combines cosine similarity and mean squared error: \[\mathcal{L}_{\text{align}} = \alpha \mathcal{L}_{\text{cosine}} + (1 - \alpha) \mathcal{L}_{\text{MSE}}\] where: \[\mathcal{L}_{\text{cosine}} = 1 - \frac{e_t^{h, \text{sem}} \cdot e_{y_t}}{\|e_t^{h, \text{sem}}\|_2 \|e_{y_t}\|_2}\] \[\mathcal{L}_{\text{MSE}} = \|e_t^{h, \text{sem}} - e_{y_t}\|_2^2\] The weighting factor \(\alpha\) balances directional alignment (cosine loss) with magnitude matching (MSE loss).

3.3.4 Motion-Action-Semantic Feature Concatenation↩︎

To provide comprehensive representations for fine-grained action segmentation, we construct a Motion-Action-Semantic (MAS) feature by concatenating three complementary feature types for each hand: \[E_t^h=[e_t^{shared}, e_t^{h,class}, e_t^{h, sem}]\in \mathbb{R}^{D+C+D_{sem}}\] where \(e_t^{\text{shared}} \in \mathbb{R}^D\) captures low-level spatio-temporal motion patterns through the shared ViT backbone, \(e_t^{h,\text{class}} \in \mathbb{R}^C\) encodes hand-specific classification logits that provide task-aware discriminative signals, and \(e_t^{h,\text{sem}} \in \mathbb{R}^{D_{sem}}\) embeds fine-grained semantics into the visual features.

3.4 Stage 3: Diffusion-Based Dual-Hand Action Segmentation↩︎

3.4.1 Architecture↩︎

Shared Encoder for Temporal Modeling. A shared encoder \(\mathcal{E}_{seg}\) is employed to process MAS features. \(\mathcal{E}_{seg}\) consists of \(L_{seg}\) mixed convolution-attention layers: \[Z^h, H^h=\mathcal{E}_{seg}(E^h)\] where \(Z^h \in \mathbb{R}^{C\times T}\) are initial action logits and \(H^h\in \mathbb{R}^{D'\times T}\) are hierarchical backbone features extracted from intermediate encoder layers.

Cross-Hand Feature Fusion. To address the challenge of complex inter-hand dependencies, we introduce a feature fusion module that enables cross-hand information exchange after hand-specific encoding: \[H^{LH} = \mathcal{F}^{LH}([H^{LH}; H^{RH}]) + H^{LH}\] \[H^{RH} = \mathcal{F}^{RH}([H^{LH}; H^{RH}]) + H^{RH}\] where \([\cdot; \cdot]\) denotes channel-wise concatenation at each time step and \(\mathcal{F}^h\) are hand-specific fusion networks implemented as: \[\mathcal{F}^h(H) = \text{Conv}_{1\times1}(\text{ReLU}(\text{Conv}_{1\times1}(H)))\] Each fusion network consists of two 1×1 convolutions with ReLU activation, projecting the concatenated \(2D'\)-dimensional features back to \(D'\) dimensions. The residual connection preserves hand-specific information while allowing the model to selectively incorporate cross-hand context. Notably, \(\mathcal{F}^{LH}\) and \(\mathcal{F}^{RH}\) are separate networks, enabling asymmetric information flow.

Forward Process. Following the improved DDPM [30] and DiffAct [6], we define a forward process that gradually adds Gaussian noise to action distributions. Given ground-truth one-hot encoded action distributions \(P^h\in \mathbb{R}^{C\times T}\), it is normalized to range \([-s_{de},s_{de}]\): \[X_{0}^h = (2P^h-1)\cdot s_{de}\] The forward process is: \[q(X_k^h | X_{0}^h)= \mathcal{N}(\sqrt{\bar{\alpha}_k}X_{0}^h,(1-\bar{\alpha}_k)\mathbf{I})\] where \(\bar{\alpha}_k= \cos^2(\frac{k/K+\delta}{1+\delta}\cdot \frac{\pi}{2})/\cos^2(\frac{\delta}{1+\delta} \cdot \frac{\pi}{2})\).

Hand-specific Denoising Decoder. Each hand-specific decoder predicts clean actions from noisy input \(\tilde{P}_k^h\) (denormalized \(X_k^h\)) conditioned on timestep \(k\) and fused features: \[\hat{\mathbf{Z}}^h = \mathcal{De}^h(\tilde{\mathbf{P}}_k^h, k, \tilde{\mathbf{H}}^h; \theta^h)\] The decoder uses sinusoidal timestep embeddings and cross-attention to query encoder features. We predict clean actions \(\hat{P}_0^h=\mathrm{Softmax}(\hat{Z}^h)\) and derive noise via: \[\hat{\epsilon}^h= \frac{X_k^h/\sqrt{\bar{\alpha}_k}-\hat{X}_0^h}{\sqrt{1/\bar{\alpha}_k-1}}\] where \(\hat{X}_0^h=\mathrm{clamp}((2\hat{P}_0^h-1)\cdot s_{de},-s_{de},s_{de})\).

DDIM Sampling. At inference, we use deterministic DDIM with \(K'=5\) steps. Starting from \(X_{K'}^h\sim \mathcal{N}(\mathbf{0},\mathbf{I})\): \[X_{k-1}^h=\sqrt{\bar{\alpha}_{k-1}}\cdot \hat{X}_0^h + \sqrt{1-\bar{\alpha}_{k-1}}\cdot \hat{\epsilon}^h\]

3.4.2 Adaptive Loss Weighting Mechanism↩︎

To balance the training progress on two hands, an adaptive bidirectional loss weighting mechanism is designed to automatically adjust loss weighting based on performance.

Performance Monitoring. We maintain a sliding window of recent validation accuracies: \[\mathcal{W}^h=\{\mathrm{Acc}_i^h\}_{i=ep-w}^{ep}\] where \(ep\) is the current epoch and \(w\) is the window size.

Adaptive Weight Computation. We compute boost factors based on the performance gaps: \[\begin{align} \beta^{LH} = \begin{cases} \min(\beta_{\max}, \max(\beta_{\min}, \frac{\bar{\mathcal{W}}^{RH}}{\bar{\mathcal{W}}^{LH}})) & \text{if } \frac{\bar{\mathcal{W}}^{LH}}{\bar{\mathcal{W}}^{RH}} < \Delta_{gap} \\ 1.0 & \text{otherwise} \end{cases} \end{align}\] \[\begin{align} \beta^{RH} = \begin{cases} \min(\beta_{\max}, \max(\beta_{\min}, \frac{\bar{\mathcal{W}}^{LH}}{\bar{\mathcal{W}}^{RH}})) & \text{if } \frac{\bar{\mathcal{W}}^{RH}}{\bar{\mathcal{W}}^{LH}} < \Delta_{gap} \\ 1.0 & \text{otherwise} \end{cases} \end{align}\] where \(\bar{\mathcal{W}}^{h}=\frac{1}{w}\sum_i \mathrm{Acc}_i^h\) is the windowed average, and \([\beta_{min},\beta_{max}]\) are bounds. This bidirectional adjustment automatically increases loss weights for the underperforming hand, preventing performance degradation.

Training Loss. The complete loss function combines encoder and decoder objectives with adaptive weighting: \[\begin{align} \mathcal{L}_{\text{total}} &= \sum_{h \in \{LH, RH\}} \beta^h (\lambda_{\text{enc}}^h \mathcal{L}_{\text{enc}}^h + \lambda_{\text{dec}}^h \mathcal{L}_{\text{dec}}^h) \end{align}\] where: \[\begin{align} \mathcal{L}_{\text{enc}}^h &= \mathcal{L}_{\text{CE}}(P^h, Y^h) + \lambda_{\text{sm}} \mathcal{L}_{\text{smooth}}(P^h) \end{align}\] \[\label{quation95loss} \begin{align} \mathcal{L}_{\text{dec}}^h = \mathcal{L}_{\text{CE}}(\hat{P}_0^h, Y^h) + \lambda_{\text{sm}} \mathcal{L}_{\text{smooth}}(\hat{P}_0^h) + \\ \lambda_{\text{bd}} \mathcal{L}_{\text{boundary}}(\hat{B}^h, B^h) \end{align}\tag{1}\] Here, \(\mathcal{L}_{\text{CE}}\) is cross-entropy with hand-specific class weights computed from training data, \(P^h=\mathrm{Softmax}(Z^h)\), \(\mathcal{L}_{\text{smooth}}(p_t)=\mathrm{MSE}(\mathrm{log}p_{t+1},\mathrm{log}p_t)\) enforces temporal smoothness. \(\mathcal{L}_{\text{boundary}}= \mathrm{BCE}(\hat{B}^h,B^h)\) penalizes boundary errors, where \(\hat{B}^h\in [0, 1]^{1\times (T-1)}\) is the predicted boundary and \(B^h\) is the ground truth. The loss weights \(\lambda_{enc}^h\), \(\lambda_{dec}^h\), \(\lambda_{sm}\) and \(\lambda_{bd}\) are scaled by \(\beta^h\) during training.

4 Experiments↩︎

4.1 Experimental Setup↩︎

Datasets. We evaluate Polyphony on three datasets covering dual-hand and single-stream scenarios: (1) HA-ViD [8] is a human assembly video dataset annotated with dual-hand actions, comprising 609 untrimmed videos from three viewpoints (side, front, top), with 75 action classes per hand. (2) ATTACH [31] is a two-handed assembly dataset containing 378 untrimmed videos across three views, with 24 action classes per hand. These two datasets are used to assess our method’s performance in dual-hand action segmentation—the primary focus of our work. (3) Breakfast [32] is a kitchen activity dataset with 1,712 untrimmed videos, covering 48 action classes. Breakfast is chosen to evaluate the generalization of our approach to conventional single-stream action segmentation in broader domains. For all datasets, we adopt the official splits.

Evaluation Metrics. Following standard protocols in temporal action segmentation, we report frame-wise accuracy (Acc), segmental edit distance (Edit), and segmental F1-scores (F1@10,25,50) [5]. For dual-hand action segmentation (HA-ViD, ATTACH), we report metrics separately for LH and RH.

Implementation Details. In Stage 1, we sample all segments plus 30, 20, and 5 random clips from each video for HA-ViD (per hand), ATTACH (per hand), and Breakfast, respectively. The visual backbone is a VideoMAE V2 ViT-Base model [4] pretrained on Kinetics-400 [33]. ADH-ViT is trained for 50 epochs with an alternation period of \(\Delta = 50\) steps, using an AdamW optimizer with a learning rate of 1e-3, cosine annealing scheduler, and weight decay of 0.1. In Stage 2, MiniLM [34] is employed to extract semantic embeddings. The temporal modeling component consists of three TCN blocks with channel dimensions [512, 128, 64] and kernel size 3. This stage is trained for 100 epochs with an AdamW optimizer (learning rate 3e-4). The adaptive alignment loss is weighted by \(\alpha = 0.7\). In Stage 3, the encoder has 10 layers and 64 channels. Each decoder contains 3 cross-attention layers. The diffusion process is configured with \(K=1000\) steps. \(\lambda_{sm}=0.05\) and \(\lambda_{bd}=0.2\) in Equation 1 . We optimize using Adam with a learning rate of 1e-3 over 1000 epochs. \(w=5\), \(\Delta_{gap}=0.95\), and \([\beta_{min},\beta_{max}]=[1,2]\) in the adaptive weighting. Three stages are trained sequentially.

4.2 Main Results↩︎

Polyphony is evaluated against state-of-the-art approaches across the three datasets on both dual-hand and single-stream action segmentation scenarios.

4.2.1 Dual-hand Action Segmentation↩︎

Tables 1 and 2 present results on HA-ViD and ATTACH. With I3D features, the same input as baseline methods, our approach achieves the best performance on most metrics, demonstrating the effectiveness of our diffusion-based segmentation model. With our MAS features, performance gains become substantial: on HA-ViD, ours surpasses the previous best method (FACT) by 12.0 and 16.8 points in left- and right-hand accuracy; on ATTACH, ours outperforms DiffAct by 5.3 and 4.8 percentage points in left- and right-hand accuracy. These improvements extend across all metrics, with Edit distance and F1 scores showing consistent gains of 2-10 points depending on the dataset. As our method performs dual-hand segmentation simultaneously through a single unified model with one shared visual backbone, it is more accurate and appropriate for dual-hand action segmentation scenarios compared with baseline approaches which train separate models for each hand.

4.2.2 Single-Stream Action Segmentation↩︎

Table 3 presents results on the Breakfast dataset. Despite being designed for dual-hand scenarios, Polyphony can adapt to single-stream settings by blocking one stream or duplicating inputs across both. Results here use the duplication strategy (details in Supplementary Material). With I3D features, we achieve 78.6% accuracy—establishing new state-of-the-art among I3D-based methods and outperforming DiffAct by 2.2 points. With MAS features, our method sets a new overall state-of-the-art at 82.5% accuracy, surpassing EAST (82.2%) despite using a substantially smaller backbone (ViT-Base with 86M parameters vs. ViT-Giant with 1B+ parameters). This demonstrates that Polyphony’s effectiveness stems from architectural design and semantic conditioning rather than relying on model scale.

4.3 Ablation Studies↩︎

We conduct comprehensive ablation studies on HA-ViD to validate our design.

4.3.1 Component Ablation↩︎

Table 4 analyzes the contribution of each component through progressive ablation. The shared motion features (MF) establish a strong baseline by capturing spatio-temporal patterns through the shared ViT backbone. Adding action features (AF) yields mixed results: while Edit distance improves for LH (+1.6%) and accuracy increases slightly for RH (+0.3%), LH accuracy marginally decreases (-0.2%). This suggests that hand-specific classification signals primarily enhance segment-level sequential accuracy rather than raw frame-level accuracy. Incorporating semantic features (SF) produces substantial gains, particularly for RH (Acc: +2.3%), validating their contribution. The asymmetric improvement (RH benefits more than LH) may reflect that RH, as the typically dominant hand, performs more diverse fine-grained manipulations that benefit from semantic grounding.

Critically, removing either cross-hand feature fusion (FF) or adaptive weighting (AW) not only degrades absolute performance but also exacerbates inter-hand performance imbalance. Without FF, the accuracy gap widens from 3.5% to 4.0%, with LH suffering disproportionate degradation (-1.6% vs. -1.1% for RH). Removing AW further widens the gap to 4.3%, with LH plummeting 1.8% compared to only 1.0% for RH. This asymmetric impact reveals that without inter-hand modeling and dynamic loss balancing, the dominant hand (RH) monopolizes gradient updates, leaving the non-dominant hand (LH) undertrained. These results validate that both components are essential not only for absolute performance but also for maintaining balanced learning across both hands.

4.3.2 ADH-ViT Study↩︎

ADH-ViT as a Standalone Action Recognition Model. To demonstrate the versatility of our approach, we evaluate ADH-ViT as a standalone action recognition model on the HA-ViD action recognition dataset [8] (Table 5). While all baseline methods employ separate models for each hand, our ADH-ViT recognizes both hands’ actions simultaneously through a single unified model. ADH-ViT (both) achieves competitive performance on both hands with the highest top-5 accuracy on LH (90.4%). Comparing ADH-ViT (seg) with (both) validates our dual sampling strategy, as incorporating random clip sampling improves top-1 accuracy by 13.1 and 15.2 percentage points for LH and RH respectively. While top-1 accuracy falls below the best baselines, this gap is offset by the reduced parameters and practicality in dual-hand action recognition tasks.

Alternating Training vs. Joint Training. To validate the efficacy of our alternating training strategy, it is compared against joint training (naive multi-task learning) on both action recognition and segmentation tasks. Table 6 demonstrates alternating training’s advantages across both dimensions. Joint training leads to a consistent performance gap favoring the right hand. Alternating training improves overall performance and reverses this imbalance in both tasks. Notably, LH benefits more from alternating training (+2.5% recognition, +3.9% segmentation) compared to RH (+1.0%, +0.3%), indicating that our mechanism specifically addresses the gradient domination problem where the dominant hand monopolizes learning in joint training.

4.3.3 Semantic Representation Study↩︎

While Table 4 validates the effectiveness of semantic features, this section provides a deeper analysis of the design choices in our semantic conditioning stage. We investigate three critical aspects: (1) the impact of action description structure (naive vs. structured compositional descriptions), (2) the choice of language models for semantic embedding, and (3) the effectiveness of semantic conditioning on addressing semantic ambiguity challenge. Our analysis combines quantitative performance metrics (Table 7 and 8) with alignment quality assessment (Figure 2). Table 7 demonstrates that structured compositional descriptions substantially outperform naive action descriptions. Figure 2 shows that structured descriptions produce sharper, more concentrated similarity distributions, indicating stronger semantic-visual alignment. Surprisingly, MiniLM-L6 achieves the best performance, outperforming larger models. This suggests that effective semantic conditioning requires balancing semantic expressiveness with visual feature compatibility—larger models with high-dimensional embeddings may diminish the visual information contribution. We evaluate performance on visually similar but semantically distinct fine-grained actions from HA-ViD side-view (e.g., "screw nut onto bolt" vs. "screw nut onto shaft"). Table 8 shows semantic features improve accuracy by 3.6 points on average, validating their role in disambiguation.

4.3.4 Qualitative Segmentation Results↩︎

Figure 3 presents qualitative segmentation results of Polyphony and DiffAct on a HA-ViD video for both hands. Polyphony (middle) produces more coherent predictions with more accurate action transitions and better inter-hand coordination compared to DiffAct (bottom), which exhibits severe over-segmentation and poor coordination.

The blue box highlights successful cases of our method: (1) correctly predicting asymmetric actions where hands perform different actions simultaneously; (2) distinguishing fine-grained semantic variations; (3) maintaining temporal synchronization when both hands execute coordinated actions. These successes collectively demonstrate that our approach addresses the core challenges of complex inter-hand dependencies, visual asymmetry, and semantic ambiguity in dual-hand action segmentation.

However, the gray boxes reveal failure modes that expose current limitations: (left) over-predicting hand coordination when the ground truth shows independent actions, suggesting our model has learned a coordination bias from training data; (middle) confusing tool-mediated actions and bare-handed actions, indicating a limited visual discrimination capability; (right) missing short, long-tail actions (rotate worm gear). These results reveal challenges with hand independence modeling, visual ambiguity, and long-tail action classes that motivate future work.

5 Conclusion↩︎

This paper introduced Polyphony, a unified three-stage method that addresses dual-hand action segmentation through an Alternating Dual-Hand Vision Transformer (ADH-ViT), Semantic Feature Conditioning, and Diffusion-based Dual-Hand Action Segmentation. Polyphony achieves state-of-the-art on both dual-hand benchmarks (HA-ViD, ATTACH) and the single-stream Breakfast benchmark, validating its effectiveness across a variety of tasks. The framework’s modular and versatile design, where each stage can be independently improved and ADH-ViT can operate as a standalone action recognition model, positions Polyphony as a foundational architecture for future research extending beyond bimanual scenarios to multi-agent collaborative behavior understanding.

While Polyphony achieves strong performance, several limitations remain. Our method exhibits learned coordination biases that over-predict hand synchronization and struggles with visually ambiguous tool-mediated actions and long-tail classes. Its reliance on manual construction of semantic descriptions limits scalability. Future directions include end-to-end approaches, self-supervised learning to reduce annotation dependence, automated semantic description generation via large language models, and extending our framework to multi-agent collaborative scenarios.

Acknowledgements↩︎

Work supported by CAIR and CQTS funded by Tamkeen NYUAD RI Award CG010 and CG008, respectively.

References↩︎