1 Introduction↩︎

Sign language translation (SLT) aims to convert continuous sign language videos into spoken language sentences, enabling more accessible communication between Deaf and hard-of-hearing signers and hearing communities [6], [7]. Recent gloss-free systems decode text directly from visual features using large pre-trained sequence-to-sequence language models [8]–[10], avoiding the costly gloss annotation required by gloss-based pipelines. However, most gloss-free benchmarks provide only natural language sentences as supervision, without gloss annotations or temporal segmentation [5], [11]. The core difficulty is therefore twofold: the semantic gap between vision and text, and the unknown correspondence between the temporal order of visual signs and the token order of the target text.

Sign languages often do not follow the word order of the corresponding spoken language: a signer may express key content words first and add grammatical relations later, so the temporal progression of gestures can differ from the target word order. As a result, visual features extracted along the video timeline are often misaligned with the text tokens that an autoregressive decoder is trained to predict. Left unresolved, the decoder must simultaneously learn translation and implicitly discover a latent cross-modal permutation, which increases optimization difficulty and degrades accuracy, especially on large-scale benchmarks such as How2Sign and OpenASL. Fig. 1 illustrates this gap: when alignment is left implicit, the cross-attention map can be noisy and the decoder may produce incorrect word ordering, whereas our approach makes alignment explicit.

Existing methods largely treat the visual stream as an ordered sequence and rely on decoder attention to bridge the modality gap. ShuBERT [3] and Uni-Sign [2] strengthen representations through large-scale pretraining but do not explicitly model frame-to-token correspondence within a sentence, while SpaMo [1] uses batch-level contrastive learning yet leaves fine-grained, sentence-internal alignment unresolved. These limitations motivate a model that learns correspondence between visual segments and text tokens and uses it to present the decoder with a semantically ordered visual sequence.

We propose VTaMo, a vision–text alignment model for gloss-free SLT that explicitly aligns and reorders visual features before text generation. Because many spoken words such as articles and prepositions have no sign-level counterpart, we align against and decode a content-word pseudo-gloss of the target rather than the raw sentence. Given a sign video, we build token-level pseudo-gloss embeddings from a frozen language model embedding layer and temporal visual embeddings from a video encoder, then estimate a soft alignment between visual positions and pseudo-gloss tokens by solving an entropy-regularized optimal transport problem [12] with a learnable null token. The null token absorbs transitional gestures and co-articulation that correspond to no explicit word, stabilizing alignment on continuous signing. Using the resulting transport plan, we reorder the visual features into a sequence that follows the target token order and feed it to the language model decoder. At inference the target token order is unknown, so the visual features are not reordered; the decoder generates pseudo-gloss directly from the signing-order features, and a lightweight text-only recovery model restores spoken word order and the omitted function words.

VTaMo is trained end to end with three complementary alignment objectives beyond the standard translation loss. First, a local optimal transport loss encourages fine-grained correspondence between temporal visual segments and text tokens while allowing unaligned frames to map to the null token. Second, a global loss calibrates the sentence-level geometry of the visual and textual embedding spaces through a learnable orthogonal transformation, supported by a memory queue that diversifies the sentence pairs used for alignment. Third, a position-aligned contrastive loss [13] sharpens token-level discriminability between reordered visual features and their text token embeddings without altering the language model embedding space. Together, these losses make the reordered visual features semantically coherent, simplifying decoder learning and improving robustness across datasets.

We evaluate on four widely used benchmarks spanning three languages and diverse data regimes: Phoenix-2014T (German), CSL-Daily (Chinese), How2Sign and OpenASL (English). Across all benchmarks, VTaMo achieves state-of-the-art performance under the gloss-free setting and yields consistent gains over strong baselines that decode without explicit alignment. Ablation studies verify that each alignment component contributes complementary improvements, and qualitative analysis of the learned transport plans reveals interpretable correspondences between signing segments and target tokens.

2 Related Work↩︎

2.1 Gloss-free SLT, Alignment, and Large-Scale Pretraining↩︎

Gloss-free sign language translation (SLT) translates continuous sign videos into spoken-language sentences without gloss supervision, but it typically underperforms gloss-based pipelines due to the absence of explicit intermediate structure [7], [14]–[20]. Prior work improves gloss-free SLT by strengthening temporal modeling [21], introducing cross-modal alignment objectives [22]–[25], incorporating large language models [8]–[10], scaling training data [4], [26], and large-scale sign pretraining such as ShuBERT [3] and Uni-Sign [2]; SpaMo [1] further emphasizes the non-trivial sign–text correspondence via contrastive objectives. These approaches—including VAP [25], which applies text-derived constraints only as visual pre-training—keep sentence-internal correspondence implicit and rely on coarse decoder attention to resolve word order. Our method instead estimates token-level alignment and reorders visual features before decoding via Sinkhorn OT with a learnable null assignment and complementary local and global objectives.

2.2 Cross-Modal and Video–Text Alignment↩︎

Our objectives build on general cross-modal alignment tools but adapt them to the structure of sign language. Entropy-regularized optimal transport with the Sinkhorn algorithm [12] is a standard device for matching features across modalities, e.g., aligning visual features with text prompts in vision–language models [27]; learnable orthogonal transformations relate two embedding spaces without distorting their geometry, as in cross-lingual word-embedding alignment [28]; and contrastive video–text frameworks such as CLIP4Clip [29], VideoCLIP [30], and X-CLIP [31] learn joint embeddings for retrieval. Closer in spirit, procedure-learning and step-grounding methods align video frames with an ordered sequence of procedure steps, e.g., grounding instructional-article steps via narrations [32] and optimal-transport-guided procedure learning [33]. These methods, however, target global or clip-level correspondence, or assume a largely fixed, monotonic step order. Sign language translation differs: the mapping is non-monotonic at the lexical level and partial, since many frames are transitional and words have no manual counterpart. VTaMo addresses both by coupling OT with a null assignment, sentence-level orthogonal calibration, and position-aligned contrastive learning to yield a reordered visual sequence.

3 Method↩︎

We present VTaMo, a framework for gloss-free sign language translation that explicitly aligns and reorders visual features before text generation. As illustrated in Fig. 2, VTaMo introduces three complementary alignment objectives: (1) local alignment via entropy-regularized optimal transport for fine-grained frame-to-token correspondences; (2) global alignment via a learnable orthogonal transformation to calibrate the embedding spaces at the sentence level; and (3) position-aligned contrastive learning for discriminative representations.

3.1 Problem Formulation↩︎

Given a sign language video, we extract frame-level features \(\mathbf{V} = \{\mathbf{v}_1, \dots, \mathbf{v}_L\}\) using a pre-trained CLIP-ViT encoder, where \(\mathbf{v}_i \in \mathbb{R}^{d_v}\). The goal is to produce the spoken language sentence corresponding to the video.

Spoken sentences contain many function words that have no direct sign-level realization, which makes a strict frame-to-word alignment ill-posed. We therefore align against, and decode, a pseudo-gloss sequence \(\mathbf{Y} = \{y_1, \dots, y_U\}\) of \(U\) tokens derived from the target sentence by a fixed offline filter. Concretely, we apply a frozen spaCy part-of-speech tagger that retains sign-relevant content tokens (NOUN, VERB, ADJ, ADV, NUM, PRON, PROPN) and removes tokens that usually lack a direct sign counterpart (DET, ADP, AUX, PART, PUNCT, CCONJ); the same filtering rule is applied across all datasets using language-specific spaCy models. Both the text tokens used for alignment and the decoder targets are pseudo-gloss, and a lightweight text-only recovery model restores fluent sentences at inference.

The visual features are projected to \(\mathbb{R}^{d_h}\) via a linear layer, temporally downsampled by an attention-based encoder to produce \(\mathbf{H} = \{\mathbf{h}_1, \dots, \mathbf{h}_M\}\) with \(M \ll L\), and mapped to the language model dimension \(d_t\) by a fusion projector. A LoRA-adapted Flan-T5 model then autoregressively generates the pseudo-gloss sequence. The central challenge is that sign language does not follow the word order of spoken language, and the temporal granularity of visual frames far exceeds that of linguistic tokens.

3.2 Attention-Based Temporal Encoding↩︎

We use an attention-based temporal convolution module with \(N_\ell\) cascaded layers, each performing \(2\times\) downsampling via learnable attention-weighted aggregation over local windows of size \(W\). For each output position \(i\), a local window \(\mathcal{W}_i\) centered at position \(2i\) is extracted, and a query is formed from the mean-pooled local context plus a learnable positional embedding \(\mathbf{p}\): \[\mathbf{q}_i = \frac{1}{|\mathcal{W}_i|}\sum_{j \in \mathcal{W}_i}\mathbf{x}_j + \mathbf{p}. \label{eq:temporal95query}\tag{1}\] Multi-head attention with relative position bias then aggregates the local context: \[\alpha_{i,j} = \mathrm{softmax}\!\left(\frac{\mathbf{q}_i^\top \mathbf{k}_j}{\sqrt{d_h / N_\text{head}}} + r_{j-c_i}\right), \qquad \mathbf{o}_i = \sum_{j \in \mathcal{W}_i}\alpha_{i,j}\,\mathbf{v}_j, \label{eq:temporal95attn}\tag{2}\] where \(c_i\) is the window center, \(r_{j-c_i}\) is a learnable relative position bias, and \(N_\text{head}\) is the number of attention heads. This ensures gradient flow to all frames, yielding more expressive temporal representations than max-pooling.

3.3 Local Alignment via Optimal Transport↩︎

Local alignment is the core component of VTaMo, establishing fine-grained correspondences between temporal visual segments and text tokens, formulated as an entropy-regularized optimal transport problem. Let \(\mathbf{S} = \{\mathbf{s}_1, \dots, \mathbf{s}_M\} \in \mathbb{R}^{M \times d_t}\) denote the visual features after fusion projection, and \(\mathbf{E} = \{\mathbf{e}_1, \dots, \mathbf{e}_U\} \in \mathbb{R}^{U \times d_t}\) the pseudo-gloss token embeddings from the frozen encoder embedding layer. Since \(\mathbf{S}\) is optimized against the fixed text embeddings \(\mathbf{E}\) under the alignment and translation losses, their cosine similarity reflects sign-level semantic correspondence, making it a meaningful transport cost. We introduce a single learnable null token \(\mathbf{e}_\varnothing \in \mathbb{R}^{d_t}\), prepended at position \(0\) of the text sequence, forming an augmented sequence \(\tilde{\mathbf{E}} \in \mathbb{R}^{K \times d_t}\) with \(K = U + 1\). The assignment is deliberately not one-to-one: frames that carry no explicit sign-language semantics—such as background, transition, or reset frames—are all allowed to align to this same null token, so the model is never forced to map every frame onto a content token. The pairwise cost matrix uses cosine distance, with a learnable bias \(b_\varnothing\) controlling null affinity: \[C_{m,k} = \begin{cases} 1 - \bar{\mathbf{s}}_m^\top \bar{\mathbf{e}}_k - b_\varnothing, & \text{if } k = 0 \text{ (null)}, \\ 1 - \bar{\mathbf{s}}_m^\top \bar{\mathbf{e}}_k, & \text{otherwise}, \end{cases} \label{eq:cost95matrix}\tag{3}\] where \(\bar{\mathbf{s}}_m\) and \(\bar{\mathbf{e}}_k\) are \(\ell_2\)-normalized vectors. We solve the entropy-regularized OT problem via the Sinkhorn algorithm in log-domain: \[\mathbf{A}^* = \arg\min_{\mathbf{A} \in \Pi(\mathbf{a}, \mathbf{b})} \; \langle \mathbf{A}, \mathbf{C} \rangle - \varepsilon\, H(\mathbf{A}), \label{eq:ot95problem}\tag{4}\] where \(\Pi(\mathbf{a}, \mathbf{b})\) is the set of transport plans with uniform marginals over valid frame and token positions, \(H(\mathbf{A}) = -\sum_{m,k} A_{m,k}\log A_{m,k}\) is the entropy term, and \(\varepsilon\) controls the sharpness of assignments. For numerical stability we iterate the dual potentials in the log domain for \(N_\text{iter}\) steps, \[\begin{align} \log u_m &\leftarrow \log a_m - \log\textstyle\sum_k \exp\!\left(-C_{m,k}/\varepsilon + \log v_k\right), \nonumber \\ \log v_k &\leftarrow \log b_k - \log\textstyle\sum_m \exp\!\left(-C_{m,k}/\varepsilon + \log u_m\right), \label{eq:sinkhorn95dual} \end{align}\tag{5}\] and recover the plan as \(A_{m,k} = \exp\!\left(\log u_m - C_{m,k}/\varepsilon + \log v_k\right)\). We adopt a multi-phase annealing schedule that progressively decreases \(\varepsilon\) from a high initial value to encourage exploration before converging to sharp, discrete assignments.

The local alignment loss combines three terms. The first is the transport cost \(\mathcal{L}_\text{trans} = \frac{1}{B}\sum_i \sum_{m,k} A_{m,k}^{(i)} C_{m,k}^{(i)}\). The second is a temporal variation regularization that penalizes adjacent frames switching tokens, computed on the row-normalized plan \(\hat{\mathbf{A}}\) restricted to real text tokens: \[\mathcal{L}_\text{tv} = \frac{1}{(M-1)(K-1)}\sum_{m=1}^{M-1}\sum_{k=1}^{K-1}\big|\hat{A}_{m+1,k} - \hat{A}_{m,k}\big|. \label{eq:tv95loss}\tag{6}\] The third is a null cost regularization that keeps the mean cost-based null affinity \(\bar{p}_\varnothing\)—computed via a row-wise softmax on the cost matrix independently of the transport plan—below a target ratio \(\rho_\text{target}\), preventing the learnable bias \(b_\varnothing\) from making the null column uniformly cheaper than all real tokens: \[\mathcal{L}_\text{null} = \big[\bar{p}_\varnothing - \rho_\text{target}\big]_{+}^{2}, \qquad \bar{p}_\varnothing = \frac{1}{M}\sum_{m=1}^{M} \frac{\exp(-C_{m,0}/\tau)}{\sum_{k=0}^{K-1}\exp(-C_{m,k}/\tau)}. \label{eq:null95loss}\tag{7}\] The complete local objective is their weighted sum: \[\mathcal{L}_\text{local} = \beta_\text{local}\, \mathcal{L}_\text{trans} + \beta_\text{tv}\, \mathcal{L}_\text{tv} + \beta_\text{null}\, \mathcal{L}_\text{null}. \label{eq:total95local95loss}\tag{8}\]

3.4 Global Alignment via Orthogonal Transformation↩︎

Because the visual and textual encoders are pre-trained on different modalities, paired sign and text sentence embeddings can suffer a coordinate (orientation) mismatch even when they are semantically equivalent. We therefore apply a learnable orthogonal transformation—a rotation of the feature space that preserves vector lengths and angular relationships—to the visual sentence embeddings before comparing the two modalities at the sentence level. We constrain the transformation to be orthogonal precisely because the goal is to correct orientation without reshaping the space: this preserves norms and angular structure for the cosine-based transport cost and avoids the spurious low-cost matches that an unconstrained projection could introduce.

We aggregate frame-level features into one sentence-level vector per video and per sentence via attention pooling, using a mean query \(\mathbf{q}\) over valid positions and the normalized weighted sum \[\mathbf{z} = \frac{\sum_l \alpha_l \mathbf{x}_l}{\big\|\sum_l \alpha_l \mathbf{x}_l\big\|_2}, \qquad \alpha_l = \mathrm{softmax}_l\!\big(\mathbf{x}_l^\top \mathbf{q}/\sqrt{d_t}\big), \label{eq:attn95pool}\tag{9}\] yielding normalized sentence-level vectors \(\mathbf{z}^\text{sign}, \mathbf{z}^\text{text} \in \mathbb{R}^{d_t}\). A FIFO memory queue stores recent sentence-level vector pairs so that the sentence-level transport is computed over a sufficiently diverse batch rather than a few in-batch pairs. A learnable matrix \(\mathbf{T} \in \mathbb{R}^{d_t \times d_t}\), initialized near identity, transforms the visual sentence vectors: \[\tilde{\mathbf{z}}_n^\text{sign} = \frac{\mathbf{z}_n^\text{sign}\, \mathbf{T}}{\|\mathbf{z}_n^\text{sign}\, \mathbf{T}\|_2}. \label{eq:global95transform}\tag{10}\] We then form the pairwise cosine cost \(C^g_{n,n'} = 1 - (\tilde{\mathbf{z}}_n^\text{sign})^\top \mathbf{z}_{n'}^\text{text}\) between the transformed visual vectors and the textual vectors and solve a Sinkhorn OT problem with uniform marginals to obtain a sentence-level transport plan \(\mathbf{P}\); the resulting Earth Mover’s Distance loss is \[\mathcal{L}_\text{EMD} = \sum_{n,n'} P_{n,n'}\, (1 - (\tilde{\mathbf{z}}_n^\text{sign})^\top \mathbf{z}_{n'}^\text{text}). \label{eq:emd95loss}\tag{11}\] An orthogonality constraint \(\mathcal{L}_\text{orth} = \|\mathbf{T}^\top \mathbf{T} - \mathbf{I}\|_F^2\) keeps \(\mathbf{T}\) close to a pure rotation and preserves pairwise distances. The global loss is activated only after local alignment stabilizes, using a scheduled ramp-up: \[\mathcal{L}_\text{global} = \lambda_g(t) \cdot \mathcal{L}_\text{EMD} + \beta_\text{orth}\, \mathcal{L}_\text{orth}. \label{eq:total95global95loss}\tag{12}\]

3.5 Window-Based Feature Reordering↩︎

Once local alignment has produced a reliable transport plan \(\mathbf{A}\), we reorder the visual features to match the target token order. This is necessary because the decoder’s autoregressive cross-entropy loss expects the input to follow the target text order; otherwise the visual features stay in temporal order and the cross-entropy gradient counteracts the alignment objectives.

Concretely, we partition the \(M\) visual frames into \(N_w = U + 2\) overlapping temporal windows, where the two extra windows absorb boundary effects; the \(w\)-th window spans \([s_w, e_w)\) with \(s_w = \lfloor w\,(M-1)/N_w + 0.5\rfloor\) and \(e_w = \lfloor (w+1)(M-1)/N_w + 0.5\rfloor + 1\). For each window we accumulate the alignment mass \(\boldsymbol{\pi}_w = \sum_{m=s_w}^{e_w-1}\mathbf{A}_{m,:} \in \mathbb{R}^{K}\) and assign it to the real text token with the largest aggregated mass, \(\phi(w) = \arg\max_{u \in \{1,\dots,U\}} \pi_{w,u}\); windows dominated by the null token are excluded. Collecting the windows assigned to each token \(u\) into \(\Omega_u = \{w \mid \phi(w) = u\}\) and traversing the token order left to right yields the reordered sequence \[\mathbf{R} = \mathrm{Concat}\big(\{\mathbf{S}_{s_w:e_w-1} \mid w \in \Omega_1^{\uparrow}\}, \dots, \{\mathbf{S}_{s_w:e_w-1} \mid w \in \Omega_U^{\uparrow}\}\big) \in \mathbb{R}^{U' \times d_t}, \label{eq:window95reorder}\tag{13}\] where \(\Omega_u^{\uparrow}\) orders the windows in \(\Omega_u\) by their original temporal index. This preserves local temporal continuity, and the reordered sequence is fed to the decoder. For the contrastive objective, frames assigned to each token \(u\) are mean-pooled to produce a single token-level representation \(\mathbf{r}_u\), yielding \(\mathbf{R} = \{\mathbf{r}_1, \dots, \mathbf{r}_U\}\) aligned one-to-one with \(\mathbf{E}\).

3.6 Position-Aligned Contrastive Learning↩︎

Because reordering is used only during training, the model must also function at inference when the target order is unknown. We therefore impose a token-level contrastive objective that binds each reordered visual feature to its text token embedding, so that even when tokens arrive out of order each is paired with its correct counterpart and the decoder can recover its lexical item; restoring fluent word order is then left to the recovery model.

We apply learnable projection heads to both modalities: \[\hat{\mathbf{r}}_u = \text{LN}\!\big(\mathbf{W}_2^\text{v}\, \sigma(\mathbf{W}_1^\text{v}\, \mathbf{r}_u)\big), \quad \hat{\mathbf{e}}_u = \text{LN}\!\big(\mathbf{W}_2^\text{t}\, \sigma(\mathbf{W}_1^\text{t}\, \mathbf{e}_u)\big),\] where \(\sigma\) is GELU activation and \(\text{LN}\) is layer normalization. After \(\ell_2\)-normalization, the InfoNCE loss is: \[\mathcal{L}_\text{contra} = -\frac{1}{|\mathcal{V}|}\sum_{i \in \mathcal{V}} \log \frac{\exp\!\big(\text{sim}(\hat{\mathbf{r}}_i, \hat{\mathbf{e}}_i) / \tau_c\big)}{\displaystyle\sum_{j \in \mathcal{V}} \exp\!\big(\text{sim}(\hat{\mathbf{r}}_i, \hat{\mathbf{e}}_j) / \tau_c\big)}, \label{eq:contrastive95loss}\tag{14}\] where \(\mathcal{V}\) denotes valid token positions and \(\tau_c\) is a temperature hyperparameter. Gradients flow through the visual branch but are blocked from the text branch to preserve the pre-trained language model representations.

3.6.0.1 Inference and pseudo-gloss recovery.

At test time the target order is unknown, so no reordering is applied: the decoder reads the visual tokens in their original temporal order and emits a pseudo-gloss sequence in visual (signing) order rather than spoken-language order. The token-level grounding above makes this feasible, since each visual token is decoded into its correct lexical item even when the tokens arrive out of order. To recover a fluent translation, we pass this video-order pseudo-gloss through a lightweight recovery model that restores spoken word order and re-inserts the function words dropped by the pseudo-gloss filter (Sec. 3.1). Crucially, this recovery model is trained purely on text and never sees sign videos: we take plain sentences, apply the same pseudo-gloss extractor, permute the content words, and train the model to reconstruct the original sentence from this shuffled input.

The overall training objective combines the translation loss with the three alignment losses: \[\mathcal{L} = \mathcal{L}_\text{T5} + \lambda_\text{local}\, \mathcal{L}_\text{local} + \lambda_g(t)\, \mathcal{L}_\text{EMD} + \beta_\text{orth}\, \mathcal{L}_\text{orth} + \beta_\text{contra}\, \mathcal{L}_\text{contra}, \label{eq:total95loss}\tag{15}\] where \(\mathcal{L}_\text{T5}\) is the cross-entropy loss for autoregressive generation. Training proceeds in two phases: in the first phase, only alignment losses are active while the language model is frozen, allowing the visual encoder to establish meaningful correspondences; in the second phase, the full objective is optimized jointly with the LoRA-adapted language model.

4 Experiments↩︎

4.1 Datasets and Metrics↩︎

We evaluate our method on four publicly available sign language translation benchmarks that span different languages, vocabulary sizes, and recording conditions: Phoenix-2014T [6] (German), CSL-Daily [7] (Chinese), How2Sign [5] and OpenASL [11] (English). We report BLEU-\(n\) scores [34] for \(n \in \{1, 2, 3, 4\}\) and ROUGE-L F1 [35] as the primary evaluation metrics. For English SLT datasets (How2Sign and OpenASL), we additionally report BLEURT [36] scores following established protocol. For every benchmark, each model trains only on its official split without additional sign-language data or task-specific pre-training.

4.2 Implementation Details↩︎

4.2.0.1 Visual features.

We extract frame-level spatial features using a pre-trained CLIP-ViT-Large model [37] with S\(^2\)-Wrapper [38] for multi-scale feature extraction at scales 1 and 2, yielding 2048-dimensional feature vectors per frame.

4.2.0.2 Language model.

We employ Flan-T5-XL [39] as the language model backbone and adapt it using LoRA [40] with rank \(r = 16\), scaling factor \(\alpha = 32\), and dropout rate 0.1. The LoRA adapters are applied to the query and value projection matrices across all transformer layers.

4.2.0.3 Architecture.

The visual features are projected from \(d_v = 2048\) to an intermediate dimension \(d_h = 768\) through a linear layer. The attention-based temporal encoder consists of \(N_\ell = 2\) cascaded layers with 4 attention heads and a window size of 8, yielding a \(4\times\) temporal downsampling. The fusion projector maps the features from \(d_h\) to the T5 embedding dimension \(d_t\). The contrastive projection heads are two-layer MLPs with GELU activation and layer normalization, projecting to 768 dimensions.

4.2.0.4 Training.

We train all models using AdamW [41] with a peak learning rate of \(6 \times 10^{-4}\), cosine learning rate schedule, and linear warm-up over the first 2,000 steps. The batch size is 6 with gradient accumulation over 2 steps, and gradient clipping is applied with a maximum norm of 1.0. All experiments were conducted on a single NVIDIA A100 GPU with mixed-precision (bf16) training.

4.2.0.5 Alignment hyperparameters.

For the local alignment, the Sinkhorn algorithm runs for \(N_\text{iter} = 10\) iterations. The epsilon annealing schedule proceeds from \(\varepsilon_\text{high} = 0.12\) to \(\varepsilon_\text{mid} = 0.10\) over the first 10 epochs and further to \(\varepsilon_\text{low} = 0.03\) over the following 80 epochs. The null ratio target is \(\rho_\text{target} = 0.2\), and the loss weights are \(\beta_\text{local} = 2.0\), \(\beta_\text{tv} = 0.1\), \(\beta_\text{null} = 0.1\), with the overall local-alignment weight \(\lambda_\text{local} = 1.0\). For the global alignment, the EMD solver uses \(\varepsilon_g = 0.05\) with 20 Sinkhorn iterations, the memory queue capacity is 256, and the weight ramps from 0 to \(\lambda_g^\text{max} = 0.1\) over 4,000 steps after warm-up. The orthogonality penalty coefficient is \(\beta_\text{orth} = 0.05\). The contrastive learning temperature is \(\tau_c = 0.1\), and the contrastive loss weight is \(\beta_\text{contra} = 1.0\).

4.2.0.6 Training cost.

On a single A100, end-to-end training takes 14 h on Phoenix-2014T, 32 h on CSL-Daily, and 50 h on How2Sign. The optimal transport solvers add negligible overhead: the local Sinkhorn step takes 54 ms (\(<\)​3.2% of the per-step time), and the global OT over the \(256\times256\) memory queue takes \(<\)​38 ms.

4.3 Comparison with State-of-the-Art↩︎

Results on Phoenix-2014T and CSL-Daily. As shown in Table 1, most prior methods rely on visual-encoder fine-tuning or gloss supervision, yet VTaMo reaches state-of-the-art on both benchmarks without either. On Phoenix-2014T it attains 28.86 BLEU-4, surpassing the strongest gloss-free method SpaMo [1] by 4.54 points and coming within 0.09 of the best gloss-based result (TS-SLT [18], 28.95). On CSL-Daily it reaches 27.16 BLEU-4, a 6.61-point gain over SpaMo and 0.80 over Uni-Sign, showing that explicit multi-granularity alignment especially helps on topically diverse data.

Results on How2Sign. How2Sign is more challenging, with open-domain topics and a much larger vocabulary. As shown in Table ¿tbl:tab:how2sign?, VTaMo achieves the best BLEU-1, BLEU-4, and BLEURT on the official test set (BLEU-4 18.47, BLEURT 50.26) using only RGB and no large-scale pre-training—an 8.36 BLEU-4 gain over SpaMo [1]—and ranks second on ROUGE-L behind SSVP-SLT [4] (38.40 vs.). It also surpasses SHuBERT [3] (16.2 / 49.9) and Uni-Sign [2] (14.9 / 49.4), both using pose+RGB and large-scale pre-training.

Results on OpenASL. OpenASL is the hardest setting, with a vocabulary exceeding 10,000 words. As shown in Table [tbl:tab:openasl], VTaMo attains 25.94 BLEU-4 and 62.48 BLEURT on the official test set, outperforming Uni-Sign [2] and SHuBERT [3] by 2.80 and 2.74 BLEU-4 despite their pose+RGB inputs and large-scale pre-training. Gains over SpaMo grow on the larger-vocabulary How2Sign and OpenASL, consistent with alignment helping more as complexity increases.

4.4 Ablation Study↩︎

All ablations are conducted on the Phoenix-2014T test set.

4.4.0.1 Alignment components.

Removing each alignment loss in turn (Table ¿tbl:tab:ablation?) shows that all three are necessary and non-substitutable. The position-aligned contrastive loss \(\mathcal{L}_\text{contra}\) matters most (\(-\)​6.30 BLEU-4), providing token-level discriminative grounding; the local OT loss \(\mathcal{L}_\text{local}\) is second (\(-\)​5.09), as it underpins the window-based reordering that feeds ordered inputs to the decoder; and the global loss \(\mathcal{L}_\text{EMD}\) (\(-\)​2.65) adds complementary embedding-space calibration.

4.4.0.2 Language-model backbone.

To test whether the gains stem from a favorable language model rather than our alignment design, we vary the backbone following SpaMo’s protocol (Table [tbl:tab:llm95ablation]). The ranking matches SpaMo—Flan-T5-XL is best, mT0-XL is close behind, and Llama-2 underperforms despite its larger size—confirming that scaling the LM alone does not help under the limited-data SLT regime. Critically, under every fixed backbone our method beats SpaMo by \(+2.49\) to \(+4.54\) BLEU-4, so the improvement comes from the alignment block.

4.4.0.3 Local alignment quality.

Three transport-plan metrics (Table ¿tbl:tab:alignment95metrics?)—peak assignment probability, normalized entropy, and segment change rate—characterize alignment sharpness. The full model is the sharpest and most confident; removing epsilon annealing yields diffuse assignments, removing the TV loss fragments them, and removing the null token forces transitional frames onto real tokens.

4.4.0.4 Global alignment components.

Table [tbl:tab:global95ablation] isolates the global module and reports the epoch of peak development BLEU-4 as an efficiency proxy. The orthogonality constraint prevents distance distortion, scheduled activation avoids premature gradients before local plans stabilize, and the memory queue is the largest contributor to faster convergence (36 vs. epochs without it).

4.5 Noise Sensitivity Analysis↩︎

Since VTaMo operates on RGB input without visual-encoder fine-tuning, we assess its robustness to appearance shifts. Using Stable Diffusion [45] with ControlNet [46] for pose-conditioned synthesis, we build four controlled conditions on the entire How2Sign test set while preserving each signer’s hand and body kinematics (Fig. 3): two replace the background (BG-1, outdoor park; BG-2, indoor kitchen) and two alter signer appearance (AP-1, full body shape and identity transfer to a synthetic signer; AP-2, clothing). All appearance modifications are entirely synthetic—the transferred identity in AP-1 does not represent any real individual—and are used solely for robustness evaluation. Table 2 reports single-run performance across all conditions: background replacement causes only marginal degradation (\(\leq\)​0.42 BLEU-4) and appearance edits even smaller drops (\(\leq\)​0.33), showing that VTaMo attends to gesture semantics rather than scene context or signer identity. Additional examples are in the supplementary material.

4.6 Qualitative Analysis↩︎

Fig. 4 (a) visualizes the learned transport plans on How2Sign: they exhibit clear block-diagonal structure—contiguous frame groups aligning to individual tokens—with the null token absorbing transitional frames, and the ordering is largely monotonic with local reorderings where signing order diverges from the target. Fig. 4 (c) marks the video segment assigned to each predicted content token along the timeline, and Fig. 4 (b) shows a t-SNE with distinct semantic clusters, indicating alignment that is both temporally coherent at the frame level and semantically organized at the sentence level.

5 Conclusion↩︎

VTaMo achieves state-of-the-art gloss-free SLT through explicit multi-granularity cross-modal alignment: local OT with a null token, global orthogonal calibration, and position-aligned contrastive learning. Experiments on four benchmarks confirm consistent gains, and ablations verify each component’s contribution, showing that explicit alignment is an effective, transferable direction for gloss-free SLT with modest overhead.

6 Limitations↩︎

Our evaluation targets offline accuracy rather than systems-level concerns such as real-time or on-device deployment; the spoken-language order is also recovered by a text-only model rather than observed directly.

Acknowledgements↩︎

This work was partially supported by ChatSign Technology, Ltd.; and the NYUAD Center for AI and Robotics (CAIR), funded by Tamkeen under the NYUAD Research Institute Award CG010. The generous computational support was provided by the HPC resources at NYU Abu Dhabi and NYU New York.

Appendix↩︎

This appendix provides expanded details and additional results complementing the main paper. Section 7 presents the full formulations for each component of VTaMo, including the attention-based temporal encoder, the complete local alignment derivation (null-token augmentation, cost matrix, Sinkhorn iterations, epsilon annealing, and loss terms), the global alignment module (attention pooling, memory queue, orthogonal mapping, and EMD loss), the window-based reordering procedure, and the position-aligned contrastive objective. Section 8 contains supplementary experimental results: a comparison of temporal-pooling strategies (attention pooling vs.max-pooling), a comparison against generic video–text alignment objectives (CLIP4Clip, VideoCLIP, X-CLIP), qualitative visualizations of the learned transport plans and t-SNE feature geometry, and representative translation examples on How2Sign comparing VTaMo against SpaMo.

7 Additional Method Details↩︎

7.1 Attention-Based Temporal Encoding↩︎

The main paper briefly introduces the attention-based temporal encoder. Here we provide the full formulation. Prior work commonly uses fixed temporal convolution with max-pooling for downsampling sign language features. Such pooling discards gradient information from non-selected frames and limits the expressiveness of the learned temporal representation. We instead use an attention-based temporal convolution module that performs learnable weighted aggregation inside each local temporal region. The module contains \(N_\ell\) cascaded attention-pooling layers. Each layer performs \(2\times\) temporal downsampling, yielding an overall downsampling factor of \(2^{N_\ell}\). Given an input sequence \(\mathbf{X} \in \mathbb{R}^{T \times d_h}\), we first apply a one-dimensional convolution with kernel size \(5\) and batch normalization. For each output position \(i\), a local window \(\mathcal{W}_i\) centered around position \(2i\) is extracted. A learnable query is then constructed from the mean-pooled local context and a positional embedding: \[\mathbf{q}_i = \frac{1}{|\mathcal{W}_i|}\sum_{j \in \mathcal{W}_i}\mathbf{x}_j + \mathbf{p},\] where \(\mathbf{p} \in \mathbb{R}^{d_h}\) denotes the positional embedding.

Multi-head attention with relative position bias is then applied within the window: \[\alpha_{i,j} = \mathrm{softmax} \left( \frac{\mathbf{q}_i^\top \mathbf{k}_j}{\sqrt{d_h / N_{\mathrm{head}}}} + r_{j-c_i} \right), \qquad \mathbf{o}_i = \sum_{j \in \mathcal{W}_i}\alpha_{i,j}\mathbf{v}_j,\] where \(c_i\) denotes the center of the local window and \(N_{\mathrm{head}}\) is the number of attention heads. The output is passed through a linear projection, layer normalization, and dropout. This design allows gradients to flow to all frames in the local region and produces more informative temporal representations than max-pooling.

7.2 Expanded Local Alignment Formulation↩︎

The main paper presents the core optimal transport formulation. This subsection gives the complete construction of the local alignment module. Let \(\mathbf{S} = \{\mathbf{s}_1, \dots, \mathbf{s}_M\} \in \mathbb{R}^{M \times d_t}\) denote the visual features after fusion projection, and let \(\mathbf{E} = \{\mathbf{e}_1, \dots, \mathbf{e}_U\} \in \mathbb{R}^{U \times d_t}\) denote the text token embeddings from the frozen encoder embedding layer of the language model.

7.2.1 Null token augmentation↩︎

Not every visual frame corresponds to a valid lexical unit. Many frames represent transitional gestures, co-articulation, or short pauses. To model this explicitly, we introduce a learnable null token \(\mathbf{e}_{\varnothing} \in \mathbb{R}^{d_t}\) and prepend it to the text sequence: \[\tilde{\mathbf{E}} = \{\mathbf{e}_{\varnothing}, \mathbf{e}_1, \dots, \mathbf{e}_U\} \in \mathbb{R}^{K \times d_t}, \qquad K = U + 1.\] A learnable scalar bias \(b_{\varnothing}\) controls the affinity of visual frames to the null token.

7.2.2 Cost matrix construction↩︎

We compute the pairwise transport cost using cosine distance. Both modalities are first \(\ell_2\) normalized: \[\bar{\mathbf{s}}_m = \frac{\mathbf{s}_m}{\|\mathbf{s}_m\|_2}, \qquad \bar{\mathbf{e}}_k = \frac{\tilde{\mathbf{e}}_k}{\|\tilde{\mathbf{e}}_k\|_2}.\] The local cost matrix \(\mathbf{C} \in \mathbb{R}^{M \times K}\) is defined as \[C_{m,k} = \begin{cases} 1 - \bar{\mathbf{s}}_m^\top \bar{\mathbf{e}}_k - b_{\varnothing}, & \text{if } k = 0, \\ 1 - \bar{\mathbf{s}}_m^\top \bar{\mathbf{e}}_k, & \text{otherwise}. \end{cases} \label{eq:supp95cost95matrix}\tag{16}\] Columns corresponding to padding tokens are assigned a very large cost to avoid spurious assignments.

7.2.3 Sinkhorn optimal transport↩︎

We solve the entropy-regularized transport problem \[\mathbf{A}^{*} = \arg\min_{\mathbf{A}\in \Pi(\mathbf{a},\mathbf{b})} \langle \mathbf{A}, \mathbf{C}\rangle - \varepsilon H(\mathbf{A}),\] where \(\Pi(\mathbf{a},\mathbf{b})\) denotes the set of transport plans with prescribed marginals, and \(H(\mathbf{A}) = -\sum_{m,k} A_{m,k}\log A_{m,k}\) is the entropy term. The marginals are initialized as uniform distributions over valid visual positions and valid token positions.

For numerical stability, we use the Sinkhorn algorithm in the log domain. Let \(\mathbf{K} = \exp(-\mathbf{C}/\varepsilon)\) denote the Gibbs kernel. The dual variables are updated as \[\begin{align} \log u_m &\leftarrow \log a_m - \log \sum_k \exp\left(-C_{m,k}/\varepsilon + \log v_k\right), \\ \log v_k &\leftarrow \log b_k - \log \sum_m \exp\left(-C_{m,k}/\varepsilon + \log u_m\right). \end{align}\] After \(N_{\mathrm{iter}}\) iterations, the transport plan is recovered as \[A_{m,k} = \exp\left(\log u_m - C_{m,k}/\varepsilon + \log v_k\right).\]

7.2.4 Epsilon annealing↩︎

To balance exploration and assignment sharpness, we use a multi-stage annealing schedule for \(\varepsilon\): \[\varepsilon(t)= \begin{cases} \varepsilon_{\mathrm{high}}, & t \le t_{\mathrm{warm}}, \\ \varepsilon_{\mathrm{high}} - \frac{\varepsilon_{\mathrm{high}}-\varepsilon_{\mathrm{mid}}}{t_2-t_{\mathrm{warm}}} \left(t-t_{\mathrm{warm}}\right), & t_{\mathrm{warm}} < t \le t_2, \\ \varepsilon_{\mathrm{mid}} - \frac{\varepsilon_{\mathrm{mid}}-\varepsilon_{\mathrm{low}}}{t_3-t_2} \left(t-t_2\right), & t_2 < t \le t_3, \\ \varepsilon_{\mathrm{low}}, & t > t_3. \end{cases} \label{eq:supp95eps95schedule}\tag{17}\] This schedule keeps the transport plan soft during early training and gradually sharpens the assignments after the alignment becomes stable.

7.2.5 Local alignment loss terms↩︎

The local alignment objective contains three terms. The transport cost term is \[\mathcal{L}_{\mathrm{trans}} = \frac{1}{B} \sum_{i=1}^{B} \sum_{m,k} A_{m,k}^{(i)} C_{m,k}^{(i)}. \label{eq:supp95local95loss}\tag{18}\] To encourage temporal smoothness, we compute a total variation penalty on the row-normalized transport plan restricted to real text tokens. Let \[\hat{A}_{m,k} = \frac{A_{m,k}}{\sum_{k'=1}^{K-1}A_{m,k'}}, \qquad k=1,\dots,K-1.\] The temporal variation term is \[\mathcal{L}_{\mathrm{tv}} = \frac{1}{(M-1)(K-1)} \sum_{m=1}^{M-1} \sum_{k=1}^{K-1} \left| \hat{A}_{m+1,k} - \hat{A}_{m,k} \right|. \label{eq:supp95tv95loss}\tag{19}\] To prevent the learnable null bias \(b_\varnothing\) from making the null token uniformly cheaper than real tokens, we regulate the cost-based null affinity (computed via row-wise softmax on the cost matrix, independently of the transport plan): \[p_{\varnothing}^{(m)} = \frac{\exp(-C_{m,0}/\tau)}{\sum_{k=0}^{K-1}\exp(-C_{m,k}/\tau)},\] \[\mathcal{L}_{\mathrm{null}} = \left[\bar{p}_{\varnothing}-\rho_{\mathrm{target}}\right]_{+}^{2}, \qquad \bar{p}_{\varnothing} = \frac{1}{M}\sum_{m=1}^{M}p_{\varnothing}^{(m)}. \label{eq:supp95null95loss}\tag{20}\] The complete local objective is \[\mathcal{L}_{\mathrm{local}} = \beta_{\mathrm{local}} \mathcal{L}_{\mathrm{trans}} + \beta_{\mathrm{tv}} \mathcal{L}_{\mathrm{tv}} + \beta_{\mathrm{null}} \mathcal{L}_{\mathrm{null}}. \label{eq:supp95total95local}\tag{21}\]

7.3 Expanded Global Alignment Formulation↩︎

The main paper introduces the sentence-level global alignment mechanism. This subsection provides the full formulation.

7.3.1 Sentence-level attention pooling↩︎

We compute sentence-level visual and textual representations using parameter-free attention pooling. Given a sequence \(\mathbf{X} = \{\mathbf{x}_1,\dots,\mathbf{x}_L\} \in \mathbb{R}^{L \times d_t}\) with validity mask \(\mathbf{m} \in \{0,1\}^{L}\), we first define a global query \[\mathbf{q} = \frac{\sum_l m_l \mathbf{x}_l}{\sum_l m_l}.\] The attention weights are then \[\alpha_l = \frac{\exp\left(\mathbf{x}_l^\top \mathbf{q}/\sqrt{d_t}\right)}{\sum_{l'}\exp\left(\mathbf{x}_{l'}^\top \mathbf{q}/\sqrt{d_t}\right)}.\] The pooled representation is \[\mathbf{z} = \frac{\sum_l \alpha_l \mathbf{x}_l}{\left\|\sum_l \alpha_l \mathbf{x}_l\right\|_2}. \label{eq:supp95attn95pool}\tag{22}\] This yields normalized sentence-level vectors \(\mathbf{z}^{\mathrm{sign}}\) and \(\mathbf{z}^{\mathrm{text}}\) for the visual and textual modalities.

7.3.2 Memory queue↩︎

The global Earth Mover’s Distance objective is most effective when computed over a sufficiently diverse set of sentence pairs. Since mini-batches are small, we maintain a first-in first-out memory queue that stores recent sentence-level vector pairs. At each iteration, the current detached sentence vectors are appended to the queue. The accumulated vectors \(\{\mathbf{z}^{\mathrm{sign}}_n\}_{n=1}^{N}\) and \(\{\mathbf{z}^{\mathrm{text}}_n\}_{n=1}^{N}\) are then used to compute the global alignment loss.

7.3.3 Learnable orthogonal mapping↩︎

We introduce a learnable transformation matrix \(\mathbf{T} \in \mathbb{R}^{d_t \times d_t}\), initialized near the identity matrix, to map the visual sentence vectors into the textual embedding space: \[\tilde{\mathbf{z}}^{\mathrm{sign}}_n = \frac{\mathbf{z}^{\mathrm{sign}}_n \mathbf{T}}{\|\mathbf{z}^{\mathrm{sign}}_n \mathbf{T}\|_2}. \label{eq:supp95global95transform}\tag{23}\]

7.3.4 Global Earth Mover’s Distance loss↩︎

Using the transformed visual vectors and the original textual vectors, we define the sentence-level cost matrix \[C^{g}_{n,n'} = 1 - \left(\tilde{\mathbf{z}}^{\mathrm{sign}}_n\right)^\top \mathbf{z}^{\mathrm{text}}_{n'}.\] A sentence-level transport plan \(\mathbf{P}\) is then computed with uniform marginals, and the global alignment loss is \[\mathcal{L}_{\mathrm{EMD}} = \sum_{n,n'} P_{n,n'} C^{g}_{n,n'}. \label{eq:supp95emd95loss}\tag{24}\]

7.3.5 Orthogonality constraint and scheduled activation↩︎

To preserve pairwise geometry, we regularize the mapping matrix with \[\mathcal{L}_{\mathrm{orth}} = \|\mathbf{T}^{\top}\mathbf{T} - \mathbf{I}\|_{F}^{2}. \label{eq:supp95orth95loss}\tag{25}\]

The global loss is activated only after local alignment becomes reliable. We use the schedule \[\lambda_g(t)= \begin{cases} 0, & t \le t_{\mathrm{warm}}, \\ \lambda_g^{\max}\cdot \frac{t-t_{\mathrm{warm}}}{t_{\mathrm{ramp}}}, & t_{\mathrm{warm}} < t \le t_{\mathrm{warm}} + t_{\mathrm{ramp}}, \\ \lambda_g^{\max}, & t > t_{\mathrm{warm}} + t_{\mathrm{ramp}}. \end{cases} \label{eq:supp95lambda95schedule}\tag{26}\] The total global objective is \[\mathcal{L}_{\mathrm{global}} = \lambda_g(t)\mathcal{L}_{\mathrm{EMD}} + \beta_{\mathrm{orth}}\mathcal{L}_{\mathrm{orth}}. \label{eq:supp95total95global}\tag{27}\]

7.4 Window-Based Reordering Guided by Vision–Text Alignment↩︎

This section details the window-based reordering module introduced in Sec. 3.5. The motivation for reordering is to resolve a fundamental mismatch between the alignment objective and the sequence generation objective. Specifically, the cross-entropy loss supervises the decoder to generate text in the ground-truth word order, whereas the temporal order of visual features does not necessarily follow that same order. This mismatch is particularly pronounced in sign language, where the semantic order expressed in visual signing may differ from the order of the target spoken-language sentence. As a result, if the visual sequence is kept in its original temporal order, the optimization target imposed by cross-entropy can become inconsistent with the correspondence learned by the vision–text alignment module.

To reduce this inconsistency, we reorder the visual feature sequence according to the alignment result before passing it to the decoder. In this way, the reordered visual sequence is made more compatible with the target text order, so that the alignment-induced correspondence and the autoregressive generation objective become mutually consistent during training.

Let \(\mathbf{S} = \{\mathbf{s}_1, \dots, \mathbf{s}_M\}\) denote the frame-level visual feature sequence after temporal encoding, and let \(\mathbf{A} \in \mathbb{R}^{M \times K}\) denote the local transport plan, where \(K = U + 1\) includes the null token and \(U\) is the number of text tokens.

Rather than performing reordering at the individual-frame level, we operate on short temporal windows. The reason is that, during inference, the target text order is unknown, and a practically useful alignment pattern should therefore associate a continuous local visual segment with the same text token. If isolated frames were reordered independently, the resulting correspondence could become temporally fragmented, making it difficult for the decoder to recover stable lexical units from the visual stream. Window-based reordering instead encourages locally coherent visual evidence to move together, which better matches the requirement of translation at test time.

We partition the \(M\) frame positions into \(N_w = U + 2\) overlapping temporal windows. The additional two windows absorb boundary effects. The boundaries of the \(w\)-th window are defined as \[s_w = \left\lfloor w \cdot \frac{M-1}{N_w} + 0.5 \right\rfloor, \qquad e_w = \left\lfloor (w+1) \cdot \frac{M-1}{N_w} + 0.5 \right\rfloor + 1.\]

For each window \(w\), we accumulate the alignment mass over the corresponding temporal region: \[\boldsymbol{\pi}_w = \sum_{m=s_w}^{e_w-1}\mathbf{A}_{m,:} \in \mathbb{R}^{K}.\] The window is then assigned to the real text token with the largest aggregated alignment mass: \[\phi(w)=\arg\max_{u \in \{1,\dots,U\}} \pi_{w,u}.\] If the dominant mass of a window lies on the null token, that window is excluded from reordering.

For each text token \(u\), we collect the set of windows assigned to that token: \[\Omega_u = \{\, w \mid \phi(w)=u \,\}.\] The reordered visual sequence is then constructed by traversing the text token order from left to right and appending the corresponding windowed visual segments: \[\mathbf{R} = \mathrm{Concat} \Big( \{\mathbf{S}_{s_w:e_w-1} \mid w \in \Omega_1^{\uparrow}\}, \{\mathbf{S}_{s_w:e_w-1} \mid w \in \Omega_2^{\uparrow}\}, \dots, \{\mathbf{S}_{s_w:e_w-1} \mid w \in \Omega_U^{\uparrow}\} \Big), \label{eq:supp95direct95window95reorder}\tag{28}\] where \(\Omega_u^{\uparrow}\) denotes the windows in \(\Omega_u\) sorted by their original temporal order, and \(\mathbf{S}_{s_w:e_w-1}\) denotes the original feature subsequence within window \(w\).

This design preserves local temporal continuity while rearranging visual segments into an order that is more compatible with the aligned text sequence. Consequently, the reordered representation serves as an intermediate form that better reconciles alignment learning with sequence generation.

7.5 Expanded Position-Aligned Contrastive Learning↩︎

After reordering, the visual sequence \(\mathbf{R} = \{\mathbf{r}_1,\dots,\mathbf{r}_U\}\) is positionally aligned with the text token sequence \(\mathbf{E} = \{\mathbf{e}_1,\dots,\mathbf{e}_U\}\). We use this structure to impose a token-level contrastive objective. Learnable projection heads are applied to both modalities: \[\hat{\mathbf{r}}_u = \mathrm{LN} \big( \mathbf{W}^{v}_2 \sigma(\mathbf{W}^{v}_1 \mathbf{r}_u) \big), \qquad \hat{\mathbf{e}}_u = \mathrm{LN} \big( \mathbf{W}^{t}_2 \sigma(\mathbf{W}^{t}_1 \mathbf{e}_u) \big), \label{eq:supp95proj95heads}\tag{29}\] where \(\sigma\) denotes GELU and \(\mathrm{LN}\) denotes layer normalization. After flattening over the batch and valid positions, the InfoNCE loss is \[\mathcal{L}_{\mathrm{contra}} = - \frac{1}{|\mathcal{V}|} \sum_{i \in \mathcal{V}} \log \frac{\exp\left(\mathrm{sim}(\hat{\mathbf{r}}_i,\hat{\mathbf{e}}_i)/\tau_c\right)}{\sum_{j \in \mathcal{V}}\exp\left(\mathrm{sim}(\hat{\mathbf{r}}_i,\hat{\mathbf{e}}_j)/\tau_c\right)}, \label{eq:supp95contrastive95loss}\tag{30}\] where \(\mathcal{V}\) denotes the set of valid token positions. Padding positions are masked out of the denominator. Gradients from this loss are applied only to the visual branch, while the text branch is detached, so that the pre-trained language model embedding space remains stable.

8 Additional Qualitative Experiments and Results↩︎

8.1 Attention Pooling versus Max-Pooling↩︎

We compare the proposed attention-based temporal convolution module (Sec. 7.1) with conventional max-pooling on the Phoenix-2014T [6] test set (Table 3). Max-pooling keeps only the maximum activation at each position and discards gradient information from all other frames in the pooling window, whereas the attention variant computes a weighted aggregation over all frames, letting gradients reach every frame and yielding more expressive temporal representations. The attention module improves BLEU-4 from 26.57 to 28.86 and ROUGE-L from 55.23 to 60.24.

8.2 Comparison with Generic Video–Text Alignment↩︎

To test whether the structured correspondence learned by VTaMo could be obtained with off-the-shelf objectives, we replace our three alignment losses with established generic video–text alignment recipes inside the same pipeline (Table 4). A CLIP-ViT-only variant with no alignment reaches only 10.45 BLEU-4 on the Phoenix-2014T [6] test set, and CLIP4Clip [29], VideoCLIP [30], and X-CLIP [31] stay at 9.47–11.13, all far below our 28.86. These recipes align a whole clip to a caption and assume a near-monotonic relation, so they do not resolve the non-monotonic, partial, token-level correspondence that sign language requires.

8.3 Qualitative Analysis of Alignment and Feature Geometry↩︎

We further provide qualitative analysis to examine the behavior of the proposed alignment framework beyond the robustness study. Fig. 5 visualizes the learned transport plans and the resulting feature organization.

The left panel shows representative transport plans between temporal visual positions and text tokens. The alignment maps exhibit clear structured correspondences, indicating that contiguous temporal regions are consistently associated with semantically relevant target tokens. The right panel shows a t-SNE visualization of the resulting visual features. Semantically related samples form compact groups, which suggests that the alignment objectives induce a more organized cross-modal representation space.

8.4 Representative Translation Examples↩︎

Table 5 presents representative translation examples on the How2Sign [5] test set. Compared with SpaMo [1], our method more faithfully preserves key content words and overall sentence meaning. The advantage is especially clear in examples that require precise lexical grounding, attribute preservation, and procedural understanding.

References↩︎

---

1. Corresponding author.↩︎