Keywords: Sign language generation, diffusion models, text-to-video synthesis, video generation, accessibility, spatio-temporal attention, transformer.

1 Introduction↩︎

Accessible sign language generation constitutes a high-impact yet technically challenging problem at the intersection of computer vision and natural language processing. The World Health Organization estimates that more than 430 million people globally require rehabilitation for disabling hearing loss, with projections exceeding 700 million by 2050 [1]. In numerous settings, communication access depends on professional interpretation services—the United States Bureau of Labor Statistics reports 75,300 interpreter and translator positions in 2024, with approximately 6,900 annual openings projected over the next decade [2]. These figures underscore the need for scalable sign language generation systems that can complement human interpretation by reducing friction in routine, text-mediated communication.

From a modeling perspective, automatic Sign Language Production (SLP) seeks to map spoken or written language into continuous signing sequences, whether represented as pose trajectories or video. Prior work has frequently introduced intermediate representations — such as glosses or skeletal poses — to stabilize long-horizon generation. Progressive Transformers [3] translate text into continuous three-dimensional skeleton pose sequences, demonstrating the importance of managing temporal drift while adopting sign-specific inductive biases. In parallel, large-scale multimodal datasets such as How2Sign [4] have enabled learning from realistic sign language videos, providing both RGB frames and aligned English translations.

Diffusion models have emerged as a powerful foundation for high-fidelity conditional generation across multiple domains [5], [6]. Video diffusion extends these ideas to temporally coherent synthesis; however, this extension incurs substantial computational and memory costs arising from spatiotemporal modeling [7]. Transformer-based backbones for diffusion (e.g., DiT [8]) improve generation quality through scaling, yet naïve attention over all video tokens remains prohibitive for practical sign language video generation.

At the same time, sign-language computing has advanced rapidly in recognition and translation. Human-keypoint-based translation systems [9], end-to-end recognition and translation transformers [10], and broader surveys of text-to-sign and assistive communication systems [11], [12] provide important context for the place of generation in the pipeline. Recent recognition work has also explored depth and 3D sensing, Fourier-domain modeling, and compact dynamic feature representations [13]–[17]. These studies show strong progress in sign understanding. Still, they do not solve the inverse problem of producing visually plausible signing from open-ended text prompts. Direct generative studies remain relatively sparse. Recent GAN-based efforts include Indian Sign Language video synthesis from text [18] and an ASL study combining GAN generators with BERT embeddings and Sora-based data augmentation [19]; however, these papers use different datasets and evaluation protocols, do not establish directly stronger results on a shared benchmark with How2Sign, and do not provide a broader directly comparable evaluation suite for sign generation. Kumar et al. describe a preliminary quantitative evaluation without a detailed comparable metric table, while Sreemathy et al. report a narrow SSIM/FID/MSE-style set rather than broader semantic or linguistic measures [18], [19]. In addition, at the time of writing, we did not identify public training code, open-source reproduction pipelines, or open-access model checkpoints for either paper. More recent work has begun to shift toward diffusion-based sign video generation. SignGen adopts latent diffusion for end-to-end signer video synthesis [20], mirroring the broader generative-vision transition in which diffusion models have surpassed GANs on standard synthesis benchmarks [21]. Earlier text-to-sign systems were often rule-driven or avatar-oriented [22], which made them easier to control but limited visual naturalism. Our work is aimed at this production setting: generating short signer video clips directly from text while staying within a single-GPU training budget.

We propose Text2Sign, a single-GPU text-to-sign language video diffusion baseline that directly addresses the spatio-temporal attention bottleneck. Our principal contributions are as follows:

1. Factorized spatio-temporal attention: We decompose full 3D attention within DiT-style blocks into separate spatial and temporal components, reducing memory consumption while preserving temporal coherence for sign language motion.

2. Frozen pretrained text conditioning: We employ a frozen CLIP text encoder that avoids additional text-encoder training overhead and achieves lower validation loss than a jointly trained custom encoder in the present short-budget comparison.

3. Comprehensive ablation study: We systematically evaluate three architectural axes—DiT block presence, text encoder strategy, and attention factorization—quantifying their individual contributions to generation quality and computational efficiency.

4. Single-GPU baseline design: The complete architecture generates temporally consistent sign language video on a single NVIDIA L4 GPU (24 GB), demonstrating feasibility for single-GPU research workflows rather than real-time deployment.

Our experimental results on the How2Sign dataset highlight useful architectural trade-offs for single-GPU sign-language video generation under constrained resources, while also making clear that temporally smooth motion does not yet imply reliable linguistic fidelity.

2 Related Work↩︎

2.1 Video Generative Methods↩︎

Diffusion models, particularly latent variants, now form the foundation of state-of-the-art text-driven image and video synthesis. Denoising Diffusion Probabilistic Models (DDPMs) established the denoising framework [5], while Latent Diffusion Models (LDMs) demonstrated that operating in compressed latent spaces substantially reduces computational requirements while preserving visual quality [6]. Dhariwal and Nichol further showed that diffusion models can surpass strong GAN baselines on standard image-synthesis benchmarks, reporting ImageNet FID 4.59 versus 6.95 for BigGAN-deep at \(256\times256\) and 7.72 versus 8.43 at \(512\times512\), while maintaining better distribution coverage [21]. The improved noise scheduling strategies introduced by Nichol and Dhariwal [23] further enhanced generation fidelity. Building on these foundations, LaVie [24] cascades latent diffusion with temporal interpolation and super-resolution to generate high-quality video clips, and Stable Video Diffusion [25] scales the approach by training in stages on large-scale datasets.

Despite their success in general scene synthesis, these models encounter significant limitations when applied to sign language generation. First, they require massive video-text corpora and considerable computational resources to capture the subtle motion patterns characteristic of signing. Second, their objective functions and sampling procedures favor broad perceptual realism over fine-grained articulatory details critical for sign language intelligibility. Third, deployment in accessibility settings — real-time captioning, on-device inference, or resource-constrained cloud APIs — requires architectures optimized for efficiency. Recent compression techniques such as TinyFusion-style pruning [26] and knowledge distillation approaches [27], [28] indicate promising directions for reducing resource requirements, while comprehensive surveys [29], [30] document the rapid expansion of this field.

The computational bottleneck in video diffusion arises primarily from attention mechanisms that scale quadratically with the number of spatiotemporal tokens. Several strategies have been proposed to mitigate this cost: factorized attention that decomposes spatial and temporal dimensions [7], structured pruning of diffusion transformer layers [26], and model compression through quantization and distillation [31]. Our work builds on the factorized attention paradigm and integrates it into a DiT-enhanced 3D UNet backbone, specifically tailored to the temporal dynamics of sign language motion.

2.2 Recent Works on Text-to-Sign↩︎

A growing body of work addresses the conversion of written text into sign-oriented visual output, though most existing systems do not produce photorealistic signer video. VisualSign [32] proposes an accessibility pipeline that processes subtitles or on-screen text to generate synchronized sign language overlays, prioritizing workflow integration for educational and broadcasting contexts by delivering via avatar-based rendering rather than full generative synthesis. Commercial platforms such as SignStream [33] and Bitmovin’s sign language integration [34] similarly focus on low-latency APIs and avatar animation using notation systems (e.g., HamNoSys, SiGML), offering scalable overlay solutions within video players. While these systems provide substantial practical value through streaming infrastructure and synchronization capabilities, they generally circumvent the full spatiotemporal complexity inherent in modeling real human signers, trading naturalism and expressive nuance for scalability.

More direct sign-video generation studies include GAN-based pipelines. Sreemathy et al. generate Indian Sign Language clips from text with conditional GANs [18], while Kumar et al. explore ASL video generation using GANs, BERT text features, and Sora-augmented training data [19]. These papers are useful contextual references, but they remain difficult to benchmark directly against our setting: the sign languages, datasets, clip construction procedures, and evaluation protocols differ; neither establishes a directly stronger metric profile on a shared How2Sign benchmark; and neither provides a richer directly comparable metric suite for sign generation, since Kumar et al. report only preliminary quantitative results while Sreemathy et al. report a narrow SSIM/FID/MSE-style set. Moreover, at the time of writing, we did not identify public training code or open-access model checkpoints for either work, which further limits exact reproduction. More recently, SignGen uses latent diffusion for end-to-end sign video synthesis and reports significant improvements over prior state-of-the-art approaches in semantic consistency, naturalness, and expressiveness across five public sign datasets [20]. Taken together with the broader diffusion literature [21], these results suggest that diffusion is a promising avenue for improving high-fidelity sign-video synthesis.

3 Methodology↩︎

This section describes our text-conditioned diffusion framework for sign language video generation. The system accepts a text prompt \(y\) and produces a sequence of RGB video frames \(\mathbf{x}_0 \in \mathbb{R}^{C \times T \times H \times W}\), where \(C\), \(T\), \(H\), and \(W\) denote color channels, temporal frames, and spatial dimensions, respectively. Our design integrates a frozen CLIP-based text encoder with a 3D UNet architecture enhanced by DiT-style transformer blocks, specifically tailored for the spatio-temporal complexity of American Sign Language (ASL) as represented in the How2Sign dataset [4].

3.1 Dataset↩︎

We evaluate on short clips extracted from the How2Sign dataset [4], a large-scale multimodal collection of ASL videos derived from instructional content, which provides parallel RGB videos and English translations. Videos are resized to \(64 \times 64\) resolution and sampled at \(T = 32\) frames per clip. The processed subset is partitioned using a signer-disjoint 90/10 protocol, with the identity metadata available after preprocessing, so that no metadata-defined identity group appears in both the training and validation sets. This protocol reduces the risk of memorizing signer appearance rather than learning text-conditioned motion. The resulting split statistics are summarized in Table 1.

The processed subset contains 231 metadata-derived identity groups in the training partition and 24 in the validation partition, with zero overlap by construction. These counts refer to effective identity labels in the processed clip subset rather than to the total number of unique human signers in the original public How2Sign release. Because the data remain limited and identity representation is uneven, this protocol should be regarded as a stronger yet imperfect estimate of generalization.

3.2 Diffusion Framework↩︎

Let \(\mathbf{x}_0 \sim q(\mathbf{x}_0)\) denote a clean video sampled from the data distribution. Following the DDPM framework [5], we define a forward diffusion process that progressively corrupts \(\mathbf{x}_0\) with Gaussian noise over discrete timesteps \(t \in \{0, \ldots, T_\mathrm{diff}-1\}\): \[q(\mathbf{x}_t \mid \mathbf{x}_{t-1}) = \mathcal{N}\bigl(\mathbf{x}_t;\, \sqrt{1-\beta_t}\,\mathbf{x}_{t-1},\, \beta_t \mathbf{I}\bigr),\] where \(\{\beta_t\}\) is a variance schedule. Defining \(\alpha_t := 1 - \beta_t\) and \(\bar{\alpha}_t := \prod_{s=0}^{t} \alpha_s\), the marginal at any timestep admits a closed-form expression: \[q(\mathbf{x}_t \mid \mathbf{x}_0) = \mathcal{N}\bigl(\mathbf{x}_t;\, \sqrt{\bar{\alpha}_t}\,\mathbf{x}_0,\, (1-\bar{\alpha}_t)\mathbf{I}\bigr),\] enabling direct sampling via \(\mathbf{x}_t = \sqrt{\bar{\alpha}_t}\,\mathbf{x}_0 + \sqrt{1-\bar{\alpha}_t}\,\boldsymbol{\epsilon}\) with \(\boldsymbol{\epsilon} \sim \mathcal{N}(\mathbf{0}, \mathbf{I})\).

3.2.0.1 Noise schedule.

We adopt the cosine schedule [23], which prevents premature signal destruction at low timesteps and ensures meaningful feature learning across all noise levels (Fig. 1): \[\bar{\alpha}_t = \frac{\cos^2\!\bigl(\tfrac{t/T_\mathrm{diff}+s}{1+s} \cdot \tfrac{\pi}{2}\bigr)}{\cos^2\!\bigl(\tfrac{s}{1+s} \cdot \tfrac{\pi}{2}\bigr)},\quad \beta_t = 1 - \frac{\bar{\alpha}_t}{\bar{\alpha}_{t-1}},\] with offset \(s\) and an upper bound on \(\beta_t\).

3.2.0.2 Reverse process.

The generative model approximates the reverse chain \(p_\theta(\mathbf{x}_{t-1} \mid \mathbf{x}_t, y)\), parameterized through a noise-prediction network \(\boldsymbol{\epsilon}_\theta\): \[\boldsymbol{\mu}_\theta(\mathbf{x}_t, t, y) = \frac{1}{\sqrt{\alpha_t}} \left(\mathbf{x}_t - \frac{\beta_t}{\sqrt{1-\bar{\alpha}_t}}\,\boldsymbol{\epsilon}_\theta(\mathbf{x}_t, t, y)\right).\]

3.3 Text-to-Video Architecture↩︎

Our architecture comprises two primary components: a text encoder that produces conditioning representations and a 3D UNet backbone enhanced with DiT-style transformer blocks. Fig. 3 provides an overview of the complete pipeline.

3.3.1 Text Encoder↩︎

We employ a frozen CLIP text encoder (ViT-B/32) [35] that maps input prompts to a fixed-dimensional representation \(\mathbf{z}_y \in \mathbb{R}^{L \times d_\mathrm{text}}\). The frozen encoder provides several advantages: (i) robust semantic representations learned from large-scale vision-language pretraining, (ii) elimination of text encoder gradient computation during training, and (iii) consistent conditioning quality independent of the diffusion training dynamics.

As demonstrated in our ablation study (Section 4.2), this frozen-pretrained approach achieves lower validation loss than the jointly trained custom transformer encoder in the present short-budget comparison while reducing the number of trainable parameters by 7.7%.

3.3.2 3D UNet with DiT-Style Transformer Blocks↩︎

The backbone is a UNet3D architecture that combines 3D convolutions with DiT-style transformer blocks. The input video tensor \(\mathbf{x}_t \in \mathbb{R}^{C \times T \times H \times W}\) is first mapped through a \(3 \times 3 \times 3\) convolutional stem to \(c_0 = 96\) base channels.

3.3.2.1 3D convolutions.

Unlike frame-wise 2D convolutions, 3D convolutions operate across the full spatio-temporal volume, capturing local motion patterns (e.g., hand trajectories through space) at the earliest network layers. This provides a strong foundation for temporal consistency in the generated output.

3.3.2.2 Factorized spatio-temporal attention.

The core innovation of our transformer blocks is a factorized attention mechanism that decomposes the full 3D attention computation into sequential spatial and temporal components:

• Spatial attention: Operates on \((H, W)\) tokens within each frame independently, modeling intra-frame dependencies for accurate hand shape and facial expression synthesis. Complexity: \(O(T \cdot (HW)^2)\).

• Temporal attention: Operates along the time axis \(T\) at each spatial location, explicitly modeling motion dynamics and inter-frame coherence. Complexity: \(O(HW \cdot T^2)\).

This factorization reduces total attention complexity from \(O((THW)^2)\) for full 3D attention to \(O(T(HW)^2 + HW \cdot T^2)\), yielding substantial memory savings. Fig. 2 illustrates the factorization schematically.

3.3.2.3 Adaptive Layer Normalization (AdaLN).

Following DiT [8], we inject timestep information through Adaptive Layer Normalization, which regresses scale (\(\gamma\)) and shift (\(\beta\)) parameters from the timestep embedding \(\mathbf{e}_t\): \[\mathrm{AdaLN}(x, t) = \gamma(\mathbf{e}_t) \cdot \mathrm{LayerNorm}(x) + \beta(\mathbf{e}_t).\]

3.3.2.4 Cross-attention conditioning.

text features \(\mathbf{z}_y\) are integrated via cross-attention layers within the transformer blocks, enabling the input prompt to guide the generation of specific sign gestures at the feature level.

3.3.3 Network Configuration↩︎

Table 2 summarizes the configuration parameters. The encoder consists of three resolution stages with channel progression \((96, 192, 384)\) and DiT blocks at spatial resolutions of \(16 \times 16\) and \(8 \times 8\). The decoder mirrors the encoder with upsampling and skip connections.

3.4 Training Objective and Sampling↩︎

3.4.1 Training Objective↩︎

The training objective follows the standard DDPM noise-prediction formulation [5]: \[\mathcal{L}(\theta) = \mathbb{E}_{\mathbf{x}_0, y, t, \boldsymbol{\epsilon}} \bigl[\lVert \boldsymbol{\epsilon} - \boldsymbol{\epsilon}_\theta(\mathbf{x}_t, t, y) \rVert_2^2\bigr]. \label{eq:loss}\tag{1}\] Training uses AdamW with a learning rate of \(5 \times 10^{-5}\), weight decay of \(10^{-2}\), a cosine schedule with a linear warmup, gradient clipping with a norm of 0.5, and AMP training. An exponential moving average (EMA) of model weights (\(\tau = 0.9999\)) is maintained for inference.

3.4.2 DDIM Sampling↩︎

Inference uses DDIM [36], which enables accelerated sampling with 50 steps: \[\mathbf{x}_{t-\Delta t} = \sqrt{\bar{\alpha}_{t-\Delta t}} \, \hat{\mathbf{x}}_0 + \sqrt{1-\bar{\alpha}_{t-\Delta t}} \, \hat{\boldsymbol{\epsilon}},\] where \(\hat{\mathbf{x}}_0 = (\mathbf{x}_t - \sqrt{1-\bar{\alpha}_t}\,\hat{\boldsymbol{\epsilon}})/\sqrt{\bar{\alpha}_t}\). Classifier-free guidance (CFG) strengthens text alignment: \[\hat{\boldsymbol{\epsilon}}_\mathrm{cfg} = \hat{\boldsymbol{\epsilon}}(y) + w \bigl[\hat{\boldsymbol{\epsilon}}(y) - \hat{\boldsymbol{\epsilon}}(\varnothing)\bigr].\]

4 Experiments↩︎

4.1 Experimental Setup↩︎

4.1.0.1 Implementation.

The model is implemented in PyTorch [37] and trained on a single NVIDIA L4 GPU (24 GB VRAM). Training uses automatic mixed-precision (AMP/bfloat16, when supported), gradient checkpointing, and gradient accumulation to manage memory usage. The short ablation variants use a fixed 3-epoch, 50-step-per-epoch budget to isolate architectural effects, while the longer signer-disjoint runs use 32-frame clips at \(64\times64\) resolution with an effective batch size of 16. Additional checkpoint analyses are conducted on a compact evaluation slice, along with short, bounded fine-tuning studies, so that the reported follow-up observations remain reproducible within the same single-GPU budget. The code associated with this work is publicly available at https://github.com/xiaruize0911/text2sign.

4.1.0.2 Evaluation metrics.

We employ the following quantitative measures:

• Video feature distance: a Fréchet-style metric computed from pretrained video features. We also report our earlier pixel-space proxy for continuity with prior internal runs, but we do not interpret it as a metric of linguistic correctness.

• Motion Magnitude: Mean absolute frame-to-frame difference measuring motion intensity.

• Spatial Gradient: Mean spatial gradient magnitude of frame differences, capturing motion complexity.

• Temporal Consistency: Normalized frame-to-frame variance where values closer to 1.0 indicate smoother transitions.

• Validation Loss: MSE on held-out data measuring denoising accuracy.

These metrics should be interpreted cautiously. Distributional video metrics and motion heuristics can indicate realism and temporal smoothness, but they do not directly measure linguistic acceptability, intelligibility, or signer preference. Accordingly, we treat them as supporting evidence rather than a substitute for expert or community-centered evaluation.

Formally, we define motion magnitude and spatial gradient from a frame-difference optical flow surrogate \(\mathbf{f}_t = \mathbf{x}_{t+1} - \mathbf{x}_t\): \[\begin{align} m_\mathrm{mag} &= \frac{1}{BCTHW}\sum_{b,c,t,h,w} |\mathbf{f}^{(b,c)}_t(h,w)|, \\ m_\mathrm{grad} &= \frac{1}{BCTHW}\sum_{b,c,t,h,w} \|\nabla_{h,w} \mathbf{f}^{(b,c)}_t(h,w)\|_2. \end{align}\] Temporal consistency is measured as: \[c_\mathrm{temp} = 1 - \frac{1}{BCT(HW)}\sum_{b,c,t,h,w} \frac{(\mathbf{x}_{t+1}-\mathbf{x}_t)^2}{\mathrm{Var}_t(\mathbf{x}) + \varepsilon}.\]

4.2 Ablation Study↩︎

To isolate each architectural component’s contribution, we conduct a systematic ablation along three axes. Four model variants are trained under strictly identical conditions (3 epochs, 50 gradient steps per epoch, identical random seed) on the same signer-disjoint data partition:

1. Ours (Full): Complete architecture with DiT-style factorized spatio-temporal transformer blocks and a frozen CLIP text encoder. Total parameters: 591.8M (528.6M UNet trainable + 63.2M frozen CLIP).

2. No DiT: Identical 3D UNet backbone with all transformer blocks disabled, relying exclusively on 3D convolutions and residual blocks. Total parameters: 498.7M (435.5M UNet trainable + 63.2M frozen CLIP). This variant isolates the contribution of attention-based feature mixing.

3. Custom TextEnc: Full DiT architecture with a jointly trained 6-layer, 8-head transformer text encoder (\(d_{\mathrm{model}}=512\)) replacing the frozen CLIP encoder. Total parameters: 572.8M, all trainable. This variant tests whether domain-specific text encoding improves conditioning.

4. Full 3D Attention: DiT blocks with non-factorized joint attention over all \(T \times H \times W\) tokens simultaneously, replacing the factorized spatial–temporal decomposition. Total parameters: 544.3M (481.1M UNet trainable + 63.2M frozen CLIP). This variant compares the cost of full attention with that of our factorized design.

4.2.1 Training Dynamics↩︎

Table 4 presents the comprehensive results across all architectural variants, while Fig. 7–8 illustrate per-step and per-epoch loss trajectories. Table 3 provides the per-epoch convergence profile for each variant.

Effect of DiT blocks. Comparing Ours (Full) against No DiT reveals that DiT-style transformer blocks contribute a 19.5% reduction in final validation loss (0.0648 vs.). Both variants start from comparable first-epoch validation losses (0.1176 and 0.1049, respectively), yet the transformer-augmented model converges to a markedly lower minimum by epoch 3. The convolutional-only variant converges to a higher loss despite a 17.6% smaller parameter count (435.5M vs.528.6M trainable parameters), confirming that self-attention is essential for capturing global spatio-temporal dependencies — particularly the coordinated hand trajectories, facial expressions, and body orientations that co-occur in natural signing.

Effect of text encoder strategy. The frozen CLIP encoder achieves lower validation loss (0.0648) than the jointly trained custom encoder (0.0728), an 11.0% relative improvement under the present short-budget comparison. Notably, the custom encoder exhibits a higher initial validation loss at epoch 1 (0.1217 vs.), suggesting that the randomly initialized text encoder impedes early conditioning quality and slows convergence. This is consistent with findings in text-to-image generation [6]: pretrained vision-language representations can stabilize optimization more effectively than encoders trained only on limited domain-specific data. At the same time, the later conditioning-control analysis on the main 100-epoch checkpoint shows only weak separation between frozen-text and random-text settings on coarse pixel-space probes, so the present result should be interpreted as a validation-loss advantage in this ablation rather than as evidence that robust semantic prompt control has already been solved.

Effect of attention factorization. Full 3D Attention achieves a validation loss of 0.0664, which is 2.5% higher than our factorized approach (0.0648). In this low-resolution, short-clip setting, factorized spatial–temporal attention therefore matches or slightly improves upon joint attention on validation loss, likely because the factored design provides a beneficial inductive bias: spatial coherence within each frame is established before temporal dynamics are modeled across frames. Furthermore, the Full 3D Attention variant exhibits higher final training loss (0.1112 vs.), suggesting that the joint attention parameterization is more difficult to optimize under limited training budgets. We do not claim that this ordering is universal outside the present setting.

3.5pt

4.2.2 Computational Efficiency↩︎

Fig. 9 presents a comparison of efficiency across all four variants. The No DiT baseline provides the fastest training throughput (1,550 ms/step) and lowest peak GPU memory consumption (8.5 GB), establishing a lower bound on the computational cost when transformer attention is absent. Our full model requires 19.0 GB of peak memory at 1,792 ms per step — a 15.6% increase in wall-clock time relative to the convolutional baseline. Yet, it remains well within the 24 GB capacity of a single NVIDIA L4 GPU.

Inference latency, measured for a single 32-frame clip using 15 DDIM sampling steps, ranges from 2,629 ms (No DiT) to 2,876 ms (Custom TextEnc). The attention overhead adds approximately 8.8% to the inference latency relative to the convolutional baseline. Notably, the Full 3D Attention variant (2,707 ms) is faster than our factorized variant (2,860 ms) during inference despite performing joint attention, because the non-factorized design executes a single attention kernel per block rather than two sequential ones; however, this advantage disappears at higher resolutions where the quadratic scaling of joint attention dominates.

The Custom TextEnc variant uses 10.8 GB peak memory — less than our full model — because jointly training the smaller text encoder (44.2M vs.63.2M) reduces the number of activations stored for backward passes. However, this memory saving comes at the expense of generation quality (Table 4). Fig. 10 visualizes the parameter composition of each variant.

Table 5 summarizes the efficiency–quality trade-offs. The data indicate that our factorized DiT design occupies a favorable position on the Pareto frontier, achieving the lowest validation loss. At the same time, its memory and latency costs remain practical for single-GPU deployment. Fig. 5 shows this trade-off visually, with our model occupying a favorable position despite its higher memory footprint. Fig. 6 compares inference latency across variants, confirming that attention overhead contributes only modest additional latency.

To make the scaling argument explicit, Table 6 compares the token-level asymptotic cost of full 3D attention with that of our factorized design at representative resolutions. Let \(N = T \cdot H \cdot W\) be the number of spatio-temporal tokens. Full attention scales as \(O(N^2)=O((THW)^2)\), whereas the factorized design scales as \(O(T(HW)^2 + HW\cdot T^2)\). At small resolutions, both are manageable, but the gap widens rapidly as spatial size increases.

3pt

4.2.3 Generation Quality↩︎

Table 8 reports generation quality metrics computed from 4 synthesized video clips per variant using 15-step DDIM sampling.

Our full model produces the most controlled motion dynamics, with the lowest motion magnitude (0.0612) and spatial gradient (0.3261), indicating generation of smooth, purposeful movements characteristic of natural signing rather than random noise or jitter. Fig. 11 presents these metrics side by side. The FVD-proxy metric (Table 8) reveals a nuanced picture: Full 3D Attention achieves the lowest distributional distance to real video statistics (7,022.1), with our factorized model close behind (7,083.2, a difference of only 0.9%). The substantially higher FVD-proxy for Custom TextEnc (9,515.6 — 34.3% worse than our model) underscores the critical importance of pretrained text representations for generating videos whose statistical distribution aligns with real signing data.

The temporal consistency metric warrants careful interpretation. Custom TextEnc achieves the highest (least negative) temporal consistency (\(-\)​0.2798), suggesting superior frame-to-frame smoothness. However, this metric must be considered alongside motion magnitude: the Custom TextEnc variant generates video with moderate motion (0.0670), and its high consistency may partly reflect under-expression of dynamic signing movements rather than genuine temporal coherence. Our full model strikes a favorable balance between motion expressiveness and temporal stability.

4.3 Qualitative Results↩︎

Table 7 places Text2Sign in the broader landscape of sign-language production systems. Prior work has frequently focused on avatar animation, pose synthesis, or deployment infrastructure rather than on direct RGB video generation. Recent GAN-based sign-video studies [18], [19] were not used as direct baselines because they were evaluated on different languages, datasets, and protocols, do not report a broader directly comparable sign-generation metric suite, and, at the time of writing, do not provide public training code or open-access checkpoints for exact reproduction. The present system, therefore, complements rather than supersedes these approaches: it offers an RGB-video diffusion research baseline, but not the controllability, linguistic reliability, or real-time guarantees associated with rule-based avatar pipelines.

Fig. 12 shows eight evenly spaced frames from the qualitatively strongest post-revision checkpoint, generated with 8-step DDIM and classifier-free guidance scale 5.0 for the prompt “Hello.” Relative to the earlier baseline checkpoint, the revised post-revision sample is visibly cleaner, more stable, and better organized at the whole-body level. At the same time, it still illustrates the central qualitative limitation of the model: the frames remain low resolution and do not recover the fine-grained hand articulation and facial detail needed for strong sign-linguistic claims. Fig. 13 extends this qualitative inspection to two prompts (“Hello” and “Thank you”) from the same checkpoint. Across both rows, the model produces coarse upper-body motion patterns and some prompt-dependent pose variation, with a clearer global structure than the earlier baseline. Yet, the frames remain too coarse to support strong claims regarding sign-linguistic correctness. Accordingly, these images are best interpreted as illustrative evidence of improved qualitative behavior under the revised checkpoint rather than as evidence of solved high-fidelity sign production.

5 Discussion↩︎

5.1 Key Findings↩︎

The ablation study yields several insights with implications for the design of efficient video diffusion architectures, particularly in resource-constrained settings. Fig. 14 provides a holistic multi-metric comparison across all variants, while Fig. 15 summarizes the three principal improvement margins.

DiT blocks are essential for sign language fidelity. The 19.5% validation loss improvement (0.0648 vs.) confirms that global attention mechanisms capture dependencies that 3D convolutions alone cannot model. Sign language communication involves coordinated, simultaneous movement across multiple articulators — hands, face, torso, and gaze — whose relationships extend beyond local receptive fields. The per-epoch convergence data (Table 3) further show that this advantage is not a transient effect: the No DiT variant consistently exhibits higher loss across all training epochs, indicating a fundamental representational limitation rather than a convergence-speed issue.

Frozen text conditioning lowers loss in the present ablation, but prompt control remains weak. Despite 7.7% fewer trainable parameters (528.6M vs.572.8M), the frozen CLIP encoder achieves 11.0% lower validation loss (0.0648 vs.) and 25.6% lower FVD-proxy (7,083.2 vs.515.6) relative to the Custom TextEnc variant in the short-budget comparison. This result is consistent with the broader observation from text-to-image generation [6] that large-scale pretrained language–vision features can stabilize optimization more effectively than encoders trained from scratch on limited data. However, the latter conditioning-control analysis reveals only a weak quantitative separation between the frozen-text and random-text conditions in coarse pixel-space probes. Accordingly, the current evidence supports a narrower claim: frozen CLIP helps the present optimization setting, but the main 100-epoch checkpoint still does not exhibit convincingly strong prompt-specific semantic control.

Factorized attention is competitive in the present regime. The marginal quality difference between factorized (0.0648) and full 3D attention (0.0664)—a 2.5% gap on validation loss in favor of factorization—contrasts with the much better theoretical scaling of the factored design. In the present low-resolution, short-clip regime, the factorized model therefore remains competitive with full 3D attention while offering a more scalable formulation. We interpret this as evidence that the factored inductive bias is effective here, not as proof that factorization will dominate across all resolutions, clip lengths, or sampling settings.

Efficiency–quality trade-offs. The efficiency–quality summary (Table 5) reveals that no single variant dominates all axes. The No DiT baseline is the most resource-efficient but suffers from substantially degraded quality. Full 3D attention offers the best FVD-proxy at moderate cost, but cannot scale to longer sequences without quadratic memory explosion. Our factorized design strikes the most favorable balance for practical single-GPU experimentation: it achieves the best validation loss while its memory footprint remains within a 24 GB budget.

5.2 Analysis of Training Dynamics↩︎

All four variants exhibit rapid initial convergence, with training loss decreasing from approximately 0.35–0.37 at epoch 1 to below 0.12 by the final epoch. However, the generalization gap—the difference between the training and validation losses—varies across variants (Fig. 4). Our full model exhibits the smallest generalization gap (0.0916 \(-\) 0.0648 \(=\) 0.0268), whereas the Full 3D Attention variant shows the largest (0.1112 \(-\) 0.0664 \(=\) 0.0448), suggesting that the joint attention parameterization is more prone to overfitting under limited-data conditions. This observation has practical implications: factorized attention may be preferable not only for computational efficiency but also for regularization when training on small-to-medium sign language corpora.

5.3 Extended 100-Epoch Run↩︎

To complement the short (3-epoch) ablation study, we trained the full model for 100 epochs on the same data configuration (batch size \(4\), 32 frames, \(64\times64\) resolution). The run lasted 76.7 hours (22,898 optimization steps). Table 9 summarizes the key outcomes; Fig. 16 shows the per-epoch loss curves, and Fig. 17 plots the train–validation gap. The best validation loss (0.00578) occurred at epoch 84, and the final validation loss at epoch 100 was 0.00768.

Across the 100 epochs, the training loss fell from \(0.141\) (epoch 2) to \(0.00748\) (epoch 100), while the validation loss reached its minimum at epoch 84 and rose modestly to \(0.00768\) by epoch 100. The generalization gap tightened to within \(2\times 10^{-4}\) in the final epochs, indicating little overfitting under this configuration. Overall, the long run shows that the architecture remains stable over extended training and that most quality gains accrue within the first 80 epochs, with diminishing returns thereafter.

5.4 Checkpoint Evaluation and Fine-Tuning Analysis↩︎

To avoid mixing evidence from different training stages, we distinguish three model states in this subsection. Unless stated otherwise, all quantitative results here—the compact validation metrics, runtime, conditioning controls, and held-out conditional-loss audit—are computed from the main signer-disjoint checkpoint obtained from the 100-epoch run in Section V-C. Two later continuation experiments are discussed only as follow-up observations: a short fine-tuning attempt from that checkpoint and a later 10-epoch post-revision run with updated conditioning modules. These auxiliary runs are not used to change the main quantitative conclusion.

For the main 100-epoch checkpoint, the compact evaluation slice produced a mean SSIM of \(0.2403\pm0.0238\) and a mean PSNR of \(15.11\pm0.42\) dB. These pixel-space values are included only as coarse reconstruction proxies; as discussed earlier, they should not be interpreted as sign-language correctness metrics because signer appearance differs between real and generated clips. More informative for motion stability, the same checkpoint yielded a temporal-consistency score of \(1.0000\pm0.0000\) and a mean frame-to-frame motion magnitude of \(0.0987\pm0.0072\). In practice, these numbers indicate very smooth clips, but smoothness alone was insufficient to produce convincing hand articulation or linguistically reliable signing in visual inspection.

The same main checkpoint was also benchmarked under an 8-step DDIM setting with a classifier-free guidance scale of 5.0, which yielded the most satisfactory qualitative trade-off among the sampled configurations. Averaged over five warm-start runs on one NVIDIA L4, this configuration required 12.60 s per 32-frame clip (2.54 frames/s) with 3.12 GB peak inference memory. Although this setting is materially lighter than high-step sampling, it remains far from interactive deployment speed and does not, by itself, resolve the qualitative fidelity limitations.

We also ran two exploratory continuation studies from the main checkpoint. A brief attempt with partial CLIP unfreezing was unstable after the trainable parameter set changed and was therefore not used to support any quantitative claim. A second short continuation with CLIP kept frozen was numerically stable, but it did not produce a clear qualitative improvement over the starting checkpoint. These exploratory continuations, therefore, served only as negative evidence: simple low-learning-rate continuation or partial CLIP unfreezing was insufficient to close the remaining fidelity gap.

The quantitative audits on the main checkpoint all lead to the same conclusion, but they do not contribute equally. Table 11 should be read only as a diagnostic negative-control analysis, not as evidence of semantic faithfulness. It compares four prompt-conditioning variants — frozen CLIP, no text, random text, and CLIP fine-tuned in the last two layers — under the same 8-step, CFG\(=5.0\) setting, using coarse pixel-space proxies. The no-text control produced the largest same-prompt drift and the highest motion magnitude, indicating noisier and less stable generations when text information was removed entirely. However, the random-text and partially fine-tuned CLIP variants were nearly indistinguishable from the frozen-CLIP baseline on these proxies. Accordingly, Table 11 supports only the narrow claim that removing text is destabilizing; it does not support a stronger claim of prompt-specific semantic separation.

Table 10 is therefore the strongest test in this subsection because it evaluates conditional denoising loss directly on held-out real validation clips rather than relying on coarse pixel-space distances. For 16 unique prompts of at most five words, removing text increased late-timestep denoising loss from 0.9875 to 0.9891, with a mean penalty of +0.00161 and a bootstrap 95% confidence interval of [0.00144, 0.00176]. In contrast, shuffled prompts remained nearly tied with the intended prompt (mean difference +0.00004; confidence interval spanning zero), and the four-way prompt-ranking test stayed close to chance. This is the clearest evidence that the checkpoint leverages text during denoising, but it still does not discriminate strongly among specific prompts.

The latter 10-epoch post-revision run under the updated conditioning path did produce a qualitatively cleaner checkpoint, which is why it is used as the default visual example elsewhere in the revision. However, its prompt-ranking audits remained near a 4-way chance, so it did not alter the main conclusion established above. Across the main checkpoint and the follow-up, the evidence is consistent: text presence matters, visual smoothness improved, but prompt-specific semantic separation remains weak.

5.5 Limitations and Future Work↩︎

5.5.0.1 Resolution, computational budget, and visual fidelity.

The combination of \(64 \times 64\) output resolution, single-GPU training budget, and limited data scale restricts the model’s ability to learn the fine-grained articulatory detail required for reliable sign generation, including finger spelling and subtle facial grammar markers. Accordingly, the present study does not consistently achieve reliable, high-fidelity signing under the computational regime considered here. Cascaded super-resolution networks, latent diffusion formulations operating on learned codebook representations, and larger-scale training budgets may help address this limitation without proportionally increasing the spatio-temporal attention cost.

5.5.0.2 Temporal length and compositional generation.

Fixed \(T=32\) clips (approximately 1.07 s at 30 fps) still constrain output to individual signs or short phrases. Generating sentence-level or discourse-level signing requires autoregressive decoding with overlap-based blending, sliding-window attention, or hierarchical latent structures that condition fine-grained motion on coarse-grained linguistic plans.

5.5.0.3 Semantic and linguistic evaluation.

Current metrics—video feature distance, motion magnitude, temporal consistency, and held-out conditional denoising loss—assess perceptual, distributional, and indirect conditioning qualities. However, they still do not directly measure linguistic correctness or intelligibility. The held-out conditional-loss audit on real clips confirms that the checkpoint uses the presence of text. However, it also shows that shuffled prompts remain nearly tied to intended prompts, indicating that prompt identity is still only weakly expressed. Future work should therefore incorporate stronger signer-disjoint recognition or translation backends and, where feasible, user studies with Deaf and hard-of-hearing participants or sign-language experts to assess communicative efficacy.

5.5.0.4 Deployment optimization.

The present system is not an edge or real-time method: even the 8-step inference setting still requires 12.60 s for a 32-frame clip on an NVIDIA L4. Achieving real-time inference (sub-100 ms per frame) for interactive applications will require quantization (INT8/FP8), structured pruning of redundant transformer heads, stronger latent compression, and compilation to optimized inference backends (ONNX Runtime, TensorRT). Our modular architecture — featuring a clearly separated text encoder, denoising backbone, and scheduler — is well-suited to such post-training optimizations.

5.5.0.5 Dataset diversity and generalization.

The How2Sign dataset features a limited number of signers performing instructional content. Signer-specific idiosyncrasies in handshape, signing speed, and spatial extent may limit cross-signer generalization. Evaluation on multi-signer corpora (e.g., PHOENIX-2014T, BSL Corpus) and data augmentation strategies (spatial jittering, temporal rate perturbation) would strengthen claims of generalizability.

6 Conclusion↩︎

This paper presents Text2Sign, a single-GPU diffusion baseline for text-to-sign-language video generation. The short (3-epoch) ablation on a signer-disjoint partition of How2Sign-derived clips (4,082 clips after preprocessing) provides indicative, though not definitive, trends: (i) DiT-style transformer blocks reduce validation loss by roughly 19.5% relative to convolution-only baselines (0.0648 vs.), suggesting the value of global attention for sign-language structure; (ii) frozen CLIP text encoders yield about 11.0% lower validation loss (0.0648 vs.) with 7.7% fewer trainable parameters in the present short-budget comparison, indicating that pretrained vision-language priors help this optimization setting; and (iii) factorized spatial–temporal attention is marginally better than full 3D attention in this regime (0.0648 vs.) while offering substantially better theoretical scaling. A longer signer-disjoint checkpoint reaches validation loss 0.00999 and, under a compact validation slice, yields SSIM \(0.2403\pm0.0238\), PSNR \(15.11\pm0.42\) dB, and temporal consistency \(1.0000\pm0.0000\). Under an 8-step DDIM setting, it generates a 32-frame clip in 12.60 s on a single NVIDIA device. The simple continuing vision checkpoint is also visibly cleaner and more stable than the earlier baseline in direct side-by-side qualitative inspection, which justifies using it as the default qualitative example in this revision. Held-out conditional-loss audits on real clips reveal a measurable null-text penalty, but only weak separation between intended and shuffled prompts. The contribution of this work should therefore be understood as a constrained-resource research baseline that clarifies architectural trade-offs and improves qualitative generation under a single-GPU budget, rather than as a complete sign-language production system. Together with the fine-tuning analyses, these findings suggest that future progress will likely require architectural, representational, evaluation, and scaling advances rather than simply continuing the same training recipe.

Acknowledgment↩︎

The author thanks Dr.Khushi Desai for advising on this research. This work was performed using NVIDIA L4 GPU resources provided by Lightning AI.

References↩︎