1 Introduction↩︎

Spoken-to-sign language gesture-based systems crucial in assistive sign language research. These systems hold significant potential to bridge the communication gap between verbal speakers and the hearing-impaired community. However, most existing systems focus solely on lexical translation, neglecting the emotional context of spoken language, an essential component of natural, human-centered communication.

Dynamic, real-time avatar generation is a critical element in enhancing the naturalness and expressiveness of sign language communication. Yet, due to the absence of emotionally expressive avatars, many current systems resemble robotic gesture mimicking rather than authentic human interaction. This gap is even more prominent in low-resource languages like Nepali and its corresponding sign language, Nepali Sign Language (NSL), where research and datasets are scarce. This pilot study proposes a low-resource multimodal translation pipeline that combines automatic speech recognition (ASR) with emotion recognition to generate dynamic sign language avatar animations. It encompasses four frequently used Nepali sign words, with corresponding facial expressions representing three emotional states: happy, sad, and neutral. The key contributions of this research are:

1. It presents first NSL-based speech dataset annotated with emotional context.

2. This has a modular, real-time pipeline with independent components for ASR, emotion recognition enabling easy scalability and upgrades.

3. The pipeline is lightweight and suitable for low-resource deployment, supporting edge applications.

The remainder of this paper is structured as follows: Section 2 reviews Related Works. Section 3 describes our Methodolog, architecture and model implemen- tation. Section 4 presents experimental results and analysis. Finally, Section 5 concludes the paper and section 6 discusses future directions.

2 Related Works↩︎

Recent research has explored integrating emotional expressions into sign language avatars to enhance comprehension and naturalness.As shown in [1] Smith & Nolan evaluated augmenting avatars with universal emotions in Irish Sign Language, finding little difference in comprehension between baseline and emotionally-enhanced avatars. Gonçalves et al. (2017)[2] proposed a facial expression parametrization method for avatars, identifying relevant facial landmarks and emotions to improve automatic sign synthesis systems [3] introduced an avatar-based Sign Language Production system for Korean Sign Language, incorporating named entity transformation and context vector generation to address out-of-vocabulary issues. While these studies demonstrate progress in integrating emotional expressions and improving avatar-based sign language systems, challenges remain in achieving natural and linguistically accurate representations.

Previous research by  [4], use a fused data set that combines the SAVEE and RAVDESS data sets (2,459 samples, 7 emotions) followed by MFCC preprocessing to encode spectral-temporal features. In general, he adopts the CNN-LSTM hybrid architecture for speech emotion recognition, achieving 61. 07% in the test set and 75.31% in the train set. A 3D avatar-based sign language learning system by  [5]. uses three modules: speech-to-text via IBM Watson, English-to-ISL translation with Lexical Functional Grammar, and Blender-based 3D avatar animation synchronized via a "motion list" for Indian Sign Language gestures. Emotionally expressive AI avatars enhance communication for hearing-impaired users, offering affordable, customizable interpreting services, but raise ethical concerns [6]. Designing for emotion and considering users’ unique needs is crucial, with emphasis on incorporating hearing-impaired individuals in the development process [7].

3 Methodology↩︎

3.1 Overview↩︎

This research proposes a novel and lightweight multimodal pipeline for translating spoken Nepali into sign language gestures with emotion-aware rendering. A shared acoustic encoder performs both automatic speech recognition (ASR) and emotion classification from the input audio. The core of the system is a unified architecture, termed NEST-V1 (Nepali Emotion and Speech Transformer – Version 1), which is jointly parameterized for both tasks via a shared encoder.

3.2 Dataset Creation↩︎

The dataset includes four commonly used Nepali words—thank you, home, me, and hello—selected based on the nature of their corresponding sign language gestures. Specifically, “thank you” and “me” are dynamic gestures involving continuous hand motion, whereas “home” and “hello” are static gestures characterized by a fixed hand pose. This distinction allows for a balanced evaluation of both motion-centric and pose-centric outputs. To ensure robustness and generalizability, audio samples were collected for all gesture–emotion combinations. Each speaker provided 12 audio samples, representing the four target words, each spoken with three emotional tones (happy, sad, and neutral). The raw recordings were captured in .m4a and .aac formats. Files were standardized to .wav format using FFmpeg and Pydub for compatibility with preprocessing and augmentation pipelines. Participants ranged in age from 15 to 45 years, ensuring age diversity. Gender distribution is summarized in Table 2.

3.3 Dataset Augmentation and Audio Characterstics↩︎

To enhance diversity and increase the volume of audio samples, this study employed both semitone shifting and Vocal Tract Length Perturbation (VTLP) for data augmentation. The resulting sample counts across the four gesture classes are summarized in Table 4. Furthermore, the dataset can be broadly categorized into four duration ranges, as detailed in Table 3. All audio samples were collected in either .aac or .m4a formats, with respective sampling rates of 44,100 Hz and 48,000 Hz. These rates were consistent across the original recordings as well as the augmented samples, including those processed via VTLP and semitone shifting.

3.3.1 Augmenting Audio Dataset with Random VTLP↩︎

Vocal Tract Length Perturbation (VTLP) simulates variations in the vocal tract length by warping the frequency spectrum of an audio signal. For each audio sample with a sampling rate \(sr = 44{,}100\) Hz or \(48{,}000\) Hz, we compute the Short-Time Fourier Transform (STFT) using an FFT size of \(N = 2048\) and a hop length of \(H = 512\). The frequency axis is linearly warped using random warping factors \(\alpha \in \{0.8, 0.9, 1.2, 1.3\}\). While standard vocal tract length normalization (VTLN) typically restricts \(\alpha\) to \([0.8, 1.2]\) [8], our goal is to introduce greater diversity in the dataset. Hence, we expand the range to \([0.8, 1.3]\).

3.4 Audio Augmentation↩︎

1. Resampling: All audio files were resampled from their original sampling rates (44.1kHz/48kHz) to 16kHz.

2. Fixed Duration: Each sample was clipped or zero-padded to a fixed duration of 2 seconds (32,000 samples at 16kHz) to handle temporal variability.

3. Hop Length: A hop length of 160 samples, equivalent to 10ms, was chosen: \[\begin{align} \text{Hop Length} &= \text{Sample Rate} \times \text{Frame Shift} \nonumber \\ &= 16{,}000 \times 0.010 = 160 \label{eq:hop} \end{align}\tag{1}\]

4. FFT Parameters: The number of FFT points (n_fft) was set to 320, resulting in a window size of 20ms.

5. Target Frame Count: Each spectrogram was configured to contain exactly 200 frames to ensure uniform dimensions.

6. Normalization: Each waveform was normalized to the range \([-1, 1]\).

All spectrograms were then converted into 2D image tensors with a fixed input size of \(128 \times 200\), where:

• 128 corresponds to the number of Mel frequency bins (vertical axis),

• 200 corresponds to the number of time frames (horizontal axis).

This standardization allowed consistent batch processing during both training and inference.

3.5 Model Architecture↩︎

3.5.1 Overview↩︎

NEST-V1 converts raw audio into visual representations using Mel spectrograms—grayscale images that capture frequency patterns over time. These spectrograms are split into smaller patches and passed through a lightweight Vision Transformer (ViT) backbone. The model learns complex time-frequency relationships and performs both keyword recognition (e.g., “Namaste”) and emotion classification using a single unified architecture. By treating audio as images and leveraging Transformers, we effectively manage multimodal audio understanding under low-resource conditions.

3.5.2 Technical Implementation↩︎

We adapt a ViT-style transformer for audio spectrograms by treating them as 2D grayscale images. The following sections detail the preprocessing, model layers, and classification heads.

Input Representation↩︎

Each audio clip is transformed into a Mel spectrogram of shape \(128 \times 200\) (Mel bands × time frames), resized to \(128 \times 128\) via bilinear interpolation to match the input requirements of the Vision Transformer. These spectrograms are treated as single-channel (grayscale) images and normalized.

Patch Embedding Layer↩︎

The spectrogram image is divided into non-overlapping patches of size \(16 \times 16\), resulting in 64 patches. A convolutional projection layer with kernel size equal to the patch size \((16 \times 16)\), stride 16, and output channels 768 is used to embed each patch: \[\text{Conv2D}(1, 768, \text{kernel}=16, \text{stride}=16)\] The patch embeddings are flattened and linearly projected to a dimension \(D = 768\).

A learnable [CLS] token is prepended for classification, and learnable positional embeddings are added to retain spatial ordering: \[\mathbf{Z}_0 = [\mathbf{z}_{\text{cls}}; \mathbf{z}_1; \dots; \mathbf{z}_{64}] + \mathbf{E}_{\text{pos}}\] Both embeddings are initialized using a truncated normal distribution with standard deviation \(\text{std} = 0.02\).

Transformer Encoder↩︎

The patch sequence, including the [CLS] token, is passed through a stack of \(L = 3\) identical Transformer blocks. Each block consists of:

• Multi-head self-attention (12 heads, with head dimension \(d = 64\); total embedding dim = 768)

• Pre-norm Layer Normalization (before both attention and MLP)

• MLP with GELU activation and 4\(\times\) expansion (768 \(\rightarrow\) 3072 \(\rightarrow\) 768)

• Residual connections across both attention and MLP sublayers

• Dropout with \(p=0.1\) for regularization

These layers capture both local and global dependencies in time-frequency space, essential for understanding speech and emotion.

Classification Heads↩︎

After the Transformer encoder, only the representation of the [CLS] token is used for classification. It is passed through a final LayerNorm and separate linear classifiers for each task: \[\hat{\mathbf{y}}_{\text{task}} = \text{softmax}(\mathbf{W}_{\text{task}} \cdot \mathbf{z}_{\text{cls}} + \mathbf{b}_{\text{task}}) \label{eq:classifier}\tag{2}\]

Two parallel heads are used:

• Emotion classification (3 classes: Happy, Neutral, Sad)

• Keyword classification (4 classes: “Hello”, “Thank you”, “House”, “Me”)

This design supports multi-task learning while sharing a common feature extractor.

Why It Works↩︎

This architecture jointly captures:

• Spectral structure via 2D patches in Mel-frequency space

• Temporal evolution across spectrogram frames

• Global dependencies through Transformer self-attention

This makes it especially suitable for tasks like emotion and keyword recognition from spoken Nepali audio, even with limited training data.

3.6 Experimental Setup↩︎

We trained a custom Audio Spectrogram Transformer (AST) model for audio classification, adapting the architecture to balance efficiency and performance. The model takes input spectrograms of size \(128 \times 128\) (single channel), divided into non-overlapping patches of size \(16 \times 16\), which are linearly projected into 768-dimensional embeddings. The Transformer backbone consists of 3 encoder layers, each with 12 attention heads, followed by task-specific classification heads. For optimization, we used the AdamW optimizer with an initial learning rate of 0.001 and a weight decay of 0.1. A cosine annealing scheduler with 10 cycles was employed to stabilize training over 25 epochs. Cross-entropy loss was used as the training objective. Batch sizes were set according to the data loader capacity. Training and validation accuracy and loss were recorded per epoch, with the best model selected based on validation accuracy. All experiments were conducted on CUDA-enabled GPUs using PyTorch, with CPU fallback where necessary. To ensure reproducibility, random seeds were fixed where applicable, and standard data splits were used for training and validation. This lightweight architecture achieves a balance between computational efficiency and representational capacity, making it well-suited for resource-constrained environments without sacrificing competitive performance.

3.7 Avatar Generation↩︎

To represent signed gestures with emotional nuance, we developed a lightweight and expressive avatar animation pipeline. This pipeline converts the audio input—specifically, the detected keyword and its emotional tone—into corresponding sign language animations using pre-rendered 2D avatar frames.

Data Preparation↩︎

For each of the four selected Nepali Sign Language gestures—Thank you, Hello, House, and Me—we prepared four distinct avatar images:

• A base pose representing the neutral standing position,

• Three emotionally expressive variants: Happy, Sad, and Neutral.

All avatars were designed with a consistent style, depicting the upper body, face, and hands to clearly convey both the sign gesture and its emotional context.

Gesture Animation Pipeline↩︎

To create fluid animations from static images, we implemented a frame-blending pipeline using Python libraries such as OpenCV and PIL. The animation process involves morphing the base (neutral) avatar image into an emotionally expressive variant using linear interpolation across frames. Specifically, the pipeline executes the following steps:

1. Load the base and target images (e.g., ghar-base.png and ghar-sad.jpeg) and resize both to a resolution of 512×512 pixels.

2. Generate 30 interpolated frames by applying alpha blending using OpenCV’s addWeighted function, where blending weights change linearly from the base to the target image.

3. Convert each blended frame to RGB and wrap it as a PIL.Image object.

4. Reverse the frame sequence to generate a smooth looping animation (i.e., forward + reverse = 60 frames total).

5. Export the sequence as a looping animated GIF using PIL’s save function, with a frame duration of 25 milliseconds.

This process is repeated for each combination of gesture and emotion, resulting in a total of 12 animated GIFs: four gestures × three emotions.

Emotion-Conditioned Playback↩︎

Once the system identifies the spoken word and corresponding emotional tone using the audio classification and emotion recognition modules, it retrieves the mapped animated GIF and plays it as output. For example, if the input audio contains the word Thank you spoken with a sad emotional tone, the system displays the thank-you-sad.gif animation.

This modular mapping approach ensures real-time responsiveness and emotional expressiveness in sign language output. Furthermore, the use of optimized, lightweight GIFs makes this method deployable on resource-constrained platforms such as web browsers and mobile devices, without requiring high-end graphics rendering.

NEST-V1 is designed for deployment in resource-constrained environments. Table 7 summarizes the computational characteristics of our model compared to typical alternatives.

3.7.1 Parameter Efficiency↩︎

Our shared encoder architecture achieves significant parameter reduction, calculated as:

\[\begin{align} \text{Parameter Reduction} &= \frac{P_{\text{separate}} - P_{\text{shared}}}{P_{\text{separate}}} \times 100\% \\ &= \frac{35.2M - 22.1M}{35.2M} \times 100\% = 37.2\% \end{align}\]

This reduction is achieved by sharing the transformer encoder between ASR and emotion recognition tasks, eliminating duplicate feature extraction layers.

3.7.2 Computational Complexity Analysis↩︎

Patch Embedding Layer

• Input: \(128 \times 128 \times 1\) spectrogram

• Patches: 64 patches of size \(16 \times 16\)

• Embedding dimension: 768

• FLOPs: \(64 \times 16 \times 16 \times 768 = 12.6 \text{M FLOPs}\)

Transformer Encoder

For each of the 3 transformer layers:

• Multi-head attention: \(\mathcal{O}(n^{2} d)\), where \(n=65\) (64 patches + CLS token), \(d=768\)

• MLP: \(\mathcal{O}(n d \times 4 d) = \mathcal{O}(4 n d^{2})\)

• Per layer FLOPs: \(65^{2} \times 768 + 4 \times 65 \times 768^{2} \approx 153.6 \text{M FLOPs}\)

• Total for 3 layers: \(3 \times 153.6 \text{M} = 460.8 \text{M FLOPs}\)

Classification Heads

• ASR head: \(768 \times 4 = 3,072\) FLOPs

• Emotion head: \(768 \times 3 = 2,304\) FLOPs

Total FLOPs: \(12.6\text{M} + 460.8\text{M} + 0.005\text{M} \approx 473.4 \text{M FLOPs}\)

3.7.3 Memory Efficiency↩︎

Model Parameters

• Patch embedding: \(16 \times 16 \times 1 \times 768 = 196{,}608\) parameters

• Positional embeddings: \(65 \times 768 = 49{,}920\) parameters

• Transformer layers: \(3 \times (768^{2} \times 4 + 768 \times 3072 \times 2) \approx 21.3 \text{M}\) parameters

• Classification heads: \(768 \times 7 = 5{,}376\) parameters

• Total: \(\sim 22.1 \text{M}\) parameters

Runtime Memory

• Input tensor: \(128 \times 128 \times 1 \times 4 \text{ bytes} = 65.5 \text{ KB}\)

• Intermediate activations: \(\sim 85 \text{ MB}\)

• Model weights: \(22.1 \times 10^{6} \times 4 \text{ bytes} = 88.4 \text{ MB}\)

• Total inference memory: \(\sim 89 \text{ MB}\)

3.7.4 Scalability Analysis↩︎

The computational complexity scales as follows with vocabulary expansion:

• ASR vocabulary scaling: \(\mathcal{O}(V)\) where \(V\) is vocabulary size

• Emotion categories scaling: \(\mathcal{O}(E)\) where \(E\) is number of emotions

• Core transformer complexity: remains constant, \(\mathcal{O}(1)\)

For deployment on edge devices, our model maintains:

• Inference time: \(< 50\) ms on modern mobile GPUs

• Memory footprint: \(< 100\) MB total

• Power consumption: estimated 2–3 W during inference

3.7.5 Deployment Considerations↩︎

Hardware Requirements

• Minimum: 4 GB RAM, ARM Cortex-A78 or equivalent

• Recommended: 6 GB RAM, Mobile GPU (Adreno 640+, Mali-G76+)

• Optimal: 8 GB RAM, Dedicated AI accelerator

Software Optimization

• Model quantization: INT8 quantization can reduce memory by 75%

• Pruning potential: estimated 30–40% parameters can be pruned

• Batch processing: supports batch sizes 1–16 for efficiency

4 Experimental analysis and results↩︎

The model was evaluated on two datasets: an ASR dataset comprising 3,107 training samples, 889 validation samples, and 447 testing samples; and an emotion dataset with 2,420 training samples, 753 validation samples, and 321 testing samples. During 25 training epochs, the model demonstrated steady performance, achieving a best training accuracy of 81.1% on the ASR dataset and 79.21% on the emotion dataset. Validation accuracies reached 79.6% for ASR and 76.54% for emotion recognition. The final loss scores for the ASR dataset were 0.3121 (training) and 0.4876 (validation), while for the emotion dataset, the loss was 0.476 (training) and 0.684 (validation).

5 Conclusion↩︎

This work presents a novel low-resource, multimodal translation pipeline that maps spoken Nepali utterances to emotion-conditioned sign language animations using a lightweight and efficient architecture. The proposed system integrates three key modules:

• a Nepali Automatic Speech Recognition (ASR) system for transcribing spoken inputs into text,

• an emotion classification module for extracting affective context from the audio signal,

• a gesture synthesis pipeline that maps the recognized text and emotion labels to pre-rendered Nepali Sign Language (NSL) avatar animations

The transformer-based hybrid model (NEST-V1) jointly optimizes ASR and emotion recognition, ensuring low-latency and resource-efficient performance suitable for deployment on edge devices. Evaluation on a constrained vocabulary of four frequent Nepali words yielded consistent classification performance, with F1-scores between 0.69 and 0.78. The incorporation of emotion conditioning into the avatar rendering phase enables expressivity beyond lexical translation, thereby improving the naturalness and communicative effectiveness of the generated sign gestures. Overall, this work demonstrates the feasibility of building an end-to-end, emotion-aware speech-to-sign language system for low-resource settings, with potential for scalability across larger vocabularies and real-world assistive applications.

6 Future Directions↩︎

Looking ahead, The authors plan to improve the current system across several key dimensions:

• Expand Vocabulary and Emotions: To include a wider set of Nepali words and extend the emotion categories to cover a broader emotional spectrum. This will help the model generalize better across real-world conversations and contexts.

• Shift to Dynamic Avatar Generation: Currently, the avatar gestures are rendered using pre-defined animated GIFs. In the next phase, the authors plan to implement a real-time avatar rendering system—possibly using 2D skeletal animation or lightweight 3D rigs—to make the signing experience more fluid and natural.

• Collect Larger and More Diverse Dataset: To improve model performance and fairness, the authors plan to collect additional speech samples from speakers across diverse age groups, dialects, and genders. This will support better generalization in low-resource scenarios.

• Introduce Human Evaluation: Alongside technical metrics, human evaluation framework involving hearing-impaired users will be introduced. Their feedback on avatar clarity, emotional accuracy, and overall usability will be critical in shaping the next version of the system.

• Optimize for Edge Deployment: Although the system is already lightweight, the authors plan to further optimize it using quantization, pruning, and efficient runtime architectures so it can run smoothly on mobile devices and embedded systems.

The goal is to maintain the balance between performance and deployability, keeping the system modular, real-time, and aligned with real-world assistive needs.

7 Acknowledgement↩︎

The authors acknowledge Dr. Manish Sakhakarmy, for his guidance and Sunway College for assiting with data collection.

References↩︎