1 Introduction↩︎

The design and development of effective tools for translating sign language are increasingly necessary to promote inclusive communication between deaf or hard-of-hearing individuals and the broader community. Sign languages are complex visual-spatial languages that rely on hand movements, facial expressions, and body posture, which make real-time interpretation challenging for people who do not understand them [1]. As a result, communication barriers often limit access to education, healthcare, public services, and employment opportunities for the deaf community. Recent advances in artificial intelligence, computer vision, and sensor technologies provide new opportunities to automatically detect and interpret hand gestures for sign language translation.

Two primary approaches have dominated the development of sign language translation systems: vision-based approaches and glove-based approaches. Vision-based approaches employ cameras together with computer vision and machine learning algorithms to interpret hand gestures directly from visual data [2], [3]. By analyzing video frames or image sequences, these systems detect and track hand movements, extract spatial and temporal features, and map the observed gestures to corresponding words or sentences. With the rapid progress of deep learning, particularly convolutional neural networks [4] and transformer-based models [5], vision-based methods have achieved significant improvements in gesture recognition accuracy and robustness [6], [7]. However, these approaches often face practical limitations, including sensitivity to lighting variations, background clutter, occlusions, and changes in camera viewpoint. Advanced sensors, such as RGB-D cameras and LiDAR [8], [9], can partially mitigate these issues. Nevertheless, continuous tracking of fine-grained hand movements remains computationally demanding and may require carefully controlled recording conditions.

These challenges have motivated the exploration of glove-based approaches for sign language recognition. Unlike vision-based systems, glove-based solutions rely on wearable sensors to directly capture hand motion and finger articulation, enabling more stable and precise measurements for gesture interpretation [10], [11]. Such systems typically incorporate multiple sensing components such as flex sensors and inertial measurement units (IMUs) to measure finger bending, hand orientation, and motion dynamics [12]. Early work demonstrated that instrumented gloves could capture detailed hand articulation and convert gestures into textual or spoken outputs to provide more reliable recognition compared with camera-based systems under challenging environmental conditions [11], [13]. Later studies have further improved sensing accuracy by introducing advanced sensing materials, integrating multiple sensor modalities, and applying machine learning algorithms for dynamic gesture classification. For example, the use of soft fiber-optic sensors combined with machine learning achieved high recognition accuracy by capturing fine-grained finger motion patterns [14]. Integration flex and IMU sensors with deep learning techniques allowed to recognize complex sign language vocabularies in real time [15]. Despite these advantages, glove-based systems may introduce usability challenges, including the need for wearable hardware, calibration requirements, and potential discomfort during prolonged use. Nevertheless, they remain an important research direction for achieving accurate and robust sign language translation, particularly in environments where vision-based approaches are less reliable [10], [16].

In this work, we propose a multimodal system that combines glove-based sensing and vision-based analysis for Vietnamese sign language (VSL) recognition. The system uses a smart glove equipped with flex sensors and an IMU to capture finger bending, hand shape, and three-dimensional hand motion. Sensor data are collected by an ESP32-C6 microcontroller and transmitted to a computer, where they are fused with facial expression information captured by the computer’s camera. A deep learning model processes these multimodal inputs to infer sign language gestures. By integrating wearable motion sensing with visual facial cues, the proposed system captures both manual and non-manual components of sign language to improve recognition accuracy and robustness.

2 System architecture↩︎

Sign languages across different countries share common characteristics, as they rely on a combination of hand movements, gestures, and facial expressions to convey meaning. Accordingly, recognizing sign language requires capturing both the structural sequencing of gestures and the temporal dynamics of hand motion, as well as non-manual cues such as facial expressions.

2.1 System overview↩︎

The overall architecture of the proposed sign language recognition system is illustrated in Fig. 1. The system combines a wearable sensing glove and a vision-based module to capture both manual and non-manual components of Vietnamese Sign Language (VSL).

The sensing glove integrates flex sensors and an IMU to measure finger articulation and hand motion. Sensor data are collected by an embedded microcontroller and transmitted via Wi-Fi to a host computer. In parallel, a camera connected to the computer captures facial expressions that provide additional contextual information in sign language communication.

On the computer, a multimodal deep learning model processes both the sensor data from the glove and the visual information from the camera. By fusing these inputs, the system interprets the performed gesture and translates it into corresponding text or speech output.

2.2 Hand glove hardware design↩︎

The hardware design of the proposed sensing glove is shown in Fig. 2. The glove is designed to capture detailed finger movements and hand orientation using multiple embedded sensors connected to an ESP32-C6 microcontroller.

• Flex sensors: Multiple SF15 flex sensors are positioned along the fingers of the glove to measure finger bending. Each sensor changes its resistance according to the degree of flexion. The resistance variation is converted into voltage signals using a voltage divider circuit and digitized by an ADS1115 analog-to-digital converter (ADC). The digitized measurements provide detailed information about finger articulation and hand shape for sign language recognition.

• IMU sensor: An IMU BNO055 is mounted on the back of the hand to track hand orientation and motion. The BNO055 integrates an accelerometer, gyroscope, and magnetometer to estimate hand rotation and movement in three-dimensional space, which allow the system to capture dynamic gestures and spatial hand movements.

• Micro controller: Sensor readings from the flex sensors and IMU are collected by an ESP32-C6 microcontroller located inside the glove enclosure. The ESP32-C6 performs sensor interfacing and data acquisition, and communicates with the ADS1115 and BNO055 via the I²C interface. The collected data are then transmitted wirelessly to the host computer for further processing.

Figure 3 shows the schematic circuit of the sensing glove. Each flex sensor is connected in a voltage divider configuration with a fixed resistor to convert resistance changes into measurable voltage signals. These analog signals are connected to the input channels of the ADS1115 ADC, which converts them into digital values. The ADS1115 communicates with the ESP32-C6 via the I²C interface. The BNO055 IMU is also connected through the same I²C bus to provide hand orientation and motion data. Power is supplied through a battery module that powers the ESP32-C6 and the connected sensors.

3 Multimodal deep learning model↩︎

The proposed system adopts a multimodal deep learning architecture that integrates hand motion signals and facial cues for gesture recognition. As illustrated in Fig. 4, the pipeline consists of five main stages: input acquisition, signal preprocessing, neural network inference, sentence construction, and output generation.

3.1 Input layer and data preprocessing↩︎

The input layer collects multimodal data from three sensing sources: flex sensors (SF15), an IMU (BNO055), and camera-based facial analysis. These signals provide complementary information about hand configuration, motion dynamics, and contextual facial expressions. Raw sensor data are streamed continuously and timestamped during acquisition. To ensure temporal consistency across samples, the signals are uniformly resampled by retaining measurements separated by \(80\,\text{ms}\) intervals. Each gesture instance consists of \(25\) temporal frames, resulting in an approximate sample duration of

\[25 \times 80\,\text{ms} = 2000\,\text{ms} \approx 2\,\text{s}.\] During preprocessing, the signals are normalized and filtered to reduce noise, and the continuous data stream is segmented into fixed-length gesture samples suitable for model training.

For facial analysis, the system uses MediaPipe[17] to detect faces and extract landmarks, including the eyebrows, eyes, and mouth. The distances between these landmarks are computed and aggregated into a feature set. The values are then averaged within a defined range. Finally, the features are encoded into discrete values from 0 to 6, representing seven distinct expressions corresponding to grammatical components.

3.2 Training data representation↩︎

Let \(T\) denote the number of temporal frames and \(d\) denote the feature dimension. Each gesture instance is represented as a multivariate time-series \(X \in \mathbb{R}^{T \times d}\), where \(T = 25\) and the feature dimension is \(d = 14\). The feature vector consists of the following measurements:

• Flex sensors: \(flex_1, flex_2, flex_3, flex_4, flex_5, flex_6, flex_7, flex_8, flex_9, flex_{10}\)

• IMU orientation signals: \(imu_x, imu_y, imu_z\)

• Facial expression indicator: \(exp\)

Thus, each gesture sample is represented as a sequence of \(25\) frames with \(14\) features per frame.

3.3 Neural network architecture↩︎

To model temporal dependencies within the input gesture sequence, the system employs a Long Short-Term Memory (LSTM) neural network, as shown in Fig. 5. LSTM networks are well suited for sequential data because they can learn temporal patterns and motion dynamics across time steps.

The first LSTM layer contains 64 hidden units and returns the full output sequence. This allows the network to preserve temporal information across all time steps. A dropout layer with a dropout rate of 0.5 is applied to reduce overfitting during training.

The output sequence is then processed by a second LSTM layer with 64 hidden units, which encodes the temporal sequence into a compact feature representation. Another dropout layer with a rate of 0.5 is applied for additional regularization.

The resulting representation is passed to a fully connected dense layer with 64 neurons and ReLU activation to learn higher-level feature interactions. Finally, a softmax output layer produces a probability distribution over the gesture classes.

The network is trained using the Adam optimizer and the categorical cross-entropy loss function, which measures the discrepancy between the predicted class probabilities and the ground-truth labels. The loss function is defined as

\[L = - \sum_{i=1}^{C} y_i \log(\hat{y}_i),\] where \(C\) denotes the number of gesture classes, \(y_i\) represents the ground-truth label in one-hot encoded form, and \(\hat{y}_i\) is the predicted probability produced by the softmax layer. Training is performed for up to 50 epochs with a batch size of 128. A validation split of 20% is used to monitor training performance, and early stopping with a patience of five epochs is applied to prevent overfitting.

4 Results↩︎

A number of experiments were conducted to evaluate the performance of the proposed approach, as described below.

4.1 Dataset preparation↩︎

The dataset used in the experiments was collected using the proposed sign language glove system. Each gesture recording consists of synchronized readings from multiple onboard sensors, including five flex sensors embedded along the fingers and an IMU providing motion data along the \(x\), \(y\), and \(z\) axes. An additional signal labeled face was also recorded to capture supplementary gesture-related information.

Three Vietnamese sign language gestures were included in the dataset: “‘Tôi’’,”Xin chào”, and “Hẹn hò”. Multiple recordings were collected for each gesture, resulting in a total of 10 gesture sequences. Each sequence contains 25 time steps, representing the temporal evolution of the hand gesture during its execution.

At each time step, a 9-dimensional feature vector is recorded: \[[f_1, f_2, f_3, f_4, f_5, x, y, z, face]\] where \(f_1\)-\(f_5\) correspond to the five flex sensor readings and \(x\), \(y\), and \(z\) represent the IMU motion measurements. Consequently, each gesture sample forms a \(25 \times 9\) time-series matrix. This structure enables the learning model to capture both finger articulation patterns and hand motion dynamics during gesture execution. Table 1 shows a short excerpt of the recorded sensor data for one gesture sequence.

Overall, the dataset contains 250 time steps of sensor measurements across all recordings. The temporal nature of the data makes it suitable for sequence learning models such as LSTM networks.

4.2 Training results↩︎

The proposed LSTM-based gesture recognition model was trained using the collected sign language glove dataset. During training, both accuracy and loss were monitored for the training and validation sets to evaluate the learning behavior and generalization performance of the network.

Fig. 6 (a) shows the training and validation accuracy and loss curves over the training epochs. The training accuracy rapidly increases during the first few epochs and gradually converges to nearly \(100\%\). The validation accuracy stabilizes around \(93\%\)-\(95\%\), indicating that the model is able to generalize well to unseen samples. Meanwhile, the training loss decreases steadily toward zero, while the validation loss stabilizes at a low value, suggesting that the model successfully learns the temporal patterns of the gesture sequences without severe overfitting.

The confusion matrix shown in Fig. 6 (b) illustrates the classification performance across the three gesture classes: Hẹn Hò, Tôi, and Xin Chào. The model achieves perfect classification for the gestures Hẹn Hò and Tôi, with all samples correctly predicted. For the Xin Chào gesture, the majority of samples are correctly classified, while a small number are misclassified as Hẹn Hò and Tôi.

Overall, the model achieves the accuracy of approximately \(95\%\). These results demonstrate that the proposed model can effectively capture both finger articulation patterns from the flex sensors and motion dynamics from the IMU data, enabling reliable recognition of VSL gestures.

4.3 Real-time deployment on hardware↩︎

To evaluate the practical applicability of the proposed system, the trained gesture recognition model was deployed on a real hardware setup using the sign language glove. The trained model was converted to TensorFlow Lite format and executed on a laptop connected to the glove system for real-time inference.

During testing, the glove collects sensor data from the flex sensors and inertial measurement unit while the user performs a gesture, as shown in Fig.7. In this example, the user performs the gesture corresponding to “Tôi”. The system successfully records the sensor sequence, loads the trained model, and performs inference in real time. The model correctly predicts the gesture as “Tôi” with a confidence score of \(95.34\%\).

In addition, the glove was tested on five participants with different hand sizes. Experiments were conducted under varying lighting conditions and distances. Optimal facial recognition performance was achieved when the user was positioned approximately one arm’s length from the camera. Users also reported that the system was comfortable to use during testing.

The results demonstrate that the proposed gesture recognition system can operate in a real-world environment. The deployment confirms that the trained LSTM model is lightweight enough for real-time execution while maintaining high prediction accuracy.

5 Conclusion↩︎

This paper presented a multimodal smart glove system for sign language recognition that integrates wearable sensing and deep learning. The proposed glove uses flex sensors and an IMU to capture finger articulation and hand motion, while facial cues are obtained from a camera. A LSTM network processes the multimodal data to recognize gesture sequences. Experimental results show that the system achieves approximately \(95\%\) recognition accuracy. The trained model was converted to TensorFlow Lite and successfully deployed for real-time inference. Future work will focus on expanding the gesture vocabulary, collecting larger datasets, and improving system portability.

References↩︎