Most sign language understanding systems operate at the level of isolated signs, limiting their usefulness in natural communication. We study sentence-level sign language translation (SLT) with the primary goal of real-time deployment rather than proposing a new translation architecture. We fine-tune a SHuBERT–ByT5 translation stack on a uniformly sampled 9,872-example subset of How2Sign, selected because of compute and storage constraints, using QLoRA while keeping SHuBERT frozen. The model obtains a validation BLEU of 16.7 and, on the test split, BLEU 15.9 and BLEURT 44.7. The main contribution is a hardware-aware streaming system: a Raspberry Pi 4B reference client provides camera capture, local text display, and speech output, while compute-intensive perception and translation run on a CPU/GPU backend. The capture protocol remains client-agnostic, so the same backend can serve a browser, phone, or laptop. Chunked ingestion, bounded queues, parallelized perception, temporal reordering, and a sentence-boundary state machine reduce mean post-finalization response latency from 1.873 to 1.354 seconds (27.71%) and P95 latency from 2.919 to 2.130 seconds (27.03%) over the complete 9,872-example working subset.

1 Introduction↩︎

Sign languages are complete natural languages with their own lexicon, grammar, and non-manual grammatical markers. Computational work on sign languages has historically concentrated on isolated sign language recognition (ISLR), in which a short, trimmed clip is mapped to a single gloss or word. This framing is convenient for data collection, annotation, training, and evaluation, because every example contains exactly one linguistic unit, allowing a model to focus on discriminative visual cues such as hand shape, hand location, movement direction, facial expression, and body posture. However, a single sign is a coarse unit of meaning: in isolation it often fails to carry the speaker’s full communicative intent.

In real interaction, signers do not produce discrete words separated by clean boundaries. They produce continuous streams of signs in which movements are chained together, co-articulated, and modulated in rhythm. For this reason we shift the target from word-level recognition to sentence-level sign language translation (SLT), where the input is an entire continuous utterance and the output is a complete text sentence [1], [2]. Operating at the sentence level lets the model learn contextual relations between signs, resolve ambiguities that arise when a sign appears alone, and produce output that can be shown as text or spoken directly, rather than a sequence of gloss labels.

This shift comes at a cost. Sentence-level SLT must process longer video with more frames, contend with fuzzy boundaries between signs, and rely on richer supervision. Non-manual channels—facial expression, mouthing, and upper-body motion—become obligatory signals rather than optional cues; a model that attends only to the hands discards a substantial part of the linguistic information.

Deployment adds a further constraint. A useful assistive system should respond quickly, yet translating each sign the instant it appears is both error-prone and unnatural. We therefore adopt an utterance-then-translate design: the system continuously ingests a signing stream, detects when an utterance ends, and emits a translation shortly afterwards. The engineering goal is to balance the contextual accuracy of a sentence-level model against a response latency low enough for near real-time communication.

To realize this design we build on SHuBERT [3], a self-supervised model that learns sign representations from roughly a thousand hours of sign-language video. We do not train a new sign encoder from scratch. Instead, we attach a byte-level decoder and fine-tune the translation stack on a resource-constrained, uniformly sampled subset of the How2Sign benchmark [4], updating only a small set of parameters while SHuBERT remains frozen. The resulting model is computationally heavy, which is precisely why the core of this work is a hardware-aware real-time stack. Our reference implementation uses a Raspberry Pi 4B with a camera, display, speaker, battery, and enclosure as the interaction endpoint, while CPU/GPU inference is offloaded to a backend. The network contract is client-agnostic: a browser, phone, or laptop can replace the Raspberry Pi as long as it samples, chunks, and uploads frames through the same interface.

1.0.0.1 Contributions.

The primary focus of this paper is real-time deployment, and our contributions reflect that emphasis.¹ (i) Real-time streaming inference (main contribution). We present a streaming pipeline for continuous signing that combines chunked ingestion, bounded queues, parallelized landmark perception, temporal reordering, and a sentence-boundary state machine. With the model and decoding settings held fixed, the optimized runtime reduces mean post-finalization response latency by 27.71% and P95 by 27.03%. (ii) Hardware-aware edge deployment. We realize the interaction loop on a portable Raspberry Pi 4B appliance with camera capture, local display, and speech output, while offloading heavy inference to a GPU backend. The hardware prototype is a reference client, not a protocol dependency; browser, phone, and laptop clients use the same HTTPS interface. (iii) How2Sign fine-tuning. We fine-tune a frozen SHuBERT encoder with a ByT5 decoder on a uniformly sampled 9,872-example How2Sign subset using QLoRA, obtaining validation BLEU 16.7 and test BLEU 15.9, together with a test BLEURT score of 44.7.

2 Related Work↩︎

2.0.0.1 From isolated recognition to translation.

Early sign-language systems targeted ISLR, mapping trimmed clips to glosses. Generic skeleton-based video encoders such as PoseConv3D [5] provide motion representations, while NLA-SLR [6] specifically targets isolated sign recognition; neither method directly generates fluent sentences. Neural SLT reframes the problem as sequence-to-sequence generation: [1] introduced neural sign language translation, and [2] jointly modeled recognition and translation with transformers. Yin and Read’s STMC-Transformer [7] strengthened translation sequence modeling and highlighted limitations of gloss-based representations. Later work removed the dependence on gloss supervision through visual–language pretraining and gloss-free end-to-end training [8], [9]. Community benchmarks such as WMT-SLT [10] have further standardized evaluation. Our system follows the gloss-free, translation-oriented line: it maps continuous signing directly to sentences without an intermediate gloss decoding step, avoiding error propagation from a gloss recognizer into the language model.

2.0.0.2 Self-supervised representations.

Self-supervised pretraining has reshaped speech and vision. HuBERT [11] learns speech representations by masked prediction over clustered targets. SHuBERT [3] carries this idea to sign language: it is a self-supervised transformer encoder trained on roughly 1,000 hours of American Sign Language video that predicts clustered targets over multiple visual streams—hands, face, and body pose—and reaches state-of-the-art results across sign language translation, isolated sign recognition, and fingerspelling detection. SHuBERT is the backbone of our system: we take its pretrained multi-stream encoder as a frozen temporal sign representation and adapt only a byte-level decoder on top of it for sentence-level translation (Section 3). On the vision side, DINOv2 [12] produces robust general-purpose image features from Vision Transformers [13] without labels; we use it as the frozen regional feature extractor that supplies SHuBERT’s per-frame inputs.

2.0.0.3 Text generation and efficient adaptation.

T5 [14] casts NLP tasks as text-to-text generation; ByT5 [15] removes the fixed subword vocabulary by operating on bytes, which suits short, free-form target sentences. Because full fine-tuning of such models is expensive, we adopt parameter-efficient adaptation: LoRA [16] injects low-rank updates into attention projections, and QLoRA [17] combines this with 4-bit quantization so that adaptation runs on modest hardware.

2.0.0.4 Perception and evaluation.

For per-frame geometry we rely on MediaPipe [18] and its pose estimator [19]. For translation quality we report BLEU [20] and BLEURT [21].

3 Translation Model↩︎

Our model does not translate raw pixels directly. Instead it first converts video into structured visual signals—hand and face geometry, upper-body pose, cropped hand and face regions, and their evolution over time—and then passes those signals through three learned stages. Figure 1 summarizes the cascade. We describe each stage in turn and clarify which components are learned during adaptation and which are frozen.

3.1 Geometric perception↩︎

The first stage is a visual perception layer that estimates the structure of the signer in every frame. We use MediaPipe [18] with three branches: a face landmarker, a hand landmarker for both hands, and a pose estimator for the upper body. These branches are not translation models; they produce geometric coordinates that tell the system where to look before any deep encoder runs.

The three branches play complementary linguistic roles. The hands are the primary carrier of lexical content. The face supplies expression, mouthing, and other non-manual markers that can flip a statement into a question or change sentiment. The pose stabilizes the interpretation of movement direction and helps detect whether the signer is active. Because these channels carry distinct information, none can be dropped without risking a loss of meaning, particularly in sentences where expression or trajectory is grammatically load-bearing.

3.2 Regional visual features↩︎

Rather than feeding whole frames to the temporal encoder, we use the landmarks to localize the hands and face, crop those regions, and resize them to \(224\times224\). Each region is encoded by DINOv2 [12], a self-supervised Vision Transformer, into a 384-dimensional feature vector that compresses hand shape, finger orientation, relative hand position, and facial expression. This has two benefits. First, discarding the background reduces environmental noise. Second, DINOv2 features are more stable than raw pixels, letting the downstream translator focus on the semantic evolution of a sign rather than re-solving low-level vision.

3.3 Temporal sign encoding↩︎

Given per-frame features, the model must understand how they change over time. This is the role of SHuBERT [3], a sequence encoder for sign language inspired by HuBERT [11]. SHuBERT consumes four channels—left hand, right hand, face, and upper-body pose. Hands and face are represented by their 384-d DINOv2 vectors, while pose is reduced to a low-dimensional geometric vector derived from nose, shoulder, elbow, and wrist points. The channels are projected into a common space, fused per time step, and encoded by a transformer [22] into a 768-dimensional contextual representation of the whole utterance. Because this representation integrates past and future frames within the same sentence, it disambiguates signs that share a hand shape but differ in trajectory or context.

3.4 Text generation↩︎

The final stage is ByT5 [15], an encoder–decoder text generator. After SHuBERT produces the contextual representation, a projection layer maps it into the decoder’s latent space, and ByT5 generates the English sentence step by step. Operating at the byte level frees the output from a fixed subword vocabulary, which is convenient for the short, free-form target sentences in our data. During real-time inference we use a small beam width to keep decoding fast; in preliminary runs, larger beams did not improve short sentences and sometimes added trailing artifacts.

3.5 Parameter-efficient adaptation↩︎

We initialize the translation stack from pretrained SHuBERT and ByT5, then fine-tune it for SLT on the uniformly sampled How2Sign working subset [4] with QLoRA [17]. This is task adaptation on How2Sign rather than training the sign encoder from scratch. The base ByT5 weights are quantized to 4-bit NF4 with an FP16 compute type, and low-rank LoRA [16] updates (\(r=4\)) are trained on the query and value projections together with the SHuBERT\(\rightarrow\)ByT5 projection layer. SHuBERT itself is frozen. This choice preserves the self-supervised sign prior, restricts trainable parameters to components that directly affect generation, and makes fine-tuning affordable: the whole run fits on two NVIDIA T4 GPUs (Section 5).

4 Real-Time System and Deployment↩︎

The learned model is accurate but heavy, so practical responsiveness depends on hardware–software co-design. We separate three concerns: (1) a capture and interaction client, (2) a CPU runtime for ingestion, perception, ordering, and boundary detection, and (3) a GPU translation stage. This separation lets the Raspberry Pi prototype provide a complete assistive interface without requiring the full model to execute on-device, while preserving the same backend contract for other client types. Figure 2 shows the resulting runtime.

4.1 Hardware prototype and client abstraction↩︎

The reference deployment is a portable Raspberry Pi 4 Model B appliance that implements the complete user-facing loop: a USB webcam captures the signer, an on-device screen displays the translated sentence, and a small speaker driven by a Class-D amplifier reads it aloud. A battery pack, step-down converter, cooling fan, and 3D-printed enclosure make the unit portable and thermally stable (Table 1). The Raspberry Pi coordinates capture, local I/O, and networking; CPU/GPU-intensive perception and translation are executed on a backend reached over Wi-Fi/HTTPS.

The hardware is deliberately separated from the client protocol. The backend requires only that a client sample frames, encode them, group them into chunks, and upload them over HTTPS. The Raspberry Pi is therefore a concrete deployment and evaluation client, while a browser, phone, or laptop can replace it without changing the model or server-side streaming logic. Throughout the remainder of the paper, client denotes this generic interface and Raspberry Pi prototype denotes the physical reference implementation.

4.2 Chunked ingestion↩︎

The client samples the incoming video below the native camera rate to reduce computation while preserving motion, normalizes each frame to a fixed size, compresses it to JPEG, and groups a small number of frames into a chunk. Chunking amortizes network cost relative to sending single frames, yet the chunks remain short enough for near real-time response. Concretely, the client samples at 15 fps, normalizes to \(640\times480\) at JPEG quality 72, and groups ten consecutive frames per chunk (Section 5).

4.3 Sentence-boundary detection↩︎

Because we translate whole utterances, the system must decide when an utterance begins and ends. A boundary detector runs as a state machine (Figure 3). In the waiting state no stable signing is observed. When hands and motion become sufficiently stable, it moves to recording and accumulates features. A brief, unstable appearance is treated as a false start and returns the machine to waiting. When the hands disappear, drop, or become nearly still beyond a threshold, the machine enters a short hold to avoid emitting duplicate sentences from a single ending gesture, and finally finalized, at which point the model generates the sentence. When the user stops manually, the system still drains already-received chunks before closing the session.

4.4 Temporal synchronization↩︎

Signing is sequential, so frame order matters as much as frame content. With several workers running in parallel, chunks may finish out of order. A reorder buffer holds completed chunks and releases them to the translator only when all earlier chunks are ready, so the system exploits parallelism while preserving the temporal logic of the utterance.

4.5 Latency optimizations↩︎

Table 2 lists the optimizations. Early loading decodes and enqueues data as soon as enough sub-frames arrive, instead of waiting for a full request. A bounded queue caps depth to keep latency stable when the arrival rate exceeds instantaneous throughput; under overload the system coalesces incoming data into the most recent chunk rather than letting a backlog grow. Two perception workers increase CPU throughput, and within each worker the face, hand, and pose modules execute concurrently through a four-thread pool, so wall-clock perception time is governed by the slowest module rather than the sum of all three. The reorder buffer preserves correctness, and a safe-drain rule finishes in-flight chunks before producing the final output so that no end-of-utterance content is lost.

5 Experimental Setup↩︎

5.1 Data↩︎

How2Sign [4] is a large-scale continuous American Sign Language corpus of instructional videos in which each example pairs a signing clip with an English sentence. The original corpus contains substantially more data than we could process within the available compute and storage budget. We therefore extract a uniformly sampled working subset of 9,872 clip–sentence pairs from the available corpus. The extracted subset contains 9 signers and is split into 7,000 training, 1,121 validation, and 1,751 test examples (Table 3). The subset retains the sentence-level clip–text task while fitting the available resource budget. Of the 9,872 target sentences, 9,746 are unique (\(\approx\)​1.013 examples per sentence), indicating low exact-sentence repetition. We report this as a dataset statistic rather than as evidence of an open-vocabulary setting.

5.2 Training configuration↩︎

Fine-tuning on the sampled How2Sign working subset follows the QLoRA recipe of Section 3.5. Table 4 lists the configuration. Training runs on two NVIDIA T4 GPUs with an effective batch size of 512, formed from a per-device batch of 16 accumulated over 16 steps across 2 devices. The base ByT5 is loaded in 4-bit NF4 with FP16 compute, LoRA rank is \(r=4\), and only the ByT5 query/value projections and the sign\(\rightarrow\)text projection are trainable; SHuBERT is frozen.

5.3 Runtime deployment configuration↩︎

The reference latency path uses the Raspberry Pi 4B client described in Section 4.1. It samples video at 15 fps, resizes frames to \(640\times480\), encodes JPEG at quality 72, and sends chunks over Wi-Fi/HTTPS to the backend. The optimized setting uses ten frames per chunk and five concurrent requests. These measurements therefore include the reference hardware/client path, while the server API remains unchanged for browser, phone, or laptop clients.

5.4 Metrics↩︎

5.4.0.1 Quality.

We report BLEU [20] and BLEURT [21]. BLEU measures \(n\)-gram overlap and applies a brevity penalty, while BLEURT is a learned BERT-based metric intended to capture semantic and fluency similarities that lexical overlap can miss. BLEURT is reported for the held-out test split.

5.4.0.2 Latency.

For real-time behavior we measure the delay between the moment the system finalizes an utterance and the moment it emits text. For evaluated clip \(i\), \[L_i = t_{\text{output},i} - t_{\text{finalize},i},\] and the mean latency over \(N\) clips is \(\tfrac{1}{N}\sum_i L_i\). We also report the 95th percentile (P95). For an aggregate latency statistic \(M\) (mean or P95), the relative reduction is \[\text{Reduction}_{M}(\%) = \frac{M_{\text{base}}-M_{\text{opt}}}{M_{\text{base}}}\times 100 .\]

5.4.0.3 Boundary-detection parameters.

The real-time evaluation uses the state-specific timing and normalized-motion thresholds in Table 5. Separate windows for absent, lowered, and idle hands avoid forcing all end-of-utterance conditions into a single timeout.

6 Results and Analysis↩︎

6.1 Translation quality↩︎

Table 6 reports the final quality after fine-tuning on the sampled How2Sign working subset. BLEU is 16.7 on validation and 15.9 on test; the held-out test split additionally obtains BLEURT 44.7. The validation and test BLEU scores differ by 0.8 points. BLEURT provides a complementary semantic measure beyond surface \(n\)-gram overlap.

6.2 Real-time latency↩︎

We compare a sequential baseline configuration with the optimized configuration; the two differ only in runtime settings (Table 7) and share the same translation model, image quality, sampling rate, frame size, and final beam width, so any difference is attributable to the streaming stack rather than the model.

Runtime evaluation covers every example in the complete 9,872-example working subset. The optimized system reduces mean latency from 1.873 s to 1.354 s (a 27.71% reduction) and P95 latency from 2.919 s to 2.130 s (27.03%); see Table 8 and Figure 4. The aggregate results therefore reflect the full sampled working corpus rather than a small sentence-level evaluation slice. This is a systems workload measurement; it should not be interpreted as an estimate of translation generalization on the training portion of the subset.

6.3 Where the gains come from↩︎

The dominant factor is perception time. In the sequential baseline, each frame waits for the face, hand, and pose branches in turn, so per-frame cost approximates the sum of the branches (about \(107\)–\(111\) ms). Running the MediaPipe modules concurrently through a four-thread pool makes wall-clock perception time track the slowest module instead, dropping it to about \(45\)–\(48\) ms/frame. Early loading and concurrent requests further shorten the client–server wait by letting the pipeline begin before a full request has arrived. Throughout these runs the raw and prepared queues never exceeded a depth of one and never saturated, confirming that the bottleneck was perception latency rather than queueing.

6.4 Qualitative examples↩︎

Table 9 shows representative sentences with their baseline and optimized latencies. The optimized system is consistently faster across short and long sentences, and longer utterances retain proportionally larger absolute savings, consistent with per-frame perception being the main cost.

6.5 Recommended defaults↩︎

From these experiments we recommend, as project defaults: two perception workers; a four-thread intra-worker pool for MediaPipe; ten frames per chunk; five concurrent client requests; a final beam of two; and state-specific finalization windows of 400/500/900 ms for hands disappearing, lowering, or remaining idle. This configuration keeps the translation model and decoding settings fixed while materially reducing latency and remaining simple enough to reason about as a real-time system.

7 Conclusion↩︎

We presented a hardware-aware, sentence-level sign language translation system whose main objective is responsive deployment. The translation stack is fine-tuned with QLoRA on a uniformly sampled 9,872-example How2Sign subset while SHuBERT remains frozen, reaching BLEU 16.7 on validation and, on test, BLEU 15.9 and BLEURT 44.7. The central contribution is the runtime and deployment design: a portable Raspberry Pi 4B interaction client paired with a client-agnostic HTTPS interface and an offloaded CPU/GPU backend. Chunked ingestion, bounded queues, concurrent perception, temporal reordering, and sentence-boundary detection reduce mean post-finalization response latency from 1.873 to 1.354 seconds (27.71%) and P95 from 2.919 to 2.130 seconds (27.03%). The hardware prototype demonstrates the complete camera-to-text-and-speech loop, while the generic client contract allows the same backend to support browser, phone, or laptop capture.

Limitations↩︎

Several limitations remain. First, the system responds after an utterance ends rather than translating incrementally while the user is still signing; truly incremental decoding is future work. Second, post-finalization latency is measured over the complete 9,872-example working subset under controlled deployment conditions; absolute numbers may shift under network variability or heavier concurrent load. Third, although the client protocol is generic, the reported hardware-backed latency measurements use the Raspberry Pi reference client; browser, phone, and laptop clients require separate benchmarking. Finally, the multi-stage pipeline is computationally non-trivial, and pushing the full model to run entirely on the edge remains challenging.

Ethics Statement↩︎

The intended use of this work is accessibility: supporting communication for deaf and hard-of-hearing signers and non-signers. Because the input is video of a person, privacy is a first-order concern; the system de-identifies faces by graying and blurring while retaining the eyes and mouth, which are linguistically necessary for non-manual markers, thereby reducing identity exposure without discarding signal. Responsible deployment requires informed consent for data collection, attention to signer diversity so the model does not systematically fail for underrepresented signers or dialects, and clear communication that automatic translations can be wrong and should not be relied upon in high-stakes settings without human confirmation.

Acknowledgments↩︎

We thank Dr.Ninh Khánh Duy for his valuable feedback and guidance throughout this project.

References↩︎

---

1. Code is available at https://github.com/NguyenDoanThanhHoang/Toward-Real-Time-Sentence-Level-Sign-Language-Translation.↩︎