Keywords: gesture detection, multimodal agent, attention mechanism, human–robot interaction

1 Introduction↩︎

Recent advances in human–robot interaction have accelerated the transition from conventional button-based interfaces to multimodal systems that combine speech, vision, and gestures. Gesture recognition provides a natural and intuitive communication channel and has broad potential in service robotics, assistive systems, and smart homes. Nevertheless, reliable gesture-based interaction in dynamic environments remains difficult because lightweight detectors may lose accuracy for small or occluded gestures, while resource-constrained robots often lack the semantic reasoning and task-planning capabilities required for complex instructions.

Deep-learning-based object detectors, particularly the YOLO family [1]–[8], have substantially advanced real-time gesture recognition. Other representative architectures, including Faster R-CNN [9], SSD [10], and DETR-based models [11], [12], also provide strong detection performance. However, illumination variations, background clutter, partial occlusion, and diverse gesture poses continue to degrade accuracy. Attention mechanisms [13]–[17] and improved bounding-box regression losses [18]–[22] have therefore been explored to strengthen feature representation and localization.

At the system level, traditional robots frequently rely on a single input modality and fixed command mappings, which limit their ability to interpret context and decompose complex tasks. Vision–language models (VLMs) and large language models (LLMs) [23]–[27] offer new opportunities for multimodal reasoning. Models such as CLIP [28], BLIP [29], [30], LLaVA [31], [32], Flamingo [33], and MiniGPT-4 [34] have demonstrated strong visual–linguistic understanding. Robotics frameworks such as SayCan [35], VoxPoser [36], RT-2 [37], and CLIPort [38] further illustrate how language-conditioned reasoning can support action generation. Directly deploying such computationally demanding models on platforms such as Raspberry Pi or TonyPi, however, remains challenging [39], [40].

To address these limitations, this study develops a unified cloud–edge interaction framework that combines refined gesture perception with coordinated LLM and VLM agents. The main contributions are as follows:

• Refined gesture detection. We construct YOLO-DC by inserting CBAM into the neck of YOLO11n and using DIoU loss for bounding-box regression. Experiments on the public and custom datasets show improved precision and mAP@0.5 in scenarios involving small targets and complex backgrounds.

• Robot-oriented multimodal agent architecture. We design a dual-agent architecture that integrates gesture detection, visual question answering, language understanding, rule-based validation, and finite-state-machine control. This architecture supports cross-modal interpretation and structured action planning.

• Cloud–edge collaborative deployment. Computationally intensive inference and reasoning are performed in the cloud, while the TonyPi platform handles sensing, communication, motion control, and feedback. A standardized JSON protocol connects the perception, reasoning, and control components.

2 Related Work↩︎

2.1 Gesture Detection via Deep Learning↩︎

The YOLO series has become a widely used foundation for real-time gesture detection because of its favorable balance between accuracy and efficiency [1]–[4]. Two-stage detectors such as Faster R-CNN [9] and Mask R-CNN [41] often provide high accuracy but can be computationally expensive for edge devices. Achieving strong generalization, high localization accuracy, and real-time performance under illumination changes, background noise, and diverse gesture poses remains an active research problem.

A variety of bounding-box regression objectives have been proposed, including focal loss [42], IoU loss [22], generalized IoU (GIoU) [18], complete IoU (CIoU) and distance IoU (DIoU) [19], and SIoU [20]. Attention mechanisms such as CBAM [13], squeeze-and-excitation modules [14], non-local networks [16], and coordinate attention [17] have also been used to emphasize informative spatial regions and feature channels. Meanwhile, ResNet [43], VGG [44], Inception [45], Vision Transformer [46], Swin Transformer [47], and ConvNeXt [48] have advanced feature extraction, whereas MobileNet [49], [50] and EfficientNet [51] provide lightweight alternatives.

This work builds on YOLO11n and targets small or partially occluded gestures by embedding CBAM in the neck and emphasizing center-distance convergence through DIoU loss. The modifications introduce limited additional parameter overhead while preserving the one-stage detection pipeline.

2.2 Multimodal Agents for Robotic Interaction↩︎

Conventional robotic systems often use locally deployed, single-modal interfaces that restrict semantic interpretation and task planning in dynamic environments. VLMs and LLMs provide a more flexible basis for integrating visual and linguistic information. CLIP [28], BLIP [29], [30], LLaVA [31], [32], Flamingo [33], and MiniGPT-4 [34] have demonstrated capabilities in visual question answering and multimodal content understanding. DINOv2 [52] and masked autoencoders [53] have further improved self-supervised visual representation learning.

In robotics, SayCan [35], VoxPoser [36], RT-2 [37], and CLIPort [38] connect language-conditioned reasoning with task execution. Reasoning strategies such as chain-of-thought prompting [54], ReAct [55], and generative agents [56] support decomposition and sequential decision-making. Nevertheless, fully local deployment of large multimodal models remains difficult on resource-limited robotic hardware. We therefore employ a cloud–edge dual-agent architecture in which the cloud performs model inference and semantic reasoning, while the robot retains low-latency sensing and motion control.

2.3 Security Considerations for Cloud–Edge Interaction↩︎

Cloud–edge robotic systems expose networked interfaces, remote inference services, and structured inter-component messages, making security validation relevant to reliable deployment. Penetration-testing workflows provide a systematic basis for identifying system-level attack surfaces and validating defensive controls [57]. At the software-analysis level, recent smart-contract studies combine semantic representations, control-flow graphs, interface information, and interaction context to identify vulnerabilities [58], [59], while chain-specific analyses emphasize that security risks and suitable tools vary across platforms [60]. Cross-contract analysis further shows the value of examining complete interaction paths rather than isolated components [61]. Although these studies target information systems and blockchain software rather than robotics, their emphasis on interface validation, interaction-aware analysis, and end-to-end attack paths motivates strict message-schema validation, command whitelisting, and cross-module consistency checks in the proposed framework.

3 Method↩︎

3.1 System Architecture↩︎

The proposed framework separates computationally demanding inference from real-time robot control. As shown in Figure 1, the local TonyPi platform captures voice and image inputs, compresses and transmits the data, receives a structured action plan, and executes the corresponding motion sequence. The cloud runs YOLO-DC for predefined gesture detection, a VLM agent for open-vocabulary scene understanding, and an LLM agent for intent interpretation and task decomposition. Their outputs are fused and checked by a rule engine and finite-state machines (FSMs) before being returned to the robot.

3.2 YOLO-DC: Refined Gesture Detection↩︎

YOLO11n provides efficient one-stage detection, but small gestures and cluttered backgrounds can still cause weak feature responses and imprecise localization. YOLO-DC addresses these limitations by integrating CBAM [13] into the neck and using DIoU loss [19] for bounding-box regression. The network structure is shown in Figure 2.

3.2.1 CBAM Attention Mechanism↩︎

CBAM sequentially applies channel and spatial attention to enhance informative feature responses. For an input feature map \(F\), the operations are \[\begin{align} F' &= M_c(F) \otimes F, \tag{1}\\ F'' &= M_s(F') \otimes F', \tag{2} \end{align}\] where \(M_c\) and \(M_s\) denote the channel and spatial-attention maps, respectively, \(\otimes\) denotes element-wise multiplication, and \(F''\) is the refined output feature map. The CBAM structure is illustrated in Figure 3.

Although the C2PSA module in YOLO11n already provides spatial attention, its feature aggregation may remain insufficient for small gestures. We therefore place CBAM after the C2PSA stage in the neck. Channel attention emphasizes feature channels associated with gesture cues, and spatial attention highlights target regions. According to the model summary, the modification adds approximately \(0.1\) million parameters.

3.2.2 DIoU-Based Bounding-Box Regression↩︎

CIoU considers overlap, center-point distance, and aspect-ratio consistency. In this study, DIoU is used to place greater emphasis on center-distance convergence and to simplify the geometric penalty term. The DIoU loss is \[\mathcal{L}_{\mathrm{DIoU}} = 1-\mathrm{IoU} + \frac{\rho^2(\mathbf{b},\mathbf{b}^{\mathrm{gt}})}{c^2}, \label{eq:diou}\tag{3}\] where \(\mathrm{IoU}\) is the intersection-over-union ratio, \(\mathbf{b}\) and \(\mathbf{b}^{\mathrm{gt}}\) are the center points of the predicted and ground-truth boxes, respectively, \(\rho(\cdot,\cdot)\) is the Euclidean distance, and \(c\) is the diagonal length of the smallest enclosing box. The center-distance penalty remains informative even when the two boxes do not overlap and encourages direct convergence of their centers.

3.3 Cloud–Edge Collaborative Task Flow↩︎

3.3.1 Local Perception and Transmission↩︎

A voice command initiates the interaction process. The local perception layer captures the audio signal, digitizes it in pulse-code modulation format, applies noise reduction, and performs speech-to-text conversion. Simultaneously, the camera captures image frames, which are denoised, resized, and JPEG-encoded. A JPEG quality factor of 85 is used to reduce communication overhead. In the reported test, this setting reduced the image size from approximately \(3.2\) MB to \(1.9\) MB while retaining sufficient visual detail for gesture detection and scene understanding. The processed text and image data are then transmitted to the cloud.

3.3.2 Cloud Fusion and Action Planning↩︎

In the cloud, YOLO-DC detects predefined hand gestures, while the VLM agent performs scene understanding and visual question answering. The LLM agent fuses the voice command, gesture results, and visual description to infer user intent and generate a candidate action sequence. For example, a query such as “What is in front of you?” produces a natural-language response grounded in the visual analysis. A behavior command such as celebrate produces a structured action token such as celebrate() together with a spoken response.

A rule engine verifies action compatibility, and FSMs enforce execution order and state consistency. Cross-agent communication uses a unified JSON schema containing the request type, recognized intent, object classes, colors, positions, confidence scores, response text, and action sequence. The local decision layer parses this message, triggers text-to-speech feedback when required, and invokes the corresponding motion-control interface.

These mechanisms provide functional validation at the message and command levels, but they do not constitute a complete cybersecurity assessment. A production deployment should additionally implement authenticated and encrypted communication, access control, rate limiting, audit logging, and systematic penetration testing [57].

4 Experiments↩︎

4.1 Experimental Setup↩︎

4.1.1 Datasets↩︎

Two datasets are used for gesture detection:

• Public dataset: a publicly available gesture dataset containing 600 images from six classes. Representative samples are shown in Figure 5.

• Custom dataset: a dataset collected to approximate practical robot-interaction conditions. It contains three classes: good, left, and palm. Videos are recorded with a Raspberry Pi camera, and one frame is extracted every five frames. Representative samples are shown in Figure 4.

For each dataset, the images are divided into training and validation subsets at a ratio of 2:1. The validation subset is used for model selection and for reporting the results in this paper.

4.1.2 Implementation Details↩︎

Experiments are conducted using an NVIDIA GeForce RTX 4090 GPU and PyTorch 2.1.0. The models are trained for 300 epochs with stochastic gradient descent, a momentum of 0.937, a batch size of 32, and an initial learning rate of 0.01. Input images are resized to \(640\times640\) pixels. A dropout rate of 0.5 is used in the training configuration to mitigate overfitting.

4.2 Gesture Detection Results↩︎

4.2.1 Comparison with Mainstream Models↩︎

YOLO-DC is compared with YOLOv5n, YOLOv8n, and the baseline YOLO11n on both datasets. As reported in Table 1, YOLO-DC achieves the best performance across all reported metrics. On the public dataset, it reaches a precision of 98.9% and an mAP@0.5 of 90.7%. On the custom dataset, it achieves a precision of 95.0% and an mAP@0.5 of 92.7%. Relative to YOLO11n on the custom dataset, the precision increases by 16.0 percentage points and the mAP@0.5 increases by 6.0 percentage points.

7pt

Visual comparisons on the public and custom datasets are shown in Figure 6 and 7, respectively. Compared with YOLO11n, YOLO-DC generally provides more precise bounding boxes and higher confidence scores in cluttered or partially occluded scenes.

4.2.2 Ablation Study↩︎

An ablation study is conducted on the public dataset to evaluate the contributions of DIoU and CBAM. As shown in Table 2, replacing the baseline regression objective with DIoU increases recall from 94.0% to 95.0% and mAP@0.5 from 88.4% to 89.6%. Adding CBAM alone increases mAP@0.5 from 88.4% to 88.9%, although precision and recall decrease slightly. Combining both modifications produces the highest precision, recall, mAP@0.5, and calculated \(F_1\) score. The \(F_1\) values in Table 2 are calculated from the reported precision and recall as \(F_1 = 2PR/(P + R)\).

8pt

The training curves in Figure 8 show stable convergence of the box-regression, classification, and distribution focal losses. Precision, recall, and mAP increase progressively and stabilize during training.

4.3 System-Level Evaluation↩︎

The multimodal interaction system is evaluated on the TonyPi platform in terms of task-execution success, response diversity, and user satisfaction.

4.3.1 Task-Execution Success Rate↩︎

Three command categories are evaluated: single-action, composite-action, and vision-dependent tasks. Table 3 shows that single-action commands achieve a success rate of 95%, composite-action commands achieve 88%, and vision-dependent tasks achieve 82%. The rule engine rejects 90% of the tested invalid action combinations, such as incompatible consecutive motion commands. Failures in vision-dependent tasks are primarily associated with visual-interpretation errors, communication delays, and response-format inconsistencies. The measured mean VLM response latency is approximately 210 ms. Figure 9 presents an example of the multimodal agent’s scene-understanding output.

4.3.2 Feedback Diversity and User Satisfaction↩︎

To assess response diversity, 500 responses are generated from 50 predefined templates. The resulting repetition rate is below 5%. In addition, 30 participants rate four predefined interaction scenarios—Wave, Kick Ball, Twist, and Celebrate—on a five-point scale. Table 4 reports the rating distributions. The mean scores calculated from these distributions are 3.83, 3.67, 3.70, and 3.57, respectively, yielding an overall mean of 3.69 out of 5.

5 Conclusion and Future Work↩︎

This paper presents a cloud–edge multimodal human–robot interaction framework that combines YOLO-DC gesture detection with coordinated LLM and VLM agents. YOLO-DC integrates CBAM into the neck of YOLO11n and uses DIoU loss to improve localization and feature discrimination for small or partially occluded gestures. On the public dataset, the model achieves a precision of 98.9% and an mAP@0.5 of 90.7%, on the custom dataset, it achieves 95.0% precision and 92.7% mAP@0.5. The cloud–edge architecture separates computationally intensive perception and reasoning from local sensing and control. System-level evaluation produces success rates of 95%, 88%, and 82% for single-action, composite-action, and vision-dependent tasks, respectively. The participant study yields an overall mean satisfaction score of 3.69 out of 5. These results support the practical feasibility of combining refined gesture detection, multimodal scene understanding, and structured action planning on resource-constrained robotic platforms.

Future work will focus on model compression, pruning, quantization, and knowledge distillation to reduce cloud dependence and enable more inference on edge hardware. Additional sensory modalities, including depth and tactile information, will be incorporated to improve robustness under occlusion and rapid environmental change. Larger-scale evaluations in industrial inspection, smart-home assistance, education, and healthcare scenarios will also be conducted to examine generalization and long-term usability.

Acknowledgments↩︎

The authors would like to express their deepest gratitude to their undergraduate instructor, Xia Ran. AI-based tools were used for language polishing during manuscript preparation.

References↩︎