1 Introduction↩︎

Recent technological advancements in analyzing human behavior have led to innovations that enhance daily life. A promising area is daily human activity monitoring [2], [3]. Traditional methods using visual sensors or wearable devices have notable success but face challenges. Wearable sensors [4], [5] are often cumbersome and uncomfortable for prolonged or high-intensity use. Visual sensors [6], [7] struggle with fixed angles, occlusions, glare, and low-light conditions, compromising image quality. Additionally, privacy concerns and regulations limit the feasibility of camera-based pose estimation where privacy is essential.

Sensor-based approaches, particularly those using WiFi CSI, have emerged as promising alternatives for human pose estimation (HPE), addressing limitations of conventional camera-based methods. For instance, RF-Pose in [8] and [9] used radio frequency signals to generate skeletal representations of the human body, showing remarkable results, especially when bodies are occluded. Additionally, an infrared-based approach [10] has proven effective for pose estimation under diverse lighting conditions, including both bright and dark environments. Despite these advancements, the adoption of these techniques faces significant hurdles due to stringent requirements, such as complex hardware (e.g. a \(16\)+\(4\) intricately designed and synchronized T-shaped antenna array) and demanding RF signals (e.g. frequency modulated continuous wave with a broad signal bandwidth of \(1.78\) GHz). The implementation of these approaches necessitates expensive equipment, challenging their practical application for everyday use.

Recently, WiFi-based approaches in [11] and [12] have emerged as promising solutions to these challenges. By leveraging ubiquitous WiFi signals, bistatic configuration methods provide a non-intrusive, privacy-aware alternative to traditional approaches. Electromagnetic interactions between WiFi signals and the human body, including reflection, refraction, and penetration, enable the extraction of pose features from channel state information (CSI) for HPE. However, real-world deployment remains challenging. CSI often lacks the granularity needed for fine-grained pose estimation, mixing human motion with environmental effects and producing unrealistic postures. Moreover, maintaining continuity and smoothness in synthesized skeleton movements is essential for reliable activity representation and usability.

1.1 Motivation↩︎

The widespread utilization of WiFi infrastructure and open-source software has significantly increased interest in exploiting CSI for human sensing [13], [14]. Many works have particularly focused on enhancing the accuracy of human sensing through WiFi-based HPE. To facilitate deployment and cost-effectiveness, the authors in [15] leveraged readily available WiFi devices to perform effective HPE tasks. Guo et al. [16] demonstrated that off-the-shelf WiFi devices could capture 2D images of human skeletons. However, this approach is limited to single-perspective poses, resulting in suboptimal performance when annotations are constrained. In contrast, the work in [15] introduced WiPose, the first 3D HPE system relying on the CSI amplitude, and has shown superior performance in controlled experimental settings. However, WiPose requires subjects to remain stationary during experiments and depends on nine distributed antennas, restricting its practicality for everyday applications such as smart homes and health monitoring. Meanwhile, the authors in [17] used two standard WiFi devices to collect CSI data, utilizing 30 subcarriers, and integrated these devices with a fully convolutional network to predict 2D human poses. Nevertheless, the key findings indicate that limited subcarrier resolution compromises result precision, highlighting the need for a WiFi acquisition system with more antennas and subcarriers. This improvement, though, introduces increased computational complexity when processing high-dimensional input with deep neural networks.

Recently, a novel approach called MetaFi was introduced in [18] to address the low resolution of subcarriers by utilizing commodity WiFi devices equipped with \(114\) subcarriers. Although this approach achieved impressive resolution, its effectiveness in HPE tasks was limited by constraints within its neural network architecture, particularly learnability and complexity. Recently, the MetaFi++ solution in [19] employed commodity WiFi devices featuring one transmission and three receiver antennas, each supporting \(114\) subcarriers. The associated neural network imposed a substantial computational burden with approximately \(26.42\) million parameters, compromising its scalability for large-scale implementation. Consequently, there remains a critical need to develop a high-resolution, easy-to-deploy commodity WiFi system, coupled with a high-performance yet lightweight neural network capable of efficiently processing high-dimensional WiFi data. These advancements are essential to unlocking the full potential of WiFi-based human sensing in real-world applications.

1.2 Main Contributions↩︎

To overcome the aforementioned shortcomings, we propose a novel WiFi-based network called “WiLHPE,” designed to generate the lightweight and efficiently human pose estimation from WiFi CSI signals and RGB images. As shown in Fig. 1, WiLHPE adopts a teacher-student architecture with two networks. A pre-trained vision-based teacher extracts pose labels from RGB images to supervise training, while the WiFi-based student predicts poses from raw WiFi signals. During testing, the student functions as a real-time pose estimator without requiring cameras.

The student network employs a channel-frequency dynamic convolution network (CF-DyNet) to learn representations from CSI efficiently. Within CF-DyNet, we introduce CF-DyConv, a convolutional architecture with multi-dimensional attention that weights input features by importance to focus on informative patterns. In CSI systems, data consists of frequency amplitude sequences across multiple antennas and short time frames (e.g., \(10\) ms in MM-Fi [20]). Although temporal resolution is limited, the channel domain provides spatial diversity to resolve occlusions, while the frequency domain captures fine-grained spectral signatures related to joint positions. Therefore, our architecture jointly exploits channel and frequency information, adaptively optimizing convolutional kernels for improved performance.

Our main contributions can be summarized as follows:

• Within the WiLHPE system, we introduce CF-DyNet sub-networks to effectively process WiFi CSI signals for human pose estimation tasks. This framework includes a novel CF-DyConv architecture that utilizes dynamic convolutional filters to adjust their kernel size based on input data characteristics. The introduced neural network is lightweight, requires no pre-processing, has high learning capability, and features a simple structure.

• We leverage the Tree-Structured Parzen Estimator (TPE) [21] method to optimize the critical hyperparameters of the considered CF-DyNet sub-networks, avoiding reliance on heuristics. This allows WiLHPE to achieve optimal results without extensive hyperparameter search time.

• We perform intensive experiments to demonstrate the effectiveness of the proposed WiLHPE framework, achieving \(85.96\%\) and \(94.27\%\) at \(\text{PCK}_{50}\) on MM-Fi [20] and WiPose [22] datasets, respectively, while balancing model performance and computational cost in human pose estimation tasks. Additionally, WiLHPE with the proposed CF-DyConv architecture exhibits robust performance on the MM-Fi dataset under challenging conditions, outperforming traditional CNNs.

1.3 Paper Organization and Mathematical Notations↩︎

The rest of this paper is organized as follows. Section 2 provides the literature review on HPE tasks and attention-based convolution. Section 3 gives the background of the CSI and dynamic convolution. The proposed WiLHPE framework is presented in Section 4. Section 5 details HPE datasets, evaluation metrics, results, and discussions, and Section 6 concludes the paper.

Mathematical notations: Throughout this paper, matrices and vectors are written as bold uppercase and lowercase letters, respectively, while the scalar number is denoted in lowercase. \(\odot\) and \(*\) denote the multiplication operation and the convolution operation, respectively. \(\delta(\cdot)\) is the Dirac delta function. The notation \(x \sim \mathcal{CN}(0,\sigma^2)\) implies that \(x\) is a circularly-symmetric complex Gaussian random variable with zero mean and variance \(\sigma^2\). \(\|\cdot\|\) represents the vector’s Euclidean norm. Finally, \(\mathbb{R}\) denotes a set of real numbers.

2 Related Work↩︎

WiFi sensing: Leveraging the widespread availability of WiFi infrastructure, numerous studies have explored the capabilities of WiFi devices over recent decades. This has enabled various sensing tasks such as indoor localization and tracking [23], [24], walking speed estimation [25], [26], inference of breathing and heartbeats [27], [28], human identification [29], [30], human presence and movement detection [31], [32], and gesture recognition [33], [34]. However, these tasks are generally considered coarse-grained person perception tasks, as they primarily detect broad actions or states without delving into detailed body dynamics. In contrast, our proposed WiLHPE framework focuses on finer-grained personal perception tasks, which involve more detailed and precise analyses of human activities. For instance, we aim to represent human pose landmarks, allowing us to capture specific body postures and movements, thereby advancing the capabilities of human pose estimation. We emphasize that our work focuses solely on HPE tasks, rather than other WiFi sensing tasks, due to the differing dataset characteristics associated with each task.

Human pose estimation: In the field of computer vision (CV), 2D HPE has been extensively explored using powerful deep learning techniques and abundantly annotated datasets from standard cameras [35], [36]. Additionally, the works [37] and [38] have implemented specialized devices to enhance the estimation performance of human motion. Despite the impressive results achieved by vision-based methods, their deployment faces significant challenges, including poor lighting, occlusion, blurred images, and privacy concerns. Efforts to address privacy issues using light-based methods (e.g. [39]) have been made, but these methods exhibit limited performance in dark or occluded conditions. While LiDAR has shown fine-grained person detection capabilities [40], the high cost and power requirements limit its practicality for daily and household use. In contrast, the WiFi-based approaches are capable of addressing these limitations by ensuring privacy and maintaining functionality in various lighting and occlusion scenarios. In particular, RF-based strategies have emerged as promising solutions to overcome these challenges. For instance, RFCapture [41] can outline human bodies even through walls, while RFPose [42] and its 3D variant [9] can extract 2D and 3D skeletons from RF signals with the aid of visual data. Nevertheless, these methods often require sophisticated hardware (e.g. a \(16\)+\(4\) intricately designed and synchronized T-shaped antenna array) and specific signal conditions (e.g. frequency modulated continuous wave with a broad signal bandwidth of \(1.78\) GHz), limiting their practical applications. In contrast, our approach framework demonstrates that robust HPE can be achieved using accessible, high-resolution WiFi systems and lightweight neural network architectures, making it more feasible for real-world deployment.

Attention-based CNN: Very recently, there has been extensive research on developing novel CNN architectures to enhance the performance of traditional CNNs, particularly through the use of attention mechanisms. A representative work is SENet [43], which introduced the Squeeze-and-Excitation (SE) module to model interdependencies among feature channels. Building on SE’s aggregation-recalibration design, [44] proposed Dynamic Convolution (DyConv), dynamically combining multiple kernels based on input attention. However, DyConv relies only on channel-domain information, limiting kernel-space modeling. To address this, ODConv [45] and F-DyConv [46] were developed as more advanced variants. Similarly, SKNet [47] fuses multi-scale features using attention across different kernel sizes, but still focuses mainly on domain-specific information, reducing versatility. F-SKNet further improved SKNet, particularly for speaker verification. In HPE-based wireless sensing, recent works in [48]–[51] employed a kernel-selective CNN to reduce model complexity by focusing on informative Wi-Fi signals, whereas [52] considered this setting in an edge-server architecture.

In this work, we propose a convolutional architecture that goes beyond channel-only processing by exploiting spatial-spectral features in both channel and frequency dimensions of short-frame CSI data. Given CSI’s limited temporal granularity, our attention mechanism selectively extracts discriminative features across these domains, enabling robust and efficient representation learning.

3 Preliminaries↩︎

3.1 Channel State Information (CSI)↩︎

In WiFi-based networks, CSI captures the unique characteristics of a communication channel, providing detailed insights into how WiFi signals propagate in a specific physical environment. This includes various phenomena such as multipath effects, reflections, diffraction, and scattering. Modern commercial WiFi devices, following the IEEE 802.11 standard, utilize OFDM at the physical layer and often employ multiple antennas. CSI data meticulously records the amplitude attenuation and phase shift of multipath signals across each communication subcarrier for every antenna pair. This detailed information allows for the detection of subtle human movements that affect WiFi signal propagation. The frequency domain represents the signal as the channel impulse response, enabling precise analysis and optimization of wireless communication performance, which can be expressed as \[h(\tau)=\sum_{l=1}^{L}\alpha_{l}e^{j\phi_l}\delta(\varphi-\varphi_{l})\] where \(\alpha_l\), \(\phi_l\), and \(\varphi\) are the amplitude, phase, and time delay of the \(l\)-th multipath component, respectively; \(L\) indicates the total number of multipath components given in the channel; and \(\delta(\varphi)\) denotes the Dirac delta function.

3.2 Overview of Dynamic Convolution↩︎

Basic Concept. Dynamic convolution (DyConv) was introduced in [44] to enhance the representational capacity of conventional convolution by dynamically adjusting the convolution kernel in response to input characteristics. Firstly, DyConv extracts attention weights based on the input and then performs a weighted summation of \(n\) basis kernels using these attention weights to derive the optimal kernel for the input, as shown in Fig. 2. Given the input and output channel numbers, denoted as \(C_{\mathtt{in}}\) and \({C_{\mathtt{out}}}\), respectively, DyConv can be mathematically expressed as: \[\boldsymbol{Y}=g(\boldsymbol{W}^{T}\boldsymbol{X}+\boldsymbol{b})=(\sum_{i=1}^{n}\alpha_{i}(\boldsymbol{W}_{i} +\boldsymbol{b}_i))\ast\boldsymbol{X}\label{eq:32basicDC}\tag{1}\] where \(\alpha_{i}=\pi_i(\boldsymbol{X})\), \(\boldsymbol{X}\in\mathbb{R}^{H\times W\times C_{\mathtt{in}}}\) and \(\boldsymbol{Y}\in\mathbb{R}^{H\times W\times C_{\mathtt{out}}}\) denote the input features and output features with \(H\) and \(W\) being the height and width of the channel, respectively. Herein, \(\mathbf{W}_i \in\mathbb{R}^{H\times W\times C_{\mathtt{in}} C_{\mathtt{out}}}\) is the \(i\)-th convolutional kernel, \(\alpha_{i}\in \mathbb{R}\) is the attention weight for the \(i\)-th convolutional kernel which is computed by the attention function \(\pi_{i}(\boldsymbol{X})\) conditioned on the input features, and \(\ast\) denotes the convolution operation.

Limitation Discussions. According to 1 , DyConv comprises two key elements: Convolutional kernels \(\mathbf{W}_i\), and attention function \(\pi_{i}(\boldsymbol{X})\) used to calculate their attention scalars \(\alpha_{i}\). With \(n\) convolutional kernels, the kernel space is characterized by four dimensions: The spatial kernel size \(k \times k\), input channel number \(C_{\mathtt{in}}\), output channel number \(C_{\mathtt{out}}\) for each kernel, and the total number of kernels, \(n\). We note that \(\pi_{i}(\boldsymbol{X})\) assigns a single attention scalar \(\alpha_{i}\) to each \(\mathbf{W}_i\), resulting in all \(C_{\mathtt{out}}\) filters \(\mathbf{W}_{i}^{m}\in\mathbb{R}^{k\times k \times C_{\mathtt{in}}}\) with \(m \in \{1,\cdots, C_{\mathtt{out}}\}\) sharing the same attention value for the input \(\boldsymbol{X}\). In simpler terms, DyConv disregards the spatial, input channel, and output channel dimensions of \(\mathbf{W}_i\), focusing solely on one remaining dimension. This narrow focus might result in inefficient utilization of the kernel space when devising attention mechanisms to empower \(n\) dynamic convolutional kernels. Meanwhile, each frame of WiFi-CSI data exists concurrently in both the frequency and channel domains, signifying pose information (i.e. frequency) from a specific viewpoint. The physical environment predominantly influences each frequency component independently at a channel level. Consequently, traditional DyConv may struggle to capture features from WiFi data effectively.

4 Proposed WiLPHE Framework↩︎

We first describe CSI preprocessing, then present the CF-DyConv implementation and complexity analysis. Finally, we introduce the WiLHPE architecture, including hyperparameter optimization and learning objectives.

4.1 Channel-Frequency Dynamic Convolution (CF-DyConv)↩︎

CF-DyConv is a key component for building an efficient neural network that learns and weights kernels based on contextual cues using diverse attention mechanisms. As illustrated in Fig. 3, the CF-DyConv architecture is divided into two distinct flows. The first flow generates new weights for 2D convolution by applying attention mechanisms in the frequency and channel domains. Initially, the input data undergoes average pooling along the frequency dimension, followed by FC layers with ReLU and softmax activations to produce frequency attention weights. A similar process is applied in the channel domain to generate channel attention weights. These attention weights are then combined to create new weights for the 2D convolutional layer. The second flow uses these newly obtained weights to perform 2D convolution on the input data, extracting relevant features efficiently.

Pre-Processing CSI Data. Let us start by considering the CSI dataset which contains \(G\) samples denoted as \(\{C_{i}\text{\}}_{i=1}^{G},\) where \(C\in\mathbb{R}^{T_{x}\times R_{x}\times F\times T}\) with \(T_x\), \(R_x\) and \(F\) are the numbers of transmitting antennas, receiving antennas and subcarriers, respectively, while \(T\) is the frame time. To simplify, we consider the \(T_x \times R_x\) dimension as the input channels \({C_{\mathtt{in}}}\). Consequently, the CSI data is represented as X \(\in\) \(\mathbb{R}^{F \times T \times {C_{\mathtt{in}}}}\), where \(F\) and \(T\) dimensions represent the values in the frequency domain and the time domain, respectively.

CF-DyConv. For a given X, we adopt a SENet [43] including the squeeze and excitation stages. SENet generates attention scalars for convolutional kernels along the channel and frequency dimensions, denoted \({\alpha_{c}(\mathbf{X})}\) and \({\alpha_{f}(\mathbf{X})}\), respectively, as shown in Fig. 3. The squeeze stage uses the global average pooling (GAP) to obtain the global spatial information, which encapsulates the global information into the channel-wise feature vector \(\boldsymbol{z}_c \in \mathbb{R}^{C_{\mathtt{in}}}\), such as \(\text{GAP}_{\texttt{\texttt{chan}}}\), and frequency-wise feature vector \(\boldsymbol{z}_f \in \mathbb{R}^F\), such as \(\text{GAP}_{\texttt{\texttt{freq}}}\). The channel-wise and frequency-wise feature vectors are computed as \[\begin{align} \boldsymbol{z}_c&=\text{GAP}_{\texttt{chan}}(\boldsymbol{X})=\frac{1}{F\times T}\sum_{f=1}^{F}\sum_{\text{t=1}}^{T}\boldsymbol{X}(f,t) \tag{2}\\ \boldsymbol{z}_f&=\text{GAP}_{\texttt{freq}}(\boldsymbol{X})=\frac{1}{C\times T}\sum_{c=1}^{C}\sum_{\text{t=1}}^{T}\boldsymbol{X}(c,t) \tag{3} \end{align}\] respectively. The excitation stage then leverages the information aggregated during the squeeze stage by using two FC layers with activation functions. These layers generate normalized attention weights for \(n\) convolution kernels, acting as a gating mechanism to capture the nonlinear interactions among multiple channels. The process can be summarized as follows: \[\begin{align} \alpha_c &= f_{\texttt{excitation}}(\mathbf{z}_{c}) =\sigma\Big(\mathbf{W}_{c2}\delta\left(\mathbf{W}_{c1} \mathbf{z}_c/\tau\right)\Big) \tag{4}\\ \alpha_f &= f_{\texttt{excitation}}(\mathbf{z}_{f}) =\sigma\Big(\mathbf{W}_{f2}\delta\left(\mathbf{W}_{f1} \mathbf{z}_f/\tau\right)\Big) \tag{5} \end{align}\] where \(\sigma\) and \(\delta\) are the softmax and ReLU activation functions, respectively; \(\mathbf{W}_{c1} \in \mathbb{R}^{\frac{C_{\mathtt{in}}}{r} \times C_{\mathtt{in}}}\), \(\mathbf{W}_{c2} \in \mathbb{R}^{C_{\mathtt{in}} \times \frac{C_{\mathtt{in}}}{r}}\), \(\mathbf{W}_{f1} \in \mathbb{R}^{\frac{F}{r} \times F}\) and \(\mathbf{W}_{f2} \in \mathbb{R}^{F \times \frac{F}{r}}\) are the learnable weights of the excitation stage. The reduction ratio \(r\) acts as a bottleneck to reduce the input dimensionality of the first FC layer. The temperature \(\tau\) is utilized to smooth the attention distribution. The attention weights \(\alpha_c\) and \(\alpha_f\) are fed to the input \(\boldsymbol{X}\) to produce more informative kernels. Note that \(\alpha_c\) is an attention scalar that weights filters according to their importance in capturing channel information, while \(\alpha_f\) weights filters based on their significance in capturing frequency information. By leveraging this attention mechanism, we enhance the kernel’s ability to capture features that are most informative and best suited for the specific dataset.

To generate convolutional kernels that capture diverse information, we combine channel and frequency scalars with the kernels after learning the attention and then aggregate the kernels. As shown in Fig. 3, the attention weights \({\alpha_c}\) and \(\alpha_f\) are multiplied with convolution kernels, providing diversity to the kernels. The output feature \(\mathbf{Y}\in\mathbb{R}^{F \times T \times C_{\mathtt{out}}}\) is computed as \[\mathbf{Y}=\mathbf{X}\ast\Big(\sum_{i=1}^{n}\alpha_{ci}\odot\alpha_{fi}\odot\mathbf{W}_{i}\Big)\label{eq:post-process4}\tag{6}\] where \(n\) is the number of convolution kernels, \(\mathbf{W}_{i}\) is the \(i\)-th convolution kernels, and \(\odot\) denotes the multiplication operations.

Complexity Analysis. The proposed CF-DyConv introduces two additional computational stages: \(i)\) computing the attention weights \(\alpha_c(\mathbf{X})\) and \(\alpha_f(\mathbf{X})\), and \(ii)\) aggregating stages. In particular, the computation cost of the first stage is estimated as \(FT C_{\mathtt{in}} + 2 C_{\mathtt{in}}^2/r + 2 F^2/r + n C_{\mathtt{in}} /r + n F /r\) Mult-Adds, while the second stage aggregates \(n\) convolution kernels with kernel size \(D_k \times D_k\), \(C_{\mathtt{in}}\) input channels and \(C_{\mathtt{out}}\) output channels requires \(n C_{\mathtt{in}} C_{\mathtt{out}} D_k^2 + n C_{\mathtt{out}}\) Mult-Adds with \(D_k\) being the kernel size. Therefore, the total computational cost of the proposed CF-DyConv is \(FT C_{\mathtt{in}} + 2 C_{\mathtt{in}}^2/r + 2 F^2/r + n C_{\mathtt{in}} /r + n F /r\) \(+\) \(n C_{\mathtt{in}} C_{\mathtt{out}} D_k^2 + n C_{\mathtt{out}}\) Mult-Adds. In comparison, basic convolution requires \(FTC_{\mathtt{in}}C_{\mathtt{out}}D_k^2\) Mult-Adds. Thus, the additional cost of our approach is negligible if \(n\ll FT\).

We note that the two key benefits of CF-DyConv to enhance the overall network performance of the WiLHPE framework are:

• \(\noindent\boldsymbol{Comprehensive Exploitation}\): CF-DyConv captures both frequency and channel information by applying dynamic convolutional filters to each frame of the input. This comprehensive approach enables the model to extract a wide range of features from the data, significantly enhancing performance in HPE tasks.

• \(\noindent\boldsymbol{Dynamic Convolutional Filters}\): CF-DyConv utilizes an attention-based mechanism to adaptively adjust its weights based on the input data. This dynamic adjustment allows the model to learn more effective feature representations with minimal additional computational cost.

4.2 WiLHPE-based Network Architecture↩︎

As shown in Fig. 4, the teacher network employs a computer vision model, such as HRNet-w48 [19], [20] to extract 2D key points from multi-view frames. These 2D key points serve as the ground truth \(\mathbf{y}\) for estimating human poses across various modalities, including pixel coordinates \((a, b)\) represented as \(\boldsymbol{y}=\{(a_{i},b_{i})|i\in[1,\cdots,P]\}\), where \(P\) denotes the number of key points. The student network is a neural network designed to predict human poses under the supervision of the teacher network. Specifically, the student network is trained using supervised learning, with the true labels provided by the teacher network’s ground truth. In Fig. 6, this network primarily comprises basic CNNs, \(M\) repeated sub-networks known as CF-DyNet to extract the feature transformation network. In particular, CF-DyNet consists of a basic CNN and a CF-DyConv, each with distinct input channels denoted as \(n_1\) and \(n_2\), respectively. Each CNN block is followed by an AvgPooling layer of size 2. The features extracted from CF-DyNet are then passed to the decoder module, which includes an FC layer with a ReLU activation function and an output size of \(2P\), as illustrated in Fig. 6. This block generates predictions of the human body’s key points \(\hat{\mathbf{y}}=\{(\hat{a}_{i},\hat{b}_{i})|i\in[1,\cdots, P]\}\).

TPE Algorithm: As previously mentioned, two key parameters influence performance: The reduction ratio \(r\) and the temperature \(\tau\). The former adjusts the capacity and computational cost of the attention module, while the latter in 4 and 5 is used to control the sparsity of the weights. The selection of these parameters and their effects are explored in the ablation experiments. In addition, to avoid manually tuning the hyperparameters of WiLHPE, we customize the TPE algorithm [21] to automatically search the hyperparameters. Instead of evaluating \(p(y|x)\) as in the traditional algorithms, TPE models \(p(x|y)\) and \(p(y)\). By two such densities, TPE defines \(p(x|y)\) as follows: \[\begin{align} p(x|y) = \begin{cases} \ell(x) & \text{if } y < y^*, \\ g(x) & \text{if } y \geq y^*. \end{cases} \end{align}\] The TPE algorithm identifies \(y^*\) as a specific quantile \(\gamma\) of the observed \(y\) value, ensuring that \(p(y < y^*) = \gamma\), where \(p(y)\) is the probability distribution of \(y\). The algorithm aims to classify sample points into the superiority set \(g(x)\) and the inferiority set \(\ell(x)\). The optimal parameters are determined through iterative successive updates to these sets, with the goal of maximizing the Expected Improvement (EI) function, which is defined as \[\begin{align} \text{EI}_{y^*}(x) &= \frac{\gamma y^* \ell(x) - \ell(x) \int_{-\infty}^{y^*} p(y) \, dy}{\gamma \ell(x) + (1 - \gamma)g(x)}\nonumber\\ &\propto \left( \gamma + \frac{g(x)}{\ell(x)}(1-\gamma) \right)^{-1}\label{eq:post-process11} \end{align}\tag{7}\] following the Bayes transformation rule [21]. Equation 7 implies that maximizing EI requires minimizing the ratio \(\frac{g(x)}{\ell(x)}\). Thus, each iteration returns a set of values for \(x\) that decrease \(g(x)\) and increase \(\ell(x)\). The hyperparameters in the set are evaluated based on the objective function. This process is then iteratively repeated to complete the hyperparameter search. The customized TPE algorithm is summarized in Algorithm 5.

4.3 Learning Objective↩︎

We employ the mean-squared error (MSE) between the prediction and the ground truth produced by the visual pose estimation model as the loss function, defined as follows: \[\mathcal{L}_{\mathtt{MSE}} = \lVert \hat{\mathbf{y}} - \mathbf{y} \rVert_2^2 \label{eq:post-process5}\tag{8}\] where \(\mathbf{\hat{y}}\) and \(\mathbf{y}\) are the predicted value and target value, respectively. Instead of using the pose adjacency matrix as done in the previous studies [17], we found that the MSE loss function yields better performance. This choice is motivated by the high granularity of the CSI data. Both the MM-Fi and WiPose datasets contain numerous subcarriers of CSI data for each antenna pair, resulting in higher-resolution data. Consequently, the task becomes more manageable even with a simple loss function.

5 Results and Discussions↩︎

To evaluate WiLHPE’s performance, we first describe the experimental setup, including the dataset, evaluation metrics and implementation details. We then conduct a comparative analysis between WiLHPE and the current state-of-the-art (SOTA) approaches to assess its effectiveness. Finally, we conduct the robustness analysis and ablation study to demonstrate the impact of CF-DyConv.

5.1 Experimental Setup↩︎

5.1.1 Dataset↩︎

We use two HPE datasets: MM-Fi [20] and WiPose [22]. A brief description of both datasets is provided in Table 1.

• MM-Fi: encompasses \(17\) skeleton points of pose annotations, derived from both camera sensor and WiFi CSI data. This dataset involves \(40\) human subjects participating in \(27\) action categories, collecting from \(14\) daily activities and \(13\) rehabilitation exercises. Furthermore, MM-Fi implements three protocols: \(i)\) Protocol 1 (P1) focuses on \(14\) daily activities, such as picking up objects and raising arms, \(ii)\) Protocol 2 (P2) involves \(13\) activities performed in fixed locations, like limb extensions and \(iii)\) Protocol 3 (P3) includes all \(27\) activities mentioned above. Each protocol utilizes two data splitting strategies: \(i)\) Setting 1 (S1) randomly divides all video samples into training and testing sets with a ratio of \(3:1\) and \(ii)\) Setting 2 (S2) splits the data by subject, allocating \(32\) subjects for training and \(8\) for testing.

• WiPose: contains \(166\)k packets with pose annotations for \(18\) skeleton points. CSI data is captured from \(12\) actions (i.e. wave, walk, throw, run, push, pull, jump, crouch, circle, sit down, stand up and bend) of \(12\) participants. The data collection system is equipped with a \(5\) GHz wireless router and a WiFi receiving host, acquiring CSI data with \(30\) subcarriers per antenna pair, resulting in each video frame corresponding to five CSI data. The training and testing sets are randomly chosen from all data.

5.1.2 Evaluation Metrics↩︎

We evaluate the system performance using three widely used metrics in HPE tasks.

• Percentage of Correct Keypoint (PCK) [53] measures the proportion of correctly detected keypoints that fall within a certain distance threshold of the ground truth keypoints \(\alpha\), which is expressed as \[\text{PCK}_{\alpha} = \frac{1}{KP}\sum_{i=1}^{K}\sum_{j=1}^{P}I\left ( \frac{\left \| pd_{i,j} - gt_{i,j} \right \|_{2}^{2} }{ \sqrt{rs^2 + lh^2} } \leq \alpha \right ) \label{eq:post-process6}\tag{9}\] where \(I(\cdot)\) represents a binary indicator function, \(K\) is the number of test frames, and \(j \in \{1, 2, \ldots, P\}\) denotes the index for body joints. The distance \(\lVert pd_{i,j} - gt_{i,j} \rVert^2_2\) signifies the Euclidean distance between the predicted results (\(pd\)) and the ground truth (\(gt\)) at the \(j\)-th body joint in the \(i\)-th test frame. Next, the term \(\sqrt{rs^2 + lh^2}\), known as the torso length, measures the distance between the right shoulder (\(rs\)) and the left hip (\(lh\)) and is used to normalize the prediction error. Given the indicator function \(I(\cdot)\) and the stricter threshold \(\alpha\), the higher the PCK score, the better the network performance.

• Mean Per Joint Position Error (MPJPE) [53], measured in millimeters (mm), evaluates the average error between the predicted and ground-truth positions for all joints after aligning the pelvis of the estimated and true pose. Therefore, a lower MPJPE metric indicates better network performance. The MPJPE is calculated as \[\text{MPJPE} = \frac{1}{KP} \sum_{i=1}^{K} \sum_{j=1}^{P} \lVert pd_{i,j} - gt_{i,j} \rVert_2 . \label{eq:post-process1}\tag{10}\]

• Procrustes-Aligned Mean Per Joint Position Error (PA-MPJPE), which is actually the MPJPE after aligning the predicted results to the ground truth through a Procrustes transformation, provides an additional alignment method to improve the accuracy of 3D body joint position estimation compared to only using basic MPJPE.

5.1.3 Benchmark Schemes↩︎

For performance comparison, we consider several SOTA WiFi-based approaches. In particular, we consider MetaFi++ [19], which is currently recognized as the most effective model for the HPE task on MM-Fi. Other well-established HPE methods are implemented, including Wi-Pose [15], Wi-Mose [54], WiSPPN [17] and WiLDAR [55].

5.1.4 Adding Noise↩︎

To analyze the impact of unfavorable conditions on WiLHPE, we introduce additional noise into the testing data to assess our model’s performance. Specifically, we employ Additive White Gaussian Noise (AWGN), denoted as \(e\), where \(e \sim \mathcal{CN}(0,\sigma_{e}^{2})\) with \(\sigma_{e}^{2}\) being the variance to simulate natural randomness typical in real-world scenarios.

We further investigate the impact of noise through adversarial attacks. Specifically, we implement the Fast Gradient Sign Method (FGSM) [56], which manipulates input images to deceive the model by introducing carefully crafted noise. Here, we apply a single-step, untargeted perturbation (FGSM-U). This method aims to increase the network loss for a given input-label pair \((\boldsymbol{I}, \boldsymbol{y})\) by adding noise with a bounded \(l_\infty\) norm. The noise is computed as the scaled \(\varepsilon\) sign of the gradient with respect to the objective, resulting in a perturbed image \(\mathbf{I}^p\): \[\mathbf{I}^p = \mathbf{I} + \varepsilon \cdot \text{sign}(\nabla_{\mathbf{I}} \mathcal{L}(f(\mathbf{I}, \theta), \mathbf{y})) \label{eq:post-process12}\tag{11}\] and iterative un-targeted perturbation FGSM-U-N. This process involves \(N\) iterations to produce the final perturbed image \(\boldsymbol{I}^p\) starting from I. The perturbed image \(\boldsymbol{I}^p_i\) at the \(i\)-th iteration for un-targeted is given as: \[\mathbf{I}_{i}^{p} = C\varepsilon\Big(\mathbf{I},\, \mathbf{I}_{i-1}^p + \alpha.\text{sign}\big(\nabla_{\mathbf{I}_{i-1}^p} \mathcal{L}(f(\mathbf{I}^{p}_{i-1}, \theta)\mathbf{y})\big)\Big) \label{eq:post-process13}\tag{12}\] where \(C_{\varepsilon}(x_0, x)\) clips \(x\) to \([x_0 - \varepsilon, x_0 + \varepsilon]\). The iterative attacks are generally more damaging compared to single-step attacks because they can distort the image in a highly non-linear manner over multiple iterations.

5.1.5 Implementation Details↩︎

All schemes are implemented using PyTorch, leveraging an Intel 13-core i9-13900K CPU (3 GHz) and a GeForce RTX 4070 GPU. The dataset is divided into three parts: 70% for training, 15% for validation, and \(15\%\) for testing. The model is trained for 50 epochs using either the stochastic gradient descent (SGD) with momentum algorithm or the Adam optimizer. The batch size, learning rate, and momentum are set to \(32\), \(0.001\) and \(0.9\), respectively. These parameters are optimized using the TPE algorithm, as outlined in Algorithm 5. Additionally, the learning rate follows a lambda decaying schedule, gradually decreasing from the initial value to 0.

For the student network, we start by implementing a standard CNN with a kernel size of \(3\) and a MaxPooling block with a size of \(2\). Next, we incorporate three identical CF-DyNet blocks (\(M\) = \(3\)), as illustrated in Fig. 6. Each CF-DyNet block consists of a basic CNN with a kernel size of \(3\) and number channels \(n_1\) of \(64\), followed by a CF-DyConv layer with a kernel size of \(5\), \(n\) (as defined in Eq. 6 ) being set to \(3\), and number channels \(n_2\) of \(128\). Within each CF-DyNet block, an AvgPooling block with a size of \(2\) is used to reduce the feature size while preserving feature quality. The decoder consists of FC layers and ReLU activation functions, as shown in Fig. 6, with hidden layers containing \(64\) units.

5.2 Performance Evaluation↩︎

5.2.1 Results on MM-Fi↩︎

We first evaluate the performance of WiLHPE on the MM-Fi dataset using the \(\text{PCK}_{20}\), \(\text{PCK}_{30}\), \(\text{PCK}_{40}\), and \(\text{PCK}_{50}\) metrics. Note that we omit the results for lower PCK thresholds due to the limitations in annotation quality and benchmark consistency of the MM-Fi dataset, as discussed in [20]. As shown in Table ¿tbl:tab:table3a?, WiLHPE exhibits impressive overall pose estimation performance, with average scores of \(79.22\%\) and \(85.96\%\) for \(\text{PCK}_{40}\) and \(\text{PCK}_{50}\), respectively. Specifically, our model shows exceptional precision in estimating the lower part of the human body (e.g. torso, hips, knees, and feet), achieving over 94% accuracy for \(\text{PCK}_{50}\). Notably, the accuracy of these specific body joints remains about \(70\%\) under more stringent conditions at \(\text{PCK}_{20}\). These results are influenced by the activities performed in MM-Fi, which primarily focus on upper body movements. Consequently, challenges arise in predicting parts of the body, such as hands and elbows, where movements occur with high intensity.

In Fig. 7 and Table 2, we further evaluate the effectiveness of WiLHPE using \(\text{PCK}_{\alpha}\), MPJPE, and PA-MPJPE metrics across different protocols. As depicted in Fig. 7, WiLHPE performs better performance in P3-S1 with \(\text{PCK}_{\alpha}\), while showing the least effectiveness in P2-S2. As observed in Table 2, WiLHPE achieves optimal results in P3-S1 with an MPJPE of \(140.71\) mm, and the weakest performance in P2-S2 at approximately \(187\) mm. However, our model achieves favorable PA-MPJPE results in P2-S2, with a value of 86.18mm. These results underscore the considerable influence of protocol configurations on the WiLHPE’s performance.

5.2.2 Results on WiPose↩︎

We now focus on individual body parts using the \(\text{PCK}_{\alpha}\) metric. As shown in Table ¿tbl:tab:table3b?, WiLHPE demonstrates impressive effectiveness, achieving average percentages of \(94.27\%\) and \(89.41\%\) for \(\text{PCK}_{50}\) and \(\text{PCK}_{20}\), respectively. Notably, WiLHPE exhibits exceptional results for body parts such as knees and hips (e.g \(100\%\) at \(\text{PCK}_{50}\)). Notably, the proposed scheme maintains commendable accuracy scores even under stricter criteria such as \(\text{PCK}_{5}\) and \(\text{PCK}_{10}\), with approximately \(69.79\%\) and \(82.46\%\), respectively. This underscores the robustness of WiLHPE’s performance on specific body joints when evaluated against rigorous standards on the WiPose dataset. This accomplishment surpasses that of MM-Fi due to differences in the methods used to acquire raw signals in the two datasets. Compared to MM-Fi, the WiPose system, with its simplified configuration and lower-intensity human activities, provides higher accuracy and reliability in HPE tasks.

5.2.3 Comparison with Benchmark Schemes↩︎

Table 3 presents results for WiLHPE and the considered benchmark schemes. Firstly, the proposed WiLHPE scheme on the MM-Fi dataset surpasses the performance of MetaFi++, which previously achieved SOTA results in HPE tasks. WiLHPE improves \(\text{PCK}_{20}\) and \(\text{PCK}_{50}\) scores by approximately \(4\%\) and \(2\%\), respectively, and reduces MPJPE by less than \(12\) mm. Notably, WiLHPE maintains significantly lower complexity, requiring fewer than \(15\) times the resources compared to MetaFi++. Notably, our method on the MM-Fi dataset exhibits a complexity level similar to WiLDAR, one of the most lightweight models in human recognition tasks, with approximately \(1.6\) million parameters. Nevertheless, WiLHPE offers superior performance over WiLDAR while requiring only half the number of FLOPs.

Next, on the WiPose dataset, WiLHPE outperforms other benchmark schemes in all \(\text{PCK}_{\alpha}\) scores, MPJPE, and PA-MPJPE metrics. Specifically, WiLHPE shows the best performance with improvements of around \(6\%\) and \(16\%\) compared to PerUnet and MetaFi++ in \(\text{PCK}_{5}\), respectively. Additionally, WiLHPE achieves MPJPE and PA-MPJPE results of \(19.21\) mm and \(15.85\) mm, respectively, significantly lower than its counterparts. Meanwhile, WiLHPE requires minimal computational resources, with only \(3.49\) million parameters, slightly larger than WiLDAR, while exhibiting outstanding efficiency compared to other benchmark schemes. These comparisons underscore the remarkable effectiveness and low complexity of our approach in HPE tasks.

5.2.4 Qualitative Result↩︎

Fig. 8 shows qualitative results by providing visual representations of human skeletons captured in various scenarios from the MM-Fi dataset. Specifically, we focus on generating 2D poses and subsequently transforming them into 3D poses by adding a constant vector as the third dimension, facilitating easier visualization. The proposed WiLHPE approach consistently and reliably generates human poses across a spectrum of daily activities and rehabilitation exercises.

5.3 Robustness Analysis↩︎

5.3.1 Impact of AWGN Noise↩︎

In Fig. 9, we vary the values of \(\sigma_{e}^2\) from \(0\) to \(0.5\) to assess the WiLHPE’s robustness. Additionally, we implement WiLHPE with a basic CNN to highlight the adaptive capabilities of CF-DyConv in severe noise conditions. As seen, CF-DyConv is only slightly affected by the AWGN noise and achieves good performance with accuracies of approximately \(78\%\), \(52\%\) and \(40\%\) at \(\text{PCK}_{50}\), \(\text{PCK}_{30}\), and \(\text{PCK}_{20}\) with \(\sigma_e^2=0.5\), respectively. In contrast, the basic CNN exhibits a drastic decline as \(\sigma_{e}^2\) gradually increases. In particular, the \(\text{PCK}_{50}\) and \(\text{PCK}_{20}\) scores of the basic CNN is around \(75\%\) and \(35\%\), respectively, at \(\sigma_{e}^2=0.2\), dropping dramatically to approximately \(60\%\) and \(9\%\), respectively, as \(\sigma_{e}^2\) increases to \(0.5\).

Next, we evaluate the WiLHPE’s robustness with the MPJPE and PA-MPJPE metrics. Fig. 10 illustrates the response of CF-DyConv to different noise levels, revealing a marginal impact with a maximum MPJPE value below \(200\) mm under the most severe condition, e.g. \({\sigma_e^2} = 0.5\). In contrast, the basic CNN exhibits more serious susceptibility to noise, as evidenced by its MPJPE value surpassing that of our network in the worst-case scenario, reaching about \(250\) mm at \({\sigma_e^2} = 0.5\). Regarding the PA-MPJPE metric, there is a notable increase of \(15\) mm for the basic CNN model, rising from \(90\) mm to \(105\) mm as \({\sigma_e^2}\) increases from \(0\) to 0\(.5\). Meanwhile, the PA-MPJPE value of CF-DyConv experiences only a slight increase, reaching \(95\) mm at \({\sigma_e^2} = 0.5\). These results confirm the robust adaptability of CF-DyConv in severe conditions with the presence of AWGN noise.

5.3.2 Adversarial Attack↩︎

To evaluate the potential impact of adversarial noise, we perform FGSM-U and FGSM-U-N on P3-S1 MM-Fi dataset, as presented previously in Section 5.1. We vary the number of iterations \(N\) from \(0\) to \(12\). Fig. 11 shows that WiLHPE under the influence of FGSM-U-N maintains a stable \(\text{PCK}_{50}\) score with a wide range of \(N\) and \(\varepsilon=0.2\), consistently exceeding approximately 78% \(N=10\). Furthermore, WiLHPE experiences only a slight drop in \(\text{PCK}_{30}\) and \(\text{PCK}_{20}\) score, decreasing by about 10% when \(N\) increases from \(0\) to \(12\). In contrast, adversarial noise significantly impacts the basic CNN’s performance, with the \(\text{PCK}{50}\) score dropping below 70% at \(N=4\). Moreover, the accuracy of this model shows a remarkable degradation in the \(\text{PCK}{20}\) and \(\text{PCK}{30}\) score, up to around \(20\%\), as \(N\) varies from \(0\) to \(12\). It is noteworthy that the influence of adversarial noise remains relatively consistent from \(N=4\) onwards at the fixed value of \(\varepsilon\).

Fig. 12 shows that the WiLHPE’s performance under the presence of adversarial noise FGSM-U-N with \(N=4\) is slightly degraded as \(\varepsilon\) increases. Specifically, the \(\text{PCK}_{50}\) score reaches \(77\%\) with \(\varepsilon = 0.3\). Moreover, we observe a reduction of around \(12\%\) in \(\text{PCK}_{20}\) as \(\varepsilon\) gradually increases from \(0\) to \(0.3\). Meanwhile, WiLHPE with basic CNN is seriously affected by adversarial noise FGSM-U-N. This model experiences a dramatic decline in \(\text{PCK}_{50}\) as \(\varepsilon\) increases (e.g \(\text{PCK}_{50}=50\%\) at \(\varepsilon = 0.3\)), and even worst with \(\text{PCK}_{20}\). These results highlight the robust adaptability of CF-DyConv compared to basic CNN, particularly under challenging conditions such as adversarial attacks.

5.4 Ablation Study↩︎

To examine the impact of CF-DyConv, we use Algorithm 5 to automatically search the structural parameters of WiLHPE, resulting in optimal values for a batch size of \(64\) and a convolution kernel size of \(5\).

5.4.1 CF-DyConv↩︎

We evaluate standard CNN and attention-based CNN models on the P3-S2 MM-Fi dataset to assess cross-subject performance, where individual frequency variations influence signals even under identical postures. As shown in Table 4, WiLHPE is compared with regular CNNs, SENet [43], SKConv [47], and DyConv [44]. SENet derives kernel attention from global features using GAP and fully connected layers with softmax activation, whereas SKConv adapts kernel sizes but relies solely on channel-domain attention, potentially losing important information. As a result, CF-DyConv significantly outperforms SENet, improving \(\text{PCK}_{20}\) by over \(20\%\) and reducing MPJPE by up to \(58\) mm. Compared with SKConv and DyConv, CF-DyConv further reduces MPJPE by \(31\) mm and \(35\) mm, respectively, demonstrating its ability to capture contextual information across both frequency and channel domains for more accurate human pose estimation.

5.4.2 Number of Convolution Kernels↩︎

From 6 , we now evaluate the impact of the number of convolution kernels \(n\). For simplicity, we select \(r = 32\) and \(\tau = 34\). We consider different values of \(n\), ranging from 1 to 4. As shown in Table 5, WiLHPE achieves optimal results in terms of \(\text{PCK}_{20}\) with \(n=3\) and \(n=4\), surpassing \(54.95\%\). This implies that WiLHPE’s performance remains consistent with \(n \geq 3\). However, due to increased model complexity with higher \(n\) values, we select \(n=3\) in our simulations.

5.4.3 Reduction Ratio and Temperature↩︎

Finally, the impact of the reduction ratio \(r\) and temperature \(\tau\) is given in Table 6. During the excitation stage of our attention module, the reduction ratio \(r\) determines the size of the convolutional layer and adjusts the complexity of WiLHPE. We conduct experiments on the MM-Fi dataset with \(r=\{4, 8, 16, 32\}\) to identify the optimal ratio. Results from Table 6 show that increasing the reduction ratio \(r\) reduces the model’s complexity without compromising performance. The optimal balance between performance and complexity is achieved with \(r=16\). This implies that adjusting the reduction ratio effectively reduces model complexity and enhances computational efficiency while retaining crucial information. Moreover, this method of dimensionality reduction simplifies the training process by decreasing the number of parameters to optimize, thereby mitigating issues related to vanishing or exploding gradients.

Table 6 also shows the impact of the temperature value \(\tau\) on kernel attention, which regulates the sparsity of the attention mechanism. We experiment with temperature values of \(\tau = \{1, 10, 20, 30\}\) to determine the optimal temperature for the proposed network. As can be seen, a temperature value of \(\tau = 30\) achieves a well-balanced network with high performance and low complexity. Consequently, we adopt \(\tau = 30\) as the default temperature value to present the results of WiLHPE.

6 Conclusions↩︎

In this paper, we have proposed WiLHPE, a novel WiFi-based model designed to derive human pose landmarks by analyzing raw WiFi signals. WiLHPE achieved state-of-the-art accuracy while maintaining a lightweight model structure, with significantly fewer parameters compared to existing models in performing HPE tasks. The proposed model integrates a kernel attention mechanism that dynamically adjusts the kernel size based on input data characteristics. Furthermore, we have implemented the TPE algorithm to optimize critical hyperparameters, enabling WiLHPE to achieve optimal performance without requiring extensive hyperparameter searches. Rigorous validation on the MM-Fi and WiPose datasets demonstrated WiLHPE’s resilience and adaptability across various experimental settings. Notably, our model has shown significant performance improvements even under severe conditions. These achievements have paved the way for real-world deployment of WiLHPE, aimed at enhancing daily experiences for individuals.

References↩︎