[orcid=0009-0002-4411-3325]

[orcid=0000-0002-6813-4203]

[orcid=0009-0007-4209-6695]

[orcid=0000-0002-7136-7876]

[orcid=0000-0002-5000-2966]

[orcid=0000-0001-9060-6361]

[orcid=0009-0005-0008-5272]

[orcid=0000-0002-6478-8699]

Audiogram and spectrum fusion , Adaptive feature fusion , Cross-domain integrating , Personalized speech enhancement

1 Introduction↩︎

According to the World Health Organization, by 2050, over 700 million people—around one in ten—are projected to experience disabling hearing loss. If left untreated, hearing loss can lead to various complications, including social isolation and even dementia. In this context, hearing aids (HAs) play a crucial role in improving the hearing-specific health-related quality of life for individuals with hearing loss [1]. By enhancing understanding during conversations, HAs enable users to connect more effectively with family, friends, and colleagues. This personalized speech enhancement (PSE) service not only improves communication but also helps reduce the risk of associated issues, fostering better overall well-being.

To restore audibility, HAs commonly employ hearing loss compensation (HLC), a nonlinear amplification strategy that reshapes the acoustic input to accommodate the limited dynamic range of listeners with hearing impairment. The most widely used approach is the wide dynamic range compression (WDRC) algorithm, which mimics the function of outer hair cells in the cochlea by providing level-dependent amplification, thereby improving audibility while maintaining loudness comfort. Despite being the most effective means of hearing rehabilitation, HLC inevitably amplifies not only speech but also background noise, which substantially degrades listening comfort and may impose additional perceptual burdens on hearing-impaired listeners under noisy conditions. To alleviate this issue, HAs typically utilize a noise reduction (NR) component to suppress background noise prior to amplification. Fig. 1 illustrates the existing topologies of NR and HLC modules in HAs. Although traditional NR algorithms [2]–[4] are very robust when dealing with stationary noise, their performance is degraded in a nonstationary noisy environment [5]. Whether configured in a series or parallel manner, the enhanced speech often retains considerable residual interference, which reveals the inherent limitations of traditional HA frameworks in adapting to complex acoustic conditions. As illustrated in Fig. 2, even when residual noise seems negligible, the WDRC algorithm can still induce pronounced spectral distortions, particularly in non-speech segments and within the high-frequency range. Consequently, HAs often perform poorly in noisy environments [6], [7], where individuals with hearing loss require the most assistance.

Recently, deep neural networks (DNN) models utilizing the frequency-temporal sequence modules, e.g., FT-LSTM [8], [9] and FT-Conformer [10], have emerged as powerful backbones for speech enhancement, providing a promising option to address NS and HLC tasks. Building on these advances, considerable efforts have been dedicated to developing DNN-based systems for HAs [11]–[14], focusing on optimizing NR, HLC, or both components. In this line, DNN-based NR models [15]–[18] have demonstrated strong performance in enhancing speech clarity under noisy conditions, while DNN-based HLC models [19], [20] have achieved notable success in improving the auditory experience of individuals with hearing loss. However, these approaches typically focus on independently optimizing a single module, thereby overlooking the crucial interplay between NR and HLC tasks, while the absence of systematic joint optimization further constrains their overall effectiveness. Various studies [21]–[23] have confirmed that simply concatenating NR and HLC components in series remains inadequate. This limitation highlights the need for a unified framework that can jointly model both processes.

In this work, we propose the affine modulation-based audiogram fusion network (AFN-HearNet), which simultaneously performs NR and HLC by fusing audiogram and spectral features within a unified architecture. One of the major challenges lies in the effective integration of audiogram information into the speech spectrum representation, which is essential for jointly addressing enhancement and compensation tasks. While spectral features are dynamic, high-dimensional, and time-dependent, the audiogram is a static, low-dimensional profile that reflects hearing loss levels at discrete frequency bands but lacks temporal structure. Current studies [24], [25] address the resolution gap by extending audiograms through repeat-embedding methods. Fletcher–Munson equal-loudness contours [26], along with subsequent psychoacoustic studies, indicate that human hearing sensitivity varies smoothly between standard audiometric test frequencies. This smoothness assumption provides a principled basis for reconstructing intermediate frequency thresholds through linear interpolation. Building on this insight, we propose an audiogram-specific encoder that transforms the sparse hearing loss profile into a deep representation, thereby mitigating the spectral alignment problem. Inspired by the auditory model CASP [27], which incorporates a modulation filterbank [28] to explain human sensitivity to temporal–spectral fluctuations, we propose the affine modulation-based audiogram fusion frequency–temporal Conformer (AMFT-Conformer) to effectively fuse audiogram and spectral features. Unlike existing in-context fusion strategies that merely concatenate shallowly aligned features before encoding, our approach first applies independent encoders to transform the audiogram and spectral modalities. It then employs affine modulation, where the audiogram serves as a conditional input to directly scale, shift, and gate the statistical distribution of deep spectral representations, functioning as a neural analogue of auditory modulation filtering to adaptively inject individualized information across both frequency and temporal dimensions. This design allows the model to adaptively inject individualized information across both frequency and temporal dimensions, providing a more natural way to capture how specific levels of hearing loss influence speech perception across different frequency bands and time segments. Furthermore, since the FIG6 prescription formula tends to introduce excessive gain in non-speech segments and thereby causes distortion, we incorporate a voice activity detection (VAD) auxiliary training task. This enables AFN-HearNet to effectively distinguish between speech and non-speech segments, ultimately improving overall performance. The main contributions of this work can be summarized as follows:

• We propose a novel audiogram-specific encoder that leverages a linear interpolation strategy to expand sparse audiogram measurements into dense representations. This interpolation-based alignment yields a physiologically plausible feature representation, enabling more effective integration with speech spectra.

• We propose an affine modulation-based audiogram fusion frequency–temporal Conformer, which incorporates audiogram information as conditional auxiliary input to modulate noisy speech spectra, thereby achieving adaptive cross-modal fusion that injects individualized hearing profiles into spectral representations across both frequency and temporal dimensions.

• We incorporate a VAD auxiliary task that implicitly drives the AMFT-Conformer to learn speech activity information, enabling AFN-HearNet to identify and focus on segments containing speech.

• Comprehensive experiments are conducted across multiple datasets to validate the effectiveness of each proposed module. The results demonstrate that the AFN-HearNet outperforms state-of-the-art (SOTA) in-context fusion joint models in key metrics such as HASQI and PESQ.

The remainder of the paper is organized as follows: Section 2 introduces related works. Section 3 presents the problem formulation, followed by a detailed description of the proposed framework in Section 4. The experimental setup is outlined in Section 5, while Section 6 covers the results and analysis. Finally, conclusions are presented in Section 7.

2 Related work↩︎

2.1 DNN-based denoising and hearing compensation methods↩︎

Due to their excellent nonlinear approximation and temporal modeling capabilities, DNN-based models [29]–[34] have shown remarkable performance in speech enhancement tasks, making them a promising solution for PSE tasks. Most existing DNN-based approaches follow the conventional HA processing pipeline, focusing on optimizing either the NR or the HLC component independently, or sequentially combining the two. For the NR task, prior studies [15]–[18] explore the potential of the DNN-based NR components for HAs, where filter coefficients are estimated by a DNN model based on input features. Regarding the HLC task, studies [19], [20] propose neural networks for compensating sensorineural hearing loss. While these approaches improve audibility, they typically optimize each component in isolation, without explicitly modeling the interactions between NR and HLC. To address the interactions between NR and HLC, two-stage joint optimization frameworks [11], [35] are proposed, in which both components are trained under a unified objective. In contrast, joint models [24], [36] employ a unified architecture to simultaneously perform both NR and HLC tasks. For instance, the HA-MGAN [24] jointly addresses NR and HLC tasks by fusing audiogram and spectral features, whereas BSRNN [36] employs a multi-task learning framework to achieve joint optimization of the two tasks. However, although the existing in-context fusion in joint models demonstrates effectiveness, it is still constrained by unresolved issues such as audiogram–spectrum alignment and temporal mismatch. Current approaches typically concatenate the two modality features directly, neglecting cross-modal interactions and relying on overly simplistic fusion mechanisms. As a result, the subsequent encoder is forced to implicitly reconcile information from misaligned features, which constrains overall performance.

In practice, it is infeasible to measure frequency-wise paired data between audiogram information and spectral features, making precise alignment unattainable. To address this resolution alignment problem, HEROS-GAN [37] leverages optimal transport theory to maximize supervisory information from unpaired data and injects Laplacian energy into the generator, transforming low-cost sensor signals into high-cost equivalents. Previous studies [19], [24], [25], [38] utilize a repeat-embedding method to expand the audiogram before concatenating it with spectral features, where each sub-band is assigned the same hearing threshold. In contrast, [36] employs a learnable fully connected layer to directly extract gated conditional features for deep spectral representations, thereby skipping the alignment step. Different from prior approaches, our method combines a psychoacoustically inspired linear interpolation strategy for audiogram expansion with a dedicated audiogram-specific encoder, enabling the extraction of discriminative patterns that more faithfully capture individualized hearing characteristics.

2.2 Fusion schemes for integrating cross-model features↩︎

Multi-modal fusion plays a critical role in personalized speech enhancement, where effective integration of audiogram and acoustic features is essential. Beyond basic in-context fusion, more sophisticated mechanisms are proposed for multi-modal feature integration. The gated fusion mechanism [39], [40] adaptively regulates the information flow by learning to select and integrate the most relevant components from noisy and enhanced features. Bilinear pooling [41]–[43] employs outer product operations to capture second-order dependencies between modalities, enabling more expressive cross-feature representations. Attention-based fusion [44], [45] leverages feature representations from one modality to compute attention weights for another, thereby guiding the fusion process by emphasizing more informative and semantically relevant components. However, most of the aforementioned fusion strategies implicitly assume temporal or semantic compatibility across modalities, relying on time-varying cues to estimate fusion weights or model cross-modal interactions. Such assumptions are not well aligned with the inherent asymmetry between audiogram and spectrum features. In contrast, affine transformation [46] offers a principled mechanism for cross-modal feature fusion, where one modality serves as conditional auxiliary information to guide the computation of the other. In parallel, auditory models [28] employ modulation operations to capture human sensitivity to temporal–spectral fluctuations and to simulate hearing loss in both normal-hearing and hearing-impaired listeners. Motivated by these insights, we introduce affine modulation into time–frequency modeling and further propose a post-gating strategy to emulate auditory masking effects, thereby enabling more effective fusion of audiogram and spectrum features.

3 Problem description↩︎

Considering a hearing aid that utilizes a serial concatenation configuration as shown in Fig. 1 (a), where NR is followed by HLC, with clean speech \(s(n)\) and ambient noise \(e(n)\), the received microphone signal \(y(n)= s(n) + e(n)\) can be expressed in the frequency domain using the short-time Fourier transform (STFT) as follows: \[Y(l, k)=S(l, k) + E(l, k),\] where \(l\) and \(k\) indicate the index of the frame and frequency bin, respectively. In the hearing aid system, a front-end NR algorithm is typically introduced to filter out noise by applying complex ratio masking \(M\) to the input spectrum: \[\hat{S}(l,k) = Y(l,k)M(l,k).\] Next, the enhanced speech is fed into a WDRC algorithm that utilizes a FIG6 fitting formula to estimate the target SPL for each sub-band based on the input sub-band SPLs and the audiogram \(HL\). Following this, the logarithmic gain \(G(l,b)\) in decibels for loudness compensation can be expressed as: \[\begin{align} G(l,b)&=\text{SPL}_{\text{target}}(l, b)-\text{SPL}_{\text{input}}(l, b), \\ \text{SPL}_{\text{target}}(l, b) &= \text{FIG6}(\hat{S}(l); HL), \end{align}\] where \(b\) denotes the sub-band index. All frequency bins within the same sub-band share the same gain. Consequently, the total gain for the PSE output spectrum \(\hat{X}\) based on the noisy microphone spectrum is given by: \[\hat{X}(l, k) = Y(l,k)\left[M\left(l,k\right)\cdot \bar{G}(l, b)\right], k\in b.\] where \(\bar{G}(l, b)=10^{\frac{G\left(l, b\right)}{20}}\). Therefore, to develop a model that jointly enhances and compensates for the noisy speech, the core task is to determine the joint masking parameters \(M(l,k)\cdot\bar{G}(l, b)\), given the individual hearing loss audiogram.

4 Methodology↩︎

4.1 Overview↩︎

The proposed audiogram fusion PSE network can be represented as a mapping function that takes a distorted noisy waveform \(Y\in\mathbb{R}^{L\times1}\) and a hearing-impaired audiogram \(HL\in\mathbb{R}^{1\times N}\) as inputs and outputs the monaural de-noised and compensated waveform \(\hat{X}\in\mathbb{R}^{L\times1}\), denoted as \(\mathcal{F}: (Y, HL)\mapsto \hat{X}\). The architecture of the AFN-HearNet is illustrated in Fig. 3 and consists of four key components: (1) a dual encoder structure that integrates a spectrum encoder to capture essential features while reducing dimensionality, along with an audiogram-specific HL-Encoder that transforms audiogram information for feature alignment; (2) cascaded AMFT-Conformer modules that fuses the cross-domain features by utilizing HL-Encoder outputs to adaptively modulate the unified deep representations during the frequency-wise and temporal-wise feature modeling process; (3) dual-path decoders that reconstruct the spectrum from both the magnitude and phase domains; and (4) an auxiliary VAD estimator that improves the information density and focus of unified deep representations by identifying speech-active regions.

4.2 Spectrum encoder and decoder↩︎

An STFT operation first converts the noisy waveform into a complex spectrogram \(Y\in\mathbb{R}^{2\times T\times F}\). We utilize the spectrum encoder and decoders from MetricGAN [10] to build the backbone network for processing and reconstructing the speech signals. The encoder begins with two cascaded down-sample convolutional blocks, followed by a dilated DenseNet [47]. Each down-sample block consists of a convolution layer, a layer normalization, and a PReLU activation function, which reduces the frequency dimension by half. Dual-path spectrum decoders operate independently to reconstruct the output components from the fused unified deep representation produced by the cascaded AMFT-Conformer. Each path incorporates a sub-pixel block consisting of two SPConvTransposed modules [48] that up-sample the frequency dimension by a factor of two. Specifically, the mask decoder is designed to predict a masking filter \(\hat{M}\in\mathbb{R}^{T\times F}\) that will be element-wise multiplied with the input magnitude \(Y_m\in\mathbb{R}^{T\times F}\). In contrast, the phase decoder estimates the real \(\hat{Y}_{r}\) and imaginary \(\hat{Y}_{i}\) components of the output spectrum, allowing for the indirect reconstruction of the phase components since structural information is absent in the phase spectrogram [49]. Finally, the spectrum reconstruction progress can be described as: \[\begin{align} \hat{X} &= (\hat{M} \odot Y_m) \cdot e^{j\angle\phi}\\ \angle\phi &= \arctan(\hat{Y}_i / \hat{Y}_r), \end{align}\] where \(\odot\) refers to the element-wise multiplication. Finally, an inverse short-time Fourier transform is applied to convert the reconstructed spectrum back into the time domain, yielding the output compensated speech.

4.3 Audiogram-specific encoder↩︎

Since audiograms quantify hearing loss thresholds at a limited set of standard clinical frequencies (typically 250, 500, 1000, 2000, 4000, and 8000 Hz), their feature representation is inherently sparse. However, speech spectrum possess a much finer spectral resolution, and perceptually relevant spectral information may occur at intermediate frequencies not directly measured in the audiogram. We propose a simple yet effective method, HL-Linear, a linear interpolation technique that extends sparse clinical measurements (\(\mathbb{R}^{6}\mapsto\mathbb{R}^{F}\)) into a high-resolution representation aligned with the spectral resolution of the input speech.

Fig. 4 illustrates the architecture of the audiogram-specific encoder. Specifically, given the input audiogram \(HL=[h_0,\cdots,h_5]\) measured at the frequency components \([f_0, \cdots, f_5]= [250, 500, 1000, 2000, 4000, 8000]\), the hearing loss value for any frequency component \(f_k\) within the \(i\)-th sub-band \([f_{i+1},f_i]\) can be computed as: \[\begin{align} HL(k) = h_i + \frac{h_{i+1}-h_i}{f_{i+1}-f_i}\cdot (f_k - f_i). \end{align}\] The interpolated audiogram is then processed by a convolutional multi-layer perceptron (ConvMLP) module, which acts as a transformation function \(\mathcal{F}:\mathbb{R}^{F}\mapsto\mathbb{R}^{C\times F}\). The ConvMLP module performs channel expansion by integrating an MLP and two convolution modules, leveraging the local perceptual capabilities of convolutional layers and the global feature learning capability of the MLP. Specifically, the extended audiogram first passes through a feature-expanding linear layer, followed by a GELU activation function. Next, it goes through two cascaded convolution blocks, each incorporating a batch normalization (BN) and a PReLU activation layer. Finally, the channel dimension is adjusted using a linear layer. The transformation progress can be expressed as: \[\begin{align} H^\prime_{e}&=GELU\left(\mathcal{R}\left(H_{e}\cdot W_f +b_f; \mathbb{R}^{CF}\mapsto\mathbb{R}^{C\times F}\right)\right), \\ H^{\prime}_{e}&=\delta_i\left(\mathcal{B}_i\left(Conv\left(H^\prime_{e};\theta_i\right)\right)\right),~~i\in[0,1]\\ \hat{H}_{e} &= \mathcal{R}\left(H^{\prime}_{e};\mathbb{R}^{C\times F}\mapsto\mathbb{R}^{F\times C}\right)\cdot W_c + b_c, \end{align}\] where \(H_e\in\mathbb{R}^{F}\) denotes the extended audiogram; \(\cal R\) refers to the rearrange operation; \(\delta_i\), \(\mathcal{B}_i\), and \(Conv(\cdot;\theta_i)\) denote the PReLU, BN, and convolution operations of \(i\)-th layer. \(W\) and \(b\) with subscripts \(f/c\) correspond to the parameters of the first and last linear layers, respectively.

4.4 Affine modulation-based audiogram fusion↩︎

Fig. 5 (a) illustrates the process of the existing in-context fusion strategy. It aligns the audiogram features with spectrum components through a feature embedding method, then concatenates these features and integrates them with a multi-layer convolutional encoder, a common fusion method for combining features from multiple models [50], [51]. However, since the fusion stage relies entirely on convolution operations and is processed only once, it fails to capture complex patterns or temporal dependencies within the features adequately, resulting in limited effectiveness. In contrast, as shown in Fig. 5 (b), the affine modulation fusion strategy aligns features in the latent domain by employing independent encoding for different features. This fusion strategy combines affine modulation with frequency and temporal-wise Conformer modules that incorporate attention mechanisms, facilitating both local and global feature integration. By utilizing the cascaded structure, it effectively merges features from different levels, refining the unified deep representation to capture complex patterns. Building on this foundation, the detailed architecture of the modulation fusion module, AMFT-Conformer, is shown in Fig. 6. It consists of two fusion stages: the frequency dimension fusion stage (FFusion-Conformer) and the temporal dimension fusion stage (TFusion-Conformer), both operating over the channel dimension. Each stage contains a stage-oriented Conformer followed by an MLP block, with an affine modulation module positioned in front of each. The AMFT-Conformer takes the down-sampled deep representations from the spectrum encoder \(Z\in\mathbb{R}^{C\times T\times\bar{F}}\) as the input feature, and the transformed audiogram \(\hat{H}_{e}\in\mathbb{R}^{C\times\bar{F}}\) from the HL-Encoder as the personalized representation.

In the FFusion-Conformer stage, the down-sampled spectrum representation \(Z\in\mathbb{R}^{C\times T\times\bar{F}}\) is first rearranged by permuting its dimensions to obtain a frequency-oriented form \(Z_f\in\mathbb{R}^{T\times\bar{F}\times C}\). The audiogram-derived representation \(\hat{H}_{e}\) is processed through a SiLU–Linear transformation module, consisting of a SiLU activation followed by a linear layer, to produce two groups of frequency-adaptive affine modulation parameters, denoted as \([\gamma_i,\beta_i,\alpha_i]\in\mathbb{R}^{1\times \bar{F}\times C}\) where \(i\in[1,2]\). This progress can be represented as: \[\left[\gamma_1,\beta_1,\alpha_1,\gamma_2,\beta_2,\alpha_2\right]^{T}=W_{mf} \sigma_{\text{SiLU}} (\hat{H}_{e}) + b_{mf},\] where \(W_{mf}\) and \(b_{mf}\) denote the parameters of the transformation linear layer. These parameters enable stage-specific feature modulation, where the first group \((\gamma_1, \beta_1, \alpha_1)\) scales and shifts the frequency-oriented representation \(Z_f\) prior to processing by the frequency-wise Conformer block. In this way, the Conformer’s self-attention and convolutional submodules operate on features that have already been re-weighted and aligned to the listener’s hearing profile. The subsequent gating operation further allows the network to selectively enhance frequency bands of higher perceptual importance while suppressing less relevant spectral components. The frequency-wise modulation process can be expressed as: \[Z^\prime_f = \left[\mathcal{T}\left(Z_f\odot\left(I+\gamma_1\right)+\beta_1\right)\right]\odot\alpha_1+Z_f,\] where \(\mathcal{T}\) refers to the Conformer block, and \(I\) denotes the all-ones matrix \(\in\mathbb{R}^{1\times \bar{F}\times C}\). Similarly, the second post-attention modulation step, parameterized by \((\gamma_2, \beta_2, \alpha_2)\), re-scales, shifts, and gates the intermediate representation, thereby introducing additional non-linear transformations that reallocate channel-wise energy while preserving the inherent sequential structure of the features. This operation can be computed as: \[\begin{align} \hat{Z}_f &= \left[MLP\left(Z^\prime_f\odot\left(I+\gamma_2\right)+\beta_2\right)\right]\odot\alpha_2+Z^\prime_f. \end{align}\]

The TFusion-Conformer stage adopts the same architecture as the FFusion-Conformer but operates along the temporal dimension, applying modulation and Conformer processing to time-oriented feature sequences in the form \(Z_t\in\mathbb{R}^{\bar{F}\times T\times C}\). While the audiogram is inherently time-invariant, effective compensation is time-dependent, as the compensation gain must be adapted to the dynamic characteristics of speech. Temporal-wise modeling enables the network to capture such variations, including transient events and fluctuations in speech energy. In the AMFT-Conformer, this capability is realized through the TFusion-Conformer stage, which performs temporal fusion to integrate personalized hearing profiles with dynamic speech features for time-aware compensation.

4.5 VAD auxiliary module↩︎

Since the FIG6 compensation algorithm estimates the target gain based on the input SPL, non-speech segments become more susceptible to the influence of residual noise. In the case of high SPL background noise, the preceding speech enhancement algorithm has already suppressed its stationary components, thereby substantially reducing the residual noise energy and bringing the SPL closer to that of quiet segments. Consequently, the SPL of non-speech segments is reduced to extremely low levels. Although these noise components are of low energy, they still fall within the FIG6 amplification region, which is typically inactive only below 20 dB SPL, a condition that is rarely met in real-world acoustic environments. As a result, these components are subject to excessive gain amplification, leading to perceptible distortion in non-speech segments, as illustrated in Fig. 2.

We propose a VAD auxiliary training module that takes the unified deep representation output by the AMFT-Conformer as input. This module implicitly encodes latent feature patterns of both speech and non-speech segments within the unified deep representation, effectively building awareness of the acoustic context in the feature latent space. This information about the acoustic context is particularly useful for addressing the noise amplification issue in silent segments caused by the WDRC loudness compensation algorithm based on the FIG6 formula while maintaining the model’s inference efficiency. Since the VAD estimator is only active during training, it introduces no additional parameters or computational overhead during deployment.

The module begins with an average pooling layer to aggregate the global features of each channel, followed by a point-wise convolution block, where each convolution operation is succeeded by layer normalization and a PReLU activation function. Next, we employ a GRU layer to model temporal dependencies, effectively capturing sequential patterns and variations over frames. Finally, the output is fed into a fully connected layer that reduces the dimensionality and passes through a sigmoid function to estimate the speech activity probabilities. Given the input unified deep representation \(H\in\mathbb{R}^{C\times T\times F}\), the process of estimating the VAD probability \(P_{vad}\in\mathbb{R}^{T\times 1}\) can be expressed as follows: \[\begin{align} P_{vad}&=\sigma\left(\text{GRU}\left(H^\prime\right)\cdot W+b\right),\\ H^\prime&=\delta\left\{\mathcal{B}_{LN}\left[\mathcal{R}\left(\text{PWConv}\left({\cal G}\left(H\right)\right); \mathbb{R}^{C\times T\times 1}\mapsto\mathbb{R}^{T\times C}\right)\right]\right\}, \end{align}\] where \(\mathcal{G}(H)=\frac{1}{F}\sum\nolimits_{i=0}^{F-1}H[..., i]\in\mathbb{R}^{C\times T\times 1}\) represents the average pool operation across the frequency dimension for each channel, and \(\text{PWConv}\) refers to the convolution layer with \((1,1)\) kernels. Symbols \(\sigma\), \(\delta\), \(\mathcal{B}_{LN}\), and \(\cal{R}\) denote the sigmoid, PReLU, layer normalization, and rearrange operations, respectively. \(W\) and \(b\) represent the parameter matrices of the linear layer.

4.6 Loss function↩︎

In this work, we introduce the Hearing Aid Speech Quality Index (HASQI) metric [52] as a key optimization target due to its established role in evaluating HLC effectiveness. Since the HASQI function is non-differentiable, we adopt the MetricGAN framework [24], [48], which enables gradient-based optimization of black-box metrics through adversarial training. For spectrum reconstruction objectives, we introduce the multi-resolution STFT loss [53], which operates on the spectrogram magnitudes. This loss is an expectation of various STFT losses configured with different frames and hop lengths, which can be computed as: \[\begin{align} \mathcal{L}_{stft}(X, \hat{X}) &= 0.5 \times(\mathcal{L}_{sc}(X, \hat{X}) + \mathcal{L}_{mag}(X, \hat{X})),\\ \mathcal{L}_{sc}(X, \hat{X}) &=\frac{\Vert~|\text{STFT}({X})|-|\text{STFT}(\hat{X})|~\Vert_{F}}{\Vert~|\text{STFT}({X})|~\Vert_{F}},\\ \mathcal{L}_{mag}(X, \hat{X}) &= \frac{1}{T} \Vert \log|\text{STFT}(X)| - \log|\text{STFT}(\hat{X})|~\Vert_{1}, \end{align}\] where the \(\Vert \cdot \Vert_F\) and \(\Vert \cdot \Vert_1\) represent the Frobenius and \(L_1\) norms, respectively. \(X\) denotes the compensated clean target and \(\hat{X}\) refers to the enhanced speech of the AFN-HearNet model. In this work, we setup the frame sizes to \([1024, 512, 256]\) and corresponding hop lengths to \([512, 256, 128]\). Additionally, we introduce the perceptual metric for speech quality evaluation (PMSQE) [54] loss to optimize for human auditory perception, and employ a focal loss to train the VAD auxiliary task.

Overall, given the hearing-impaired audiogram \(HL\), compensated clean speech \(X\), and enhanced output \(\hat{X}\), and the ground-truth as well as estimated VAD labels \(V\) and \(\hat{V}\), the joint losses for the entire framework are described as follows: \[\label{loss95eqa} \begin{align} {\cal L}_G &=\alpha\cdot \mathbb{E}\left[ \left\| D_{\text{fix}}\left( X,\hat{X},HL \right) - 1 \right\|^2 \right]\\ &+ \lambda\cdot{\cal L}_{PMSQE}\left( X,\hat{X} \right) + {\cal L}_{stft}\left(X,\hat{X}\right)+0.3\cdot\mathcal{L}_{focal}(V,\hat{V}),\\ {\cal L}_D &= \mathbb{E}\left[\left\| {D\left(X,X,HL\right)-1} \right\|^2\right] \\ &+ \mathbb{E}\left[\left\| D\left(X,\hat{X},HL\right)-\text{HASQI}\left(X,\hat{X},HL\right) \right\|^2 \right], \end{align}\tag{1}\] where \({\cal L}_G\) and \({\cal L}_D\) denote the loss functions to train the generator of AFN-HearNet and a corresponding discriminator of \(D\) proposed in MetricGAN, respectively. \(\alpha\) and \(\lambda\) are used to balance the various loss values. Given that GAN training is highly sensitive to the weighting of loss terms [55], a further analysis is presented in Appendix 9.

5 Experiments↩︎

5.1 Datasets↩︎

To conduct the experiments, we utilized two synthetic datasets derived from widely used public datasets: one dataset based on the DNS-Challenge [56] and another based on the LibriSpeech [57]+Demand [58] dataset. The DNS-Challenge¹ dataset consists of 500 h clean speech from 2150 speakers and 180 h noise. The LibriSpeech² corpus consists of approximately 1000 h of read English speech, and we utilize the clean speech from several subsets, i.e., train-clean-100, dev-clean, and test-clean, along with noise from the Demand³ corpus. The audiograms used in the experiments were sourced from the National Health and Nutrition Examination Survey (NHANES) [24]. The dataset consists of 114 audiograms obtained following standard measurement procedures for pure-tone audiometry, covering ascending, sloping, and flat audiogram curve types. Each participant contributed both left- and right-ear audiograms. The average age of the hearing-impaired participants was 68.4 years, with 27 males and 30 females. The mean hearing threshold at 500 Hz, 1 kHz, 2 kHz, and 4 kHz is used to calculate pure tone average, and the degree of deafness was categorized according to the severity of the hearing loss: 10 audiograms with thresholds below 40 dB were classified as mild or less severe; 61 audiograms with thresholds between 40 dB and 70 dB as moderate or moderately severe; and 43 audiograms exceeding 70 dB as severe or profound [59].

The detailed procedures for synthesizing datasets are outlined in Algorithm 7. First, sample a clean speech \(s\) that exhibits over 60 % speech activity, along with a noise \(e\). To simulate different scenarios of loudness compensation, where small signals are released while large signals are attenuated according to their SPL, we transform the clean speech \(s\) by either amplifying, suppressing, or leaving unchanged, with respective probabilities of \([0.4, 0.3, 0.3]\). The transformed clean speech is denoted as \(x_G\). During the amplification phase, the clean speech energy is increased by at least 5 dB, but not exceeding an upper limit of -10 dB. Conversely, in the suppression phase, the clean speech energy decreases by at least 5 dB, reaching a minimum of \(-35\) dB. After that, with a probability of 0.7, Gaussian white noise is added to the sampled noise. The noisy \(x_{noisy}\) is synthesized by combining the clean speech with this noise according to an SNR sampled from a uniform distribution ranging from \([-5,15]\) dB. Then, we apply the WDRC algorithm, based on the FIG6 fitting formula, which takes the transformed speech \(s_G\) and audiogram vector \(hl\) as inputs, to generate the target speech \(s_{hl}\). To address the problem of excessive gain in silent segments resulting from the FIG6 fitting formula, we replace the compensated non-speech segments of target speech with the original speech segments before compensation. In total, we synthesized 72000 training samples (\(\sim\)​100 h) and 1440 validation samples (\(\sim\)​2 h), each lasting 5 s, for both the DNS-Challenge dataset and LibriSpeech+Demand dataset. The no-reverberation test set of the DNS-Challenge dataset contains 150 samples, while the LibriSpeech+Demand test set includes 1440 samples (\(\sim\)​2 h) for comparison with other baselines. Ablation studies are conducted on the DNS-Challenge dataset, with evaluations performed on both the no-reverberation DNS-Challenge test set and a synthesized test. This synthesized test set combines speech from the LibriSpeech test-clean subset with noise from the NoiseX-92 corpus [60] under SNRs from \(-5\) dB to 15 dB in 5 dB increments, with each SNR containing 720 samples (\(\sim\)​1 h). The NoiseX-92 corpus consists of 15 different types of real-world noise recordings, including babble, factory noise, and street noise.

5.2 Implementation details↩︎

All speech recordings were sampled at a frequency of 16 kHz. The audio waveform is converted to the frequency domain using STFT with a Hanning window of 512 (32 ms) and a hop size of 256 (16 ms). In our work, we set the number of middle channels to 48 and the number of cascaded AMFT-Conformer modules to 2. A more detailed description of the network parameters is given in Table 1. The hyperparameters of convolution layers for the encoder and decoder are specified as follows: filterHeight \(\times\) filterWidth, stride (along frame, frequency), and dilation (along frame, frequency) if applicable. The parameters \(\alpha, \lambda\), and \(\mu\) for the discriminator, PMSQE, and multi-resolution STFT loss are set to 0.5, 0.3, and 1, respectively. We utilize the Adam optimizer with a learning rate of \(5e^{-4}\), which is decayed by a factor of 0.5 every 10 epochs. The proposed model is trained for a total of 25 epochs.

5.3 Comparative baselines↩︎

To assess the performance of both benchmarks fully, we conducted a fair comparison with several baselines that jointly address the NR and HLC tasks using the in-context fusion strategy. This comparison includes classical models such as CMGAN, as well as SOTA models like SEMamba and MP-SENet.

For the DNS-Challenge benchmark, we consider the following seven baselines: WDRC-FIG6 [61], a traditional HLC algorithm that employs the FIG6 fitting formula; CRN [62], the first model developed using the encoder-decoder architecture in the field of speech enhancement; DCCRN [63], which revises both convolutional and recurrent structures to effectively handle complex-valued operations; CMGAN [10], a Conformer-based metric generative adversarial network that employs two-stage Conformer blocks to model temporal and frequency dependencies; NUNet_TLS [64] that utilizes two-level skip connections in a U-Net architecture, along with causal time-frequency attention, to improve the dynamic representation of speech context across multiple scales; CompNet [65], a dual-stage model that enhances noisy speech in the time domain and uses a post-processing module to filter magnitude and refine phase; HA-MGAN [24], the first joint framework that applies a metric generative adversarial strategy for simultaneous NR and HLC tasks. For the LibriSpeech+Demand benchmark, we introduce the five additional baselines. DB-AIAT [66] is a dual-branch attention-in-attention transformer-based model developed to handle both coarse- and fine-grained regions of the spectrum in parallel. MP-SENet [67] concurrently denoises magnitude and phase spectra through a codec architecture with convolution-augmented transformers, and utilizes multi-level losses for joint training. FSPEN [68] is an ultra-lightweight network designed for real-time speech enhancement in resource-constrained environments, utilizing both full-band and sub-band structures to extract global and local features. SEMamba [69] is a scalable state-space model developed for speech enhancement, utilizing a Mamba-based regression approach to characterize speech signals while incorporating signal-level distances and metric-oriented loss functions. PrimeK-Net [70] is a speech enhancement framework centered on computational efficiency, and it incorporates a group prime kernel feedforward network that extracts deep temporal and frequency features with linear complexity.

5.4 Evaluation metrics↩︎

We employed a comprehensive set of objective metrics to evaluate both individual module contributions and overall system performance. For speech enhancement quality assessment, we adopt the Perceptual Evaluation of Speech Quality (PESQ) [71] in both narrow and wide bands configurations, along with the Short-Time Objective Intelligibility (STOI) [72] metric for quantifying speech intelligibility improvements. To assess speech quality for hearing-impaired listeners, we incorporate the HASQI [52] metric, a key indicator of hearing aid performance. Furthermore, we introduce the signal-to-distortion ratio (SDR) [73], which measures the overall distortions of the enhanced signal relative to the clean reference, and scale-invariant signal-to-noise ratio (SI-SNR) [74], which provides a robust assessment of amplitude scaling.

6 Results and analysis↩︎

The experimental results are analyzed and discussed in detail in this section.

6.1 Ablation Study↩︎

For our ablation study, we develop various variants trained on the DNS-Challenge dataset to investigate the effectiveness of each proposed component. These variants are built on two base models: AFN-HearNet-base, revised from the widely used FT-Conformer backbone network of CMGAN [10], and HA-MGAN [24], the FT-LSTM backbone network that first jointly performs NR and HLC tasks. Both models utilize the HL-Embedded method [24] to extend the input hearing loss audiogram. For each base model, the +HL-Linear variant replaces the HL-Embedded module with linear interpolation to transform the audiogram features, while the +VAD variant incorporates the VAD auxiliary task during training. The +AMFT-Conformer&HL-Encoder variant combines the HL-Encoder and AMFT-Conformer with the base model, whereas the +AMFT-LSTM&HL-Encoder variant integrates the HL-Encoder and AMFT-LSTM that replaces the Conformer module in AMFT-Conformer with an LSTM sequence module. Both variants replace the in-context fusion strategy with our proposed affine modulation fusion strategy.

To ensure the integrity of the experiments, the ablation study is conducted on two test sets aimed at validating the effectiveness of each block and assessing their generalization capabilities across different noisy distributions. The first experiment was conducted on the DNS-Challenge validation and test sets, while the second was performed on the LibriSpeech+NoiseX-92 test set. The detailed results are shown in Table 2 and Table 3. Additionally, Table 4 illustrates the Student’s t-test significance analysis results of each proposed component.

6.1.1 Impact of audiogram linear fitting interpolation for feature alignment↩︎

The audiogram linear interpolation method first extends the sparse audiogram features and aligns them with the spectrum features in terms of frequency components. By comparing the results of the two baselines to those of the +HL-Linear variants in Table 2 and Table 3, we can conclude that the linear interpolation method outperforms the HL-Embedded interpolation method across both baselines. It significantly improves the performance (\(p<0.05\)) of the FT-LSTM baseline across all metrics on both test sets. Specifically, there are relative improvements of 10.1% in HASQI, 7.1% in WB-PESQ, 4.7% in NB-PESQ, 17.1% in SDR, 20.7% in SI-SNR, and 1.7% in STOI on the DNS-Challenge test set. In the case of the FT-Conformer baseline, the linear interpolation method shows marginal significance (\(p<0.07\)) in improving the WB- and NB-PESQ metrics, along with significant improvements in other metrics.

These findings underscore the importance of the proposed linear interpolation strategy. By transforming sparse clinical measurements into dense, spectrum-aligned representations, the HL-Linear method provides a more physiologically plausible approach to audiogram-based feature expansion. This not only enhances speech quality and intelligibility metrics but also demonstrates the general utility of explicit audiogram interpolation across different neural architectures.

6.1.2 Impact of affine modulation-based adaptive audiogram fusion↩︎

The results on the DNS-Challenge test set in Table 2 illustrate that both +AMFT-LSTM and +AMFT-Conformer variants significantly outperform their baselines using the in-context fusion strategy (\(p<0.05\)) across all metrics. Specifically, the +AMFT-LSTM variant demonstrates notable improvements with HASQI, WB-PESQ, NB-PESQ, SDR, SI-SNR, and STOI metrics increasing by 11.6%, 10.0%, 6.7%, 21.1%, 25.3%, and 2.0%, respectively. While maintaining the high score of 0.81 in the HASQI metric, the +AMFT-Conformer variant increases the WB-PESQ metric from 2.70 to 2.75, NB-PESQ from 3.23 to 3.27, SDR from 13.57 to 13.92, SI-SNR from 13.33 to 13.68, and STOI from 95.17 to 95.55 compared to its baseline. Moreover, Table 3 demonstrates the consistency results that both the +AMFT-Conformer and +AMFT-LSTM variants significantly outperform their respective baselines across all SNR conditions and evaluation metrics.

These cross-dataset consistency results confirm the superiority of the affine modulation-based fusion strategy compared to the in-context fusion strategy, demonstrating its effectiveness in enhancing both perceptual quality and intelligibility while maintaining robustness across diverse acoustic conditions.

6.1.3 Impact of VAD auxiliary training task↩︎

The VAD auxiliary training task implicitly drives the AMFT-Sequence modules to incorporate speech activity information into the unified deep representation. It is interesting to find out that the effectiveness of the VAD auxiliary training task varies across the two baselines. For the FT-LSTM baseline using an in-context fusion strategy, the results of the +VAD variant in Table 2 and Table 3 demonstrate that incorporating the speech activity information significantly increases the performance across all metrics. The +VAD variant improves the HASQI, WB-PESQ, NB-PESQ, SDR, SNR, and STOI scores on both the validation and test sets, with notable increases in SDR from 10.11 to 12.18 on the validation set and SI-SNR from 9.08 to 11.48. However, the improvements on the FT-Conformer baseline are modest, with SDR increasing only from 13.57 to 13.63 and STOI from 95.17 to 95.31 on the test set.

The VAD auxiliary training achieves similar performance results with the affine modulation-based feature fusion strategy. The improvements for the +ACFT-Conforme&HL-Encoder&VAD are more modest compared to the +ACFT-Conforme&HL-Encoder variant. Table 2 illustrates that on the validation set, NB-PESQ increases from 3.20 to 3.22, and STOI rises from 92.51 to 92.57. On the test set, NB-PESQ improves from 3.27 to 3.29, and STOI increases from 95.51 to 95.55. These observations suggest that the Conformer’s self-attention mechanism may already be implicitly learning to differentiate speech from non-speech segments by leveraging its capacity to model long-range dependencies and global context. Specifically, inter-frame attention computes pairwise similarities between frames via query-key dot products, effectively capturing relationships rooted in diverse acoustic representations and temporal dynamics. As a result, after applying a sigmoid activation (or softmax normalization), the attention weights corresponding to speech–speech interactions are considerably higher than those between speech and noise, or between noise frames [75]. This inherent characteristic of self-attention may already compensate for the benefits provided by explicit VAD information. Consequently, incorporating VAD supervision yields relatively smaller performance gains, as its information becomes partially redundant. This highlights the strong representational capacity of the Conformer, which reduces the marginal utility of auxiliary cues that overlap with the patterns already captured by the architecture itself. In summary, the VAD auxiliary training task is a valuable addition, particularly for the simpler FT-LSTM base model, while its utility for the FT-Conformer base model is more limited but still positive.

To further improve the effectiveness of auxiliary supervision, future work may explore tasks that offer more orthogonal information to the Conformer’s internal representations. For instance, noise type classification or SNR estimation has been investigated in prior works as complementary information for guiding speech enhancement [76], [77]. These tasks introduce domain-relevant cues that are less likely to be implicitly modeled by the self-attention mechanism, and thus may offer additional improvements when combined with architectures like the Conformer.

6.1.4 Generalization gap analysis↩︎

To assess the generation gap [78] of each variant under unseen acoustic conditions, all variants are further trained on the LibriSpeech+DEMAND dataset. We define the DNS-Challenge dataset as the double-mismatch acoustic condition, \(\mathcal{C}_{d} = \left(\mathcal{S}^{\text{speech}}_{\text{DNS}}, \mathcal{S}^{\text{noise}}_{\text{DNS}}\right)\), where both the speech and noise sources differ from those in the LibriSpeech+Demand test set, i.e., \(\mathcal{S}^{\text{speech}}_{\text{DNS}} \neq \mathcal{S}^{\text{speech}}_{\text{LibriSpeech}},\mathcal{S}^{\text{noise}}_{\text{DNS}} \neq \mathcal{S}^{\text{noise}}_{\text{Demand}}\). The variants trained on LibriSpeech+DEMAND can thus be regarded as reference variants, enabling a clearer separation between the effect of exposure to new data and the effect of increased task difficulty in speech enhancement.

The improvements between the unprocessed noisy signal and the enhanced output are denoted as \(\Delta\)metric, such as \(\Delta\)HASQI and \(\Delta\)SDR, as illustrated in Fig. 8. Across all variants, the matched conditions, i.e., DNS-DNS and Libri-Libri, consistently achieve higher metrics than the mismatched setting \(\mathcal{C}_d\), reflecting the performance degradation caused by condition mismatch. Although all variants experience performance drops under speech and noise mismatches, the +AMFT-Conformer & VAD, i.e., AFN-HearNet, variant maintains relatively higher scores, indicating stronger generalization to unseen acoustic conditions. Additionally, the +AMFT-Conformer & VAD variant is the only one that outperforms its corresponding reference model across all metrics on the mismatched test set, with the exception of HASQI. This is evidenced by the generation gap \(G_E = \tfrac{\Delta E - \Delta E_{\text{ref}}}{\Delta E_{\text{ref}}}\), yielding –11.11% for HASQI, 3.29% for WB-PESQ, 2.65% for NB-PESQ, 3.00% for SDR, 2.79% for SI-SNR, and 4.61% for STOI.

These findings suggest that although the VAD auxiliary training task does not yield substantial improvements for the Conformer backbone under matched conditions, it nevertheless provides complementary benefits by enhancing robustness under mismatched conditions. This advantage may be attributed to the fact that the Conformer primarily learns speech distribution patterns through implicit learning during sequence modeling, whereas the VAD module leverages explicit supervision to discriminate speech from noise, thereby yielding stronger generalization capability across unseen domains.

6.1.5 Complexity analysis↩︎

For the HAs system, there is a crucial trade-off between performance and model statistical characteristics, including model size and complexity. The computational complexity and parameter size of the AFN-HearNet are primarily influenced by two key factors: the number of intermediate channels and the dimensionality of the downsampled frequency bins. The intermediate channels determine the feature processing dimension within the AMFT-Conformer block, with higher dimensions necessitating greater computational and memory resources. Additionally, the dimensionality of the downsampled frequency bins directly affects both the temporal steps in the F-Conformer and the number of parallel processing units in the T-Conformer.

To evaluate the model’s sensitivity to these factors and gain insights into the trade-offs between model complexity and performance, we conducted experiments with different configurations, denoted as C\(m\)DF\(n\), derived from the Cartesian product of middle channels \(m\in[32, 48, 60]\) and frequency dimension down-sampled factor \(n\in[4, 8]\). For example, C36DF8 refers to the model with 36 intermediate channels and frequency down-sampled by a factor of 8, achieved by doubling the number of stacked down-sampling layers in the spectrum encoder. The results in Fig. 9 (a)-(c) demonstrate that as the number of channels increases from C36 to C60 in the DF4 configuration, the parameters rise from 0.91 M to 1.76 M, representing a 1.9\(\times\) increase, while the floating point operations per second (FLOPs) increase from 1.93 G to 5.25 G, reflecting a 2.7\(\times\) surge. Similarly, in the DF8 configuration, this channel expansion rises parameters from 0.94 M to 1.83 M, also a 1.9\(\times\) increment, and FLOPs from 1.18 G to 3.20 G, marking a 2.7\(\times\) increment. Additionally, performance metrics indicate that this increase in complexity yields diminishing returns. Fig. 9 (d) illustrates that while C60DF4 achieves the highest scores across all metrics, C48DF4 manages to retain 90% of these scores with 40% fewer FLOPs. Although the DF8 variants consistently demonstrate better efficiency, requiring 39% fewer FLOPs than the DF4 variants with the same number of channels, there are substantial decreases in all metrics, which is unacceptable. Therefore, adjusting the channel dimension is a more effective way to reduce model complexity. Consequently, we choose the C48DF4 configuration as the AFN-HearNet, and the C36DF4 variant as the AFN-HearNet-small, given its performance at a lower complexity level.

To further analyze the computational characteristics of the proposed model, we decompose the multiply–accumulate operations (MACs) by major network modules and present their proportional contributions to the overall computational cost, as shown in Fig. 10. Under the C48D4 configuration (i.e., AFN-HearNet), 29.28% of the total parameters and 39.16% of the total MACs are contributed by the intermediate AMFT-Conformer module. Within this module, the frequency–temporal modeling Conformer accounts for 30.8% of the overall computational complexity, while the embedded modulation fusion submodule contributes 8.36%. Furthermore, the analysis shows that although the dilated DenseNet exhibits strong global feature extraction capabilities, it is a computation-intensive module, accounting for 15.13% of the total computational cost despite having relatively few parameters. Consequently, the dilated DenseNet-based SpectrumEncoder, MaskDecoder, and PhaseDecoder contribute 15.99%, 20.73%, and 20.81% of the total MACs, respectively. In this work, we propose a personalized speech enhancement model operating under causal constraints with a theoretical latency of 16 ms, prioritizing performance while maintaining flexibility in optimizing the core computational load. Specifically, the modulation fusion can be decoupled from the FT-modeling module, and the Conformer component can be substituted with a more lightweight alternative. In addition, model compression can be further achieved through knowledge distillation, enabling a lightweight student model to learn from the original network, which may achieve a 2–4× reduction in model size with only a slight degradation in quality [79].

6.2 Comparisons with advanced approaches↩︎

6.2.1 Results on DNS-challenge dataset↩︎

Fig. 12 presents the violin plots of results for all baselines on the DNS-Challenge test set, effectively illustrating the distribution of objective metrics across different methods. The plots indicate that AFN-HearNet not only achieves a better central tendency but also exhibits a relatively narrow distribution compared to all baselines. Table 5 shows the mean results of each model on the DNS-Challenge test set. The results demonstrate that the AFN-HearNet achieves the highest scores across key speech quality metrics, including HASQI, WB-PESQ, NB-PESQ, and STOI, and considerably outperforms the second-best baseline, HA-MGAN, with improvements of 5.19%, 3.38%, 2.49%, and 1.76%, respectively. While the NUNet_TLS achieves marginally higher scores in signal-level metrics, with an SDR of 13.98 and an SI-SNR of 13.71, the proposed AFN-HearNet demonstrates competitive performance with an SDR of 13.86 and an SI-SNR of 13.62. Moreover, the AFN-HearNet delivers substantial improvements across all other key metrics. Specifically, the AFN-HearNet outperforms the NUNet_TLS by 3.85% in HASQI, achieving 0.81 compared to 0.78; 29.11% in WB-PESQ, with 2.75 compared to 2.13; 17.92% in NB-PESQ, with 3.29 compared to 2.79; and 1.67% in STOI, achieving 95.55 compared to 93.98. Additionally, the AFN-HearNet exhibits advantages in parameter count and computational complexity, requiring only 45% and 62% of the resources used by NUNet_TLS, respectively, highlighting the superiority of our model compared to NUNet_TLS.

A two-tailed Student’s t-test is employed to rigorously assess the statistical significance of the observed performance differences between AFN-HearNet and each baseline method, with the corresponding results reported in Table 6. Overall, AFN-HearNet significantly outperforms all baselines in most metrics, especially in perceptual quality scores, such as HASQI, WB-PESQ, NB-PESQ, and STOI, where significant improvements (\(p<0.05\)) are observed across nearly all comparisons. For the SDR and SI-SNR metrics, AFN-HearNet achieves statistically significant improvements over most methods; however, for a few methods (e.g., DCCRN, CMGAN, and NUNet_TLS), the differences are not statistically significant (\(p > 0.1\)), indicating negligible performance gaps.

Fig. 11 illustrates the performance–efficiency trade-off among all compared methods, where the horizontal axis represents computational cost in GFLOPs and the vertical axis denotes the metric scores. The red dashed line highlights the Pareto frontier, indicating methods that achieve the best possible performance for a given computational budget. Both AFN-HearNet and its small variant lie on the Pareto frontier, demonstrating their competitive efficiency–performance profiles. Although NUNet_TLS also appears on the Pareto frontier for the SDR metric, it requires 61.65% more computational resources than AFN-HearNet, yet offers only a negligible SDR gain of 0.12 dB

Unlike above masking- or mapping-based speech enhancement baselines that directly estimate clean spectral components from noisy inputs, the diffusion model [80]–[82] adopts a fundamentally different generative paradigm, reconstructing enhanced speech through iterative denoising. We further train the SGMSE [80] diffusion model on the DNS-Challenge dataset as a standard speech enhancement module, followed by the FIG6 HLC algorithm to achieve personalized speech enhancement. This two-stage configuration conforms to the classic hearing-aid compensation framework, enabling evaluation of a diffusion model–based approach within this established paradigm. Under conditions of a large model footprint and non-causal processing, diffusion models tend to deliver superior performance, benefiting from their iterative refinement process and strong generative prior. However, by comparing the results of SGMSE and SGMSE+FIG6, we observe that although SGMSE alone achieves strong denoising performance, the subsequent application of the HLC algorithm consistently degrades all quality metrics. Notably, WB/NB PESQ scores decrease from 2.77/3.31 to 2.56/3.17. This degradation suggests that the two-stage compensation paradigm, in which speech enhancement and hearing compensation operate as decoupled processes, fails to preserve the intricate interdependencies between noise suppression and perceptual compensation, thereby introducing additional distortions. In contrast, the jointly optimized AFN-HearNet demonstrates a clear advantage in speech quality, indicating that integrating enhancement and compensation within a unified framework better preserves perceptual fidelity.

6.2.2 Results on LibriSpeech+Demand dataset↩︎

Fig. 14 visualizes the results of various models on the LibriSpeech+Demand test set, demonstrating that our AFN-HearNet and DB-AIAT joint model outperform others in terms of speech compensation. Additionally, the AFN-HearNet shows better performance than DB-AIAT in the NR task. This finding aligns with Table 8, which illustrates the detailed results of various baselines on the LibriSpeech+Demand test set. The statistical analysis confirms that AFN-HearNet achieves significant improvements (\(p\) < 0.05) over all baselines in HASQI, WB-PESQ, and NB-PESQ, with the exception of PrimeK-Net, for which the differences are not statistically significant (\(p\) > 0.1). Although the DB-AIAT baseline achieves the highest scores in SDR at 14.32 and SI-SNR at 13.86, the AFN-HearNet closely follows with scores of 13.72 and 13.28. Despite the marginal difference in SDR and SI-SNR, the AFN-HearNet demonstrates considerable overall performance across other metrics, highlighting its competitiveness. Specifically, in terms of HASQI, WB-PESQ, and NB-PESQ, the AFN-HearNet significantly outperforms the DB-AIAT by 8.45%, 49.16%, and 35.02%, respectively. Compared to the PrimeK-Net’s STOI performance of 93.06, the AFN-HearNet maintains a comparable intelligibility score of 92.35 and excels in other key metrics, achieving improvements of 6.94% in HASQI, 1.91% in WB-PESQ, 19.72% in SDR, and 31.23% in SI-SNR.

These results demonstrate that the AFN-HearNet offers a more comprehensive performance with lower computational costs. Specifically, it attains the highest HASQI and WB-PESQ scores while requiring only 3.39 G FLOPs, which is less than one-third of PrimeK-Net (11.18 G) and approximately one-quarter of DB-AIAT (13.12 G). This corresponds to computational cost reductions of 69.69% and 74.16%, respectively Additionally, Fig. 13 illustrates the performance–efficiency trade-offs of AFN-HearNet and the baselines on the LibriSpeech test set, with FLOPs plotted against HASQI, WB-PESQ, and SDR. Both AFN-HearNet and its small variant occupy positions along the Pareto frontier. These results demonstrate that the AFN-HearNet offers a more comprehensive performance with lower computational costs.

7 Conclusion↩︎

In this paper, we propose a novel joint model named AFN-HearNet that delivers PSE tailored to the individual’s hearing capabilities. By fusing cross-domain features of the spectrum and the hearing loss audiogram, the AFN-HearNet tackles the NR and HLC tasks simultaneously, effectively addressing the interactions between these two tasks and enabling systematic optimization. We propose an audiogram-specific encoder that transforms the sparse audiogram features into personalized deep representations, addressing the alignment problem in cross-domain feature fusion. Following the encoders, the AMFT-Conformer integrates affine modulation and Conformer’s attention mechanisms, iteratively fusing the spectrum deep representations and the audiogram-derived personalized representations at different levels. In the speech reconstruction stage, we introduce a VAD auxiliary training task that implicitly embeds speech and non-speech patterns into the unified deep representations, which helps the model better distinguish between speech and non-speech segments, further improving the overall performance. Experimental results show that the AFN-HearNet significantly outperforms SOTA in-context fusion joint approaches across the key metrics of HASQI and PESQ. Moreover, it maintains a low parameter count and computational complexity, representing a considerable trade-off between performance and efficiency. In future work, we plan to explore extensions for binaural HA scenarios by leveraging spatial audio cues to enhance the listening experience. Additionally, we aim to investigate knowledge distillation techniques to reduce model complexity, making it more suitable for resource-constrained devices.

8 Acknowledgments↩︎

This work was supported in part by the National Key Research and Development Program of China under Grant No. 2020YFC2004003, the National Natural Science Foundation of China under Grant No. 61871213, and by the China Scholarship Council under Grant No. 202406090186.

Extra information↩︎

9 Impact of different weighting factors on generative loss↩︎

Among the generative loss function defined in Equation 1 , the adversarial loss \(\mathcal{L}_{adv}\) and the perceptual loss \(\mathcal{L}_{pmsqe}\) aim to jointly optimize the key metrics HASQI and PESQ, whose performance is directly influenced by the weighting factors (i.e., \(\alpha\) and \(\lambda\)). We evaluate different weight combinations by first fixing \(\alpha=1.00\) with \(\lambda \in \{0.30, 0.45, 2.00\}\), and then fixing \(\lambda=0.30\) with \(\alpha \in \{0.50, 1.00, 1.50, 2.50\}\). Results in Table 10 indicate that, when fixing the HASQI adversarial loss weight \(\alpha=1.00\), increasing the PMSQE loss weight \(\lambda\) leads to consistent improvements in WB/NB-PESQ, SDR, and SI-SNR metrics, but with a slight decrease in the STOI metric. These results demonstrate that emphasizing the PMSQE perceptual loss facilitates improvements in perceptual quality. Nevertheless, they also highlight that the optimization objectives for perceptual speech quality and intelligibility are not inherently aligned. As \(\alpha\) increases from 0.5 to 1.5, the evaluation metrics remain relatively stable. Nevertheless, a substantial degradation is observed at \(\alpha=2.5\), suggesting that an overly large adversarial loss weight adversely affects the overall reconstruction performance.

Ye Ni received the M.S.degree from Nanjing University, Nanjing, China, in 2022. He is currently working toward a Ph.D.degree from Southeast University, Nanjing, China. His research interests include deep learning-based speech enhancement, acoustic echo cancellation, and signal processing.

Ruiyu Liang (Member, IEEE) received the Ph.D.degree from Southeast University, Nanjing, China, in 2012. He is a Professor at the Nanjing Institute of Technology, Nanjing, China. His research interests include speech signal processing and signal processing for hearing aids.

Xiaoshuai Hao received his Ph.D. from the Institute of Information Engineering, Chinese Academy of Sciences, in 2023. He is currently a research expert in multimodal algorithms for autonomous driving and robotics at Xiaomi EV. His research interests include embodied intelligence, multimodal learning, and autonomous driving. Dr. Hao has published over 30 papers in top-tier journals and conferences, including TIP, Information Fusion, NeurIPS, ICLR, ICML, CVPR, ICCV, ECCV, ACL, AAAI, and ICRA. He has achieved significant success in international competitions, securing top-three placements at prestigious conferences such as CVPR and ICCV. Additionally, he serves on the editorial board of Data Intelligence and is an organizer for the RoDGE Workshop at ICCV 2025 and the RoboSense Challenge at IROS 2025.

Jiaming Cheng received the B.E.degree from Nanjing Normal University, Nanjing, China, in 2019. He is currently working toward a Ph.D.degree from Southeast University, Nanjing, China. His research interests include speech enhancement and deep learning.

Qingyun Wang Qingyun Wang received the M.S. degree in Computer Engineering and the Ph. D. degree in Information and Communication Engineering from Southeast University, Nanjing, China, in 2001 and 2011 respectively. Now she is a professor of Nanjing Institute of Technology, Nanjing, China. Her current research interests include acoustic signal processing, speech enhancement and microphone array signal processing.

Chengwei Huang (Member, IEEE) received his undergraduate degree in 2006, and Ph.D.for speech emotion recognition from Southeast University (China) in 2013. His main research interests include affective computing, signal processing, and digital health. He conducted research on human-computer interaction as an associate professor in Soochow University from 2013 to 2014. He is currently with the Zhejiang Laboratory as a principal investigator studying digital health, learning, and well-being.

Cairong Zou received the B.S., M.S., and Ph.D.degrees from Southeast University, China, in 1984, 1987, and 1991, respectively, in electrical engineering. He was a Postdoctoral Fellow with the Department of Electrical and Computer Engineering, Concordia University, Canada, in 1992. He is currently a Professor with Southeast University.

Wei Zhou (IEEE Senior Member) is an Assistant Professor at Cardiff University, United Kingdom. Previously, Wei studied and worked at other institutions such as the University of Waterloo (Canada), the National Institute of Informatics (Japan), the University of Science and Technology of China, Intel, Microsoft Research, and Alibaba Group. Dr Zhou is now an Associate Editor of IEEE Transactions on Neural Networks and Learning Systems (TNNLS), ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), and Pattern Recognition. Wei’s research interests span multimedia computing, perceptual image processing, and computational vision.

Weiping Ding (M’16-SM’19) received the Ph.D. degree in Computer Science, Nanjing University of Aeronautics and Astronautics, Nanjing, China, in 2013. From 2014 to 2015, he was a Postdoctoral Researcher at the Brain Research Center, National Chiao Tung University, Hsinchu, Taiwan, China. In 2016, he was a Visiting Scholar at National University of Singapore, Singapore. From 2017 to 2018, he was a Visiting Professor at University of Technology Sydney, Australia. Now he is the Full Professor of Nantong University. His main research directions involve granular data mining and multimodal machine learning. He has published over 380 articles, including over 190 IEEE Transactions papers. His twenty authored/co-authored papers have been selected as ESI Highly Cited Papers. He serves as an Associate Editor/Area Editor/Editorial Board member of more than 10 international prestigious journals, such as IEEE Transactions on Neural Networks and Learning Systems, IEEE Transactions on Fuzzy Systems, IEEE Transactions on Intelligent Transportation Systems, Information Fusion, Neurocomputing, Applied Soft Computing, et al. He was the Leading Guest Editor of Special Issues in several prestigious journals, including IEEE Transactions on Evolutionary Computation, IEEE Transactions on Fuzzy Systems, et al.

Björn W.Schuller (Fellow, IEEE) received the diploma, the doctoral degree, and the habilitation and Adjunct Teaching Professorship in the subject area of signal processing and machine intelligence, all in electrical engineering and information technology from the Technical University of Munich (TUM), Germany, in 1999, 2006, and 2012, respectively. He is a Full Professor of artificial intelligence, and the Head of GLAM – the Group on Language, Audio & Music, Imperial College London, U.K., a Full Professor and Chair of Health Informatics at TUM, co-founding CEO and current CSO of audEERING. He is also with Munich’s MCML, MDSI, and MIBE. Previous stations include the University of Augsburg and Passau, Germany, as Full Professor, the French CNRS, and Joanneum Research in Graz, Austria. He is President Emeritus and Fellow of the AAAC, Fellow of the ACM, BCS, DIRDI, ELLIS, IEEE, and ISCA. He authored or coauthored five books and more than 1500 publications in peer reviewed books, journals, and conference proceedings leading to more than 65k citations (h-index = 116).

References↩︎