1 Introduction↩︎

In surveillance scenarios, the presence of noise, occlusions, variations in illumination, and varying poses poses a challenge even for sota face recognition algorithms, leading to a significant decline in performance [1]. The most critical issue is the low-resolution and low-quality images acquired in real-world scenarios. Thus, the idea of utilizing sr algorithms as a pre-processing step for face recognition is not new [2] and arose as a natural solution to generate images with higher quality. However, the super-resolution problem is inherently ill-posed, making the recovery of fine details and a reliable identity quite challenging [3]–[6]. In this sense, some works have focused on performing super-resolution by leveraging soft attributes such as eyeglasses, beards, mustaches, gender, and others as an additional source of information to reduce ambiguity and provide more robust results [7]–[10]. Nevertheless, facial attributes are often indistinct in lr images, making reliable identification challenging. Also, obtaining these attributes requires a classifier or manual extraction, which is not very efficient [10]. However, many characteristics can be employed to assist super-resolution algorithms. These include subtle facial proportions, skin textures, shapes, and other high-level, more abstract features that are not easily labeled or categorized.

Given the scarcity of robust super-resolution methods capable of preserving identity, this work aims to address this gap by focusing on identity preservation. We develop srname, a robust sr algorithm that recovers crucial features for face recognition. It is more effective because, in addition to the lr image, it also takes as input a reliable vector of facial features derived from a set of lr images, such as a series of video frames or independent photos of an individual. This new vector has a higher signal-to-noise ratio than each individual vector. We incorporate it into the network, merging its information with the lr image to generate an sr version. In this way, our algorithm effectively recovers facial information from an image, yielding higher-quality results with minimal distortion of identity. Notably, srname employs a diffusion model based on a sde and does not require a classifier to guide the reverse diffusion process.

A preliminary version of this work, introducing the fasr method, was published at the 2024 Conference on Graphics, Patterns and Images (SIBGRAPI) [11]. This paper builds upon that work with the following key improvements: (i) we develop a neural network specifically designed to enhance feature integration in low-resolution images, enabling more effective fusion and recovery of high-frequency details. A hypothesis test validates its ability to merge complementary features; (ii) we incorporate additional evaluation metrics, expanding beyond recognition and verification performance to include image quality assessments such as Structural Similarity Index Measure (SSIM), Peak Signal-to-Noise Ratio (PSNR), and Learned Perceptual Image Patch Similarity (LPIPS) [12]; (iii) we conduct a comprehensive ablation study, demonstrating that the proposed method significantly outperforms the original FASR and other baseline models.

Our method’s effectiveness has been validated on the popular CelebA [13] and Quis-Campi [14] datasets, and the key contributions are as follows:

• We present a neural network that fuses and enhances low-resolution features, and we demonstrate its effectiveness in feature integration;

• We introduce FASR++, an improvement of FASR [11], which utilizes the proposed neural network to generate high-quality merged features for assisting the reconstruction of super-resolution images in the diffusion model;

• Our approach achieves superior qualitative results, producing more natural images with less distortion compared to sota super-resolution algorithms;

• Our quantitative results outperform sota algorithms in both image quality metrics, such as PSNR, SSIM and LPIPS, and identity-related metrics, including auc in 1:1 verification, and accuracy in the 1:N identification.

2 Related Work↩︎

[15] introduced a generative model based on principles from non-equilibrium thermodynamics in their seminal work. Two other influential studies in the field of diffusion models are Denoising Diffusion Probabilistic Models (DDPMs) [16] and Score-Based Generative Models (SGMs) [17], [18]. In [19], DDPM and SGM are generalized for continuous time steps and noise levels using sdes, expanding the range of research possibilities in diffusion models.

Due to the rapid evolution of diffusion models, various opportunities for their application have emerged. Recent works include the generation of audio, graphs, and shapes, as well as image synthesis, solutions of general inverse problems, and applications in medical images [16], [17], [19]–[22]. The full potential of diffusion models can also be leveraged through multi-domain data integration, such as text-to-image translation [23] and image editing [24]. Additionally, [25] combines audio-visual information for speech enhancement.

sr is another important application of diffusion models and is investigated in this work. In [26], an adaptation of the DDPM model produces high-quality sr images. Similarly, SRDiff [27] employs diffusion models to estimate the difference between the original lr image and a hr image, resulting in an sr image. In [28], sdes were used to generate sr images. Additionally, [10] performs sr by incorporating attribute information such as beard, gender, and the presence of eyeglasses to generate high-quality images. However, their approach has the drawback that these attributes must be explicitly provided to the algorithm, which cannot be easily estimated in lr images.

In [29], an identity-preserving sr method was developed. In both [10] and [29], a gradient must be calculated during the image reconstruction phase, which can increase computational cost. In this study, we develop an algorithm that restores image attributes by supplying a compact descriptor of facial features for the algorithm.

Despite the impressive results achieved by diffusion models, their primary drawback is the high execution time caused by their iterative nature. Nevertheless, this issue is expected to be mitigated in the near future, as many studies focus on improving the computational efficiency of these methods. For a more in-depth discussion on accelerating sampling and enhancing efficiency in diffusion models, refer to [30]–[32].

3 Proposed Method↩︎

In this section, we present the general concept of the proposed method, followed by the description of a Feature Combiner module, the theoretical background on diffusion models formulated as SDEs, the model architecture, and the conditioning mechanisms based on low-resolution images, time, and feature embeddings.

3.1 General Idea↩︎

As previously noted, images captured in surveillance environments are often of low quality. Nevertheless, in certain instances, a video of a particular person can provide multiple low-resolution images that, when combined, can increase the valuable information necessary to recognize an individual.

In this work, we employ an sr algorithm to enhance a low-resolution image (LR\(_0\)), recovering useful information for face recognition (see Figure 1). We use a set of low-resolution auxiliary images LR\(_1,\dots,\)LR\(_N\) of the same individual to extract a set of compact descriptors \(\mathbf{F}_1,\dots,\mathbf{F}_{N}\). We train a Feature Combiner (FC) module (see Figure 2) to merge two vectors and recover the image’s high-frequency information. The vectors \(\mathbf{F}_1,\dots,\mathbf{F}_{N}\) are combined through an ensemble of FCs to generate a representative compact descriptor \(\boldsymbol{F}_M\). The reference low-resolution image LR\(_0\) and the merged vector \(\boldsymbol{F}_M\) are input into a diffusion model, which generates super-resolution images with minimal distortions in the identity. Due to the stochasticity of diffusion models and the ill-posed nature of super-resolution problems, we can generate different solutions with small variations consistent with the same input LR\(_0\). To increase the method’s robustness, we generate two super-resolution images, SR\(_a\) and SR\(_b\), and combine them through an average to generate the final image, SR. This general framework leads to enhanced reliability concerning person identification. Moreover, the algorithm successfully retrieves high-level features that might not be clearly visible but significantly enhance recognition accuracy and image quality.

3.2 Feature Combiner (FC)↩︎

None

To train the fc module, we need a collection of low-resolution images and their high-resolution versions. Specifically, for a given identity, let \(\text{LR}_1, \dots, \text{LR}_N\) denote the set of \(N\) low-resolution images, and let \(\text{HR}_1, \dots, \text{HR}_N\) denote the respective set of high-resolution images. We then extract a set of compact face descriptors from these images using a pre-trained visual encoder to obtain the descriptors \(\mathbf{F}_1,\dots,\mathbf{F}_{N},\mathbf{F}_1^\text{HR},\dots,\mathbf{F}_{N}^\text{HR}\) respectively. The feature vectors from low-resolution images are combined in pairs using an ensemble of FC modules and posteriorly averaged to obtain an approximation of a reliable descriptor of a high-resolution image.

More specifically, given two feature vectors \(\mathbf{F}_i\) and \(\mathbf{F}_{i+1}\), the FC module takes them as input and produces a single merged representation \(\mathbf{F}_{\text{out},i'}\), where \(i' = (i+1)/2\), as illustrated in Figure 2. The FC module consists of the mean of the input features and a refinement network \(\delta\). The mean operation was adopted as a baseline because the input low-resolution images, and consequently their extracted features, often contain noise and small distortions. Averaging allows the model to amplify the facial components that are common across different images of the same identity while attenuating uncorrelated noise and spurious variations. The FC then combines this mean with a learnable correction term generated by the neural network \(\delta\), which captures nonlinear relationships between the two feature vectors, modeling complex dependencies that cannot be represented by simple averaging. Formally, the operation is \[\mathbf{F}_{\text{out},i'} = \text{FC}\left(\mathbf{F}_i, \mathbf{F}_{i+1}\right) = \frac{\mathbf{F}_i + \mathbf{F}_{i+1}}{2} + \eta \cdot \delta\left(\mathbf{F}_i, \mathbf{F}_{i+1}\right), \label{eqmerge}\tag{1}\] where \(\delta(\cdot)\) denotes the neural network and \(\eta \in \{0,1\}\) is a control parameter used to assess the influence of \(\delta\) on the final results.

The architecture of \(\delta\), presented in Figure 2, consists of three fully connected layers. The input is formed by concatenating the feature vectors \(\mathbf{F}_i\) and \(\mathbf{F}_{i+1}\), producing a 1024-dimensional representation. This concatenated vector is processed by two sequential linear transformations, each followed by Batch Normalization (BN) and ReLU activation, with a dropout rate of 0.2 applied after each nonlinear operation to improve generalization and reduce overfitting. The chosen dropout rate of 0.2 was determined through validation experiments and aligns with the typical values reported by [33]. A final linear transformation produces a 512-dimensional feature refinement, which is combined with the mean \((\mathbf{F}_i + \mathbf{F}_{i+1}) / 2\) to generate the fused representation \(\mathbf{F}_{\text{out},i'}\). Finally, an \(L_2\) normalization is applied to ensure unit-length embeddings.

To optimize the network parameters, we employ a triplet loss, aiming to maximize the similarity between positive matches while minimizing it for negative ones. In this framework, the anchor, positive and negative samples are:

• Anchor: \(\mathbf{F}_{\text{out},i'}=\text{FC}\left(\mathbf{F}_i, \mathbf{F}_{i+1}\right)\), the combined feature representation generated by the FC module;

• Positive sample: The arithmetic mean of feature vectors from the respective HR image, i.e.: \[\frac{\mathbf{F}_i^\text{HR} + \mathbf{F}_{i+1}^\text{HR}}{2};\]

• Negative sample: A feature vector from a HR image of a different identity.

After the network \(\delta\) is trained and each FC module effectively merges two feature vectors, an ensemble of FCs is employed to obtain the overall merged representation for a given individual. The final feature vector is computed by averaging all \(N/2\) merged outputs \(\mathbf{F}_{\text{out},i'}\), resulting in the final merged feature \(\mathbf{F}_M\), as expressed by \[\mathbf{F}_{M} = \frac{2}{N} \sum_{i'=1}^{N/2} \mathbf{F}_{\text{out},i'} = \frac{1}{N} \sum_{i=1}^{N} \mathbf{F}_{i} + \frac{2\eta}{N} \sum_{i=1}^{N/2} \delta(\mathbf{F}_{2i-1}, \mathbf{F}_{2i}), \label{eqfm}\tag{2}\] for even \(N\). When \(N\) is odd, the last feature vector \(\mathbf{F}_{N}\) is averaged together with the \((N-1)/2\) merged features \(\mathbf{F}_{\text{out},i'}\). This formulation is general and applies to any number of input images. The averaging operation in Equation 2 can be interpreted as analogous to the average pooling mechanism commonly employed in convolutional neural networks, where feature-wise averaging effectively reduces noise while preserving the dominant structural information.

In Equation 2 , other operations, such as the maximum or the sum, could also be used. However, based on experimental analysis, we found that the mean yields superior results compared to these alternatives. Using the maximum may distort the features by emphasizing individual noisy components rather than the common underlying structure. A simple sum, in turn, would unnecessarily amplify the magnitude of the features as \(N\) increases. In contrast, the mean provides a stable aggregation that yields merged features \(\mathbf{F}_{\text{out},i'}\) closer to the target high-resolution ground truths, ensuring a coherent estimate of the underlying representation. Since the individual feature vectors are normalized, the mean effectively balances their contributions without being dominated by any single component. Moreover, residual noise components may still be present, and averaging serves as a natural denoising mechanism that increases the signal-to-noise ratio and stabilizes the overall representation across different image conditions.

From Equation 2 , it follows that the mean of \(\mathbf{F}_{\text{out},i'}\) is equivalent to a single global mean over all input features \(\mathbf{F}_{i}\), plus an additional term involving \(\delta(\cdot,\cdot)\) that represents the nonlinear refinement computed from sequential feature pairs by the Feature Combiner. Therefore, if another nonlinear function were more suitable for feature aggregation than the mean, the \(\delta\) network would implicitly compensate for it.

This formulation ensures that the learned representation effectively consolidates the features while preserving identity and enhancing discriminability among individuals.

3.3 Theoretical Background↩︎

In the context of image generation, diffusion models have two phases: forward diffusion and reverse diffusion. During forward diffusion, Gaussian noise is added to the image, and a network is trained to predict this noise. In reverse diffusion, an image composed purely of noise is iteratively denoised and transformed into an image that follows a distribution similar to the images in the training set. If the diffusion procedure is continuous, it can be modeled using an sde.

According to [19], [34], a forward diffusion process \(\{\mathbf{x}(t)\}_{t=0}^T\) and its reverse are, respectively, modeled using the following sdes: \[\mathrm{d}\mathbf{x}=\mathbf{f}(\mathbf{x},t)\mathrm{d}t+g(t)\mathrm{d}\mathbf{w}, \label{eqsde}\tag{3}\] \[\mathrm{d}\mathbf{x}=[\mathbf{f}(\mathbf{x},t)-g(t)^2\nabla_{\mathbf{x}}\log p_t(\mathbf{x})]\mathrm{d}t+g(t)\mathrm{d}\bar{\mathbf{w}}, \label{revsde}\tag{4}\] where \(\mathbf{f}(\mathbf{x},t)\) is the drift coefficient, \(g(t)\) is a diffusion coefficient, \(\mathbf{w}\) and \(\bar{\mathbf{w}}\) are Wiener process (the latter runs backward in time) and \(p_t\) is the probability density of \(\mathbf{x}(t)\). [35], [36] supply more details about Itô sdes and the Wiener process.

Here, we consider \(\mathbf{x}_t\) as an image to be denoised. At \(t=0\), the noise level in the image is zero, and at \(t=T\), the noise is at its maximum, and there is no information on the image. To obtain a super-resolved image, we solve the reverse diffusion process defined in Equation 4 . For this purpose, a deep neural network \(s_\theta\) is employed to approximate the score function \(\nabla_{\mathbf{x}}\log p_t(\mathbf{x})\). During this reverse process, \(s_\theta\) is conditioned on both the reference low-resolution image \(\text{LR}_0\), denoted by \(\mathbf{y}\), and the merged feature vector \(\mathbf{F}_M\), which provides complementary guidance.

The training of the neural network \(s_{\theta}\) is achieved by optimizing the following loss function [37]: \[\begin{align} \min_{\theta} \mathbb{E}_{t\sim \mathcal{U}[0,T]} \mathbb{E}_{\mathbf{x}(0)\sim p(\mathbf{x}(0))}\mathbb{E}_{\mathbf{x}(t)\sim p_t(\mathbf{x}(t)|\mathbf{x}(0))} \big[\lambda(t)\quad\nonumber\\ \times\, \|s_\theta(\mathbf{x}(t),\mathbf{y},\mathbf{F}_\mathbf{y}^\text{HR},t)-\nabla_{\mathbf{x}(t)}\log p_{}(\mathbf{x}(t)|\mathbf{x}(0))\|_2^2\big], \label{eqloss} \end{align}\tag{5}\] where \(\lambda(t)\) is a positive weighting function, \(p_{}(\mathbf{x}(t)|\mathbf{x}(0))\) is the transition kernel from \(\mathbf{x}(0)\) to \(\mathbf{x}(t)\) and \(\mathbf{F}_{\mathbf{y}}^{\text{HR}}\) represents the ground-truth features extracted from the high-resolution version of \(\mathbf{y}\).

Here, we use the ve case described in [19] with \(\mathbf{f}(\mathbf{x},t)\) and \(g(t)\) given respectively by: \[\mathbf{f}(\mathbf{x},t)=\mathbf{0},\quad g(t)=\sqrt{\frac{\mathrm{d}\sigma^2(t)}{\mathrm{d}t}} \, ,\] where \(\sigma(t)=\sigma_{min}\left(\sigma_{max}/\sigma_{min}\right)^t\) denotes the noise level of the image at the time \(t\).

For \(\mathbf{f}(\mathbf{x},t)\) and \(g(t)\) described above, the mean and variance of \(p(\mathbf{x}(t)|\mathbf{x}(0))\) are given by [19]: \[\pmb{\mu}(t) = \mathbf{x}(0), \,\, \pmb{\Sigma}(t)=[\sigma^2(t)-\sigma^2(0)]\mathbf{I}. \label{eq95mean}\tag{6}\] Thus, we can analytically compute \(\nabla_{\mathbf{x}}\log p(\mathbf{x}(t)|\mathbf{x}(0))\) in Equation 5 , allowing for efficient model training. Once the network well estimates the gradient, we generate an SR image \(\mathbf{x}(0)\) by changing \(\nabla_{\mathbf{x}}\log p_t(\mathbf{x})\) by \(s_\theta(\mathbf{x}(t),\mathbf{y},\mathbf{F}_M,t)\) in the reverse process (Equation 4 ) and solving it from \(t=T\) to \(t=0\) using the Euler-Maruyama method [35], [36].

3.4 Diffusion Model Architecture↩︎

The proposed model adopts the NCSN++ [19] backbone, implemented as a U-Net [38] with residual and attention blocks distributed in multiple resolutions. The network comprises seven resolution levels (\(128\!\to\!64\!\to\!32\!\to\!16\!\to\!8\!\to\!4\!\to\!2\)), with channel widths scaled by \((1, 1, 2, 2, 2, 2, 2)\) relative to an initial number of 128 feature maps. Each level contains two residual blocks, followed by either downsampling or upsampling operations between successive resolutions.

Each residual block follows a standard design consisting of Group Normalization, a SiLU nonlinearity, and two \(3\times3\) convolutional layers. A dropout rate of \(0.1\) is applied within each block to improve regularization, while a \(1\times1\) convolution is used for channel projection whenever the input and output dimensions differ. In addition, self-attention layers are inserted at the \(16\times16\) resolution to capture long-range spatial dependencies across feature maps [39].

The network is conditioned on three signals: the low-resolution input image, the time embedding, and the merged feature vector. In each residual block, temporal and feature signals are injected through independent dense layers after SiLU activation, as described in Subsection 3.5.

Finally, a \(3\times3\) convolution maps the feature maps back to the RGB space. This configuration, combining residual, attention and multi-scale conditioning mechanisms, improves representational capacity and training stability [40]–[42].

3.5 Model Conditioning↩︎

To guide the diffusion process, our model employs three complementary conditioning mechanisms: \((i)\) the low-resolution image, which provides spatial and textural priors; \((ii)\) the time embedding, which encodes the current denoising stage; and \((iii)\) the merged feature vector, which conveys high-level identity information obtained from the ensemble of feature combiners. Each of these conditioning signals plays a distinct role in steering the generation process toward faithful and identity-preserving reconstructions.

3.5.0.1 LR Image Conditioning.

To condition the model on low-resolution input, we follow a strategy similar to that adopted in [26], [28], where the conditioning image is directly concatenated with the noisy input image along the channel dimension. Specifically, the low-resolution image \(\mathbf{y}\) and the noisy image \(\mathbf{x}_T\) (the sample being progressively denoised) are concatenated to form a six-channel tensor \[\text{concatenate}[\mathbf{y}, \mathbf{x}_T] \in \mathbb{R}^{6\times H\times W},\] where \(H\) and \(W\) denote the spatial dimensions of the image. This tensor serves as the input to the U-Net backbone, allowing the network to take advantage of spatial and textural cues from the LR image throughout the denoising process.

3.5.0.2 Time Conditioning.

The temporal conditioning follows the strategy adopted in other diffusion models [19], where the diffusion timestep \(t \in [0, 1]\) is mapped into a high-dimensional embedding using Gaussian Fourier features [43]. For a given timestep \({t}\), the Fourier mapping is computed as \[\mathbf{E}_t = \text{concatenate}[\sin(2\pi \boldsymbol{\omega} t), \; \cos(2\pi\boldsymbol{\omega} t)],\] where \(\boldsymbol{\omega} \in \mathbb{R}^{d}\) is a fixed, non-trainable vector of random frequencies drawn from a normal distribution. This produces a time embedding \(\mathbf{E}_t \in \mathbb{R}^{2d}\) (with \(d = 256\) in our implementation, yielding 512 dimensions).

The embedding \(\mathbf{E}_t\) is then processed by a SiLU activation followed by a single linear layer that projects it into the same channel dimension \(C\) as the U-Net feature maps. To enable element-wise addition with the convolutional features, \(\mathbf{E}_t\) is reshaped to \(\mathbb{R}^{C\times1\times1}\) and broadcast across \(H\) and \(W\), ensuring that the same temporal modulation is applied uniformly to all spatial positions within each residual block.

Time conditioning plays a crucial role in diffusion models, as it allows the network to adapt its denoising behavior according to the current noise level along the diffusion trajectory, improving both temporal coherence and reconstruction quality.

3.5.0.3 Feature Conditioning.

Analogously, the model is also conditioned on the merged feature vector \(\mathbf{F}_M\) obtained from the ensemble of feature combiners. This vector is processed through a SiLU activation followed by a single linear layer, projecting it into the same channel dimension \(C\) as the network feature maps. Then it is reshaped to \(\mathbb{R}^{C\times1\times1}\) and broadcast across \(H\) and \(W\) to allow element-wise addition with the U-Net feature maps. In this way, the conditioning provided by \(\mathbf{F}_M\) acts as a global control signal, influencing all spatial locations uniformly while preserving the channel-wise semantics learned by the network.

3.5.0.4 Integration within Residual Blocks.

We now describe how these conditioning signals are incorporated into the network. At each residual block of the U-Net, let \(h\) denote the intermediate feature map after normalization and activation. The temporal embedding and the merged feature vector are injected into these residual blocks through independent linear layers D\(_t\) and D\(_f\), as follows: \[h \leftarrow h + \text{D}_t(\text{SiLU}(\mathbf{E}_t))_{\text{reshaped}} + \text{D}_f(\text{SiLU}(\mathbf{F}_M))_{\text{reshaped}}.\] The layers \(\text{D}_t\) and \(\text{D}_f\) project these embeddings into the same channel dimension as \(h\), and the resulting tensors are reshaped to match the spatial dimensions of the feature maps. This mechanism allows both the diffusion timestep and the identity-related information to condition the U-Net feature maps across multiple scales, as illustrated in Figure 3.

3.5.0.5 Qualitative Demonstration of Feature Conditioning.

To demonstrate the effectiveness of using the feature vector to generate an SR image, we trained our model with \(\mathbf{y}=0\) in Equation 5 , i.e., without using the LR image, and employed the ground-truth feature vectors to produce images, as illustrated in Figure 4. These images demonstrate the algorithm’s ability to reconstruct high-level features that encode identity-specific semantics, including facial geometry, landmark configuration, and characteristic texture patterns. The low-resolution image, in turn, provides the global spatial layout and low-frequency information required to preserve geometric coherence, pose, and illumination consistency during reconstruction. It defines the structural basis for the generative process. When both signals are combined, they enable the model to generate visually convincing and identity-consistent faces.

4 Experiments and Results↩︎

This section describes the experimental setup and the results obtained on two different datasets. Lastly, we present the ablation study and examine some extreme cases where the algorithm may fail.

4.1 Experiments↩︎

In this study, we explored four datasets: FFHQ [44], CASIA-WebFace [45], CelebA [13], and Quis-Campi [14], the latter of which originates from surveillance scenarios. The FFHQ dataset was used for the diffusion model training, with \(10^6\) training steps performed, while the CASIA-WebFace dataset was specifically used to train the \(\delta\) network. Further details on the \(\delta\) training procedure are provided in the next section. CelebA was used to test our approach, with \(500\) identities selected. Each identity comprises multiple images, with one randomly chosen as the gallery image. A second image is downsampled to create an lr probe image. The remaining images were also downsampled and used to extract features, assisting the reconstruction of the LR probe image.

A complementary test to further validate our algorithm was conducted on a real-world scenario from the Quis-Campi dataset, where the images pose additional challenges for SR and face recognition algorithms [14]. We selected \(90\) identities and used five downsampled images as probe images for each identity. These images were then used to calculate an average feature vector, which was utilized to support the generation of the SR image. In addition, the dataset already contains gallery images obtained in a controlled environment for each identity.

The parameters controlling the noise level over time were set at \(\sigma_{min}=0.001\) and \(\sigma_{max} = 348\). We worked with images of \(128 \times 128\) pixels. For producing LR images, we applied \(8\times8\) downsampling followed by upsampling using bicubic interpolation to achieve a final size of \(128 \times 128\) pixels. We used \(2{,}000\) steps to solve the SDE for image reconstruction.

The feature vector used for both training the SR algorithm and facial recognition consists of a 512-dimensional vector generated through AdaFace [46] with a ResNet backbone [40] trained on the CASIA-WebFace dataset [45]. Image descriptors were compared using the cosine similarity metric. For the recognition task, we compare the SR-recovered images against the gallery images. Our proposed algorithm is compared against sota algorithms: SPARNET [47], GFPGAN [48], SwinIR [42], SDE-SR [28], IDM [49], SR3 [26], and SRDG [10].

4.2 Training and Evaluation of the \(\delta\) Network↩︎

To train the \(\delta\) network, we used the CASIA-WebFace dataset [45]. In our experiments a total of 490,623 images from 10,572 identities were used, corresponding to an average of 46.41 images per subject. The dataset was divided into training (85%) and validation (15%) subsets. The model was trained for up to 20 epochs, with early stopping applied when no improvement in the validation loss was observed for five consecutive epochs.

To compute the triplet loss, two high-resolution images were randomly selected for each identity. The mean of their feature vectors was used as the positive sample, while the features extracted from their corresponding low-resolution versions were used as inputs to the Feature Combiner module to generate the anchor representation. The negative sample was obtained from the high-resolution features of a different identity.

The work of [50] shows that training can collapse when the margin is too large relative to the initial embedding spread, preventing the embeddings from separating properly. Following this observation, the margin was empirically set slightly above the average gap between the anchor–positive and anchor–negative distances to enhance training stability, resulting in a final value of 0.495.

To demonstrate the effectiveness of the proposed ensemble of FC modules in merging low-resolution features, we conducted experiments on the CelebA dataset using the identities selected in Section 4.1. For each identity, the features of the low-resolution images \(\text{LR}_1, \dots, \text{LR}_N\) are fused using the ensemble of FCs, and the resulting feature representation is compared to the corresponding ground truth gallery feature via cosine similarity. We considered two approaches: \((i)\) \(\eta = 0\), where only the arithmetic mean is used for merging low-resolution features, and \((ii)\) \(\eta = 1\), where the network \(\delta\) refines the arithmetic mean during feature integration. This comparison highlights the advantage of using the network \(\delta\) for refinement over a simple averaging scheme.

The results show that the mean similarity score increased from 0.162 for \(\eta = 0\) to 0.350 for \(\eta = 1\), representing a 116% improvement. A paired t-test confirmed that this difference is highly significant (\(t=-46.33\), \(p<10^{-12}\)), demonstrating that the proposed \(\delta\) network effectively enhances feature merging and produces representations that more closely match the high-resolution gallery features. The histogram analysis in Figure 5 further supports this conclusion, showing a clear rightward shift in the score distribution when the \(\delta\) network is applied to refine the merging process. This improvement in the feature space ultimately translates into better face recognition performance, as discussed in Section 4.3.

4.3 General Results↩︎

In this section, we present the results obtained in our experiments. Tables 1 to 4 summarize the main findings. Dark gray highlights the best values, while light gray indicates the second-best.

Table 1 shows the quantitative results of our proposed method on the CelebA dataset. We used the gallery image as a reference for the first three metrics, auc, Rank-1, and Rank-5, and the high-resolution ground truth image as a reference for the remaining three metrics: PSNR, SSIM, and LPIPS. Compared to other algorithms, FASR++ provides superior results in all metrics, demonstrating its ability to recover discriminative identity features while maintaining high perceptual and structural image quality.

In general, the parameters of super-resolution algorithms are optimized to improve image quality metrics such as PSNR and SSIM; however, these improvements may be accompanied by distortions in identity and impaired recognition performance. Moreover, [28] exemplifies that for a given set of solutions for a super-resolution problem, the images with higher values for PSNR and SSIM metrics are not the images that provide the highest value for recognition metrics. Remarkably, FASR++ maintains superior PSNR and SSIM values, and lower LPIPS scores, while still achieving optimal recognition performance by generating high-quality features through an ensemble of fc modules to assist the diffusion model.

Table 2 presents the quantitative results on the Quis-Campi dataset. The SRDG algorithm uses soft attributes as input to guide its diffusion process during image reconstruction. Although SRDG benefits from this additional conditioning, our method surpasses SRDG and the other competing algorithms in both recognition metrics (Rank-1 and Rank-5) and image quality metrics (PSNR, SSIM, and LPIPS).

To further assess the reliability of the image quality improvements, a paired t-test was conducted to evaluate the statistical significance of the results across both datasets. FASR++ was compared against the second-best performing methods: FASR for PSNR and SSIM, and SR3 for LPIPS. On the CelebA dataset, the improvements in PSNR and SSIM were confirmed to be statistically significant relative to FASR (\(t_\text{PSNR}=28.9\), \(t_\text{SSIM}=22.4\)), while the difference in perceptual quality, measured by LPIPS, was also statistically significant compared to SR3 (\(t_\text{LPIPS}=-4.4\)). Similarly, on the Quis-Campi dataset, statistically significant differences were observed when comparing FASR++ to FASR for PSNR and SSIM (\(t_\text{PSNR}=26.4\), \(t_\text{SSIM}=8.9\)), and to SR3 for LPIPS (\(t_\text{LPIPS}=-16.1\)). In all cases, the differences were highly significant (\(p<10^{-5}\)), confirming that the observed gains are not due to random variation but instead establish the proposed method as the sota in both reconstruction fidelity and perceptual quality.

In Figures 6 and 7, we present the qualitative comparison of our method srname against other super-resolution algorithms for the Celeba and Quis-Campi datasets, respectively. While all other methods are effective to some extent, they often introduce artifacts or noise into the facial images, typical issues encountered in sr algorithms. For instance, in most examples, the images generated by other algorithms exhibit distortions, mainly in the eye region, providing an artificial and distorted appearance. In Figure 6, GFPGAN yields distortions regarding the person’s age. In contrast, srname stands out as the only approach that produces natural-looking images without noticeable artificiality. It preserves symmetries and successfully recovers details without introducing artifacts or distorting facial features.

Due to the ill-posed nature of the SR problem, many SR algorithms suffer from bias issues and struggle to recover a person’s identity accurately. In contrast, our algorithm effectively tackles these challenges, mitigating identity-related problems and yielding superior quantitative and qualitative results.

4.4 Ablation Study↩︎

This section analyzes the impact of the \(\delta\) network and the choice of dataset on both recognition accuracy and image quality, followed by an evaluation of how the number of training samples used in the \(\delta\) network influences its effectiveness and the overall performance of the model.

4.4.0.1 Influence of the \(\delta\) Network and Dataset.

In this section, we conduct an incremental study to examine the impact of each enhancement applied to our proposed method on the CelebA and Quis-Campi datasets. We start with FASR, where low-resolution features are combined using an arithmetic mean to produce a single super-resolution output. Next, we introduce FASR++ (\(\eta=0\)), which still combines low-resolution features via arithmetic mean but generates two super-resolution images that are averaged to create a final image. We also present two versions of our complete approach, FASR++ and FASR\(^\dagger\)++. In both cases, an ensemble of Feature Combiner modules integrated with the \(\delta\) network (\(\eta=1\)) is employed to estimate a reliable descriptor and to produce two super-resolution images. As described in Section 4.2, in FASR++ the \(\delta\) network is trained on the CASIA-WebFace dataset using 10,572 identities, with an average of 46.41 images per subject. Alternatively, in FASR\(^\dagger\)++, the \(\delta\) network is trained on a subset of the CelebA dataset, where we selected 2,000 identities with an average of 20 images per identity. This experiment aims to analyze the influence of dataset limitations such as fewer identities, reduced intra-class variability, and a smaller number of images per identity on the performance of the proposed fusion mechanism.

To further examine the role of the weighting factor \(\eta\) in the fusion process, we performed additional experiments with intermediate values \(\eta \in \{0, 0.25, 0.5, 0.75, 1.0\}\) using the same experimental setup described in Section 4.2. For \(\eta \neq 0\), the resulting cosine similarity distributions were largely overlapping, indicating that \(\eta\) has little effect on the fused representations. This behavior occurs because the FC module is trained to optimize the combined feature representation as a whole, so any scaling change in \(\eta\) tends to be internally compensated by the learned weights of the \(\delta\) network, maintaining consistent FC outputs across different configurations. Therefore, \(\eta = 1\) was adopted in all experiments.

Tables 3 and 4 present the verification and recognition results along with the corresponding image quality metrics. In addition, the detailed CMC curves for the CelebA and Quis-Campi datasets are shown in Figure 8. From Table 3, we observe a consistent improvement as each component is introduced in our incremental study. Averaging two images in FASR++ (\(\eta=0\)) slightly enhances reconstruction quality compared to the baseline FASR, while maintaining similar recognition performance. When the learned fusion mechanism is activated (\(\eta=1\)) in FASR\(^\dagger\)++, both recognition accuracy and perceptual quality improve, with a more pronounced effect on recognition (a \(4\%\) increase in Rank-1 accuracy). Finally, training \(\delta\) on the larger CASIA-WebFace dataset (FASR++) yields the best overall Rank-1 accuracy (a \(6.4\%\) improvement over the baseline), whereas for the other metrics, both FASR\(^\dagger\)++ and FASR++ achieve very similar results.

From Table 4, we observe a similar incremental behavior on the Quis-Campi dataset. Averaging two images in FASR++ (\(\eta=0\)) results in a clear perceptual improvement, yielding a substantial reduction in LPIPS (from \(0.1303\) to \(0.1004\)) compared to FASR. When the learned fusion mechanism is enabled (\(\eta=1\)) in FASR\(^\dagger\)++, the gain is more pronounced in Rank-5 accuracy. Finally, training \(\delta\) on the larger CASIA-WebFace dataset (FASR++) yields the best overall results, except for Rank-5 accuracy.

For both datasets, we observe that using the \(\delta\) network (\(\eta=1\)) provides superior results compared to the other configurations. This outcome was expected since, as discussed in Section 4.2, the \(\delta\) network combines features efficiently. The improved results in both recognition and image quality metrics confirm that the combined features are being effectively leveraged by the diffusion model. Moreover, the larger number of images and the greater identity diversity of the CASIA-WebFace dataset contribute to the superior results of FASR++ compared to FASR\(^\dagger\)++, mainly in terms of Rank-1 accuracy.

We can also observe that the superior results of FASR++ are more pronounced for the CelebA dataset and more subtle for Quis-Campi. This can be attributed to two main reasons: \((i)\) as will be described in the next section, the Quis-Campi dataset is a real-world surveillance dataset, which is more challenging, containing variations in lighting and pose, as well as a higher level of noise compared to CelebA. Consequently, the facial features in Quis-Campi can be noisier and more difficult to reconstruct accurately; \((ii)\) for the Quis-Campi dataset, only five features from low-resolution images were used to assist in the reconstruction of the super-resolution images, while more than eight low-resolution images were used for CelebA, providing a richer source of information for the algorithm.

4.4.0.2 Effect of the Training Dataset Size on the \(\delta\) Network.

We now analyze how the total number of samples used for training the \(\delta\) network affects its ability to effectively merge features and influence the final results of the FASR++ framework, using the CelebA dataset as the evaluation benchmark. In this experiment, we limit the number of images per identity in the CASIA-WebFace dataset and evaluate the impact on the PSNR, SSIM, Rank-1, and Rank-5 metrics. Figure 9 shows that as the number of training samples increases, both image quality metrics (PSNR and SSIM) and recognition metrics (Rank-1 and Rank-5 accuracies) remain relatively stable with only minor fluctuations, suggesting that approximately 50k images are sufficient to achieve optimal performance in both reconstruction quality and face recognition. In this range, the Rank-1 and Rank-5 scores converge to around 60% and 75%, respectively, while statistical analysis confirmed that, despite small variations in PSNR and SSIM, no significant differences were observed when training with 50k images or more (\(p > 0.05\)).

4.5 Failure Cases↩︎

Figure 10 shows some failure cases of our algorithm compared to SRDG and SDE-SR. In the first row, srname fails to recover the eyeglasses correctly, whereas SRDG successfully recovers this attribute. However, it is important to note that SRDG requires explicit information on whether the person is wearing eyeglasses. This information is not always discernible from lr images in surveillance scenarios.

In the second row of Figure 10, we observe a failure case of srname compared to SDE-SR. The image in question shows significant pose variation and highly heterogeneous illumination. srname produces smoother images with less noise than the other algorithms, causing the information about eyeglasses and the sun’s reflection to spread across the periocular region.

Upon closer examination of the cases where our algorithm fails in Rank-5, we observed that most images share characteristics similar to those described in the previous paragraphs. Thus, srname provides better results for recognition accuracy but may be more sensitive to variations in pose and lighting.

5 Conclusions↩︎

In this work, we introduced srname, an algorithm that effectively combines multiple features through a neural network to produce a reliable and representative feature vector. This vector is then integrated with a reference low-resolution image in a diffusion model to generate high-quality super-resolution images. A key advantage of our algorithm is its independence from explicitly provided facial attributes; instead, it implicitly extracts high-level information through a visual encoder. This methodology enables our algorithm to preserve individuals’ identities more effectively than other methods, resulting in high-quality SR images with enhanced face symmetry, reduced noise and minimized distortion of facial attributes. We validated our approach on the CelebA and Quis-Campi datasets and achieved state-of-the-art results for visual quality and recognition metrics, demonstrating its potential for applications in real-world surveillance scenarios.

Declarations↩︎

References↩︎