1 Introduction↩︎

Blind super-restoration (BSR) aims to convert low-resolution (LR) images that have undergone complex and unknown degradation into clear high-resolution (HR) versions. Differing from classical super-resolution [1]–[3], where the degradation process is singular and known, BSR is designed to super-resolve real-world degraded images which have more practical value. In real-world scenarios, achieving both effectiveness and efficiency poses a key challenge for BSR.

Generative models, generative adversarial network (GAN) and diffusion model, have demonstrated significant superiority in BSR task due to their ability to generate realistic details. However, they both have disadvantages. GAN-based methods [4]–[9] incorporate adversarial training to learn a network that fits the mapping function from the distribution of input LR images to that of HR images. While GAN-based methods require only one-step inference, they often struggle to generate satisfactory results when handling natural images with intricate textures (e.g., Fig. 1).

Recently, diffusion models (DM) have garnered significant attention owing to their potent generative capabilities and the ability to combine information from multiple modalities. DM-based BSR can be roughly divided into two categories: without Stable Diffusion (SD) prior [10]–[12] and with SD prior [13]–[16]. SD prior can significantly assist the model in capturing the distribution of natural images, enabling the generation of HR images with realistic details. Due to the iterative refinement characteristic of DM, diffusion-based methods can generally achieve better results than GAN-based methods, albeit with a noticeable drop in efficiency as shown in left of Fig. 1.

As analyzed above, designing a BSR model with both excellent restoration effect and high efficiency is urgently needed for real-world applications. To fill this blank, we propose a novel \(\mathtt{AddSR}\) based on adversarial diffusion distillation (ADD), which enhances restoration effects and accelerates inference speed simultaneously. However, incorporating ADD directly into BSR poses two crucial challenges, which are both addressed in our proposed \(\mathtt{AddSR}\):

(\(1\)) Task inconsistency: There are two differences between ADD and \(\mathtt{AddSR}\). a) Original ADD is proposed for efficient text-to-image generation, while \(\mathtt{AddSR}\) deals with image-to-image restoration. b) Original ADD does not require LR images as input, while \(\mathtt{AddSR}\) incorporates ControlNet to receive LR and HR images in the student and teacher models respectively to support BSR.

Specifically, we incorporate the ideas of distillation and ControlNet. On one hand, we design a simple yet effective method called prediction-based self-refinement (PSR) for the student model. Unlike previous SD-based methods (e.g., SeeSR) that receive the fixed LR images as input for BSR, we estimate HR images from the predicted noise to provide high-frequency information during training. On the other hand, with incorporation of the teacher model, we can directly send the HR image as its input to additionally supply high-frequency prior during training, which is not available in previous SD-based frameworks. This strategy can help the student model produce high perception quality results closely resembling those of the teacher model.

(\(2\)) Perception-distortion imbalance: The perceptual quality of images generated by the diffusion model increases with the number of steps. Constraining generated images in different timesteps with the same GAN loss weight in ADD may lead to insufficient perceptual constraint with small step (e.g., \(1\)) to generate blurry results, and overly restrictive constraint with larger step (e.g., \(4\)) to produce hallucinated results. To tackle this challenge, we develop a timestep-adapting (TA) loss to alleviate this issue. The core idea of TA loss involves employing varying weights for the GAN loss and distillation loss at each timestep, enhancing the constraint of the adversarial loss on the model in the early stages and reducing it in the later stages.

Our main contributions can be summarized as threefold:

\(\bullet\) To our best knowledge, \(\mathtt{AddSR}\) is the first SD-based BSR model that achieves high perception quality within \(0.8\)s, achieving \(\times\) \(7\) faster than the previous state-of-the-art model SeeSR, along with better restoration performance.

\(\bullet\) We propose prediction-based self-refinement to provide high quality controlling signal for regulating the model output and refine the teacher model training process. Moreover, we propose the timestep-adapting loss for achieving the perception-distortion trade-off.

\(\bullet\) Experiments on popular benchmarks against extensive competitors demonstrate that our proposed \(\mathtt{AddSR}\) achieves superior restoration effect and high efficiency simultaneously.

2 Related Work↩︎

GAN-based BSR. In recent years, BSR have drawn much attention due to their practicability. Adversarial training [17] is introduced in BSR task to avoid generating over-smooth results. BSRGAN [4] designs a random shuffle strategy to enlarge the degradation space for training a comprehensive SR model. Real-ESRGAN [5] presents a more practical degradation process called “high-order” to synthesize realistic LR images. Both of them utilize the GAN to learn the projection from the distribution of degraded LR images to the distribution of HR images. KDSRGAN [6] estimates the implicit degradation representation to assist the restoration process. While GAN-based BSR methods require only one step to restore the LR image, their capability to super-resolve complex natural images is limited. In this work, our \(\mathtt{AddSR}\) seamlessly attains superior restoration performance with a slight dip in speed, making it a compelling choice. Diffusion-based BSR. Diffusion models has demonstrated significant advantages in image generation tasks (e.g., text-to-image). Recently, some diffusion-based BSR works have been proposed. One common approach [10], [18], [19] is training a non-multimodal diffusion model from scratch, which takes the concatenation of a LR image and noise as input in every step. Another approach [13]–[16] fully leverages the prior knowledge from a pre-trained multimodal diffusion model (i.e., SD model), which requires training a ControlNet and incorporates new adaptive structures (e.g., cross-attention). SD-based methods excel in performance compared to the aforementioned approaches, as they effectively incorporate high-level information. However, the large number of model parameters and the need for numerous sampling steps pose substantial challenges to their application in the real world.

Efficient Diffusion Models. Several works [20]–[23] are proposed to accelerate the inference process of DM. Although these methods can reduce the sampling steps from thousands to \(20\)-\(50\), the restoration effect will deteriorate dramatically. A very recent technique called adversarial diffusion distillation [24] are proposed to achieve \(1\) \(\sim\) \(4\) steps inference while maintaining satisfactory generating ability. However, ADD was originally designed for the text-to-image task. Considering the multifaceted nature of BSR, such as image quality, degradation, or the trade-off between fidelity and realness, employing ADD to expedite the SD-based model for BSR is non-trivial. In summary, \(\mathtt{AddSR}\) introduces two pivotal designs to adapt ADD into BSR tasks, making it both effective and efficient.

3 Methodology↩︎

Here, we outline \(\mathtt{AddSR}\). First, we provide a brief introduction to stable diffusion and adversarial diffusion distillation in Sec. 3.1. Next, in Sec. 3.2, we introduce the Image Quality-Adjusted ADD, an acceleration strategy refined from ADD to suit the BSR task. Then, we describe and discuss our proposed prediction-based self-refinement and timesteps-adapting loss in Sec. 3.3 and Sec. 3.4, respectively.

3.1 sec:Preliminary↩︎

Stable Diffusion. The goal of \(\mathtt{AddSR}\) is to design an effective and efficient model with Stable Diffusion prior for BSR task. Stable Diffusion is a large-scale text-to-image latent diffusion model. Its forward and denoising process are operated in latent space through an encoder \(\mathcal{E}\) [25] that translates the pixel space \(x\) into latent space \(z\) . The forward process is a Markovian chain that adds Gaussian noise to the latent tensor \(z\) at time \(t\) to generate the noisy state: \[z_t = \sqrt{\overline{\alpha}_t}z+\sqrt{1-\overline{\alpha}_t}\epsilon, ~~\overline{\alpha}_t=\prod \limits_{i=1}^t (1-\beta_i) \label{eq1},\tag{1}\] where \(z=\mathcal{E}(x)\), \(x\) is the sampled data, \(\beta_i \in (0,1)\) is the variance of the Gaussian noise and \(\epsilon\sim(0, \boldsymbol{I})\). Then, a U-Net \(\theta\) is employed to predict the added noise with text prompt \(\tau\). The loss function is specified as follows: \[\mathcal{L}_{SD} = \mathbb{E}_{z,t,\tau,\epsilon}[\Vert \epsilon-\epsilon_\theta(z_t, t, \tau) \Vert_2^2],\] where \(t\) is uniformly sampled. Once the model is trained, we can input descriptions of desired scenes and obtain the corresponding images.

Adversarial Diffusion Distillation. The denoising process is crucial to the diffusion model, while adversarial training plays a central role in GANs. Recently, adversarial diffusion dstillation is designed to combine the above two approaches for accelerating the denoising process. There are two models in ADD, including a ADD-student and a ADD-teacher. Specifically, ADD first chooses a student timestep \(s\) from a set \(T_{student}=\lbrace t_1, t_2, t_3, t_4\rbrace\), adds noise to convert \(x\) to \(x_s\) and send it into ADD-student. Second, ADD-student generates \(\hat{x}_\theta(x_s, s)\), diffuses the result to noisy state \(x_t\) and takes the ADD-teacher’s output \(\hat{x}_\phi(x_t, t)\) as the distillation target. Finally, ADD passes the student result \(\hat{x}_\theta(x_s, s)\) and ground truth \(x_0\) to a discriminator to employ adversarial loss as follows: \[\mathcal{L}_{ADD} = \mathcal{L}_{dis}(\hat{x}_\theta(x_s, s), \hat{x}_\phi(x_t, t)) + \lambda\mathcal{L}_{adv}(\hat{x}_\theta(x_s, s), x_0, \psi), \label{eq3}\tag{2}\] where \(\theta\) is the weight set of student model which is initialized from teacher model, \(\phi\) is the weight set of teacher model, \(\psi\) is the weight set of discriminator and \(\lambda\) is the balance weight. We empirically set \(\lambda\) to \(0.02\).

3.2 Image Quality-Adjusted ADD↩︎

ADD is proposed to expedite the inference process of DM for the text-to-image task, without consideration of receiving low-quality images. Conversely, image quality is an inevitable factor in the BSR task. To design an acceleration strategy for the BSR task, we propose image quality-adjusted ADD, as illustrated in Fig. 2. Specifically, if we do not consider the image quality and directly employ ADD for the BSR task, which means using the same input (i.e., LR images) to control the student and teacher models, we fail to fully utilize the high-frequency information from HR images available in the training phase. In Fig. 2, we substitute LR images with their HR counterparts (i.e., the input of RAM, text encoder and ControlNet) to regulate the output of the teacher model. This compels the student model, controlled with LR images, to produce similar results as the teacher model controlled with HR images.

3.3 Prediction-based Self-Refinement↩︎

As shown in Fig. 3, SD-based methods usually use LR images to control the output of DM in each inference step. However, LR images may suffer from multiple degradations, which can disrupt the restoration process and limit their ability to provide only low-frequency information for controlling the model output (e.g., the first line of Fig. 4). Previous methods [14], [16] employ a degradation removal structure to preprocess the LR image, aiming to mitigate the impact of degradation. However, these approaches often sacrifice efficiency, which is unacceptable when designing an efficient model.

To address this issue, we propose a novel prediction-based self-refinement approach, which incurs only tiny efficiency overhead. The core idea of PSR is to fully utilize the predicted noise to enhance image restoration. Specifically, we use the following equation

\[x_0 = (x_t-\sqrt{1-\overline{\alpha}_t}\epsilon_{\theta,t}) / \sqrt{\overline{\alpha}_t} \label{eq4}\tag{3}\]

to estimate the HR image \(\hat{x}_0\) from predicted noise in each step, and then control the model output in next step, where \(x_t\) is the noisy state and \(\epsilon_{\theta,t}\) is the predicted noise at time \(t\). The \(\hat{x}_0\) in each step has more high-frequency information to better control the model output (e.g., the right side of Fig. 3 and also Fig. 4). Moreover, the additional processing time, solely attributed to solving Eq. 3 , is nearly negligible. By leveraging our simple yet effective PSR, \(\mathtt{AddSR}\) can notably enhance the restoration outcomes without compromising efficiency (see Tab. 5).

3.4 Timestep-Adapting Loss Mechanism↩︎

Motivation. Perception-distortion trade-off [26] is a well-known phenomenon in SR task. Ideally, a good model can generate image with both high perception and fidelity. However, with the increasing realness of restored image (e.g., by elevating the inference step in diffusion model), the fidelity may deteriorate. We observed that training BSR task with ADD directly exacerbates this phenomenon, as shown in Fig. 5. Specifically, during the initial stages of inference, there is a significant decrease in fidelity, accompanied by improvement in perception quality. In later stages of inference, fidelity remains at a low level, while perception quality undergoes a dramatic increase. The aforementioned scenario may give rise to two issues: (1) When the inference step is small, the quality of restored image is subpar. (2) As the inference step increases, the generated images may exhibit “hallucinations”.

The main reason could be attributed to ADD, which employs the same weight for GAN loss and distillation loss across different student timesteps. Since the perception quality of diffusion model-generated images typically increase when elevating the student timeteps, the introduction of weight-invariant ADD may result in insufficient adversarial constraints on the student model during the initial stages of inference, resulting in the inability to generate perceptually high-quality images. However, during later inference stage, the constraints on model from adversarial training may be too strong to produce “hallucinations”.

Timestep-adapting Loss. To address this issue, we dynamically adjust the weight of distillation loss across different student timesteps. In particular, the weight of the distillation loss decreases as the number of timesteps decreases. The loss function can be reformulated as follows:

\[\mathcal{L}_{ADD} = \gamma\mathcal{L}_{dis}(\hat{x}_\theta(x_s, s), \hat{x}_\phi(x_t, t)) + \lambda\mathcal{L}_{adv}(\hat{x}_\theta(x_s, s), x_0, \psi) \label{dqsbvcyj}\tag{4}\]

where \(\gamma\) is a weight that is negatively correlated with timestep. Extensive experiments demonstrate the proposed loss is simple yet effective (see Sec. 4.5).

4 Experiments↩︎

4.1 Experimental Settings↩︎

Training Datasets. We adopt DIV2K [27], Flickr2K [28], first \(20\)K images from LSDIR [29] and first \(10\)K face images from FFHQ [30] for training. We use the same degradation model as Real-ESRGAN [5] to synthesize HR-LR pairs.

Test Datasets. We evaluate \(\mathtt{AddSR}\) on \(4\) datasets: DIV2K-val [27], DRealSR [31], RealSR [32] and RealLR200 [15]. It is worth noting that we conduct \(4\) degradation types on DIV2K-val to comprehensively assess \(\mathtt{AddSR}\), and except RealLR200, all datasets are cropped to \(512\) \(\times\) \(512\) and degraded to \(128\) \(\times\) \(128\) LR image.

Implementation Details. We adopt SeeSR [15] as the teacher model. Note that our approach is applicable to most of the existing SD-based BSR methods for improving restoration results and acceleration. The student model is initialized from the teacher model, and fine-tuned with Adam optimizer for \(50\)K iterations. The batch size and learning rate are set to \(6\) and \(2\) \(\times10^{-5}\), respectively. \(\mathtt{AddSR}\) is trained under \(512\) \(\times\) \(512\) resolution images with \(3\) NVIDIA A\(100\) GPUs (\(40\)G).

Evaluation Metrics. We employ non-reference metrics (e.g., MANIQA [33], MUSIQ [34], CLIPIQA [35]) and reference metrics (e.g., LPIPS [36], PSNR, SSIM [37]) to comprehensively evaluate our \(\mathtt{AddSR}\). We consider non-reference metrics as the primary metrics since they are closer to human perception. Compared Methods. We demonstrate the superiority of \(\mathtt{AddSR}\) against several state-of-the-art BSR methods, including GAN-based methods: BSRGAN [4], Real-ESRGAN [5], MM-RealSR [38], LDL [7], FeMaSR [8] and diffusion-based methods: StableSR [16], ResShift [10], PASD [13], DiffBIR [14], SeeSR [15].

4.2 Evaluation on Synthetic Data↩︎

To demonstrate the superiority of the proposed \(\mathtt{AddSR}\) in handling various degradation cases, we synthesized \(4\) test datasets using the DIV2K-val dataset with different degradation processes. The quantitative results are summarized in Tab. 2. The conclusions include: (\(1\)) Our \(\mathtt{AddSR}\)-\(4\) achieves the highest scores in MANIQA, MUSIQ and CLIPIQA across \(4\) degradation cases, except for a 0.3\(\%\) and 2\(\%\) lower score in MUSIQ and CLIPIQA under noise degradation, recpectively. Especially for MANIQA, \(\mathtt{AddSR}\) surpasses the second-best method by more than 16\(\%\) on average. (\(2\)) The highest scores in PSNR, SSIM and LPIPS are usually generated by GAN-based method. The reason may be that due to the powerful generative abilities of diffusion-based models, some realistic details are generated but not exist in the ground truth, thereby leading to lower scores in full-reference metrics. (\(3\)) \(\mathtt{AddSR}\)-\(1\) can generate comparative results against other SD-based methods except SeeSR, but significantly reduces the sampling steps (i.e., from \(\geq\) \(15\) steps to only \(1\) step).

For a more intuitive comparison, we provide visual results in Fig. 6. One can see that GAN-based method like FeMaSR fails to reconstruct the clean and detailed HR images of the three displayed LR images. As for SD-based method DiffBIR, it tends to generate blurry results. This is mainly because DiffBIR uses a degradation removal structure to remove the degradation of LR images. However, the processed LR image is blurry, which may lead to the blurry results. Thanks to our proposed PSR, \(\mathtt{AddSR}\) uses the predicted \(\hat{x}_0\) to control the model output, which has more high-frequency information and nearly no extra time cost. With TA loss, \(\mathtt{AddSR}\) can generate precise images and rich details. In a nutshell, \(\mathtt{AddSR}\) can produce images with better perceptual quality than the state-of-the-art models while requiring fewer inference steps and less time.

4.3 Evaluation on Real-World Data↩︎

Tab. 3 shows the quantitative results on \(3\) real-world datasets. We can see that our \(\mathtt{AddSR}\) achieves the best scores in MANIQA, MUSIQ and CLIPIQA, the same as in the synthetic degradation cases. This demonstrates that \(\mathtt{AddSR}\) has an excellent generalization ability to handle unknown complex degradations, making it practical in real-world scenarios. Additionally, \(\mathtt{AddSR}\)-\(1\) surpasses the previous GAN-based methods, primarily due to the integration of IQ-ADD that combines a diffusion model with adversarial training. This integration enables \(\mathtt{AddSR}\) to leverage high-level information to enhance the restoration process and generate high perception quality images, even through a one-step inference.

Fig. 7 shows the visualization results. We consider \(3\) kinds of images including building, car and face to comprehensively compare various methods. A noticeable observation is that \(\mathtt{AddSR}\) generate more clear and regular line, as evidenced by the linear pattern of the building in the first example and the intake grille of a car in the second example. In the third example, the original LR image is heavily degraded, FeMaSR and ResShift fail to generate the human face, showing only the blurry outline of the face. DiffBIR can generate more details, yet still unclear. The image generated by SeeSR exhibits artifacts, which may be caused by the LR images degradation. Conversely, our \(\mathtt{AddSR}\) can generate comparative results with FeMaSR and ResShift in one-step. As evaluating the inference steps, \(\mathtt{AddSR}\) generates clearer and more detailed human face, which significantly surpasses the aforementioned methods.

4.4 Prompt-Guided Restoration↩︎

One of the advantages of diffusion model is its ability to integrate with text. In Fig. 8, we demonstrate that our \(\mathtt{AddSR}\) can efficiently achieve more precise restoration results in \(4\) steps by incorporating with manual prompt, i.e., we can manually input the text description of the LR image to assist the restoration process. Specifically, Fig. 8 (a) shows that the plaid shirt in the restored image with the RAM prompt can be edited to camouflage through the manual prompt. In Fig. 8 (b), the Spider-Man restored with the original RAM prompt has a mouth and beard, whereas with the manual prompt, the man is accurately restored wearing a spiderweb suit. In Fig. 8 (c), the word on the chip can be corrected from ‘2ALC515’ to ‘24LC515’ with the manual prompt. In Fig. 8 (d), the mushroom’s background should appear blurry, while the RAM prompt renders the tree branches sharply. Conversely, the manual prompt maintains the background’s intended blur, aligning with the Ground Truth.

4.5 Ablation Study↩︎

Effectiveness of Refined Training Process. To make teacher model provide more information for student model to learn, we refine the training process to replace the LR image of ControlNet, RAM and text encoder input with HR image. Since SeeSR is adopted as the baseline, we also replace the LR image of RAM input with HR image. The quantitative results are shown in Tab. 4. Moreover, the visual comparisons are shown in Fig. 9 (a). We can see that with the supervision from HR input, the restored image of student model become clearer with more details. The quantitative results also demonstrate this conclusion due to the perception quality becomes better.

Effectiveness of PSR. As shown in Tab. 5, PSR can improve the restored LR image’s perception quality notably at only \(0.03\)s slower than the original LR controlling strategy. Fidelity of restored images deteriorates after using PSR, since PSR uses the former step’s output that has more high-frequency information to control next step’s result, and may generate some extra details, as discussed in Sec. 4.2. The visual comparisons are shown in Fig. 9, we observe that the output of the first step, whether or not PSR is employed, remains nearly the same, as both are generated by controlling the LR image. However, when the step reaches \(4\), the model with PSR generates image showing high perception quality.

Importance of TA Loss. The TA loss aims to increase the fidelity of restored images while maintaining the perception quality. The quantitative results are shown in Tab. 6. As can be seen, although we enlarge the weight of distillation loss in the last two steps, the perception quality still becomes better. The reason may be that the first two steps generate well enough perception quality images, which can provide more information by combining PSR, thus the last two steps can generate high perception quality while achieving good fidelity.

The visual results are shown in Fig. 10. For the left six images, the original background is rock. However, without utilizing TA loss, \(\mathtt{AddSR}\)might hallucinate the background as another blurred wolf. Consequently, in the final step, the background transforms into the eye of a wolf. For the right six images, the image content is a statue. Yet, in the first row, the model hallucinates its hand as a bird. Such hallucination compromises the fidelity of the restored images, as illustrated in Tab. 6. Conversely, with the help of TA loss, the restored images can generate more consistent contents with GTs. As depicted in the second row of Fig. 10, TA loss constrains the model from excessively leveraging its generative capabilities, thereby preserving more information in the image content, aligning closely with the GTs. Specifically, using TA loss, background of the left image retains the rock with out-of-focus appearance, and texture of the statue’s hand in the right image remains unchanged.

5 Conclusion↩︎

We propose \(\mathtt{AddSR}\), an effective and efficient model based on Stable Diffusion prior for blind super-resolution. In contrast to current SD-based BSR approaches that employ LR images to regulate each inference step’s output, \(\mathtt{AddSR}\) substitutes the LR image with the HR image calculated in the preceding step to control the model output and mitigate the degradation impact of LR images. To further augment the model’s restoration capacity, we introduce Timestep-adaptive loss, which assigns distinct weights to GAN loss and distillation loss across different steps, addressing the perception-distortion imbalance issue due to the introduction of original ADD. Additionally, to tailor ADD for the BSR task, we propose an image quality-adjusted strategy, enabling the teacher model to provide better supervision to the student model. Extensive experiments demonstrate that our proposed \(\mathtt{AddSR}\) can generate superior results within \(1\) \(\sim\) \(4\) steps in various degradation scenarios and real-world low-quality images.

5.0.0.1 Limitations.

Our \(\mathtt{AddSR}\) designs an acceleration strategy incorporated with PSR and TA loss for BSR task to generate high perception quality results. Although the inference speed surpasses all of the existing SD-based methods remarkably, there still exists a gap between GAN-based methods and \(\mathtt{AddSR}\). The primary factor is that \(\mathtt{AddSR}\) is built upon Stable Diffusion and ControlNet, which, due to its substantial model parameters and intricate network structure, noticeably hinders the inference time. In the future, we are exploring a more streamlined network architecture to enhance overall efficiency.

6 Comparisons among ADD, SeeSR and \(\mathtt{AddSR}\)↩︎

In this section, we provide a comparison among ADD, SeeSR, and our proposed \(\mathtt{AddSR}\). Their architecture diagrams are depicted in Fig. 11. Firstly, the distinctions between ADD and \(\mathtt{AddSR}\) primarily lie in two aspects: \(1\)) Introduction of ControlNet: ADD is originally developed for text-to-image task, which typically only takes text as input. In contrast, \(\mathtt{AddSR}\) is an image-to-image model that requires the additional ControlNet to receive information from the LR image. \(2\)) Perception-distortion Trade-off: ADD aims to generate photo-realistic images from texts. However, introducing ADD into blind SR brings the perception-distortion imbalance issue (please refer to Sec. \(3.4\) in our submission), which is addressed by our proposed timestep-adapting loss in \(\mathtt{AddSR}\).

Secondly, the key differences between SeeSR and \(\mathtt{AddSR}\) are: \(1\))Introduction of Distillation: SeeSR is trained based on vanilla SD model that needs \(50\) inference steps, while \(\mathtt{AddSR}\) utilizes a teacher model to distill an efficient student model to achieve just \(1\) \(\sim\) \(4\) steps. \(2\)) High-frequency Information: SeeSR uses the LR image \(y\) as the input of the ControlNet. In contrast, \(\mathtt{AddSR}\) on one hand adopts the HR image \(x_0\) as the input of the teacher model’s ControlNet to supply the high-frequency signals since the teacher model is not required during inference. On the other hand, \(\mathtt{AddSR}\) proposes a novel prediction-based self-refinement (PSR) to further provide high-frequency information by replacing the LR image with the predicted image as the input of the student model’s ControlNet. Therefore, \(\mathtt{AddSR}\) has the ability to generate results with more realistic details.

7 Effectiveness of Prediction-based Self-Refinement↩︎

Our PSR is proposed to remove the impact of LR degradation and enhance high-frequency signals to regulate the student model output. As shown in Fig. 12, the restored images generated with PSR exhibit more details and sharper edges, while the images generated without PSR tend to be blurry with fewer details.

8 Comparison with SeeSR-Turbo↩︎

A very recent SD-based method named SeeSR-Turbo² has been introduced for blind super-resolution through 2-steps inference. To demonstrate the superiority of our \(\mathtt{AddSR}\), we conduct a comparison between SeeSR-Turbo and \(\mathtt{AddSR}\). The quantitative and qualitative results are shown in Tab. 7 and Fig. 13, respectively. One can see that our \(\mathtt{AddSR}\) surpasses SeeSR-Turbo with significant margins on NIQE, MANIQA, MUSIQ and CLIPIQA. In addition, \(\mathtt{AddSR}\) can generate realistic results by \(2\) steps, while SeeSR-Turbo tends to generate blurry results.

9 More Examples↩︎

We provide more visual results in Fig. 14. These examples demonstrate \(\mathtt{AddSR}\) can produce more photo-realistic restored images within \(4\) steps, compared with the SOTA competitors.

References↩︎

---

1. Corresponding author. Work done when Rui Xie was an intern at Nanjing University.↩︎

2. SeeSR-Turbo is open-sourced on March 10th, 2024.↩︎