1 Introduction↩︎

Super-Resolution (SR) aims to reconstruct high-quality images from low-resolution inputs. While deep learning has significantly improved SR performance, the ill-posed nature of SR often yields overly smooth outputs, especially at extreme downsampling rates (e.g., \(\times 16\)). To enhance perceptual quality, existing single-modal methods are typified by adversarial architectures[1] while the multi-modal ones generally leverage textual semantics as priors[2].

Recent breakthrough of pre-trained multi-modal large language models reveals the potential of textual semantics for image restoration[2]. However, they suffer from sensitivity to erroneous priors and substantial annotation overhead. Moreover, existing multi-modal methods struggle with insufficient attention to critical details due to semantic biases during text-image fusion, as demonstrated in Figure 1 (b).

To this end, we propose a novel multi-modal super-resolution framework termed Spatially Adaptive Modulation and Visual Prompt Adaptation (Simon-SR). Existing text-driven SR methods assume texts as ground-truth semantic priors, whereas Simon-SR treats texts as latent, learnable semantic variables jointly optimized with image restoration. As illustrated in Figure 1 (c), our method efficiently extracts textual features with minimal computational overhead while adaptively modulating image features.

In summary, our contributions are threefold:

(1) We propose a learnable prompt-based approach for SR that extracts textual semantics from unannotated images, effectively avoiding semantic biases from human annotations and pre-trained multi-modal large language models.

(2) We propose a spatially adaptive text-image fusion mechanism via attention-based affine transformations, enhancing both semantic relevance and textual utilization.

(3) Extensive experiments demonstrate state-of-the-art performance across multiple benchmarks and downsampling rates.

2 Related Works↩︎

2.1 From Single-Modal to Prior-Guided Multi-Modal↩︎

Early deep learning-based SR mainly relied on single-modal paradigms. Classic architectures [3], [4] primarily focused on pixel-level reconstruction, which often produces overly smooth outputs under severe degradation. To overcome this limitation, [5] introduced adversarial training to improve realism. However, it still suffered from structural distortion and artifacts due to the absence of external supervision. Subsequent multi-modal methods [6] incorporated auxiliary high-resolution images as structural priors, but were sensitive to spatial misalignment.

2.2 Text-Driven Super-Resolution↩︎

To ease the inherently ill-posed problem, early text-driven approaches made use of structured semantic cues such as segmentation maps[7], yet remained limited to specific categories. The development of vision-language models such as CLIP enabled text-guided SR by aligning visual and textual embeddings. For instance, [8] incorporated text descriptions to guide semantic-aware reconstruction. Recent diffusion-based methods  [9], while generating perceptually realistic textures, still depend on accurate text annotations and susceptible to semantic biases. These approaches suffer from inherent limitations: susceptibility to erroneous text priors and hallucination of implausible details.

3 Proposed Methods↩︎

3.1 Overview↩︎

As illustrated in Figure 2, our framework consists of two stages: Contrastive Prompt Learning (CPL) and Prompt-Guided Spatially Adaptive Refinement (PSAR). CPL initially uses CLIP to extract instance-level visual semantics from unannotated data \(\{(I_x^{\text{LR}},I_x^{\text{HR}})\}_{x=1}^{n_t}\). Distinct from recognition-oriented prompt tuning, the proposed prompts are optimized for robust cross-modal alignment, serving as intermediate semantic anchors instead of explicit supervision. Subsequently, PSAR adaptively fuses textual and visual features via Progressive Text-Aware Refinement Blocks (PTRBlock), where learned prompts are injected into affine transformations to enable fine-grained semantic-aware enhancement.

3.2 Contrastive Prompt Learning↩︎

Firstly, textual semantics are extracted adaptively via learnable prompts. To be specific, each instance is associated with \(\{[X]_i\}_{i=1}^M\). Both the input image \(x\) and textual description, formulated as “A photo of a \([X]_1[X]_2\dots[X]_M\)”, are fed into the frozen CLIP image encoder \(\mathcal{I(\cdot)}\) and text encoder \(\mathcal{T}(\cdot)\). Prompts are optimized by minimizing: \[\mathcal{L}_{con}=\sum_x\mathcal{L}_{i2t}(x)+\sum_x\mathcal{L}_{t2i}(x), \label{eq:stage195loss}\tag{1}\] where \(\mathcal{L}_{i2t}\) and \(\mathcal{L}_{t2i}\) denote the contrastive losses, defined as: \[\mathcal{L}_{i2t}=-\log \frac{\exp(s(V_x,T_x))}{\sum_{a=1}^B \exp(s(V_x,T_a))}, \mathcal{L}_{t2i}=-\log \frac{\exp(s(V_x,T_x))}{\sum_{a=1}^B \exp(s(V_a,T_x))}\] where \(\{V_x, T_x\}\) represent {visual, textual} features and \(s(\cdot,\cdot)\) the cosine similarity. Since all prompts are learned by the model itself, CPL avoids semantic biases from human annotations, thereby eliminating interference with model performance.

3.3 Prompt-Guided Spatially Adaptive Refinement↩︎

Preliminary Cross-Modal Alignment. Prompts and their corresponding inputs \(I^{\text{LR}}_x\) are encoded into \(\boldsymbol{f}_{txt}^{(l)}\in\mathbb{R}^{B\times d_{proj}}\) and \(\boldsymbol{f}_{img}^{(l)}\in\mathbb{R}^{B\times C\times H\times W}\), respectively. These features are fused via PTRBlock for initial textual-visual alignment, and then jointly input to frozen CLIP-ViT to establish a unified embedding space for refinement.

Progressive Text-Aware Refinement. Following the U-Net architecture, the refinement procedure is made up of PTRBlocks, as illustrated in Figure 2 (c). To obtain the modulated feature \(\hat{\boldsymbol{f}}_{img}^{(l)}\), a novel spatially adaptive affine transformation is proposed. Specifically, we project \(\boldsymbol{f}_{img}^{(l)}\) into query space through \(\mathbf{Q}=\text{Conv}_{1\times1}(\boldsymbol{f}_{img}^{(l)})\in\mathbb{R}^{B\times (C/r)\times H\times W}\), and \(\boldsymbol{f}_{txt}^{(l)}\) into key space through \(\mathbf{K}=\text{Linear}(\boldsymbol{f}_{txt}^{(l)})\in \mathbb{R}^{B\times C/r}\), where \(r\) is the reduction factor. After reshaping, where \(\mathbf{Q}=\mathbb{R}^{B\times HW \times C/r}\) and \(\mathbf{K}=\mathbb{R}^{B\times 1 \times C/r}\), the spatial attention is computed as: \[\mathcal{A}=\sigma(\frac{\mathbf{Q}\cdot\mathbf{K}^{\top}}{\sqrt{C/r}})\in \mathbb{R}^{B\times 1\times H\times W}\] where \(\sigma(\cdot)\) is the softmax function. \(\mathcal{A}\) encodes the spatial correlation between visual content and textual counterparts. We then define the affine parameters as \(\Delta \boldsymbol{\Gamma}=P_{\gamma}(\boldsymbol{f}_{img}^{(l)})\) and \(\Delta \boldsymbol{B}=P_{\beta}(\boldsymbol{f}_{txt}^{(l)})\), where \(P_{\gamma}(\cdot)\) and \(P_{\beta}(\cdot)\) denote two distinguished linear projections and \(\Delta \boldsymbol{\Gamma},\Delta \boldsymbol{B}\) are initialized with zero weights. Inspired by  [11], the final transformation is formulated as: \[\hat{\boldsymbol{f}}_{img}^{(l)}=(\boldsymbol{I}+\Delta {\boldsymbol{\Gamma}}\otimes \mathcal{A})\otimes \boldsymbol{f}_{img}^{(l)}\oplus \Delta \boldsymbol{B} \otimes \mathcal{A} \label{eq:spatial}\tag{2}\] where \(\oplus\) and \(\otimes\) denote element-wise addition and multiplication, and \(\boldsymbol{I}\) is the identity matrix. Since \(\Delta \boldsymbol{\Gamma},\Delta \boldsymbol{B}\) are zero-initialized and gated by cross-modal attention, regions with low relevance naturally converge toward \(\hat{\boldsymbol{f}}_{img}^{(l)}=\boldsymbol{f}_{img}^{(l)}\), and salient regions receive amplified or suppressed responses. Different from FiLM-style conditioning or cross-attention fusion, PSAR selectively enhances semantically relevant regions while leaving irrelevant areas unaffected, as illustrated in Figure 1 (c).

Adversarial Authenticity Discrimination. Following previous works [10], triple losses are adopted: reconstruction (sr) loss, perceptual loss, and adversarial loss, which are defined as: \[\mathcal{L}_{sr}=\mathbb{E}[||\mathbf{M}(I_x^{\text{LR}},\Theta)-I_{x}^{\text{HR}}||_1]\]

6pt

where \(\mathbf{M}(I_x^{\text{LR}},\Theta)\) stands for the output of the complete network, \(||\cdot||_1\) represents the pixel-wise \(\mathcal{L}_1\)-norm. \[\mathcal{L}_{percep}=\mathbb{E}\Big[\sum_i \mu_i||\alpha_i(\mathbf{M}(I_x^{\text{LR}},\Theta))-\alpha_i(I^{\text{HR}}_x)||_1\Big]\] where \(\mu_i\) is a hyper-parameter controlling the contribution of the \(i-\)th layer, \(\alpha_i(\cdot)\) denotes the feature map extracted from the \(i-\)th VGG-19 layer. \[\mathcal{L}_{adv}=-\mathbb{E}_{\hat{I}_X^{\text{SR}}\sim P_g}\big[\mathbf{D}\Big(\mathbf{C}(\hat{I}_x^{\text{SR}}),\boldsymbol{f}_{txt}^{(l)}\Big)\big]-\alpha \mathbb{E}_{\hat{I}_x^{\text{SR}}\sim P_g}\big[\text{Sim}(\mathbf{V}(\hat{I}_x^{\text{SR}},\boldsymbol{f}_{txt}^{(l)}))\big]\] where \(\mathbf{D}(\cdot,\cdot)\), \(\mathbf{C}(\cdot)\), \(\mathbf{V}(\cdot)\) respectively stand for the discriminator, frozen CLIP-ViT and image feature extractor within the discriminator, while \(\hat{I}_x^{\text{SR}}=\mathbf{M}(I_{x}^{\text{LR}},\Theta)\).

The total loss is proposed as: \[\mathcal{L} = \mathcal{L}_{sr} + \mathcal{L}_{percep} + \lambda \times \mathcal{L}_{adv}\] where \(\lambda\) is the hyper-parameter with \(\lambda=0.02\) by default.

4 Experiments↩︎

We evaluate our model on three datasets: DIV2K, CUB, and COCO2017. Since [2] requires text annotations, we generate captions for DIV2K images using BLIP-2 to ensure a fair comparison, while the proposed Simon-SR does not rely on any textual annotations, as prompts are learned directly from images. For evaluation, we adopt PSNR, SSIM, and LPIPS as metrics, where PSNR and SSIM measure structural fidelity, and LPIPS reflects perceptual realism. All experiments are conducted on two NVIDIA 4090 GPUs. For a fair comparison, we reproduce all baseline results using their official implementations, and generate LR images via bicubic downsampling with respective scaling factors while resizing all training and validation images to \(256\times 256\). The complete evaluation scripts will be released together with our codebase to facilitate reproducibility.

4.1 Quantitative Evaluation↩︎

As illustrated in Table 1, our model achieves new state-of-the-art across multiple datasets and downsampling rates. Compared to  [3], which focuses on structural fidelity, our method yields PSNR/SSIM improvements of up to 0.52/0.0353 (COCO2017, \(\times 8\)). Relative to CLIP-SR, which emphasizes perceptual realism, our method achieves up to a 0.0695 reduction in LPIPS (DIV2K, \(\times 16\)). Furthermore, relative to CLIP-SR, our model also shows substantial gains in structural fidelity, with maximum PSNR/SSIM increases of 0.50/0.0133 (CUB, \(\times 16\) & COCO2017, \(\times 16\)). These improvements are attributed to the proposed CPL and PSAR mechanism. Specifically, the learnable prompts mitigate erroneous textual priors, while PSAR dynamically enhances text-image modulation, as validated by ablation studies presented in Table 2.

4.2 Qualitative Evaluation↩︎

As shown in Figure 3, conventional methods often generate overly smooth images, while diffusion-based models are significantly influenced by erroneous textual priors, resulting in degraded reconstructions. In contrast, our model attains a balanced compromise between smoothness and perceptual realism, effectively avoiding unnecessary artifacts while recovering abundant details across different downsampling rates.

5 Conclusion↩︎

In this study, we propose Simon-SR, a novel multi-modal super-resolution framework where learnable prompts are adopted for efficient semantic mining and robust text-guided refinement. The key idea is to combine contrastive prompt learning with spatially adaptive affine transformations in PTRBlocks to enhance multi-modal perception and fusion. This work demonstrates the effectiveness of learnable prompts in multi-modal SR without additional annotation cost and computational overhead.

References↩︎