1 Introduction↩︎

Despite advancements in camera technology, the quality of images may still fall short of human expectations, often requiring zooming to reveal finer details. To address this, early super-resolution (SISR or SR) methods [1]–[6] have focused on recovering high-resolution (HR) images from degraded low-resolution (LR) counterparts, where LR images were generated by degrading HR images [7]–[11].

Specifically, early LR images [12] are constructed by downsampling (e.g., bicubic and bilinear) HR images, failing to capture real-world degradations. To overcome this, later methods have explored two main strategies for constructing LR images that more accurately reflect real-world degradations. One method [7], [8] synthetically simulates image signal processor (ISP) pipelines, incorporating various degradation processes (e.g., noise, blur, and compression). However, they often fail to capture the full complexity of real-world degradations. The other methods utilize specialized camera systems with beam splitters [9], [10] or multiple focal lengths [11] to capture realistic LR images. However, they require sophisticated hardware and still rely on a specific camera model, restricting their ability to generalize to complex real-world degradations.

On this basis, we propose a novel SR framework, the Language Assistant for SR (LA-SR), to address this. Inspired by DGDML-SR [13], we observe that even high-quality images often contain both real-LR and HR regions due to varying subject-to-camera distances, as shown in Figure 1 (e.g., distant grass and close tiger). This depth variation causes different regions within the same image to exhibit distinct levels of detail, effectively serving as unpaired LR-HR patches. However, the absence of explicit LR-HR pairs makes direct supervised training difficult. Although DGDML-SR [13] addresses this issue by identifying similar content patterns within the same image to construct LR-HR pairs, this assumption rarely holds in real-world scenarios, limiting the applicability of their approach. Instead, to address this limitation, we leverage powerful vision-language models [14]–[17], which are capable of understanding both the semantic content and perceptual quality of images, enabling effective supervision without requiring explicit LR-HR pairings.

To elaborate, we utilize the correlation between language and image in our LA-SR framework, as shown in Figure 1. Here, considering content and quality as crucial factors in image restoration [18], [19], we present two loss terms: linguistic-content loss to preserve the content of the input images in the SR images and linguistic-quality loss to ensure the high quality of the SR images. Specifically, using a depth map obtained from a pre-trained model [20], we extracted LR-HR patches from high-quality images: large HR patches (e.g., blue patch in Figure 1) from near depths and small LR patches (e.g., yellow patch in Figure 1) from distant depths. Then, we prepare two types of textual descriptions: content texts, derived from each patch using a pre-trained image-to-text model [15]–[17], and pre-defined quality texts, categorizing patches as either high or low quality. Assuming distant LR patches contain fewer details than closer ones, we use contrastive learning to encode LR patches with strong correlations to their respective content and low-quality texts, while maintaining weak correlations with other texts (and vice versa for HR patches) (e.g., \(\text{\ding{110}}\) and \(\text{\ding{110}}\) in Figure 1). Finally, we train our LA-SR framework to produce SR images (e.g., red patch in Figure 1) that align closely with their corresponding content and high-quality texts (e.g., \(\text{\ding{54}}\) in Figure 1), thereby maintaining the content of input patches and preserving the details of HR patches.

LA-SR effectively produces realistic results from diverse real-world inputs by training with real LR patches and incorporating our language-based loss terms: linguistic-content and linguistic-quality losses. Instead of relying on previously used synthetically degraded LR benchmarks [7], [8], [12], we show the efficacy of LA-SR on natural images, including various benchmark datasets. We summarize our contributions as follows:

• Unpaired Real-World SR Framework: We propose LA-SR, a novel framework that learns from real LR patches extracted from high-quality images using depth information, eliminating the need for synthetic degradations.

• Language-Space Alignment with Guided Losses: We formulate unpaired SR as a language-space alignment problem using a pretrained vision-language model, and design content and quality losses to keep semantics and enhance realism, enabling high-quality SR.

• Superior SR Performance: Our LA-SR delivers superior SR performance across various benchmarks, excelling in both perceptual metrics and visual quality, demonstrating strong generalization to real-world LR inputs.

2 Related works↩︎

Real-world Super-Resolution. Since the early breakthrough of deep learning-based SR methods such as VDSR [21], EDSR [1], and SRGAN [2], many subsequent SR methods [4], [5], [22] have supervised their networks by minimizing distance-based losses (e.g., pixel-wise and VGG) between super-resolved and high-resolution images. Although they achieve strong performance, they often struggle with real-world images due to reliance on LR images [12], [23], [24] generated through simple downsampling that fail to replicate real degradations. A straightforward solution is to create a dataset that better reflects real LR images. To do this, several methods [7], [8] apply degradation-based augmentations (e.g., noise, resizing, and JPEG compression) but fail to capture real-world complexities. Others use specialized camera setups (e.g., beam splitters [9], [10] or multiple focal lengths [11]) to capture LR images with actual ISP degradations. However, they require advanced equipment and produce LR images that are specific to particular camera models, limiting generalizability.

Self-supervised Super-Resolution. Without requiring LR-HR pairs, several zero-shot methods [25], [26] have proposed training strategies using only LR images. ZSSR [25] trains an image-specific CNN on a given LR image, while MZSR [26] uses an initialization from a large-scale external dataset [12] to tailor the model to given LR images. However, they assume that LR images are degraded with a known degradation, which is often impractical. To tackle images with unknown degradations, subsequent methods use auxiliary reference images, either captured via dual-camera setups [27], [28] or retrieved using patch similarity [29], and train SR networks in a self-supervised manner. However, obtaining reference images is challenging, and they still rely heavily on manually degraded LR images, which limits their applicability to real-world images. DGDML-SR [13] takes a different approach by using depth-based LR-HR patch and employing a cyclic GAN [30]. However, due to the instability of the adversarial training, their method restricts the LR-HR patch pairs to regions with highly similar content (e.g., identically shaped windows), which is an unrealistic constraint in most real-world scenarios.

Language in image restoration. Although early image restoration methods focused solely on image data, recent approaches recognize the strong correlation between image and language and use language to enhance restoration. Specifically, CoSeR [31] utilizes cognitive embeddings with semantic and textural information for super-resolution. LLMRA [32] leverages language priors based on user dialogue to restore various degraded images. Similarly, LM4LV [33] uses a frozen large-language model (LLM) to generate visual tokens, which are then decoded into restored images. These advances highlight the potential for further integration of language assistance into image restoration.

Contrastive Language-Image training. Image-based contrastive learning aims to learn discriminative representations to differentiate an image from others, proving effective across various tasks. Starting from simple classifications [34] that map images to corresponding labels (e.g., cat and dog), later works [15], [35], [36] have adopted contrastive learning to capture the correlation between images and text descriptions. Specifically, they jointly train image and text encoders to predict correlated pairs from a batch of (image, text) data. They combine pairs of images and corresponding text descriptions in a shared embedding space while pushing apart pairs that do not match. In other words, instead of learning a one-to-one mapping between image and text, these approaches determine how multiple images are relatively correlated to each text.

3 Proposed Method↩︎

Figure 2 shows the overall pipeline of our LA-SR. We utilize an existing SR network [37] without modifications for our SR network, while introducing two novel loss terms: linguistic-content and linguistic-quality losses.

3.1 Preparing LR-HR patches↩︎

We begin by extracting LR patch \(\mathbf{I}_{\text{LR}}\) and HR patch \(\mathbf{I}_{\text{HR}}\) from high-quality images. To achieve this, we first use a pre-trained depth estimation network [20] to determine the relative distance of each pixel from the camera, selecting distant patches as \(\mathbf{I}_{\text{LR}}\) and closer patches as \(\mathbf{I}_{\text{HR}}\), as shown in Figure 2 (a). Here, unlike the previous depth-based approach [13], which constrains LR and HR patches to share similar textures (i.e., patterns), our method imposes no such constraint, allowing LR-HR patches to be extracted from any image regardless of content similarity. To capture the depth of pixels within each patch, we apply a convolution with a large fixed kernel (i.e., all values set to 1 and of size \(\mathbb{R}^{1 \times 3 \times 29 \times 29}\)), enabling depth estimation based on surrounding pixels, resulting in the final depth map \(\mathbf{D}\). Then, using the given image and estimated depth map \(\mathbf{D}\), we extract two types of patches: small patches from distant regions \(\mathbf{I}_{\text{LR}} \in \mathbb{R}^{h \times w \times 3}\) and large patches from closer regions \(\mathbf{I}_{\text{HR}} \in \mathbb{R}^{H \times W \times 3}\). Here, \((h, w)\) and \((H, W)=(sh, sw)\) are the (height, width) of each patch with \(s>1\) as a fixed scaling factor. Specifically, we select \(\mathbf{I}_{\text{LR}}\) and \(\mathbf{I}_{\text{HR}}\) from the depth map \(\mathbf{D}\) using a top-k algorithm, where the k farthest pixels (i.e., bright regions in Figure 2 (a)) are assigned as centers of \(\mathbf{I}_{\text{LR}}\) and the k nearest pixels (i.e., dark regions in Figure 2 (a)) are selected as centers of \(\mathbf{I}_{\text{HR}}\). During training, we super-resolve \(\mathbf{I}_{\text{LR}}\) into a higher-resolution SR image \(\mathbf{I}_{\text{SR}} \in \mathbb{R}^{H \times W \times 3}\). Training supervises this process to ensure that \(\mathbf{I}_{\text{SR}}\) retains the content of \(\mathbf{I}_{\text{LR}}\) via linguistic-content loss while also adhering to the details of \(\mathbf{I}_{\text{HR}}\) through linguistic-quality loss, as shown in Figure 2 (b).

3.2 Architecture of encoders↩︎

LA-SR employs contrastive learning between images and text, using the image encoder \(\mathcal{E}_{\text{I}}\) and text encoder \(\mathcal{E}_{\text{T}}\) from the pre-trained CLIP model [15], both designed with a Transformer [38]. Here, since the original CLIP model has a fixed input size (e.g., \(224 \times 224\)), we follow the approach in [39] by removing the positional embedding from the image encoder \(\mathcal{E}_{\text{I}}\), allowing it to accept variable input sizes.

3.3 Linguistic-content loss↩︎

Previous SR methods [3], [5] directly compare \(\mathbf{I}_{\text{SR}}\) and \(\mathbf{I}_{\text{HR}}\) to preserve content, which is not applicable in our case due to unpaired LR-HR patches. To address this, we introduce a linguistic-content loss \(\mathcal{L}_{\text{LC}}\) to ensure that \(\mathbf{I}_{\text{SR}}\) retains the content of \(\mathbf{I}_{\text{LR}}\). To achieve this, we first generate content text from \(\mathbf{I}_{\text{LR}}\) and \(\mathbf{I}_{\text{HR}}\) using an image-to-text model [14]. Then, the text encoder \(\mathcal{E}_{\text{T}}\) processes these content texts, producing \(\mathbf{C}_{\text{LR}}\) from \(\mathbf{I}_{\text{LR}}\) and \(\mathbf{C}_{\text{HR}}\) from \(\mathbf{I}_{\text{HR}}\). Similarly, image features \(\mathbf{F}_{\text{LR}}\), \(\mathbf{F}_{\text{HR}}\), and \(\mathbf{F}_{\text{SR}}\) are extracted from \(\mathbf{I}_{\text{LR}}\), \(\mathbf{I}_{\text{HR}}\), and \(\mathbf{I}_{\text{SR}}\), respectively, using the image encoder \(\mathcal{E}_{\text{I}}\). Note that all the above features are \(\in \mathbb{R}^{1 \times c}\), where \(c\) represents the channel dimension and \(1\) represents the batch size when the batch size is set to 1.

Afterward, we jointly train the encoders and the SR network using text and image features. For the encoders, the loss function \(\mathcal{L}^{\mathcal{E}}_{\text{LC}}\) is optimized using a contrastive function as: \[\begin{align} \mathrm{Cont}(\mathbf{A}, \mathbf{B}) &= - \frac{1}{\ast}\sum^{\ast}_{i=1}\bigg(\mathrm{log} \frac{e^{\mathrm{cos}(\mathbf{A}_{i}, \mathbf{B}_{i})}}{\sum^{\ast}_{k=1 [k \neq i]}e^{\mathrm{cos}(\mathbf{A}_{i}, \mathbf{B}_{k})}} \bigg), \\ \mathcal{L}^{\mathcal{E}}_{\text{LC}} &= \mathrm{Cont}(\mathbf{F}_{\text{LR}} \parallel {\mathbf{F}_{\text{HR}}}, \mathbf{C}_{\text{LR}} \parallel {\mathbf{C}_{\text{HR}}}), \end{align} \label{eq:clip95content}\tag{1}\] where \(\mathbf{A}\) and \(\mathbf{B}\) represent any features in \(\mathbb{R}^{\ast \times c}\), and \(\mathrm{cos}(\cdot, \cdot)\) denote cosine similarity, where \(\mathrm{cos}(\mathbf{A}_{i}, \mathbf{B}_{i}) = \frac{\mathbf{A}_{i} \cdot {\mathbf{B}_{i}}^T}{\norm{\mathbf{A}_{i}} \norm{\mathbf{B}_{i}}}\). Moreover, \(\parallel\) denotes concatenation along the batch dimension. The proposed \(\mathcal{L}^{\mathcal{E}}_{\text{LC}}\) trains the encoders to align images with their corresponding content texts, while contrasting them with unrelated images and content texts.

For the SR network, we compute the loss function \(\mathcal{L}^{\text{SR}}_{\text{LC}}\) to ensure that the SR image \(\mathbf{I}_{\text{SR}}\) maintains strong correlation with the corresponding content text feature \(\mathbf{C}_{\text{LR}}\) as the inputs \(\mathbf{I}_{\text{LR}}\), ensuring both share the same content as: \[\mathcal{L}^{\text{SR}}_{\text{LC}} = \mathrm{Cont(\mathbf{F}_{\text{SR}}, \mathbf{C}_{\text{LR}})}. \label{eq:usr95content}\tag{2}\] Finally, the linguistic-content loss \(\mathcal{L}_{\text{LC}}\) is defined as sum of Equations 1 and 2  (\(\mathcal{L}_{\text{LC}}\)=\(\mathcal{L}^{\mathcal{E}}_{\text{LC}} + \mathcal{L}^{\text{SR}}_{\text{LC}}\)).

3.4 Linguistic-quality loss↩︎

For the quality of \(\mathbf{I}_{\text{SR}}\), we design linguistic-quality loss \(\mathcal{L}_{\text{LQ}}\). Unlike the content, where \(\mathbf{I}_{\text{SR}}\) and \(\mathbf{I}_{\text{LR}}\) should share the same content, \(\mathbf{I}_{\text{SR}}\) should contain more details than \(\mathbf{I}_\text{LR}\), as it uses more pixels to represent the same content. Therefore, assuming that patches from closer distances \(\mathbf{I}_{\text{HR}}\) contain more details than those from distant ones \(\mathbf{I}_{\text{LR}}\), we train the encoders to distinguish between \(\mathbf{I}_{\text{LR}}\) and \(\mathbf{I}_{\text{HR}}\), while contrastively training \(\mathbf{I}_{\text{SR}}\) to align with the distribution of \(\mathbf{I}_{\text{HR}}\).

To do this, we first prepare two sets of text for image quality, one for low-quality (e.g., \(\left\{bad \right\}\)) and the other for high-quality  (e.g., \(\left\{good \right\}\)). Then, we encode them into low-quality text features \(\mathbf{Q}_{\text{-}}\) and high-quality text features \(\mathbf{Q}_{\text{+}}\), respectively, each in \(\mathbb{R}^{1 \times c}\), using the text encoder \(\mathcal{E}_{\text{T}}\). Using the quality text features and the image features, we train the encoders to identify \(\mathbf{F}_\text{HR}\) as higher quality than \(\mathbf{F}_\text{LR}\) by optimizing the loss function \(\mathcal{L}^{\mathcal{E}}_{\text{LQ}}\) as: \[\mathcal{L}^{\mathcal{E}}_{\text{LQ}} = \mathrm{Cont}(\mathbf{F}_{\text{LR}}\parallel\mathbf{F}_{\text{HR}}, \mathbf{Q}_{\text{-}}\parallel\mathbf{Q}_{\text{+}}). \label{eq:clip95quality}\tag{3}\] Note that Equation 3 shows the case when the batch size is 1. In our actual training, we extend \(\mathbf{Q}_{\text{-}}\) and \(\mathbf{Q}_{\text{+}}\) to \(N\) batches.

Together with \(\mathcal{L}^{\mathcal{E}}_{\text{LQ}}\), we compute the loss function \(\mathcal{L}^{\text{SR}}_{\text{LQ}}\) to train the SR network, encouraging the image encoder \(\mathcal{E}_{\text{T}}\) to classify \(\mathbf{I}_{\text{SR}}\) as \(\mathbf{I}_\text{HR}\). Unlike Equation 2 , where each image feature is matched to a unique content text feature, \(\mathcal{L}^{\text{SR}}_{\text{LQ}}\) should ensure that every \(\mathbf{I}_{\text{SR}}\) aligns with a single high-quality text feature \(\mathbf{Q}_{\text{+}}\). To this end, we modify Equation 2 for \(\mathcal{L}^{\text{SR}}_{\text{LQ}}\) as: \[\mathcal{L}^{\text{SR}}_{\text{LQ}} = - \frac{1}{\ast} \bigg(\mathrm{log} \frac{e^{\mathrm{cos}(\mathbf{F}_{\text{SR}}, \mathbf{Q}_{\text{+}})}}{e^{\mathrm{cos}(\mathbf{F}_{\text{SR}}, \mathbf{Q}_{\text{-}})}} \bigg). \label{eq:usr95quality}\tag{4}\] Similar to Equation 3 , note that during actual training, \(\mathbf{Q}_{\text{-}}\) and \(\mathbf{Q}_{\text{+}}\) are extended to \(N\) batches, and \(\ast\) in Equation 4 is set to \(N\). Finally, the linguistic-quality loss \(\mathcal{L}_{\text{LQ}}\) is defined as the sum of Equations 3 and 4  (\(\mathcal{L}_{\text{LQ}} = \mathcal{L}^{\mathcal{E}}_{\text{LQ}} + \mathcal{L}^{\text{SR}}_{\text{LQ}}\)).

3.5 Modified VGG loss↩︎

While the proposed linguistic losses, \(\mathcal{L}_{\text{LC}}\) and \(\mathcal{L}_{\text{LQ}}\), help guide the SR network to produce visually pleasant SR images, relying solely on them may not ensure the preservation of low-frequency information (e.g., structure and color). To address this, we modify the conventional perceptual loss [2] \(\mathcal{L}_{\text{VGG}}\) and apply it between \(\mathbf{I}_{\text{SR}}\) and \(\mathbf{I}_{\text{LR}}\) in our LA-SR framework. Given the difference in resolution between \(\mathbf{I}_{\text{SR}}\) and \(\mathbf{I}_{\text{LR}}\), we incorporate additional pooling steps to extract VGG features for \(\mathbf{I}_{\text{SR}}\).

Finally, we combine all the aforementioned loss functions to formulate the total loss \(\mathcal{L}_{\text{tot}}\) as follows: \[\mathcal{L}_{\text{tot}} = \lambda_\text{LC}\mathcal{L}_{\text{LC}} + \lambda_\text{LQ}\mathcal{L}_{\text{LQ}} + \lambda_\text{VGG}\mathcal{L}_{\text{VGG}}, \label{eq:final95loss}\tag{5}\] where \(\lambda_\text{LC}=0.5\), \(\lambda_\text{LQ}=0.5\), and \(\lambda_\text{VGG}=0.1\) are empirically chosen hyperparameters of the corresponding loss functions.

4 Experiments↩︎

Datasets. As LA-SR only requires high-quality images, we train our model using high-quality images from DF2K [12] and LSDIR [40]. For evaluation, we apply SR networks to various benchmarks, including Set5 [41], Set14 [42], BSD100 [43], General100 [44], Urban100 [45], and DIV2K [12]. Unlike previous studies [3], [7], [8], which focus on bicubic-degraded images, we show results on their natural high-quality images. Moreover, we compare performance on real low-quality images from OST [23], DRealSR [46], and CameraFusion [27].

Metrics. As our goal is to generate photo-realistic SR images, we focus on evaluating visual quality using non-reference-based perceptual metrics. We employ BRISQUE [47], CLIP-IQA [48], TOPIQ (NR) [49], and MUSIQ [50], which are metrics for assessing perceptual quality in a reference-free setting. Furthermore, while the aforementioned perceptual metrics are central to our evaluation, for completeness, we also report reference-based distortion metrics, such as PSNR and SSIM, even though they are not our main focus. Additionally, we include reference-based perceptual metrics, including LPIPS [51], DISTS [52], and TOPIQ (FR) [49].

Experimental configurations. Like prior SR methods [7], [8], we initialize our SR network using PSNR-oriented pre-trained SR models, which are trained to reconstruct HR images from bicubic-downsampled LR images. We then fine-tune the SR network using the LA-SR framework under Equation 5 . For training, we use \(\mathbf{I}_{\text{LR}}\) sized \(56 \times 56\) and \(\mathbf{I}_{\text{HR}}\) sized \(224 \times 224\) with a batch size of 16. The model is trained for 300,000 iterations with a fixed learning rate of \(5 \times 10^{-5}\), and the entire training process takes 50 hours using four Quadro RTX 8000 GPUs.

4.1 Comparison with previous SR methods↩︎

Comparison with un- and self-supervised SR methods. Since LA-SR framework is designed to trained with unpaired low- and high-resolution images, it naturally shares conceptual similarities with both unsupervised and self-supervised SR approaches. To highlight these connections, we first present a comprehensive comparison with prior unsupervised SR methods [53], [54] and self-supervised SR methods [27], [29], [55] in Table 3. When compared to unsupervised methods [53]–[55], which learn SR without any paired HR supervision, LA-SR consistently delivers higher-quality reconstructions on real-world LR images, demonstrating stronger generalization to natural degradations. Furthermore, against self-supervised methods [27], [29], which typically rely on reference images or synthetic degradations to guide the learning process, LA-SR achieves competitive or superior performance while requiring no additional reference inputs. This highlights LA-SR’s ability to leverage language-based supervision to bridge the gap between unpaired LR-HR domains, achieving effective real-world SR without the constraints of explicit pairing or handcrafted degradations. Moreover, Figure [fig:qualitaive95masa] provides visual comparisons. As shown, previous self-supervised approaches [27], [29] struggle to accurately recover fine text details, whereas LA-SR produces sharper and more readable results.

Comparison with supervised SR methods. Furthermore, we extend our comparison to supervised SR methods that are trained with explicit LR–HR pairs constructed in different ways. In particular, using the same RRDB backbone [37] as our SR network for a fair evaluation, we compare LA-SR against three representative settings: (1) methods trained on bicubic-downsampled LR images [37], (2) methods trained on real LR images captured using specialized camera (denoted as ESRGAN\(^\dagger\)), and (3) methods trained on synthetically degraded LR images with more complex degradation models [8] setups [46].

First, as shown in Table 1, ESRGAN [37] trained on bicubic-downsampled degraded images consistently performs the worst due to a large gap between its training and testing domains. Similarly, ESRGAN\(^\dagger\), despite being trained on real-degraded images [46] from specific camera models, shows limited generalizability on other images. Likewise, models [8] trained on synthetic datasets often struggle to super-resolve high-quality images. Second, we evaluate performance on real low-quality images using the OST [23] and DRealSR [46] datasets. Since LA-SR is trained on naturally degraded LR images influenced by subject-to-camera distance, it also performs well on real low-quality images.

Moreover, Figure 4 provides visual comparisons across different supervised methods. As shown, even when compared with supervised methods, LA-SR consistently reconstructs finer structures, such as hair strands and subtle textures (i.e., the texture of hair and flower in the first and last row of Figure 4), yielding more natural and detailed outputs. For further analysis of another key aspect (i.e., distortion reduction), please refer to Appendix 9. For more visual comparisons, please refer to Appendix 13.

Comparison on generated images. With the increasing prevalence of AI-generated images, the ability to perform SR on such synthetic content has become increasingly important. To evaluate this capability, the lower portion of Table 1 shows SR performance on synthetically generated images, including the Dataset Diffusion [56] and a curated set of 100 images generated by LDM [57]. As shown, LA-SR consistently delivers superior performance not only on real-world degraded images but also on these high-quality synthetic samples. This demonstrates that LA-SR effectively generalizes across both natural and diffusion-generated domains.

4.2 Effect of depth-based LR-HR preparation↩︎

Comparison with differently prepared LR-HR patches. Table [tab:patch95selection] compares different approaches for obtaining LR-HR patches. While we use depth to assign distant patches as \(\mathbf{I}_{\text{LR}}\) and closer patches as \(\mathbf{I}_{\text{HR}}\), we also evaluate three alternatives: overlap, inclusion, and random. In the overlap (Figure [fig:sampling95a]), \(\mathbf{I}_{\text{HR}}\) is randomly chosen and \(\mathbf{I}_{\text{LR}}\) is selected to overlap with it beyond a certain threshold (i.e., 0.4). On the other hand, in the inclusion (Figure [fig:sampling95b]), \(\mathbf{I}_{\text{HR}}\) is randomly selected and \(\mathbf{I}_{\text{LR}}\) is confined within HR patches. As shown in the table, both the overlap and inclusion approaches result in excessive similarity between \(\mathbf{I}_{\text{LR}}\) and \(\mathbf{I}_{\text{HR}}\), making differentiation challenging. In the random (in Figure [fig:sampling95c]), \(\mathbf{I}_{\text{LR}}\) and \(\mathbf{I}_{\text{HR}}\) are selected randomly. While the random approach performs better than overlap and inclusion, our depth-based approach (Figure [fig:sampling95d]) consistently delivers more reliable results.

Background filtering. To train LA-SR, we extract LR from distant depth. However, since background areas are typically located farther from the camera, this depth-based sampling strategy tends to select patches that are dominated by background content, which may contain fewer meaningful structures or fine details compared to foreground regions. To examine the impact of this potential bias, we conduct an empirical study comparing a naive sampling strategy with a background filtering approach in Table ¿tbl:sup95tab:background?. As shown, while filtering out background regions provides a slight performance gain, background patches still contribute valuable contextual cues that benefit super-resolution training.

Distance-Irrelevant Degradations. Since the real LR patches \(\mathbf{I}_{\text{LR}}\) used in our LA-SR framework are primarily derived from naturally captured images, they may not fully represent extreme degradations such as those found in animation frames or old film footage. To improve robustness under sevely degraded conditions, we incorporate synthetic degradations following the strategy of RealESRGAN. As shown in Table [tbl:sup95tab:degrade], introducing such synthetic degradations significantly enhances performance on heavily degraded datasets, especially Real38 dataset, which includes diverse and challenging artifacts. However, this augmentation also introduces a trade-off: it causes a slight performance drop on datasets like DRealSR, where the degradations are primarily natural (e.g., distance-based blur) and lack the complex, domain-specific artifacts modeled by synthetic processes.

4.3 Effect of the proposed loss functions↩︎

Table 6 shows a quantitative analysis of our proposed loss functions: \(\mathcal{L}_{\text{LC}}\), \(\mathcal{L}_{\text{LQ}}\), and \(\mathcal{L}_{\text{VGG}}\), by evaluating performance when each term is ommitted. While using all loss terms consistently yields the best perceptual performance, Figure [fig:loss95fig] further shows the individual contributions of each loss function. First, omitting \(\mathcal{L}_{\text{LQ}}\) (Figure [fig:loss95fig95b]) results in severe artifacts. This likely occurs because multiple images can match the same content description (e.g., different images could correspond to “a bird" while still satisfying perceptual loss). Therefore, without \(\mathcal{L}_{\text{LQ}}\), the network lacks a strong regularizer to discourage implausible high-frequency details, leading to noisy or distorted reconstructions. Similarly, omitting \(\mathcal{L}_{\text{LC}}\) (Figure [fig:loss95fig95c]) causes the network to generate irrelevant details. Without explicit content alignment, the reconstruction drifts away from the intended structure, producing objects that are irrelevant to the input. Moreover, omitting \(\mathcal{L}_{\text{VGG}}\) (Figure [fig:loss95fig95d]) hinders the network’s ability to recover low-frequency details, causing noticeable color shifts. In contrast, our full model (Figure [fig:loss95fig95e]) trained with all three loss components, successfully balances structural fidelity, perceptual realism, and semantic alignment.

4.4 Visualization of encoded features↩︎

Figure 7 provides a visual analysis of the encoded features to show the role of our encoders. Figure [fig:distribution95a] shows a t-SNE [58] projection of two image features \(\mathbf{F}_{\text{LR}}\) and \(\mathbf{F}_{\text{HR}}\). As shown, our encoders effectively distinguish between \(\mathbf{F}_{\text{LR}}\) and \(\mathbf{F}_{\text{HR}}\), mapping them into the separate spaces (e.g., red and blue clusters). This indicates that encoders classify LR and HR images based on quality, enabling meaningful backpropagation in \(\mathcal{L}_{\text{LQ}}^{\text{SR}}\). Figure [fig:distribution95b] shows the distribution of features \(\mathbf{F}_{\text{HR}}\), where images with the same color share the same keywords in their context texts. For example, the image-to-text model [14] produces context texts that include ‘Water’ for all the purple images. Interestingly, while the majority of images are distinctly divided by keywords, indicating that our encoders organize images according to their corresponding content, the orange and yellow patches are positioned closer together, reflecting similar properties between ‘Cat’ and ‘Dog’.

5 Limitations↩︎

While LA-SR shows strong SR performance (Tables 3, 1, and Figure 4), our LR patches are mainly extracted from high-quality images using depth information. For cases like animation or old films, different unseen degradation patterns can cause ringing and artifacts (also described in Section 4.2). We plan to expand the dataset to cover more diverse degradations in future work.

6 Conclusion↩︎

In this work, we present LA-SR, a novel SR framework that uses real LR images naturally occurring in photographs due to subject-camera distance. To handle the challenge of unpaired LR-HR images in our framework, we present linguistic-content and linguistic-quality losses, which guide content preservation and quality—two key factors in SR. Extensive experiments show that LA-SR achieves superior SR performance across various images. Also, our framework can be extended to other image restoration tasks, including deblurring and denoising.

7 Appendix↩︎

In this appendix, we provide discussions, details, and more visual results that could not be included in the main manuscript due to lack of space.

8 Comparison with Diffusion-based SR↩︎

In the main paper, we ensure a fair comparison by evaluating our LA-SR against methods that employ the same RRDB architecture. Table 2 further compares LA-SR with recent diffusion-based super-resolution approaches, including LDM [57], StableSR [59], and SeeSR [60]. As shown, although diffusion-based methods generally achieve stronger perceptual quality, our LA-SR attains competitive results, performing on par with LDM and StableSR and even surpassing them in several non-reference metrics (e.g., TOPIQ (NR) and MUSIQ). Considering the substantially lower parameter count and faster runtime of LA-SR compared to diffusion-based approaches, these results highlight its favorable balance between efficiency and performance.

9 Distortion-based Comparison↩︎

Table 3 presents various reference-based metrics, including distortion-based measures (e.g., PSNR and SSIM) and perceptual metrics (e.g., LPIPS [51], DISTS [52], and TOPIQ (FR) [49]). Although LA-SR may not achieve the highest distortion-based metric scores due to the trade-off between perceptual and distortion quality under limited network capacity [61], it remains competitive and excels in DISTS and TOPIQ (FR).

10 Reliance on Depth Estimator↩︎

We construct the unpaired LR-HR pairs using depth information. Since the accuracy of the predicted depth depends on the depth estimator used, we provide a quantitative comparison across different estimators. As shown in Table 4, our LA-SR demonstrates robust performance regardless of the chosen depth estimator, indicating that substituting DepthFM with other networks, Depth Anything V2 [62], yields comparable results on DIV2K.

11 Practical applications↩︎

11.1 Fine-tuning on specific images↩︎

As LA-SR framework operates in an unpaired setting, it allows training directly on the target image intended for super-resolution. To demonstrate this, we randomly sample 10 images and fine-tune our SR network on these images, with evaluations shown on the same set in Figure 8. As shown, the quantitative progression over iterations shows consistent perceptual improvements with fine-tuning. This occurs because our SR network can adapt to specific degradation patterns present in the given images, further enhancing the results. Due to the small size of the dataset, we use a learning rate of \(1 \times 10^{-7}\), training only the SR network while keeping the encoders fixed.

11.2 Iterative approach.↩︎

Unlike other methods that train on LR patches that are always degraded, LA-SR includes a range of LR patches, from severely degraded images to relatively high-quality ones when all subjects in the image are closer to the camera. As a result, while previous methods are specifically tailored to degraded images, LA-SR remains effective even on high-quality images. Based on this, considering conventional SR methods mapping low-quality images to high-quality ones, LA-SR can be iteratively applied to its own super-resolved outputs, further enhancing image quality.

To demonstrate this, we apply \(\times 4\) SR networks twice to generate \(\times 16\) SR results in Figure 9. As shown, previous methods [37], [59] face challenges due to the significant domain gap between their training LR images and the recovered SR images, which often produces blurry results. Furthermore, the diffusion-based StableSR [59] alters the original image structure (e.g., transforming stars into circles in the second row of Figure 9). In contrast, LA-SR shows the most visually appealing results.

12 Details of Image-to-text model↩︎

We utilize an image-to-text model to extract content texts from images. Specifically, we employ the CLIP [15]-Interrogator, which evaluates the correlation between words in a dictionary and a given image, selecting the top-\(k\) (\(k=24\)) most highly correlated words. However, since computing correlations between all words and images is computationally expensive, we manually curate a set of 1,000 representative texts in our dictionary to reduce the overall computational burden.

13 Additional Visual Comparison↩︎

Figures 11, 12, and 13 present additional visual comparisons between our LA-SR and previous SR methods. The results demonstrate that LA-SR consistently generates perceptually superior outputs compared to existing approaches.

References↩︎