Single Image Super-Resolution (SISR) aims to recover a high-resolution (HR) image from its low-resolution (LR) input image, and it is useful in many important applications, such as medical imaging and object detection [1], [2]. Formally, the degradation process of the HR image can be defined as \[\label{eq95hr95degradation} \boldsymbol{I}=\boldsymbol{D}(\boldsymbol{H};\boldsymbol{\beta}),\tag{1}\] where \(\boldsymbol{I}\in R^{h \times w}\) is an observed LR image and \(\boldsymbol{H}\in R^{sh \times sw}\) is a corresponding HR image. \(s\) is a down-scaling factor. \(\boldsymbol{D}(\cdot)\) denotes the degradation operation with parameters \(\boldsymbol{\beta}\) and the default choice for \(\boldsymbol{D}(\cdot)\) in previous researches[3]–[6] is the bicubic downsampling operator. If we consider the degradation under non-ideal conditions (e.g. blurring and additive noise), which is closer to the real-world scenario, the degradation can be generalized as follows \[\label{eq95hr95degradation95with95noise95blur} \boldsymbol{I}=\boldsymbol{D}(\boldsymbol{k}*\boldsymbol{H};\boldsymbol{\beta}) + \boldsymbol{n},\tag{2}\] where \(*\) denotes the convolution operation, \(\boldsymbol{k}\) is a blur kernel with low pass filter, and \(\boldsymbol{n}\) is an additive noise.

The SISR tasks, recovering \(\boldsymbol{H}\) from \(\boldsymbol{I}\), are regarded as severly ill-posed problems, particularly when the scaling factor is large. To generate visually pleasing results, it is common practice to use natural image priors such as the representative self-similarity. Thus, many self-similarity-based SISR methods [3], [6]–[9] were proposed to address the ill-posedness and achieved satisfactory reconstruction results. The self-similarity is based on an observation that small textures in a natural image tend to recur within and across scales of the image[8], [10]. These repeated textures can provide valuable internal examples for a more faithful reconstruction. Essentially, the self-similarity provides valuable priors for exploring non-local image information. For example, when repairing hair textures, the related non-local hair regions are obviously more meaningful than low-frequency face regions or structured architectural regions. However, in SISR problems, previous studies usually assumed that the importance of non-local textures is positively related to their similarity scores, ignoring the flaws of this assumption, i.e., non-local textures with low-similarity may provide more accurate and richer details than the ones with high-similarity, especially when the query textures are severely damaged.

In this paper, we seek a deeper understanding of the role that the self-similarity plays in SISR tasks and expand its applicability in deep SISR models. In existing deep SISR models, the self-similarity is often integrated by non-local attention (NLA)[11], which was first proposed to explicitly model the long-range feature dependencies for high-level computer vision and also proven to be effective in SISR[9], [12], [13]. These NLA-based SISR models achieved satisfactory reconstruction performance by utilizing the NLA to capture the self-similarity priors. However, the previous methods are all based on a basic assumption to explore the self-similarity: non-local textures that are more similar to the query textures can provide richer information. We argue that the basic assumption is not always valid for SISR tasks. As illustrated in 1, when a similarity-based non-local search is performed on the severely damaged query textures, non-local textures with high-similarity will get higher similarity scores. However, these high-similarity non-local textures obviously cannot provide the critical information for repairing the severely damaged query textures. Essentially, the reason for this defect is that the basic assumption of the self-similarity cannot handle the situation where low-similarity textures are more accurate and informative than high-similarity textures. Repairing such severely damaged textures is a critical and extremely difficult task for SISR, so it is meaningful to design a non-local textures exploration scheme that can handle this situation. Another problem of NLA-based SISR models is that the computational complexity is quadratic to the size of the input image, which is generally unacceptable for SISR tasks.

To address these issues, we proposed a Global Learnable Attention (GLA) module, which can modify similarity score between two features adaptively during training and has asymptotic linear computational complexity with respect to the size of the input image. Our GLA module consists of two core parts: a Learnable Similarity Scoring (LSS) function and a Super-Bit Locality-Sensitive Hashing (SB-LSH)[14]. With the SB-LSH, our GLA module can perform hashing on input features and compute similarity only within the same hash bucket. The SB-LSH not only can reduce the computational complexity of the standard self-attention from quadratic to asymptotic linear, but also eliminate a large amount of redundant and irrelevant non-local information. The proposed LSS function aims to address the inherent limitations of the self-similarity (see 1) by providing trainable parameters to adaptively modify similarity scores between input features. Furthermore, we constructed a Global Learnable Attention based Features Fusion Module (GLAFFM) to fuse local adjacency information and learnable non-local self-similarity information. The local adjacency information is captured by the common component of Local Features Fusion Block (LFFB) that consists of several simple residual convolution layers. Finally, we integrated some GLAFFMs into a residual backbone to build our Deep Learnable Similarity Network (DLSN) (see 2) for SISR tasks.

The effectiveness of our DLSN will be verified in the experiment section for different degradation types of SISR tasks: bicubic-downscale, blur-downscale, and noisy-downscale. In all degradation types, our DLSN outperforms other state-of-the-art SISR models [9], [13], [15], [16] by a large margin. In addition, to verify the universality of our GLA in deep SISR, we integrated our GLA into some representative deep SISR models, such as FSRCNN [4], EDSR[5], and RCAN[16]. Experimental results demonstrate that our GLA can significantly improve the SR performance of various deep SISR models. Ablation studies are also conducted to analyze the impact of the proposed LFFB and GLA on the reconstruction results. The main contributions of this paper are summarized as follows.

• We provide some new insights into the self-similarity-based SISR solutions and argue that the basic assumption of the self-similarity is not flexible enough for SISR tasks. (As shown in 1)

• The proposed Global Learnable Attention (GLA) with Super-Bit Locality-Sensitive Hashing (SB-LSH) can adaptively modify the similarity scores during training and has asymptotic linear computational complexity.

• A new Deep Learnable Similarity Network (DLSN) is designed by using our Global Learnable Attention-based Features Fusion Modules (GLAFFMs) and achieves the state-of-the-art SR performance both quantitatively and qualitatively.

1 Related Work↩︎

The self-similarity has been widely applied to image generation problems. It assumes that similar texture patterns frequently recur within and across scales of the same image. Many classical SISR methods achieve satisfactory SR performance by exploring the self-similarity[3], [7], [8], [17], [18]. The difference between these classical SISR methods in exploring self-similarity is mainly in the range of non-local search space. For example, to reduce the computational complexity of exploring the self-similarity, the search space is usually constrained to some specified local regions[17], [18]. To achieve more accurate reconstruction quality, researchers usually extend the search space of the self-similarity to cross-scale images[7], [8]. In addition, the search space can be further expanded by modeling geometric transformations[3].

Although these classical SISR methods differ in the range of the self-similarity search space, they all follow the same principle of using the self-similarity, which assigns larger weights to non-local textures that are more similar to the query textures. This principle is still used in deep SISR models for exploring the self-similarity information[6], [9], [12], [13], [19]. Specifically, these deep SISR models use cosine similarity to measure the similarity scores between two features and assign higher weights to features with high-similarity. However, as illustrated in 1, we argue that this principle is not reasonable enough for SISR tasks. This motivates us to explore a learnable similarity scoring method to address this drawback.

In deep learning-based SISR, the non-local attention (NLA)[11] is commonly used to explore the self-similarity. However, the computational complexity of the NLA is quadratic to the size of the input image [9], [12], [13], which significantly affects the application of the NLA in deep SISR. Fortunately, there are many approaches to reduce the computational complexity of the NLA by using sparse attention, such as locality sensitive hashing attention[6], [20], routing attention[21], and BigBird[22]. For example, in routing attention[21], the query feature is routed to a limited number of context elements through its cluster assigned by the spherical k-means clustering. Compared with the standard NLA, these sparse attention methods can reduce the complexity significantly by making each feature interact with less but more similar features. Motivated by sparse attention, we propose to adopt the Super-Bit Locality-Sensitive Hashing (SB-LSH)[14] as a preprocessing method of our Global Learnable Attention (GLA). The reason we used SB-LSH to provide sparse attention is that it has two advantages: (1) It only adds negligible extra computation. (2) It is theoretically guaranteed [14] that the SB-LSH can achieve a small hashing variance.

2 Methodology↩︎

In this section, we will introduce our Deep Learnable Similarity Network (DLSN) in details. The DLSN consists of a residual backbone with our Global Learnable Attention-based Features Fusion Modules (GLAFFMs). For SISR tasks, there are two kinds of information which are critical for improving the reconstruction results: local adjacency information and learnable non-local self-similarity information. The GLAFFM is designed for fusing these two types of critical information captured by the Local Features Fusion Block (LFFB) and our Global Learnable Attention (GLA), respectively. As discussed in most previous studies, we used the LFFB as a common component to capture the local adjacency information. We start with an overview of the proposed DLSN and then introduce the details of each component in the GLAFFM.

2.1 Overview of DLSN↩︎

As shown in 2, our DLSN is an end-to-end SR framework which is composed of three parts: low-resolution (LR) features extraction, local and global deep features fusion, and HR image reconstruction. As suggested in [5], [16], only one convolutional layer with trainable parameters \(\boldsymbol{\alpha}\) is used to extract the shallow feature \(\boldsymbol{F}_0\) from the given LR image \(\boldsymbol{I}\). This procedure can be formally defined as follows \[\boldsymbol{F}_0=\Phi(\boldsymbol{I};\boldsymbol{\alpha}), \label{eq95shallow95feature95extract}\tag{3}\] where \(\Phi(\cdot)\) is the convolution operation in LR features extraction part. Then, \(\boldsymbol{F}_0\) is fed into the local and global deep features fusion part with \(m\) GLAFFMs to obtain refined deep features \(\boldsymbol{F}_m\) \[\boldsymbol{F}_m=\Psi(\boldsymbol{F}_0;\boldsymbol{\beta}), \label{eq95deep95feature95extract}\tag{4}\] where \(\Psi(\cdot)\) represents the function of our local and global deep feature fusion part with trainable parameters \(\boldsymbol{\beta}\). Finally, we upscale the obtained deep features \(\boldsymbol{F}_m\) by sub-pixel convolution \(\uparrow\) [23] and then use it to generate a high-resolution image \(\boldsymbol{\hat{I}}\) in the HR image reconstruction part as \[\boldsymbol{\hat{I}}=\Omega(\boldsymbol{F}_m\uparrow;\boldsymbol{\gamma}), \label{eq95hr95image95reconstruction}\tag{5}\] where \(\Omega(\cdot)\) denotes the HR image reconstruction part, which is implemented by a convolutional layer of 3 filters with trainable parameters \(\boldsymbol{\gamma}\) for the final RGB image reconstruction. We can combine the three parts above in our DLSN as \[\boldsymbol{\hat{I}}=\text{DLSN}(\boldsymbol{I};(\boldsymbol{\alpha},\boldsymbol{\beta},\boldsymbol{\gamma},\uparrow)), \label{eq95dlsn}\tag{6}\] where \(\text{DLSN}(\cdot)\) is the function of our deep learnable similarity network. In addition, to focus on learning high-frequency information and preventing gradients from exploding, we use a long skip connection in the DLSN to directly bypass abundant low-frequency information.

2.2 Global Learnable Attention-based Features Fusion Module (GLAFFM)↩︎

The structure of GLAFFM is shown in 2, from which we can see that the GLAFFM serves as a basic module of our DLSN. Specifically, each GLAFFM is also with the residual architecture and consists of a Local Features Fusion Block (LFFB), a Global Learnable Attention (GLA) and a feature refinement convlution layer. The LFFB is responsible for capturing locality inductive bias, while the GLA is exploring global information.

The trainable parameters of GLAFFM are omitted for simplicity and the corresponding function in the i-th GLAFFM can be defined as \[\boldsymbol{F}_i=\Psi_i(\boldsymbol{F}_{i-1})=\Psi_i(\Psi_{i-1}(\cdots \Psi_2(\Psi_1(\boldsymbol{F}_0)))), \label{eq95glaffm}\tag{7}\] where \(\Psi_i\) represents the function of i-th GLAFFM. \(\boldsymbol{F}_{i-1}\) and \(\boldsymbol{F}_{i}\) are the input and the output of the i-th GLAFFM.

2.2.1 Local features fusion block (LFFB)↩︎

As discussed in most previous studies [5], [24], we used the LFFB to exploit the locality of the convolution for capturing the local information of nature images. The LFFB is the cornerstone of SR reconstruction, which is formed by stacking \(n\) simplified residual blocks (see 2 for more details).

2.2.2 Global learnable attention (GLA)↩︎

Our GLA can explore global information by summarizing related features from input feature maps. Given the input feature maps \(\boldsymbol{X}\in R^{h\times w\times c}\), we reshap it into a 1-D feature \(\boldsymbol{X^{'}}\in R^{hw\times c}\) for illustration purposes. Then, the attention process of the query feature vector \(\boldsymbol{x}_i\) can be generally formulated as \[f(\boldsymbol{x}_i)=\sum_{j=1}^{n}\frac{{\rm exp}(s(\boldsymbol{x}_i,\boldsymbol{x}_j))}{\sum_{k=1}^{n}{\rm exp}(s(\boldsymbol{x}_i,\boldsymbol{x}_k))}\phi_v(\boldsymbol{x}_j), \label{eq95pixel95wise95similarity}\tag{8}\] where \(n=hw\), \(\boldsymbol{x}_j\) and \(\boldsymbol{x}_k\) are the j-th and k-th feature vectors on \(\boldsymbol{X^{'}}\) respectively. \(\phi_v(\cdot)\) is a feature transformation function that generates value vectors. \(s(\cdot , \cdot )\) is used to measure similarity and composes a learnable similarity scoring function \(s_l(\boldsymbol{x}_i)\) and a fixed dot product similarity scoring function \(s_f(\boldsymbol{x}_i,\boldsymbol{x}_j)\) \[s(\boldsymbol{x}_i,\boldsymbol{x}_j)=s^{j}_l(\boldsymbol{x}_i)+s_f(\boldsymbol{x}_i,\boldsymbol{x}_j), \label{eq95learnable95fc95and95fixed95dot95product}\tag{9}\] where \(s^{j}_l(\cdot)\) represents the j-th component in \(s_l(\cdot)\), and \(s_f(\cdot,\cdot)\) is the standard fixed dot product similarity scoring function, which can be defined as \[s_f(\boldsymbol{x}_i,\boldsymbol{x}_j)=\phi_q(\boldsymbol{x}_i)^\mathrm{T}\phi_k(\boldsymbol{x}_j), \label{eq95fixed95dot95product}\tag{10}\] where \(\phi_q(\cdot)\) and \(\phi_k(\cdot)\) are feature transformations which we use to generate query and key vectors. In practice, our GLA shared the parameters in \(\phi_q(\cdot)\) and \(\phi_k(\cdot)\) to reduce the computational cost, and we found that this modification didn’t reduce the SR performance. Existing non-local deep SISR methods[6], [9], [12], [13] assume \(s(\cdot,\cdot)=s_f(\cdot,\cdot)\), and we have discussed the limitations in Section [sec:intro]. To overcome the limitations, we proposed a learnable similarity scoring function \(s_l(\cdot)\) to adaptively modify similarity scores. It is worth noting that the similarity scoring function is highly nonlinear and varies from object to object. Here we used a single hidden layer feedforward neural network (FNN) that has achieved excellent representation ability to revise the fixed dot product similarity scoring. In addition, the modified term learned from the transformed feature \(s_l(\cdot)\) directly through the FNN with a single hidden layer (deeper networks are also possible) has linear computational complexity with respect to the size of the input image. Formally, the learnable similarity scoring function \(s_l(\cdot)\) can be defined as \[s_l(\boldsymbol{x}_i)=\boldsymbol{W}_2\sigma(\boldsymbol{W}_1\phi_l(\boldsymbol{x}_i)+\boldsymbol{b}_1)+\boldsymbol{b}_2, \label{eq95learnable95fc}\tag{11}\] where \(\sigma(\cdot)\) is the ReLU activation and \(\boldsymbol{x}_i\in R^{c}\), \(\boldsymbol{W}_1\in R^{n\times c}\), \(\boldsymbol{b}_1\in R^{n}\),\(\boldsymbol{W}_2\in R^{n\times n}\), \(\boldsymbol{b}_2\in R^{n}\).

2.3 Super-Bit Locality-Sensitive Hashing (SB-LSH)↩︎

As mentioned in Section 1, the sparse attention has been widely used in deep learning[6], [20]–[22] to reduce the computational complexity of the standard non-local attention from quadratic to asymptotic linear by computing the similarity only within each bucket. In this paper, we propose to adopt the SB-LSH[14] to hash global related features into the bucket of the query feature. The SB-LSH provides an estimation of angular similarity with negligible extra computation and shows that the more similar elements are more likely to fall into the same hash bucket. Thus, the SB-LSH can be used as a suitable preprocessing method for our Global Learnable Attention (GLA). Given \(b\) hash buckets, we first project the query feature \(\boldsymbol{x}_i\) with dimension \(c\) onto an orthonormal basis \(\boldsymbol{M}\in R^{b\times c}\): \[\boldsymbol{x}_i^{'}=\boldsymbol{M}\boldsymbol{x}_i. \label{eq95ortho95project}\tag{12}\]

Then, the assigned hash bucket of \(\boldsymbol{x}_i\) can be expressed as \(h(\boldsymbol{x}_i^{'})= \rm argmax(\boldsymbol{x}_i^{'})\), where \(\rm argmax(\cdot)\) finds the index that gives the maximum value from \(\boldsymbol{x}_i^{'}\). Finally, global features which are assigned in the same hash bucket \(\lambda_i\) with the query feature \(\boldsymbol{x_i}\) can be defined as \[\lambda_i=\{\boldsymbol{x}_j|h(\boldsymbol{x}_i^{'})=h(\boldsymbol{x}_j^{'})\}. \label{eq95query95hash95buckets}\tag{13}\]

The SB-LSH is simultaneously performed for all query features with batch matrix multiplication, which only adds negligible extra computation. With the preprocessing method SB-LSH, our GLA can achieve asymptotic linear computational complexity with respect to the size of the input image by computing the non-local attention only within the same hash bucket. Formally, the GLA with the preprocessing method SB-LSH can be derived from Eq. 8 and Eq. 13 as follows \[f(\boldsymbol{x}_i)=\sum_{\boldsymbol{x}_j \in \lambda_i}\frac{{\rm exp}(s(\boldsymbol{x}_i,\boldsymbol{x}_j))}{\sum_{\boldsymbol{x}_k \in \lambda_i}{\rm exp}(s(\boldsymbol{x}_i,\boldsymbol{x}_k))}\phi_v(\boldsymbol{x}_j), \label{eq95hash95pixel95wise95similarity}\tag{14}\] where \(\lambda_i\) contains the features which are assigned in the same hash bucket with the query feature \(\boldsymbol{x}_i\). Next, we will introduce some solutions to the problems which arise when using the SB-LSH. In practice, the size of hash buckets tend to be uneven, which makes it difficult to perform parallel computing. To solve the problems, we sort all query features by their bucket number and the sorted permutation is denoted as \(\xi:i\rightarrow \xi(i)\), where \(\xi(i)\) is the new position corresponding to the original index \(i\). Given buckets size \(l\), the sorted features in the k-th chunk \(C_k\) can be expressed as \[C_k=\{\boldsymbol{x}_{lk+1},\boldsymbol{x}_{lk+2},...,\boldsymbol{x}_{l(k+1)}\}. \label{eq95k95th95chunk95features}\tag{15}\]

Finally, the hash bucket \(\lambda_i\) of the query feature \(\boldsymbol{x}_i\) can be updated as \[\lambda_i=C_k, \label{eq95update95att95bucket}\tag{16}\] where the sorted index \(\xi(i)\) of \(\boldsymbol{x}_i\) is between \(lk+1\) and \(l(k+1)\). We can now obtain hash buckets of the same size for parallel computing. Some new hash buckets may cross the original bucket boundaries, which can be solved by extending the attention over adjacent buckets. After using adjacent buckets, the search space of non-local features is expanded to \(3l\).

The hash operation based on SB-LSH has a small probability that similar features are not split into the same bucket. This probability can be reduced by merging the results from multiple rounds of hashing. In the r-th round of hashing, the result of the query feature \(\boldsymbol{x}_i\) processed by our GLA (see Eq. 14 ) is defined as \(f_{r}(\boldsymbol{x}_i)\). Then, the results of our GLA with multi-hash SB-LSH is regarded as the weighted sum of each hashing attention result. Formally, the multi-hash GLA results \(\boldsymbol{\hat{x}}_i\) can be obtained by \[\boldsymbol{\hat{x}}_i=\sum_{r}\omega_rf_{r}(\boldsymbol{x}_i), \label{eq95multi95hash95gla}\tag{17}\] where \(\omega_r\) is the weight coefficient of r-th round hashing. The \(\omega_r\) represents the normalized similarity between the query feature and all features in its assigned r-th round bucket \[\omega_r=\frac{\sum_{\boldsymbol{x}_j \in \lambda^{r}_i} s(\boldsymbol{x}_i,\boldsymbol{x}_j)}{\sum_{\hat{r}=1}^{h}\sum_{\boldsymbol{x}_k \in \lambda^{\hat{r}}_i} s(\boldsymbol{x}_i,\boldsymbol{x}_k)}, \label{eq95r95round95weight}\tag{18}\] where \(\lambda^{r}_i\) is a set of global features which are assigned in the same hash bucket with the query feature \(\boldsymbol{x}_i\) in the r-th round of hashing. \(h\) is the number of hashing rounds. The effectiveness of the multi-hash SB-LSH will be verified in the experiment section.

More implementation details of our GLA with SB-LSH can be found in 3. For illustration, we only show the process of the i-th bucket in GLA, the remaining buckets are executed in parallel in the same way.

2.4 Computational Complexity of Similarity Scoring↩︎

Given an input feature \(\boldsymbol{X}\in R^{h\times w\times c}\), the standard non-local attention on \(\boldsymbol{X}\) is \(O((hw)^2c)\) in computational complexity. After hashing the input feature \(\boldsymbol{X}\) into \(\frac{hw}{l}\) buckets with size \(l\), the computational complexity of the standard fixed dot product similarity scoring \(s_f(\cdot,\cdot)\) and our learnable similarity scoring function \(s_l(\cdot)\) are \(O(hwlc)\) and \(O(hwlc+hwl^2)\), respectively. Thus, the computational cost of our GLA in calculating the similarity scores is \(O(2hwlc+hwl^2)\). It means that our GLA with SB-LSH has asymptotic linear computational complexity with respect to the size of the input image.

3 Experiments↩︎

3.1 Datasets and Evaluation Metrics↩︎

Following previous studies[5], [6], [13], [16], we used 800 images from DIV2K [25] as training datasets. Then, we tested our model on five standard SISR benchmarks: Set5[26], Set14[27], B100[28], Urban100[3], and Manga109[29]. All the results are evaluated by SSIM[30] and PSNR metrics on the Y channel in YCbCr space.

3.2 Implementation Details↩︎

Our final DLSN was built on a residual backbone with 10 GLAFFMs. In each GLAFFM, the number of residual blocks in LFFB was set to 4 empirically. All intermediate features have 256 channels, except those in our GLA, which have 64 channels. The last convolution layer in our DLSN has 3 filters to transfer deep features into a 3-channel RGB image. All the convolutional kernel sizes were set to \(3 \times 3\).

During training, a mini-batch consists of 16 images with patch size \(48\times 48\) randomly cropped from the training datasets and was augmented by random rotation of 90, 180, and 270 degrees and horizontal flipping. The mean absolute error (MAE) was used as the loss function to train our DLSN. We used ADAM optimizer[31] with \(\beta_1= 0.9\), \(\beta_2= 0.999\), and \(\epsilon=10^{-8}\) to optimize our model. In scale factor \(\times 2\), the initial learning rate was set to \(10^{-4}\) and reduced to half every 300 epochs until the training stops at 1500 epochs. When training our models for scale factor \(\times 3\) and \(\times 4\), we initialized the model parameters with pre-trained \(\times 2\) network and the learning rate \(10^{-4}\) was reduced to half every 50 epochs until the fine-tunning stops at 200 epochs. All our models were implemented by PyTorch and trained on Nvidia 3090 GPUs.

3.3 Ablation Study and Analysis↩︎

In ablation study, we trained our DLSN on DIV2K[25] for classical SISR with scale factor \(\times 2\) and observed the best PSNR (dB) values on Set14[27] in \(5\times10^4\) iterations.

3.3.1 Impact of LFFB and GLA↩︎

The effects of Local Features Fusion Block (LFFB) and Global Learnable Attention (GLA) in our basic unit Global Learnable Attention-based Features Fusion Module (GLAFFM) on SR performance are shown in 1.

Note: \(s_f(\cdot,\cdot)\) is the fixed dot product similarity scoring (see Eq.[eq95fixed95dot95product]) and \(s_l(\cdot)\) is our learnable similarity scoring (see Eq.[eq95learnable95fc]).

By comparing the PSNR of the first and second columns in 1, we can find that our learnable similarity scoring function \(s_l(\cdot)\) can bring 0.16dB performance improvement on Set14, which is remarkable for Set14 reconstruction. Furthermore, the zoomed in results on Set14[27], Manga109[29] and B100[28] datasets for SR \(\times 4\) are shown in 4, from which we can see that the network with using our learnable similarity scoring (LSS) can correct some inaccurate textures. Take ‘78004’ (bottom) in B100[28] as an example, our DLSN with using the LSS successfully recovers the structured architectural textures missed by the network without using the LSS.

To observe the mechanism of our LSS for non-local similarity modification, we show the locations of non-local features (red dots) corresponding to the top 128 weights associated with the query feature (green dot) in the last GLA. As shown in 5, when the fixed dot product is used for similarity scoring (see the first line in 5), the associated non-local features are relatively concentrated around the query feature, that is, the fixed dot product tends to give higher weights to non-local features similar to the query feature. On the contrary, after introducing our LSS (see the second line in 5), an expected change occurs, our DLSN tends to assign higher weights to more informative but low-similarity non-local regions. These results indicate that our LSS can indeed correct the weights of non-local textures, and enable our DLSN to reconstruct more accurate textures by exploring non-local regions with low-similarity but richer textures information.

In 1, we can see that locality inductive bias is crucial to SR performance. Without using our LFFB to explore the locality, the SR performance will degrade severely by about 0.3dB. The last column correspond to the GLAFFM used in our DLSN, which can explore local and learnable non-local information simultaneously. We also found that the fixed dot product similarity scoring function \(s_f(\cdot,\cdot)\) is still helpful, and the SR performance can be further improved from 33.65 to 33.77 when composing our learnable similarity scoring function \(s_l(\cdot)\) with \(s_f(\cdot,\cdot)\).

3.3.2 Impact of multi-hash SB-LSH and bucket size↩︎

Multi-hash SB-LSH. As discussed in the 2.3, increasing the number of hashing rounds can improve the robustness and effectiveness of our GLA. The number of hashing rounds \(h\) can be set flexibly in the inference stage to find the trade-offs between getting accurate SR performance and reducing computational complexity. The ablation study on hashing rounds \(h\) is shown in 2, from which we can see that increasing the number of hashing rounds in training and testing can both improve SR performance. Considering the SR performance and computational complexity, we set \(h\) to be 3 in both training and testing phases in our final DLSN model.

Bucket Size. As discussed in the 2.3, the bucket size \(l\) determine the number of non-local features that the query feature can explore. The effects of different \(l\) are shown in 7, from which we can see that the SR performance of our DLSN peaks at \(l=128\). When we further enlarging the bucket size, the SR performance starts to degrade. This is mainly because a larger \(l\) reduces the effectiveness of our SB-LSH by merging features across multiple bucket boundaries. Adjusting the bucket size may further improve the reconstruction results, however, it will take a lot of training time to determine the optimal bucket size for each practical dataset. Therefore, we recommend training our DLSN with a relatively appropriate bucket size obtained by our ablation studies. In the extreme case when \(l\) is equal to the size of the input image, our SB-LSH will be ineffective by making our GLA explore all global features.

3.3.3 Versatility of GLA↩︎

To analyze the versatility of our GLA, we integrate the GLA into existing representative deep SISR models with the different depths and parameters, such as FSRCNN[4], EDSR[5], and RCAN[16]. From 6, we observe that our GLA can improve the SR performance of these SISR models significantly. Specifically, our GAL brings 0.13dB, 0.14dB, and 0.17dB improvement for FSRCNN[4], EDSR[5], and RCAN[16], respectively. These results demonstrate that our GLA can be used as an efficient generic block to explore non-local information in deep SISR models.

3.4 Efficiency Analysis↩︎

Model Parameters Comparisons. We compare the model size and SR performance of our DLSN with other state-of-the-art deep SISR models including EDSR[5], RDN[15], RCAN[16], DBPN[32], RNAN[33], SAN[13], NLSN[6], IGNN[19]. As shown in 8, the SR performance of our DLSN (C=256) on Manga109 (\(\times 4\)) is significantly better than other deep SISR models. In addition, the smaller DLSN (C=128) which can still achieve remarkable SR performance as compared to the prior state-of-the-art models. Specifically, our DLSN (C=128, about 18.27M parameters) brings 0.38dB improvement in SR performance with much lower parameters than NLSN[6] (about 44.9M parameters). It means that the improvement in SR performance for our DLSN is not simply the result of having more parameters in the network.

Inference Time and Memory Comparisons. To analysis the efficiency of our DLSN, we compare the inference time and memory consumption of recently competitive SISR models on the Manga109 dataset with scale factor 4. We also provide a smaller version of our DLSN by settting the number of channels C to 128. The inferences of all models are conducted in the same environment with Nvidia 1070Ti GPUs, and the inference time is shown in 9. Specifically, by comparing RCAN[16] and NLSN[6], we found that the reconstruction performance of NLSN is 0.05dB higher than that of RCAN at a cost of 114.80 seconds. Furthermore, the reconstruction performance of our DLSN (C=128) is not only 0.38dB higher than that of NLSN, but also reduces the inference time by about 18.4 seconds. These results indicate that our DLSN is very efficient in improving SR performance, and the similar conclusion can also be obtained by comparing the memory consumption (see [fig:memory-consumption]).

3.5 Comparisons with State-of-the-art↩︎

3.5.1 Bicubic-downscale degradation↩︎

To verify the effectiveness of the proposed DLSN, we compare it with 17 state-of-the-art methods including FSRCNN[4], VDSR[34], LapSRN[35], EDSR[5], MemNet[36], SRMDNF[37], DBPN[32], RDN[15], RCAN[16], SAN[13], OISR[38],IGNN[19], CSNLN[9], HAN[39], NLSN[6], DRLN[24], and SwinIR[40]. DLSN+ is the self ensemble results of our DLSN.

Quantitative Evaluations. The quantitative comparisons with different scale factors are shown in ¿tbl:tab:x295x395x495psnr95ssim?, where we report the quantitative comparisons between our DLSN and 17 state-of-the-art deep SISR methods on five benchmark datasets for scale factor \(\times 2\), \(\times 3\) and \(\times 4\). From ¿tbl:tab:x295x395x495psnr95ssim?, we can see that our DLSN outperforms other state-of-the-art deep SISR models by a large margin on almost all benchmarks and scale factors. For example, compared with impressive NLSN[6] in scale factor \(\times 3\), our DLSN has 0.07dB, 0.10dB, 0.07dB, 0.29dB and 0.33dB performance improvement on Set5, Set14, B100, Urban100 and Manga109 datasets, respectively. The proposed DLSN can achieve significant improvements on challenging datasets Urban100 and Manga109, which contain extensive repeated textures.

On Urban100 (\(\times 2\)) dataset, which is designed for analyzing the self-similarity, we can see that recent deep SISR models have made very limited improvements on this dataset. For example, even though NLSN[6] integrates impressive sparse non-local attention in deep SISR models for exploring the self-similarity, PSNR only increases by 0.08dB from RCAN (ECCV2018)[16] to NLSN (CVPR2021)[6]. In contrast, our DLSN achieves a significant improvement on this dataset: we bring 0.36dB improvement in PSNR compared to RCAN[16]. The improvement is consistent with our motivation to design the GLA, which aims to capture the self-similarity information in LR images efficiently.

Qualitative Evaluations. Visual comparisons on Urban100 and Mange109 with scale factor \(\times 4\) are shown in 11 and 12, from both of which we can see that the proposed DLSN can restore the severely damaged textures when the corresponding non-local information can be found in LR images. On the contrary, deep SISR models without non-local attention cannot reconstruct severely damaged textures accurately. For example, by comparing the reconstruction results of image ‘img_092’ in 11, we observe that the generated results of our DLSN is very close to the HR, but the other very competitive deep SISR models without non-local attention such as EDSR[5], RCAN[16] are not suitable for recovering such severely damaged regions. Moreover, compared with the other non-local deep SISR methods such as SAN[13] and NLSN[6], our DLSN still maintains better reconstruction quality with more accurate textures. These comparison shows that our DLSN is more efficient in repairing severely damaged regions by exploring the self-similarity information with the proposed non-local method GLA.

The visual results demonstrate that our DLSN not only outperforms in quantitative metrics, but also perceptually better than the other deep SISR models by a large margin. In addition, more visual comparisons between our DLSN and some significant SISR methods including DRLN[24] and SwinIR[40] can be found in our supplementary file.

3.5.2 Blur-downscale degradation↩︎

Our DLSN is verified on blur-downscale degradation SISR tasks with scale factor \(\times 3\) and the gaussian standard deviation is set to 1.6 as discussed in SRMDNF[37] and RCAN[16]. The quantitative results of our DLSN are compared with the other 8 state-of-the-art methods: FSRCNN[4], VDSR[34], EDSR[5], SRMDNF[37], RDN[15], RCAN[16], SAN[13] and HAN[39].

Quantitative Evaluations. The quantitative comparisons of blur-downscale degradation with scale factor \(\times 3\) are shown in 3. From 3, we can see that our DLSN outperforms the very deep SISR model HAN[39], which has a well-engineered structure with the holistic attention. Specifically, compared with HAN, our DLSN has 0.17dB, 0.10dB, 0.08dB, 0.39dB and 0.42dB improvement on Set5, Set14, B100, Urban100 and Manga109 datasets, respectively. These results mean that our DLSN can still achieve very impressive SR performance when solving SISR tasks with blur-downscale degradation.

Qualitative Evaluations. Visual comparisons on Set14 datasets with blur-downscale degradation are shown in 10, from which we can see that our DLSN generates the most visual pleasing textures with accurate image details. From 10, we can also find that EDSR[5] cannot restore textures which are severely damaged by blur-downscale degradation, even though the selected region has informative repeated textures in the tablecloth of the input LR image. These visual results show that our DLSN is indeed effective when dealing with blur-downscale SISR tasks.

3.5.3 Noisy-downscale degradation↩︎

To verify the robustness of our DLSN in handling noisy-downscale degradation, we reimplemented some state-of-the-art deep SISR models under the same training datasets and compared them with our DLSN. 4 shows the SR performance of these state-of-the-art SISR models at noise levels of 10, 15, 20, and 25, respectively. From 4, we can see that our DLSN outperforms other state-of-the-art SISR models at all noise levels, which indicates that our DLSN is robust in handling SISR tasks with different noise levels.

3.5.4 Reference-based SISR↩︎

We also provided the comparisons with some significant reference-based SISR methods, including MASA[41]. Compared with reference-based SISR methods, our DLSN can still achieve competitive reconstruction performance without using the reference images. Please refer to the supplementary file for detailed comparisons.

3.5.5 Real-world images super-resolution↩︎

In this section, we provided the performance comparisons on real-world historic images with JPEG compression artifacts, as discussed in MS-LapSRN[42] and DRLN[24]. From 13, we can see that on the top LR input, our DLSN reconstructs more accurate structured architectural textures than the competitive MS-LapSRN and DRLN. Furthermore, compared with the results of Bicubic, we can observe that when the input letters are seriously damaged, our DLSN can still generate sharp and clear edges of the letters.

3.6 Limitations↩︎

Although our DLSN can reconstruct visually pleasing results, it struggles to "hallucinate" fine details if the input textures are completely destroyed. This is a common limitation shared by classic deep SISR methods including RCAN[16], NLSN[6] and DRLN[24]. For example, as shown in 14, all the classic deep SISR methods failed to repair the completely destroyed structured architectural region. The main reason for the limitation is that the completely destroyed input region lacks the basic texture patterns for the reconstruction.

4 Conclusion↩︎

In this paper, we provided new insights into the self-similarity in SISR tasks and found some critical limitations presenting in the existing deep self-similarity-based methods. To overcome these drawbacks, we design a flexible global learnable attention-based features fusion module (GLAFFM) that can make our deep learnable similarity network (DLSN) focus on more valuable non-local textures to repair severely damaged regions. Furthermore, with the super-bit locality-sensitive hashing (SB-LSH), our GLAFFM can achieve asymptotic linear computational complexity with respect to the image size when computing non-local attention. In addition, extensive experiments demonstrate that our GLA can handle SISR tasks with different degradation types (e.g. blur and noise), and can be integrated as an efficient general building block in deep SISR models.

References↩︎