Fast and Accurate Image Restoration and Generation with Rank Enhanced Linear Attention
May 22, 2025
Abstract
Transformer-based models have made remarkable progress in image restoration (IR) tasks. However, the quadratic complexity of self-attention in Transformer hinders its applicability to high-resolution images. Existing methods mitigate this issue with sparse or window-based attention, yet inherently limit global context modeling. Linear attention, a variant of softmax attention, demonstrates promise in global context modeling while maintaining linear complexity, offering a potential solution to the above challenge. Despite its efficiency benefits, vanilla linear attention suffers from a significant performance drop in IR, largely due to the low-rank nature of its attention map. To counter this, we propose Rank Enhanced Linear Attention (RELA), a simple yet effective method that enriches feature representations by integrating a lightweight depthwise convolution. Building upon RELA, we propose an efficient and effective Vision Transformer, named LAformer. LAformer eliminates hardware-inefficient operations such as softmax and window shifting, enabling efficient processing of high-resolution images. Extensive experiments across 7 IR tasks and 21 benchmarks demonstrate that LAformer outperforms SOTA methods and offers significant computational advantages. Furthermore, we extend LAformer to diffusion-based and flow-based visual generation, showcasing its strong potential as a competitive alternative to DiT and SiT. Code and models are available at https://github.com/shallowdream204/LAformer.
1 Introduction↩︎
Image restoration (IR) aims to recover a high-quality (HQ) image by removing degradations from its degraded low-quality (LQ) counterpart. As a fundamental task in low-level vision, image restoration has a wide array of real-world applications, spanning fields such as autonomous driving [1], surveillance monitoring [2], medical imaging [3], . Consequently, developing hardware-efficient IR networks holds substantial practical value.
Over the past decade, convolutional neural networks (CNNs) have achieved remarkable progress in various image restoration tasks [4]–[6]. However, CNNs are often limited by their constrained receptive field, making it challenging to capture long-range dependencies. Recently, Transformer-based methods [7]–[9] have significantly advanced the performance of image restoration.
While Transformer offers the advantage of a global receptive field, its core self-attention mechanism introduces quadratic computational complexity. Directly applying self-attention globally incurs substantial overhead, particularly when handling high-resolution images. To mitigate this, some methods compute self-attention within local windows [10]–[12], while others adopt sparse attention patterns [13], [14]. While effectively reducing computational costs, these methods either limit global perception or are sensitive to degradation types due to specific attention patterns [15].
The above analysis leads to a crucial question: Is it possible to design a hardware-efficient image restoration model that simultaneously preserves strong global perceptive capabilities? In this paper, we explore linear attention [16] as a potential solution to this issue. Compared to standard softmax attention [17], linear attention replaces the softmax operation with simpler activation functions (, ReLU [18], ELU [19]) and reorders the computation to first calculate \(K^{\top} V\). As shown in Fig. 2, linear attention reduces the computational complexity to a linear scale with respect to the number of visual tokens \(N\), while preserving the global contextual advantages of softmax attention. When processing high-resolution images, the global receptive field of linear attention enables the capture of long-range relationships between pixels, thereby enhancing restoration quality. These attributes motivate us to employ linear attention in developing an efficient and effective image restoration model.
However, standard linear attention mechanisms appear suboptimal for image restoration, resulting in a notable performance drop relative to softmax attention (see Tab. ¿tbl:tab:ablation95model?). Through rank analysis [20], we find that the low-rank nature of the attention map in linear attention severely limits the diversity of its output features, thereby diminishing its representation capacity (see Fig. 5). To overcome this limitation, we propose Rank Enhanced Linear Attention (RELA), which incorporates a lightweight depthwise convolution to effectively enhance the rank of output feature. RELA retains linear complexity while boosting the representation capacity of linear attention, enabling improved modeling of complex degradation patterns.
We further propose LAformer, an efficient Vision Transformer structured within the widely adopted U-Net architecture [8], [21]. As shown in Fig. 3, LAformer incorporates a Dual-Attention (DA) Block that operates across spatial and channel dimensions, leveraging both RELA and channel attention [22] concurrently to facilitate efficient global information capture. Recognizing the crucial role of local information in restoring texture details, we introduce a Convolutional Gated Feed-Forward Network (CG-FFN) to strengthen LAformer’s capability in modeling spatial locality. This architecture enables LAformer to effectively integrate local and global information, enhancing its representational power. Notably, LAformer circumvents hardware-inefficient operations like softmax and window shifting, significantly boosting its efficiency for high-resolution image processing. As shown in Fig. 1, our LAformer surpasses SOTA Transformer-based IR methods and significantly reduces computational overhead. Furthermore, we extend LAformer for diffusion and flow-based visual generation, demonstrating its remarkable efficiency and scalability.
Our main contributions can be summarized as follows:
We propose LAformer, a simple, efficient, and strong Vision Transformer that combines powerful representation capability with linear complexity.
We propose Rank Enhanced Linear Attention (RELA), a simple yet effective mechanism that boosts feature diversity in linear attention by enhancing the rank of features.
Extensive experiments across 7 IR tasks and 21 benchmarks validate that LAformer achieves SOTA performance while maintaining high efficiency.
When extended to visual generation, LAformer demonstrates impressive efficiency and scalability, highlighting its potential as a competitive alternative to DiT and SiT.
2 Related Work↩︎
Image Restoration. Images captured in real-world scenarios often suffer from various degradations, such as blur, low-light, and adverse weather conditions. These issues inevitably degrade the performance of downstream models and diminish the effectiveness of many vision systems [1], [23]–[25]. SRCNN [4] and DnCNN [5] are pioneering works that introduce CNN into image restoration and achieve impressive results. Subsequently, a plethora of CNN-based architectures incorporating meticulously designed modules have been introduced to enhance IR performance. Notable examples include the residual block [26]–[28], dense block [29]–[31], channel attention [22], and non-local attention mechanisms [32]–[34]. Despite their success, CNNs remain constrained by their inherently limited receptive field, posing challenges in effectively capturing long-range dependencies. Recently, researchers have begun to use self-attention mechanisms in place of convolutions to enhance IR performance [8], [10], [11], [35].
Vision Transformer. Since its introduction by NLP researchers, Transformer [17] has been widely adopted in computer vision tasks [36]–[40]. IPT [35] is the pioneering work that introduces Transformer to image restoration by processing images in small patches. Since then, many works [8], [20], [41]–[43] have focused on improving self-attention, aiming to reduce its significant computational overhead. SwinIR [10] and Uformer [11] perform self-attention in non-overlapping local windows. HAT [12] proposes overlapping cross-attention to establish cross-window interaction. ART [13] introduces sparse attention to IR for a large receptive field. AST [15] proposes an adaptive sparse self-attention to filter out irrelevant tokens. Although effective, a trade-off between global receptive field and computational efficiency persists.
Due to its efficiency and ability to capture global context, Linear attention [16] has garnered significant interest from the computer vision community. Shen [44] introduce linear attention to object detection and instance segmentation. Bolya [45] propose hydra attention by maximizing the number of linear attention heads. Han [46] propose a focused linear attention module to enhance efficiency and expressiveness. EfficientViT [47] finds that linear attention cannot produce sharp attention distributions and proposes multi-scale linear attention to enhance local information extraction. Distinct from prior works, ours is the first to demonstrate the efficacy of linear attention in high-resolution image restoration. Additionally, we investigate the limitation of linear attention in representation capacity and propose novel approaches to overcome the challenge.
3 Method↩︎
We propose LAformer, an efficient framework for high-resolution IR. First, we analyze the softmax attention and linear attention mechanisms in Sec. 3.1. Then Sec. 3.2 delves into the inherent low-rank characteristics of linear attention, motivating the development of Rank Enhanced Linear Attention (RELA) as a solution. Finally, in Sec. 3.3, we present LAformer, a model built on RELA that achieves explicit global modeling with linear complexity.
3.1 Softmax vs. Linear Attention↩︎
Softmax attention [17] is a widely used mechanism in Transformer. For a given input \(X\in\mathbb{R}^{N\times C}\), where \(N\) represents the number of visual tokens and \(C\) denotes the channel dimension, it can be generally formulated as \[\label{eq:softmax} Y_i=\sum_{j=1}^{N} \frac{\mathrm{Sim}(Q_i,K_j) }{\sum_{j=1}^{N}\mathrm{Sim}(Q_i,K_j)} V_j,\tag{1}\] where \(Q=XW_Q, K=XW_K, V=XW_V\), \(W_{Q,K,V}\in\mathbb{R}^{C\times C}\) are learnable linear projection matrices, \(\mathrm{Sim}(\cdot,\cdot)\) is the similarity function, \(i,j\) index visual tokens. When the similarity function is set as \(\mathrm{Sim}(Q_i,K_j)=\mathrm{exp}(\frac{Q_iK_j^{\top}}{\sqrt{C}})\), Eq. (1 ) represents the standard softmax attention. Due to the need to compute similarities between all queries and keys, softmax attention incurs a \(\mathcal{O}(N^2)\) complexity.
Linear attention [16] uses simple activation functions to approximate the similarity function as \(\mathrm{Sim}(Q_i,K_j)=\psi(Q_i)\psi(K_j)^{\top}\), where \(\psi(\cdot)\) is the activation function. With this assumption, we can rewrite Eq. (1 ) as \[\label{eq:linear} \begin{align} Y_i &= \sum_{j=1}^{N}\frac{\psi(Q_i)\psi(K_j)^{\top}}{\sum_{j=1}^{N}\psi(Q_i)\psi(K_j)^{\top} }V_j \\ &= \frac{\psi(Q_i)\left(\sum_{j=1}^{N}\psi (K_j)^{\top} V_j\right)}{\psi (Q_i)\left(\sum _{j=1}^N\psi (K_j)^{\top}\right)}. \end{align}\tag{2}\]
Compared to softmax attention, linear attention modifies the computation order from \((QK^{\top})V\) to \(Q(K^{\top}V)\), reducing the complexity from \(\mathcal{O}(N^2)\) to \(\mathcal{O}(N)\) while preserving the global modeling capabilities. However, the improvement in computational efficiency is accompanied by a noticeable drop in performance (see Tab. ¿tbl:tab:ablation95model?).
3.2 Rank Enhanced Linear Attention↩︎
Following [20], we conduct rank analysis for the linear attention map \(M\), which is computed as \(\psi(Q)\psi(K)^{\top}\in \mathbb{R}^{N\times N}\). According to basic matrix theory, it follows that \[\begin{align} \mathrm{Rank}(M) &\le \mathrm{min}( \mathrm{Rank}(\psi(Q)), \mathrm{Rank}(\psi(K)^{\top})\\ &\le \mathrm{min}(N,C). \end{align}\]
For high-resolution IR, the number of visual tokens \(N\) is typically much larger than \(C\), which results in the low-rank nature of the attention map in linear attention (see Fig. 4). Since the attention output is a weighted sum of \(V\) using the attention map, the highly correlated weights inevitably limit the expressive capacity of the output features. As shown in Fig. 5, the rank of output features produced by linear attention is notably lower than that of softmax attention, suggesting a reduced diversity in features.
To address this limitation, we introduce Rank Enhanced Linear Attention (RELA), depicted in Fig. 3 (c). We integrate a lightweight depth-wise convolution (DWC) into the computation process, which can be formulated as \[Y = (1+\mathrm{ELU} (Q))(1+\mathrm{ELU} (K))^{\top}V+W_dV, \label{eq:rela}\tag{3}\] where \(W_d\) represents the depth-wise convolution. Following [16], we use \(1+\mathrm{ELU}(\cdot)\) as the activation function.
The DWC can be interpreted as a local attention operation, enriching the output through local feature combinations. We can rewrite Eq. (3 ) as \(Y=(\psi(Q)\psi(K)^{\top}+M_{DWC})V=M_{attn}V\), where \(M_{DWC}\) represents the sparse matrix derived from the DWC operation, and \(M_{attn}\) denotes the resulting full attention matrix. Since \(M_{DWC}\) can potentially be of full rank, its incorporation effectively raises the upper bound of the rank of the overall attention map \(M_{attn}\). Fig. 4 and Fig. 5 illustrate that our RELA successfully restores the attention map and output features to a full-rank state, substantially enhancing the diversity of learned features. It is worth noting that our RELA introduces only a minimal increase in parameters and computational overhead compared to standard linear attention, yet it brings a significant performance boost (see Tab. ¿tbl:tab:ablation95model?).
3.3 Network Architecture↩︎
3.3.1 Overall Architecture↩︎
The overall architecture of our LAformer is depicted in Fig. 3. To rigorously assess the effectiveness of our proposed method, we adopt the classical U-Net architecture as the backbone, aligning with previous approaches [8], [11], [48]. For an input low-quality (LQ) image \(I_{lq}\in \mathbb{R}^{H\times W\times 3}\), LAformer initially applies a \(3\times 3\) convolution for overlapping patch embedding, transforming \(I_{lq}\) into a feature map with \(C\) channels. In both the encoder and decoder modules, we incorporate Dual Attention (DA) Blocks to efficiently capture both global and local degradation patterns. Within each stage, skip connections link the encoder and decoder, enabling seamless information flow across intermediate features. Across stages, pixel-unshuffle and pixel-shuffle operations are employed for effective feature down-sampling and up-sampling.
3.3.2 Dual Attention Block↩︎
To improve both global and local feature extraction in LAformer, we integrate a dual attention module and a Convolutional Gated Feed-Forward Network (CG-FFN), which address global and local feature learning, respectively. The details of these components are outlined below.
Dual Attention. We use the proposed RELA to achieve efficient global modeling in the spatial dimension. Considering that channel-wise information also plays an important role in IR, we introduce channel attention [22] into the DA Block, forming the Channel Attention Block (CAB) in conjunction with convolutions, as depicted in Fig. 3 (d). The process of CAB can be formulated as \[Y=W_{p_2}\mathrm{CA}(\mathrm{GELU}(W_dW_{p_1}X)),\] where \(W_{p_{(\cdot)}}\) is the \(1\times1\) point-wise convolution, \(W_d\) is the \(3\times3\) depth-wise convolution, \(\mathrm{CA}(\cdot)\) denotes the channel attention operation. Channel attention can be interpreted as a mechanism for modeling global dependencies, as it aggregates token information across the spatial domain through global average pooling. In the DA Block, we combine RELA and CAB to effectively model and aggregate global information across both spatial and channel dimensions, achieving efficient global modeling and aggregation.
Convolutional Gated FFN. As discussed in [47], compared to softmax attention, linear attention tends to produce less sharp attention maps, which hinders its ability to effectively capture local information. However, rich local information is crucial for restoring texture details. To address this, we propose a simple Convolutional Gated Feed-Forward Network (CG-FFN) to enhance the local fitting capability of LAformer. As illustrated in Fig. 3 (e), CG-FFN integrates depth-wise convolution with Gated Linear Unit (GLU) [49], which can be formulated as \[Y=W_{p_2}(\mathrm{GELU}( W_dW_{p_1}X)\odot W_dW_{p_1}X).\] Compared to MLP, CG-FFN incorporates depth-wise convolution to enhance LAformer’s capacity for capturing fine-grained local details while reducing computational cost, thereby complementing the linear attention mechanism.
Dual-Attention Block. Our DA Block is equipped with RELA, CAB, and CG-FFN, which is defined as \[\begin{align} X^\prime &= \mathrm{RELA}(\mathrm{LN}(X)) + \mathrm{CAB}(\mathrm{LN}(X)) + X, \\ Y &= \mathrm{CGFFN}(\mathrm{LN}(X^\prime)) + X^\prime. \end{align}\]
Built upon the stacked DA Blocks, LAformer avoids hardware-inefficient operations (, softmax, window shifting) and explicitly models spatial-wise global features, channel-wise global features, and local features, all with linear complexity. This enables LAformer to maintain high efficiency while possessing strong representation capability.
3pt
c c c c c c c c c c c c >cc & & DeblurGAN-v2 & SRN & MIMO-UNet+ & MPRNet & DeepRFT+ & BANet & MSSNet & Stripformer & FFTformer & GRL & LAformer-B
& & [50] & [51] & [52] & [53] & [54]& [55] & [56] & [57] &[58]& [20] & Ours
RealBlur-R & PSNR & 36.44 & 38.65 & N/A & 39.31 & 39.55 & 39.76 & 39.84& 39.84 & 40.11 & 40.20 & 41.07
[59] & SSIM & 0.935 & 0.965 & N/A & 0.972 & 0.971 & 0.972& 0.972&0.974 &0.973& 0.974 &
0.977
RealBlur-J & PSNR & 29.69 & 31.38 & 31.92 & 31.76 & 32.00 &32.10 &32.19& 32.48 &32.62&32.82 &32.92
[59] & SSIM & 0.870 & 0.909 & 0.919 & 0.922 & 0.923 & 0.928 &0.931& 0.929 &0.933&0.932&
0.933
4 Experiments↩︎
4.1 Experimental Setup↩︎
We evaluate the proposed LAformer on various datasets for 7 image restoration tasks: (1) motion deblurring, (2) defocus deblurring, (3) image dehazing, (4) image desnowing, (5) raindrop removal, (6) low-light enhancement, and (7) all-in-one image restoration. Furthermore, we extend LAformer for diffusion-based and flow-based visual generation. Additional details about the training dataset, training protocols, and more visual results can be found in the Appendix due to space limits.
Implementation Details. The kernel size for the depth-wise convolution in RELA is set to \(5 \times 5\). To align with SOTA models for each specific task, we adapt the number of blocks and channels, considering variants of different sizes, including Tiny, Small, and Base versions (LAformer-T/S/B). All experiments are conducted on A100 GPUs. Additional task-specific training hyperparameters and details of the model architecture can be found in the Appendix.
![Figure 6: Top row: Visual comparison on the DPDD [60] dataset for single-image defocus deblurring. Bottom row: Visual comparison on the GoPro [61] dataset for motion deblurring. Zoom in for a better view.](/article/2505.16157/_pdflatex/mbuikxop.png "fig:")
width=0.7
2pt
| Method | Indoor Scenes | Outdoor Scenes | Combined | |||||||||
| PSNR \(\textcolor{black}{\uparrow}\) | SSIM \(\textcolor{black}{\uparrow}\) | MAE \(\textcolor{black}{\downarrow}\) | LPIPS \(\textcolor{black}{\downarrow}\) | PSNR \(\textcolor{black}{\uparrow}\) | SSIM \(\textcolor{black}{\uparrow}\) | MAE \(\textcolor{black}{\downarrow}\) | LPIPS \(\textcolor{black}{\downarrow}\) | PSNR \(\textcolor{black}{\uparrow}\) | SSIM \(\textcolor{black}{\uparrow}\) | MAE \(\textcolor{black}{\downarrow}\) | LPIPS \(\textcolor{black}{\downarrow}\) | |
| DRBNet\(_S\) [62] | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | 25.73 | 0.791 | N/A | 0.183 |
| Restormer\(_S\) [8] | 28.87 | 0.882 | 0.025 | 0.145 | 23.24 | 0.743 | 0.050 | 0.209 | 25.98 | 0.811 | 0.038 | 0.178 |
| NRKNet\(_S\) [63] | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | 26.11 | 0.810 | N/A | 0.210 |
| INIKNet\(_S\) [64] | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | 26.05 | 0.803 | N/A | 0.185 |
| MPerceiver\(_S\) [65] | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | 25.88 | 0.803 | N/A | 0.190 |
| LAformer-B\(_S\) | 29.13 | 0.886 | 0.024 | 0.146 | 23.57 | 0.759 | 0.048 | 0.207 | 26.28 | 0.821 | 0.036 | 0.177 |
| RDPD\(_D\) [66] | 28.10 | 0.843 | 0.027 | 0.210 | 22.82 | 0.704 | 0.053 | 0.298 | 25.39 | 0.772 | 0.040 | 0.255 |
| Uformer\(_D\) [11] | 28.23 | 0.860 | 0.026 | 0.199 | 23.10 | 0.728 | 0.051 | 0.285 | 25.65 | 0.795 | 0.039 | 0.243 |
| IFAN\(_D\) [67] | 28.66 | 0.868 | 0.025 | 0.172 | 23.46 | 0.743 | 0.049 | 0.240 | 25.99 | 0.804 | 0.037 | 0.207 |
| Restormer\(_D\) [8] | 29.48 | 0.895 | 0.023 | 0.134 | 23.97 | 0.773 | 0.047 | 0.175 | 26.66 | 0.833 | 0.035 | 0.155 |
| LAformer-B\(_D\) | 29.53 | 0.898 | 0.023 | 0.125 | 24.49 | 0.795 | 0.044 | 0.160 | 26.95 | 0.845 | 0.034 | 0.143 |
4.2 Image Restoration↩︎
Motion Deblurring. Tab. ¿tbl:tab:motion? and Tab. ¿tbl:tab:motion95real? show the motion deblurring results on the synthetic and real-world datasets, respectively. When trained on the synthetic GoPro [61] dataset, our model outperforms all comparison methods on both the GoPro and HIDE [68] datasets. When trained on the real RealBlur [59] dataset, our model achieves the best results, surpassing GRL [20] by 0.87 dB in PSNR on RealBlur-R. Fig. 6 shows that LAformer produces sharper and clearer results compared to other methods.
We present a comparison of model efficiency on the GoPro dataset in Tab. ¿tbl:tab:efficiency?, where FLOPs are computed using 256×256 input images. Our method demonstrates superior efficiency compared to other approaches.
Defocus Deblurring. Tab. ¿tbl:table:dpdeblurring? shows the comparison results on the DPDD [60] dataset. Our LAformer consistently outperforms SOTA methods across all three scene types. Specifically, in the combined scenes, LAformer achieves a PSNR improvement of 0.3 dB over the Transformer-based method Restormer [8] in the single-image setting, and a 0.29 dB improvement in the dual-pixel setting. Fig. 6 shows that LAformer restores clear text, whereas other methods result in noticeable blurring or artifacts.
Image Dehazing. Tab. ¿tbl:tab:haze95syn? and Tab. [tbl:tab:haze95real] present the dehazing results on the synthetic and real-world datasets, respectively. LAformer achieves the best performance for all datasets. Fig. 7 shows that LAformer achieves the dehazing result closest to the ground truth compared to other methods.
Image Desnowing. Tab. [tbl:tab:snow] shows the desnowing results on three widely used datasets. We follow the dataset split used in FocalNet [69]. Our method achieves the best performance across three datasets.
Raindrop Removal. Tab. ¿tbl:tab:raindrop? shows the raindrop removal results on the Raindrop [70] dataset. Compared to recent methods AST [15] and Histoformer [71], LAformer achieves a PSNR gain of 1.06 dB and 0.32 dB, respectively.
Low-light Enhancement. As shown in Tab. [tbl:tab:lowlight], LAformer achieves the best PSNR score for all datasets. Fig. 7 shows that LAformer produces the clearest visual results, while the results of the comparative methods contain noticeable noise and artifacts.
All-in-One IR. We further validate the effectiveness of LAformer on the all-in-one IR task. Following the five-task setting of [72], [73], we train a single LAformer model to address five different types of degradations. As shown in Tab. 1, our LAformer outperforms DiffUIR across all tasks, delivering significant performance gains. This further highlights the strong representation capacity of LAformer, which achieves superior results without requiring any specialized design for the all-in-one IR task.
![Figure 7: Top row: Visual comparison on the LOL-v1 [74] dataset for low-light enhancement. Bottom row: Visual comparison on the Dense-Haze [75] dataset for image dehazing. Zoom in for a better view.](/article/2505.16157/_pdflatex/ynejipql.png "fig:")
| Method | Venue | Deraining \((5sets)\) | Enhancement | Desnowing \((2sets)\) | Dehazing | Deblurring | ||||||
| PSNR \(\uparrow\) | SSIM \(\uparrow\) | PSNR \(\uparrow\) | SSIM \(\uparrow\) | PSNR \(\uparrow\) | SSIM \(\uparrow\) | PSNR \(\uparrow\) | SSIM \(\uparrow\) | PSNR \(\uparrow\) | SSIM \(\uparrow\) | |||
| Restormer [8] | CVPR’22 | 27.10 | 0.843 | 17.63 | 0.542 | 28.61 | 0.876 | 22.79 | 0.706 | 26.36 | 0.814 | |
| AirNet [76] | CVPR’22 | 24.87 | 0.773 | 14.83 | 0.767 | 27.63 | 0.860 | 25.47 | 0.923 | 26.92 | 0.811 | |
| Painter [77] | CVPR’23 | 29.49 | 0.868 | 22.40 | 0.872 | N/A | N/A | N/A | N/A | N/A | N/A | |
| IDR [78] | CVPR’23 | N/A | N/A | 21.34 | 0.826 | N/A | N/A | 25.24 | 0.943 | 27.87 | 0.846 | |
| ProRes [79] | arXiv’23 | 30.67 | 0.891 | 22.73 | 0.877 | N/A | N/A | N/A | N/A | 27.53 | 0.851 | |
| PromptIR [80] | NeurIPS’23 | 29.56 | 0.888 | 22.89 | 0.847 | 31.98 | 0.924 | 32.02 | 0.952 | 27.21 | 0.817 | |
| DACLIP [81] | ICLR’24 | 28.96 | 0.853 | 24.17 | 0.882 | 30.80 | 0.888 | 31.39 | 0.983 | 25.39 | 0.805 | |
| DiffUIR [72] | CVPR’24 | 31.03 | 0.904 | 25.12 | 0.907 | 32.65 | 0.927 | 32.94 | 0.956 | 29.17 | 0.864 | |
| Varformer [82] | CVPR’25 | 31.33 | 0.913 | 25.13 | 0.917 | N/A | N/A | 32.96 | 0.956 | N/A | N/A | |
| - | 31.55 | 0.914 | 26.81 | 0.924 | 34.24 | 0.942 | 34.28 | 0.984 | 30.92 | 0.910 | ||
3pt
| Method | Venue | Params | FLOPs | FID\(\downarrow\) | IS\(\uparrow\) | Prec.\(\uparrow\) | Rec.\(\uparrow\) |
|---|---|---|---|---|---|---|---|
| DiT-S/2 [83] | ICCV’23 | 33M | 6.06G | 68.40 | - | - | - |
| DiC-S [84] | CVPR’25 | 33M | 5.9G | 58.68 | 25.82 | - | - |
| DiG-S/2 [85] | CVPR’25 | 33M | 4.30G | 62.06 | 22.81 | 0.39 | 0.56 |
| LAformer | - | 33M | 4.32G | 50.39 | 28.64 | 0.44 | 0.59 |
| DiT-B/2 | ICCV’23 | 130M | 23.02G | 43.47 | - | - | - |
| DiC-B | CVPR’25 | 130M | 23.5G | 32.33 | 48.72 | - | - |
| DiG-B/2 | CVPR’25 | 130M | 17.07G | 39.50 | 37.21 | 0.51 | 0.63 |
| LAformer | - | 131M | 17.32G | 26.92 | 53.17 | 0.56 | 0.64 |
| DiT-L/2 | ICCV’23 | 458M | 80.73G | 23.33 | - | - | - |
| DiG-L/2 | CVPR’25 | 458M | 61.66G | 22.90 | 59.87 | 0.60 | 0.64 |
| LAformer | - | 463M | 60.98G | 13.42 | 85.52 | 0.66 | 0.65 |
| DiT-XL/2 | ICCV’23 | 675M | 118.66G | 19.47 | - | - | - |
| DiC-XL | CVPR’25 | 702M | 116.1G | 13.11 | 100.15 | - | - |
| DiG-XL/2 | CVPR’25 | 676M | 89.40G | 18.53 | 68.53 | 0.63 | 0.64 |
| LAformer | - | 682M | 89.91G | 11.31 | 98.21 | 0.67 | 0.66 |
5pt
| Method | Venue | Params | FLOPs | FID\(\downarrow\) | IS\(\uparrow\) | Prec\(\uparrow\) | Rec\(\uparrow\) |
|---|---|---|---|---|---|---|---|
| SiT-S/2 [86] | ECCV’24 | 33M | 6.06G | 57.64 | 24.78 | 0.41 | 0.60 |
| FlowDCN-S/2 [87] | NeurIPS’24 | 30.3M | 4.36G | 54.6 | 26.4 | - | - |
| LAformer | - | 33M | 4.32G | 46.92 | 31.74 | 0.46 | 0.61 |
| SiT-B/2 | ECCV’24 | 130M | 23.02G | 33.02 | 43.71 | 0.53 | 0.63 |
| FlowDCN-B/2 | NeurIPS’24 | 130M | 17.87G | 28.5 | 51 | - | - |
| LAformer | - | 131M | 17.32G | 24.37 | 55.53 | 0.57 | 0.64 |
| SiT-L/2 | ECCV’24 | 458M | 80.73G | 18.79 | 72.02 | 0.64 | 0.64 |
| FlowDCN-L/2 | NeurIPS’24 | 421M | 63.51G | 13.8 | 85 | - | - |
| LAformer | - | 463M | 60.98G | 12.38 | 87.34 | 0.66 | 0.65 |
| SiT-XL/2 | ECCV’24 | 675M | 118.66G | 17.19 | 76.52 | 0.65 | 0.63 |
| FlowDCN-XL/2 | NeurIPS’24 | 618M | 93.24G | 11.3 | 97 | - | - |
| LAformer | - | 682M | 89.91G | 10.98 | 102.04 | 0.67 | 0.66 |
5pt
3pt
| RELA | Params (M) | FLOPs (G) | PSNR (dB) |
|---|---|---|---|
| \(\psi(\cdot)=\mathrm{ReLU}(\cdot)\) | 24.89 | 144.33 | 33.38 |
| \(\psi(\cdot)=1+\mathrm{ELU}(\cdot)\) | 24.89 | 144.33 | 33.40 |
| 3\(\times\)3 DWC | 24.79 | 143.11 | 33.35 |
| 5\(\times\)5 DWC | 24.89 | 144.33 | 33.40 |
| 7\(\times\)7 DWC | 25.03 | 146.16 | 33.41 |
3pt
| Method | Params (M) | FLOPs (G) | PSNR (dB) |
|---|---|---|---|
| Vanilla LA [16] | 6.18 | 29.08 | 39.28 |
| Window SA [10] | 6.20 | 30.19 | 40.76 |
| Transposed SA [8] | 6.26 | 29.87 | 40.52 |
| Rank Enhanced LA | 6.25 | 29.84 | 41.17 |
| w/o CAB | 6.13 | 30.79 | 40.81 |
| Dual Attention | 6.25 | 29.84 | 41.17 |
| MLP | 6.59 | 30.96 | 40.20 |
| CG-FFN w/o DWC | 6.24 | 30.16 | 40.11 |
| CG-FFN w/o GLU | 6.28 | 29.60 | 40.86 |
| CG-FFN | 6.25 | 29.84 | 41.17 |
4.3 Visual Generation↩︎
Following [83], [86], [89], we conduct class-conditional image generation experiments on ImageNet [90] at 256\(\times\)256 resolution. We adjust the depth and width of LAformer to match the parameter count and computational cost of baselines. We follow exactly the same experimental setup as DiT and SiT to ensure a fair comparison. Tab. 2 shows that across models of varying scales (ranging from approximately 33M to 680M parameters), our LAformer achieves superior generation performance with lower computational cost compared to DiT. This demonstrates the efficiency and scalability of LAformer in both diffusion-based and flow-based visual generation. Fig. 8 shows that our method is capable of generating high-quality images efficiently.
4.4 Ablation Study↩︎
As shown in Tab. ¿tbl:tab:ablation95model? and Tab. ¿tbl:tab:ablation95rela?, we perform detailed ablation studies on SOTS-Indoor [88] and GoPro [61] datasets to analyze the role of each component in LAformer. For fair comparisons, all models are designed with comparable complexity. FLOPs are computed on 256 \(\times\) 256 images. More ablations and analysis are provided in the Appendix.
Rank Enhanced Linear Attention. As shown in Tab. ¿tbl:tab:ablation95model?, RELA significantly improves performance over vanilla linear attention, achieving a 1.89 dB PSNR gain on SOTS-Indoor with only a modest increase of 0.07M parameters and 0.76G FLOPs. We also compare RELA with the commonly used window self-attention [10] and transposed self-attention [8] in image restoration. With similar parameter counts and FLOPs, our RELA achieves a PSNR improvement of 0.41 dB and 0.65 dB, respectively. This demonstrates the global perception capability of RELA, which contributes to improved restoration performance.
We also conduct ablations on the design of RELA, as shown in Tab. ¿tbl:tab:ablation95rela?. We analyze the choice of activation function \(\psi(\cdot)\), with experimental results indicating minimal impact on PSNR performance. This suggests that the effectiveness of our RELA is not dependent on meticulously chosen activation functions. We further analyze the impact of the depth-wise convolution (DWC) kernel size on the results. Considering both computational efficiency and performance, we ultimately select a \(5 \times 5\) kernel.
Channel Attention Block. Tab. ¿tbl:tab:ablation95model? shows that removing CAB results in a 0.36 dB drop in PSNR, highlighting the importance of channel-wise global information for IR.
Convolutional Gated FFN. Tab. ¿tbl:tab:ablation95model? indicates that the DWC in CG-FFN significantly enhances LAformer’s ability to extract local information, markedly improving model performance. Removing it results in nearly a 1 dB drop in PSNR.
5 Conclusion↩︎
In this paper, we introduce linear attention to image restoration, aiming to overcome the complexity limitations of Transformer. We uncover the inherent low-rank limitations of vanilla linear attention and propose a simple yet effective solution: Rank Enhanced Linear Attention (RELA). RELA enhances the rank of features through a lightweight depth-wise convolution, thereby improving its representation capability. Building upon RELA, we introduce LAformer, a Vision Transformer designed to achieve effective global perception through the integration of linear attention and channel attention. Comprehensive experiments show that LAformer surpasses SOTA IR methods in performance, while also offering considerable computational efficiency gains. Furthermore, we extend LAformer to visual generation, showcasing its excellent efficiency and scalability. We aim to further scale up LAformer and extend it to broader generative tasks, including text-to-image generation.