1 Introduction↩︎

Light field imaging records the spatial-angular distribution of light rays, enabling sophisticated applications such as depth estimation and 3D reconstruction [1], [2]. However, hardware-constrained trade-offs between spatial resolution and angular density typically result in low-resolution sub-aperture images (SAIs), which significantly limit downstream performance. Consequently, Light Field Super-Resolution (LFSR) has emerged as a pivotal technique to restore high-frequency details while preserving 4D epipolar geometry, thereby overcoming hardware bottlenecks and enabling high-quality applications such as virtual reality and autonomous driving. Despite its promise, LFSR is hindered by insufficient spatial-angular modeling, particularly in occluded regions [3].

To address this core issue, early CNN-based methods (e.g., DistgSSR [4]) employed specific convolution kernels to decouple the 4D light field subspaces. However, due to local receptive fields, they struggle to capture long-range cross-view dependencies. Subsequent transformer-based architectures typically adopt a decoupled processing paradigm for spatial and angular information. For instance, EPIT [2] employs serial spatial convolution and angular transformers, while LFT [5] utilizes recurrent spatial and angular transformer modules for feature interaction. Although these methods mitigate the deficiencies in long-range context modeling, they primarily rely on shallow feature concatenation mechanisms when integrating the decoupled branches. Consequently, this decoupling paradigm lacks deep geometric awareness and spatial-angular synergy, limiting the ability to achieve global consistency and accurate reconstruction of high-frequency details.

Furthermore, within the realm of LFSR, deep spatial-angular consistency is directly manifested as distinct linear structures in Epipolar Plane Images (EPIs). Leveraging the long-range linearity of EPIs, State Space Models (SSMs) [6] offer an efficient sequence modeling framework with linear complexity. However, existing state-of-the-art architectures based on SSMs [7], [8] have failed to surmount this geometric representation hurdle. As shown in Fig. 1 (a), scene information in the EPI domain manifests as continuous slanted trajectories governed by disparity, rather than being distributed across a regular orthogonal grid. Regrettably, most current SSM-based LFSR methods directly inherit the rigid grid-based scanning patterns from Visual Mamba [6], resulting in a fundamental scanning-geometry mismatch. Specifically, whether referring to MLFSR [7], which introduces bidirectional scanning to capture long-range dependencies, or LFmamba [8], which employs channel-wise lightweight SSMs to facilitate feature fusion, these approaches artificially fragment the physical manifold of EPIs. This isolation of correlated pixels during serialization dilutes information and scatters attention, hindering focus on geometrically consistent epipolar lines and constraining feature extraction precision.

To bridge these gaps, we propose SMART (Slope-Guided Mamba and Angular-Refined Transformer). This hybrid architecture unifies geometry-aligned sequence modeling with structure-aware optimization, synergizing transformer’s texture extraction with mamba’s geometric consistency.

Specifically, to rectify the insufficient spatial-angular synergy inherent in existing decoupled methods, we propose the Angular-Modulated Spatial module (AMS). While leveraging Query-Key based attention to capture intra-view spatial context, this module injects angular priors derived from the macro-pixel layout into the attention maps as structural biases. By incorporating these angular constraints, the model effectively reconstructs high-frequency details in occluded regions, ensuring comprehensive feature completion and geometric consistency. Simultaneously, we propose the Manifold-Aligned Trajectory module (MAT), which innovatively employs a slope-guided geometric scanning strategy to address the scan-geometry mismatch. Capitalizing on the unique EPI structure, we explicitly estimate the local slope \(k\) governed by disparity and utilize it as a baseline for sampling offsets. This allows the scanning path to adaptively warp, breaking rigid grid constraints to ensure state transitions occur strictly along geometrically consistent epipolar lines rather than irrelevant background pixels. As illustrated in Fig. 1(a), this mechanism aggregates features from the same spatial point across views along their precise geometric trajectories. Experiments on five standard datasets validate the effectiveness of our approach. As shown in Fig. 1(b), SMART achieves a new SOTA of 33.04 dB / 0.9478 with efficient parameters and GFLOPs, surpassing L\(^2\)Fmamba [9] by 0.42 dB. Qualitative results in Fig. 3 confirm that SMART restores sharp text outlines and intricate textures while effectively suppressing distortion artifacts. The contributions can be summarized as follows:

(1) We propose SMART, a hybrid network that synergizes the content refinement capabilities of transformers with the geometric consistency modeling of mamba to preserve 4D light field coherence. (2) We design a slope-guided adaptive scanning mechanism that utilizes estimated EPI slopes \(k\) to break rigid grid constraints, enabling mamba to aggregate features along geometrically consistent trajectories. (3) Extensive evaluations on five benchmark datasets demonstrate that SMART achieves new SOTA results in PSNR and SSIM, significantly outperforming existing methods.

2 Methods↩︎

2.1 Methodology Overview↩︎

Problem Formulation. The light field feature is generally represented as a 4D tensor \(L \in \mathbb{R}^{U \times V \times H \times W}\), where \((U, V)\) and \((H, W)\) denote the angular and spatial resolutions, respectively. The goal of LFSR is to reconstruct a high-resolution light field \(L_{HR} \in \mathbb{R}^{U \times V \times \alpha H \times \alpha W}\) from its low-resolution counterpart \(L_{LR} \in \mathbb{R}^{U \times V \times H \times W}\), where \(\alpha\) represents the upscaling factor. This problem defines a mapping function \(\mathcal{F}\): \[L_{SR} = \mathcal{F}(L_{LR}; \Theta),\] where \(L_{SR}\) denotes the reconstructed LF and \(\Theta\) represents the learnable parameters of the network.

SMART Architecture. To address this problem, we propose SMART, a cascaded network for LFSR. As shown in Fig.2, the pipeline comprises three stages: feature extraction, deep structure interaction, and reconstruction. \[\begin{gather} F_0 = H_{Sf}(L_{LR}) \tag{1}, \\ F_i = AMS_i \left( MAT_i (F_{i-1}) \right), \quad i=1, \dots, N \tag{2}, \\ L_{HR} = H_{Rec}(F_{N}), \tag{3} \end{gather}\] where \(H_{Sf}\) extracts shallow features from the input \(L_{LR}\) using convolution layers. The interaction stage employs MAT module to reinforce epipolar consistency using EPI features, followed by AMS module for disparity and texture encoding. Finally, \(H_{Rec}\) upsample \(F_{N}\) via sub-pixel convolution to recover the high-resolution \(L_{HR}\).

2.2 Manifold-Aligned Trajectory Module↩︎

Applying SSMs [6] to LFSR faces a mismatch between fixed scanning trajectories and the underlying epipolar geometry of the light field. Specifically, scene points in EPIs manifest as slanted lines rather than orthogonal grid patterns. Consequently, standard raster scans compel SSMs to model spatially adjacent but geometrically unrelated pixels, disrupting the inherent manifold continuity. To address this, we propose the MAT module. It shifts sequence modeling from blind spatial scanning to geometry-aware manifold traversal, guiding feature extraction along continuous epipolar structures.

To leverage the inherent manifold structure, we reorganize \(F_{i-1}\) into a horizontal EPI stack \(X_{EPI}^{(H)} \in \mathbb{R}^{(U \cdot H) \times W \times V \times C}\), mapping scene depth to linear slopes in the \(V \text{-} W\) plane. We define a disparity hypothesis set \(\mathcal{K} = \{k_1, \dots, k_M\}\). For each pixel \(p\), we sample features along the trajectory corresponding to \(k_m\) within an angular neighborhood \(\Omega\). We quantify the degree of consistency among pixels along the trajectory by computing the consistency cost \(\mathcal{E}\): \[\mathcal{E}(p, k_m) = \frac{1}{C} \sum_{c=1}^{C} \text{Var} \left( { X_{EPI}^{(H)}(p + \delta \cdot \vec{v}_{k_m}) \mid \delta \in \Omega } \right),\] where \(\vec{v}_{k_m}\) is the direction vector defined by \(k_m\), and \(\text{Var}(\cdot)\) computes the trajectory variance. \(p\) represents the currently processed pixel point, \(C\) denotes the number of feature channels, and \(\delta\) is the offset within the angular neighborhood \(\Omega\) used for sampling. Lower \(\mathcal{E}\) indicates minimal feature dispersion, corresponding to the true scene geometry.

We convert the cost \(\mathcal{E}\) into a probability distribution \(P\) via Softmax and derive the optimal slope \(\mathbf{S}\) as the expectation: \[\begin{align} P(p, k_m) &= \frac{\exp (-\lambda \mathcal{E}(p, k_m))}{\sum_{j=1}^{M} \exp (-\lambda \mathcal{E}(p, k_j))},\\ \small \mathbf{S}(p) &= \sum_{m=1}^{M} k_m \cdot P(p, k_m) , \end{align}\] where \(\lambda\) is a learnable parameter, \(\mathbf{S} \in \mathbb{R}^{U \times V\times H \times W}\) encodes dominant EPI directions and provides a continuous and differentiable geometric manifold constraint for feature scanning.

Leveraging the geometric prior \(\mathbf{S}\), our slope-guided adaptive scanning mechanism to align SSM receptive fields with the manifold. The input \(X_{EPI}^{(H)}\) are flattened into three parallel sequences: forward \(X_{fwd}\), backward \(X_{bwd}\), and the slope-guided path \(X_{sg}\). While the raster paths capture bidirectional context, \(X_{sg}\) transcends grid limits, utilizing geometric priors to dynamically reconstruct the scanning trajectory.

The slope prior \(\mathbf{S}\) generates coordinate offsets and sorting indices, allowing the SSM to transcend grid limits. This facilitates scanning along epipolar lines, aggregating long-range information from geometrically correlated regions. Specifically, \(\mathbf{S}\) is embedded via an MLP and concatenated with feature \(F\). This combined input drives an offset network to produce spatial offsets \(\Delta P \in \mathbb{R}^{2 \times H \times W}\) and temporal sorting indices \(\mathcal{I} \in \mathbb{R}^{1 \times H \times W}\). \([\cdot ; \cdot]\) denotes channel-wise concatenation. \[(\Delta P, \mathcal{I}) = \Psi \left( [\text{MLP}(\mathbf{S}); F_{i-1}] \right),\] where \(\Psi\) denotes the offset and index generation network. We resample the original features \(F_{i-1}\) using \(\Delta P\) to achieve feature aggregation along the slope direction, yielding the aligned features \(F_{align}\). Subsequently, \(F_{align}\) is reordered based on \(\mathcal{I}\) to establish the slope-guided dynamic scanning trajectory.

These sequences are processed by a shared selective scan kernel, fused via summation, refined by an MLP, and reshaped, incorporating a residual connection to get the output \(F_{H}\): \[\small F_{H} = \text{MLP} ( \sum_{d \in {fwd, bwd, sg}} \text{SSM}(X_d) ) + F_{i-1},\]

where\(X_d\) denotes the input sequences. \(fwd\), \(bwd\), and \(sg\) denote the forward, backward, and slope-guided scanning orders, respectively. Analogously, for the vertical EPI stack, we transform the features \(F_{H}\) into \(X_{EPI}^{(V)}\) to obtain the corresponding output \(F_V\). The final output of the MAT is the fused representation \(F_{HV}\), which combines both \(F_H\) and \(F_V\).

This design aligns feature extraction with light transport geometry, facilitating consistent high-frequency reconstruction.

max width=

2.3 Angular-Modulated Spatial Module↩︎

Existing LFSR methods typically decouple spatial and angular processing, fusing them only at the final stage. This separation ignores the 4D coherence, leading to artifacts caused by geometric inconsistency. To address this, we introduce the AMS module, which intertwines spatial with angular geometry by leveraging angular correlations as attention biases.

Specifically, for an input \(F_{HV} \in \mathbb{R}^{U \times V \times C \times H \times W}\), we first reshape it into \(X_{S} \in \mathbb{R}^{(UV) \times C \times H \times W}\). We employ depth-wise separable convolutions \(W_{(\cdot)}\) to project \(X_S\) into query (\(Q\)), key (\(K\)), and value (\(V\)) embeddings: \[Q = W_Q(X_S), \quad K = W_K(X_S), \quad V = W_V(X_S).\]

However, spatial attention derived solely from \(Q\) and \(K\) is limited to capturing intra-view texture similarities, often overlooking the geometric consistency inherent in light fields. To bridge this gap, we introduce a parallel angular prior branch. Specifically, we reshape the input \(F_{HV}\) into macropixels \(X_{A} \in \mathbb{R}^{(HW) \times C \times U \times V}\) to explicitly model angular patterns, which are extracted via convolution \(f_{ang}\) and compressed by projection \(\phi\). Finally, we employ an additive broadcasting operation \(\mathcal{R}\) to generate the pairwise attention bias, defined as \(B_{i,j} = \phi(x_i) + \phi(x_j)\) for spatial positions \(i\) and \(j\). The formulation of the angular prior \(A_{prior}\) is given by: \[A_{prior} = \mathcal{R}\left( \phi \left( f_{ang}(X_A) \right) \right).\]

The angular prior \(A_{prior}\) is integrated into the spatial attention mechanism to impose geometric constraints. We concatenate the spatial affinity \(Q \cdot K^T\) with \(A_{prior}\) and apply a fusion layer \(\mathcal{F}_{mix}\) to generate the attention map: \[Attn_{map} = \sigma \left( \mathcal{F}_{mix}([Q \cdot K^T, A_{prior}]) \right),\] \[F_{i} = Attn_{map} \odot V + F_{HV},\] where \(\sigma\) denotes the activation function and \(\odot\) represents element-wise multiplication. This modulation integrates texture similarity with angular consistency, thereby enhancing reconstruction in texture-less regions with depth structures.

3 Experiments↩︎

3.1 Experimental Settings↩︎

Datasets and Preprocessing. We evaluate our method on five publicly available datasets: EPFL [10], INRIA [11], STFgantry [12], HCInew [13], and HCIold [14]. Experiments use the central \(5 \times 5\) SAIs, with training patches extracted via a sliding window from high-resolution images and downsampled by a factor of 4 using bicubic interpolation. The proposed model is trained for 100 epochs on 4 NVIDIA GeForce RTX 3090 GPUs. We employ the Adam optimizer with \(\beta_1 = 0.9\) and \(\beta_2 = 0.999\), and use the L1 loss as our optimization objective. The initial learning rate is set to \(2 \times 10^{-4}\) and is decayed by 50% every 25 epochs. The batch size is set to 2. For the disparity hypothesis set, we initialize \(K = \{-2, -1, 0, 1, 2\}\).

3.2 Comparison with State-of-the-Art Methods↩︎

Quantitative Results. As summarized in Tab. ¿tbl:tab:comparison95results?, SMART exhibits significant superiority over more than 20 representative LFSR methods, achieving an average PSNR of 33.040 dB and an SSIM of 0.9478. With a comparable parameter count, SMART achieves a PSNR gain of 0.417 dB over the previous state-of-the-art method, \(L^2\)Fmamba. This substantial performance gain validates the exceptional capability of SMART in modeling complex geometric consistency and reconstructing high-frequency texture details.

max width=

Qualitative Results. Fig. 3 illustrates the visual comparison of super-resolution reconstruction results on two real-world datasets. When dealing with fine structures such as text, typical methods, including those based on mamba, often suffer from blurred strokes or distortion artifacts. In contrast, the SMART, by accurately modeling EPI features, is able to restore text outlines, producing reconstruction details that are most consistent with the GT.

Robustness to Scene Geometry. Fig. 4 illustrates the performance consistency concerning scene geometry on the STFgantry dataset. It demonstrates the PSNR distribution across varying disparity levels. It can be observed that SMART maintains higher PSNR compared to other methods across disparity range. This confirms that slope-guided adaptive scanning effectively aggregates features along epipolar lines, mitigating performance loss in complex-depth regions.

3.3 Ablation Study↩︎

Component Effectiveness. As shown in Tab. 1, the spatial feature extraction establishes a solid foundation for light field reconstruction, yielding significant PSNR improvements of 2.7–4.2 dB across datasets. The Angle Prior guidance brings further gains of 1.5–2.6 dB, verifying the critical role of angular information in distinguishing complex light field structures. Finally, integrating the slope-guided adaptive scanning strategy boosts the average PSNR by 0.5 dB. This confirms that this strategy strictly aggregates features along epipolar lines to preserve geometric continuity, achieving more precise reconstruction than generic long-range modeling.

Impact of Cascade Blocks. Tab. 2 shows that as \(N\) increases from 3 to 5, the average PSNR rises from \(32.451\) dB to \(33.040\) dB. Since performance saturates at \(N=6\), we select \(N=5\) to balance accuracy and efficiency.

Scanning Strategies. As shown in Tab. ¿tbl:tab:scanning95strategies?, the standard rigid scanning (F+B) yields limited performance. Replacing either fixed direction with the slope-guided boosts performance to 30.014/32.251 dB. The full strategy achieves optimal results, demonstrating that the slope-guided deformable scanning adaptively aligns with light field disparity structures, compensating for the limitations of traditional fixed scanning.

4 Conclusion↩︎

In this paper, we propose SMART to address the challenge of maintaining high-dimensional geometric consistency in light field super-resolution. SMART incorporates angular priors and epipolar-guided scanning to strengthen spatial-angular correlation and consistent sequence modeling, enabling precise feature aggregation along the intrinsic 4D light field manifold. Experimental results demonstrate that SMART outperforms state-of-the-art methods in reconstruction accuracy, while exhibiting robust capabilities in restoring intricate textures and mitigating geometric inconsistencies.

References↩︎