1 Introduction↩︎

Light-field cameras capture both spatial appearance and angular variation of incoming rays, enabling dense multi-view reasoning from a single acquisition [1]–[3]. Under the standard two-plane parameterization, a light field is represented over spatial and angular coordinates, and epipolar plane images (EPIs) reveal a direct geometric relationship between line slope and disparity [4]–[6]. This makes light fields attractive for depth estimation, 3D reconstruction, computational photography, robotic perception, refocusing, and view synthesis [6]–[9].

Despite this rich angular structure, robust light-field disparity estimation remains challenging. Smooth surfaces, textureless regions, and large planar areas require globally consistent disparity estimates, while occlusion boundaries, thin structures, repetitive patterns, and abrupt foreground-background transitions require local precision [6], [9], [10]. Textureless regions provide weak local evidence, and occluded regions produce inconsistent observations across viewpoints [8], [11], [12]. These challenges indicate that light-field disparity is not simply a pixel-wise regression problem. The target disparity map is a structured spatial field that should combine long-range consistency with local spatial detail.

Current light-field disparity formulations largely focus on where to match across views or how to aggregate angular evidence. Local EPI and convolutional methods exploit short-range angular patterns [6], [10], [13]; cost-volume and focal-stack methods explicitly evaluate candidate disparities and regularize matching evidence [9], [14], [15]; attention-based methods select or aggregate reliable angular evidence [8], [11], [13]; and recent implicit neural field approaches represent disparity continuously but often rely on iterative optimization [16]. These approaches have substantially advanced light-field disparity estimation. However, they often rely on local matching windows, discrete disparity hypotheses, memory-intensive volumes, attention-based aggregation, or iterative optimization. This motivates an alternative field-level formulation: predicting disparity from globally and locally updated EPI-derived latent features rather than from an explicitly constructed disparity volume.

We introduce FreqLF, an EPI-guided Fourier-local framework for light-field disparity estimation. FreqLF first extracts angular parallax cues from horizontal and vertical EPI stacks, together with central-view appearance features. The resulting spatial feature field is projected to a latent feature field and updated by stacked hybrid Fourier-local layers. Each layer updates the same latent representation through two complementary branches: a Fourier-domain branch that applies learned weights to retained low-frequency Fourier modes for global feature interaction, and a local convolutional branch that applies spatial \(3 \times 3\) updates to preserve local spatial variations needed for fine disparity detail. A coordinate-conditioned Gaussian-mixture decoder then predicts disparity from the updated latent field, with the mixture mean used as the final disparity estimate. FreqLF therefore shifts the main computation from explicit disparity-volume construction to latent field updating. The Fourier and convolutional branches operate on a shared EPI-derived latent feature field: the Fourier branch introduces low-mode global feature interaction, while the local branch provides spatially localized updates. Their outputs are combined at each layer, producing a representation that carries both long-range field context and local spatial detail before disparity decoding. In its base form, this allows FreqLF to estimate disparity without explicitly enumerating candidate disparities in a cost volume. We evaluate FreqLF on the HCI 4D Light Field Benchmark [7]. The results show that FreqLF approaches the accuracy of strong supervised baselines while avoiding explicit cost-volume construction in the base model. Beyond average accuracy, we analyze the proposed formulation through branch ablations, Fourier-local complementarity, scaling behavior, and region-specific diagnostics. These experiments suggest that Fourier-local latent field learning is a practical alternative design principle for light-field disparity estimation, especially when global field interaction is desired without explicit disparity-volume construction. In summary, our contributions are:

• We proposed a cost-volume-free, field-level formulation for light-field disparity estimation, where disparity is decoded from EPI-derived latent feature fields updated by complementary global Fourier-domain and local convolutional operations.

• We introduce FreqLF, an EPI-guided Fourier-local architecture that encodes horizontal and vertical angular parallax cues, projects the fused spatial features to a latent field, and updates this field through stacked hybrid Fourier-local layers before disparity decoding.

• We demonstrate that the base FreqLF model approaches the accuracy of strong supervised light-field disparity baselines without explicit cost-volume construction, showing that discrete disparity-volume enumeration can be traded for global-local feature updates over EPI-derived latent fields.

• We analyze the proposed design through branch ablations, Fourier-local complementarity, scaling experiments, and region-specific diagnostics, identifying both the benefits of the Fourier-local mechanism and the remaining limitations of the current model.

2 Related Work↩︎

2.0.0.1 Geometry-driven light-field disparity.

Light fields provide a dense sampling of scene radiance over spatial and angular dimensions [1]–[3]. Their epipolar plane images (EPIs) expose a direct relationship between angular line slope and disparity [4], [5]. Classical light-field depth methods exploit this structure through EPI orientation analysis, structure tensors, variational regularization, defocus-correspondence cues, and geometric consistency [10], [17]–[19]. These methods are interpretable and closely tied to light-field geometry, but their local or handcrafted assumptions make them less robust under occlusion, repeated textures, and thin structures. FreqLF retains the geometric advantage of EPI features, but uses them to condition a learned frequency-structured disparity field rather than to perform local slope selection.

2.0.0.2 Learned spatial-angular representations.

Deep learning methods replace handcrafted EPI filters with learned spatial-angular representations. [6] introduced EPINET, a fully convolutional network that processes EPI stacks from multiple angular directions and established deep EPI processing as a strong light-field disparity baseline. Later methods improve efficiency or representation capacity through lightweight EPINET variants, directional EPI modeling, and spatial-angular disentanglement [13], [20], [21]. These methods learn useful angular features, but remain primarily convolutional or feature-disentanglement pipelines. In contrast, FreqLF uses EPI-derived features as input to a spectral-local field operator, explicitly separating smooth global structure from high-frequency disparity discontinuities.

2.0.0.3 Cost-volume and occlusion-aware matching.

Cost-volume methods estimate disparity by constructing matching evidence over candidate disparities and regularizing the resulting volume. This provides strong multi-view matching, but increases memory usage and ties prediction to discrete disparity hypotheses. [9] proposed OACC-Net, an occlusion-aware cost constructor for light-field depth estimation. [14] model sub-pixel disparity distributions using focal-stack and cost-volume information. Other recent methods refine cost construction through cascade estimation or EPI-guided cost volumes [15], [22]. These methods remain centered on constructing and regularizing matching costs. In contrast, the base version of FreqLF avoids explicit cost-volume construction and replaces discrete disparity enumeration with field-level spectral-local reasoning over EPI-derived latent features.

2.0.0.4 Attention, view aggregation, and self-supervision.

Attention-based methods improve robustness by selecting or aggregating reliable angular evidence. [8] introduced attention-based view selection for light-field disparity estimation, while later methods combine attention with EPI constraints or multi-scale aggregation [12], [13], [23]. Self-supervised and unsupervised approaches reduce dependence on ground-truth disparity by using photometric consistency, occlusion-aware losses, or epipolar consistency [11], [12]. These methods focus on view reliability, angular aggregation, or supervision design. FreqLF is complementary: it does not primarily ask which view to trust, but how the output disparity field should be represented and processed once angular evidence has been encoded.

2.0.0.5 Continuous fields and neural operators.

Implicit neural representations model continuous signals and query outputs at arbitrary coordinates [24], [25]. Recent light-field disparity work has explored continuous neural disparity fields, often through iterative or test-time optimization [16]. Neural operators instead learn mappings between function spaces, with Fourier Neural Operators providing global receptive fields through frequency-domain convolution [26], [27]. FreqLF takes inspiration from this operator view, but adapts it to light-field disparity through an EPI-guided hybrid design: global geometry is modeled by a Fourier branch, while boundary detail is preserved by a local convolutional branch. Thus, our goal is not merely continuous querying or applying a Fourier-local latent-field backbone, but frequency-structured disparity field learning.

3 Proposed Method↩︎

3.0.0.1 Problem formulation.

We consider a densely sampled light field \[\mathcal{L} \in \mathbb{R}^{A_v \times A_h \times H \times W \times 3},\] where \(A_v\) and \(A_h\) denote the number of angular views along the vertical and horizontal axes, and \(H \times W\) is the spatial resolution of each sub-aperture image. The central view is denoted by \(I_c \in \mathbb{R}^{3 \times H \times W}\). The goal is to estimate the central-view disparity field \[d: \Omega \rightarrow \mathbb{R}, \qquad \Omega = [-1,1]^2,\] which maps each normalized image coordinate \(\mathbf{q}=(q_x,q_y)\in\Omega\) to a scalar disparity value. A common formulation is to infer \(d(\mathbf{q})\) by local angular matching, direct regression from learned image features, or selection from a discrete set of candidate disparities. We instead view disparity estimation as prediction from a structured spatial field representation. The target disparity map should remain globally consistent over smooth surfaces, textureless regions, and large planar areas, while preserving sharp local changes at occlusion boundaries, thin structures, and foreground-background discontinuities. This creates a modeling tension. Local operators preserve spatial detail but have limited global context. Fourier-domain updates provide efficient full-field interaction through retained low-frequency modes, but low-mode truncation alone may lose fine spatial variations. FreqLF addresses this tension at the latent feature level: a Fourier-domain branch provides global feature interaction, while a local convolutional branch provides spatially localized feature updates before disparity decoding.

3.0.0.2 Overview of FreqLF.

The central component of FreqLF is a stack of hybrid Fourier-local layers applied to an EPI-derived latent feature field. The EPI encoder provides angular evidence, the Fourier-local layers update the latent field globally and locally, and the decoder predicts disparity from the updated representation. FreqLF follows a three-stage design, illustrated in Fig. 1. First, an EPI-guided encoder extracts angular parallax cues from horizontal and vertical EPI stacks, together with appearance features from the central view. Second, the fused spatial feature field is projected to a latent feature field and updated by stacked hybrid Fourier-local layers. Third, a coordinate-conditioned decoder bilinearly samples the updated latent field and predicts disparity values.

We construct horizontal and vertical EPI stacks through the central angular row and central angular column: \[E_h = \mathcal{L}[u_c, :, :, :, :] \in \mathbb{R}^{A_h \times 3 \times H \times W}, \qquad E_v = \mathcal{L}[:, v_c, :, :, :] \in \mathbb{R}^{A_v \times 3 \times H \times W}.\] Here, \(u_c\) and \(v_c\) denote the central vertical and horizontal angular indices. Each stack contains sub-aperture views along one central angular line. Classical 2D EPI slices are obtained by fixing a spatial row or column within these stacks. Given the central view \(I_c\), horizontal EPI stack \(E_h\), vertical EPI stack \(E_v\), and query coordinates \(\mathcal{Q}=\{\mathbf{q}_i\}_{i=1}^{N_q}\), FreqLF predicts \[\hat{d}(\mathbf{q}_i) = f_{\theta}(I_c, E_h, E_v, \mathbf{q}_i).\] In its base form, FreqLF avoids explicit cost-volume construction. It does not enumerate candidate disparities or build a multi-view matching volume. Instead, it encodes angular geometry into a spatial latent field and performs Fourier-local feature updating before disparity decoding.

3.0.0.3 EPI-guided feature encoding.

The encoder maps the light-field input to a spatial feature field \[\mathbf{F} \in \mathbb{R}^{C_F \times H \times W}.\] It combines three complementary sources of evidence: central-view appearance, horizontal EPI geometry, and vertical EPI geometry. The central view is processed by a spatial stream \[\mathbf{F}_s = \phi_s(I_c), \qquad \mathbf{F}_s \in \mathbb{R}^{C_s \times H \times W},\] which extracts local texture, edge, and appearance features. The horizontal and vertical EPI stacks are processed by angular streams \[\mathbf{F}_h = \phi_h(E_h), \qquad \mathbf{F}_v = \phi_v(E_v), \qquad \mathbf{F}_h \in \mathbb{R}^{C_h \times H \times W}, \qquad \mathbf{F}_v \in \mathbb{R}^{C_v \times H \times W}.\] The angular streams use 3D convolutions to aggregate information along the angular dimension while preserving spatial resolution. These streams encode EPI line-slope patterns, which provide direct geometric cues for disparity. The three feature maps are concatenated and fused: \[\mathbf{F}_{cat} = [\mathbf{F}_s;\mathbf{F}_h;\mathbf{F}_v], \qquad \mathbf{F} = \phi_f(\mathbf{F}_{cat}), \qquad \mathbf{F} \in \mathbb{R}^{C_F \times H \times W}.\] The fusion module uses a channel projection followed by channel-wise reweighting. This allows the model to adaptively balance appearance information from the central view and angular-geometric information from the EPI streams before applying Fourier-local field processing.

3.0.0.4 Hybrid Fourier-local layers.

The encoder output is first projected to the latent feature dimension by a \(1\times1\) convolution: \[\mathbf{z}^{0} = W_{\mathrm{proj}}(\mathbf{F}), \qquad \mathbf{z}^{0} \in \mathbb{R}^{C_z \times H \times W}.\] FreqLF then applies \(L\) hybrid Fourier-local layers. Each layer updates the same latent feature field through a Fourier-domain branch, a local convolutional branch, and a pointwise residual projection: \[\mathbf{z}^{\ell+1} = \sigma\left( \mathcal{K}_{\ell}(\mathbf{z}^{\ell}) + \mathcal{C}_{\ell}(\mathbf{z}^{\ell}) + W_{\ell}^{1\times1}(\mathbf{z}^{\ell}) \right), \qquad \ell=0,\ldots,L-1,\] where \(\sigma\) is a nonlinear activation, \(\mathcal{K}_{\ell}\) is the Fourier-domain low-mode update, \(\mathcal{C}_{\ell}\) is the local convolutional update, and \(W_{\ell}^{1\times1}\) is a pointwise channel-mixing projection. The Fourier branch provides a learned global update to the latent feature field. It applies a 2D Fourier transform over the spatial dimensions, learns complex-valued weights on a truncated set of low-frequency modes, and maps the result back to the spatial domain: \[\mathcal{K}_{\ell}(\mathbf{z}) = \operatorname{Re}\left[ \mathcal{F}^{-1} \left( R_{\ell} \cdot \mathcal{F}(\mathbf{z}) \right) \right],\] where \(\mathcal{F}\) and \(\mathcal{F}^{-1}\) denote the Fourier transform and inverse transform over the spatial dimensions. The product \(R_{\ell} \cdot \mathcal{F}(\mathbf{z})\) denotes mode-wise multiplication with learned channel mixing over the retained Fourier modes. For retained modes, the Fourier weights have shape \[R_{\ell} \in \mathbb{C}^{C_z \times C_z \times k_1 \times k_2}.\] In practice, \(R_{\ell}\) is applied only to the retained low-frequency modes, while the remaining modes are set to zero before the inverse transform. Since each retained Fourier mode has global spatial support, this branch allows distant spatial locations to interact through low-mode feature components with \(O(N\log N)\) scaling for \(N=HW\).

The local branch provides spatially localized feature updates: \[\mathcal{C}_{\ell}(\mathbf{z}) = \mathrm{Conv}_{3\times3}^{\ell}(\mathbf{z}).\] This branch complements the Fourier branch. Low-mode Fourier updates provide global field interaction, but they may suppress fine spatial variations when used alone. The local convolutional branch preserves local feature variations that are important for boundaries, occlusions, and thin structures. The two branches jointly update the same latent feature field before disparity decoding, producing a representation that combines long-range field context with local spatial detail.

3.0.0.5 Coordinate-conditioned disparity decoder.

After \(L\) hybrid Fourier-local layers, the final latent field \(\mathbf{z}^{L}\) is defined on the image grid. For each coordinate \(\mathbf{q}\in[-1,1]^2\), we obtain a feature vector by bilinear sampling: \[\mathbf{h}(\mathbf{q}) = \mathrm{bilinear\_sample}(\mathbf{z}^{L}, \mathbf{q}),\] and concatenate the sampled feature with the coordinate: \[\tilde{\mathbf{h}}(\mathbf{q}) = [\mathbf{h}(\mathbf{q}); \mathbf{q}].\]

A multilayer perceptron maps this coordinate-conditioned feature to the parameters of a Gaussian mixture: \[\left( \boldsymbol{\pi}(\mathbf{q}), \boldsymbol{\mu}(\mathbf{q}), \boldsymbol{\sigma}(\mathbf{q}) \right) = g_{\theta}\left(\tilde{\mathbf{h}}(\mathbf{q})\right).\] Here, \(f_{\theta}\) denotes the full FreqLF model, while \(g_{\theta}\) denotes the final MLP decoder. The mixture weights are normalized with a softmax, ensuring \[\sum_{k=1}^{K}\pi_k(\mathbf{q}) = 1, \qquad \pi_k(\mathbf{q}) \geq 0,\] and the standard deviations are parameterized with a softplus function to ensure \(\sigma_k(\mathbf{q}) > 0.\)

The decoder predicts disparity by bilinearly sampling the learned latent field rather than selecting from a discrete disparity volume. Since the latent representation is computed on an \(H\times W\) grid, FreqLF should be interpreted as a coordinate-conditioned decoder over a grid-based latent field, rather than as a fully implicit continuous representation.

3.0.0.6 Probabilistic output and training objective.

The predicted mixture parameters define the conditional disparity distribution \[p(d \mid \mathbf{q}) = \sum_{k=1}^{K} \pi_k(\mathbf{q}) \mathcal{N} \left( d \mid \mu_k(\mathbf{q}), \sigma_k^2(\mathbf{q}) \right),\] where \(\pi_k\) are mixture weights, \(\mu_k\) are component means, and \(\sigma_k\) are component standard deviations. At inference, the final disparity prediction is the mixture mean: \[\hat{d}(\mathbf{q}) = \sum_{k=1}^{K} \pi_k(\mathbf{q}) \mu_k(\mathbf{q}).\]

The model is trained by minimizing the negative log-likelihood over valid ground-truth pixels: \[\mathcal{L}_{\mathrm{NLL}} = - \frac{1}{|\mathcal{V}|} \sum_{\mathbf{q}_i \in \mathcal{V}} \log \left[ \sum_{k=1}^{K} \pi_k(\mathbf{q}_i) \mathcal{N} \left( d_i \mid \mu_k(\mathbf{q}_i), \sigma_k^2(\mathbf{q}_i) \right) \right],\] where \(\mathcal{V}\subset\Omega\) denotes the set of normalized coordinates corresponding to valid ground-truth pixels. We use the mixture mean for disparity evaluation, while uncertainty estimates derived from the predicted mixture distribution are used only for diagnostic analysis and are not part of the primary benchmark metric.

3.0.0.7 Enhanced front-end variant.

The base FreqLF model uses lightweight EPI streams and avoids explicit cost-volume construction. To test whether the proposed Fourier-local latent field backbone is compatible with stronger angular encoders, we also evaluate an enhanced variant, FreqLF+. In FreqLF+, the lightweight EPI encoder is replaced by an occlusion-aware front-end, while the Fourier-local latent-field backbone and coordinate-conditioned decoder remain unchanged. FreqLF+ is included as a controlled enhanced-front-end variant to test whether the proposed Fourier-local backbone benefits from stronger angular features. It is not used to support the cost-volume-free claim; all comparisons involving cost-volume-free reasoning refer to the base FreqLF model.

3.0.0.8 Implementation summary.

Unless otherwise stated, FreqLF uses \(C_z=128\) latent channels, \(L=4\) hybrid Fourier-local layers, \(k_1=k_2=16\) retained Fourier modes, and a \(K=5\)-component Gaussian mixture decoder. All models are trained end-to-end with Adam and a negative log-likelihood loss. Full architecture, training, and augmentation details are provided in Appendix 6.2.

4 Experiments↩︎

We evaluate FreqLF on the HCI 4D Light Field Benchmark [7]. The experiments address four questions: (i) whether Fourier-local latent field learning approaches strong light-field disparity baselines, (ii) whether the Fourier and local branches provide complementary updates, (iii) whether the base cost-volume-free model scales with spatial resolution, and (iv) where the method succeeds or remains limited beyond average error.

4.0.0.1 Dataset and protocol.

We use the four HCI stratified scenes: Backgammon, Dots, Pyramids, and Stripes. We train on the HCI training scenes and evaluate on the held-out stratified scenes. The reported evaluation scenes are not used for training or hyperparameter selection. Each light field contains a \(9\times9\) grid of sub-aperture views at \(512\times512\) resolution. We estimate central-view disparity and exclude invalid pixels and an \(11\)-pixel image border. For broader coverage, we also evaluate FreqLF+ on the 16-scene HCI additional split and report the full results in Appendix.

4.0.0.2 Metrics.

We report MSE\(\times10^2\) and BadPix\((\tau)\): \[\mathrm{MSE}\times 10^2 = 10^2 \cdot \frac{1}{|\mathcal{V}|} \sum_{\mathbf{q}_i\in\mathcal{V}} \left(\hat{d}_i-d_i\right)^2 ,\] \[\mathrm{BadPix}(\tau) = 100\cdot \frac{1}{|\mathcal{V}|} \sum_{\mathbf{q}_i\in\mathcal{V}} \mathbb{1}\left[ |\hat{d}_i-d_i|>\tau \right],\] where \(\mathcal{V}\) is the set of valid pixels. MSE emphasizes large squared errors, while BadPix\((0.03)\) and BadPix\((0.07)\) measure strict and more tolerant error rates.

4.0.0.3 Model variants.

FreqLF denotes the base model with an EPI-guided feature encoder, hybrid Fourier-local layers, and a coordinate-conditioned Gaussian-mixture decoder. FreqLF+ keeps the same Fourier-local backbone and decoder but replaces the lightweight EPI encoder with an enhanced EPI front-end. FreqLF+ tests whether the proposed latent field backbone benefits from stronger angular encoding; cost-volume-free claims refer only to the base FreqLF model.

1.8pt

4.0.0.4 Comparison with existing methods.

We compare against traditional, optimization-based, supervised, unsupervised, and iterative light-field disparity methods using reported per-scene HCI values [14], [16]. Table 1 shows that FreqLF+ ranks third by average MSE, within \(2.0\%\) of the best reported average score, while the base FreqLF ranks fifth, within \(4.1\%\). This supports our claim that FreqLF approaches strong supervised baselines while replacing explicit disparity-volume construction in the base model with Fourier-local updates over EPI-derived latent fields. Among EPI-centered methods (\(^{\dagger}\)), FreqLF+ achieves the best average MSE, and the base FreqLF remains competitive with Epinet-fcn-m while improving substantially over traditional EPI geometry and EPI orientation regularization.

2.0pt

Table 2 reports BadPix\((0.03)\) and BadPix\((0.07)\) on the same scenes. FreqLF+ obtains near-best average scores at both thresholds and the best Stripes result for BadPix\((0.07)\). Among EPI-centered methods (\(^{\dagger}\)), FreqLF+ achieves the best average BadPix scores, while the base FreqLF remains competitive with the strongest EPI-stack baseline. These results suggest that Fourier-local latent field processing is a useful extension of EPI-derived feature encoding.

4.0.0.5 Ablation study.

Table 3 isolates the contribution of each component. Removing the Fourier branch increases mean MSE from \(1.553\) to \(1.821\), showing that low-mode global feature interaction improves the latent field. Removing the local CNN branch causes a larger degradation to \(6.148\), confirming that spatially local updates are essential for preserving fine disparity detail. SE fusion and the probabilistic head also improve accuracy, but their contributions are smaller than the Fourier-local backbone.

4pt

4pt

4.0.0.6 Fourier bandwidth and computational scaling.

Table ¿tbl:tab:spectral95and95scaling? shows that \(k_{\max}=16\) retained Fourier modes gives the best validation MSE across the four stratified scenes. The same table reports inference time and peak GPU memory at increasing spatial resolutions on a single NVIDIA A100 GPU. The base FreqLF remains below \(20\) GB at \(1024\times1024\), while FreqLF+ is more expensive due to its enhanced front-end. This highlights the main trade-off: FreqLF+ improves accuracy with stronger angular encoding, whereas the base model provides the cost-volume-free and more scalable formulation.

4.0.0.7 Frequency and region diagnostics.

Figure 2 analyzes the Fourier-domain error on Stripes. The Fourier-only variant leaves mid-to-high-frequency errors, while the CNN-only variant shows stronger low-frequency bias; the full model gives the most balanced spectrum, supporting complementary Fourier and local updates. Figure 3 shows the masks used for targeted diagnostics. Boundary pixels are the top \(10\%\) of ground-truth disparity gradient magnitude, and textureless pixels are the bottom \(15\%\) of local central-view variance using a \(7\times7\) window. These masks are not official HCI labels, but they help inspect behavior in high-frequency discontinuities and low-texture regions separately.

5 Conclusion↩︎

We presented FreqLF, an EPI-guided Fourier-local framework for light-field disparity estimation. The main contribution is a field-level alternative to disparity-volume-centered design: instead of explicitly enumerating candidate disparities in the base model, FreqLF predicts disparity from EPI-derived latent features updated by complementary Fourier-domain global interactions and local convolutional operations. Experiments on the HCI 4D Light Field Benchmark show that this formulation approaches strong supervised baselines while retaining a cost-volume-free base architecture. FreqLF+ further improves accuracy with stronger angular encoding, indicating that the proposed Fourier-local backbone can benefit from enhanced EPI features. The MSE and BadPix results, together with branch ablations and scaling analysis, show that the proposed design trades explicit disparity-volume construction for competitive global-local latent field processing. Additional implementation details, extended results, diagnostic analyses, and limitations are provided in the appendix.

6 Technical Appendices and Supplementary Material↩︎

6.1 Detailed Architecture↩︎

The main paper presents the conceptual overview of FreqLF. Here, we provide stage-wise architectural details for reproducibility and to clarify the relation between the base model and the enhanced variant. FreqLF consists of three stages: an EPI-guided feature encoder, a hybrid Fourier-local operator backbone, and a coordinate-conditioned probabilistic decoder. FreqLF+ replaces the lightweight EPI encoder with an enhanced occlusion-aware angular front-end, while keeping the hybrid Fourier-local backbone and coordinate-conditioned decoder unchanged.

6.1.0.1 EPI-guided feature encoder.

The base FreqLF encoder processes three input streams: the central view, the horizontal EPI stack, and the vertical EPI stack. The central-view stream extracts spatial appearance features using 2D convolutions. The horizontal and vertical EPI streams use 3D convolutions to aggregate angular information while preserving the spatial grid. These angular streams encode EPI line-slope patterns, which provide geometric cues for disparity. The resulting feature maps are concatenated, projected to the latent channel dimension, and reweighted using channel-wise attention. This produces an EPI-conditioned spatial feature field used by the hybrid operator backbone.

6.1.0.2 Hybrid Fourier-local backbone.

The fused feature map is lifted to a latent field with \(C=128\) channels. The backbone uses \(L=4\) hybrid Fourier-local layers. Each layer contains a Fourier branch, a local \(3\times3\) convolutional branch, and a \(1\times1\) residual projection. The Fourier branch applies a 2D FFT over the spatial dimensions, retains \(k_1=k_2=16\) low-frequency modes, applies learned complex-valued weights with channel mixing, and maps the result back to the spatial domain with an inverse FFT. The local branch applies a spatial \(3\times3\) convolution to the same latent field. All layers operate at the full spatial resolution without downsampling, which preserves spatial detail while allowing global field-level interaction.

6.1.0.3 Coordinate-conditioned probabilistic decoder.

The final latent grid is queried at normalized image coordinates by bilinear sampling. The sampled latent feature is concatenated with the query coordinate and passed to an MLP decoder. In the probabilistic version, the decoder outputs the parameters of a \(K=5\)-component Gaussian mixture model. The final disparity prediction is the mixture mean. Since the latent representation remains an \(H\times W\) grid, FreqLF should be interpreted as a coordinate-conditioned decoder over a grid-based latent field, rather than as a fully implicit continuous scene representation.

6.1.0.4 FreqLF+.

FreqLF+ is an enhanced front-end variant. It replaces the lightweight EPI encoder with an occlusion-aware front-end, while keeping the hybrid Fourier-local backbone, Fourier mode configuration, decoder, and training objective unchanged. We use this variant only as a stronger angular feature extractor to test whether the proposed Fourier-local backbone benefits from improved angular evidence. Therefore, the cost-volume-free claim refers to the base FreqLF model, while FreqLF+ is reported as a modular enhanced variant.

Algorithm 8 summarizes the forward pass of the base FreqLF model.

6.2 Implementation Details↩︎

6.2.0.1 Model configuration.

Unless otherwise stated, all FreqLF experiments use the configuration in Table 4. The default Fourier bandwidth is \(k_1=k_2=16\), which is also the best-performing setting in the spectral bandwidth study.

5pt

6.2.0.2 Training setup.

All models are trained with Adam using the default momentum parameters \(\beta_1=0.9\), \(\beta_2=0.999\), and \(\epsilon=10^{-8}\). The learning rate is fixed to \(10^{-4}\), with no scheduler. The batch size is 1, and models are trained for 500 epochs. The training loss is the negative log-likelihood of the ground-truth disparity under the predicted Gaussian mixture. During each iteration, the model processes one light field, queries all valid spatial coordinates, computes the NLL loss, and updates all learnable parameters end-to-end.

5pt

6.2.0.3 Data augmentation.

We use multi-scale random cropping during training. Let \(S=\min(H,W)\). At each iteration, one of the crop strategies in Table 6 is sampled. The same crop is applied to the central view, EPI stacks, disparity map, and valid mask.

6pt

For ablation experiments, all variants are trained from scratch under identical hyperparameters, changing only the component under study. To reduce wall-clock time, ablations use a forced small-crop setting with crop sizes sampled in \([64,0.7S]\). This keeps the relative comparison controlled while reducing training cost.

6.2.0.4 Evaluation protocol.

We follow the HCI 4D Light Field Benchmark protocol. The reported stratified scenes are used only for evaluation and are not used during training or hyperparameter selection. All evaluations use the valid pixel mask provided by the benchmark and exclude an 11-pixel image border. Unless otherwise stated, inference is performed at the native \(512\times512\) resolution with a single model and a single forward pass. No test-time ensembling, multi-model fusion, or post-processing is applied.

6.3 Additional HCI Results↩︎

In addition to the four HCI stratified scenes used in the main paper, we evaluate FreqLF+ on the 16-scene HCI additional split. Table 7 reports per-scene MSE\(\times10^2\) and BadPix\((0.07)\). These results provide broader coverage across different textures, geometries, and occlusion conditions.

8pt

The additional split shows that FreqLF+ performs strongly on several scenes with clean geometry and moderate texture, such as Boardgames, Pillows, Tomb, and Platonic. The largest errors occur on Kitchen and Vinyl, which contain challenging specular, semi-transparent, or out-of-distribution appearance effects. Moderate errors occur on Rosemary, Museum, and Greek, which contain fine-scale or high-frequency geometry. These results indicate that FreqLF+ generalizes well across many HCI scenes, while complex materials and difficult occlusion patterns remain challenging.

6.4 Ablation and Diagnostic Protocols↩︎

This section provides additional details on the ablation and diagnostic protocols used to interpret the Fourier-local design. These analyses are intended to clarify why both branches are included and how the diagnostic masks are constructed.

6.4.0.1 Component ablations.

The component ablation in the main paper isolates the effect of the Fourier branch, local CNN branch, SE fusion, and probabilistic head. All ablated variants are trained from scratch using the same optimizer, learning rate, number of epochs, data processing, and evaluation protocol. Only the component under study is changed. This keeps the comparison focused on the contribution of each module.

6.4.0.2 Interpretation of the Fourier branch.

The Fourier branch is designed to update the latent field using low-frequency modes with global spatial support. This is useful for disparity regions that require long-range consistency, such as smooth surfaces, planar structures, and textureless areas where local evidence can be weak. Removing the Fourier branch increases the average error, indicating that global low-mode interaction improves the latent field beyond purely local processing.

6.4.0.3 Interpretation of the local branch.

The local \(3\times3\) branch is designed to preserve and refine spatial variations. This is especially important near occlusion boundaries, thin structures, and abrupt foreground-background transitions. Removing the local branch leads to the largest degradation in the ablation study, which confirms that low-mode Fourier updates alone are insufficient for accurate disparity estimation. The full model therefore combines global Fourier-domain interaction with spatially localized updates.

6.4.0.4 Fourier bandwidth.

The spectral bandwidth study varies the retained Fourier modes \(k_{\max}\). Small bandwidths provide strong smoothing but can under-represent spatial detail. Larger bandwidths increase the number of retained frequency components, but they do not necessarily improve validation accuracy. In our experiments, \(k_{\max}=16\) gives the best balance between global interaction, local detail preservation, and computational cost.

6.4.0.5 Boundary mask definition.

Boundary diagnostics are computed from the ground-truth disparity map. Let \(d\) denote the ground-truth disparity and let \[G(\mathbf{q}) = \sqrt{ \left(\partial_x d(\mathbf{q})\right)^2 + \left(\partial_y d(\mathbf{q})\right)^2 }\] denote the disparity-gradient magnitude. Boundary pixels are selected as the top \(10\%\) of valid pixels according to \(G(\mathbf{q})\). These pixels approximate high-frequency disparity transitions such as object boundaries and foreground-background discontinuities.

6.4.0.6 Textureless mask definition.

The HCI benchmark does not provide official textureless-region annotations. We therefore define textureless regions diagnostically from the central view. We convert the central view to intensity and compute the local intensity variance in a \(7\times7\) window. Textureless pixels are selected as the bottom \(15\%\) of valid pixels according to this local variance. These pixels approximate regions where local appearance cues are weak and angular or global field information becomes more important.

6.4.0.7 Use of diagnostic masks.

The boundary and textureless masks are not official HCI semantic labels. They are used only as diagnostic tools to separate two common light-field disparity failure modes: high-frequency discontinuities and low-texture regions. The masks help evaluate whether the Fourier-local design behaves as intended. The Fourier branch is expected to support globally consistent estimates in weakly textured or smooth regions, while the local branch is expected to improve boundary and thin-structure behavior.

6.4.0.8 Region-specific quantitative results.

Table 8 reports region-specific MSE\(\times10^2\) and BadPix\((0.07)\) for FreqLF+ on the four HCI stratified scenes. Errors are substantially higher in boundary regions than in textureless regions. The mean boundary error is \(5.254\) MSE\(\times10^2\) and \(11.467\%\) BadPix\((0.07)\), while the mean textureless-region error is \(0.171\) MSE\(\times10^2\) and \(0.601\%\) BadPix\((0.07)\). This suggests that the model handles weak-texture regions well, while sharp disparity transitions remain the main source of residual error.

6pt

Backgammon and Dots have the highest boundary errors, reflecting the difficulty of sharp, high-contrast depth edges and sparse repeated structures. Pyramids has the lowest boundary error because its disparity changes are smoother and less dominated by abrupt occlusion boundaries. Textureless regions show consistently low error across all scenes, supporting the view that EPI-derived global field processing is effective when local appearance evidence is weak.

6.4.0.9 Frequency-domain error analysis.

For frequency-domain diagnostics, we compute the absolute disparity error map for each model variant and visualize its Fourier magnitude spectrum. This analysis is used to inspect whether errors are concentrated in low-frequency components, high-frequency components, or both. The Fourier-only variant tends to leave mid-to-high-frequency errors, while the CNN-only variant shows stronger low-frequency bias. The full model gives a more balanced error spectrum, supporting the complementary role of the Fourier and local branches.

6.5 Mathematical Interpretation of the Fourier-Local Update↩︎

FreqLF does not rely on a formal convergence guarantee or a theoretical optimality proof. However, the hybrid update can be interpreted as a decomposition of the latent-field update into a globally supported low-frequency component and a spatially local component. For a latent field \(\mathbf{z}^{\ell} \in \mathbb{R}^{C \times H \times W}\), each hybrid layer applies \[\mathbf{z}^{\ell+1} = \sigma\!\left( \mathcal{K}_{\ell}(\mathbf{z}^{\ell}) + \mathcal{C}_{\ell}(\mathbf{z}^{\ell}) + W^{1\times1}_{\ell}(\mathbf{z}^{\ell}) \right),\] where \(\mathcal{K}_{\ell}\) is the Fourier-domain branch, \(\mathcal{C}_{\ell}\) is the local convolutional branch, and \(W^{1\times1}_{\ell}\) is a pointwise residual projection.

Ignoring the nonlinearity and channel mixing for interpretation, the Fourier branch applies a learned update on a retained set of low-frequency spatial modes: \[\mathcal{K}_{\ell}(\mathbf{z}) = \mathcal{F}^{-1} \left( \mathbf{M}_{k} \odot \mathbf{R}_{\ell}\mathcal{F}(\mathbf{z}) \right),\] where \(\mathcal{F}\) and \(\mathcal{F}^{-1}\) denote the 2D Fourier transform and inverse transform over the spatial dimensions, \(\mathbf{M}_{k}\) is a binary mask selecting the retained modes, and \(\mathbf{R}_{\ell}\) denotes learned complex-valued mode weights with channel mixing. Since each Fourier basis function has global spatial support, this branch allows distant spatial locations to interact through the retained frequency components. Low-frequency modes are well suited to representing smooth field components such as large planar regions, global disparity trends, and textureless areas.

The local convolutional branch is given by \[\mathcal{C}_{\ell}(\mathbf{z}) = \mathrm{Conv}^{3\times3}_{\ell}(\mathbf{z}),\] and has compact spatial support. It therefore provides localized corrections that are important for high-frequency disparity structures, including occlusion boundaries, thin structures, and abrupt foreground-background transitions.

Thus, the hybrid layer can be viewed as a learned spectral-local update. The Fourier branch updates globally supported smooth components of the latent field, while the convolutional branch updates local spatial detail. This interpretation does not imply an exact frequency separation or a formal optimality guarantee, because both branches operate on learned latent features and are followed by nonlinearities. Instead, it motivates the architecture, while the ablation results empirically verify that both branches provide complementary contributions.

6.6 Uncertainty Diagnostics↩︎

The probabilistic decoder predicts a Gaussian mixture distribution over disparity at each query coordinate. These predictions are used to compute the final disparity estimate and to visualize uncertainty. We use uncertainty only as an auxiliary diagnostic signal, not as a calibrated confidence estimate.

For each query coordinate, the decoder predicts a \(K\)-component Gaussian mixture distribution, \[p(d\mid \mathbf{q}) = \sum_{k=1}^{K} \pi_k(\mathbf{q}) \mathcal{N} \left( d \mid \mu_k(\mathbf{q}), \sigma_k^2(\mathbf{q}) \right).\] The point disparity prediction is the mixture mean, \[\hat{d}(\mathbf{q}) = \sum_{k=1}^{K} \pi_k(\mathbf{q})\mu_k(\mathbf{q}).\] For visualization, we compute the predictive variance using the law of total variance: \[\mathrm{Var}(d\mid\mathbf{q}) = \sum_{k=1}^{K} \pi_k(\mathbf{q})\sigma_k^2(\mathbf{q}) + \sum_{k=1}^{K} \pi_k(\mathbf{q})\mu_k^2(\mathbf{q}) - \left( \sum_{k=1}^{K} \pi_k(\mathbf{q})\mu_k(\mathbf{q}) \right)^2 .\]

The uncertainty maps provide useful qualitative diagnostics, especially near boundaries, thin structures, and ambiguous regions. However, qualitative uncertainty maps alone do not imply calibrated confidence. We therefore additionally evaluate the relation between predicted variance and actual disparity error.

6.6.0.1 Uncertainty-error correlation.

Table 9 reports the Spearman and Pearson correlations between predicted variance and absolute disparity error. The pooled Spearman correlation is \(0.069\), and the pooled Pearson correlation is \(0.218\), indicating that predicted variance is only weakly correlated with the actual error. Scene-wise correlations are also inconsistent, including negative Spearman correlations for Backgammon and Stripes.

8pt

6.6.0.2 Coverage-based calibration.

We also evaluate coverage-based calibration. For each confidence level \(p\), we measure the fraction of pixels satisfying \[|\hat{d} - d| \leq z_p \hat{\sigma},\] where \(z_p\) is the corresponding Gaussian interval coefficient and \(\hat{\sigma}\) is the predicted standard deviation. Perfect calibration would produce empirical coverage close to the nominal confidence level. In our experiments, the empirical coverage curves lie above the ideal diagonal, indicating underconfidence: the predicted uncertainty overestimates the actual error spread.

8pt

Overall, these results show that the probabilistic decoder provides useful qualitative uncertainty maps, but the predicted variance should not be interpreted as a calibrated confidence estimate. Calibrated uncertainty estimation, including improved uncertainty-error alignment and calibration-error reduction, is left for future work.

6.7 Additional Qualitative Results↩︎

Figure 12 shows additional qualitative results. The visual examples include the central view, ground-truth disparity, predicted disparity, absolute error, and uncertainty map. These examples complement the quantitative results in the main paper by showing where errors concentrate spatially.

The qualitative results show that most errors occur near disparity discontinuities, repetitive structures, and ambiguous angular evidence. Pyramids is generally easier because the geometry is smooth and texture is regular, while Backgammon, Dots, and Stripes expose different failure modes: matching ambiguity, sparse repeated structures, and high-frequency stripe boundaries.

6.8 Computational Details and Complexity↩︎

6.8.0.1 Runtime and memory protocol.

Runtime and memory measurements are performed on a single NVIDIA A100 GPU with 80 GB HBM2e memory. For each input resolution, one warm-up pass is performed before measurement. The reported runtime is the mean over 10 repeated runs. Peak GPU memory is measured during inference. The same trained model is evaluated at different spatial resolutions.

6.8.0.2 Complexity.

Let \(N=HW\) denote the number of spatial pixels. The dense encoder and decoder scale approximately linearly with \(N\). In each hybrid operator layer, the FFT-based Fourier branch contributes an \(O(N\log N)\) term, while the local convolutional branch scales linearly in \(N\) for fixed channel count and kernel size. Thus, the dominant spatial scaling of the base FreqLF backbone is approximately \(O(N\log N)\). This scaling is one reason the base model remains tractable at resolutions beyond the native \(512\times512\) benchmark resolution.

6.8.0.3 Memory behavior.

The base FreqLF model avoids explicit disparity-volume construction and therefore does not allocate memory proportional to a dense set of candidate disparity hypotheses. Instead, its main memory usage comes from the spatial feature maps, latent field, FFT buffers, and decoder queries. FreqLF+ requires more memory because its enhanced front-end processes richer angular evidence before the same Fourier-local backbone and decoder are applied.

6.8.0.4 FreqLF+ overhead.

FreqLF+ improves accuracy by using a stronger occlusion-aware front-end. This front-end processes more angular evidence and substantially increases runtime and memory consumption. In the main paper, FreqLF+ is therefore interpreted as a modular enhanced variant rather than as the primary cost-volume-free model. The base FreqLF model is the variant used to support the cost-volume-free field-learning claim.

6.8.0.5 Runtime interpretation.

Reported runtimes for prior methods are taken from their original reports when available and may not be measured on identical hardware. Therefore, they should be interpreted only as indicative values. The controlled runtime and memory analysis for our own models is reported separately in the main paper.

6.9 Discussion, Scope, and Limitations↩︎

6.9.0.1 Scope of the cost-volume-free claim.

The base FreqLF model avoids explicit cost-volume construction: it does not enumerate candidate disparities and does not build a multi-view matching volume over discrete disparity hypotheses. Instead, it encodes angular geometry into EPI-derived spatial features and updates the resulting latent field through Fourier-local layers before coordinate-conditioned decoding. The cost-volume-free claim therefore refers to the base FreqLF model. FreqLF+ is included only as an enhanced front-end variant to test whether the proposed Fourier-local backbone benefits from stronger angular features.

6.9.0.2 Dependence on light-field calibration.

FreqLF relies on EPI-derived angular structure and therefore assumes that the light-field views are sufficiently calibrated and rectified so that EPI line-slope patterns provide reliable disparity cues. Inaccurate camera calibration, view misalignment, or non-uniform angular sampling can distort EPI geometry and may degrade performance. Cost-volume-based approaches can be more robust in such settings because they explicitly search over candidate disparities and aggregate matching evidence across views. In contrast, the base FreqLF model trades explicit disparity-volume construction for Fourier-local updates over EPI-derived latent fields. Improving robustness to imperfect calibration is an important direction for future work.

6.9.0.3 Why Fourier-local processing?

Light-field disparity contains both low-frequency and high-frequency structure. Smooth surfaces, planar regions, and textureless areas require long-range consistency, while occlusion boundaries, thin structures, and foreground-background transitions require local precision. The Fourier branch provides global interaction through retained low-frequency modes, whereas the local CNN branch preserves spatially localized detail. The ablation results support this design: removing the Fourier branch weakens global field modeling, while removing the local branch produces a substantially larger degradation because fine disparity detail is no longer sufficiently preserved.

6.9.0.4 Relation to Fourier Neural Operators.

FreqLF is inspired by the operator-learning view of Fourier-domain feature updates, but it is not simply a generic hybrid Fourier-local backbone applied to light-field data. The model is specifically conditioned on light-field geometry through horizontal and vertical EPI stacks. It combines EPI-derived angular features, central-view appearance features, local convolutional refinement, and coordinate-conditioned probabilistic decoding. The goal is therefore not only to apply a Fourier operator, but to use Fourier-local latent field learning as a disparity-specific alternative to explicit disparity-volume construction.

6.9.0.5 Relation to implicit neural representations.

FreqLF uses query coordinates in the decoder, but it should not be interpreted as a fully implicit continuous scene representation. The latent representation is computed on an \(H\times W\) spatial grid, and the decoder obtains features by bilinear sampling from this grid. Thus, FreqLF is a coordinate-conditioned decoder over a grid-based latent field. This design allows coordinate injection and probabilistic prediction while retaining efficient dense image-grid processing.

6.9.0.6 Role of the probabilistic decoder.

The probabilistic decoder is used to obtain a mixture-based disparity estimate and auxiliary uncertainty maps. The mixture mean is used as the final disparity prediction for all benchmark metrics. In this work, uncertainty is treated as a diagnostic signal rather than as a reliable calibrated confidence estimate. The predicted variance shows weak correlation with absolute error, and coverage-based calibration indicates underconfidence. More rigorous uncertainty modeling and calibration are therefore left for future work.

6.9.0.7 Limitations and future work.

FreqLF is primarily evaluated on the HCI 4D Light Field Benchmark, and broader validation on real-world light-field data remains future work. Real-world light-field data may introduce calibration errors, sensor noise, sparse angular sampling, non-Lambertian effects, and domain shifts that are not fully captured by the benchmark. Boundary regions and thin structures remain challenging, as indicated by the region-specific diagnostics, although the Fourier-local design improves performance on high-frequency scenes such as Stripes. FreqLF+ shows that stronger angular encoding can improve accuracy, but it also increases runtime and memory consumption. Future work will focus on boundary-aware refinement, improved robustness to imperfect light-field calibration, improved uncertainty calibration, more efficient enhanced front-ends, and validation on real-world light-field acquisitions.

References↩︎

---

1. Corresponding author: sara.monjiazad@medma.uni-heidelberg.de↩︎