1 Introduction↩︎

3D reconstruction of the environment and motion estimation are essential for real-time localization in robotics, autonomous vehicles, drones, virtual reality, and augmented reality. While lidars, radars, GPS, and inertial sensors are widely used, camera-based slam and vo have become widely adopted for mapping and localization. Passive cameras offer high resolution, rich visual information, versatility, and a cost-effective, lightweight, and compact sensing solution. However, monocular cameras still cannot recover absolute scale without additional information.

Alternatives such as stereo, RGB-D, and ToF cameras address the scale ambiguity of monocular vision but remain constrained by the stereo baseline, the structured light range, or the trade-off between range and accuracy. By placing a mla between the main lens and the sensor of a camera, a plenoptic camera extends conventional imaging by simultaneously capturing spatial and angular information of the scene. This produces multiple viewpoints in the form of micro-images, also offering a very wide depth of field. This makes them well suited for true-to-scale robust slam in compact systems.

To exploit this advantage, we propose PRISM (Plenoptic Reconstruction via Inverse-depth Sparse Mapping) visual odometry, a new pure optimization-based photometric vo approach for focused plenoptic cameras. It combines disparity between micro-images for absolute scale estimation and larger baseline from temporal disparity to improve tracking accuracy and robustness. It enables reliable sparse metric reconstruction (see 1), which is not possible with a standard monocular camera. Our main contributions are:

• A sparse and photometric plenoptic bundle adjustment formulation that jointly optimizes camera poses and environment points.

• Tight integration of the plenoptic camera model into front-end tracking and back-end bundle adjustment of the vo pipeline.

• Depth refinement using variance-weighted plenoptic depth residuals combined with temporal optimization.

We extensively evaluate the method on several datasets containing various types of scenes and different cameras. We demonstrate excellent performance compared to state-of-the-art methods by reducing the errors accumulated along the trajectory while providing a metric scale.

2 Related Work↩︎

This section reviews classical slam, plenoptic and light-field methods, and prior sfm and vo approaches using plenoptic cameras.

2.1 Classical slam Approaches↩︎

Early visual slam systems relied on feature-based approaches (also called indirect methods), such as PTAM [2] and ORB-SLAM [3], which estimate camera motion by tracking sparse keypoints across frames. Later, direct methods such as DTAM [4] and LSD-SLAM [5] bypassed feature extraction by optimizing over image intensities. DSO [6] further improved robustness due to photometric bundle adjustment. Methods such as DROID-SLAM [7] rely on deep learning and achieve state-of-the-art accuracy through end-to-end optimization of camera poses and dense depth. More recently, visual–inertial slam frameworks such as DM-VIO [8] have shown real-time performance in challenging environments.

Despite their success, these methods rely on pinhole or fisheye camera models. When applied to plenoptic cameras, their geometric and radiance assumptions no longer hold, motivating the need for adapted formulations.

2.2 Plenoptic and Light-Field Vision↩︎

We distinguish two main configurations of plenoptic cameras: unfocused (plenoptic camera 1.0) and focused (plenoptic camera 2.0).

Prior works studied unfocused plenoptic cameras, where the main lens is focused on the mla. The mla is itself focused at infinity. Calibration methods typically reconstruct sub-aperture images to detect features [9]–[11] or detect features directly in the micro-images [12]–[14]. However, they are less used today due to low spatial resolution.

Focused plenoptic cameras improve the trade-off between spatial and angular resolution by focusing micro-lenses on the main lens image plane [15], [16]. Subsequent work has established new light-field models to estimate all intrinsic, extrinsic, and scene parameters from reconstructed images [17]–[19] or directly from raw images [20]–[22]. To eliminate the need for a calibration target, [23], [24] use sub-aperture views, and LiFCal [25] allows recalibration on any scene.

Several methods exploit light-field data for depth estimation within a single shot [26], [27]. In addition, applications such as post-capture refocusing [28], super resolution [29] or view synthesis [30] highlight the versatility of light-field data.

2.3 sfm and vo with Plenoptic Cameras↩︎

Few works directly integrate plenoptic cameras into vo/slam. Early approaches proposed closed-form 6-DOF VO [31] or multi-scale motion estimation for low-resolution plenoptic vision [32], while [33] introduced a light-field front-end for indirect slam. However, these rely on custom setups rather than true plenoptic cameras. SPO, A semi-dense direct plenoptic odometry method, introduced in [19] and improved in [1], operates on micro-images and recovers metric scale from the light-field geometry. Nevertheless, this method is limited to projecting points from the penultimate image onto the last image without performing a bundle adjustment across multiple frames, which makes it less robust and more sensitive to drift. [34] relies on unsupervised learning for depth estimation and vo but is tailored to sparse light-field cameras. An indirect vo framework leveraging light-field cameras is introduced in [35], which enables metric-scale translation estimation and simplifies calibration from a novel keypoint extraction. However, this approach requires an additional step of feature detection depending on textured scenes. The methods in [36] and [25] also perform reconstruction but operate as SfM pipelines without considering temporal links between images.

Overall, light-field sensing shows strong potential for robust tracking and scale estimation. However, existing approaches target different camera setups, require training, or rely on highly textured environments.

3 Preliminaries↩︎

We first provide an overview of the plenoptic camera model used in the vo pipeline (3.1). Then, we introduce the notation used in the paper (3.2).

3.1 The Focused Plenoptic Camera↩︎

A plenoptic camera extends the conventional imaging model by capturing not only the intensity of light rays on the sensor but also their directions within the optical system by placing an mla between the main lens and the sensor.

Throughout the paper, we consider a focused plenoptic camera in Galilean mode and base our method on the plenoptic camera model introduced in [25], modeling the main lens as a thin lens and the micro-lenses as pinholes. The plenoptic camera model is defined by key geometric parameters (see 2): \(f_L\) (main lens focal length), \(b_{L0}\) (distance from main lens to MLA), and \(B\) (distance from MLA to sensor). The main lens forms a virtual image of the scene at a distance \(b_L\) from the main lens and \(b\) from the mla. The virtual depth \(v\), introduced in  [37] as \(v=\frac{b}{B}\), maps the distance \(z_C\) between the real 3D point in camera metric coordinates \(\mathbf{x}_C=[x_C, y_C, z_C]^T\) in the 3D virtual image space \(\mathbf{x}_V=[x_V, y_V, z_V]^T\). This projection is denoted as \(\Pi_{\text{pl}}(\cdot)\). The virtual depth allows a depth map to be generated without prior metric calibration of the camera. The main lens principal point is denoted as \(\mathbf{c}_L=[c_x,c_y]^T\) in pixels.

Each micro-lens forms a small image on the sensor (see 3 (a)). Neighboring micro-images represent the same points from slightly different perspectives. We rely on the depth-estimation pipelines of [38] and [25] to compute virtual depth maps (3 (c)), the so-called totally focused image (3 (b)) created by the main lens and the virtual depth uncertainty maps (3 (d)) from a single raw plenoptic image. More details about the camera model are provided in [25] and in the supplementary material.

3.2 Notation↩︎

In the remainder of this document, vectors are denoted in bold lowercase letters (\(\mathbf{x}\)), matrices in bold uppercase letters (\(\mathbf{J}\)) and scalars in non-bold letters (\(v\)). Indexed quantities use subscript \(i\) (\(i \in \{1,\dots,n\}\)) for all indexed variables, including scalars (\(v_i\)) and vectors (\(\mathbf{x}_{C,i}\)).

4 PRISM-VO: Plenoptic Direct Visual Odometry↩︎

We first give an overview of the pipeline in 4.1, followed by the main components: the multi-scale image representation (4.2), the point selection (4.3), and the plenoptic bundle adjustment formulation (4.4).

4.1 Method Overview↩︎

PRISM-VO uses a novel sparse and photometric plenoptic bundle adjustment formulation within a keyframe-based front-end/back-end architecture. It integrates the plenoptic camera model into both tracking and mapping (4). This tight coupling is required because the plenoptic camera model differs fundamentally from pinhole stereo/RGB-D models.

Each input raw image is processed into a totally focused image, virtual depth map, and virtual depth uncertainty map in a similar way as in [38] and [25]. The totally focused image drives photometric tracking, while depth and uncertainty provide geometric priors for both front-end tracking and back-end optimization.

The front-end estimates camera pose via direct image alignment against reference keyframes. To increase robustness and convergence, it uses a coarse-to-fine approach. During initialization, points are selected according to their gradient and their depth information. Keyframes are added based on field of view changes, occlusions, and exposure variations in a manner inspired by [6].

The back-end maintains an active window of keyframes. As new keyframes are added, depth hypotheses are refined by combining multi-view photometric consistency with plenoptic depth cues. Camera poses and scene geometry are jointly optimized through a novel sparse photometric plenoptic bundle adjustment. Older keyframes are then marginalized, with their information retained in the optimization via prior constraints.

4.2 Multi-Scale Image Representation↩︎

To enable robust multi-scale processing, we construct image pyramids for the totally focused image, virtual depth map, and virtual depth uncertainty map. Each pyramid level is half the resolution of the previous level in both dimensions.

Totally Focused Image Pyramid. The totally focused image pyramid is built by averaging \(2\times2\) pixel blocks from the previous level.

Virtual Depth Map Pyramid. Since the inverse virtual depth is proportional to the micro-image disparity, one can assume it to be normally distributed. Virtual depth maps are downsampled via variance-weighted averaging, giving more weight to reliable measurements and ignoring invalid pixels. The inverse virtual depth \(\bar{\rho}_{v,i}\) at the coarser level is computed with 1 , where \(N_x^{(i)}\) denotes the neighborhood pixels with valid inverse virtual depth \(\rho_{v,i}\) and the corresponding variance \(\sigma_{\rho_{v,i}}^2\).

\[\bar{\rho}_{v,i+1} = \frac{\sum_{k \in N_x^{(i)}} \rho_{v,k} \cdot \left( \sigma_{\rho_{v,k}}^2 \right)^{-1}}{\sum_{k \in N_x^{(i)}} \left( \sigma_{\rho_{v,k}}^2 \right)^{-1}}, \qquad \bar{\sigma}_{\rho_{v,i}}^2 = \frac{\left|N_x^{(i)}\right|}{\sum_{k \in N_x^{(i)}} \left( \sigma_{\rho_{v,k}}^2 \right)^{-1}} \label{eq:rho95bar95and95sigma95rho}\tag{1}\]

Virtual Depth Uncertainty Map Pyramid. The inverse virtual depth variance \(\bar{\sigma}_{\rho_{v,i}}\) at coarser levels is computed using a harmonic mean over valid child pixels with 1 , where \(\left|N_x^{(i)}\right|\) is the cardinal of \(N_x^{(i)}\). This approach ensures that regions with high uncertainty are appropriately reflected in coarser levels, assuming a strong correlation between depths of neighboring pixels.

4.3 Point Selection↩︎

Adaptive Depth Cutoff. We compute an adaptive inverse depth cutoff based on the mean inverse virtual depth \(\rho_{v,\text{av}}\) across valid points to favor points near the camera with lower variance. Using a normalized weighting factor \(w_\text{cut} \in [0,1]\), we define a percentile \(p_a\) with 2 between \(p_{\min}\) and \(p_{\max}\) that reduces the influence of distant outliers for reliable depth selection (see 5). The values \(\rho_{v,\text{n}}\) (near) and \(\rho_{v,\text{f}}\) (far) represent reference inverse virtual depth bounds.

\[\begin{align} p_{\text{a}} = p_{\min} + w_\text{cut} \cdot (p_{\max} - p_{\min}), \qquad \text{with} \qquad w_\text{cut} = \frac{\rho_{v,\text{av}} - \rho_{v,\text{f}}}{\rho_{v,\text{n}} - \rho_{v,\text{f}}} \label{eq:p95a} \end{align}\tag{2}\]

Point Penalization. The gradient-based score \(s_{\text{grad}}(\mathbf{x}_{V})\) of a point \(\mathbf{x}_{V}\) is defined from the totally focused image gradient \(\nabla I(\mathbf{x}_{V})\) and a randomly sampled unit direction \(\mathbf{d}\) to encourage orientation diversity. The score of a pixel \(\mathbf{x}_{V}\) containing valid depth is weighted more heavily by a constant factor \(w_d(\mathbf{x}_{V})\) than those without depth information. The final selection score used for ranking candidate pixels is therefore computed with 3 . Points without depth data are also selected (but with a lower weight \(w_d(\mathbf{x}_{V})\)) to ensure good distribution of points across the image. 5 illustrates the resulting point selection. This approach favors reliable nearby points while ensuring coverage in regions with noisy, missing, or distant depth.

\[s(\mathbf{x}_{V}) = \left| \nabla I(\mathbf{x}_{V})^\top \mathbf{d} \right| \, w_d(\mathbf{x}_{V}) \label{eq:selection95score}\tag{3}\]

4.4 Optimization Formulation↩︎

The state vector, including camera poses, inverse depths, and brightness parameters (to account for exposure changes), is estimated by minimizing a joint nonlinear least-squares problem. A point \(\mathbf{x}_{V,i}\) in the host frame \(i\) is projected with 4 to a point \(\mathbf{x}_{V,j}\) in a target frame \(j\) using the plenoptic camera model introduced in 3.1. \(\mathbf{R},\mathbf{t} \in SE(3)\) denote the relative pose and \(\Pi_{\text{pl}}(\cdot)\) denotes the plenoptic projection (see supplementary material for details).

\[\begin{align} \mathbf{x}_{V,j} = \Pi_{\text{pl}}\left(\mathbf{R} \cdot \Pi_{\text{pl}}^{-1}\left(\mathbf{x}_{V,i} \right) + \mathbf{t}\right) \label{eq:projection} \end{align}\tag{4}\]

The photometric residual \(r^{\text{photo}}\) for a point \(\mathbf{x}_{V,i}\) in a reference frame \(I_i\) observed in a target frame \(I_j\) is defined in 5 . The affine brightness parameters \(a_i\), \(a_j\), \(b_i\) and \(b_j\) are inspired by [6].

\[r^{\text{photo}} = \left( I_j \left[ \Pi_{\text{pl}}(\mathbf{x}_{V,i}) \right] - b_j \right) - \frac{e^{a_j}}{e^{a_i}} \left( I_i[\mathbf{x}_{V,i}] - b_i \right) \label{eq:r95photo}\tag{5}\]

The depth residual is formulated in the inverse virtual depth space as defined in 6 . Here, \(\rho_v\) is the inverse virtual depth resulting from the projection and \(\rho_v^{\text{meas}}\) the corresponding measured value from the depth map.

\[r^{\text{depth}} = \rho_v - \rho_v^{\text{meas}} \label{eq:r95depth}\tag{6}\]

The total energy is given by

\[E = \sum_k w_k^{\text{photo}} \, \| r_k^{\text{photo}} \|_\gamma + \eta \sum_l w_l^{\text{depth}} \, \| r_l^{\text{depth}} \|_\gamma, \label{eq:energy95residuals}\tag{7}\]

where \(w_k^{\text{photo}}\) and \(w_l^{\text{depth}}\) are per-residual weights. The quadratic error of the inverse virtual depth can be minimized as it is normally distributed. The inverse virtual depth residual is weighted by the inverse virtual depth variance, resulting in \(w_l^{\text{depth}} = \left( \sigma_{\rho_{v,l}}^2 \right)^{-1}\), so more reliable depth measurements have greater influence. The symbol \(\|\cdot\|_\gamma\) denotes the Huber norm, and \(\eta\) balances the photometric and depth residual terms.

Adaptive Cross-Modal Weighting. To balance photometric and depth residuals in the joint optimization, we introduce a scalar weight \(\eta\). It is adapted online by matching the average Gauss–Newton curvature, determined by \(\mathbf{J}^T\mathbf{J}\), of both residual types with 8 .

\[\eta = c \cdot \frac{\sum_i \| J_i^{\text{photo}} \|_2^2}{\sum_j \| J_j^{\text{depth}} \|_2^2} \quad \text{with} \quad J_i^{\text{photo}} = \frac{\partial r_i^{\text{photo}}}{\partial \rho_{C,i}} \quad \text{and} \quad J_i^{\text{depth}} = \frac{\partial r_j^{\text{depth}}}{\partial \rho_{C,j}} \label{eq:eta95weight}\tag{8}\]

Here, \(c\) is a constant factor and \(J_i^{\text{photo}}\) and \(J_j^{\text{depth}}\) are respectively the Jacobian contributions of the photometric and the inverse virtual depth residuals related to the inverse point depths.

For temporal stability, the weight \(\eta\) is updated with

\[\eta_{i+1} = \exp\!\left( (1-\alpha)\log \eta_i + \alpha \log \eta \right), \label{eq:eta95kp1}\tag{9}\]

using a fixed sensitivity parameter \(\alpha\) and exponential smoothing in the logarithmic domain to decrease the weight for older values. This results in a scene-adaptive balance between photometric and depth constraints without manual tuning as shown in 6. Since inverse depth is more sensitive to nearby points due to stronger parallax, scenes dominated by close objects increase the scaling factor, giving depth residuals greater influence (see 6).

Levenberg–Marquardt Optimization. The nonlinear problem is solved with a Levenberg–Marquardt optimization in a sliding window, jointly optimizing camera poses \(\boldsymbol{\xi}\) in the tangent space, affine brightness parameters \(a\), \(b\), and inverse depths \(\rho_{C}\). At each iteration, all residuals are linearized around the current state \(\mathbf{s}\) as

\[r(\mathbf{s} + \delta \mathbf{s}) \approx r(\mathbf{s}) + \mathbf{J} \, \delta \mathbf{s},\]

where \(\mathbf{J}\) stacks the photometric and depth Jacobians (see the supplementary material for the complete derivation). The inverse virtual depth residual \(r^{\text{depth}}\) depends only on inverse depth and does not depend on \(a\), \(b\) or the pose \(\boldsymbol{\xi}\). The overall structure of the Hessian matrix is therefore preserved, making the integration very efficient to implement. Stacking all residuals yields the 10 , where \(\mathbf{H}\) is the approximate Gauss-Newton Hessian, \(\mathbf{b}\) the gradient vector and \(\mathbf{W}\) the weight matrix. The Schur complement is used to efficiently calculate the increment of the state vector by taking advantage of the sparsity of \(\mathbf{H}\) and to marginalize keyframes that leave the optimization window, similar to [6].

\[(\mathbf{H} + \mu \mathbf{I}) \, \delta \mathbf{s} = - \mathbf{b}, \qquad \text{with} \qquad \mathbf{H} = \mathbf{J}^\top \mathbf{W} \mathbf{J}, \qquad \mathbf{b} = \mathbf{J}^\top \mathbf{W} \mathbf{r} \label{eq:gauss95newton}\tag{10}\]

5 Evaluation↩︎

We first qualitatively demonstrate the advantages of the plenoptic camera. Then, we evaluate PRISM-VO against plenoptic-specific methods and state-of-the-art monocular vo/slam pipelines and perform an ablation study of its key components. Our experiments rely on two plenoptic datasets with synchronized multi-sensor data and ground truth.

• A Synchronized Stereo and Plenoptic Visual Odometry Dataset [39]. It consists of 11 indoor and outdoor sequences acquired with a Raytrix R5 plenoptic camera and a synchronized stereo camera pair, featuring large loops and depths up to several hundred meters for drift and scale evaluation.

• LiFMCR Dataset [40]. It contains 7 scenes recorded with two high-resolution Raytrix R32 plenoptic cameras, providing a precise 6-DoF ground truth from a Vicon system for precise pose accuracy assessment.

These datasets provide complementary scenarios for assessing long-term drift, scale consistency, and pose accuracy. Qualitative 3D reconstructions are shown in 7 and 8 for both datasets respectively.

5.1 Qualitative Results Against Other Sensors↩︎

To demonstrate the benefits of the plenoptic technology, we recorded a sequence using a synchronized plenoptic-monocular rig consisting of an R32 Raytrix camera and a Basler acA1920-40gc camera. As shown in 9, DSO and ORB-SLAM3 fail in a challenging fan scene, despite a wide field of view, due to parallax at intersecting grilles, transparent fan blades, and reflective surfaces. DPVO remains stable but cannot recover metric scale, unlike PRISM-VO.

We also captured the same scene from the same viewpoint using the R32 Raytrix camera and an RGB-D camera (Intel RealSense D455). While the RGB-D camera produces unstable depth, the plenoptic system succeeds (see 10).

5.2 Quantitative Drift Study↩︎

To evaluate the accumulated drift over the complete vo pipeline, we rely on the dataset from [39]. Each sequence starts and ends at the same location, forming a large loop, allowing a quantitative assessment of accumulated drift compared to the ground truth obtained by loop closure. Following [39], we estimate two similarity transformations, \(\mathbf{T}_{\mathrm{start}}^{\mathrm{gt}}\) (Eq. 11 ) and \(\mathbf{T}_{\mathrm{end}}^{\mathrm{gt}}\) (Eq. 12 ), aligning estimated 3D points \(\mathbf{x}_{C,i} \in \mathbb{R}^3\) and ground-truth points \(\mathbf{x}_{C,i}^{\mathrm{gt}} \in \mathbb{R}^3\) at the beginning and end of each sequence respectively. The start segment is defined by the index set \(S\), and the end segment by the index set \(E\).

\[\mathbf{T}_{\mathrm{start}}^{\mathrm{gt}} := \arg \min_{\mathbf{T} \in \mathrm{Sim}(3)} \sum_{i \in S} \left\| \mathbf{T} \mathbf{x}_{C,i} - \mathbf{x}_{C,i}^{\mathrm{gt}} \right\|_2^2 \label{eq:Tstart}\tag{11}\]

\[\mathbf{T}_{\mathrm{end}}^{\mathrm{gt}} := \arg \min_{\mathbf{T} \in \mathrm{Sim}(3)} \sum_{i \in E} \left\| \mathbf{T} \mathbf{x}_{C,i} - \mathbf{x}_{C,i}^{\mathrm{gt}} \right\|_2^2 \label{eq:Tend}\tag{12}\]

From \(\mathbf{T}_{\mathrm{start}}^{\mathrm{gt}}\) and \(\mathbf{T}_{\mathrm{end}}^{\mathrm{gt}}\), we compute the accumulated drift transformation \(\mathbf{T}_{\mathrm{drift}} \in \mathrm{Sim}(3)\) over the entire trajectory with 13 .

\[\mathbf{T}_{\mathrm{drift}} := \begin{bmatrix} e_s \mathbf{R} & \mathbf{t} \\ 0 & 1 \end{bmatrix} = \mathbf{T}_{\mathrm{end}}^{\mathrm{gt}} \left( \mathbf{T}_{\mathrm{start}}^{\mathrm{gt}} \right)^{-1} = \begin{bmatrix} s_e \mathbf{R}_e & \mathbf{t}_e \\ 0 & 1 \end{bmatrix} \begin{bmatrix} s_s \mathbf{R}_s & \mathbf{t}_s \\ 0 & 1 \end{bmatrix}^{-1} \label{eq:Tdrift}\tag{13}\]

For evaluation, we define the following metrics: absolute scale error \(d_s = \sqrt{s_e \cdot s_s}\), scale drift \(e_s = \frac{s_e}{s_s}\), rotation error \(e_r\) defined as the rotation angle around the Euler axis associated with the rotation matrix \(\mathbf{R} \in \mathrm{SO}(3)\) and a combined alignment error \(e_{\mathrm{align}}\) captures overall trajectory inconsistency (14 ). To facilitate interpretation, we define \(d_s' := \max\{d_s,\; d^{-1}_s\}\) and \(e_s' := \max\{e_s,\; e^{-1}_s\}\).

\[e_{\mathrm{align}} := \sqrt{ \frac{1}{N} \sum_{i=1}^{N} \left\| \mathbf{T}_{\mathrm{start}}^{\mathrm{gt}} \mathbf{x}_{C,i} - \mathbf{T}_{\mathrm{end}}^{\mathrm{gt}} \mathbf{x}_{C,i} \right\|_2^2 } \label{eq:ealign}\tag{14}\]

PRISM-VO is first compared against the plenoptic vo method SPO [1] using the dataset introduced in [39]. 1 reports the number of sequences that satisfy high, medium, and coarse precision thresholds for the different metrics. This evaluation protocol is adopted because only per-sequence results are available for SPO. Both methods recover the absolute scale accurately (scale close to 1), with PRISM-VO attaining slightly better results with \(d_s' \leq 1.10\) on 8/11 sequences and \(d_s' \leq 1.30\) for 10/11 sequences. More notably, PRISM-VO clearly reduces accumulated rotation error (\(e_r < 4^\circ\) on 8/10 sequences) and lowers worst-case rotation. The scale drift is comparable to SPO with 7/11 sequences achieving \(e_s' \leq 1.05\). The alignment error is also consistently smaller, especially at high and coarse precision. Additionally, PRISM-VO succeeds on one sequence where SPO fails, indicating greater robustness and stability beyond pure accuracy metrics. These results indicate that PRISM-VO provides more accurate and robust pose estimation, particularly in terms of alignment and rotational consistency, demonstrating the benefits of the proposed bundle-adjustment formulation for globally consistent vo. Therefore, PRISM-VO establishes a new state-of-the-art in plenoptic camera-based vo.

Apart from PRISM-VO and SPO, no other plenoptic vo algorithms performing on this type of scene are available. Therefore, we also compare PRISM-VO with state-of-the-art monocular methods, namely DSO [6], ORB-SLAM3 [41] and DPVO [42]. For a fair comparison of the tracking performance, large-scale-loop-closure is disabled in ORB-SLAM3. For the comparison methods, we use the provided monocular images and crop them to match the field of view of the plenoptic images (while adapting the intrinsics to preserve the pinhole model) to be able to compare the methods. To mitigate non-deterministic effects, each pipeline is run 10 times per sequence, with aggregated results shown in 11. ORB-SLAM3 fails on all the sequences of [39], likely due to the limited field of view. However, PRISM-VO is able to reliably recover the absolute scale of the scene. Furthermore, PRISM-VO exhibits a consistently lower scale drift over the optimization-based method DSO and performs on par with the deep-learning-based approach DPVO. With respect to the overall alignment error, PRISM-VO exhibits a performance similar to that of DSO and is only slightly worse than DPVO, which likely benefits from learned geometric priors.

5.3 Quantitative Pose Estimation Error↩︎

We further evaluate PRISM-VO on the LiFMCR dataset [40] to assess generalization to other plenoptic cameras. Raw images are calibrated with LiFCal [25]. The dataset only contains plenoptic data. For comparison with conventional methods, we generate totally focused images using a central perspective projection on a common image plane through the main lens center. This emulates a pinhole camera model and allows standard monocular methods to be applied (see supplementary material).

Ground-truth poses for all camera views are provided by a Vicon motion capture system, enabling per-frame pose evaluation. We evaluate absolute pose errors over the full trajectory and report the rmse denoted as \(\mathrm{RMSE}_t\) and \(\mathrm{RMSE}_r\) for the translation and rotation errors respectively in ¿tbl:tab:abs95error95lifmcr?. PRISM-VO achieves an error of a few millimeters and less than three degrees on most sequences. Although the camera model is not exactly a pinhole model for monocular methods, the comparison gives a rough idea. Overall, PRISM-VO consistently achieves the lowest translational error, obtaining the best \(\mathrm{RMSE}_t\) on six out of seven sequences. Rotational accuracy is also competitive, achieving the best \(\mathrm{RMSE}_r\) in four sequences and the second-best results in most remaining cases. The results demonstrate the advantages for plenoptic cameras in close range scenes, where reliable depth can be measured.

5.4 Ablation Study↩︎

We assess PRISM-VO components through incremental ablation, isolating each contribution and measuring its impact on accuracy and drift reduction.

• (1) Pinhole camera model: Baseline using a central perspective camera model without plenoptic geometry.

• (2) Plenoptic depth initialization and model integration: Metric scale from the plenoptic depth and deep integration of the plenoptic camera model for the tracking and the bundle adjustment.

• (3) Full PRISM-VO algorithm: Adds an inverse virtual depth residual term to the optimization. Automatically balances depth residuals by their uncertainty, assigning higher influence to points with lower variance.

12 reports the cumulative plots of the used metrics for the ablation stages. Compared to the pinhole baseline (1), adding plenoptic depth initialization and explicit plenoptic projection (2) reduces scale drift and makes absolute scale observable, confirming the benefit of exploiting plenoptic data. The full PRISM-VO algorithm (3) further improves all metrics, most notably scale consistency and alignment, through variance-weighted inverse virtual depth residuals and adaptive residual balancing. This uncertainty-aware integration of plenoptic depth within bundle adjustment enhances global consistency and reduces accumulated drift. Overall, each component contributes positively, with the largest gains from jointly optimizing and probabilistically weighting depth information.

6 Conclusion↩︎

We presented PRISM-VO, a novel vo method that tightly integrates plenoptic camera measurements into a photometric bundle adjustment. By explicitly modeling the plenoptic geometry, the method jointly leverages photometric and depth information via an inverse virtual depth residual. Experiments show improved trajectory consistency and accurate metric scale on challenging indoor and outdoor sequences. PRISM-VO outperforms the leading plenoptic VO pipeline and matches or surpasses monocular optimization and learning-based methods while reliably recovering metric scale. These results highlight the potential of tightly coupled plenoptic sensing and optimization for robust pose estimation and metric reconstruction with a single camera, despite limitations such as reduced resolution, a smaller field of view, and limited long-range depth accuracy.

Acknowledgements↩︎

This research was partially funded by the Federal Ministry of Research, Technology and Space of Germany in its program "FH-Kooperativ".

References↩︎