1 Introduction↩︎

High-quality 3D scene reconstruction is one of the most important and challenging applications in computer vision. The accuracy of 3D reconstruction hinges on the quality of essential components, such as camera poses, RGB images, and depth maps. These elements are interconnected and influence each other; a failure in one component can adversely impact the others. For example, blurry RGB images or noisy depth maps can significantly impair camera pose estimation, which, in turn, degrades overall reconstruction quality. This challenge is particularly pronounced when input data is captured in low-light conditions or during rapid camera movement—common scenarios in everyday, casual capture. Such conditions result in degraded color and depth frames, leading to poorly estimated camera poses and reconstructions with blurry textures and distorted or smoothed geometry. Therefore, achieving high-quality 3D reconstruction requires clear RGB images and accurate depth maps.

The interdependence of these input components suggests an opportunity for compensating or restoring degraded elements. However, this process presents a chicken-and-egg problem, complicating the reconstruction further. For instance, accurate camera pose estimation requires sharp images, while deblurring motion-blurred images depend on reliable camera pose information. Although numerous methods [1], [2] exist for recovering sharp images from motion-blurred ones, they often rely on supervised learning models trained on specific datasets. Consequently, their deblurring performance can degrade significantly when faced with new cameras or motion blur scenarios that differ from the training data.

Recent works [3]–[6] have made advances in addressing these challenges by optimizing camera trajectories during exposure while simultaneously learning sharp RGB colors. This enables simultaneous camera pose estimation and image deblurring. However, these methods require an initial camera pose and, when using Gaussian Splatting [7], also need a sparse point cloud as input. This requirement limits their applicability, especially when severe motion blur prevents structure-from-motion (SfM) methods like COLMAP [8] from functioning effectively, making it impossible to start the optimization process.

To overcome these limitations, we introduce a robust approach for 3D scene reconstruction that effectively compensates for severe motion blur and addresses the challenges of camera pose estimation without relying on a precise initial pose. Our method leverages RGB-D inputs and incorporates both optical flow and depth information to align camera poses accurately, even in the presence of challenging motion blur and lighting conditions. We first perform a global alignment between consecutive frames using optical flow and the ICP algorithm, which enables effective local alignment of point clouds generated from the depth maps.

After this initial alignment, we refine the camera poses and 3D geometry by integrating them into a Gaussian Splatting pipeline. This approach allows us to initialize dense 3D Gaussians from depth maps, which we scale to ensure a detailed representation of the scene geometry. Our refinement process iteratively adjusts both camera poses and the positions of 3D Gaussians, achieving global consistency by minimizing a depth alignment loss that compares rendered depth maps with input depth measurements. Through this adjustment, we eliminate loop-closure artifacts and reduce the drift that often accumulates in traditional SLAM-based methods.

For scenes with significant motion blur, we further optimize the deblurring process by modeling the camera poses at the start and end of each frame’s exposure. This temporal modeling allows us to simulate the effects of motion during the exposure period, which we incorporate into our Gaussian Splatting framework. We minimize the difference between the observed motion-blurred image and the average of a set of sharp images rendered from multiple viewpoints along the exposure trajectory, resulting in more accurate, geometrically consistent deblurring. Our deblur loss function combines image alignment, structural similarity, and depth consistency, ensuring that the final reconstructions retain sharpness and fidelity to the original scene structure.

Through extensive evaluation on a newly constructed RGB-D dataset featuring extreme motion blur, we demonstrate that our approach significantly outperforms existing methods in both accuracy and robustness. By addressing the key challenges of pose estimation, depth alignment, and image deblurring, our method provides a versatile and effective solution for high-quality 3D reconstruction under real-world capture conditions, such as low light and fast motion. The proposed method holds promising applications in areas requiring reliable 3D mapping and reconstruction, including robotics, autonomous navigation, and augmented reality. We will make both our dataset and implementation publicly available to foster further research and development in this field.

2 Related Work↩︎

A popular approach to 3D geometry reconstruction defines a 3D voxel grid and constructs a signed distance function (SDF) volume by accumulating depth estimates captured by sensors. KinectFusion [9] uses a Kinect camera to capture RGB-D images, then estimates relative camera poses between adjacent frames with the ICP algorithm. Next, it constructs and updates a truncated signed distance function (TSDF) volume, enabling real-time 3D reconstruction of objects. Niessner et al. [10] improve memory efficiency by combining hashing algorithms to store and update SDF values only where needed, thus reducing memory requirements.

Several methods [11]–[14] adopt memory-efficient, hierarchical data structures for depth fusion. BundleFusion [15] simultaneously refines camera poses and geometry using bundle adjustment, improving reconstruction accuracy and addressing the loop closure problem caused by accumulated camera pose and depth estimation errors. ElasticFusion [16] builds a map with surfels—point primitives containing position, normal, and radius—and updates the map through a deformation graph to minimize error and perform loop closure. Other methods [17]–[19] directly fuse depth estimates into point clouds, optimizing memory use. However, these methods generally require slow camera motion to capture sharp images and accurately estimate camera transformations.

Simultaneous localization and mapping (SLAM) methods [20]–[24] track camera motion and construct a map in real-time by extracting visual features from input images to establish 3D point correspondences. CodeSLAM [25] and DeepTAM [26] use neural networks to combine depth maps with color images, while Azinović et al. [27] train multi-layer perceptrons (MLP) to learn SDF and color values for scene reconstruction. Their approach also optimizes camera transformation correction variables during training to refine poses. However, these methods struggle with input images that contain motion blur. Nice-SLAM [28], Point-SLAM [29], and DROID-SLAM [30] use neural networks to track camera poses. SplaTAM [31] and MonoGS [32] introduced SLAM solutions based on 3D Gaussian Splatting. However, these methods are not designed to handle motion blur.

Motion blur occurs when camera motion during the exposure time changes the projected pixel coordinates of rays on the camera sensor. Early studies [33]–[35] tackle this issue by developing kernels to restore blurry images. Convolutional neural networks (CNNs) demonstrate powerful performance in learning deblur kernels [36], [37], and deep learning techniques such as ResNet with skip connections and multi-scale networks further improved restoration quality [38], [39]. SRN-DeblurNet [1] adopt recurrent networks [40], [41] and combine them with an encoder-decoder network structure, while NAFNet [2] simplify the network structure, extracting only essential components and proposing a nonlinear activation-free network. However, since these methods rely on curated datasets like the GoPro dataset [39], they struggle to effectively deblur images with extreme motion blur or those captured on different cameras.

There are NeRF-based approaches [42] that tackle motion blur in input images. Deblur-NeRF [43] and DP-NeRF [44] use a set of motion-blurred images along with camera poses estimated by COLMAP as input to restore sharp images through NeRF optimization. BAD-NeRF [4] enhances this by jointly optimizing virtual camera poses and radiance fields during exposure. The latest approaches utilize Gaussian Splatting [7], which offers faster training times than NeRF by rendering a set of 3D Gaussians with a dedicated rasterizer rather than optimizing a neural network. Gaussian Splatting-based deblurring methods [3], [5], [6] optimize the virtual camera trajectory and minimize differences between input blurry images and the average of rendered images at each virtual camera position.

All of these methods rely on initial camera poses, typically obtained from COLMAP, and for Gaussian Splatting-based methods, a sparse point cloud as well. However, structure-from-motion methods like COLMAP often fail with blurry inputs—such as images captured by fast-moving cameras or in low-light conditions that require long exposure times. Even with dense depth inputs, ICP requires acceptable global alignment, and recent point cloud matching methods [45], [46] cannot accurately determine relative camera transformations between two point clouds if they consist mainly of planar structures.

To address these issues, we propose an effective camera pose estimation algorithm tailored for RGB-D image sequences that jointly refines both the camera trajectory and the reconstructed 3D point cloud, resulting in improved sharpness in 3D reconstruction.

3 Method↩︎

In datasets with significant motion blur, using COLMAP to estimate camera poses becomes impractical. Additionally, rapid camera motion and the limited field of view of depth sensors further complicate this task, making ICP registration less effective. To address these challenges, we first perform global alignment using both RGB and depth images, which establishes sufficient initial alignment for the ICP algorithm to operate effectively. After achieving this global alignment, we apply ICP again to refine and accurately estimate the camera poses. However, estimating camera poses by analyzing only two consecutive frames in a sequence can result in accumulated errors, leading to drift over time. To resolve this issue, it is crucial to adjust the 3D geometry and align the camera poses across the entire sequence so that depth maps and camera views from all perspectives remain well-aligned. We accomplish this through bundle adjustment, optimizing the positions of 3D Gaussians and camera poses within the Gaussian Splatting pipeline. Finally, to account for motion blur, we approximate the camera trajectory over time and optimize the movement during exposure, enabling effective deblurring. See Figure 1 for an overview.

3.1 Local Pose Estimation↩︎

To estimate relative camera transformations, we select consecutive frames from the input RGB-D image sequence. Our approach begins with global alignment using optical flow, followed by local alignment of the two-point clouds using the ICP algorithm. For each consecutive RGB image and depth map pair at times \(t\) and \(t+1\), we optimize the transformation \(\boldsymbol{\xi} \in \mathfrak{se}(3)\) between the frames. Here, \({\boldsymbol{\xi} _{t \to t + 1}} = {[{\mathbf{x}^{\intercal}},{\boldsymbol{\omega} ^{\intercal}}]^{\intercal}}\), where \(\mathbf{x}\) represents the translation vector between the two camera origins, and \(\boldsymbol{\omega}\) is the rotation element of \(\mathfrak{se}(3)\). We ultimately calculate a camera-to-world transformation matrix \(\boldsymbol{\xi}_t\) for each timestamp \(t\).

To achieve this, we first compute the difference between the estimated optical flow and the reprojected pixel from the camera at time \(t\) to \(t+1\) using the depth estimate for each pixel in the depth map \({D}_{t}\) and the intrinsic matrix \(\mathbf{K}\). A pixel \(\mathbf{p}_i\) in the depth map is backprojected to 3D space as: \({\mathbf{P}_i} = {\pi ^{ - 1}}(\mathbf{K},\mathbf{p}_i,{D}_t(\mathbf{p}_i))\). We then project this 3D point to the camera at \(t+1\): \(\mathbf{p_j} = \pi (\mathbf{K},{\text{exp}} (\boldsymbol{\xi} _{t \rightarrow t + 1}^ \wedge )\mathbf{P}_i)\), where \(\pi\) and \(\pi^{-1}\) represent the projection and backprojection operators, respectively. The pixel movement from \(\mathbf{p}_i\) to \(\mathbf{p}_j\) is then compared with the estimated optical flow \(F_{t \to t+1}(\mathbf{p}_i)\) to compute the optical flow loss: \[{\mathcal{L}_F} = \sum_i{M_t(\mathbf{p}_i)M_{t+1}(\mathbf{p}_j){\left\| {{F_{t \to t + 1}}({\mathbf{p}_i}) - ({\mathbf{p}_j} - \mathbf{p}_i)} \right\|_2}}.\nonumber\]

In this formulation, \(M_t\) is a mask indicating valid depth values in \({D}_t\) at pixel \(\mathbf{p}_i\). The mask \(M_t\) dynamically updates during optimization, as \(\mathbf{p}_j\) depends on the transformation \(\boldsymbol{\xi}\): \(M_t(\mathbf{p}_i) = {D}_t(\mathbf{p}_i) > 0\).

We employ a state-of-the-art pretrained optical flow estimation model [47] designed hierarchically to produce reliable optical flow even from two blurry input images. Next, we calculate the photometric loss between color values at pixel \(\mathbf{p}_i\) in image \({I}_t\) and pixel \(\mathbf{p}_j\) in image \({I}_{t+1}\): \[{\mathcal{L}_I} = \sum_i {M_t(\mathbf{p}_i)M_{t+1}(\mathbf{p}_j){\left\| { {I}_t(\mathbf{p}_i) - {I}_{t+1}(\mathbf{p}_j) } \right\|_1}}.\nonumber\]

To ensure geometric consistency, we introduce a depth consistency loss by using the two depth maps captured by the depth camera to optimize relative poses. Specifically, as illustrated in Figure 2, we compute and compare the vertical components \(\overline{{\mathbf{P}_i\mathbf{H}_i}}\) and \(\overline{{\mathbf{P}_j\mathbf{H}_j}}\) based on the baseline vector \(\overline{{\mathbf{O}_t\mathbf{O}_{t+1}}}\) between the two cameras. We calculate \(\overline{{\mathbf{P}_i\mathbf{H}_i}}\) as follows: \[\overline{{\mathbf{P}_i\mathbf{H}_i}} = {{{\left\| {\mathbf{P}_i} \right\|}_2}\sin \left({{\cos }^{ - 1}}\left(\frac{{\mathbf{P}_i \cdot \mathbf{x}}}{{{{\left\| {\mathbf{P}_i} \right\|}_2}{{\left\| \mathbf{x} \right\|}_2}}}\right)\right)}.\nonumber\] Similarly, we compute \(\overline{{\mathbf{P}_j\mathbf{H}_j}}\) as: \[{!}{ \fontsize{10}{12}\selectfont\overline{{\mathbf{P}_j\mathbf{H}_j}} = {{{\left\| {\mathbf{P}_j} \right\|}_2}\sin \left( {{\cos }^{ - 1}}\left(\frac{{\mathbf{P}_j \cdot ( - {\text{exp}} {{(\boldsymbol{\omega} )}^{\intercal}}\mathbf{x})}}{{{{\left\| {\mathbf{P}_j} \right\|}_2}{{\left\| { - {\text{exp}} {{(\boldsymbol{\omega} )}^{\intercal}}\mathbf{x}} \right\|}_2}}}\right) \right) }. } \nonumber\] This depth consistency loss is calculated as follows: \[\mathcal{L}_{C} = \sum_i {M_t(\mathbf{p}_i)M_{t+1}(\mathbf{p}_j){\left\| \overline{{\mathbf{P}_i\mathbf{H}_i}} - \overline{{\mathbf{P}_j\mathbf{H}_j}} \right\|}},\] where \({\text{exp}}()\) is the exponential mapping operator in Lie algebra, which helps maintain alignment. Our final loss function combines these terms with weighted coefficients: \[\mathcal{L}_\text{pose}= {\mathcal{L}_F} + {\lambda _I}{\mathcal{L}_I} + {\lambda _C}{\mathcal{L}_C}.\]

We start optimization with \({\lambda _I} = 0\) and \({\lambda _C} = 0\), allowing only the flow loss \({\mathcal{L}_F}\) to guide the relative pose alignment with global optical flow. After initial convergence, we increase weights to \({\lambda _I} = 75\) and \({\lambda _C} = 25\) and continue optimization to a second convergence.

Finally, we apply a point-to-plane ICP algorithm on the two globally aligned point clouds to estimate camera poses accurately. Since optical flow estimation between two blurry images introduces some errors, this final camera pose estimation relies exclusively on depth maps.

3.2 Global Pose Estimation/Geometry Refinement↩︎

Starting from the initial camera poses and point clouds obtained in the previous step, we merge them into a global point cloud within the world coordinate system (Figure 3). To manage memory efficiently, we downsample the point cloud with a specified voxel size \(s\). However, due to accumulated errors in the initial camera poses, these poses may not align well with the global point cloud, leading to seam artifacts at image boundaries and a drifting phenomenon that creates loop-closure issues. To resolve this, we refine both the camera poses and 3D point cloud positions simultaneously within the Gaussian Splatting pipeline, ensuring global and geometric consistency.

We initialize the Gaussian Splatting pipeline by loading the point cloud along with the estimated camera poses. To capture every depth value per pixel from the dense depth maps, we initialize 3D Gaussians, fixing their sizes and setting their scale to \(s/2\). This dense arrangement of 3D Gaussians provides a detailed geometric representation. We then train only the camera poses, 3D positions, and opacities of the Gaussians by comparing a rendered depth map \({\widetilde{D}}\) with the input depth map \({D}\).

During training, we randomly select camera viewpoints from among the trainable camera pose variables to render each depth map. For a depth value \({\widetilde{D}}_t(\mathbf{p}_i)\) at pixel \(\mathbf{p}_i\) in the rendered map, we calculate the depth loss as: \[{\mathcal{L}_{D}} = {\sum\limits_i {{M_t}(\mathbf{p}_i){{\left\| { {{D_t}}(\mathbf{p}_i) - {\widetilde{D}}_{t}(\mathbf{p}_i)} \right\|}_1}} }.\] We also initialize each 3D Gaussian’s opacity \(o_j\) to 0.5, optimizing them to values of 0 or 1: \[\label{eq:opacity95loss} {\mathcal{L}_\text{opacity}} = \sum\limits_j {o_j^2{{(1 - {o_j})}^2}}.\tag{1}\]

This optimization aims to retain only Gaussians with an opacity of 1, pruning others to reduce ambiguity for the subsequent deblurring stage. We prune Gaussians with an opacity less than 0.8 and reset opacity with 0.5 again every certain amount of iteration. As iterations progress, the rendered depth maps increasingly align with the input depth maps, indicating that all 3D Gaussian positions and camera poses are globally well-aligned (Figure 4).

The final refinement loss is a weighted combination of the depth and opacity losses: \[\label{eq:refinement95loss} \mathcal{L}_\text{refinement }= {\lambda _D}{\mathcal{L}_D} + {\lambda_o}{\mathcal{L}_\text{opacity}}.\tag{2}\]

3.3 Image Deblur↩︎

Motion blur caused by camera movement can generally be approximated as: \[I(t) \approx B(t) = \frac{1}{k}\sum\limits_{i = 0}^k {{C_i}(t)}\] Here, \(B(t)\) represents the motion-blurred image, which we model as the average of \(k\) sharp images \(C_i(t)\) rendered from the 3D Gaussians at \(k\) virtual camera poses interpolated between the camera poses at the start and end of the exposure time period \({\boldsymbol{\xi}} _t^s\) and \({\boldsymbol{\xi}} _t^e\). This approximation allows \(B(t)\) to closely match the input image \(I(t)\).

Using the refined camera poses and point cloud from the previous step, we incorporate depth information to achieve geometrically accurate deblurring, following the approach in recent work [3]. Specifically, we minimize the difference between the input image \(I(t)\) and the averaged rendered image \(B(t)\), as shown below: \[{\mathcal{L}_{B}} = \sum\limits_{t} {\sum\limits_i {{{\left\| { {{I_t}}(\mathbf{p}_i) - {B}_{t}(\mathbf{p}_i)} \right\|}_1}} }.\] The final deblur loss function integrates this alignment loss with additional terms for structural similarity and depth consistency: \[\label{eq:deblur95loss} \mathcal{L}_\text{deblur }= {(1 - \lambda _B)}{\mathcal{L}_B} + {\lambda_B}{\mathcal{L_\text{D-SSIM}}} + {\lambda _D}{\mathcal{L}_D}.\tag{3}\] This combined loss helps ensure that the deblurred output aligns well with both the input image and depth structure, leading to sharper and more geometrically consistent results.

4 Results↩︎

We prepare both synthetic and real-world datasets to evaluate our method’s performance. In the absence of an RGB-D dataset with extreme motion blur, we generate a synthetic reference dataset by rendering directly from 3D models. We create RGB-D image sequences and record per-frame camera trajectories to evaluate our pose estimation and deblurring accuracy. The sequences, ranging from 50 to 150 frames, are captured along simulated trajectories that include rapid translations, 360-degree rotations, and human movement paths (using Blender’s Walk Navigation feature). The RGB and depth images have a resolution of \(640\times480\), and motion blur is applied by enabling Blender’s motion blur setting. The 3D models are sourced from the McGuire archive [48] and the Blender archive [49].

For real scene evaluation, we used an Azure Kinect RGB-D camera with a 33.3ms fixed exposure time, 5000K fixed white balance, NFOV-\(2\times2\)-binned depth capture mode, and \(1280\times720\) RGB resolution. We used the provided intrinsic camera parameters to undistort the images and aligned depth maps to color images using the SDK’s transformation functions. Depth maps were eroded three times to remove outliers, which commonly occur along edges.

Our code is based on 3D Gaussian Splatting [7], with data processing partially adapted from CF-3DGS [7]. For initial camera pose estimation, we use the Unimatch [47] optical flow network and the ICP [50] algorithm from Open3D as part of the optimization process. For an input image sequence of \(N\) frames, we prune Gaussians every 100\(N\) iterations, reset opacity to 0.5 every 200\(N\) iterations, and run a total of 600\(N\) iterations (Section 3.2). The number of virtual views for deblurring is set to 15 for synthetic dataset evaluation and 17 for real-scene comparison. All experiments were conducted on an NVIDIA RTX A6000 GPU (48GB), with initial camera pose optimization taking 11 seconds per frame, 0.1 seconds per iteration for pose and geometry refinement, and 0.26 seconds per iteration for deblurring at \(640\times480\) resolution.

4.1 Quantitative Evaluation↩︎

We evaluate the camera pose accuracy and deblurring performance on three synthetic scenes from our dataset, each rendered with extreme motion blur. For pose accuracy, we compare our method against a variety of approaches capable of generating a 3D point cloud and camera trajectory: COLMAP [8], ICP with FPFH feature-based global alignment [51], ORB-SLAM3 [24], ElasticFusion [16], and several RGB-D SLAM methods, including NICE-SLAM [28], Point-SLAM [29], and DROID-SLAM [30]. Additionally, we include recent Gaussian Splatting-based SLAM approaches such as MonoGS [32] and SplaTAM [31]. For each method, we calculate absolute trajectory error (ATE) and relative pose error (RPE) in meter and degree units, using the ground truth camera trajectories as references.

To evaluate deblurring accuracy, we modify a state-of-the-art Gaussian Splatting-based image restoration method [3] to operate in a depth-aware manner using undistorted color and depth images. We select one out of every three images as a test view and render the corresponding novel view for comparison with the ground truth. For quantitative metrics, we calculate PSNR, SSIM, and LPIPS on the rendered images, as well as RMSE between the rendered and ground truth depth maps in the inverse depth domain (\(m^{-1}\)).

4.2 Ablation Study↩︎

To validate the contributions of each component in our method, we perform an ablation study evaluating camera pose and deblurring accuracy by selectively omitting key elements. Specifically, we conduct experiments without fixing the scale of the Gaussians (Section 3.2), without the pose and geometry refinement process (Equation 2 ), and without depth loss (Equation 3 ).

We assess the effectiveness of these components by performing deblurring based on results from Section 3.1—omitting refinement of camera poses and geometry—and by allowing the scale of the 3D Gaussians to be optimized freely. Table 3 demonstrates that both the global refinement process and fixed Gaussian scale are essential for achieving high accuracy. Notably, during pose refinement, the algorithm reduces absolute pose error even at the expense of a slight increase in relative pose error, underscoring the importance of our refinement approach for stable, high-quality reconstruction.

Our method achieves superior RPE scores across all scenes, as shown in Table 1, outperforming other approaches in pose and geometry estimation under severe motion blur. While our ATE scores are comparable to those of DROID-SLAM, which uses a bundle adjustment (BA) module, ATE alone does not correlate strongly with deblurring quality. Our method consistently produces more accurate and denser point clouds, which directly contributes to improved deblurring performance, as evidenced by the metrics in Table 2. These results confirm that our integrated approach to pose estimation and deblurring is more resilient to challenging conditions than other methods.

4.3 Qualitative Evaluation↩︎

Figures 5 and 6 present qualitative comparisons of our method’s performance in synthetic and real scenes. The densification capability of Gaussian Splatting provides detailed scene representation even from a relatively sparse point cloud; however, our approach, with its initially dense and accurate point cloud generation, achieves notably superior deblurring performance, especially in restoring high-frequency details. Our method consistently demonstrates a significant advantage in visual fidelity and detail restoration across diverse conditions.

5 Conclusion↩︎

We have introduced a robust method for 3D scene reconstruction from RGB-D image sequences that effectively addresses the challenges of extreme motion blur and low-light conditions. Our approach leverages optical flow from color images and a carefully designed geometric loss to achieve accurate global alignment, followed by local refinement of camera poses and geometry using ICP. By integrating these steps into a Gaussian Splatting pipeline, we further refine camera poses and 3D geometry by minimizing depth and opacity losses. Additionally, fixing the scale of the 3D Gaussians ensures that the depth information from the RGB-D input is fully utilized, allowing for precise camera pose estimation and improved deblurring performance. Our method demonstrates superior performance on real and synthetic RGB-D scenes with significant motion blur, outperforming existing approaches.

Acknowledgements↩︎

Min H. Kim acknowledges the Samsung Research Funding & Incubation Center (SRFC-IT2402-02), the Korea NRF grant (RS-2024-00357548), the MSIT/IITP of Korea (RS-2022-00155620, RS-2024-00398830, 2022-0-00058, and 2017-0-00072), Microsoft Research Asia, and Samsung Electronics.

References↩︎