1 Introduction↩︎

Neural fields [1] have recently gained a lot of attention given its ability to encode implicit representations of complex 3D scenes using deep neural networks. Additionally, they have been shown very effective in addressing multiple problems such as 3D reconstruction [2], [3], scene rendering [4], [5] or human body representation [6], [7]. Among different methods built upon this framework [8]–[10], Neural Radiance Fields (NeRF) [11] has obtained impressive results in generating photo-realistic views of 3D scenes and solving downstream tasks such as depth estimation [12], scene editing [13], [14] or pose prediction [15], [16].

Despite its increasing popularity, a critical limitation of NeRF holding back its application to real-world problems has been recently pointed out by [17]. Concretely, current NeRF-based methods are not able to quantify the uncertainty associated with the model estimates. This information is crucial in several scenarios such as robotics [18], [19], medical diagnosis [20] or autonomous driving [21] where, to reduce potentially catastrophic failures, the confidence on the model outputs must be included into the decision-making process.

To address this limitation, recent works [5], [17] have explored different strategies to incorporate uncertainty quantification into NeRF. Remarkably, Stochastic NeRF [17] (S-NeRF) has recently obtained state-of-the-art results by employing a probabilistic model to learn a simple distribution over radiance fields. However, in order to make this problem tractable, S-NeRF makes strong assumptions over the aforementioned distribution, thus limiting model expressivity and leading to sub-optimal results for complex 3D scenes.

In this context, we propose Conditional-Flow NeRF (CF-NeRF), a novel probabilistic framework to incorporate uncertainty quantification without sacrificing model flexibility. As a result, our approach enables both effective 3D modelling and reliable uncertainty estimates (see Fig. 1). In the following, we summarise the main technical contributions of our method:

Modelling Radiance-Density Distributions with Normalizing Flows: In contrast to S-NeRF where the radiance and density in the scene are assumed to follow simple distributions (see Fig. 2 (c-Top)), our method learns them in a flexible and fully data-driven manner using conditional normalizing flows [22]. This allows CF-NeRF to learn arbitrarily complex radiance and density distributions (see Fig. 2 (c-Bottom)) which have the capacity to model scenes with complex geometry and appearance.

Latent Variable Modelling for Radiance Fields Distributions: Motivated by De Finetti’s Theorem [23], CF-NeRF incorporates a global latent variable in order to efficiently model the joint distribution over the radiance-density variables for all the spatial locations in the scene. This contrasts with the approach followed by S-NeRF, where these variables are considered independent for different 3D coordinates. Intuitively, the independence assumption violates the fact that changes in the density and radiance between adjacent locations are usually smooth. As a consequence, S-NeRF tends to generate low-quality and noisy predictions (see Fig. 2 (a,b)-Top). In contrast, the global latent variable used in CF-NeRF allows to efficiently model the complex joint distribution of these variables, leading to spatially-smooth uncertainty estimates and more realistic results (Fig. 2 (a,b)-Bottom).

In our experiments, we evaluate CF-NeRF over different benchmark datasets containing scenes with increasing complexity. Our qualitative and quantitative results show that our method obtains significantly better uncertainty estimations for rendered views and predicted depth maps compared to S-NeRF and other previous methods for uncertainty quantification. Additionally, CF-NeRF provides better image quality and depth estimation precision. These results confirm that our method is able to incorporate uncertainty quantification into NeRF without sacrificing model expressivity.

2 Related Work↩︎

NeRF [11] has become a popular approach for 3D scene modelling due to its simplicity and its impressive performance on several tasks. By using a sparse collection of 2D views, NeRF learns a deep neural network encoding a representation of the 3D scene. This network is composed of a set of fully-connected layers that output the volume density and emitted radiance for any input 3D spatial location and 2D viewing direction in the scene. Subsequently, novel views are generated by applying volumetric rendering  [24] to the estimated density and radiance values obtained by the network.

Recently, several extensions over original NeRF have been proposed  [25]–[33]. For instance, different works have focused on accelerating NeRF training  [34]–[36] or modelling dynamic scenes  [37]–[41]. Despite the advances achieved at different levels, the application of current NeRF-based methods in real scenarios is still limited since they are unable to quantify the uncertainty associated with the rendered views or estimated geometry. Our proposed CF-NeRF explicitly addresses this problem and can be easily combined with most of current NeRF-based approaches.

Uncertainty estimation has been extensively studied in deep learning [42]–[46] and have been applied to different computer vision tasks  [47]–[49]. Early works proposed to use Bayesian Neural Networks (BNN) [50], [51] to estimate the uncertainty of both network weights and outputs by approximating their marginal distributions. However, training BNNs is typically difficult and computationally expensive. As a consequence, their use in large deep neural networks is limited.

More recently, efficient strategies have been proposed to incorporate uncertainty estimation into deep neural networks  [52], [53]. Among them, MC-Dropout  [54] and Deep Ensembles  [55] are two of the most popular approaches given that they are agnostic to the specific network architecture  [56]–[59]. More concretely, MC-Dropout adds stochastic dropout during inference into the intermediate network layers. By doing multiple forward passes over the same input with different dropout configurations, the variance over the obtained set of outputs is used as the output uncertainty. On the other hand, Deep Ensembles applies a similar strategy by computing a set of outputs from multiple neural networks that are independently trained using different parameters initializations. In contrast to MC-Dropout and Deep Ensembles, however, we do not train independent networks to generate the samples. Instead, CF-NeRF explicitly encodes the distribution of the radiance fields into a single probabilistic model.

Recently, some works [5], [17] have attempted to incorporate uncertainty estimation into Neural Radiance Fields. NeRF-in-the-Wild (NeRF-W) [5] modelled uncertainty at pixel-level in order to detect potential transient objects such as cars or pedestrians in modelled scenes. For this purpose, NeRF-W employs a neural network to estimate an uncertainty value for each 3D spatial location. Subsequently, it computes a confidence score for each pixel in a rendered image by using an alpha-compositing strategy similar to the one used to estimate the pixel color in original NeRF. However, this approach is not theoretically grounded since there is no intuitive justification to apply the volume rendering process over uncertainty values. Additionally, NeRF-W is not able to evaluate the confidence associated with estimated depth maps.

On the other hand, S-NeRF [17] has achieved state-of-the-art performance in quantifying the uncertainty of novel rendered views and depth-maps. Instead of learning a single radiance field as in original NeRF [11], S-NeRF models a distribution over all the possible radiance fields explaining the scene. During inference, this distribution is used to sample multiple color or depth predictions and compute a confidence score based on their associated variance. However, S-NeRF imposes strong constraints over the radiance field distribution and, as a consequence, this limits its ability to model scenes with complex appearance and geometry. In contrast, the proposed CF-NeRF combines a conditional normalizing flow and latent variable modelling to learn arbitrary complex radiance fields distributions without any prior assumption.

Normalizing Flows (NF) have been extensively used to model complex distributions with unknown explicit forms. For this purpose, NF uses a sequence of invertible functions to transform samples from a simple known distribution to variables following an arbitrary probability density function. Additionally, the change-of-variables formula can be used to compute the likelihood of different samples according to the transformed distribution. Given its flexibility, Normalizing Flows have been used in several 3D modelling tasks. For instance, [60], [61] introduced different flow-based generative models to model 3D point cloud distributions and sample from them. More recently, [62] used a normalizing flow in order to avoid color shifts at unseen viewpoints in the context of NeRF. To the best of our knowledge, our proposed CF-NeRF is the first approach employing Normalizing Flows in order to learn radiance fields distributions for 3D scene modelling.

3 Deterministic and Stochastic Neural Radiance Fields↩︎

Standard NeRF [11] represents a 3D volumetric scene as \(\mathcal{F} = \{ (\mathbf{r}(\mathbf{x},\mathbf{d}), \alpha(\mathbf{x})): \mathbf{x} \in \mathbb{R}^3, \mathbf{d} \in \mathbb{R}^2\), where \(\mathcal{F}\) is a set containing the volume density \(\alpha(\mathbf{x}) \in \mathbb{R}^+\) and RGB radiance \(\mathbf{r}(\mathbf{x},\mathbf{d}) \in \mathbb{R}^3\) from all the spatial locations \(\boldsymbol{x}\) and view directions \(\mathbf{d}\) in the scene.

In order to implicitly model the infinite set \(\mathcal{F}\), NeRF employs a neural network \(f_{\boldsymbol{\theta}}(\mathbf{x},\mathbf{d})\) with parameters \({\boldsymbol{\theta}}\) which outputs the density \(\alpha\) and radiance \(\mathbf{r}\) for any given input location-view pair \(\{\mathbf{x},\mathbf{d}\}\). Using this network, NeRF is able to estimate the color \(\mathbf{c}(\mathbf{x}_o,\mathbf{d})\) for any given pixel defined by a 3D camera position \(\mathbf{x}_o\) and view direction \(\mathbf{d}\) using the volumetric rendering function: \[\label{eq:vol95rendering} \mathbf{c} (\mathbf{x}_o,\mathbf{d}) = \int_{t_n}^{t_f} T(t) \alpha(\mathbf{x}_t) \mathbf{r}(\mathbf{x}_t, \mathbf{d}) dt,\text{where}T(t) = \text{exp}(-\int_{t_n}^{t} \alpha(\mathbf{x}_s) ds),\tag{1}\] where \(\mathbf{x}_t = \mathbf{x}_o + t \mathbf{d}\) corresponds to 3D locations along a ray with direction \(\mathbf{d}\) originated at the camera origin and intersecting with the pixel at \(\mathbf{x}_o\).

During training, NeRF optimizes the network parameters \({\boldsymbol{\theta}}\) using Eq. ([eq:vol_rendering]) by leveraging Maximum Likelihood Estimation (MLE) over a training set \(\mathcal{T}\) containing ground-truth images depicting views of the scene captured from different camera positions. More details about NeRF and its learning procedure can be found in the original paper [11].

Deterministic NeRF is not able to provide information about the underlying uncertainty associated with the rendered views or estimated depth maps. The reason is that the network \(f_{\boldsymbol{\theta}}(\mathbf{x},\mathbf{d})\) is trained using Maximum Likelihood Estimation and thus, the learning process performs a single point estimate over all the plausible radiance fields \(\mathcal{F}\) given the training set \(\mathcal{T}\). As a consequence, model predictions are deterministic and it is not possible to sample multiple outputs to compute their associated variance. To address this limitation, S-NeRF [17] employs Bayesian Learning to model the posterior distribution \(p_{\boldsymbol{\theta}}(\mathcal{F} | \mathcal{T})\) over all the plausible radiance fields explaining the scene given the training set. In this manner, uncertainty estimations can be obtained during inference by computing the variance over multiple predictions obtained from different radiance fields \(\mathcal{F}\) sampled from this distribution.

In order to make this problem tractable, S-NeRF uses Variational Inference to approximate the posterior \(p_{\boldsymbol{\theta}}(\mathcal{F} | \mathcal{T})\) with a parametric distribution \(q_{\boldsymbol{\theta}}(\mathcal{F})\) implemented by a deep neural network. However, S-NeRF imposes two limiting constraints over \(q_{\boldsymbol{\theta}}(\mathcal{F})\). Firstly, it models the radiance \(\mathbf{r}\) and density \(\alpha\) for each location-view pair in the scene as independent variables. In particular, the probability of a radiance field given \(\mathcal{T}\) is modelled with a fully-factorized distribution: \[\label{eq:apost95definition} q_{\boldsymbol{\theta}}(\mathcal{F}) = \prod_{\mathbf{x} \in \mathbb{R}^3} \prod_{\mathbf{d} \in \mathbb{R}^2} q_{\boldsymbol{\theta}}(\mathbf{r}|\mathbf{x},\mathbf{d}) q_{\boldsymbol{\theta}}(\alpha|\mathbf{x}).\tag{2}\] Whereas this conditional-independence assumption simplifies the optimization process, it ignores the fact that radiance and density values in adjacent spatial locations are usually correlated. As a consequence, S-NeRF tends to render noisy and low-quality images and depth maps (see Fig. 2).

4 Conditional Flow-based Neural Radiance Fields↩︎

Our proposed Conditional Flow-based Neural Radiance Fields incorporates uncertainty estimation by using a similar strategy than S-NeRF. In particular, CF-NeRF learns a parametric distribution \(q_{\boldsymbol{\theta}}(\mathcal{F})\) approximating the posterior distribution \(p(\mathcal{F}|\mathcal{T})\) over all the possible radiance fields given the training views. Different from S-NeRF, however, our method does not assume a simple and fully-factorized distribution for \(q_{\boldsymbol{\theta}}(\mathcal{F})\), but it relies on Conditional Normalizing Flows and Latent Variable Modelling to preserve model expressivity. In the following, we discuss the technical details of our approach (Sec. [sec:cf_nerf]), the optimizing process following S-NeRF (Sec. [sec:optimization]) and inference procedure used to compute the uncertainty associated with novel rendered views and depth maps (Sec. [sec:inference]).

4.1 Modelling Flexible Radiance Field Distributions with CF-NeRF↩︎

As discussed in Sec. [sec:nerf], S-NeRF formulation assumes that radiance and density variables at each spatial location in the field follow a fully-factorized distribution ( Eq. ([eq:apost_definition])). As a consequence, they are forced to be independent from each other. This simplification was motivated by the fact that modelling the joint distribution over the infinite set of radiance and density variables in the field \(\mathcal{F}\) is not trivial. To address this problem, our proposed CF-NeRF leverages De Finetti’s representation theorem [23]. In particular, the theorem states that any joint distribution over sets of exchangeable variables can be written as a fully-factorized distribution conditioned on a global latent variable. Based on this observation, we define the approximate posterior \(q_{\boldsymbol{\theta}}(\mathcal{F})\): \[\label{eq:factored95distribution} q_{\boldsymbol{\theta}}(\mathcal{F}) = \int_{\mathbf{z}} q_\vartheta( \mathbf{z}) q_{\boldsymbol{\theta}}(\mathcal{F}| \mathbf{z})d \mathbf{z} = \int_{\mathbf{z}} q_\vartheta(\boldsymbol{z}) \prod_{\mathbf{x} \in \mathbb{R}^3} \prod_{\mathbf{d} \in \mathbb{R}^2} q_{\boldsymbol{\theta}}(\mathbf{r}|\mathbf{x},\mathbf{d},\mathbf{z}) q_{\boldsymbol{\theta}}(\alpha|\mathbf{x}, \mathbf{z})d \mathbf{z} \, ,\tag{3}\] where the latent variable \(\mathbf{z}\) is sampled from a Gaussian prior \(q_\vartheta(\mathbf{z})=\{\mathbf{\mu}_z , \mathbf{\sigma}_z\}\) with learned mean and variance. In this manner, all the 3D spatial-locations \(\mathbf{x}\) and viewing-directions \(\mathbf{d}\) generated from the same shared latent variable \(\mathbf{z}\) and thus, they are conditionally dependent between them. This approach allows CF-NeRF to efficiently model the distribution over the radiance-density variables in the radiance field without any independence assumption. It is also worth mentioning that De Fineti’s theorem has been also used in previous works learning joint distributions over sets for 3D point cloud modelling [63].

Given that the radiance-density distributions can be highly complicated for complex scenes, modelling them with variants of Gaussian distributions as in Stochastic NeRF can lead to sub-optimal results. For this reason, CF-NeRF models \(q_{\boldsymbol{\theta}}(\mathbf{r}|\mathbf{x},\mathbf{d},\mathbf{z})\) and \(q_{\boldsymbol{\theta}}(\alpha|\mathbf{x},\mathbf{z})\) using a Conditional Normalizing Flow (CNF) [64]. This allows to learn arbitrarily complex distributions in a fully data-driven manner without any prior assumption.

For any input location-direction pair \((\mathbf{x},\mathbf{d})\), we use a sequence of \(K\) CNFs that transform the global latent variable \(\mathbf{z}\) into samples from the radiance \(\boldsymbol{r}\) or density \(\alpha\) distribution \(q_{\boldsymbol{\theta}}(\mathbf{r}|\mathbf{x},\mathbf{d},\mathbf{z})\) or \(q_{\boldsymbol{\theta}}(\alpha|\mathbf{x},\mathbf{z})\). More formally, each flow is defined as an invertible parametric function \(\mathbf{z}_k = f_k(\mathbf{z}_{k-1},\mathbf{\mathbf{x}, \mathbf{d}})\), where \(f_k\) maps a random variable \(\boldsymbol{z}_{k-1}\) into another one \(\boldsymbol{z}_k\) with a more complex distribution. Note that the each flow \(f_k\) is conditioned on the location and view direction \((\mathbf{x},\mathbf{d})\). Finally, \(\boldsymbol{z}_K\) is followed by a Sigmoid and Softplus activation functions for radiance and density samples, respectively. This process is detailed in Fig. [fig:conditional_nf].

Using the introduced CNF, radiance and density probabilities \(q_{\boldsymbol{\theta}}(\mathbf{r}|\mathbf{x},\mathbf{d},\mathbf{z})\) and \(q_{\boldsymbol{\theta}}(\alpha|\mathbf{x},\mathbf{z})\) can be computed with the change-of-variables formula typically used in normalizing flows as: \[\begin{align} \label{eq:radiance95probability} q_{\boldsymbol{\theta}}(\mathbf{r}|\mathbf{x},\mathbf{d}, \mathbf{z}) &= q_\vartheta(\mathbf{z}) \left| \det \frac{\partial \mathbf{z}}{\partial \mathbf{r}} \right| = q_\vartheta(\mathbf{z}) \left| \det \frac{\partial \mathbf{r}}{\partial \mathbf{z}_{K}} \right|^{-1} \prod_{k=1}^K \left| \det \frac{\partial \mathbf{z}_k}{\partial \mathbf{z}_{k-1}} \right|^{-1} , \end{align}\tag{4}\] where \(\left| \det (\partial \mathbf{z}_k / \partial \mathbf{z}_{k-1}) \right|\) measures the Jacobian determinant of the transformation function \(f_k\). Note that \(q_{\boldsymbol{\theta}}(\alpha|\mathbf{x},\mathbf{z})\) can be computed in a similar manner.

4.2 Optimizing Conditional-Flow NeRF↩︎

We adopt a Variational Bayesian approach to learn the parameters of the posterior distribution \(q_{\boldsymbol{\theta}}(\mathcal{F})\) defined in Eq. ([eq:factored_distribution]). Note that the optimized \(\mathbf{\theta}\) corresponds to the parameters of the CNFs defined in the previous section. More formally, we solve an optimization problem where we minimize the Kullback-Leibler (KL) divergence between \(q_{\boldsymbol{\theta}}(\mathcal{F})\) and the true posterior distribution \(p(\mathcal{F} | \mathcal{T})\) of radiance fields given the training set: \[\begin{align} \label{eq:kl95fnerf} &\min_{{\boldsymbol{\theta}}}\mathbb{KL}(q_{\boldsymbol{\theta}}(\mathcal{F}) || p( \mathcal{F} | \mathcal{T})) \nonumber \\ &= \min_{{\boldsymbol{\theta}}} - \mathbb{E}_{q_{\boldsymbol{\theta}}(\mathcal{F})} \log p( \mathcal{T} | \mathcal{F}) + \mathbb{E}_{q_{\boldsymbol{\theta}}(\mathcal{F})} \log q_{\boldsymbol{\theta}}(\mathcal{F}) - \mathbb{E}_{q_{\boldsymbol{\theta}}(\mathcal{F})} \log p(\mathcal{F}) . \end{align}\tag{5}\]

The first term in Eq. ([eq:kl_fnerf]) measures the expected log-likelihood of the training set \(\mathcal{T}\) over the distribution of radiance fields \(q_{\boldsymbol{\theta}}(\mathcal{F})\). The second term indicates the negative Entropy of the approximated posterior. Intuitively, maximizing the Entropy term allows uncertainty estimation by preventing the optimized distribution to degenerate into a deterministic function where all the probability is assigned into a single radiance field \(\mathcal{F}\). Finally, the third term corresponds to the cross-entropy between \(q_{\boldsymbol{\theta}}(\mathcal{F})\) and a prior over radiance fields \(p(\mathcal{F})\). Given that it is hard to choose a proper prior in this case, we assume \(p(\mathcal{F})\) to follow uniform distribution and thus, this term can be ignored during optimization. In the following, we detail how the first two terms are computed during training.

Assuming that the training set \(\mathcal{T}\) is composed by \(N\) triplets \(\{\mathbf{C}_n,\mathbf{x}_n^o ,\mathbf{d}_n\}\) representing the color, camera origin and view direction of a given pixel in a training image, the log-likelihood term in Eq. ([eq:kl_fnerf]) is equivalent to: \(\frac{1}{N} \sum_{n=1}^N \log p(\boldsymbol{C}_n | \mathcal{F})\). In order to compute an individual \(p(\mathbf{C}_n | \mathcal{F})\) we use the following procedure.

Firstly, we sample a set of variables \(\boldsymbol{z}^{1:K}\) from the global latent distribution \(q_\vartheta(\mathbf{z})\). Secondly, given a training triplet \(\{\mathbf{C},\mathbf{x}^o,\mathbf{d}\}\) and a set of 3D spatial locations along a ray defined by \(\boldsymbol{x}_t=\boldsymbol{x}_o + t\boldsymbol{d}\), we sample a set of density and radiance values \(\{\alpha^{1:K}\}_{t_n:t_f}\) and \(\{\boldsymbol{r}^{1:K}\}_{t_n:t_f}\) by using the Conditional Flow introduced in Sec. [sec:cf_nerf]. In particular, we perform a forward pass over the flow with inputs \(\boldsymbol{z}^k\) conditioned to \(\boldsymbol{x}^*_t\) for all \(t\). Subsequently, a set of \(K\) color estimates \(\hat{\boldsymbol{C}}_{1:K}\) are obtained by using, for every \(k\), the volume rendering formula in Eq. ([eq:vol_rendering]) over the sampled radiance and density values \(\{\alpha^k\}_{t_n:t_f}\) and \(\{\boldsymbol{r}^k\}_{t_n:t_f}\), respectively. Finally, with the set of obtained color estimations \(\hat{\boldsymbol{C}}_{1:K}\), we use a non-parametric kernel density estimator as in [17] to approximate the log-likelihood \(\log p(\mathbf{C} | \mathcal{F})\) of the pixel color. An illustration of this procedure is shown in Fig. 4.

In order to compute the entropy term in Eq. ([eq:kl_fnerf]), we use a Monte Carlo approximation with \(M\) samples as: \[\begin{align} \mathbb{E}_{q_{\boldsymbol{\theta}}(\mathcal{F})} \log q_{\boldsymbol{\theta}}(\mathcal{F}) &= \mathbb{E}_{q_{\boldsymbol{\vartheta}}(\mathbf{z})} \prod_{\mathbf{x} \in \mathbb{R}^3} \prod_{\mathbf{d} \in \mathbb{R}^2} \log q_{\boldsymbol{\theta}}(\mathbf{r}|\mathbf{x},\mathbf{d},\boldsymbol{z}) + \mathbb{E}_{q_{\boldsymbol{\vartheta}}(\mathbf{z^{\alpha}})} \prod_{\mathbf{x} \in \mathbb{R}^3} \prod_{\mathbf{d} \in \mathbb{R}^2} \log q_{\boldsymbol{\theta}}(\alpha|\mathbf{x},\boldsymbol{z}) \nonumber \\ &\sim \frac{1}{M} \sum_{m=1}^M \bigg( \log q_{\boldsymbol{\theta}}(\mathbf{r}_m|\mathbf{x}_m,\mathbf{d}_m,\boldsymbol{z}_m) + \log q_{\boldsymbol{\theta}}(\alpha_m|\mathbf{x}_m,\mathbf{z}_m) \bigg) \end{align}\] where \(\mathbf{z}_m\) are obtained from the latent variable distribution \(q_{\boldsymbol{\vartheta}}(\mathbf{z})\) and \(\{\mathbf{x}_m,\mathbf{d}_m\}\) are possible 3D locations and view directions in the scene which are randomly sampled. Finally, \(\alpha_m\) and \(\mathbf{r}_m\) are density and radiance values obtained by applying our conditional flow with inputs \((\mathbf{z}_m\),\(\boldsymbol{x}_m\),\(\mathbf{d}_m)\). Their probabilities \(q_{\boldsymbol{\theta}}(\alpha_m|\mathbf{x}_m,\mathbf{z}_m)\) and \(q_{\boldsymbol{\theta}}(\mathbf{r}_m|\mathbf{x}_m,\mathbf{d}_m,\boldsymbol{z}_m)\) can be computed by using Eq. ([eq:radiance_probability]).

4.3 Inference and Uncertainty Estimation↩︎

CF-NeRF allows us to quantify the uncertainty associated with rendered images and depth-maps. As described in Sec. [sec:log_likelihood], given any camera pose, we can obtain a set of color estimates \(\hat{\boldsymbol{C}}_{1:K}\) for every pixel in a rendered image. These estimates are obtained by sampling a set of random variables \(\boldsymbol{z}_{1:K}\) from latent distribution \(q_\vartheta(\mathbf{z})\) and applying CF-NeRF. Finally, the mean and variance of the pixel colors over the K samples are treated as the predicted color and its associated uncertainty, respectively. For depth-map generation, we sample a sequence of density values \(\{\alpha_{k}\}_{t_n:t_f}\) along a camera ray and we compute the expected termination depth of the ray as in  [11]. In this way, we obtain a set of depth-values \(d_{1:K}\) for each pixel in the depth-map for the rendered image. As in the case of the RGB color, its mean and variance correspond to the estimated depth and its associated uncertainty.

5 Experiments↩︎

We conduct a set of exhaustive experiments over two benchmark databases typically used to evaluate NeRF-based approaches: the LLFF dataset from the original NeRF paper [11] and the Light Field LF dataset  [25], [65]. The first is composed of 8 relatively simple scenes with multiple-view forward-facing images. On the other hand, from the LF dataset we use 4 scenes: Africa, Basket, Statue and Torch. The evaluation over the LF scenes is motivated by the fact that they have a longer depth range compared to the ones in LLFF and, typically, they present more complicated geometries and appearance. As a consequence, the evaluation over LLFF gives more insights into the ability of NeRF-based methods to model complex 3D scenes. Same as  [17], we use a sparse number of scene views (\(\sim\)​4) for training and the last adjacent views for testing. As discussed in the cited paper, training in an extremely low-data regime is a challenging task and provides an ideal setup to evaluate the ability of the compared methods to quantify the uncertainty associated with the model predictions. While the ground truth of depth is not available in the original datasets, we compute pseudo ground truth by using the method in  [12] trained on all the available views per scene.

We compare our method with the state-of-the-art S-NeRF [17], NeRF-W [5] and two other methods, Deep-Ensembles(D.E.) [55] and MC-Dropout(Drop.) [54], which are generic approaches for uncertainty quantification in deep learning. For NeRF-W, we remove the latent embedding component of their approach and keep only the uncertainty estimation layers. For Deep-Ensembles, we train 3 different NeRF models in the ensemble in order to have a similar computational cost during training and testing compared to the rest of compared methods. In the case of MC-Dropout, we manually add one dropout layer after each odd layer in the network to sample multiple outputs using random dropout configurations. The described setup and hyper-parameters for the different baselines is the same previously employed in S-NeRF [17].

In order to implement CF-NeRF and the different baselines, we inherit the same network architecture and hyper-parameters than the one employed in the original NeRF¹ implementation. In CF-NeRF, we use this architecture to compute the conditional part given \(\mathbf{x}\) and \(\mathbf{d}\). Since CF-NeRF uses an additional normalizing flow following this network, we add additional layers for the rest of baselines so that they have a similar computational complexity than our method. During training and inference in CF-NeRF, we sample 32 radiance-density pairs for mean and variance estimation for each ray. We optimize all the models for 100,000-200,000 steps with a batch size of 512 and uniformly sampled 128 points across each ray using Adam optimizer with default hyper-parameters. See the supplementary material for additional details on hyper-parameters of our conditional normalizing flow.

In our experiments, we address two different tasks using the compared methods: novel view synthesis and depth-map estimation. In particular, we evaluate the quality of generated images/depth-maps and the reliability of the quantified uncertainty using the following metrics:

Quality metrics: We use standard metrics in the literature. Specifically, we report PSNR, SSIM [66], and LPIPS [67] for generated synthetic views. On the other hand, we compute RMSE, MAE and \(\delta\)-threshold [68] to evaluate the quality of the generated depth-maps.

Uncertainty Quantification: To evaluate the reliability of the uncertainty estimated by each model we report two widely used metrics for this purpose. Firstly, we use the negative log likelihood (NLL) which is a proper scoring rule  [69], [70] and has been previously used to evaluate the quality of model uncertainty [17]. Secondly, we evaluate the compared methods using sparsification curves [47], [48], [71]. Concretely, given an error metric (e.g. RMSE), we obtain two lists by sorting the values of all the pixels according to their uncertainty and the error computed from the ground-truth. By removing the top \(t\% (t=1\sim 100)\) of the errors in each vector and repeatedly computing the average of the last subset, we can obtain the sparsification curve and the oracle curve respectively. The area between them is the AUSE, which evaluates how much the uncertainty is correlated with the predicted error.

5.1 Results over LLFF dataset↩︎

Following a similar experimental setup than [17], we firstly evaluate the performance of the compared methods on the simple scenes in the LLFF dataset. All the views in each scene are captured in a restricted setting where all cameras are forward-facing towards a plane, which allows to use Normalized Device Coordinates (NDC) to bound all coordinates to a very small range(0-1). In  Table [tab:table_llff_rgb_uncertainty], we report the performance of the different evaluated metrics averaged over all the scenes. As can be seen, S-NeRF achieves the second best performance in terms of uncertainty quantification with quality metrics comparable to the rest of baselines. These results are consistent with the results reported in their original paper and demonstrate that the expressivity of S-NeRF on these simple scenes is not severely limited by the imposed strong assumptions over the radiance field distribution. Nonetheless, our CF-NeRF still achieves a significant improvement over most of the metrics, showing the advantages of modelling the radiance field distribution using the proposed latent variable modelling and CNFs.

5.2 Results over LF dataset↩︎

To further test the performance of different approaches on larger scenes with more complex appearances and geometries, we evaluate the different methods on the LF dataset. All images in these scenes have a large depth range and are randomly captured towards the target. As a consequence, coordinates cannot be normalized with NDC.

Results over rendered RGB images and estimated depth-maps for all the scenes and evaluated metrics are shown in Table [tab:table_lf_rgb] and Table [tab:table_lf_depth], respectively. As can be seen, S-NeRF obtains poor performance in terms of quality and uncertainty estimation. This clearly shows that the imposed assumptions over the radiance fields distribution are sub-optimal in complex scenes and lead to inaccurate model predictions and unreliable uncertainty estimations. In contrast, our CF-NeRF outperforms all the previous approaches by large margins across all the scenes and metrics both on image rendering and depth-maps estimation tasks. The better results obtained by our method can be explained because the use of latent variable modelling and conditional normalizing flows allows to learn arbitrarily complex radiance field distributions, which are necessary to model scenes with complicated appearance and geometry. This partly explains that our method achieves better performance, particularly an average 69\(\%\) improvement compared to the second best method Deep-Ensembles on LPIPS, which measures the perceptual similarity at the image level. On the other hand, the performance of Deep-Ensembles and MC-Dropout is largely limited by the number of the trained models or dropout samples, which cannot be increased infinitely in practical cases. In contrast, the explicitly encoding of the radiance field distribution into a single probabilistic model with CF-NeRF allows to use a higher-number of samples in a more efficient manner.

In order to give more insights into the advantages of our proposed CF-NeRF, Fig. 5 shows qualitative results obtained by the compared methods on an example testing view of the LF dataset. By looking at the estimated uncertainty for RGB images, we can observe that NeRF-W obtains a higher prediction error which, additionally, is not correlated with the uncertainty estimates. On the other hand, Deep Ensembles and MC-Dropout render images with a lower predictive error compared to NeRF-W. As can be observed, however, their estimated uncertainties are noticeably not correlated with their errors. Interestingly, S-NeRF seems to obtain uncertainty estimates where the values are more correlated with the predictive error. However, the quality of the rendered image is poor, obviously shown in the background. In contrast, our CF-NeRF is the only method able to render both high-quality images and uncertainty estimates that better correlate with the prediction error.

By looking at the results for generated depth-maps, we can see that our method also obtains the most accurate depth estimates, followed by Deep Ensembles. Nonetheless, the latter generates depth maps with obvious errors in the background. Regarding the uncertainty estimation, we can observe a high correlation between the predicted error and the estimated uncertainty maps generated by our CF-NeRF. This contrasts with the rest methods which estimate low confidence values in the background of the scene inconsistent with their low prediction errors on this specific area. In conclusion, our results demonstrate the ability of our proposed method to provide both reliable uncertainty estimates and accurate model predictions.

6 Conclusion↩︎

We have presented CF-NeRF, a novel probabilistic model to address the problem of uncertainty quantification in 3D modelling using Neural Radiance Fields. In contrast to previous works employing models with limited expressivity, our method couples Latent Variable Modelling and Conditional Normalizing Flows in order to learn complex radiance fields distributions in a flexible and fully data-driven manner. In our experimental results, we show that this strategy allows CF-NeRF to obtain significantly better results than state-of-the-art approaches, particularly on scenes with complicated appearance and geometry. Finally, it is also worth mentioning that our proposed framework can be easily combined with other NeRF-based architectures in order to provide them with uncertainty quantification capabilities.

Supplementary Materials↩︎

In this Supplementary Materials, we firstly give more additional details about our CF-NeRF implementation (Sec. [sec:implementation]), then we describe a set of ablation studies to give more insights about the performance of our model (Sec. [sec:ablations]) and, finally, we provide a set of additional qualitative results (Sec. [sec:more_results]).

7 Additional Implementation Details↩︎

7.0.1 Training details↩︎

As mentioned in Sec. [sec:implementations], we use the same MLP-based architecture used in original NeRF [11] as a backbone network for our CF-NeRF and the rest baselines. In particular, we use 512 hidden units for all layers. For CF-NeRF, each sample from the latent prior distribution is shared for different spatial-location and viewing-direction inputs in each batch during training. To avoid overfitting with the sparse number of training views used in our experiments, we employ an additional depth loss based on [72] during optimization. This loss is weighted with a value of \(1e-2\) for our method and the rest baselines. Additionally, we set a value of 0.01 as the weight for the Entropy term in Eq. ([eq:kl_fnerf]).

7.0.2 Conditional Normalizing Flows↩︎

As for invertible transformation functions in our Conditional Normalizing Flow(CNF), we use the Sylvester Flows [64] defined as: \[\boldsymbol{z}_k = \boldsymbol{z}_{k-1} + \boldsymbol{A} h(\boldsymbol{B} \boldsymbol{z}_{k-1} + b),\] where \(\boldsymbol{A}\), \(\boldsymbol{B}\) and \(\boldsymbol{b}\) are flow parameters of each transformation function. Additionally, \(h\) is an hyperbolic tangent activate function. These flow parameters are conditional functions of the 5D location-direction pairs, while the samples from the latent distributions are transformed to radiance and density by sequentially using these transformation functions \(f_{1:K}\) described in Sec. [sec:conditional_flow]. In our CF-NeRF, we use four flows for the radiance and density CNFs (see Fig. [fig:conditional_nf]) with the dimensions of the conditional feature into each flow set to 64.

7.0.3 Metrics↩︎

As a metric used to assess the quality of the depth prediction, we use the \(\delta\)-threshold [68]. This metric is defined as follows: \[\% \text{ of } y_i \text{ s.t. } max(\frac{y_i^*}{y_i}, \frac{y_i}{y_i^*}) = \delta_{k=1,2,3} < \tau^{k=1,2,3},\] where we set the threshold \(\tau=1.25\) as done in previous works [68]. Note that we only report \(\delta_3\) due to space limitations in the main paper.

8 Ablations↩︎

In order to give insights into some design decisions of our proposed CF-NeRF, we provide results for two ablation experiments. Concretely, we conduct experiments over the LF dataset. Results are shown in Table 4. In the following, we describe each of the experiments in more detail.

We remove the entropy term in Eq. ([eq:kl_fnerf]) and train our CF-NeRF only using the NLL as the training loss. On both generated RGB images and depth-maps, we achieve better performance by using the Entropy term as well across all metrics, including the prediction error and its associated uncertainty. This is consistent with what we have discussed in Sec. [sec:optimization] that, maximizing the Entropy term intuitively prevents the optimized distribution to degenerate into a deterministic function where all the probability is assigned into a single radiance field \(\mathcal{F}\), thus losing the ability to quantify correct uncertainty.

As discussed in Sec. [sec:cf_nerf], our CF-NeRF uses two conditional normalizing flows(CNF) for modelling the distribution of radiance and density. However, a more efficient strategy could be to jointly model their distributions using a single flow in order to take into account the possible dependence between them. As we can see in Table 4, this variant obtains worse performance compared to our CF-NeRF with two CNFs in terms of prediction quality and uncertainty estimation. This drop in performance is especially high in the case of depth-map estimation. This can be explained because using a single CNF for radiance and density distribution contradicts the fact that the volume density must be independent of the emitted radiance to obtain optimal results, as was previously discussed in [11], [25].

Table 4: Results of our ablation studies: Quality and uncertainty quantification metrics on rendered images and depth-maps over LF dataset. Best results are shown in bold. See text for more details.

	Methods		Quality Metrics				Uncertainty Metrics
			PSNR\(\uparrow\)	SSIM\(\uparrow\)	LPIPS\(\downarrow\)		AUSE RMSE\(\downarrow\)	AUSE MAE\(\downarrow\)	NLL\(\downarrow\)
RGB images	CF-NeRF w/o Entropy		23.40	0.81	0.258		0.068	0.048	-0.448
	CF-NeRF w/ Single Flow		23.82	0.83	0.228		0.081	0.039	-0.578
	CF-NeRF		24.78	0.86	0.168		0.051	0.026	-0.710
			RMSE\(\downarrow\)	MAE\(\downarrow\)	\(\delta_3\) \(\uparrow\)		AUSE RMSE\(\downarrow\)	AUSE MAE\(\downarrow\)	NLL\(\downarrow\)
Depth	CF-NeRF w/o Entropy		0.121	0.078	0.76		0.224	0.143	7.88
	CF-NeRF w/ Single Flow		0.170	0.111	0.64		0.229	0.138	8.16
	CF-NeRF		0.118	0.074	0.81		0.110	0.071	5.09

9 More Results↩︎

9.0.1 Interpolation videos↩︎

An intuitive advantage of the explicit distribution modelling over the radiance fields in our CF-NeRF is that, we can conveniently analyze the learned radiance fields by interpolating in the latent space  [73], [74]. The shared latent variable allows to model the joint distribution of all the radiance-density pairs in the scene in contrast to S-NeRF and hence could avoid the noisy results as discussed in Sec. [sec:intro] and illustrated in Fig. 2. More formally, we define the interpolation value as, \[f_{L}(\boldsymbol{z}_1,\boldsymbol{z}_2,\lambda) = \lambda \boldsymbol{z}_1 + (1-\lambda) \boldsymbol{z}_2\] where \(\boldsymbol{z}_1\) and \(\boldsymbol{z}_2\) are two random samples from the latent distribution with \(\lambda \in [0,1]\). Then the density and radiance can be obtained through our proposed CNF, following our inference process as discussed in Sec. [sec:inference] to render novel views and depth. To see the dynamic interpolation results we provide a video attached to this supplementary material. By looking at different frames in the dynamic interpolated results, S-NeRF tends to generate noisy image and depth predictions with random and incoherent changes between adjacent frames obtained using two adjacent interpolation values. In contrast, our CF-NeRF can generate more coherent and smoothly changing frames, both on rendered RGB images and estimated depth-maps. This clearly demonstrates the advantages of our proposed Latent Variable Modelling for CF-NeRF in order to efficiently model the joint distribution over all the possible radiance and density pairs in the scene.

9.0.2 Sparsification plots↩︎

Fig. [fig:ause_rgb] shows the additional related sparsification curves on the synthetic novel views and estimated depth averagely over the LF dataset. Note that NeRF-W is not able to estimate uncertainty on depth as analyzed in Sec. [sec:related_work] and hence cannot generate the sparsification curve on depth. When evaluated over all pixels, all methods perform similarly. As we remove the pixels with high uncertainty from 1\(\%\) to 100\(\%\), our method always obtains the lowest value and fits closest with the oracle curve. This demonstrates that our estimated uncertainty correlates significantly better with the prediction error than the others.

9.0.3 More qualitative results↩︎

Fig. [fig:supplem_llff] shows more qualitative results obtained by our CF-NeRF for the scenes in the simple LLFF dataset. Moreover, Fig. [fig:supplem_lf] shows additional qualitative results obtained by our CF-NeRF across other scenes in the LF dataset: Africa, Statue, Torch. For each scene, we show not only the predicted RGB views and the estimated depth-maps, but also their associated uncertainty estimations.

References↩︎