1 Introduction↩︎

Neural Fields [1] parameterize implicit functions over continuous domains using neural networks and have gained popularity due to their ability to represent various types of signals. To represent arbitrary high-frequency functions, positional encoding [2], Fourier features [3], Gaussian activations [4], [5], among other alternatives [6]–[8], are used to overcome the spectral bias [9]. Compared to their explicit counterparts, implicit representations are more memory-efficient and can achieve higher quality, but regrettably suffer from slower inference. In addition, as the signal is encoded in the weights and biases of a neural network, every change to a parameter has a non-local effect, which makes implicit representations hard to modify.

For novel view synthesis problems, the 5D plenoptic function mapping positions and directions to radiance and density can be learned from posed images leveraging a neural field. NeRFs [2] have recently demonstrated impressive performance for real-world and synthetic datasets. However, the naïve approach of sampling the scene densely is slow and inefficient due to the prohibitive number of required samples. A coarse-to-fine sampling strategy is adopted to overcome these limitations, estimating the density distribution along a ray by uniformly sampling a coarse NeRF, which generates a new set of samples for a subsequent fine NeRF using inverse transform sampling. Although performance is improved substantially, the number of required samples still prohibits real-time rendering. Furthermore, as no ground-truth density information is available, the coarse NeRF is supervised with an MSE reconstruction loss.

Reducing the number of required samples for NeRFs is a fruitful endeavor to lower the computational requirements. Sampling networks [10]–[13], which predict volumetric density along a ray to produce a set of samples for a subsequent shading network, prove highly effective for this task. The network capacity for a sampling network is significantly reduced, as solely predicting volumetric density is view-independent and thus easier to learn. For supervising these sampling networks ground-truth depth data [13], a pre-trained NeRF [14], multi-view stereo [15] or distillation from a concurrently trained shading network [10]–[12] may be used. Alternatively, occupancy grids [16] allow for efficient empty-space skipping by encoding the expected density within a discretized region.

A clear disadvantage of the coarse-to-fine NeRF pipeline is the use of the coarse model for density prediction only, even though it is optimized to reconstruct the ground-truth image. Sampling networks, on the other hand, map sample positions to volumetric density using a network with reduced capacity and can be supervised by the shading network. Given that sampling networks show that density can be modelled with lower capacity, a subset of a coarse NeRF should be able to model density accurately as well — particularly since recent work has found that NeRF-MLPs naturally partition themselves into structural and color layers [17].

To this end, we analyze the intermediate activations of coarse NeRFs: We derive a representation from these activations and show how it is tied to the density distribution along a view ray. Building on this crucial finding, we devise a novel method to extract density estimates given intermediate activations of coarse NeRFs. By elegantly combining activation analysis and our method to extract densities, our approach can reduce inference times while maintaining high quality at test-time. We demonstrate that our approach is not only applicable to NeRF and Mip-NeRF [18], but produces similar results when applied to proposal network samplers as well, with extensive evaluation supporting our claim. To the best of our knowledge, our approach is the first to use intermediate activations to accelerate inference for coordinate-based MLPs.

To summarize, our contributions are as follows:

• We present a novel method for visualizing and analyzing the activations of coordinate-based ReLU-MLPs.

• We present an approach for extracting a density estimate for fine re-sampling from activations in early layers of coarse NeRFs, significantly reducing the inference time with little decrease in quality. Consequently, our method provides valuable insight into the inner workings of NeRFs.

• We demonstrate the effectiveness of our method for several architectures with real-world and synthetic data.

2 Related Work↩︎

In the following, we review related work, focusing on real-time rendering and efficient sampling for NeRFs.

2.0.0.1 NeRFs for Real-time Rendering.

Instead of the implicit representation of radiance fields used in NeRF [2], explicit representations with grid-like data structures [19], [20] or hybrid representation [16], [21]–[24] are often used to accelerate inference and training. Most prominently of the aforementioned, Instant-NGP [16] replaces the fixed positional encoding used by NeRF with a learnable hash encoding, allowing for the use of much smaller MLPs and thus rapid convergence. Alternatively, 3D Gaussian Splatting [25] has recently demonstrated state-of-the-art rendering quality in addition to fast training and rendering using a fully explicit representation relying on anisotropic 3D Gaussians. Other works [26], [27] suggest a divide-and-conquer approach, utilizing many tiny MLPs responsible for a small region instead of a large MLP for the whole scene. Distillation of a NeRF into a render-friendly format has been explored extensively [28], [29], but these approaches are memory-intensive and require an additional distillation step after training. Recent work [30], [31] has also investigated approximate geometry equipped with neural features and small, global decoders. Finally, mesh-based view synthesis [32]–[34] has recently demonstrated real-time rendering on low-power devices, relying on approximate meshes extracted from radiance fields and the well-known graphics pipeline.

2.0.0.2 Efficient Sampling.

Several works investigate a learnable proposal sampler, which can predict volumetric density [11] or sample locations [10]. DONeRF [13] learns a density volume as a multi-class classification task to guide sample placement with direct depth supervision. Building on DONeRF, AdaNeRF [12] mitigates the requirement of pre-training and depth supervision for the sampling network with a 4-phase training recipe. EfficientNeRF [35] proposes valid sampling, caching densities of sample positions and exclusively evaluating those with positive density — this approach shares similarities with our method in that it is motivated by the density distribution of coarse NeRFs. TermiNeRF [14] distills a NeRF into a sampling and shading network. For their sampling network, they perform distribution matching, effectively reparameterizing NeRFs intervals to allow for fewer samples. Lin et al. [15] learn a proxy geometry with a cost volume for depth guidance, uniformly sampling a smaller depth range with a lower sample count. A better sampling strategy is also attained by unifying implicit surface models and radiance fields [36]. To enable mesh-based view synthesis, SDFs [32] or Binary Opacity Grids [34] can accurately locate surfaces. Finally, by restricting volume rendering to a small band around the surface [37], the number of samples can be reduced significantly.

3 Preliminaries↩︎

In the following section, we briefly summarize the most important architectures and concepts which build the foundation for our work.

3.1 NeRF↩︎

NeRFs [2] learn a function \[\Theta_{\text{NeRF}}: \mathbb{R}^5 \rightarrow \mathbb{R}^4,\;({\mathbf{p}}, {\mathbf{d}}) \mapsto ({\mathbf{c}}, \sigma),\] where \({\mathbf{p}} = (x,y,z)\) and \({\mathbf{d}} = (\theta, \phi)\) are the sample position and viewing direction, \({\mathbf{c}} \in [0,1]^3\) denotes the predicted output color and \(\sigma \in \mathbb{R}\) is the volumetric density. NeRF uses a ReLU-MLP with 8 layers and \(56\) hidden units to predict a feature vector \({\mathbf{f}} \in \mathbb{R}^{256}\) and \(\sigma\). This feature vector is then concatenated with \({\mathbf{d}}\) and passed through a small ReLU-MLP to produce \({\mathbf{c}}\), which allows view-dependent phenomena to be modelled. For each pixel, we cast a ray \({\mathbf{r}}(t) = {\mathbf{o}} + t{\mathbf{d}}\) from the origin \({\mathbf{o}}\) in the direction of \({\mathbf{d}}\). We evaluate \(\Theta_{\text{NeRF}}\) at different sample positions \(t \in [t_n, t_f]\) between the near bound \(t_n\) and the far bound \(t_f\). The color \(\hat{\mathcal{C}}\) of a ray \({\mathbf{r}}\) can be estimated using \(N_s\) samples with \[\hat{\mathcal{C}}({\mathbf{r}}) = \sum_{i=1}^{N_s} T_i (1 - \exp({-\sigma_i \delta_i})){\mathbf{c}}_i,\] where \(\delta_i\) and \(T_i\) are given as \[\delta_i = t_{i+1}-t_i, \quad T_i = \exp{\left(- \sum_{j=1}^{i-1} {\sigma_j \delta_j}\right)}.\] To increase rendering efficiency, two NeRFs are concurrently optimized. The coarse NeRF \(\Theta_{\text{coarse}}\) places \(N_c\) samples uniformly for each ray \({\mathbf{r}}\), which produces a set of weights \({\mathbf{w}} \in \mathbb{R}^{N_c}\) with \[\mathrm{w}_i = { T_i(1 - \exp(-\sigma_i \delta_i))}, \label{eq:3:weights}\tag{1}\] used to express the estimated coarse radiance as a weighted sum \[\hat{\mathcal{C}}_c({\mathbf{r}}) = \sum_{i=1}^{N_c} \mathrm{w}_i {\mathbf{c}}_i.\] The weights \({\mathbf{w}}\) are normalized such that \({\mathbf{1}}^T {\mathbf{w}}= 1\) to obtain a piecewise-constant PDF along \({\mathbf{r}}\). This distribution is used to produce a new set of \(N_f\) samples with inverse transform sampling. The final rendered color is produced by \(\Theta_{\text{fine}}\) using all \(N_c + N_f\) samples.

NeRF applies positional encoding \(\gamma(\cdot)\) to \({\mathbf{p}}\) and \({\mathbf{d}}\), allowing the network to model high-frequency details effectively [3]. The network is supervised with the MSE loss for both \(\Theta_{\text{coarse}}\) and \(\Theta_{\text{fine}}\), i.e. \(\sum_{{\mathbf{r}}}\|\hat{\mathcal{C}}_c({\mathbf{r}}) - {\mathcal{C}}({\mathbf{r}})\|_2^2 + \|\hat{\mathcal{C}}_f({\mathbf{r}}) - {\mathcal{C}}({\mathbf{r}})\|_2^2\), with \(\mathcal{C}({\mathbf{r}})\) denoting the ground truth color for \({\mathbf{r}}\) and \(\hat{\mathcal{C}}_f({\mathbf{r}})\) denoting the estimated fine radiance of \({\mathbf{r}}\).

3.2 Mip-NeRF↩︎

Mip-NeRF [18] builds on NeRF and tackles the problem of aliasing when scene content is observed at various resolutions. The standard computer graphics solution of super-sampling is prohibitively expensive when considering NeRFs volume rendering. Instead, Mip-NeRF introduces a novel Integrated Positional Encoding (IPE), which is the expected positional encoding of all coordinates within a conical frustum. Each conical frustum is approximated with a multivariate Gaussian, and the IPE feature serves as the encoded input for \(\Theta_{\text{NeRF}}\). This modification allows the MLP to reason about the size and shape of the encoded region, efficiently combating aliasing which occurs for ambiguous point samples.

As the network can now learn a multi-scale scene representation, \(\Theta_{\text{NeRF}}\) performs both coarse and fine sampling, reducing the model size by \(50\%\) — however, the network is still supervised with an MSE loss.

3.3 Mip-NeRF 360↩︎

NeRF and Mip-NeRF require that 3D coordinates exist in a bounded domain for sampling — an assumption which is violated in unbounded 3D scenes, where objects may exist at any distance from the camera. Therefore, Mip-NeRF 360 [11] proposes a spatial contraction \(\text{contract}(\cdot)\), which maps coordinates from \(\mathbb{R}^3\) to a ball of radius \(2\): \[\begin{align} \text{contract}({\mathbf{x}}) = \begin{cases} {\mathbf{x}} & \|{\mathbf{x}} \| \leq 1 \\ \left( 2 - \frac{1}{\|{\mathbf{x}} \|}\right)\left( \frac{{\mathbf{x}}}{\|{\mathbf{x}} \|}\right) & \|{\mathbf{x}} \| > 1 \\ \end{cases}. \end{align}\] To handle the increased capacity required for unbounded scenes and increase rendering efficiency, Mip-NeRF 360 concurrently optimizes two MLPs, \(\Theta_{\text{prop}}\) and \(\Theta_{\text{NeRF}}\). Their proposal network sampler \(\Theta_{\text{prop}}\) learns a function \[\Theta_{\text{prop}}: \mathbb{R}^3 \rightarrow \mathbb{R},\;{\mathbf{p}} \mapsto \sigma,\] and is optimized to bound the weights \({\mathbf{w}}\) produced by \(\Theta_{\text{NeRF}}\). Two subsequent rounds of proposal sampling produce two weights \(\{\hat{{\mathbf{w}}}_1, \hat{{\mathbf{w}}}_2\}\), which is efficient due to the small size of \(\Theta_{\text{prop}}\) with 4 layers and 256 hidden units.

4 Activation Analysis↩︎

In this section, we present our method for visualizing and analyzing the intermediate activations for ReLU-MLPs. In addition, we show how we can leverage these intermediate activations to accelerate inference for pre-trained NeRFs.

4.1 Visualizing and Analyzing Activations↩︎

Benefiting from the inherent spatial structure baked into their building blocks, several works [38], [39] investigate the visualization of intermediate features for Convolutional Neural Networks (CNNs). These methods typically operate by back-projection from feature space to pixel space, allowing informative insight into the inner workings of these models. For MLPs, which are mostly used as universal function approximators, practitioners have not been as interested in their activations. To this end, we propose a novel method for visualizing activations of coordinate-based MLPs.

We analyze the hidden layer activations of NeRF MLPs as follows: For each pixel \((x,y)\), a ray \({\mathbf{r}}\) is generated, positional encoding \(\gamma(\cdot)\) is applied and each sample along the ray is passed through \(\Theta_{\text{NeRF}}\). In the following, we describe the intermediate activations in terms of a single ray/pixel. After each linear layer, the activation \({\mathbf{A}}^{(\ell)}\) is of size \(\mathbb{R}^{N_s \times N_h}\), where \(N_s\) and \(N_h\) denote the number of samples and the number of hidden units, respectively. To obtain a feature representation \({\mathbf{f}}_{\ell}\in \mathbb{R}^{N_s}\) for the activation in layer \(\ell\), we compute the mean over the dimension \(N_h\): \[{\mathbf{f}}_{\ell}= \frac{1}{N_h}\;\sum_{i=1}^{N_h}\;{{\mathbf{A}}^{(\ell)}_{i}}. \label{eq:feature95per95sample}\tag{2}\] Further, we can produce a scalar \(v_{\ell}\in \mathbb{R}\) for each ray \({\mathbf{r}}\) for visualization purposes with \[v_{\ell}= \frac{1}{N_s}\;\sum_{i=1}^{N_s}\;\sum_{j=1}^{N_h}\;{{\mathbf{A}}^{(\ell)}_{i, j}}. \label{eq:activation}\tag{3}\] We visualize the activations of different layers \(\ell\) for some example views using Eqn. 3 in Fig. 1. As we can see, intermediate activations exhibit notable structural information. We note that when recording activations \({\mathbf{A}}^{(\ell)}\), we have the option to apply or not apply the activation function. Unless stated otherwise, we work with activations after the ReLU has been applied, thus \({\mathbf{A}}^{(\ell)} \in \mathbb{R}_+^{N_s \times N_h}\). Although unintuitive due to the information loss caused by the ReLU, early experiments showed that applying the activation function resulted in more consistent histograms for \({\mathbf{f}}_{\ell}\).

4.25pt

4.2 Reducing Unnecessary Network Evaluations using Activations↩︎

As a first step towards speeding up the coarse-to-fine NeRF rendering pipeline, we analyze the densities \(\sigma\) predicted by \(\Theta_{\text{coarse}}\). Take, for example, any of the synthetic scenes from the Blender dataset [2]. Along rays \({\mathbf{r}}\) in empty space, we do not expect any sample with significant volumetric density. We can sum the number of densities along each ray \({\mathbf{r}}\) larger than a small threshold \(\tau\) to obtain a mapping \(g: {\mathbf{r}} \mapsto \{0, \ldots, N_c\}\). If for any of the rays the result is \({0}\), the corresponding pixel likely belongs to the transparent background. We see in Fig. 2 that when the value of \(g\) is high, the value of \(v_{\ell}\) is low, and vice-versa.

For each ray \({\mathbf{r}}\), we record the predicted coarse radiance \(\hat{\mathcal{C}}_c\) and the activations \(v_{\ell}\) from \(\Theta_{\text{coarse}}\). We choose a threshold \(\tau\), which indicates whether we observe scene content along \({\mathbf{r}}\). We choose \(\tau = \mu({\mathbf{V}}_\ell)\), the mean activation of the activation image \({\mathbf{V}}_\ell\). We obtain a set of rays \(P = \{{\mathbf{r}}: v_{\ell}< \tau\}\), which we treat as a mask. If \({\mathbf{r}} \in P\), we evaluate \(\Theta_{\text{fine}}\). Otherwise, we treat the coarse radiance \(\hat{\mathcal{C}}_c\) as our final output. Using this approach, we can significantly reduce the number of evaluations for \(\Theta_{\text{fine}}\), which uses \(N_c + N_f = 192\) samples, which results in a significant performance improvement. As we cover most of the object in the scene with our derived mask \(P\), the image quality loss is negligible. We report the results for the lego scene from the Blender dataset [2] in Tab. 1 and visualize our approach in Fig. 2. This experiment demonstrates that early intermediate activations already contain notable density information and reveals an interesting connection between density and activations.

4.3 Approximating Density using Activations↩︎

Clearly, the approach from Sec. 4.2 heavily relies on the transparent background present in synthetic scenes. Real-world captures, on the other hand, often include highly detailed backgrounds. Therefore, we go beyond this basic approach by analyzing the activation features \({\mathbf{f}}_{\ell}\) along a ray \({\mathbf{r}}\). We accomplish this by examining histograms of the predicted density \(\sigma\) and the activation features \({\mathbf{f}}_{\ell}\), as can be seen in Fig. 3. We notice a trend throughout multiple scenes: Activation feature space minima indicate samples with significant density. If we can find a function to transform \({\mathbf{f}}_{\ell}\) to a reasonable density estimate \(\hat{{\mathbf{d}}}\), we can effectively replace the expensive evaluation for \(\Theta_{\text{coarse}}\) with a pass through a smaller MLP. Ideally, this function should transform \({\mathbf{f}}_{\ell}\) to a vector of mostly zeros, except for locations where \(\sigma\) is significant. To this end, we propose 3 different functions \(f_i\) to map the intermediate activations directly to estimated densities \(\hat{{\mathbf{d}}}\), which were handcrafted based on the observed relationship between activations and density: \[\begin{align} f_1\left( {\mathbf{f}}_{\ell}\right) &= \text{ReLU} \left( \left(\mu({\mathbf{f}}_{\ell}) - \sigma({\mathbf{f}}_{\ell})\right) - {\mathbf{f}}_{\ell}\right), \tag{4}\\ f_2\left( {\mathbf{f}}_{\ell}\right) &= \text{ReLU} \left( \left(\mu({\mathbf{f}}_{\ell}) - \frac{\sigma({\mathbf{f}}_{\ell})}{2}\right) - {\mathbf{f}}_{\ell}\right), \tag{5}\\ f_3\left( {\mathbf{f}}_{\ell}\right) &= \text{ReLU} \left( \left(\mu({\mathbf{f}}_{\ell}) - \frac{\sigma({\mathbf{f}}_{\ell})}{2}\right) - {\mathbf{f}}_{\ell}\right)^2, \tag{6} \end{align}\] where \({\mathbf{f}}_{\ell}\) denotes the intermediate activation feature for layer \(\ell\) and \(\mu(\cdot),\;\sigma(\cdot)\) denote the mean and standard deviation along the last dimension (i.e. along the \(N_s\) samples of the ray \({\mathbf{r}}\)). We visualize our approach for a toy example in Fig. [fig:act95informed].

Using Eqn. 4 results in fewer intervals with density along the ray, but is prone to failures if the distribution along a ray exhibits high standard deviation. On the other hand, Eqns. 5  and 6 extract density estimates more conservatively but might lead to insufficient detail, as the interval between fine samples grows larger.

Given a well-behaved distribution, we can estimate densities sufficiently close to \(\sigma\) such that we get good reconstruction quality when converting \(\hat{{\mathbf{d}}}\) to a weight estimate \(\hat{{\mathbf{w}}}\). Further, we can significantly reduce the required inference time by using only a small subset of the large \(\Theta_{\text{coarse}}\) MLP. Clearly, the potential speedup is dependent on the number of samples \(N_s\), which is large for NeRFs.

4.3.0.1 Our Proposed Method for Weight Estimation.

First, we extract the activation features \({\mathbf{f}}_{\ell}\) and apply a function \(f_i\) to transform this to a density estimate \(\hat{{\mathbf{d}}}\) for each ray \({\mathbf{r}}\). To obtain our estimated weights \(\hat{{\mathbf{w}}}\), which should conform to a piecewise-constant PDF along the ray \({\mathbf{r}}\), we use \(\hat{w}_i = \hat{d}_i / \sum_{j=1}^{N_s}\hat{d}_j\) and subsequently perform inverse transform sampling, as done in NeRF [2] (c.f. Fig. [fig:act95informed]).

5 Experiments↩︎

In the following, we apply our presented method to NeRF [2] and Mip-NeRF [18]. Additionally, we apply our method to proposal network samplers, utilizing a nerfacto model [40] with a Mip-NeRF 360 [11] proposal sampler, which we dub \(\text{nerfacto}^\dagger\). We only apply our density extraction method to the first 3 layers: Recent work by Wang et al. [17] has found that NeRF MLPs naturally store structural information within the first MLP layers. In addition, when applying our method to earlier layers, we can accelerate inference more substantially.

5.0.0.1 Datasets.

For our evaluation, we use four diverse datasets, which are well-established in the novel view synthesis literature: The Blender dataset [2] consists of \(360^\circ\) captures of synthetic objects with complex geometry, realistic non-Lambertian effects and transparent backgrounds in a bounded domain. LLFF [41] is a dataset of forward-facing, real-world scenes in high resolution. The Mip-NeRF 360 dataset [11] contains \(360^\circ\) captures of real-world, unbounded scenes and is the most challenging benchmark in our evaluation. Finally, we also use 2 large-scale, bounded indoor scenes from Deep Blending [42] (playroom & drjohnson). For both LLFF and Mip-NeRF 360, we use the \(4\times\) downsampled images. We report per-scene results in the supplementary material.

5.0.0.2 Implementation Details.

We use the NeRF and Mip-NeRF implementations from Nerfstudio [40] and use the hyperparameter settings as reported in the respective works [2], [18]. To facilitate sampling in NDC, as NeRF does for LLFF [41], we implement a custom LLFF dataloader adapted from the nerf-pytorch codebase [43].

For our experiments with proposal network samplers, we adapt the nerfacto model from Nerfstudio [40]. This architecture unifies recent NeRF-related research [11], [16], [44]–[46] into a single, universally-applicable method. For our \(\text{nerfacto}^\dagger\) model, we replace the HashMLPDensityField with a 4-layer ReLU-MLP with 256 hidden units, following Mip-NeRF 360 [11], and use positional encoding [2]. Note that this configuration is slower than nerfacto due to the large MLP, however, it enables experiments with our proposed method for proposal network samplers.

5.1 Speeding up NeRF and Mip-NeRF↩︎

2.8pt

We evaluate our approach for NeRF and Mip-NeRF using the Blender dataset [2] and the LLFF dataset [41]. For the NeRF experiment, we could either record the activations from \(\Theta_{\text{coarse}}\) or \(\Theta_{\text{fine}}\). However, only \(\Theta_{\text{coarse}}\) is trained for uniform samples, hence we use the coarse MLP, which also performed better in our experiments.

We report quantitative results in Tab. ¿tbl:tab:results? and show our visual results in Fig. 4. For NeRF, the performance gain is not as significant as for Mip-NeRF due to the different number of samples \(N_c, N_f\): For Mip-NeRF, \(N_c = N_f = 128\), contrary to \(N_c=64, N_f = 128+64\) for NeRF. Hence, our approach performs favorably for Mip-NeRF due to its more expensive coarse network evaluation. We find that Eqn. 5 for \(\ell=2\) performs best in our experiments. All run-time results for NeRF and Mip-NeRF were obtained on an NVIDIA Quadro RTX 8000.

5.2 Speeding up Proposal Networks↩︎

First, we evaluate our approach for \(\text{nerfacto}^\dagger\) using the Blender dataset [2] and the LLFF dataset [41]. For our experiments, we approximate the first iteration of proposal sampling: As the proposal sampler iterations use \(\{256, 96\}\) samples, respectively, approximating the first round yields more performance improvements¹. In addition, our functions \(f_i\) are biased towards uniform or piecewise-linear sample placement, and lead to unfavorable results when interval sizes vary significantly.

We report quantitative results in Tab. ¿tbl:tab:results? and show our visual results in Fig. 4. For the \(\text{nerfacto}^\dagger\) model, we observe a huge performance gain for the first layer, which declines sharply — this is due to the large number of samples for the first proposal sampler iteration and the small shading network. We find that Eqn. 5 applied in layer \(\ell=2\) performed best when considering standard image metrics. The reported run-time results for \(\text{nerfacto}^\dagger\) were obtained on an NVIDIA RTX 2070 Super.

Further, we evaluate our method on unbounded, real-world scenes, leveraging the Mip-NeRF 360 [11] and Deep Blending [42] datasets. We adapt \(\text{nerfacto}^\dagger\) in the following ways: We use \(7 \times 10^{5}\) training iterations, disable the appearance embedding, camera optimizer, distortion loss and the collider. We present our results in Tab. 2 and show example outputs in Fig. 5. Compared to the results presented for the LLFF dataset [41], our method performs less favorably in this scenario. We attribute this to the large sampling domain in these datasets, which requires more precise density estimates. Performance-wise, we observe the same trends as for the previous experiments. We used an NVIDIA RTX 4090 for these two datasets.

2.8pt

5.3 Ablation studies↩︎

To support our quantitative evaluation, we conduct additional ablation studies to analyze the effect of varying network capacity and the use of the ReLU function in our proposed functions \(f_i\). More details and ablation studies are provided in the supplementary material.

5.3.0.1 Varying Network Capacity.

To investigate the effect of network capacity, we vary the number of layers for the proposal network sampler of \(\text{nerfacto}^\dagger\) using the Mip-NeRF 360 dataset [11]. We report results in Tab. 3 — remarkably, lower network capacity allows for better visual quality retention for our best configuration. As the decomposition property of NeRFs [17] does not apply to proposal network samplers, lower network capacity forces the proposal network sampler to learn a more informative latent representation of density. We report results for all configurations of \(\ell\) and \(f_i\) in the supplementary material — the configuration \(\ell=1\) and \(f_2\) performs best for all tested network capacities.

4pt

5.3.0.2 ReLU or no ReLU.

We also conduct an additional ablation study to test the effectiveness of applying the ReLU in Eqns. 4 , 5 and 6 : We remove the ReLU for all \(f_i\), use the \(\text{nerfacto}^\dagger\) model for the Mip-NeRF 360 dataset [11] and report the results in Tab. 4.

4.5pt

As we can see, not applying the ReLU leads to worse outputs for every configuration we tested. Empirically, this is caused by the more inconsistent histograms and larger standard deviation \(\sigma({\mathbf{f}}_{\ell})\). We note that the configuration \(\ell=1, f_2\) performs best for both variants of our method.

6 Limitations↩︎

Our method is not without its limitations. For certain test set views, using Eqn. 4 results in rays \({\mathbf{r}}\) with \(\hat{{\mathbf{w}}} = \mathbf{0}\), which does not produce a good set of samples for the subsequent shading pass. We indicate these cases in Tab. ¿tbl:tab:results? and show examples in Fig. 6. These failures are particularly evident for Mip-NeRF, as the fine pass does not include the uniformly distributed samples. Eqns. 5  and 6 mitigate this issue effectively, but might lead to outputs with less detail due to their more conservative formulation.

Although our approach performs well for a variety of architectures and datasets, extending it to hybrid or explicit representations such as [16], [19], [20], [24] is not trivial, as these approaches replace the MLP with more efficient alternatives. Further, our handcrafted functions for density estimation were created intuitively and are biased towards uniform or piecewise-linear sample placement.

7 Conclusion and Outlook↩︎

We have presented a generally applicable framework to analyze and visualize the intermediate activations of coordinate-based ReLU-MLPs. Building on our findings, we proposed a novel method to obtain density estimates using our derived activation features. We demonstrated that our approach produces faithful renderings with improved performance and provided valuable insight into the internals of neural fields. We believe our concepts could be applied to other tasks involving coordinate-based MLPs.

8 Additional Implementation Details for nerfacto\(^\dagger\)↩︎

8.0.0.1 Blender.

For the Blender dataset [2], we set \(t_n = 2.0,\;t_f = 6.0\) for all scenes. We disable the camera optimizer, the appearance embedding, the distortion loss and the scene contraction. We manually set the background color to white. For all scenes, we use a batch size of \(2048\) rays during training and use a uniform initial proposal network sampler. We train \(\text{nerfacto}^\dagger\) for \(16.5 \times 10^{3}\) epochs, as in [40].

8.0.0.2 LLFF.

For the LLFF dataset [41], we disable the camera optimizer, the appearance embedding and the distortion loss. We use a \(\ell_\infty\) scene contraction as default for nerfacto [40], contrary to the use of NDC coordinates in NeRF and Mip-NeRF. We do not use our custom LLFF dataparser, we instead opt for the nerfstudio dataparser, which requires pre-processing the data. For all scenes, we use a batch size of \(1024\) rays during training and a batch size of \(2048\) during evaluation. We train \(\text{nerfacto}^\dagger\) for \(30 \times 10^{3}\) epochs, as in [40]. For the leaves scenes, we use a uniform initial proposal network sampler, as the piecewise-uniform sampler did not result in stable optimization.

8.0.0.3 Mip-NeRF 360 and Deep Blending.

For Mip-NeRF 360 [11] and Deep Blending [42], we use the Nerfstudio data processing pipeline [40] to extract the dataset using the given SfM output. As we use an MLP with NeRFs positional encoding [2], the hyperparameters reported in [40] do not necessarily provide the best results. We experimented with different hyperparameters and ultimately used the configuration reported in the main paper, as it resulted in the best performance for \(\text{nerfacto}^\dagger\). Also, note that the results for Mip-NeRF 360 reported in [40] do not include the scenes flowers and treehill, for which \(\text{nerfacto}^\dagger\) exhibits the worst image metrics (c.f. Sec. 10).

8.0.0.4 Comparison with nerfacto.

To investigate the effect of the MLP introduced for \(\text{nerfacto}^\dagger\), we compare the results with nerfacto (with HashMLPDensityField) in Tab. 5. We use the Mip-NeRF 360 dataset [11] and use the same hyperparameter settings as reported in the main paper. Both variants achieve comparable results, although nerfacto is significantly faster.

9 Ablation Studies↩︎

First, we present an extensions to our method, which reduces processing time even further. In addition, we also report more detailed results for the ablation studies in Sec. 5.3.

5pt

9.1 Upsampling Activations↩︎

As our approach builds on the intermediate activations for a proposal of density along a ray, a natural extension to our method is to obtain the activations \({\mathbf{A}}^{(\ell)}\) at a lower resolution. To this end, we perform an ablation study, where we record \({\mathbf{A}}^{(\ell)}\) at a resolution of \(\frac{w}{2} \times \frac{h}{2} \times \frac{N_c}{2} \times N_h\), then perform upsampling to obtain a full-resolution activation \({\mathbf{A}}^{(\ell)}\). As our interpolation method, we choose nearest neighbor upsampling due to its negligible cost. Similarly to this approach, DONeRF [13] filters along the depth axis and across neighboring rays to obtain a smoothed depth target for their oracle network. For the Blender dataset [2] and the LLFF dataset [41], we use our method with \(\ell =1\) and \(f_2\) and report results in Tab. 6. We perform this experiments for NeRF [2] and Mip-NeRF [18]: As we can see, this modifications impacts visual quality only slightly.

2.7pt

9.2 Impact of Capacity↩︎

We provide the full results for our capacity experiment from the main material in Tab. 7. As we can see, layer \(\ell = 1, f_2\) is the best configuration for our method across various scenarios. This further indicates that capacity has minimal impact on our approach.

9.3 ReLU or no ReLU↩︎

We provide two example renderings comparing our method with the variant without the ReLU in Fig. 7. As can clearly be seen, not using the ReLU leads to more artefacts, particularly around sharp edges, causing the drop in all image metrics. This is caused by the more inconsistent histograms and the larger standard deviation.

10 Per-Scene Results↩︎

In Tabs. 8 and 9, we show per-scene results for the Blender dataset [2] and the LLFF dataset [41]. For the drums scene, we manage to outperform Mip-NeRF quantitatively. In Tabs. 10 and 11, we also report the per-scene results for Mip-NeRF 360 [11] and Deep Blending [42].

5pt

References↩︎

---

1. Please note that \(\text{nerfacto}^\dagger\) is significantly slower than nerfacto due to the use of an MLP.↩︎