1 Introduction↩︎

Differentiable rendering shows great promise for accurate multi-view 3D reconstruction from image observations. NeRF [1] use differentiable volume rendering to create high quality view interpolation through neural, density-based light-fields. Surface-based methods apply signed distance fields [2]–[6] or triangle meshes [7], [8] to capture high quality geometry. Recent work [7], [9], [10] further decompose these representations into geometry, material, and environment light.

Most aforementioned methods rely on appearance baked into neural light fields or apply simple shading models. Typically, direct lighting without shadows is considered, combined with pre-filtered representations of environment lighting [6], [7]. Some methods account for shadowing and indirect illumination [9]–[11], but often lock the geometry optimization when sampling the shadow term, or rely on learned representations of irradiance. While results are impressive, the deviations from physically-based shading models makes it harder for these methods to plausibly disentangle shape, material and lighting, as shown in Figure 1.

In theory, it is straightforward to replace the rendering engines of these 3D reconstruction methods with photorealistic differentiable renderers [13]–[18] and optimize in a setting with more accurate light simulation, including global illumination effects. In practice, however, the noise in multi-bounce Monte Carlo rendering makes gradient-based optimization challenging. Very high sample counts are required, which result in intractable iteration times.

In this paper, we bridge the gap between current multi-view 3D reconstruction and physically-based differentiable rendering, and demonstrate high-quality reconstructions at competitive runtime performance. We attack the challenging case of extracting explicit triangle meshes, PBR materials and environment lighting from a set of multi-view images, in a format directly compatible with current DCC tools and game engines. For improved visual fidelity, we compute direct illumination using Monte Carlo integration with ray tracing, and add several techniques to combat the increased noise levels. By carefully trading variance for bias, we enable efficient gradient-based optimization in a physically-based inverse rendering pipeline. Compared to previous 3D reconstruction methods, our formulation primarily improves material and light separation.

Concretely, we reduce variance by combining multiple importance sampling [19] and differentiable denoisers. We evaluate both neural denoisers [20]–[22] and cross-bilateral filters [23] in our pipeline. Furthermore, we decompose the rendering equation into albedo, demodulated diffuse lighting and specular lighting, which enable precise regularization to improve light and material separation.

2 Previous Work↩︎

2.0.0.1 Neural methods for multi-view reconstruction

These methods fall in two categories: implicit or explicit scene representations. NeRF [1] and follow-ups [24]–[33], use volumetric representations and compute radiance by ray marching through a neurally encoded 5D light field. While achieving impressive results on novel view synthesis, geometric quality suffers from the ambiguity of volume rendering [27]. Surface-based rendering methods [2]–[5] optimizing the underlying surface directly using implicit differentiation, or gradually morph from a volumetric representation into a surface representation. Methods with explicit representation estimate 3D meshes from images, where most approaches assume a given mesh topology [34]–[36], but recent work also include topology optimization [7], [8], [37], [38].

2.0.0.2 BRDF and lighting estimation

To estimate surface radiometric properties from images, previous work on BTF and SVBRDF estimation rely on special viewing configurations, lighting patterns or complex capturing setups [39]–[47]. Recent methods exploit neural networks to predict BRDFs from images [48]–[53]. Differentiable rendering methods [34]–[36], [54], [55] learn to predict geometry, SVBRDF and, in some cases, lighting via photometric loss.

Most related to our work are neural 3D reconstruction methods with intrinsic decomposition of shape, materials, and lighting from images [6], [7], [9]–[11], [56]. Illumination is represented using mixtures of spherical Gaussians [6], [7], [9], [11], pre-filtered approximations [7], [56], or low resolution environment maps [10]. When the shadowing term is accounted for [10], optimization is split into two passes where geometry is locked before the shadow term is sampled. Other approaches represent indirect illumination with neural networks [11], [57].

2.0.0.3 Image denoisers

Denoisers are essential tools in both real-time- and production renderers. Traditionally, variants of cross-bilateral filters are used [58], which require scene-specific manual adjustments. More recently, neural denoisers [20]–[22] trained on large datasets have shown impressive quality without the need for manual tuning, and are now incorporated in most production renderers. We directly incorporate differentiable versions of these denoisers in our pipeline. We are currently unaware of image denoisers applied in differentiable rendering, but we see a lot of potential for denoisers in physically-based inverse rendering going forward.

3 System↩︎

We target the challenging task of joint optimization of shape, material and environment lighting from a set of multi-view images with known foreground segmentation masks and camera poses. Our goal is to use physically-based rendering techniques to improve the intrinsic decomposition of lighting and materials, producing assets that can be relit, edited, animated, or used in simulation. As a proof-of-concept, we extend a recent approach, nvdiffrec [7], which targets the same optimization task (shape, materials and environment lighting). Notably, they directly optimize a triangular 3D model, which has obvious benefits: it is easy to import and modify the reconstructed models in existing DCC tools, and a triangular representation can exploit hardware-accelerated differentiable rasterization [59]. In our setting, triangular 3D models also means we can leverage hardware-accelerated ray-tracing for efficient shadow tests. nvdiffrec reports competitive results on view interpolation, material reconstruction, and relighting, and we will use their pipeline as a baseline in our evaluations.

Our system is summarized in Figure 2. A triangular mesh with arbitrary topology is optimized from a set of images through 2D supervision. Geometry is represented by a signed distance field defined on a three-dimensional grid and reduced to a triangular surface mesh through deep marching tetrahedra (DMTet) [8]. Next, the extracted surface mesh is rendered in a differentiable renderer, using the physically-based (PBR) material model from Disney [60]. This material model combines a diffuse term with an isotropic, specular GGX lobe [61]. A tangent space normal map is also included to capture high frequency shading detail. Finally, the rendered image is evaluated against a reference image using a photometric loss. In contrast to nvdiffrec, which uses a simple renderer with deferred shading and the split-sum approximation for direct lighting (without shadows), we instead leverage a renderer which evaluates the direct lighting integral using Monte Carlo integration and ray tracing (shadow rays). We represent the scene lighting using a high dynamic range light probe stored as a floating point texture, typically at a resolution of 256\(\times\)​256 texels. Finally, to combat the inherent variance that comes with Monte Carlo integration, we leverage differentiable image denoising and multiple importance sampling.

3.0.0.1 Optimization task

Let \(\phi\) denote the optimization parameters (shape, spatially varying materials and light probe). For a given camera pose, \(c\), our differentiable renderer produces an image \(I_{\phi}(c)\). Given that we use Monte Carlo integration during rendering, this image inherently includes noise, and we apply a differentiable image denoiser, \(D_{\theta}\), with parameters, \(\theta\), to reduce the variance, \(I_{\phi}^{\mathrm{denoised}}(c) = D_{\theta}(I_{\phi}(c))\). The reference image \(I_{\mathrm{ref}}(c)\) is a view from the same camera. Given a photometric loss function \(L\), we minimize the empirical risk \[\underset{\phi, \theta}{\mathrm{argmin}}\;\mathbb{E}_{c}\big[L\big(D_\theta(I_{\phi}(c)), I_{\mathrm{ref}}(c)\big)\big] \label{eq:optimization}\tag{1}\] using Adam [62] based on gradients w.r.t. the optimization parameters, \(\partial L/\partial\phi\), and \(\partial L/\partial\theta\), which are obtained through differentiable rendering. We use the same loss function as nvdiffrec. An example of the optimization process is illustrated in Figure 3,

3.1 Direct Illumination↩︎

The outgoing radiance \(L(\omega_o)\) in direction \(\omega_o\) can be expressed using the rendering equation [64] as: \[L(\omega_o) = \int_\Omega L_i(\omega_i)f(\omega_i,\omega_o) (\omega_i \cdot \mathbf{n}) d\omega_i. \label{eq:ibl}\tag{2}\] This is an integral of the product of the incident radiance, \(L_i(\omega_i)\) from direction \(\omega_i\) and the BSDF \(f(\omega_i, \omega_o)\). The integration domain is the hemisphere \(\Omega\) around the surface normal, \(\mathbf{n}\).

Spherical Harmonics (SH) [35] or Spherical Gaussians (SG) [6], [9] are often used as efficient approximations of direct illumination, but only work well for low- to medium-frequency lighting. In contrast, the split sum approximation [7], [65] captures all-frequency image based lighting, but does not incorporate shadows. Our goal is all-frequency lighting including shadows, which we tackle by evaluating the rendering equation using Monte Carlo integration: \[L(\omega_o) \approx \frac{1}{N} \sum_{i=1}^{N} \frac{L_i(\omega_i)f(\omega_i,\omega_o) (\omega_i \cdot \mathbf{n})}{p(\omega_i)},\] with samples drawn from some distribution \(p(\omega_i)\). Note that \(L_i(\omega_i)\) includes a visibility test, which can be evaluated by tracing a shadow ray in direction \(\omega_i\). Unfortunately, the variance levels in Monte Carlo integration with low number of samples makes gradient-based optimization hard, particularly with complex lighting. In Section 4 we propose several variance reduction techniques, which enable an inverse rendering pipeline that efficiently reconstructs complex geometry, a wide range of lighting conditions and spatially-varying BSDFs.

3.1.0.1 Shadow gradients

In single view optimization [66], shape from shadows [67], or direct optimization of the position/direction of analytical light sources, shadow ray visibility gradients [14], [68] are highly beneficial. However, in our multi-view setting (50+ views), similar to Loubet et al. [14], we observed that gradients of diffuse scattering are negligible compared to the gradients of primary visibility. Hence, for performance reasons, in the experiment presented in this paper, the shadow ray visibility gradients are detached when evaluating the hemisphere integral, and shape optimization is driven by primary visibility gradients, obtained from nvdiffrast [59].

4 Variance Reduction↩︎

We evaluate direct illumination with high frequency environment map lighting combined with a wide range of materials (diffuse, dielectrics, and metals). Strong directional sunlight, highly specular, mirror-like materials, and the visibility component can all introduce significant levels of noise. To enable optimization in an inverse rendering setting at practical sample counts, we carefully sample each of these contributing factors to obtain a signal with low variance. Below, we describe how we combat noise by using multiple importance sampling and denoising.

4.1 Multiple Importance Sampling↩︎

We leverage multiple importance sampling [19] (MIS), a framework to weigh a set of different sampling techniques to reduce variance in Monte Carlo integration. Given a set of sampling techniques, each with a sampling distribution \(p_i\), the Monte Carlo estimator for an integral \(\int_\Omega g(x) dx\) given by MIS is \[\sum_{i=1}^n \frac{1}{n_i} \sum_{j=1}^{n_i} w_i(X_{i,j})\frac{g(X_{i,j})}{p_i(X_{i,j})}, \;\;\;\;\; w_i(x) = \frac{n_i p_i(x)}{\sum_k n_k p_k(x)}.\] The weighting functions \(w_i(x)\) are chosen using the balance heuristic. Please refer to Veach’s thesis [19] or the excellent PBRT book [69] for further details.

In our case, we apply MIS with three sampling techniques: light importance sampling, \(p_\mathrm{light}(\omega)\), using a piecewise-constant 2D distribution sampling technique [69], cosine sampling, \(p_\mathrm{diffuse}(\omega)\), for the diffuse lobe, and GGX importance sampling [70], \(p_\mathrm{specular}(\omega)\), for the specular lobe. Unlike in forward rendering with known materials and lights, our material and light parameters are optimization variables. Thus, the sampling distributions, \(p_i\), are recomputed in each optimization iteration. Following the taxonomy of differentiable Monte Carlo estimators of Zeltner et al. [17], our importance sampling is detached, i.e., we do not back-propagate gradients to the scene parameters in the sampling step, only in the material evaluation. Please refer to Zeltner et al. for a careful analysis of Monte Carlo estimators for differentiable light transport.

MIS is unbiased, but chaining multiple iterations of our algorithm exhibits bias, as the current, importance sampled, iteration dictates the sampling distributions used in the next iteration. Light probe intensities, for example, are optimized based on an importance sampled image, which are then used to construct the sampling distribution for the subsequent pass. For unbiased rendering, the sampling distribution must be estimated using a second set of uncorrelated samples instead. Furthermore, we explicitly re-use the random seed from the forward pass during gradient backpropagation to scatter gradients to the exact same set of parameters that contributed to the forward rendering. This approach is clearly biased [71], but we empirically note that this is very effective in reducing variance, and in our setting this variance-bias trade-off works in our favor.

4.2 Denoising↩︎

For differentiable rendering, the benefits of denoising are twofold. First, it improves the image quality of the rendering in the forward pass, reducing the optimization error and the gradient noise introduced in the image loss. Second, as gradients back-propagate through the denoiser’s spatial filter kernel, gradient sharing between neighboring pixels is enabled. To see this, let’s consider a simple denoiser, \(O = X \circledast F\), where the noisy rendered image \(X\) is filtered by a low-pass filter, \(F\) (\(\circledast\) represents an image space 2D convolution). Given a loss gradient, \(\frac{\partial L}{\partial O}\), the gradients propagated back to the renderer are \(\frac{\partial L}{\partial X} = \frac{\partial L}{\partial O} \circledast F^T\), which applies the same low-pass filter in case the filter is rotationally symmetric (\(F = F^T\)). In other words, the renderer sees filtered loss gradients in a local spatial footprint. While denoisers inherently trade a reduction in variance for increased bias, we empirically note that denoising significantly helps convergence at lower sample counts, and help to reconstruct higher frequency environment lighting. We show an illustrative example in Figure 9.

Following previous work in denoising for production rendering [20], Figure 4 shows how we separate lighting into diffuse, \(\mathbf{c}_d\), and specular, \(\mathbf{c}_s\) terms. This lets us denoise each term separately, creating denoised buffers, \(D_{\theta}(\mathbf{c}_d)\), and \(D_{\theta}(\mathbf{c}_s)\). More importantly, we can use demodulated diffuse lighting, which means that the lighting term has not yet been multiplied by material diffuse albedo, \(\mathbf{k}_d\). In forward rendering, this is important as it decorrelates the noisy lighting from material textures, thus selectively denoising the noisy Monte-Carlo estimates and avoiding to low-pass filter high-frequent texture information. In inverse rendering, we can additionally use it to improve material and light decomposition by adding regularization on the lighting terms, as disucussed in Section 5. We compose the final image as \(\mathbf{c} = \mathbf{k}_d \cdot D_{\theta}(\mathbf{c}_d) + D_{\theta}(\mathbf{c}_s)\). We currently do not demodulate specular lighting because of the view dependent Fresnel term, but expect this to be improved in future work.

4.2.0.1 Cross-bilateral filters

Cross-bilateral filters are commonly used to remove noise in rendered images [58]. To evaluate this family of denoisers in our inverse pipeline, we adapted Spatio-temporal Variance-Guided Filtering [23] (SVGF), which is a popular cross-bilateral filter using surface normals and per-pixel depth as edge-stopping guides. The denoiser is applied to demodulated diffuse lighting, to avoid smearing texture detail. We disabled the temporal component of SVGF, as we focus on single frame rendering, and implemented the filter as a differentiable module to allow for loss gradients to propagate back to our scene parameters.

4.2.0.2 Neural denoisers

As a representative neural denoiser, we deploy the Open Image Denoiser (OIDN) [72], which is a U-Net [73] pre-trained on a large corpus of rendered images. The denoiser is applied to the rendered image before computing the image space loss. As the network model is fully convolutional, it is trivially differentiable, and we can propagate gradients from the image space loss, through the denoiser back to the renderer.

In Figure 5, we provide an overview on the effect of denoisers during 3D scene reconstruction. We observe that denoising is especially helpful at low sample counts, where we obtain similar reconstruction results at 8 spp with denoising, compared to 32 spp without a denoiser. At higher sample counts, however, the benefit from denoising diminishes, as variance in the Monte-Carlo estimates decreases. Given that denoisers enable significantly faster iteration times, we consider it a valuable tool for saving computational resources when fine-tuning model parameters for subsequent runs with high sample counts. Additionally, we empirically found that using a denoiser during reconstruction yields higher quality light probes, as can be seen in Figure 9, both at low and high sample counts.

We can also jointly optimize the denoiser parameters, \(\theta\), with the 3D scene reconstruction task. i.e., the denoiser is fine-tuned for the current scene. Unfortunately, this approach has undesirable side-effects: Features tend to get baked into the denoiser network weights instead of the materials or light probe. This is especially apparent with the OIDN [72] denoiser, which produced color shifts due to lack of regularization on the output. We got notably better results with the hierarchical kernel prediction architecture from Hasselgren et al. [22], which is more constrained. However, the results still lagged behind denoisers with locked weights. We refer to the supplemental material for details.

5 Priors↩︎

In our setting: 3D reconstruction from multi-view images with constant lighting, regularization is essential in order to disentangle lighting and materials. Following previous work [7], [10], we apply smoothness priors for albedo, specular, and normal map textures. Taking the albedo as an example, if \(\boldsymbol{k}_{d}\left(\mathbf{x}\right)\) denotes the diffuse albedo at world space position, \(\mathbf{x}\), and \(\mathbf{\epsilon}\) is a small random displacement vector, we define the smoothness prior for albedo as: \[L_{\boldsymbol{k}_{d}} = \sum_{\mathbf{x}_\text{surf}} \left|\boldsymbol{k}_{d}\left(\mathbf{x}_\text{surf}\right) - \boldsymbol{k}_{d}\left(\mathbf{x}_\text{surf} + \mathbf{\epsilon}\right)\right|,\] where \(\mathbf{x}_\text{surf}\) are the world space positions at the primary hit point on the object. We note that the smoothness prior is not sufficient to disentangle material parameters and light, especially for scenes with high frequency lighting and sharp shadows. Optimization tends to bake shadows into the albedo texture (easy) rather than reconstruct a high intensity, small area in the environment map (hard). To enable high quality relighting, we explicitly want to enforce shading detail represented by lighting, and only bake remaining details into the material textures. We propose a novel regularizer term that is surprisingly effective. We compute a monochrome image loss between the demodulated lighting terms and the reference image: \[L_\text{light} = |\mathrm{Y}\left(\mathbf{c}_d + \mathbf{c}_s\right) - \mathrm{V}\left(I_{\mathrm{ref}}\right)|.\] Here, \(Y\left(\mathbf{x}\right) = \left(\mathbf{x}_r + \mathbf{x}_g + \mathbf{x}_b\right) / 3\) is a simple luminance operator, and \(V\left(\mathbf{x}\right) = \max\left(\mathbf{x}_r, \mathbf{x}_g, \mathbf{x}_b\right)\) is the HSV value component. The rationale for using HSV-value for the reference image is that the \(\max\) operation approximates demodulation, e.g., a red and white pixel have identical values. We assume that the demodulated lighting is mostly monochrome, in which case \(Y\left(\mathbf{x}\right) \sim V\left(\mathbf{x}\right)\), and given that we need to propagate gradients to \(\mathbf{c}_d\) and \(\mathbf{c}_s\), \(Y\) avoids discontinuities. This regularizer is limited by our inability to demodulate the reference image. The HSV-value ignores chrominance, but we cannot separate a shadow from a darker material. This has not been a problem in our tests, but could interfere with optimization if the regularizer is given too much weight. Please refer to the supplemental materials for complete regularizer details.

6 Experiments↩︎

1mm

In our experiments, we use nvdiffrec [7] as a baseline, and refer to their work for thorough comparisons against related work. We focus the evaluation on the quality of material and light separation. At test time, all view interpolation results are generated without denoising at 2k spp. All relighting results are rendered in Blender Cycles at 64 spp with denoising [72]. Table 1 shows a quantitative comparison with nvdiffrec and NeRFactor [10] on the NeRFactor relighting setup. Note that there is an indeterminate scale factor between material reflectance (e.g., albedo) and the light intensity. This is accounted for by scaling each image to match the average luminance of the reference for the corresponding scene. The same methodology is applied for all algorithms in our comparisons. We outperform previous work, providing better material reconstruction. Figure 6 shows visual examples.

We additionally perform relighting on the synthetic NeRF dataset, which is substantially more challenging than the NeRFactor variant, due to high frequency lighting and global illumination effects. nvdiffrec produces severe artifacts, as exemplified by the Hotdog scene in Figure 6. Table 1 shows a significant increase in image quality for our approach. The visual examples show that our results are plausible, though not without artifacts. Finally, we constructed a novel synthetic dataset with three scenes with highly complex geometry to stress-test the system. Each scene contains 200 training views and 200 novel views for evaluation. The three scenes are shown in Figures 1 , 3 ,and 9. Quantitatively we outperform previous work by a larger margin, and Figure 1 shows very little shading in the albedo textures.

In Figures 7 and 8, we apply our method to datasets with real photos. These sets are more difficult, due to inaccuracies in foreground segmentation masks and camera poses. We extract triangle meshes that can be trivially edited in 3D modeling software, and in Figure 7 we use Blender to perform material editing and relighting with convincing results. Figure 8 shows a breakdown of geometry, material parameters, and environment light.

To study the impact of denoising, we optimized the Porsche scene w/ and w/o denoising. As shown in Figure 9, denoising improves both visual quality and environment lighting detail at equal sample counts. The noise levels varies throughout the scene, and we note that denoising is particularly helpful in regions with complex lighting or occlusion, such as the specular highlight and cockpit.

1mm

Neural light-fields, e.g., Mip-NeRF [76] excel at view interpolation. We enforce material/light separation through additional regularization, which slightly degrades view interpolation results, as shown in Table 2. Our scores are slightly below nvdiffrec, but, as shown above, we provide considerably better material and lighting separation.

6.0.0.1 Compute resources

Tracing rays to evaluate the direct illumination is considerably more expensive than pre-filtered environment light approaches. We leverage hardware-accelerated ray intersections, but note that our implementation is far from fully optimized. Our method scales linearly with sample count, which gives us a simple way to trade quality for performance. With a batch size 8 at a rendering resolution of 512\(\times\)​512, we get the following iteration times on a single NVIDIA A6000 GPU.

Our denoising strategies enable optimization at low sample counts. Unless otherwise mentioned, for the results presented in the paper, we use high quality settings of 128+ rays per pixel with 5000\(\times\)​2 iterations (second pass with fixed topology and 2D textures), which takes \(\sim\)​4 hours (A6000).

7 Conclusions↩︎

Our restriction to direct illumination shows up in scenes with global illumination. Similarly, we do not handle specular chains (geometry seen through a glass window). For this, we need to integrate multi-bounce path tracing, which is a clear avenue for future work, but comes with additional challenges in increased noise-levels, visibility gradients through specular chains, and drastically increased iteration times. Our renderer is intentionally biased to improve optimization times, but unbiased rendering could expect to generate better results for very high sample counts. Other limitations include lack of efficient regularization of material specular parameters and reliance on a foreground segmentation mask. Our approach is computationally intense, requiring a high-end GPU for optimization runs.

To summarize, we have shown that differentiable Monte-Carlo rendering combined with variance-reduction techniques is practical and applicable to multi-view 3D object reconstruction of explicit triangular 3D models. Our physically-based renderer clearly improves material and light reconstruction over previous work. By leveraging hardware accelerated ray-tracing and differentiable image denoisers, we remain competitive to previous work in terms of optimization time.

8 Optimization and Regularization↩︎

8.0.0.1 Image Loss

Our renderer uses physically-based shading and produces images with high dynamic range. Therefore, the objective function must be robust to the full range of floating-point values. Following recent work in differentiable rendering [7], [55], our image space loss, \(L_\mathrm{image}\), computes the \(L_1\) norm on tonemapped colors. As tone map operator, \(T\), we transform linear radiance values, \(x\), according to \(T(x) = \Gamma(\log(x + 1))\), where \(\Gamma(x)\) is the sRGB transfer function [77]: \[\begin{align} \Gamma(x) &=& \begin{cases} 12.92x & x \leq 0.0031308 \\ (1+a)x^{1/2.4} -a & x > 0.0031308 \end{cases} \\ a &=& 0.055. \nonumber \end{align}\]

8.0.0.2 Regularizers

While it is desirable to minimize regularization, in our setting with multi-view images with constant lighting, we have to rely on several priors to guide optimization towards a result with a good separation of geometry, materials, and lighting. As mentioned in the paper, we rely on smoothness regularizers for albedo, \(\boldsymbol{k}_{d}\), specular parameters, \(\boldsymbol{k}_{\mathrm{orm}}\) and geometric surface normal, \(\mathbf{n}\), following: \[L_{\boldsymbol{k}_{d}} = \frac{1}{\left|\mathbf{x}_\text{surf}\right|}\sum_{\mathbf{x}_\text{surf}} \left|\boldsymbol{k}_{d}\left(\mathbf{x}_\text{surf}\right) - \boldsymbol{k}_{d}\left(\mathbf{x}_\text{surf} + \boldsymbol{\epsilon}\right)\right|,\] where \(\mathbf{x}_\text{surf}\) is a world space position on the surface of the object and \(\boldsymbol{\epsilon} \sim \mathcal{N}(0,\sigma\!\!=\!\!0.01)\) is a small random displacement vector. Regularizing the geometric surface normal (before normal map perturbation) is novel compared to nvdiffrec and helps enforce smoother geometry, particularly in early training.

We note that normal mapping (surface normals perturbed through a texture lookup) also benefits from regularization. While normal mapping is a powerful tool for simulating local micro-geometry, the decorrelation of the geometry and surface normal can be problematic. In some optimization runs, we observe that with normal mapping enabled, the environment light is (incorrectly) leveraged as a color dictionary, where the normal perturbation re-orients the normals of a geometric element to look up a desired color. To prevent this behavior, we use the following regularizer. Given a normal perturbation \(\mathbf{n}'\) in tangent space, our loss is defined as \[L_{\mathbf{n}'} = \frac{1}{\left|\mathbf{x}_\text{surf}\right|}\sum_{\mathbf{x}_\text{surf}} 1 - \underbrace{ \frac{\mathbf{n}'\left(\mathbf{x}_\text{surf}\right) + \mathbf{n}'\left(\mathbf{x}_\text{surf} + \boldsymbol{\epsilon}\right)}{\left|\mathbf{n}'\left(\mathbf{x}_\text{surf}\right) + \mathbf{n}'\left(\mathbf{x}_\text{surf} + \boldsymbol{\epsilon}\right)\right|} }_{\textrm{Half-angle vector}} \cdot (0,0,1). \label{eq:normalsmooth}\tag{3}\] Intuitively, we enforce that normal perturbations, modeling micro-geometry, randomly selected in a small local area, have an expected value of the unperturbed tangent space surface normal, \((0,0,1)\).

As mentioned in the paper, we additionally regularize based on monochrome image loss between the demodulated lighting terms and the reference image: \[L_\text{light} = |Y\left(T(\mathbf{c}_d + \mathbf{c}_s)\right) - V\left(T(I_{\mathrm{ref}})\right)|_1.\] Here, \(T\) is the tonemap operator described in the previous paragraph. \(\mathbf{c}_d\) is demodulated diffuse lighting, \(\mathbf{c}_s\) is specular lighting, and \(I_{\mathrm{ref}}\) is the target reference image. Monochrome images are computed through the simple luminance operator, \(Y\left(\mathbf{x}\right) = \left(\mathbf{x}_r + \mathbf{x}_g + \mathbf{x}_b\right) / 3\), and HSV-value, \(V\left(\mathbf{x}\right) = \max\left(\mathbf{x}_r, \mathbf{x}_g, \mathbf{x}_b\right)\). While the regularizer is limited by our inability to demodulate the reference image, we show in Figure 10 that it greatly increases lighting detail, which is particularly effective in datasets with high frequency lighting. Figure 27 shows the impact of the regularizer on a larger set of scenes.

We compute the final loss as a weighted combination of the image loss and regularizer terms: \[L = L_\text{image} + \underbrace{\lambda_{\boldsymbol{k}_{d}}}_{=0.1}L_{\boldsymbol{k}_{d}} + \underbrace{\lambda_{\boldsymbol{k}_{\mathrm{orm}}}}_{=0.05}L_{\boldsymbol{k}_{\mathrm{orm}}} + \underbrace{\lambda_{\mathbf{n}}}_{=0.025}L_{\mathbf{n}} + \underbrace{\lambda_{\mathbf{n}'}}_{=0.25}L_{\mathbf{n}'} + \underbrace{\lambda_\text{light}}_{=0.15}L_\text{light},\] where we show the default weights used for most results (unless otherwise noted).

8.0.0.3 Optimization details

We use Adam [62] (default settings) to optimize parameters for geometry, material, and lighting. Referring to DMTet [8], geometry is parameterized as SDF values on a three-dimensional grid, with a perturbation vector per grid-vertex. Material and lighting are encoded in textures, or neural network (e.g., an MLP with positional encoding) encoded high-frequency functions. We typically use different learning rates for geometry, material, and lighting, with lighting having the highest learning rate and material the lowest. We closely follow the publically available nvdiffrec code base and refer to that for details. In Figure 11, we visualize two examples of optimizing the light probe, and note that we can capture high-frequency lighting details for specular objects.

We note that the early phases of optimization (starting from random geometry with large topology changes) are crucial in producing a good final result. The shadow test poses a challenge since changes in geometry may drastically change the lighting intensity in a previously shadowed area, causing the optimization process to get stuck in bad local minima. This is shown in Figure 12, where the optimizer fails to carve out the shape of the object. We combat this by incrementally blending in the shadow term as optimization progresses.

Recall from the main paper that we compute the color according to the rendering equation: \[L(\omega_o) = \int_\Omega L_i(\omega_i)f(\omega_i,\omega_o) (\omega_i \cdot \mathbf{n}) d\omega_i, \label{wfhtvldg}\tag{4}\] where \(L_i(\omega_i) = L_i'(\omega_i)H(\omega_i)\) can be separated into a lighting term, \(L_i'(\omega_i)\), and visibility term, \(H(\omega_i)\). Rather than using binary visibility, we introduce a light leakage term, \(\tau\), and linearly fade in the shadow contributions over the first 1750 iterations: \[\begin{align} H(\omega_i, \tau) &=& \begin{cases} 1-\tau & \text{if intersect\_ray}(\omega_i)\\ 1 & \text{otherwise} \end{cases}. \end{align}\]

In Figure 12 we note that gradually blending in the shadow term has a large impact on early convergence. In particular, since we start from a random topology, carving out empty space or adding geometry may have a large impact on overall shading, causing spiky and noisy gradients.

The denoiser may also interfere with early topology optimization (blurred visibility gradients). This is particularly prominent when the denoiser parameters are trained along with scene parameters. Therefore, we similarly, ramp up the spatial footprint, \(\sigma\), in the case of a bilateral denoiser. For neural denoisers, which have no easily configurable filter width, we instead linearly blend between the noisy and denoised images to create a smooth progression.

9 Denoising↩︎

9.0.0.1 Bilateral Denoiser

As a representative bilateral denoiser, we adapt the spatial component of Spatiotemporal Variance-guided Filtering [23] (SVGF), Our filter is given by \[\hat{\mathbf{c}}\left(p\right) = \frac{ \sum_{q \in \Omega} \mathbf{c}\left(q\right) \cdot w\left(p, q\right) }{ \sum_{q \in \Omega} w\left(p, q\right) },\] where \(\mathbf{c}\) are the pixel colors, \(w\left(p, q\right)\) are the bilateral weights between pixels \(p\) and \(q\), and \(\Omega\) denotes the filter footprint. The bilateral weight is split into three components as follows \[w\left(p, q\right) = \underbrace{ \vphantom{\max(0, \mathbf{n}\left(p\right) \cdot \mathbf{n}\left(q\right))^{\sigma_\mathbf{n}}} e^{-\frac{|p - q|^2}{2\sigma^2}} }_{w} \underbrace{ \vphantom{\max(0, \mathbf{n}\left(p\right) \cdot \mathbf{n}\left(q\right))^{\sigma_\mathbf{n}}} e^{-\frac{\left|z\left(p\right) - z\left(q\right)\right|}{\sigma_{z}\left|\nabla z\left(p\right) \cdot (p - q)\right|}} }_{w_{z}} \underbrace{ \max(0, \mathbf{n}\left(p\right) \cdot \mathbf{n}\left(q\right))^{\sigma_\mathbf{n}} }_{w_\mathbf{n}}.\] The spatial component, \(w\), is a Gaussian filter with its footprint controlled by \(\sigma\). We linearly increase \(\sigma\) from 1e-4 to 2.0 over the first 1750 iterations. The depth component, \(w_z\) implements an edge stopping filter based on image space depth. It is a soft test, comparing the depth differences between pixels \(z(p)\) and \(z(q)\) with a linear prediction based on the local depth gradient \(\nabla z(p)\). The final component, \(w_\mathbf{n}\), is computed as the scalar product over surface normals \(\mathbf{n}(p)\) and \(\mathbf{n}(q)\). Following SVGF, we use \(\sigma_z=1\) and \(\sigma_\mathbf{n} = 128\). We also omitted the Á-Trous wavelet filtering of SVGF, which is primarily a run-time performance optimization, and evaluate the filter densely in the local footprint. We propagate gradients back to the renderer, i.e., the noisy input colors, but no gradients to the filter weights.

9.0.0.2 Neural denoisers

For OIDN, we use the pre-trained weights from the official code [72], and wrap the model (a direct-prediction U-Net) in a PyTorch module to allow for gradient propagation to the input colors. We also propagate gradients to the network weights, in order to fine-tune the denoiser per-scene, but unfortunately, the direct-prediction architecture of OIDN made live-training highly unstable in our setting.

For the kernel-predicting neural denoiser, we leverage the architecture from Hasselgren et al. [22], but omit the adaptive sampling part and the recurrent feedback loop. The network weights were initialized to random values using Xavier initialization. The hierarchical kernel prediction in this architecture constrains the denoiser, and enables live-trained neural denoising in our setting. That said, in our evaluations, the live-trained variant produce worse reconstructions than the pre-trained denoisers.

We study the impact of different denoising algorithms on scene reconstruction, including jointly optimizing the denoising network, in Figure 13. Unfortunately, when fine-tuning the denoiser to the specific scene as part of optimization, we note a tendency for features to get baked into the denoiser’s network weights instead of the desired scene parameters (overfitting), negatively impacting the reconstruction quality. We showcase the isolated effect of denoising on light probe optimization in Figure 14. Figure 15 illustrates the effect of denoising when jointly optimizing geometry, material and lighting.

10 Visibility Gradients from Shadows↩︎

In Figure 16 we show the impact of the visibility gradients [14], [68] for the shadow test on a few targeted reconstruction examples. In this ablation, we disabled denoising, and applied a full shadow term directly (instead of linearly ramping up the shadow contribution, as discussed in Section 8). While clearly beneficial in the simpler examples, e.g., single-view reconstruction and targeted optimization (finding the light position), for multi-view reconstruction with joint shape, material, and environment light optimization, the benefits of the shadow visibility gradients is less clear. In our experiments, gradients from the shadow rays in the evaluation of direct lighting in the hemisphere, typically have negligible impact compared to gradients from primary visibility. Below, we report the view-interpolation scores over the validation sets for three scenes.

In contrast, we observed that the shadow ray gradients sometimes increase noise levels and degrade reconstruction quality. We show a visual example in Figure 17.

11 Sampling↩︎

In Figure 18, we examine the effect of importance sampling strategies on reconstruction quality. We compare cosine sampling, BSDF importance sampling [70], light probe importance sampling (using piecewise-constant 2D distribution sampling [69]), and multiple importance sampling [19] (MIS) using the balance heuristic. MIS consistently generates high quality reconstruction for a wide range of materials.

For unbiased results, the forward and backward passes of the renderer should use independent (decorrelated) random samples. However, we observe drastically reduced noise levels, hence improved convergence rates, when using correlated samples, i.e., re-using the same random seeds in the forward and backward pass, especially for low sample count optimization runs. This effect mainly stems from gradients flowing back to the exact same set of parameters which contributed to the forward rendering, instead of randomly selecting another set of parameters, which naturally increases variance. In Figure 19, we compare the two approaches over a range of sample counts.

12 Geometry↩︎

In Table 3 we relate our method to Table 8 (supplemental) of the nvdiffrec paper [7], and note similar geometric quality as nvdiffrec. The Lego scene is an outlier caused by our normal smoothness regularizer from Eq. 3 . The scene is particularly challenging for geometric smoothing since it contains very complex geometry full of holes and sharp edges. Please refer to Figure 20 for a visual comparison.

13 Relighting and View Interpolation↩︎

We present relighting results with per scene breakdown for all synthetic datasets in Table ¿tbl:tab:all95relighting?. Figures 21, 22, and 23 show visual examples of the NeRFactor synthetic dataset. This dataset is intentionally designed to simplify light and material separation (low frequency lighting, no prominent shadowing, direct lighting only). Figures 24, 25, and 26 show visual examples from the NeRF synthetic dataset. This dataset was designed view-interpolation, and contains particularly hard scenes with all-frequency lighting. Table ¿tbl:tab:all95view95interp? presents per-scene view interpolation scores for the synthetic datasets.

14 Scene Credits↩︎

Mori Knob from Yasotoshi Mori (CC BY-3.0). Bob model (CC0) by Keenan Crane. Rollercoaster and Porsche scenes from LDraw resources (CC BY-2.0) by Philippe Hurbain. Apollo and Damicornis models courtesy of the Smithsonian 3D repository [63] (CC0). The Family scene is part of the Tanks&Temples dataset [74] (CC BY-NC-SA 3.0), the Character scene is part of the BlendedMVS dataset [75] (CC BY-4.0) and the Gold Cape scene is part of the NeRD dataset [9] (CC BY-NC-SA 4.0). The NeRF [1] and NeRFactor [10] datasets (CC BY-3.0) contain renders from modified blender models located on blendswap.com: chair by 1DInc (CC0), drums by bryanajones (CC-BY), ficus by Herberhold (CC0), hotdog by erickfree (CC0), lego by Heinzelnisse (CC-BY-NC), materials by elbrujodelatribu (CC0), mic by up3d.de (CC0), ship by gregzaal (CC-BY-SA). Light probes from Poly Haven [78]: Aerodynamics workshop and Boiler room by Oliksiy Yakovlyev, Dreifaltkeitsberg by Andreas Mischok, and Music hall by Sergej Majboroda (all CC0). The probes provided in the NeRFactor dataset are modified from the probes (CC0) shipped with Blender.

References↩︎