1 Introduction↩︎

Authoring materials for 3D objects is a challenging artistic task, which involves creating spatially-varying details and assigning appropriate material parameters to its different parts. Current interactive applications, e.g., games, commonly apply a physically-based (PBR) material model, where a material is defined by diffuse base color, roughness, and metallicity stored in spatially varying textures. This allows a large variety of materials, including metals, woods, and plastics, but is still far from the appearance quality in offline rendering, e.g., film production, where artists design large material node graphs with multiple layers to capture the real-world appearance of effects like skin, eyes, and car paint.

To bridge the gap between offline and real-time material quality, neural material representations [1]–[4] have recently been proposed. The idea is to bake complex material graphs into a compact neural representation, consisting of a set of latent texture maps and a small neural network. This results in a unified format efficiently executable on modern graphics hardware at real-time rates, e.g., inside a path tracing real-time renderer [3]. Neural materials are typically baked from large material node graphs by densely sampling the 6D space of spatial variation (2D), and incoming and outgoing directions (4D). For each sample, the spatially varying bidirectional reflection/scattering function (SVBSDF) is evaluated and the neural material is optimized to match the reference material’s SVBSDF, as closely as possible.

In this work, we ask whether the inverse rendering machinery can be extended to extract neural material representations, which offer far greater expressiveness. To help asset creation, we introduce the first pipeline to extract neural materials from multi-view images. On a high level, this process can be described as photogrammetry for neural materials, however, the devil is in the details. Many recent works have successfully extracted PBR materials from multi-view images by leveraging inverse rendering and the highly constrained PBR material representation. Neural materials on the other hand are much more expressive, with additional degrees of freedom, and the latent space representing the manifold of valid materials can be fractured, which require careful initializations for inverse rendering optimization to converge. We address this problem by combining feed-forward reconstruction model priors with differentiable optimization to robustly extract neural materials from images.

Specifically, we have constructed a large feed-forward material reconstruction model (LMRM) which produces 1) a high quality initial guess of the neural material from a set of image observations, and 2) uncertainty estimates of the predictions. In a second step, we further refine the material prediction using Monte-Carlo based inverse rendering, guided by uncertainty. We leverage a recent neural material basis [4] which focuses on high quality specular appearance, including clear-coat, dust, fuzz, and scatter effects (see [fig:teaser]). Our extracted neural materials can be directly deployed inside a real-time path tracer. We summarize our main contributions as follows:

• We introduce the first neural material extraction pipeline combining a pre-trained prior with test-time optimization, going beyond standard PBR representations.

• We train an LMRM to estimate neural materials from multi-view images providing good initialization for our subsequent inverse path tracing.

• We use uncertainty-based regularization to better constrain inverse path tracing and avoid baked-in effects.

2 Related Work↩︎

Our work targets extracting neural material from images by combining inverse rendering with a large feed-forward reconstruction model. We briefly review the related fields.

****Material extraction from images.   A long-standing goal in inverse rendering is to extract spatially varying materials from images of 3D shapes. Several techniques estimate surface radiometric properties from images. Previous work on SVBRDF estimation rely on special viewing configurations, lighting patterns or complex capturing setups [5]–[12]. More recently, optimization-based methods jointly recover PBR texture maps, environment lighting and geometry from casually captured images [13]–[20] using inverse rendering. Radiance field-based methods factorize scene representations into intrinsic components without relying on explicit geometry [21]–[25].

Optimization-based methods are sensitive to ambiguous data and often require well-calibrated observations to reliably disentangle lighting, reflectance, and shape. To overcome this, a plethora of methods rely on neural networks to predict BRDFs from images [26]–[31]. Additionally, many recent methods use feed-forward or generative approaches to directly predict material intrinsics in a single forward pass [32]–[42].

While the field has advanced substantially, these methods universally target standard PBR material representations, i.e., a diffuse term paired with a single-lobe specular component, which limits the complexity of the extracted materials.

****Neural materials.   Neural material models represent SVBSDFs using latent textures for spatial variation and neural decoders for directional variation [1]–[3]. To improve rendering performance, the decoders are highly optimized MLPs designed around hardware constraints [3], and the complexity of the compressed latent space typically requires special consideration during optimization [43].

Recent work propose methods for generative neural materials, including models that synthesize neural latents directly or extract them from generative image and video priors [4], [44], [45]. However, the nonlinear neural latent space in these approaches poses a challenging problem [4], especially with small, highly optimized MLPs targeting real-time deployment. Combining inverse rendering with neural materials remains an under-explored research field. In this work, we note that the latent spaces of recent neural material representations further exaggerate the problem of optimization-based approaches getting stuck in local minima.

****Relightable neural representations.   Several works use neural representations for relighting surface and volumetric assets [46]–[50]. They typically treat assets as atomic units and precompute light transport, which makes animation, local lighting, and object interaction difficult. For instance, reconstructed objects generally do not support mutual global illumination effects. In addition, asset generation often requires controlled-lighting setups [46], [49], [50], typically with point light sources, complicating real-world data capture. In contrast, our method relies on inverse rendering [16]–[19] and explicitly reconstructs the local SVBSDF, making the resulting assets compatible with off-the-shelf animation and light-transport algorithms.

3 Overview of NeuMatEx↩︎

In this section, we briefly introduce the Neural Material representation that NeuMatEx builds upon, followed by an overview of the two-stage framework of NeuMatEx.

Preliminary: Neural Materials (NM).   Different from standard PBR materials [51], [52] parameterized by explicit maps, e.g., base color, roughness, metallicity, a Neural Material (NM) [1]–[3] can represent a richer SVBSDF expressed as a set of latent textures and a small neural network. In this work, we leverage a recent NM representation [4] which combines a Lambertian diffuse lobe with an expressive neural specular component: \[\label{eq:svbsdf} f({\mathbf{p}}, \omega_i, \omega_o) = T_\mathrm{neu}({\mathbf{p}}, \omega_i)\,\frac{\rho_d({\mathbf{p}})}{\pi} + f_\mathrm{neu}({\mathbf{p}}, \omega_i, \omega_o),\tag{1}\] where \(\rho_d\) is the diffuse base color, \(f_\mathrm{neu}\) the specular BSDF, and \(T_\mathrm{neu}\) the transmission albedo (to enforce energy conservation). These values are decoded by a pre-trained universal decoder MLP \(\mathcal{D}_\text{neu}\): \[\label{eq:nm95fwd} (f_\mathrm{neu}, T_\mathrm{neu}, R_\mathrm{neu}) = \mathcal{D}_\text{neu}(\ell({\mathbf{p}}),\, \omega_i, \omega_o),\tag{2}\] where \(\ell({\mathbf{p}})\in\mathbb{R}^{6}\) is a per-point specular latent code that compresses 22 analytical parameters [4] into a compact differentiable representation, and \(R_{\mathrm{neu}}(\omega_i)\) is a scalar measure of reflected energy in \(f_{\mathrm{neu}}\), used for importance sampling between the diffuse and neural specular lobe during rendering.

Trained on a large-scale procedurally enhanced NM dataset [4] spanning diverse multi-lobe specular effects, e.g., clearcoat, inner-/subcutaneous scattering, this NM latent space covers significantly richer appearance than PBR’s single GGX lobe. Rendering NM is detailed later in 5.

NeuMatEx Pipeline.   NeuMatEx consists of two stages (see 1). In the first stage (1), given multi-view images and the corresponding 3D mesh geometry, a Large Material Reconstruction Model (LMRM) predicts a feature triplane in a single forward pass. Two lightweight MLPs decode this triplane to jointly produce an initial neural material, i.e., per-point base color and neural specular latents, and a material uncertainty that flags ambiguous surface regions. In the second stage (1), we perform test-time optimization of the initialized triplane using differentiable neural material path tracing, regularized by the LMRM’s uncertainty prediction from the first stage: discouraging material drifts for confident regions while letting uncertain ones adapt, to avoid poor local minima. We detail the initialization stage in 4 and the test-time optimization in 5.

4 Large Material Reconstruction Model↩︎

We introduce a large feed-forward material reconstruction model (LMRM), that takes multi-view images and jointly predicts 1) a high quality initial guess of the neural material, and 2) uncertainty estimates of the material predictions.

Model architecture.   Following recent work [45], [53], [54], we leverage a pre-trained diffusion transformer (DiT) [55] for text-to-video, Wan2.1-1.3B [56], repurposed as a single-step model. We finetune the model to take a set of input views, \(\mathbf{I}\), and to output a neural material parameterized in a triplane representation. We note that Mullia et al [46] also parametrize a neural appearance representation using triplanes. We drop the text conditioning.

The model comprises a VAE encoder-decoder pair, \((\mathcal{E}_\text{VAE}, \mathcal{D}_\text{VAE})\), and a transformer-based denoising function, \(\mathcal{F}_{\theta}\). We use the encoder \(\mathcal{E}_\text{VAE}\) to encode the input image views, \(\mathbf{I}\), into a latent tensor, \(\boldsymbol{z}^{\mathbf{I}}\). Specifically, we encode \(F=17\) views, evenly spaced in a 360 degree orbit around the object, alongside the six canonical views. The 17 views are encoded using the video VAE, and the canonical views are each encoded with the image VAE for additional quality [57]. We then run a single step of the transformer-based denoising function, \(\mathcal{F}_{\theta}\) to produce an output latent, \[\boldsymbol{z}^{\mathbf{tri}}(\theta) = \mathcal{F}_{\theta}(\boldsymbol{z}^{\mathbf{I}}).\] This latent is decoded via the VAE decoder, \(\mathcal{D}_\text{VAE}\), into three triplane features: \[\mathbf{T}_\mathrm{XY}, \mathbf{T}_\mathrm{YZ}, \mathbf{T}_\mathrm{XZ}= \mathcal{D}_\text{VAE}(\hat{\boldsymbol{z}}^{\mathbf{tri}}(\theta)),\] representing the \(\mathrm{XY}\), \(\mathrm{YZ}\), and \(\mathrm{XZ}\) planes respectively.

We query the triplanes at each surface point, \({\mathbf{p}}\), for each covered pixel in each view. These three point feature tensors are then concatenated to form a point feature, which is decoded by two small multi-layered perceptrons (MLPs) sharing the same input: a material decoder, \(\mathcal{M}_{\phi}^{\text{mat}}\), and an uncertainty decoder, \(\mathcal{M}_{\psi}^{\text{unc}}\). By querying triplane features for all the surface points, the material decoder produces G-buffer images of the material parameters for each covered pixel, \[\mathbf{G}_{\text{mat}}({\mathbf{p}}) = \mathcal{M}_{\phi}^{\text{mat}}([\mathbf{T}_\mathrm{XY}(\mathbf{p}), \mathbf{T}_\mathrm{YZ}(\mathbf{p}), \mathbf{T}_\mathrm{XZ}(\mathbf{p})]),\] while the uncertainty decoder produces G-buffer images of the per-material uncertainty measure, i.e., log-variance, \[\mathbf{G}_{\text{unc}}({\mathbf{p}}) = \mathcal{M}_{\psi}^{\text{unc}}([\mathbf{T}_\mathrm{XY}(\mathbf{p}), \mathbf{T}_\mathrm{YZ}(\mathbf{p}), \mathbf{T}_\mathrm{XZ}(\mathbf{p})]), \label{eq:uncertainty}\tag{3}\] over the same material parameters. Note that we train over a large collection of 3D shapes, so both MLPs, \(\mathcal{M}_{\phi}^{\text{mat}}\) and \(\mathcal{M}_{\psi}^{\text{unc}}\), represent shared, universal decoders of triplane features, while the predicted triplanes are unique per 3D object.

Training LMRM.   We jointly train the parameters of the LMRM network, \(\theta\), and the decoder MLPs \(\phi\) and \(\psi\), by minimizing the training objective: \[\mathcal{L}(\theta, \phi, \psi) = \mathcal{L}_{\mathrm{mat}} + \lambda_{\mathrm{unc}}\mathcal{L}_{\mathrm{unc}}, \label{eq:objective}\tag{4}\] which combines a material regression loss and an uncertainty loss. The material loss supervises the mean prediction of \(\mathcal{M}_{\phi}^{\text{mat}}\) against the reference, i.e., ground-truth, G-buffers, \[\mathcal{L}_{\mathrm{mat}} = \mathbb{E}_{\mathbf{G}_{\text{mat}}^{\mathrm{ref}}\sim p_{\text{data}}} \lVert \mathbf{G}_{\text{mat}}(\theta, \phi) - \mathbf{G}_{\text{mat}}^{\mathrm{ref}} \rVert_{2}^{2} , \label{eq:loss95mat}\tag{5}\] where \(\mathbf{G}_{\text{mat}}^{\mathrm{ref}}\) are G-buffer views of the reference material parameters. The uncertainty loss supervises the per-pixel material uncertainties, i.e., log-variance (Eq. 3 ), following the \(\beta\)-NLL formulation of Seitzer et al. [58]: \[\label{eq:loss95unc} \mathcal{L}_{\mathrm{unc}}{=}\mathbb{E}\left[\lfloor \exp{(\beta\mathbf{G}_{\text{unc}})} \rfloor \left( \frac{\mathbf{G}_{\text{unc}}}{2} {+} \frac{\lVert \lfloor \mathbf{G}_{\text{mat}}\rfloor - \mathbf{G}_{\text{mat}}^{\mathrm{ref}}\rVert_{2}^2}{2\exp(\mathbf{G}_{\text{unc}})} \right) \right],\tag{6}\] where \(\beta=0.5\) and \(\lfloor\cdot\rfloor\) denotes the stop-gradient operation, applied per material channel. We apply the stop-gradient to the material mean \(\mathbf{G}_{\text{mat}}\) inside \(\mathcal{L}_{\mathrm{unc}}\) so that it trains only the uncertainty decoder \(\mathcal{M}_{\psi}^{\text{unc}}\), while \(\mathcal{L}_{\mathrm{mat}}\) trains the mean; we refer to Seitzer et al. [58] for details.

We adopt a two-stage training curriculum. We first pre-train LMRM on large-scale 3D shape datasets with annotated PBR materials [34], [59], [60], where \(\mathcal{M}_{\phi}^{\text{mat}}\) predicts the standard PBR G-buffers. This PBR pre-training stage lets the model learn a robust image-to-material mapping before facing the more expressive neural material target. We then fine-tune LMRM on 3D assets annotated with procedurally enhanced neural materials [4], where \(\mathcal{M}_{\phi}^{\text{mat}}\) predicts the neural material G-buffer, i.e., diffuse base color \(\rho_d\) and the specular latent code \(\ell\), exposing \(\mathcal{M}_{\psi}^{\text{unc}}\) to the increased ambiguity of recovering multi-lobe reflectance.

5 Test-Time Optimization of Neural Materials↩︎

Given initial neural material predictions from LMRM, our test-time optimization (TTO) performs differentiable Monte Carlo inverse rendering to further refine material details and improve decomposition (see 2). Our MC inverse renderer follows NVDiffRecMC [18] but with the PBR specular lobe replaced by a universal neural material basis [4].

Forward rendering of NM.   For our path tracer with neural materials, the outgoing radiance \(L(\omega_o, {\mathbf{p}})\) at a surface point \({\mathbf{p}}\) in direction \(\omega_o\) is defined by the rendering equation [61]: \[L(\omega_o, {\mathbf{p}}) = \int_\Omega L_i(\omega_i, {\mathbf{p}})f({\mathbf{p}}, \omega_i,\omega_o) (\omega_i \cdot \mathbf{n}) d\omega_i. \label{eq:ibl}\tag{7}\] This is an integral of the product of the incident radiance, \(L_i(\omega_i, {\mathbf{p}})\) from direction \(\omega_i\) and the neural material SVBSDF, \(f({\mathbf{p}}, \omega_i, \omega_o)\) (Eq. 1 ). The integration domain is the hemisphere \(\Omega\) around the surface normal, \(\mathbf{n}\). We evaluate the rendering equation using Monte Carlo integration: \[L(\omega_o,{\mathbf{p}}) \approx \frac{1}{N} \sum_{i=1}^{N} \frac{L_i(\omega_i,{\mathbf{p}})f({\mathbf{p}},\omega_i,\omega_o) (\omega_i \cdot \mathbf{n})}{p(\omega_i)},\] with samples drawn from some distribution \(p(\omega_i)\).

We leverage multiple importance sampling [62] (MIS), a framework for combining multiple sampling techniques to reduce variance in Monte Carlo integration. In our case, we apply MIS with two sampling techniques: light importance sampling, \(p_\mathrm{light}(\omega)\), using a piecewise-constant 2D distribution sampling technique [63], and BSDF sampling, \(p_\mathrm{bsdf}(\omega)\). The weighting functions \(w_i(x)\), for each technique are chosen using the balance heuristic [62], [63].

Following Zeltner et al. [3], our BSDF sampler uses a frozen sampling decoder to extract parameters for two independent GGX lobes for the neural part, then selects one component among the diffuse cosine lobe and the two GGX lobes according to the predicted mixture weights (Eq. 2 ) and samples it. Using the taxonomy of differentiable Monte Carlo estimators [64], our importance sampling is detached, i.e., gradients are not back-propagated to scene parameters in the sampling step, only in the material evaluation.

Inverse rendering of NM: Optimization task.   In our setup, the neural component representing non-diffuse contributions consists of a universal, frozen decoder MLP \(\mathcal{D}_\text{neu}\) operating on a \(6\)D latent code \(\ell({\mathbf{p}})\) (Eq. 2 ). Rather than optimizing the per-point material attributes directly, we expose them through our triplane representation \(\mathbf{T}\) (see 3). Together with the frozen material decoder \(\mathcal{M}_{\phi}^{\text{mat}}\), the triplane yields the per-point base color \(\rho_d({\mathbf{p}})\) of the traditional diffuse lobe [18] and the specular latent \(\ell({\mathbf{p}})\), \[\rho_d({\mathbf{p}}),\, \ell({\mathbf{p}}) = \mathcal{M}_{\phi}^{\text{mat}}(\mathbf{T}({\mathbf{p}})), \label{eq:tto95decode}\tag{8}\] so that the LMRM-initialized triplane \(\mathbf{T}\) ([sec:lmrm]) forms the only optimization parameter.

For a given camera pose \(c\), our differentiable path tracer produces an image \(\mathbf{I}(\mathbf{T}; c)\). We light the scene with a known high dynamic range (HDR) environment map, and the renderer produces an output image with linear HDR values. We also emphasize that our neural materials are operating in linear HDR, e.g., strong specular peaks can have significant energy. The reference image \(\mathbf{I}_\mathrm{ref}(c)\), a view from the same camera, is typically stored in LDR. We apply a tonemap operator \({\mathcal{T}}\) that maps the renderer output to LDR. The tonemapped rendering loss to optimize the triplane is: \[\mathcal{L}_\mathrm{photo} = \mathbb{E}_{c}\Big[\lVert{\mathcal{T}}\big(\mathbf{I}(\mathbf{T}; c)\big) - \mathbf{I}_\mathrm{ref}(c)\rVert_{2}^{2}\Big]. \label{eq:obj95photo}\tag{9}\] We minimize the rendering loss using Adam [65], with gradients \(\partial\mathcal{L}_\mathrm{photo}/\partial\mathbf{T}\) obtained through differentiable path tracing. Note that both the triplane material decoder \(\mathcal{M}_{\phi}^{\text{mat}}\) and the neural material decoder \(\mathcal{D}_\text{neu}\) remain frozen throughout; only the triplane \(\mathbf{T}\) is updated.

Uncertainty-guided material regularization.   The photometric objective in Eq. 9 is under-constrained at the neural material level: due to the rich multi-lobe specular effects, many combinations of base color \(\rho_d\) and specular latent \(\ell\) can explain the same image-space observations. Thus, the test-time optimization may drift toward an implausible material while still minimizing the photometric loss. To mitigate this amibiguity, we leverage the joint material-uncertainty prediction from LMRM (Eq. 3 ). In a single forward pass, LMRM predicts both the initial material mean \(\mathbf{G}_{\text{mat}}^\text{LMRM}({\mathbf{p}})\) and the log-variance \(\mathbf{G}_{\text{unc}}^\text{LMRM}({\mathbf{p}})\). We freeze both as constants during TTO and use them to anchor the current material \(\mathbf{G}_{\text{mat}}(\mathbf{T})\), decoded from the optimized triplane, through a uncertainty-weighted regularization term: \[\mathcal{L}_\mathrm{reg} = \mathbb{E}_{{\mathbf{p}}}\!\left[\,\frac{\lVert\mathbf{G}_{\text{mat}}(\mathbf{T}; {\mathbf{p}}) - \mathbf{G}_{\text{mat}}^\text{LMRM}({\mathbf{p}})\rVert_2^2}{\exp\!\big(\mathbf{G}_{\text{unc}}^\text{LMRM}({\mathbf{p}})\big)}\,\right], \label{eq:obj95reg}\tag{10}\] where the error is computed per material channel and reciprocally weighted by the LMRM’s predicted uncertainty \(\exp(\mathbf{G}_{\text{unc}}^\text{LMRM})\), mirroring the \(\beta\)-NLL term used at training time (Eq. 3 ). Intuitively, Eq. 10 acts as a Gaussian prior centered at the LMRM prediction: confident triplane regions (small \(\exp(\mathbf{G}_{\text{unc}}^\text{LMRM})\)) are strongly anchored, suppressing material drift, while uncertain regions (large \(\exp(\mathbf{G}_{\text{unc}}^\text{LMRM})\)) are loosely constrained and free to explore and refine the details missing from the feed-forward initialization (see 3).

Overall, the complete test-time optimization objective combines the photometric data term (Eq. 9 ) with the uncertainty-guided material regularizer (Eq. 10 ), \[\operatornamewithlimits{\arg \min}_{\mathbf{T}}\;\mathcal{L}_\mathrm{photo} + \lambda_\mathrm{reg}\,\mathcal{L}_\mathrm{reg}, \label{eq:tto95total}\tag{11}\] where \(\lambda_\mathrm{reg}\) weights uncertainty-guided material regularization for achieving robust material decomposition.

6 Experiments↩︎

By combining a novel feed-forward model (LMRM) which predicts neural materials from multi-view observations, with test-time optimization, we can robustly extract high quality neural materials (4). In experiments, we visualize renderings of neural materials extracted by NeuMatEx and compare with PBR-based material extraction methods on synthetic and real-world images. We also investigate the benefits of NeuMatEx’s core design choices by analyzing the impact of material initialization and regularization.

6.1 Neural Material Rendering Performance↩︎

We evaluated rendering performance of the neural materials at 1080p resolution on an RTX 5090 using the Japanese tabletop scene in 5, which consists of four different neural materials and runs at 3.88 ms per frame at 1 spp in our path tracer with 10 bounces. Note that this number reflects the combined cost of both the neural evaluation and the underlying path-tracing computation. The universal neural material basis consists of MLPs with 4 layers of width 64 (4×64) for BSDF evaluation, a 3×64 auxiliary network for estimating transmission albedo, and a 2×32 sampling network for importance sampling. This clearly indicates that the neural material representation we extract allows for real-time performance within a path-tracing context. Please refer to Zeltner et al. [3] for a thorough analysis of neural material runtime performance and implementation details.

6.2 Comparison with PBR-based Methods↩︎

As representative examples of recent feed-forward PBR material generation methods, we choose Hunyuan3D-Paint 2.1 [41] and TRELLIS.2 [42], using their PBR texture generation mode with known geometry. Note that both Hunyuan3D and TRELLIS.2 are conditioned on a single input view, while our method uses 17+6 views in the LMRM stage and 17 views for test-time optimization; therefore, this comparison is to highlight the limitations of PBR rather than a direct comparison. We also adapt NVDiffRecMC [18], initialized with a PBR prediction from our LMRM, with known geometry and lighting, optimizing only the materials. We denote this version NVDiffRecMC++, and argue that it is a strong method for optimization-based PBR extraction.

In 6, we show renderings of extracted materials alongside the diffuse base color for assets with complex specular shading effects, including haze, clearcoat, fuzz, and scatter. We visualize objects with ground-truth (reference) neural materials expressed in the basis from [4]; all methods are rendered from the input view of Hunyuan3D and TRELLIS.2 for the fairest possible visual comparison. Our method, NeuMatEx, consistently reconstructs materials that are close to the reference, successfully models the specular effects, and separates the diffuse and specular material terms, without baking specular shading into the diffuse base color. In 1, we report quantitative results, i.e., PSNR scores for pre-defined 17 orbital views, for both the path-traced renderings and base color. NeuMatEx consistently outperforms all the PBR-based methods, showing that the expressiveness of the neural material is essential for faithfully reproducing complex appearance effects that PBR cannot capture.

6.3 Ablations↩︎

Impact of design choices.   In 2, we ablate the design choices of NeuMatEx: the triplane material representation, the impact of the initial material prediction from the LMRM, and the uncertainty regularization. Neural materials are significantly harder to extract from multi-view images through inverse rendering, and we note that the strong prior from the LMRM initialization greatly improves both the render quality and the material decomposition compared to optimizing from randomly initialized materials ((a),(d) vs. (b),(e)). The impact of uncertainty guided optimization is more subtle, but increases the quality of the material decomposition ((b),(e) vs. (c),(f)). The triplane representation provides a significant quality improvement, since it enforces stricter spatial coherence than naïve UV texture mapping, which helps regularize test-time optimization ((a)-(c) vs. (d)-(f)).

Impact of uncertainty-guided regularization.   In 7, we qualitatively show a clear improvement in material decomposition from uncertainty-guided regularization. In 3, we ablate the benefit of uncertainty-guided regularization for test-time optimization. The unconstrained TTO (row (b)) already outperforms the LMRM prediction alone, and adding uncertainty-guided regularization further improves material decomposition while retaining competitive render quality (row (c)). Also, we evaluate two regularization strategies using: our predicted uncertainty \(\mathbf{G}_{\text{unc}}^\text{LMRM}\), and an oracle \(\mathbf{G}_{\text{unc}}^\text{oracle}\) that uses material G-buffer errors as uncertainty, i.e., ideal upper bound case up to scale (row (d)). Our TTO with predicted \(\mathbf{G}_{\text{unc}}^\text{LMRM}\) performs close to the ideal case, supporting that LMRM produces well-calibrated uncertainty estimates that effectively guide TTO.

6.4 Neural Materials from Real-world Captures↩︎

Real-world objects can exhibit complex reflection properties that go beyond the capabilities of the standard PBR model. Extracting expressive neural materials directly from photographs can pave a way for future data-driven SVBSDF model authoring. As a proof-of-concept, we apply NeuMatEx to extract neural materials from real-world objects. We use the Digital Twin Catalog (DTC) [66] consisting of high-quality multi-view captures (\(\sim120\) photographs per object) together with meshes, camera poses, and measured environment lighting. First, we use novel-view synthesis to obtain the preset views, i.e., \(17\) orbital views and \(6\) orthogonal views, needed for our LMRM. Specifically, we use 2D Gaussian Splatting [67], where we initialize the primitives at the mesh surface and the normals with the mesh normals. Then, we run our LMRM on the rendered views to estimate the initial material. During test-time optimization, we optimize against all real photographs to avoid baking any novel-view-synthesis artifacts.

We compare NeuMatEx against NVDiffRecMC++ in 8. While the PBR materials can reconstruct many simple appearances, they struggle with complex glossy effects and layered reflectance, e.g., the clearcoat lobe visible on the teapot. In contrast, our extracted neural materials are expressive enough to capture these effects, producing more faithful reflections. In 9, the relighting results show that these materials remain stable under illumination changes, yielding realistic reflections without baking in the original lighting.

7 Discussion and Limitations↩︎

We present NeuMatEx, demonstrating that inverse rendering can be extended to recover expressive neural materials from multi-view images and photographs. This is made possible by combining a strong feed-forward initialization from the Large Material Reconstruction Model with uncertainty-guided regularization during test-time optimization, stabilizing the otherwise ill-posed optimization in the neural material latent space. NeuMatEx outperforms PBR-based methods and recovers richer materials in both synthetic and real-world scenarios, cleanly separating lighting and material response. Although we build on Yu et al.’s [4] SVBSDF representation, our method is not inherently tied to this basis, and we believe it can be extended to other neural material models. In future work we aim to improve initial material prediction quality, and increase the spatial resolution of the recovered materials to capture finer-scale appearance details.

Limitations.   The complexity of the neural basis still poses a challenge even with our careful initialization. As shown in 10, LMRM predictions on out-of-domain inputs, such as real-world photographs, can exhibit specular artifacts, usually yielding a red tint, similar to the artifacts observed by Yu et al. [4]. Our test-time optimization often reduces this issue, but slight artifacts can still be visible. Learning a more robust neural material model, ideally from real-world observations, is therefore an important direction toward truly photorealistic reflectance recovery.

References↩︎