1 Introduction↩︎

Procedural materials power the visual surfaces of films, games, product design, and virtual worlds by encoding appearance as parameterized node graphs rather than fixed bitmaps. Their flexibility makes them essential for scalable, editable, and physically consistent asset creation. Yet crafting these materials remains specialist work: experienced artists rely on visual intuition, procedural reasoning, and domain-specific workflows. Both aspiring and established creators continually refine their craft by observing how experts layer textures, tune parameters, and reason about physical effects. Such demonstrations convey process knowledge that extends far beyond the final graph. Figure 1 illustrates our central premise: material generation should reflect the process expertise artists use when constructing procedural materials. Current methods, even when trained on large procedural material datasets, operate in isolation at inference time: they generate materials but do not incorporate such expertise into the generation.

If process knowledge is central to high-quality material design, how can it be reflected automatically during generation? We address this by framing procedural material generation as retrieval-time process reasoning. We introduce MaterialApprentice, a multimodal framework that integrates expert-derived process traces into the generative loop. Rather than operating solely in graph space, MaterialApprentice retrieves relevant traces, synthesizes a target trace aligned with the user’s intent, and grounds it into an executable material graph. As illustrated in Fig. 2, MaterialApprenticedistinguishes three representations: a process trace, which records construction steps, parameters, and design intent; a material graph, the executable shader-node layout compiled from the trace; and a procedural material, the resulting editable shader asset. Treating process traces as a first-class representation provides a foundation for higher-quality, more editable material generation.

Expert procedural knowledge is typically acquired by observing skilled artists as they construct materials. In practice, this expertise is often externalized through tutorial videos, which provide a temporally ordered and multimodal record of the creation process: the evolving node graph, parameter adjustments, synchronized narration explaining intent, and visual focus on individual operations. These signals indicate not only the final artifact but the reasoning trajectory behind it.

In our setting, we approximate this observational learning by analyzing Blender tutorial videos and extracting structured, agent-friendly process traces using automated video analysis tools. Rather than forming a training dataset, these traces serve as demonstrations retrieved at inference time. In practice, only a small number of relevant exemplars is sufficient to guide synthesis, forming a lightweight knowledge base for retrieval-time reasoning.

Technically, we instantiate retrieval-time process reasoning through a retrieval-augmented large language model (LLM) framework. Given a user query, MaterialApprenticeretrieves relevant process traces and conditions a pretrained LLM-based ProcessSynthesizerto synthesize a process trace aligned with the intended material. This trace is then grounded into an executable representation with a pretrained LLM-based Compiler(differing only in prompting), which translates it into a structured material graph implemented using Blender shader nodes. This design separates reasoning from execution while integrating process-level conditioning directly into the generative loop. Importantly, beyond one-shot synthesis, MaterialApprenticecan also demonstrate the process, producing step-by-step constructions that mirror expert workflows and support structured editing. In contrast, prior systems such as VLMaterial [2], BlenderAlchemy [3], and Blender-MCP [1] are limited to operation in graph space.

We evaluate MaterialApprenticeon a suite of Blender material creation tasks, including multimodal material generation and text-driven editing. Across quantitative metrics and perceptual studies, MaterialApprenticeconsistently outperforms VLMaterial [2], BlenderAlchemy [3], and Blender-MCP [1]. In addition, a study with experienced Blender artists shows that process traces generated by MaterialApprenticemore closely follow expert constructions and require fewer manual edits than alternatives.

In summary, we frame procedural material generation as retrieval-time process reasoning over expert demonstrations. Our contributions are:

• A formulation of procedural material generation as retrieval-time process reasoning, elevating process to a first-class representation beyond graphs.

• A retrieval-augmented architecture with an explicit process-trace representation and compiler that separate reasoning from execution and ground generated traces into executable Blender shader graphs.

• A curated corpus of Blender tutorial videos and video analysis tools for extracting agent-friendly process traces that enable inference-time conditioning.

• Comprehensive evaluation, including with Blender artists, showing process-based synthesis better aligns with expert workflows than graph-based systems.

2 Related Work↩︎

2.0.0.1 Generative Materials

Recent work directly synthesizes material appearances. MaterialGAN [4] extends StyleGAN2 [5] to generate SVBRDF textures, but operates in an opaque latent space. Diffusion-based methods improve visual quality: ControlMat [6] uses ControlNet-guided diffusion for SVBRDF generation with limited structural control, DreamMat [7] maps text prompts to point-wise BRDF parameters in a hash grid, and MaterialPicker [8] fine-tunes a video generator for text- and image-conditioned material textures. Huang et al. [9] generate textures directly in UV space, improving surface consistency. However, these methods produce non-procedural, sample-based or latent representations and do not expose editable procedural graphs.

2.0.0.2 Generative Procedural Materials

Closest to our work are methods that synthesize procedural material graphs. Hu et al. [10] generate node-based materials conditioned on text prompts, example images, or partial node graphs, using node graphs as defined in Adobe Substance 3D [11] and the Matformer representation [12]. Li et al. [13] build on these ideas, using synthetic data to train a transformer model for material graph generation and applying RL post-training on both real and synthetic data to improve realism and stability. While these methods produce procedural representations that are more editable and interpretable than purely texture-based outputs, they rely on large-scale training and custom models.

2.0.0.3 LLMs for Program Synthesis

Large language models (LLMs) have demonstrated strong capabilities in program synthesis [14], motivating their use as engines for 3D content creation. Early systems directly generate Python code for Blender, such as 3DGPT [15]. However, operating in a general-purpose, low-level language makes it difficult to maintain structure, guarantee validity, and scale to complex scenes. To address these limitations, several works introduce domain-specific languages (DSLs) tailored for 3D generation. Holodeck [16] and L3GO [17] use LLMs to author scenes in compact DSLs, simplifying high-level planning and constraint reasoning. While DSLs can make LLM outputs more structured, they typically lack large-scale training corpus, which can lead to unexpected behavior [18].

2.0.0.4 LLMs as Planners

Another line of work treats LLMs as planners that orchestrate external tools for 3D creation. LayoutGPT [19] uses LLMs to generate layouts, while SceneCraft [20] and LayoutVLM [21] adopt an analysis-by-synthesis loop in which a vision-language critic evaluates rendered scenes and guides refinement. When combined with tool-use frameworks like ReAct [22], LLM agents can scale to interactive, photorealistic environments [23].

2.0.0.5 VLMs for Materials

LLMs and vision-language models have also been applied directly to material synthesis. VLMaterial [2] fine-tunes a VLM to generate procedural materials represented as node graphs and BlenderAlchemy [3] iteratively samples and refines node graphs using VLM-based evaluation of rendered outputs, demonstrating that large models can reason about procedural material programs and visual feedback. In contrast, our method is training-free, leverages process-trace reasoning, and uses in-context learning from expert material generation workflows to achieve high-quality procedural material synthesis and editing.

3 Method↩︎

Procedural material creation is not merely graph assembly, but the execution of a structured reasoning trajectory. Skilled artists iteratively refine structure, adjust parameters, and articulate intent; the final node graph is only the endpoint of this process. We therefore distinguish the three representations used by MaterialApprentice: a process trace, which records construction steps, parameters, and design intent; a material graph, the executable shader-node layout compiled from the trace; and a procedural material, the resulting editable shader asset. While prior systems operate directly in graph space, we perform generation in process space.

To operationalize this formulation, MaterialApprenticeinstantiates retrieval-time process reasoning. As summarized in Figure 3, given a user query, it retrieves relevant process traces and synthesizes a process trace via ProcessSynthesizeraligned with the intended material. This trace is then grounded into an executable Blender shader graph through the Compiler, explicitly separating reasoning from execution. Process traces are derived from expert video demonstrations using video analysis tools which convert raw tutorials into structured, agent-friendly textual representations. These traces form the knowledge base for retrieval-time reasoning. Figure 4 details the ProcessSynthesizer, DeCompiler, and video analysis tools, which are described in the following subsections.

3.1 Process Synthesizer↩︎

The ProcessSynthesizeris the core reasoning module of MaterialApprentice, synthesizing process traces aligned with user intent. Given a query \(q\), a Retrieverselects relevant reference traces \(P\) from the demonstration corpus \(C_P\). Conditioned on \(q\) and \(P\), the Editorsynthesizes a target trace \(p\) describing how to construct the desired material.

For editing, ProcessSynthesizeradditionally accepts an input graph \(g_i\). In this setting, the Retrieverselects process traces relevant to \(g_i\), and a DeCompilermodule reconstructs a corresponding process trace \(p_i\). This inferred process trace, together with the user’s edit instruction, conditions the Editorto synthesize an updated process trace consistent with the requested modification.

3.1.0.1 Retriever.

The Retrieverselects relevant demonstrations for both generation and editing. Rather than ranking examples by surface appearance alone, it uses the pretrained LLM to compare how materials are constructed. Given material names and descriptions in \(C_P\), it predicts which expert demonstrations best match the user’s intent, desired material properties, and construction strategy. The top-\(K\) corresponding process traces are selected to condition synthesis.

3.1.0.2 Editor.

The Editoris a pretrained LLM prompted to perform in-context synthesis in the process domain. Conditioned on \(P\) and \(q\), it generates a target process trace \(p\) specifying the construction sequence, parameter choices, and compositional structure required to realize the requested material. As illustrated in Fig. 5, the Editorcombines construction patterns from retrieved traces into a target trace before compilation. Here, lowercase \(p\) denotes a single trace, while \(P\) refers to sets of retrieved process traces from \(C_P\). To improve reliability, the Editoremploys a lightweight self-refinement step, critiquing and revising its draft before finalizing \(p\), enhancing consistency and adherence to the query. Because the Editoroperates purely in process space, editing requires representing existing graphs in the same modality, motivating the DeCompilermodule.

3.1.0.3 DeCompiler.

The DeCompilerconverts an input material graph \(g_i\) into a process trace, enabling editing within a unified representation. Given \(g_i\), it reconstructs a plausible trace \(p_i\) using exemplar graph–trace pairs from the reference corpus. Each reference trace is first compiled into graph form via the Compiler, yielding exemplar pairs \(E = \{(g_j, p_j)\}_{j=1}^k\). The exemplars are selected to match the input graph in construction strategy rather than surface appearance. A specialized LLM module, Narrator, then reconstructs \(p_i = \mathrm{\small Narrator}(g_i; E)\) by matching substructures of \(g_i\) to similar graph patterns in the exemplars and transferring their associated construction steps. The resulting \(p_i\) captures the operations, parameters, and intent implied by the input graph.

3.2 Grounding Process Traces in Material Graphs↩︎

The Compilertranslates textual process traces into executable Blender shader graphs. It is implemented as a pretrained LLM prompted to generate Blender Python graph-construction code conditioned on a trace \(p\). A lightweight self-refinement stage enforces consistency between node topology, parameter definitions, and layer ordering. Given \(p\), the compiler produces an executable graph \(g\). The Compilergenerates multiple candidate programs \(N\) per synthesized process trace and selects the first syntactically valid and executable instance. The compiler targets Blender versions 3.0–4.5, with custom node groups for compatibility across API changes.

3.3 Video Tutorials for Extracting Process Traces↩︎

To support retrieval-time process reasoning, we construct a corpus of expert Blender material tutorials. The corpus consists of 158 publicly available YouTube tutorials from established creators, selected for clear stepwise instruction, synchronized narration, and focused visual presentation (e.g., zoomed-in node editing, explicit parameter explanation, and structured workflow progression). It spans 29 material categories (e.g., metals, fabrics, organic surfaces), with an average of 5.5 tutorials for each. Tutorials average 12.6 minutes in duration, with materials containing on average 14.3 shader nodes, providing rich signals about compositional structure, parameter tuning, and procedural reasoning.

In addition, we create 10 original tutorials for which we hold full rights and will release publicly to ensure long-term reproducibility. Publicly sourced tutorials are used exclusively as a retrieval-time knowledge source; no model training is performed on raw video content. We release YouTube IDs and category annotations to enable reconstruction of the corpus while preserving attribution to original creators. Detailed statistics are provided in the supplementary.

3.4 Process Extraction via Video Analysis Tools↩︎

To derive process traces from expert demonstrations, MaterialApprenticeuses automatic video analysis tools to convert tutorials into structured text that LLM agents can directly retrieve and reason over. This component is primarily an engineering step rather than a core research contribution: the goal is simply to automatically extract procedural signals from demonstrations so that they can be used for retrieval-time reasoning. In practice, Blender tutorials exhibit a simple box-and-wire interface, allowing standard detection and segmentation models to reliably recover graph structure. Given a tutorial video \(v\), the pipeline extracts keyframes, reconstructs intermediate graph states, and aligns visual edits with narrated explanations.

3.4.0.1 Video Decomposition.

Videos are first decomposed into frames and audio narration. Informative keyframes are selected to capture meaningful transitions in the material construction process while removing redundant views. The audio stream is transcribed to preserve the artist’s verbal explanations and design rationale.

3.4.0.2 Graph Reconstruction.

The visual stream is analyzed to recover the evolving node graph. From selected keyframes, the system identifies node layouts, connectivity patterns, and parameter states using standard detection and segmentation models fine-tuned for the Blender shader editor. This produces a sequence of intermediate graph representations reflecting the progressive assembly of the material.

3.4.0.3 Process Summarization.

A Summarizermodule integrates the reconstructed graph states with the transcribed narration to produce a compact textual process trace \(p_v\), capturing both the sequence of construction steps and the accompanying procedural reasoning expressed by the artist. These traces populate the retrieval corpus used by the ProcessSynthesizerat inference time.

3.4.0.4 Robustness of Process Extraction.

Three design choices make extraction reliable. (1) Grounding: frame understanding relies on explicit vision tasks – node, socket, and wire detection – rather than direct LLM inference. Given the structured box-and-wire layout of the Blender shader editor, these detectors achieve mAP50 = 0.98 and mIoU = 0.87. (2) Selection: a fine-tuned frame-selection model retains only visually parseable frames robust to zoom, pan, and occlusion. (3) Redundancy: frames are parsed at 0.2 Hz and combined with narration through an LLM summarizer, mitigating localized perception errors and producing stable process traces. Further, while the pipeline contains multiple stages, errors in early stages are mitigated by the process abstraction. The ProcessSynthesizerreasons over construction patterns rather than exact node configurations, allowing it to tolerate minor inaccuracies in reconstructed graph states or narration alignment. In practice, we observe that small parsing errors rarely propagate to catastrophic failures in the final material graph. Further details are provided in the supplementary.

3.5 Implementation Details↩︎

All LLM-based components—Retriever, Editor, DeCompiler, Compiler, and Summarizer—use the same pretrained GPT-5 model, differing only in prompt structure and contextual conditioning. The Retrieverselects the top \(K=3\) process traces per query, balancing contextual coverage and computational efficiency. The compiler generates \(N=10\) programs. While processes synthesized by the ProcessSynthesizerare typically coherent and faithful to the query, final material quality may be further refined through established procedural optimization techniques [3]. A single process trace is \(\approx 8k\) tokens, a single run costs \(\approx 2\)USD and runs for \(30\) mins.

4 Experiments↩︎

Our experiments probe the implications of viewing procedural material generation as retrieval-time process reasoning. Section 4.1 evaluates the key design choices implied by this formulation—reasoning in process space rather than graph space and grounding execution through the Compiler. Section [sec:baseline95comp] then situates MaterialApprenticewithin existing procedural material systems through comparisons on established generation and editing benchmarks. Finally, Section 4.3 presents ablations analyzing generalization, retrieval behavior, the impact of video analysis tools, and the reproducibility of our framework through experiments on in-house tutorial videos.

4.1 Reasoning in the Process Domain↩︎

3.5pt

To evaluate process-space reasoning, we isolate the representation used for retrieval while keeping the output modality fixed. Our method retrieves process traces, while the baseline retrieves material graphs; both variants synthesize process traces that artists follow to construct and refine material graphs. This isolates the effect of structured procedural knowledge relative to raw graph structure.

4.1.0.1 Evaluation setup.

We recruit five Blender artists (avg. 7.5 years of procedural-material experience) and evaluate 60 synthesized process traces from 30 prompts under two retrieval conditions. For each prompt, artists see anonymized, randomly ordered process traces from the two variants. The study has two stages: artists first construct a material graph by following each trace, then refine the graph for up to ten minutes to better match the input prompt. We record atomic graph edits—node edits, connection edits, and parameter refinements—as a direct measure of corrective editing effort.

Artists also follow a think-aloud protocol[24] during construction and refinement, verbalizing their understanding of each trace, the rationale behind their edits, and their assessment of the resulting graph. We ask them to comment on implementation clarity, procedural strategy, expected outcome quality, parameter control, and pedagogical value. An LLM then analyzes the recorded narrations with a fixed rubric to assign structured pairwise preferences and editing-friction labels. Final render preference is collected separately after refinement. Table 1 summarizes the study along three axes: edit effort, expert preference, and think-aloud sentiment. We provide the GPT-5 analysis prompts in the supplement.

4.1.0.2 Editing effort.

The left block of Table 1 reports edits required to refine the generated graphs. Node and connection edits modify graph structure and therefore reflect changes to the underlying procedural strategy, while parameter refinements tune existing controls such as sliders and color ramps. Process-conditioned traces require fewer structural edits than graph-conditioned traces (7.84 vs. 10.84) and fewer total edits overall (13.58 vs. 15.67), indicating that they produce graphs closer to the intended procedural structure. They require slightly more parameter tweaks (5.75 vs. 4.83), which we interpret as a favorable shift: artists spend less effort repairing structure and more effort tuning exposed controls.

4.1.0.3 Expert preference.

The middle block reports LLM-assigned preferences from artists’ think-aloud commentary; final render preference is provided directly by artists after refinement. Process-conditioned workflows are favored across all criteria, especially implementation clarity (100%) and outcome quality (92%), with strong preferences for procedural strategy (83%), parameter control (83%), pedagogical value (84%), and final render quality (83%). This indicates that process traces provide clearer construction guidance beyond final appearance.

4.1.0.4 Editing sentiment.

The right block analyzes editing friction from think-aloud narrations. An LLM assigns low/medium/high labels to four signals: confusion and hesitation, which indicate difficulty understanding the graph or choosing the next edit, and satisfaction and completion, which indicate successful refinement and confidence in the result. After mapping labels to 1/2/3, process-conditioned workflows reduce confusion (\(-0.15\)) and hesitation (\(-0.10\)), while increasing satisfaction (\(+0.40\)) and completion (\(+0.25\)). This suggests that process-derived graphs are easier for artists to understand, modify, and finalize.

4.1.0.5 Fully automatic generation.

Figure 6 evaluates the complete pipeline without manual editing. For 50 diverse material prompts, we compare paired outputs from process-conditioned and graph-conditioned retrieval. Three experts judge each pair against the input prompt, with majority vote determining preference. Process-conditioned outputs are preferred in 72% of comparisons, showing that process retrieval improves not only artist-facing editability but also automatic prompt alignment. Qualitatively, process-conditioned retrieval better captures hierarchical procedural structure, including directional fibers, layered corrosion, and structured surface noise, whereas graph-conditioned retrieval often yields simpler or structurally incorrect patterns.

4.1.0.6 Need for Compiler.

We evaluate a variant where the Editordirectly generates material graphs instead of processes. We measure reliability by execution rate: the fraction of prompts for which at least one candidate program runs successfully in Blender. Across 100 prompts, direct graph generation fails in 30% of cases, whereas our full process-to-compiler pipeline achieves a 100% execution rate.

4.2 Comparison with Prior Procedural Material Systems↩︎

3pt

We compare MaterialApprenticeagainst three state-of-the-art procedural material systems: BlenderAlchemy [3], BlenderMCP [1], and VLMaterial [2]. We also report a Nearest-Neighbor baseline (the closest retrieved process) to verify that MaterialApprenticeisn’t simply copying demonstrations. All baselines and MaterialApprenticemodules use a GPT-5 backbone for fairness.

For generation, perceptual similarity is measured following [2] using CLIP cosine similarity, style loss, and sliced Wasserstein distance (SWD). For editing, where no reference render exists, we follow [3] and compute CLIP similarity between the edit instruction and the generated output. We additionally conduct GPT-5 and user forced-choice studies: for generation, evaluators receive the target and two outputs; for editing, they receive the initial material, the edit instruction, and two candidate results. We recruit 150 users on Amazon MT, with comparisons rated by three annotators and aggregated by majority vote.

4.2.0.1 Procedural Material Generation.

We evaluate all methods, excluding BlenderAlchemy (which does not support prompt-based generation), on two benchmarks: (i) 110 rendered materials from BlenderKit and (ii) 35 in-the-wild close-up photographs. Each prompt includes a short text description (e.g., “banana skin’’) and a reference image. Figure 7 shows text-to-material generation results. MaterialApprenticeproduces materials more closely aligned with textual prompts, achieving higher CLIP similarity (0.806 vs. for nearest neighbor and 0.781 for BlenderMCP). Users prefer our results in 86% and 92% of comparisons against nearest neighbor and BlenderMCP, respectively. Table ¿tbl:tab:comp95table? reports quantitative results. MaterialApprenticeachieves the strongest performance on the challenging in-the-wild benchmark and is competitive with VLMaterial on BlenderKit, where VLMaterial is fine-tuned. CLIP, Style Loss, and SWD capture image-level similarity and are sensitive to scale, alignment, and rendering, while GPT-5 and user preferences reflect prompt faithfulness and procedural plausibility; these metrics therefore provide complementary views of quality. Across both preference studies, MaterialApprenticeis consistently favored over competing methods.

Figure 8 (left) shows qualitative comparisons. MaterialApprenticereproduces both macro- and micro-scale structures—from nacre patterns to dalmatian textures—while VLMaterial often produces coarse approximations and BlenderMCP drifts from the target appearance. Our outputs remain clearly distinct from nearest-neighbor examples, indicating genuine synthesis rather than retrieval.

4.2.0.2 Procedural Material Editing.

We evaluate BlenderAlchemy [3], BlenderMCP [1], and MaterialApprenticeon two editing tasks: 25 coarse edits (e.g., wood \(\rightarrow\) marble) and 25 fine-grained edits (e.g., adding moss to brick), using base materials from BlenderKit. Coarse edits require substantial restructuring of the shader graph, while fine edits involve localized modifications.

As shown in Table ¿tbl:tab:comp95table?, GPT-5 consistently prefers edits produced by MaterialApprenticeover BlenderMCP and nearest-neighbor baselines. Qualitative examples (Fig. 8, right) show that baselines often fail to incorporate key prompt attributes—for example, missing structural features—while MaterialApprenticeintroduces both large-scale structural changes and subtle surface details.

4.3 Ablations↩︎

We now analyze several aspects of MaterialApprentice  including its ability to generalize beyond the materials present in the process corpus and the effect of the number of retrieved demonstrations used during synthesis.

4.3.0.1 Generalization beyond the process corpus.

To test whether MaterialApprenticerelies on category-level cues from the corpus, we remove entire material categories corresponding to the evaluation prompts—Organic (blue jay egg), Concrete (cracked concrete), and Marble (midnight gold marble). Figure 9 shows that even under this setting, MaterialApprenticecontinues to synthesize plausible procedural structures that capture the essential characteristics of the input images, indicating that the model generalizes procedural strategies rather than memorizing category-specific examples.

4.3.0.2 Effect of the number of retrieved demonstrations.

We next vary the number of demonstrations provided as context during synthesis (Fig. 9). Even a single retrieved process trace substantially improves generation quality compared to using no demonstrations. However, the benefit saturates quickly, with limited improvement beyond three retrieved examples, suggesting that a small number of relevant demonstrations is sufficient to guide effective process synthesis.

4.3.0.3 Reproducibility with a curated video set.

To further demonstrate reproducibility, we evaluate MaterialApprenticeusing a curated set of tutorial videos that we created ourselves and release publicly with full rights. Figure 10 shows text-conditioned generation results when the reference pool is restricted to this small set. Despite the limited corpus, MaterialApprenticecontinues to synthesize diverse materials that differ clearly from both nearest-neighbor retrieval and generation without demonstrations, indicating that even a handful of shareable tutorials provides sufficient process-level cues for effective material synthesis.

4.3.0.4 Impact of Video Analysis Tools.

We analyze the contribution of our video analysis pipeline by constructing three variants: (1) Audio-only, which ignores the visual stream and relies solely on narration; (2) GPT-5 Frame Parsing, which replaces our specialized detectors with GPT-5-based frame interpretation; and (3) Random Keyframe Selection, which removes our informativeness-based keyframe filtering. We evaluate all variants on 100 tutorial videos by reconstructing material graphs from extracted process traces and comparing them with ground-truth graphs. Node detection and topology reconstruction are evaluated using precision, recall, and F1 scores. Parameter fidelity is measured as the normalized absolute difference between predicted and ground-truth parameters, while perceptual similarity is computed using sliced Wasserstein distance (SWD). We additionally report graph edit distance, defined as the number of atomic operations required to transform the reconstructed graph into the ground-truth graph (node additions/removals, connection changes, and parameter adjustments), estimated by prompting an LLM to produce the minimal edit sequence. As shown in Fig. 11, our full pipeline consistently achieves the best performance across all metrics and produces graphs that most closely match the original materials. These results highlight the importance of jointly modeling visual structure, narration, and keyframe selection for reliable process extraction from tutorials.

5 Conclusion↩︎

Procedural material design is inherently process-driven: artists construct materials through layered operations, parameter tuning, and physical reasoning. We frame procedural material generation as retrieval-time process reasoning over expert demonstrations, elevating process to a first-class representation beyond graph-only synthesis. By retrieving process traces and grounding them into executable shader graphs, MaterialApprenticeproduces materials that better reflect expert workflows while remaining interpretable and editable. More broadly, this perspective suggests generative systems that learn directly from creative practice rather than static artifacts alone. Extending process reasoning to multi-step design workflows, additional authoring tools beyond Blender, and larger corpora of demonstrations are promising directions for future work. Failure cases are discussed in the supplementary. Code, models, and data will be publicly released.

6 Acknowledgments↩︎

This work was supported in part by NSF grant 2402583 and gifts from Qualcomm, Google, and Adobe. We are grateful to the Blender experts who participated in our expert study: Barnabas Thomas, John Arquero, Benjamin Olawuni, Ricardo Raimundo, and Tanushree Roychoudhury. We also thank Krishna Chaitanya, Harsh Sinha, Kanav, and Diptimayee Gupta for their inspiration, support, and helpful discussions throughout this work.

7 Supplementary Video↩︎

Please watch the supplementary video for an overview of the method and highlights about key experiments.

8 Expert Study↩︎

8.1 Matched Graph-Space Baseline↩︎

To further isolate the role of the intermediate representation, we evaluate a matched Graph\(\rightarrow\)Graph baseline in which both retrieval and generation operate directly in graph space. This baseline retrieves material graphs and directly generates a material graph, removing the intermediate process representation. We compare it with Graph\(\rightarrow\)Process, which retrieves graphs but synthesizes process traces, and our full Process\(\rightarrow\)Process method, which retrieves and synthesizes in process space.

8.1.0.1 Evaluation protocol.

We evaluate 17 materials across the three methods. For each material, artists are given the final generated graph from each method and perform two independent editing tasks. In Target Refinement, artists edit the graph to better match the reference material. In Instructional Edit, artists apply a semantic edit using only parameter changes. Both tasks start from the same initial output graph and are limited to ten minutes. Separately, two artists choose the best result within each method triplet at three stages: the initial output, after target refinement, and after instructional edit.

8.1.0.2 Results.

6pt

Table 2 shows that Process\(\rightarrow\)Process is preferred at all stages: 76% for the initial output, 65% after target refinement, and 82% after instructional edit. It also requires the fewest node edits, connection edits, structural edits, and total edits during target refinement, indicating that process-space reasoning produces graphs that are closer to the intended procedural structure. Graph\(\rightarrow\)Graph requires fewer parameter-only edits during instructional editing, but this reflects the fact that its simpler graphs expose fewer tunable controls; despite requiring fewer parameter tweaks, its final edited outputs are rarely preferred. These results show that the advantage of process-space reasoning persists under a matched graph-space baseline and improves both initial graph quality and artist-facing editability.

8.2 Parameter-only Semantic Editing↩︎

To further evaluate whether generated materials expose useful controls, we conduct a parameter-only semantic editing study. Unlike the main expert study, where artists may modify graph structure, this study fixes the generated material graph and permits only parameter changes. This isolates whether the graph produced by MaterialApprenticecontains editable controls that support meaningful downstream refinement.

We evaluate 6 generated materials with 4 semantic edits per material, for a total of 24 editing tasks. Two Blender artists perform each edit using only exposed parameters, with at most 3 refinement attempts and a 5-minute time limit per edit. After each task, artists rate satisfaction with the final edit and ease of use on a 5-point Likert scale.

Artists report an average satisfaction score of 4.10/5 and an ease-of-use score of 3.75/5, with an average editing time of 1:48 minutes per edit. These results suggest that MaterialApprenticedoes not merely produce executable graphs, but also exposes parameters that support efficient semantic refinement without requiring structural graph edits.

8.3 Qualitative Examples↩︎

Figure 12 provides representative examples from the expert study comparing graph-conditioned and process-conditioned retrieval. For each prompt, artists first constructed a material graph by following the generated process trace and then refined the graph to better match the target material. The figure shows the input prompt, the initial and edited materials, the number of edits and editing time, and summarized think-aloud feedback from the artists.

Across these examples, process-conditioned retrieval typically produces graphs that start closer to the target appearance and require fewer corrective edits. For instance, in the Alien Rock and Rough Leather examples, graph-conditioned retrieval leads to substantial restructuring, including replacing procedural textures, reconnecting nodes, and debugging unexpected node behavior. In contrast, process-conditioned retrieval more often yields a graph with the right high-level structure, allowing artists to focus on parameter tuning such as scale, color, roughness, and bump strength.

The artist comments further illustrate this difference. Graph-conditioned outputs often cause confusion or trial-and-error editing when the node organization is difficult to interpret or when parameters have unclear effects. Process-conditioned outputs are described as easier to follow and quicker to refine, suggesting that process traces transfer not only node-level operations but also higher-level construction logic. These qualitative examples complement the quantitative results in Table 1, showing how process-conditioned retrieval reduces structural repair and improves artist-facing editability.

9 Video Tutorial Corpus↩︎

To enable MaterialApprenticeto reflect on expert workflows, we curate a tutorial video corpus containing 158 publicly available Blender material–creation tutorials from established YouTube creators such as Ryan King Art, Ducky 3D, Sam Bowman, PIXXO 3D, BlenderBiteSize, and others. These videos provide diverse demonstrations of real expert practice that MaterialApprenticeretrieves at inference time.

The corpus spans 29 material categories, with each category averaging 5.5 tutorials. Tutorials have an average duration of 12.57 minutes, and the demonstrated materials contain on average 14.32 nodes, offering rich signals about procedural structure, parameter tuning, and tacit artistic reasoning (see Table [tbl:tab:yt-dataset]). We will release the YouTube IDs and category labels for all videos, allowing the corpus to be reconstructed while preserving attribution to the original creators.

This corpus is strictly used for retrieving process traces at inference time. Experiments in section 4.3/ Figure 10 (main paper) show that MaterialApprenticeis able to generate novel materials with even a small fraction of the corpus which is further demonstrated by running MaterialApprenticeon the corpus of videos made by us (Figure 9).

10 Video Analysis Tools for Process Extraction↩︎

This section provides implementation details for the video analysis pipeline used to extract process traces from tutorial videos. As discussed in Sec. 3.4 of the main paper, this component is primarily an engineering step that converts demonstrations into structured signals usable for retrieval-time process reasoning.

Blender tutorial videos exhibit a consistent box-and-wire interface, allowing procedural structure to be reliably recovered using standard computer vision tools. Our pipeline automatically decomposes each tutorial video \(v\) into frames and narration, reconstructs intermediate node graphs, and summarizes these multimodal signals into a textual process trace \(p_v\) describing the material construction workflow.

The pipeline relies on lightweight pretrained detectors and segmentation models fine-tuned on Blender shader editor screenshots. Because the interface exhibits highly consistent layout, color schemes, and typography, standard architectures achieve high accuracy without extensive tuning. All stages operate automatically and require no manual annotation. We note that minor perception errors rarely propagate to the final generation stage, as the ProcessSynthesizerreasons over construction patterns rather than exact node configurations.

10.1 Video Decomposition↩︎

Each tutorial video \(v\) is decomposed into visual frames and audio narration. Let \(X_v = \{(t_k, I_k)\}_{k=1}^{T_v}\) denote the frame sequence and \(a_v\) the audio stream.

10.1.0.1 Keyframe Selection.

Tutorial videos often contain frames that are difficult to parse due to occlusions, extreme zoom, or limited visibility of the node graph. We therefore first identify frames that are suitable for reliable graph reconstruction. Frames are categorized according to their suitability for parsing, ranging from useless to excellent, as illustrated in Fig. 13.

To automatically estimate this quality, each frame \(I_k\) is assigned an informativeness score \[s_k = \Phi_{\text{score}}(I_k),\] which evaluates node visibility, text readability, graph coverage, and overall visual clarity. Technically, this is achieved via the scoring network \(\Phi_{\text{score}}\) uses a YOLOv8n [25] backbone fine-tuned as a five-class classifier to predict frame informativeness (excellent, good, fair, poor, useless). The scoring process is illustrated in Fig. 14. We train the score model on our internally created and annotated dataset consisting of \(1000\) synthesized Blender screenshots labeled across the five categories for 50 epochs at \(1080\)p resolution with a batch size of 128, achieving \(70\%\) top-1 accuracy.

Frames with scores above a threshold \(\tau_s\) form a candidate set \[\mathcal{C}_v = \{(t_k,I_k)\mid s_k \ge \tau_s\}.\] Even after filtering, tutorials may still contain many visually similar frames where the node graph remains unchanged while the instructor explains a concept. To reduce redundancy and improve efficiency, candidate frames are embedded using \(\Phi_{\text{emb}}\) (CLIP) and clustered by cosine similarity. The final frame from each cluster is retained, capturing the most evolved graph state at that stage of the tutorial.

10.1.0.2 Narration Transcription.

The audio stream \(a_v\) is transcribed using the Whisper-Large automatic speech recognition model, producing a time-aligned transcript \[\tau_v = \Phi_{\text{asr}}(a_v).\] The outputs of this stage are the keyframe set \(X_v^{\mathcal{K}}\) and the aligned narration transcript \(\tau_v\).

10.2 Graph Reconstruction↩︎

For each selected keyframe \(I_k\), the pipeline reconstructs the corresponding node-graph state by detecting nodes, recovering UI parameters, and inferring graph connectivity.

10.2.0.1 Image-Level Parsing.

We first identify the structural components of the shader graph. An object detector localizes nodes together with their input and output sockets, while a segmentation model extracts connection wires (see figure on the right):

\[\mathcal{N}_k,\mathcal{S}_k^{in},\mathcal{S}_k^{out} = \Phi_{\text{det}}^{img}(I_k), \quad \mathcal{W}_k = \Phi_{\text{seg}}^{wire}(I_k).\]

OCR is then applied within detected node regions to recover node names and socket labels.

10.2.0.2 Node-Level Parameter Extraction.

Each detected node region \(I_n\) is further analyzed to recover UI elements encoding parameter values. A lightweight node-level detector identifies interface components such as sliders, color ramps, checkboxes, and color tabs (see figure on the left):

\[\mathcal{U}_n = \Phi_{\text{det}}^{node}(I_n).\]

Slider positions are normalized within their UI bounds, while color tabs are converted to RGB values, producing node parameter states \(\Theta_n\).

Both the image-level detector \(\Phi_{\text{det}}^{img}\) and the node-level detector \(\Phi_{\text{det}}^{node}\) use YOLOv8n trained at \(1080\)p resolution for 50 epochs. Both achieve mAP\(_{50}\) scores near \(0.98\) with precision and recall in the \(0.95\)–\(0.98\) range. Connection wires are segmented using Mask2Former [26] (Swin-B backbone) trained on \(950/50\) train/validation frames using AdamW with cosine decay and standard augmentations, achieving a mean IoU of \(0.87\).

10.2.0.3 Topology Reconstruction.

Graph connectivity is inferred from geometric relationships between nodes and sockets. Socket–node associations are determined via spatial overlap, while socket–socket connections are inferred by fitting candidate Bézier curves between output and input sockets and measuring their overlap with the wire segmentation mask.

Combining detected nodes, sockets, inferred edges, and parameter states yields the graph representation (see figure on the right):

\[G_k = (\mathcal{N}_k, \mathcal{S}_k, E_k, \Theta_k),\]

where \(E_k\) represents graph connectivity and \(\Theta_k\) stores node parameters. Each pair \((t_k,G_k)\) therefore represents a temporally grounded snapshot of the evolving material graph.

10.3 Process Summarization↩︎

Finally, the sequence of reconstructed graph states is fused with the narration transcript to produce a textual process trace. Given \(\{(t_k,G_k)\}\) and the narration transcript \(\tau_v\), a language-model summarizer integrates these multimodal signals:

\[p_v = \Phi_{\text{sum}}(\{(t_k,G_k)\}, \tau_v).\]

The summarizer aligns narrated explanations with observed graph edits and produces a compact description of the construction process. Self-refinement prompts ensure that the generated process remains consistent with both the transcript and the reconstructed graph sequence. The resulting trace \(p_v\) captures both explicit construction steps and the procedural reasoning expressed by the artist.

Figures 33 and 34 show example outputs before and after summarization. Raw narration transcripts are typically long (often exceeding \(60\)k tokens for a \(10\) minute tutorial), while the summarized process traces are significantly shorter (typically \(\sim8\)k tokens), making them easier to condition the ProcessSynthesizerduring retrieval-time reasoning.

11 Test Prompts used for Evaluation↩︎

All experimental evaluations—including text-conditioned generation, image-conditioned generation, and ablation studies—use materials sourced from the BlenderKit dataset. Table [tbl:tab:text95prompts95stats] summarizes the distribution of test prompts across the 29 BlenderKit material categories used in our benchmark, averaging 3.9 prompts per category. The source materials used in both coarse and fine-grained editing experiments are also drawn from BlenderKit.

For image-conditioned generation, we additionally evaluate on a set of in-the-wild close-up material photographs collected from the internet. The category distribution of these images is also reported in Table [tbl:tab:text95prompts95stats].

12 Ablations↩︎

In addition to the ablations presented in the main paper, we analyze the contribution of the video analysis tools used for process extraction, the effect of summarizer self-refinement, and the role of the DeCompilerin enabling reliable editing.

12.1 Ablation of Video Analysis Tools↩︎

The video analysis pipeline converts a tutorial video into a structured process trace through three stages. First, Video Decomposition extracts frames and audio, scores frames for informativeness, removes near-duplicates using perceptual embeddings, and produces a time-aligned transcript via ASR. Second, Graph Reconstruction parses each selected keyframe using lightweight detectors, segmentors, and OCR to recover the evolving node graph—its nodes, socket connections, and parameter states such as sliders and color ramps. Finally, Process Summarization fuses these graph states with the narration using an LLM with self-refinement to produce a concise textual process that captures both explicit editing steps and the expert’s underlying rationale.

To assess the role of each stage, we evaluate four ablations of this pipeline and analyze their impact on text-conditioned material generation:

• Audio-only: removes all visual analysis and reduces the pipeline to ASR on the narration followed by process summarization.

• Random Keyframe Sampling: replaces the scored and deduplicated keyframe selection in video decomposition with uniform random sampling.

• GPT-based Frame Parsing: replaces the domain-adapted detectors and segmentors used for graph reconstruction with GPT-5-based vision parsing of each frame.

• w/o Summarizer Self-Refine: removes the iterative self-refinement stage in process summarization, forcing the summarizer to compress the raw multimodal trace in a single pass.

Each ablation isolates a single stage while keeping the rest of the pipeline unchanged. The resulting process traces are then used by MaterialApprenticeduring material generation.

Figures 18–22 illustrates how these components affect generation quality.

Audio-only performs the worst: without spatial information about the node graph, the model cannot resolve references such as “connect the output of the color ramp to the mix factor,” especially when multiple such nodes exist, resulting in severely degenerate graphs.

Random Keyframe Sampling improves over Audio-only by exposing the correct set of nodes, but randomly selected frames rarely capture meaningful structural cues—many contain isolated nodes, occlusions, or uninformative UI states—leading to incorrect or incomplete graph connectivity.

GPT-based Frame Parsing further improves quality since it analyzes visual frames rather than relying purely on narration, but generic vision parsing often misreads Blender UI elements such as ColorRamp stops, color tabs, and slider values, producing inaccurate graph states that degrade the extracted procedural cues.

w/o Summarizer Self-Refine starts from the strongest multimodal trace, but a single-pass summary often overlooks subtle design rationale and parameter intent, yielding weaker guidance for generation.

In contrast, the full pipeline—combining importance-weighted keyframe selection, domain-adapted graph reconstruction, and iterative process summarization—produces the most faithful process traces and consistently delivers the highest-quality generated materials.

To illustrate the role of each component more concretely, we analyze a representative example: generating a raw–beef material. This example is diagnostically useful because high-quality results require (1) multi-scale feature construction, (2) correct node-graph topology, and (3) coherent cross-channel reasoning across color, roughness, bump, and displacement. Even small parsing failures therefore produce large and visually noticeable artifacts. Figures 18–22 show qualitative results for each ablation.

Audio-Only. With only narration available, the system has no access to graph structure or spatial cues present in the tutorial frames. As shown in Figure 18, the reconstructed graph collapses into a minimal structure driven almost entirely by a single reused ColorRamp mask, which is incorrectly repurposed across color, roughness, bump, and displacement. The rendered output becomes a flat two-tone pattern with no fat content, strand flow, or multi-scale variation. Audio alone does not describe the topology, grouping, or numeric structure required for procedural reconstruction.

Random Keyframe Sampling (Figure 19). Using visual frames but selecting them uniformly at random exposes many relevant nodes but rarely captures the structural relationships between them. Important edges and parameter interactions simply never appear in the sampled frames. As shown in Figure 19, the system assembles the correct primitives (Noise, Musgrave, Voronoi) but connects them incorrectly: displacement becomes disconnected from color reasoning, and the intended directional warp collapses into an isotropic bump field. The result resembles a lumpy sausage interior rather than raw beef. This demonstrates that which frames are selected is as important as having frames at all.

GPT-5 Frame Parsing. Replacing the domain-adapted detectors with generic GPT-5 vision parsing yields noisier node states. GPT-5 frequently misreads sliders, color ramps, and numeric parameters—particularly in complex nodes such as ColorRamp or Musgrave. As a result (Figure 20), frequency values become exaggerated, color-ramp thresholds collapse to black, and vector outputs are misinterpreted as scalar fields. The resulting material resembles an over-sharpened sponge rather than meat. This highlights the need for domain-adapted parsing capable of reliably extracting Blender-specific parameters.

No Self-Refinement in Process Summarization. When keyframes and graph reconstruction are correct but the summarizer compresses the verbose process trace in a single pass, subtle structural dependencies are lost. As shown in Figure 21, the graph contains the correct families of nodes (fat masks, noise layers, fiber masks) but their relationships become mis-sequenced or collapsed. The rendering resembles ground meat—plausible in isolation but lacking the anisotropic strand-aligned structure of raw beef. Iterative refinement is required to preserve long-range dependencies across channels.

Full Pipeline. Only the complete pipeline—importance-weighted keyframes, domain-adapted graph reconstruction, accurate color and parameter extraction, and iterative process summarization—successfully reconstructs the procedural logic of the tutorial. As shown in Figure 22, the recovered graph captures the intended multi-scale structure: a warped vector field for strand direction, a mid-frequency Musgrave mask for elongated fibers, Voronoi fat specks modulating color and roughness, micro-noise for fine grain, and consistent cross-channel connections tying these elements together. The resulting render exhibits anatomically plausible marbling, strand flow, and shading, demonstrating that all stages of the pipeline are required to recover the full procedural structure of expert workflows.

12.2 Refinement↩︎

Refinement plays a key role in improving generation quality in MaterialApprentice. We study two forms of refinement: (1) Self-Refine[27], where an LLM critiques and revises its own output, and (2) iterative material-graph refinement using BlenderAlchemy[3]. Motivated by the gains observed from self-refine in the Summarizer, we apply the same strategy to both the Editorand Compilerand evaluate their individual contributions. Figure 23 shows qualitative results across five text-to-material examples. Removing self-refine from either the Editoror Compilercauses a clear drop in fidelity and coherence. Although the raw outputs from the Compilerare already strong, subsequent iterative refinement further improves realism and prompt alignment, demonstrating complementary benefits between LLM-based self-correction and post-hoc graph optimization.

12.3 Decompiler↩︎

The DeCompilerconverts an input material graph \(g_i\) into a process trace \(p_i\), allowing the Editorto operate in the same modality as the retrieved process traces. Using a small bank of exemplar (graph, process) pairs, it aligns substructures in \(g_i\) with analogous patterns in these exemplars to reconstruct a plausible authoring process. Without this step, the Editormust edit graphs directly, which leads to unstable edits and semantic drift. Figure 24 shows this effect on the same editing setup and prompts used in the main paper: bypassing the DeCompilerand supplying the initial graph directly to the Editorproduces noticeably weaker and less controlled edits compared to our full pipeline.

13 Comparison to PhotoMat↩︎

We provide qualitative comparisons with PhotoMat [28], a state-of-the-art non-procedural texture generation method based on diffusion models. Unlike MaterialApprentice, which generates procedural shader graphs, PhotoMat directly synthesizes texture maps from image inputs. Figure 25 shows representative results on several materials. While PhotoMat often captures coarse color statistics, it frequently struggles to reproduce the structured, hierarchical patterns present in many materials. In contrast, MaterialApprenticeproduces outputs that better preserve directional structures and characteristic construction patterns (e.g., the layered fibers in Salmon or the clustered patterns in Salamander Skin), highlighting the advantages of process-based procedural generation.

14 Additional Results↩︎

Additional qualitative results for text conditioned generation, image conditioned generation and text based editing are shown in Figures 26, 27, 28 respectively.

15 Limitations↩︎

MaterialApprenticeis the first system to explicitly leverage tacit expertise in procedural material creation to synthesize new materials, but several limitations remain.

First, expert-made materials often fall into two extremes: highly elaborate graphs (see Fig29) with many interacting layers, or extremely concise graphs that use a few nodes to express rich structure. MaterialApprenticestruggles in both regimes. Large graphs introduce too many degrees of freedom for reliable synthesis, while compact, elegant graphs reflect artistic intuition developed over years—intuition that cannot be captured from a small number of demonstrations 29. Second, as shown in Figure 30, while MaterialApprenticecreated materials are often structurally close, they sometimes lack adequate parameter tuning which may require some human inputs to achieve desired quality. Lastly, MaterialApprenticedepends on a powerful but slow and expensive backbone LLM (GPT-5) for both video understanding and process synthesis in a single forward pass. Future work could explore more efficient multi-agent or modular architectures that decompose the task—e.g., separating video parsing, structural reasoning, and graph generation—allowing these components to be handled by smaller, faster, and more accessible models.

References↩︎