1 Introduction↩︎

As the primary factor to determine the visual appearance of in-the-wild scenes, an accurate lighting representation is the foundational cornerstone for photorealistic augmented reality [1], virtual object insertion [2], and scene relighting [3], [4]. However, recovering the complete environment map from a single in-the-wild video remains a significant challenge, due to the inherently partial illumination observations and the complex interactions among camera motion, object occlusions, and diverse material properties.

Classical physics-based methods [5] typically solve a simplified problem, assuming videos have static lighting, Lambertian surfaces, or known geometry. Meanwhile, multi-view approaches [6] solve the problem by introducing external guidance (, images from additional perspectives). However, both paradigms fundamentally deviate from the intended task of processing an in-the-wild video. In contrast, recent work [7] feeds videos as conditions to generate low dynamic range (LDR) results for high dynamic range (HDR) fusion. While this approach presents a practical solution, it treats the model as a simple translator, ignoring its inherent generative priors for illumination and thus potentially producing implausible results, such as exposure-fusion artifacts, color bias, and temporally unstable reflections.

Recent video generation models [8]–[12] have demonstrated an impressive capability to produce photorealistic scenes with complex light and shadow interactions, which highlights their implicit understanding of illumination and suggests an inherent capability for lighting estimation. This motivates us to explore these capabilities to directly generate HDR environment maps. However, extending existing video generation frameworks for lighting estimation introduces two key challenges: (i) adapting the model to handle HDR processing, and (ii) enabling HDR representations within the latent diffusion model.

In this paper, we introduce V-LITE (Video generation models are inherent LIghTing Estimators), a unified HDR-integrated framework that estimates environment maps directly from a single in-the-wild video. We are inspired by the foundational practice in the visual effects industry, where a light probe (typically a chrome ball) is inserted into a scene to capture the complete surrounding illumination [13], [14]. This enables rendering virtual objects with photorealistic lighting consistency. Therefore, we reformulate the lighting estimation problem as a video inpainting task, requiring the model to fill in a synthetic chrome ball, which is then unwrapped to produce the final environment map. This approach forces the video generation model to consider both spatial context and temporal consistency when plausibly inserting the ball, thereby inherently leveraging its generative priors. As shown in 1, our method robustly generates high-quality environment maps even in extreme cases (, fast camera motion).

We build our framework upon a diffusion-based network for LDR video generation [11]. To process HDR videos with original LDR Variational AutoEncoder (VAE), we design and integrate learnable tonemap adapters, ensuring the fine-grained illumination cues (, color temperature variations across time) are faithfully preserved in the latent space. To further align the pretrained latent space with these HDR latents, we introduce a LoRA-based fine-tuning scheme [15]. This effectively enhances the lighting understanding while preserving its original generative priors for spatial and temporal context. To facilitate the model training and evaluation, we also construct V-LITESet, a dataset including over 8K HDR video pairs with dynamic temporal lighting and 800 static HDR images, totaling over 648K frames for effective co-training and evaluation.

Our contributions can be summarized as follows:

• We reformulate video lighting estimation as a light probe inpainting task, producing a dynamic environment map from a single in-the-wild video.

• We design an HDR-aware VAE to preserve lighting cues and a LoRA-based fine-tuning scheme to align the latent space with HDR representations.

• We construct a hybrid dataset comprising 8K videos with dynamic temporal lighting and 800 images with diverse luminance distributions, facilitating robust model training and comprehensive evaluation.

2 Related Work↩︎

2.1 Lighting Estimation from Images↩︎

Estimating scene illumination from a single image has been studied extensively in both vision and graphics. Early deep methods directly regress HDR indoor environment maps from limited-FoV LDR photos, showing that plausible light probes can be inferred without explicit geometry or material supervision [16]. Subsequent works decompose the task into geometry/visibility completion and LDR-to-HDR mapping to better preserve high-frequency cues and material-dependent effects [17], [18]. To improve spatial fidelity, fast indoor lighting estimation is explored [19], while HDR panorama generation enables lighting editing and robust HDR mapping [20]–[22]. Neural inverse rendering provides complementary, physics-grounded priors by jointly reasoning about geometry, BRDFs, and illumination from a single image [23]. For outdoor scenes, single-image methods often predict parametric sun/sky lighting [24]. Despite these advances, single-image approaches typically lack temporal coupling and rarely provide native HDR training/inference under unconstrained conditions.

2.2 Lighting Estimation from Videos↩︎

Moving from images to videos introduces the challenge of maintaining temporal consistency while still modeling dynamic illumination. Early progress leverages multi-view or stereo cues to infer lighting volumes with 3D coherence [6]. Closer to our setting, Li  [5] represent video illumination as a spatiotemporally consistent spherical-Gaussian lighting volume and train recurrent priors to suppress flicker and enforce cross-frame consistency for indoor HDR lighting. Recent work further studies spatiotemporally consistent indoor lighting estimation directly from in-the-wild videos, emphasizing smoothness of predicted lighting fields under changing viewpoints and local light variations [25]. In parallel, sequence-based HDR lighting reconstruction methods encode spatially-varying illumination as Gaussian splats from image sequences [26], but typically require controlled capture settings and do not target generic casually captured videos. Methods that aim at lightweight 3D-coherent representations can also improve spatial consistency (, LightOctree), but they are primarily designed for single-image inputs and do not directly couple predictions across time [27]. However, robust estimation for in-the-wild videos that is simultaneously temporally consistent and supports native HDR remains under-explored, which we address by leveraging video diffusion priors and task-specific adaptation.

2.3 Generative Diffusion Priors↩︎

Generative diffusion priors have been widely used in various computer vision tasks. Due to their strong capability to model the complex distribution of natural images, they are naturally explored to handle low-level vision tasks (, dehaze [28], colorization [29], and super-resolution [30]). This further motivates researchers to extend their application to classic vision tasks (, object detection [31], segmentation [32], and pose estimation [33]) and even 3D vision for reconstruction [34] and point cloud completion [35]. Recently, researchers have also demonstrated that video diffusion models have zero-shot reasoning capabilities for the visual world, implicitly encoding complex physics, geometry, and material optics [36]. Based on these previous works, we reformulate the video lighting estimation problem and explore the approach to leverage the generative priors of video diffusion models to capture the dynamic illumination.

3 V-LITESet Dataset↩︎

Existing large-scale video generation datasets [37], [38] primarily focus on LDR video content, making them ill-suited for developing HDR models. In contrast, existing HDR video datasets either focus on indoor scenarios [16] or provide limited in-the-wild samples [39]. To overcome these limitations, we introduce V-LITESet, a high-quality dataset specifically designed for training and evaluating HDR video lighting estimation.

Dynamic spatiotemporal subset. We build the dynamic subset of V-LITESet upon the PanoVid dataset [40], which includes diverse in-the-wild LDR panoramic videos. To lift these LDR videos to HDR, we employ a rigorous offline curation pipeline. Rather than assuming a single camera response function, we process the source panoramas using three distinct inverse tone-mapping curves (linear, \(\gamma=2.2\), and filmic) to span a diverse set of candidate HDR distributions. To filter out severe photometric losses, we map these candidates back to the LDR domain, and query a VLM [41] to evaluate the physical plausibility of the frames (, unnatural color banding, severe overexposure, and irreversible detail loss). Approximately 57% of the raw sequences are discarded due to low confidence scores. The remaining videos provide robust dynamic spatiotemporal context.

Static photometric subset. Since photometric approximations for tonemapped HDR videos may introduce certain artifacts, we additionally integrate 800 high-quality HDR panoramic images from PolyHaven [39] to form the static subset of V-LITESet. We adapt these images into static video sequences to match the input format of our video diffusion backbone. These real-world samples serve as absolute physical anchors during training, enabling the model to effectively learn the intrinsic luminance distributions and HDR contrasts of light sources (, the accurate intensity ratio between the sun and the sky).

Summary. The final V-LITESet comprises over 8K video pairs at \(480 \times 832\) resolution with dynamic temporal lighting, alongside 800 HDR static sequences with diverse luminance distributions. This amounts to over 648K frames, split into 567K training and 81K evaluation samples. This provides a robust and comprehensive foundation for advancing research in dynamic HDR lighting estimation from a single video.

4 Methodology↩︎

In this section, we introduce the details of our V-LITE framework. We first review the flow-matching video generation and editing models that form our foundation (4.1). Next, we propose the end-to-end VAE for HDR video processing to adapt the pretrained video backbone into the HDR domain (4.2). Finally, we present our strategy to reformulate lighting estimation as a video inpainting task, leveraging its inherent generative priors to accurately capture the scene’s dynamic lighting (4.3). The overall pipeline is shown in 3.

4.1 Preliminaries↩︎

Video generation models Video generation models have demonstrated strong capabilities in producing diverse and high-quality dynamic content. As a representative practice of the video backbone, Wan 2.1 [11] adopts a diffusion Transformer architecture, modeling complex spatio-temporal dynamics through a latent-space training scheme based on the flow matching framework [42].

Given a target video \(x_1\), the pretrained VAE encoder \(\mathcal{E}\) first maps it into a latent code \(z_1 = \mathcal{E}(x_1)\). During training, the model randomly samples a noise latent \(z_{0} \sim \mathcal{N}(\mathbf{0}, \mathbf{I})\), and creates an intermediate latent code via linear interpolation for a timestep \(t \in [0,1]\): \(z_t = t \cdot z_1 + (1 - t) \cdot z_0\). The model is trained to predict the ground-truth velocity field associated with the interpolation path. Formally, the training objective is: \[\begin{align} \mathcal{L} = \mathbb{E}_{z_0, z_1, c_{\mathrm{txt}}, t} \big [ \| v_\theta(z_t, c_{\mathrm{txt}}, t) - (z_1 - z_0) \|^2 \big ], \end{align}\] where \(c_{\mathrm{txt}}\) denotes the text condition and \(\theta\) represents the model parameters. \(v_\theta(\cdot)\) is the diffusion Transformer to predict the velocity field at latent code \(z_t\).

Video editing models Video editing aims to modify the video content based on the text description indicating the editing direction and the semantics of the original video. To selectively transform targeted regions while maintaining temporal and spatial coherence in unedited areas, a recent method [43] proposes to incorporate text, mask, and other visual conditions into a unified conditioning representation. This conditioning representation is injected into the video backbone via an additional adapter. The training objective is thus formulated as: \[\begin{align} \label{eq:editing} \mathcal{L} = \mathbb{E}_{z_0, z_1, \mathcal{V}, t} \big[ \| v_\theta(z_t, \mathcal{V}, t) - (z_1 - z_0) \|^2 \big], \end{align}\tag{1}\] where \(\mathcal{V} = (T, V, M)\) denotes the conditioning input, with \(T\) representing the text description, \(V\) denoting the visual conditions, and \(M\) providing the corresponding masks.

4.2 HDR VAE Processing↩︎

The success of leveraging inherent generative priors for lighting estimation depends critically on operating in the HDR domain. However, since existing video generation backbones [8], [11] are designed for LDR content, it is essential to adapt their architectures to accurately support end-to-end HDR processing.

Existing methods [7], [14], [44] typically address this gap by producing multiple LDR variants (, exposure-bracketed frames) that are subsequently fused to approximate an HDR result. However, such approaches require multiple inference passes or an additional fusion stage, leading to increased computational cost and potential error accumulation across the reconstruction pipeline. In contrast, we propose to integrate the HDR domain conversion directly into the VAE, bridging the distribution gap within the latent space. As presented in 3 (a), this design inherently makes the LDR-trained video backbone compatible with HDR data.

Tonemap adapters. We first introduce a pair of learnable tonemap adapters and integrate them into the VAE [11] for end-to-end HDR video processing. Specifically, given a linear HDR video \(x_\mathrm{HDR}\), our tonemap adapter converts it to the log domain. It then applies a single 3D convolutional layer \(\mathcal{F}\), a learnable scale \(\mathbf{s}\) and bias \(\mathbf{b}\) to calculate the log-domain representation: \[\begin{align} \label{eq:tonemap} x_{\log} = \mathbf{s} \odot \left[\log(x_\mathrm{HDR}+\epsilon) + \mathcal{F}\big(\log(x_\mathrm{HDR}+\epsilon)\big)\right] + \mathbf{b}. \end{align}\tag{2}\] This adaptive mapping enables the encoder to dynamically stretch and compress signal ranges. The log domain representation \(x_{\log}\) is fed into the LDR VAE encoder \(\mathcal{E}\) to generate the compressed latent code \(z=\mathcal{E}(x_{\log})\). After that, the HDR VAE decoder \(\mathcal{D}_\mathrm{HDR}\) produces the log-domain representation \(x'_{\log} = \mathcal{D}_\mathrm{HDR}(z)\). The inverse tonemap adapter then maps this representation back to the linear HDR domain: \[\begin{align} \label{eq:inverse95remap} x'_{\mathrm{HDR}} = \exp\!\left[ \frac{x'_{\log} - \mathbf{b}}{\mathbf{s}} + \mathcal{G}\!\left( \frac{x'_{\log} - \mathbf{b}}{\mathbf{s}} \right) \right], \end{align}\tag{3}\]

where \(\mathcal{G}\) is a single 3D convolutional layer and \(x'_{\mathrm{HDR}}\) is the reconstructed HDR video. During the training process, these lightweight adapters learn the dynamic illumination and extreme ranges within a stable log-domain latent space. Notably, our HDR VAE decoder has the exact same architecture as the pretrained LDR VAE from the video backbone [11].

Training scheme. The VAE training is formulated as a reconstruction task, requiring the model to minimize the error between the input video \(x_{\mathrm{HDR}}\) and the reconstructed video \(x'_{\mathrm{HDR}}\). To adapt our VAE with tonemap adapters for stable HDR processing, we employ a two-stage finetuning strategy. In the first stage, we freeze all VAE parameters and train the two tonemap adapters to establish a stable log-domain representation. In the second stage, we unfreeze the VAE decoder and jointly finetune it with the adapters. After finetuning, the combined VAE and tonemap adapters operate directly in the HDR domain, preserving visual fidelity across extreme luminance ranges typical of HDR data.

4.3 Inherent Lighting Estimation↩︎

Previous work [7] employs video generation models as a video translator for lighting estimation. In contrast, observing that recent video generation models [11] demonstrate an implicit ability to model lighting, producing temporally and spatially light-coherent results, we are motivated to directly exploit their internal lighting modeling capability rather than formulating the task as a translation problem. As illustrated in 3 (b), we reformulate the task of video lighting estimation as an inpainting problem.

Since the central region of the scene is typically irrelevant for global illumination, we leverage a central mask \(M\) to define the inpainting region for the synthesis of a physically plausible chrome ball. Given an in-the-wild LDR video \(x_{\mathrm{LDR}}\), we construct the masked input as \(x_{\mathrm{input}} = x_{\mathrm{LDR}} \odot (1 - M) + C \odot M\), where \(C\) is a constant gray value. We then decouple this input into the environment region \(x_e = x_{\mathrm{input}} \odot M\) and the scene content \(x_s = x_{\mathrm{input}} \odot (1 - M)\) as two conditions for the generation model. This setup allows the model to simultaneously reconstruct the HDR environment map from the surrounding regions and enhance the dynamic range of the original scene.

Diffusion process. During training and inference, both \(x_s\) and \(x_e\) are represented in the latent space as \(z_s\) and \(z_e\) using the pretrained LDR VAE encoder \(\mathcal{E}\). At each inference step \(t\), we utilize a condition adapter [43] to inject these latent codes into the video backbone to predict the velocity field: \[\begin{align} \hat{v} = f_{\theta}\bigl(\hat{z}_t, f_{c}([z_s, z_e, c_{\mathrm{txt}}]), c_{\mathrm{txt}})\bigr), \end{align}\] where \(f_{\theta}\) denotes the diffusion Transformer of the video backbone, \(f_{c}\) denotes the condition adapter for video inpainting, and \(c_{\mathrm{txt}}\) represents the text embedding driving the model to fill the chrome ball. During inference, we iteratively solve the ordinary differential equation \(\textrm{d}z_t/\textrm{d}t = \hat{v}\) from \(t = 1\) to \(t = 0\) using a numerical solver to synthesize the chrome ball. In particular, the prompt we provide is “A perfect mirrored reflective chrome sphere, near the camera, on top of everything”, which explicitly specifies the appearance and placement of the chrome ball.

After the diffusion process, our pretrained VAE decoder \(\mathcal{D}_\mathrm{HDR}\) is employed to reconstruct HDR video frames \(\hat{x}_{\mathrm{HDR}}\). To extract the final HDR environment map \(\mathbf{E}\), we perform a post-processing step, where the chrome ball is isolated from the reconstructed HDR frames using the corresponding mask and re-projected into the environment map: \[\begin{align} \mathbf{E} = f_{\mathcal{R}}(\hat{x}_{\mathrm{HDR}} \odot M), \end{align}\] where \(f_{\mathcal{R}}\) denotes the equirectangular transformation. This procedure produces a high-fidelity and temporally coherent sequence of HDR environment maps that faithfully capture the dynamic lighting of the scene.

Optimization process. We train only the core video diffusion Transformer using LoRA-based fine-tuning [15] to align its generative priors with the HDR latent distribution. During training, we randomly mix samples from the dynamic spatiotemporal and static photometric subsets at a \(10:1\) ratio. To enable this joint training, we replicate the HDR images within the photometric subset to form static video sequences. Consequently, this mixed-data training strategy ensures that the model preserves its inherent temporal consistency priors through the dynamic videos, while utilizing the static sequences as physical anchors to learn accurate luminance distributions (, precise sun intensities).

To simultaneously reconstruct the HDR environment map and enhance the dynamic range of the original scene context, we define our objective as follows: \[\begin{align} \mathcal{L} &= \mathbb{E}_{z, \epsilon, t} \big[ \|M' \odot (v_\theta(z_t, \mathcal{C}, t) - v )\|^2 \big], \end{align}\] where \(\mathcal{C}=[z_\text{cond}, c_{\mathrm{txt}}]\) denotes the full conditioning input, \(z_\text{cond} = \{z_e,z_s\}\), and \(\epsilon \sim \mathcal{N}(\mathbf{0}, \mathbf{I})\) represents a latent noise sample. Here, \(v=z-\epsilon\) denotes the target velocity, \(v_\theta\) is the predicted velocity field, and \(M'=1+(\alpha-1)M\) is a modulation mask with a weighting factor \(\alpha=5\) to explicitly emphasize the lighting estimation within the central inpainting region.

5 Experiments↩︎

In this section, we conduct a series of experiments to validate the effectiveness of our proposed method, V-LITE. We first introduce the experimental setup, including our implementation details, the dataset used, and the evaluation metrics. We then present comprehensive quantitative and qualitative comparisons against state-of-the-art methods. Finally, we perform detailed ablation studies to analyze the contribution of each key component in our framework.

We compare V-LITE against DiffusionLight [14], DiffusionLightTurbo [44] and StyleLight [20]. Since most existing methods do not support dynamic lighting from a single video, we adapt state-of-the-art single-image lighting estimation methods to the video domain by applying them in a per-frame manner.

5.1 Implementation Details↩︎

Our framework is built upon the pretrained Wan 2.1-1.3B video model [11]. Our custom-trained HDR-aware VAE is specifically designed to operate in the linear Rec. 2020 color space, which provides a wide-gamut, scene-linear representation aligned with HDR standards, ensuring physically accurate color reproduction. To effectively model the high dynamic range, the VAE’s latent codes are compressed into a log space before being processed by the core diffusion model. For HDR-aware VAE fine-tuning, we adopt a two-stage training scheme with \(10\)k steps in the first stage and \(5\)k steps in the second. We optimize the model using the standard VAE objective, which includes the reconstruction loss and KL divergence loss, identical to the losses used in the original VAE training. For diffusion Transformer fine-tuning, we employ LoRA [15] with a rank of \(r=32\) and apply it to the attention layers of the diffusion Transformer. The model is trained on our V-LITESet for 100K steps with a global batch size of 64 on 8 NVIDIA H100 GPUs. We use the AdamW optimizer with a learning rate of \(1 \times 10^{-5}\). During training and inference, the input videos containing the masked sphere are processed at a resolution of \(480 \times 832\). After the model inpaints the sphere, its reflection is unwrapped to produce the final HDR environment map at a resolution of \(256 \times 512\), which is commonly used in prior work [7], [14], [20], [44].

5.2 Quantitative Evaluation↩︎

Evaluation protocol. We conduct quantitative evaluations on two public benchmarks following the evaluation protocol of DiffusionLight [14]. (i) Editable Indoor assesses illumination quality through scene rendering and reports MSE, SI-MSE, Angular Error in Radians (AER), and Lighting Stability (LS, the standard deviation of SI-MSE). (ii) EnvMapNet estimates the dominant lighting direction from the environment maps and reports both the Angular Error in Degrees (AED) and Angle Stability (AS, the standard deviation of AED).

Baselines. We compare our method against DiffusionLight [14], DiffusionLight-Turbo [44], and StyleLight [20]. Note that DiffusionLight is extremely time-consuming (, over one day for a single video), so we report comparisons on a subset of 10 videos. The quantitative results are provided in ¿tbl:tab:main95quantitative95results?. Compared to DiffusionLight, our model achieves competitive lighting accuracy and superior temporal stability while requiring only a fraction of the computation time. Compared to DiffusionLight-Turbo and StyleLight, our model attains the best overall performance.

5.3 Qualitative Evaluation↩︎

We provide a qualitative comparison with state-of-the-art image relighting methods in 4. The baselines including DiffusionLight [14], DiffusionLight-Turbo [44], and StyleLight [20], are designed for single-image manipulation. To adapt them for video evaluation, we apply each method independently to each frame of source video sequences. As illustrated, this per-frame approach leads to significant temporal artifacts. The baseline methods exhibit noticeable flickering, where the intensity and color of highlights and shadows change erratically between consecutive frames. This is an expected outcome, as these models lack any inherent mechanism to enforce temporal consistency.

Generalization to in-the-wild videos. As shown in 5, V-LITE generalizes to diverse in-the-wild videos under both day and night conditions, producing spatially accurate and temporally coherent lighting estimates even in complex real-world footage.

5.4 Ablation Study↩︎

To validate our key design choices, we conduct a set of ablation experiments that isolate the contributions of the HDR-aware VAE, the LoRA-based fine-tuning procedure, and our mixed-data training strategy. We evaluate three ablated variants against our full model and report the quantitative results in ¿tbl:tab:main95quantitative95results?.

LDR baseline. We first evaluate the original pretrained model of Jiang  [43] without both our HDR VAE and the fine-tuning stage. Although this LDR baseline can inpaint chrome balls in videos, it cannot generate HDR content. To enable a comparison with HDR-capable models, we apply the inverse tone-mapping derived from the HDR ground truth to lift its LDR predictions into the HDR domain. The results support our claim that while video diffusion models inherently encode lighting information, they lack the representational precision to express it in the HDR space.

Frozen backbone. We incorporate our pretrained HDR VAE directly into the baseline model without any task-specific fine-tuning. This configuration introduces a domain mismatch between the original LDR training distribution and the new HDR latent space, significantly degrading the generative performance.

Video-only. To evaluate the impact of our mixed-data training strategy, we fine-tune the model equipped with the HDR VAE exclusively on the video dataset. While this variant aligns the generative priors to the video inpainting task, relying solely on tonemapped HDR video data limits the model’s ability to predict accurate physical HDR values.

Instead, our full model incorporates both the HDR VAE and the LoRA-based fine-tuning, utilizing a mixed-data training strategy. It outperforms all ablation variants, demonstrating that all proposed components are indispensable.

5.5 Virtual Object Insertion↩︎

We demonstrate the effectiveness of our estimated HDR environment maps by inserting virtual objects into real-world videos. We first process the input LDR video to solve for the 3D camera motion and establish a ground plane using built-in motion tracker from Blender [45]. The dynamic HDR environment map sequence generated by V-LITE is then used as the exclusive lighting source for a virtual object. This object is rendered from the solved camera’s perspective and composited back onto the original video. The results in 6 show that the virtual objects are seamlessly integrated, with lighting and shadows that are both physically plausible and temporally consistent.

To rigorously validate perceptual realism and lighting consistency, we conducted a user study with 10 participants across 20 diverse videos. For perceptual realism, 64% of responses rated our insertions as “Perfect”, and 28% as “Acceptable”. For lighting consistency, 66.5% rated the results as “Perfect”, with less than 2% of total responses considered failures.

5.6 Discussion↩︎

Tonemapped HDR videos and mixed-data training. Capturing in-the-wild HDR videos requires specialized multi-exposure cameras, which fundamentally limits dataset scale and scene diversity. To learn complex dynamic lighting, our V-LITESet utilizes computationally tonemapped HDR videos. While these are not photometrically perfect HDRs, our rigorous VLM-based filtering explicitly discards 57% of heavily clipped data to ensure reasonable intensity ratios. By mixing these dynamic tonemapped videos with static real-world HDR images, V-LITE effectively integrates absolute photometric priors to overcome the physical scarcity of real-world HDR video data.

Ill-posed estimation and visual trade-offs. Estimating an HDR environment from a single LDR observation is an inherently ill-posed problem, as the original tone-mapping process is non-invertible. Because V-LITE leverages the generative priors of diffusion models to naturally resolve this ambiguity rather than performing a strict pixel-to-pixel translation, the output may exhibit slight background visual shifts (, lower contrast on certain bright surfaces). These are expected physical trade-offs for accurately extracting HDR reflections and dominant light directions. As evidenced by our high-fidelity virtual object insertions, V-LITE robustly estimates practical dynamic lighting.

Zero-shot generalization and spatial variance. We further examine the influence of the probe’s position and size, observing that the inpainted reflections dynamically respond to its spatial positioning. As depicted in 7, rows 1–3 illustrate off-center placements from top-left to top-right, while row 4 explores different mask coverage ratios (30%, 50%, and 70%). All preserve structural integrity and realistic illumination, implying that V-LITE inherently captures underlying scene geometry.

6 Conclusion↩︎

We present V-LITE, a unified HDR-aware framework that produces dynamic HDR environment maps from in-the-wild videos. We also introduce V-LITESet, a hybrid dataset of tonemapped HDR videos and real-world HDR images curated to advance video-based lighting estimation. Our formulation natively treats lighting recovery as a latent video-inpainting problem, letting the model fully exploit the generative priors of video diffusion models. Experiments show that V-LITE delivers high-fidelity, temporally coherent environment maps in challenging dynamic scenes.

Limitations and failure cases. While V-LITE demonstrates strong generalization, its data-driven nature makes it sensitive to out-of-distribution inputs. As illustrated in 8, videos with extreme illumination conditions can occasionally lead to mismatched environment maps. Additionally, since our underlying video backbones [8], [11] struggle with long-duration generation, V-LITE’s ability to estimate temporally stable lighting for extended sequences remains constrained.

Acknowledgment↩︎

This work is supported by National Natural Science Foundation of China (Grant No. 62136001) and Beijing Major Science and Technology Project (Grant No. Z251100008). PKU-affiliated authors thank openbayes.com for providing computing resources.

References↩︎