1 Introduction↩︎

Reconstructing dynamic human-scene environments from monocular videos is an important problem in computer vision, with applications in novel view synthesis [1], [2], virtual and augmented reality [3], and digital human modeling [4]. Given only a monocular video, the goal of dynamic human-scene reconstruction is to jointly recover scene geometry, camera motion, and dynamic humans, while enabling photorealistic novel view synthesis.

Existing 3D/4D reconstruction approaches can be broadly categorized into optimization-based and feed-forward methods. Optimization-based methods, including Neural Radiance Fields (NeRF) [5] and 3D Gaussian Splatting (3DGS) [6]-based approaches, optimize 3D/4D scene representations for each scene [5]–[9]. While these methods achieve high reconstruction quality, the optimization process is costly, making them impractical for fast or real-time inference [10]–[12]. Moreover, in the 4D reconstruction setting, most methods require multi-view videos [7], [8], [13] or additional sensors such as LiDAR or depth [14], limiting their applicability in real-world scenarios.

In contrast, feed-forward methods predict 3D geometry directly from images in a single forward pass, enabling fast inference [1], [15]–[20]. These approaches generalize well to unseen scenes by leveraging strong geometric priors learned from large-scale data [21]–[36]. Yet, handling dynamic humans and achieving photorealistic rendering quality at the same time remains a significant challenge [1], [37].

As summarized in Table 1, existing feed-forward methods either do not explicitly model dynamic humans or do not provide photorealistic renderable representations. In this work, we take a first step toward feed-forward photorealistic, renderable dynamic human-scene reconstruction from monocular videos. To this end, we leverage geometric and human priors [37] and lift them into a unified 3DGS representation. Our representation consists of dynamic human Gaussians and static scene Gaussians, whose appearance is predicted by respective decoders for humans and scenes. Our method enables feed-forward reconstruction of dynamic human-scene environments while preserving the photorealistic rendering quality of a 3DGS representation.

Our contributions are as follows:

• We tackle a new problem setting, feed-forward photorealistic, renderable dynamic human-scene reconstruction from monocular videos, and establish a strong baseline.

• We design an architecture that bridges human-scene foundation models and photorealistic rendering by leveraging geometric priors and SMPL-X [38] representations.

• We demonstrate that our method achieves competitive novel view synthesis quality against decomposition-based feed-forward baselines and an optimization-based human-scene baseline, while being significantly more efficient.

r0.45

2 Related Work↩︎

2.1 3D Reconstruction↩︎

Early approaches to 3D reconstruction typically rely on multi-view geometry pipelines such as structure-from-motion (SfM) [39]–[42] and multi-view stereo (MVS) [43], [44], which recover camera poses and explicit 3D structure from image correspondences. Neural rendering methods such as NeRF [5] and Gaussian-based methods such as 3DGS [6] have also been introduced, which represent scenes using continuous or point-based representations optimized through differentiable rendering, enabling photorealistic novel view synthesis. While these approaches produce high-quality reconstructions, they rely on iterative optimization over camera parameters and scene representations, making inference computationally expensive [10], [45].

Recent feed-forward reconstruction methods [15]–[17], [46] predict scene geometry, including point maps and camera parameters, directly from input images, enabling fast inference and generalization to unseen scenes without per-scene optimization. They achieve this by learning strong geometric priors from large-scale data [21]–[36] and leveraging transformer-based architectures with the help of strong backbone features [47], [48]. Feed-forward approaches have also been extended to photorealistic rendering with 3DGS representations [1], [18]. For example, AnySplat [1] directly predicts Gaussian parameters from images, using the geometric priors of VGGT [17].

However, these feed-forward approaches primarily focus on static scenes and often struggle when dynamic objects are present, leading to inconsistent geometry and degraded reconstruction quality [1], [18]. Our work addresses this limitation by explicitly disentangling dynamic humans from static scenes and models them separately within a unified 3DGS framework.

2.2 4D Reconstruction↩︎

Reconstructing dynamic scenes from image sequences has also been widely studied. Many existing approaches [7], [9], [49]–[51] rely on optimization-based pipelines that model scene dynamics using deformation fields or canonical representations. They typically require multi-view synchronized video inputs [7], [51]–[53] and costly optimization, limiting their scalability and applicability in real-world scenarios. More recently, several works [19], [20], [54], [55] attempt feed-forward reconstruction of dynamic scenes from monocular videos. For example, CUT3R [19] directly predicts time-varying scene structures from input images using a recurrent architecture by introducing state tokens.

However, existing methods either rely on expensive optimization for high-quality reconstruction [7], [49] or adopt simplified geometric representations in feed-forward settings [19], [20], resulting in a trade-off between efficiency and representation quality. In contrast, our method adopts a 3D Gaussian representation within a feed-forward framework, enabling both efficient inference and photorealistic rendering.

2.3 Human-Scene Reconstruction↩︎

Human-scene reconstruction aims to jointly recover the 3D scene geometry, human motion, and camera poses from visual observations. Early approaches typically formulate this problem as a global optimization over multiple elements, including camera poses, reconstructed scenes [15], [46], [56], and SMPL [38], [57] mesh parameters [58], [59], often regularized with learned motion priors. Several works further adopt renderable representations such as NeRF or 3DGS [4], [11], [12], [60], achieving high visual fidelity but requiring costly test-time optimization.

To improve efficiency, recent works explore feed-forward alternatives such as HAMSt3R [61] and JOSH3R [62]. More recently, Human3R [37] proposes a one-step feed-forward model that predicts static scenes as point clouds and dynamic humans as SMPL-X meshes [38] in a single forward pass. While these methods provide strong geometric priors, their outputs are not a unified renderable representation, since scenes and humans are represented as point clouds and parametric meshes, respectively. In contrast, our method transforms both static scenes and dynamic humans into a common 3D Gaussian representation, enabling coherent and photorealistic human-scene reconstruction in a feed-forward manner.

3 Method↩︎

3.1 Overview↩︎

Given a monocular video \(\{\boldsymbol{I}_t\}\), where \(\boldsymbol{I}_t\) denotes the input frame at time \(t\), our goal is to reconstruct a dynamic human-scene representation with accurate geometry and photorealistic appearance. We decompose the entire scene \(\boldsymbol{G}_{t}\) into a static scene and dynamic humans, represented as a set of scene Gaussians \(\boldsymbol{G}^{\mathrm{s}}_{t}\) aggregated across time until frame \(t\) and time-dependent human Gaussians \(\{\boldsymbol{G}^{\mathrm{h}}_{t,k}\}\) for each frame \(t\), where \(k\) indexes individual humans.

We build upon a human-scene foundation model [37] that provides geometric priors and structured token representations. Our key idea is to leverage these priors while disentangling appearance modeling for the static scene and dynamic humans. Specifically, we introduce a Scene Gaussian Decoder and a Human Gaussian Decoder, detailed in Sec. 3.3 and Sec. 3.4, respectively. These designs allow us to reconstruct the full scene in a feed-forward manner from monocular input. An overview is shown in Fig. 1.

3.2 Preliminaries↩︎

Human-Scene Foundation Model. We build upon a unified feed-forward model for human-scene reconstruction, referred to as Human3R [37]. Given a monocular video \(\{\boldsymbol{I}_t\}\), the model jointly estimates camera pose \(\boldsymbol{T}_t\), scene point maps \(\boldsymbol{X}_t\), and a set of human meshes as vertices \(\{\boldsymbol{V}_{t, k}\}\) where \(t\) indexes time and \(k\) indexes individual humans.

Scene geometry is represented as point clouds, and human bodies are represented as parametric SMPL-X [38] meshes. Given the local human pose (relative axis-angle rotations) \(\boldsymbol{\theta} \in \mathbb{R}^{52 \times 3}\), shape \(\boldsymbol{\beta} \in \mathbb{R}^{10}\), facial expression \(\boldsymbol{\alpha} \in \mathbb{R}^{10}\), and global human root transformation \(\boldsymbol{P} = [\boldsymbol{R} | \boldsymbol{t}] \in \mathrm{SE}(3)\) parameterized by global orientation \(\boldsymbol{R} \in \mathrm{SO}(3)\) and translation \(\boldsymbol{t} \in \mathbb{R}^3\), the model outputs an expressive 3D human mesh with 10,475 vertices \(\boldsymbol{V}_{t, k}\): \[\boldsymbol{V}_{t, k} = \mathrm{SMPL\text{-}X}(\boldsymbol{\theta}, \boldsymbol{\beta}, \boldsymbol{\alpha}, \boldsymbol{P}).\]

The backbone [48], [59] follows a recurrent formulation. Each frame \(\boldsymbol{I}_t\) is encoded into image tokens \(\boldsymbol{F}_t\). Then the model obtains human detections using a detector [59] with a confidence threshold, which produces a set of 2D locations \(\boldsymbol{u}_{t,k}\) for each frame. Each human token \(\boldsymbol{h}_{t,k}\) is obtained by sampling \(\boldsymbol{F}_t\) at the detected human location \(\boldsymbol{u}_{t,k}\). All human tokens at frame \(t\) are concatenated as \(\boldsymbol{H}_t = \{\boldsymbol{h}_{t, k} \}\). At the same time, the model maintains a persistent state \(\boldsymbol{S}_t\) and a learnable parameter \(\boldsymbol{z}_t\) corresponding to camera state, jointly updating all tokens through a decoder: \[[\boldsymbol{F}_t', \boldsymbol{z}_t', \boldsymbol{H}_t'], \boldsymbol{S}_t = \mathrm{Decoder}([\boldsymbol{F}_t, \boldsymbol{z}_t, \boldsymbol{H}_t], \boldsymbol{S}_{t-1}).\] From the updated image, camera, and human tokens, Human3R predicts scene point maps \(\boldsymbol{X}_t\), camera poses \(\boldsymbol{T}_t\), and human parameters \(\{\boldsymbol{p}_{t, k}\}\) for all detected humans. This design enables disentangled representations of scene and human via image and human tokens, respectively, while jointly reasoning through a shared recurrent state. In our method, we leverage the reconstructed point clouds \(\boldsymbol{X}\) and human mesh vertices \(\boldsymbol{V}\) as geometric priors and further utilize both image tokens \(\boldsymbol{F}\) and human tokens \(\boldsymbol{H}\) to extract appearance features for downstream modeling.

3D Gaussian Splatting. To achieve photorealistic rendering of both scene and human, we adopt 3D Gaussian Splatting (3DGS) [6], which represents a 3D scene as a set of anisotropic Gaussian primitives. Each Gaussian \(\boldsymbol{g}\) models a local volumetric element and is parameterized as: \(\boldsymbol{\mu}_g, \boldsymbol{\alpha}_g, \boldsymbol{q}_g, \boldsymbol{s}_g, \boldsymbol{c}_g\) where \(\boldsymbol{\mu}_g \in \mathbb{R}^3\) denotes the 3D position, \(\boldsymbol{\alpha}_g \in \mathbb{R}\) represents opacity, \(\boldsymbol{q}_g \in \mathbb{R}^4\) encodes rotation as a quaternion, and \(\boldsymbol{s}_g \in \mathbb{R}^3\) defines anisotropic scaling. The appearance \(\boldsymbol{c}_g \in \mathbb{R}^3\) is modeled using view-dependent color, typically parameterized with spherical harmonics.

3.3 3D Scene Reconstruction↩︎

Scene Gaussian Decoder. We represent the static scene by progressively aggregating per-frame Gaussians \(\boldsymbol{g}^s_{t,i}\), each corresponding to pixel \(i\) in frame \(t\), into a unified static Gaussian set \(\boldsymbol{G}_t^s\). Given the reconstructed scene point maps \(\boldsymbol{X}_t=\{\boldsymbol{x}_{t,i}\}\) obtained from the foundation model, we initialize each Gaussian center \(\boldsymbol{\mu}_{t,i}\) directly with the corresponding 3D point \(\boldsymbol{x}_{t,i}\), where \(\boldsymbol{\mu}_{t, i}\) denotes the center of the Gaussian corresponding to pixel \(i\) at time \(t\).

Next, we decode the Human3R image tokens \(\boldsymbol{F}_t'\) into per-pixel features \(\boldsymbol{C}^D_{t, i}\) using a DPT-style decoder [63]. To incorporate direct appearance cues from the input image, we also extract CNN image features \(\boldsymbol{C}^I_t\) from \(\boldsymbol{I}_t\) using a CNN-based image encoder. We fuse the two feature maps and predict the Gaussian parameters using an MLP \(F_G\): \[\boldsymbol{\alpha}_{t, i}, \boldsymbol{q}_{t, i}, \boldsymbol{s}_{t, i}, \boldsymbol{c}_{t, i} = F_G(\boldsymbol{C}^D_{t, i}, \boldsymbol{C}^I_{t, i}).\] These parameters define a Gaussian primitive: \[\boldsymbol{g}^s_{t,i} = (\boldsymbol{\mu}_{t,i}, \boldsymbol{q}_{t,i}, \boldsymbol{s}_{t,i}, \boldsymbol{\alpha}_{t,i}, \boldsymbol{c}_{t,i}).\]

Filtering and Voxelization. At each timestep \(t\), newly predicted Gaussians \(\hat{\boldsymbol{G}}_t^s= \{\boldsymbol{g}^s_{t,i}\}_{i}\) are filtered using confidence scores [19] and human detection scores [59] to suppress contributions from dynamic human regions. In addition, to handle long video sequences while maintaining memory efficiency, we introduce a voxelization scheme for the accumulated scene Gaussians inspired by AnySplat [1]. Since our model operates in a metric space, we define a fixed voxel size in real-world scale and discretize the 3D space accordingly. This allows consistent aggregation of Gaussians across frames without scale ambiguity and for unseen scenes. At each timestep, newly predicted Gaussians are merged with the existing set through voxelization: \[\boldsymbol{G}_t^s \leftarrow \mathrm{Voxelize}(\boldsymbol{G}_{t-1}^s, \tilde{\boldsymbol{G}}^s_t),\] where \(\boldsymbol{G}_t^s\) denotes the accumulated scene Gaussian set up to timestep \(t\), and \(\tilde{\boldsymbol{G}}_t^s\) denotes the filtered per-frame Gaussians newly predicted at timestep \(t\). Within each voxel, we retain the Gaussian center corresponding to the highest confidence, while the remaining parameters are aggregated using confidence-weighted averaging. This filtering and voxelization strategy preserves high-quality geometry while preventing memory growth from scaling linearly with the input size.

3.4 Dynamic Human Reconstruction↩︎

Human Gaussian Decoder. For each \(k\)-th human at timestamp \(t\), we used the SMPL-X [38] mesh vertices \(\boldsymbol{V}_{t, k}\) as geometric anchors for human Gaussians. Given these vertex anchors, the Human Gaussian Decoder transfers image appearance features to the canonical body space and predicts the corresponding Gaussian attributes.

We define four types of tokens to capture different aspects of the human representation of the \(k\)-th human at timestamp \(t\):

• Human tokens \(\boldsymbol{h}_{t,k}\) — representing the person-level context from Human3R (where the human is)

• Vertex tokens \(\boldsymbol{V}_{t,k}\) — representing 3D SMPL-X vertices in the canonical A-pose with positional encoding (which body part it corresponds to)

• Image tokens \(\boldsymbol{F}_t'\) — extracted from the input image (what the human looks like)

• Memory tokens \(\boldsymbol{m}_{k}\) — storing accumulated appearance features for each person over time (what the human looked like before)

We apply a cross-attention transformer, named Human Gaussian Transformer (HGT), where human, vertex, and memory tokens serve as queries, and image tokens serve as keys and values: \[\boldsymbol{V}_{t,k}' = \mathrm{HGT}( \boldsymbol{Q} = [\boldsymbol{h}_{t,k}, \boldsymbol{V}_{t,k}, \boldsymbol{m}_k], \; \boldsymbol{K} = \boldsymbol{F}_t', \; \boldsymbol{V} = \boldsymbol{F}_t' ).\]

We denote the vertex features extracted from the transformer output as \(\boldsymbol{V}'_{t,k}\). These vertex features are fed into an MLP \(F_H\) to predict Gaussian parameters for each vertex: \[(\boldsymbol{\alpha}_{t,k,v}, \boldsymbol{q}_{t,k,v}, \boldsymbol{s}_{t,k,v}, \boldsymbol{c}_{t,k,v}) = F_H(\boldsymbol{V}'_{t,k,v}),\] where \(v\) indexes the vertices of the canonical SMPL-X mesh. The Gaussians are defined in the canonical A-pose space and are transformed to the posed space via linear blend skinning (LBS) using SMPL-X parameters.

Appearance Memory. To maintain a consistent appearance over time, we assign memory tokens \(\boldsymbol{m}_k\) to each tracked person. The memory tokens implicitly carry appearance information accumulated from previous frames and are used as an additional query token in the Human Gaussian Decoder. During inference, identities are associated across frames using matching based on SMPL-X parameters. When a person is matched to a previous track, the corresponding memory token is reused and updated, allowing the model to preserve person-specific appearance even under occlusion or viewpoint changes.

3.5 Training Details↩︎

Training Setup. We train the Scene Gaussian Decoder and the Human Gaussian Decoder separately while keeping the foundation model, Human3R [37] frozen. This allows each decoder to specialize in its own representation without disrupting the shared geometric prior. For both training stages, we input sequential images to ensure the model understands the temporal relationships.

Scene Gaussian Decoder. We train the Scene Gaussian Decoder using the following objective: \[\mathcal{L}_{\mathrm{scene}} = \lambda_{\mathrm{mse}}\mathcal{L}_{\mathrm{mse}} + \lambda_{\mathrm{lpips}}\mathcal{L}_{\mathrm{lpips}} + \lambda_{\mathrm{dep}}\mathcal{L}_{\mathrm{dep}} + \lambda_{\mathrm{reg}}\mathcal{L}_{\mathrm{reg}}.\] Here, \(\mathcal{L}_{\mathrm{mse}}\) and \(\mathcal{L}_{\mathrm{lpips}}\) [64] supervise the rendered scene against the input image, while \(\mathcal{L}_{\mathrm{dep}}\) supervises the rendered depth against the ground truth (GT) depth to enforce the geometric consistency. For all these loss functions, we use GT masks to supervise only background regions. \(\mathcal{L}_{\mathrm{reg}}\) regularizes Gaussian scales to avoid degenerate elongated Gaussians [65]: \[\mathcal{L}_{\mathrm{reg}} = \frac{1}{N}\sum_{i} \max\!\left( \frac{\max(\boldsymbol{s}_i)}{\min(\boldsymbol{s}_i)} - \tau,\; 0 \right),\] where \(\boldsymbol{s}_i\) denotes the scale parameters of the \(i\)-th Gaussian and \(\tau\) is a threshold hyperparameter.

We train the Scene Gaussian Decoder on the BEDLAM [66] dataset following Human3R [37], which consists of monocular video sequences with diverse human motions and appearances. In addition, we use DL3DV [23], a multi-view image dataset of real-world scenes, to improve generalization to real-world settings.

Human Gaussian Decoder. We train the Human Gaussian Decoder using the following objective, which is defined for each person at each frame \[\mathcal{L}_{\mathrm{human}} = \lambda_{\mathrm{mse}}\mathcal{L}_{\mathrm{mse}} + \lambda_{\mathrm{part}}\mathcal{L}_{\mathrm{part}} + \lambda_{\mathrm{sil}}\mathcal{L}_{\mathrm{sil}} + \lambda_{\mathrm{reg}}\mathcal{L}_{\mathrm{reg}}.\] Here, \(\mathcal{L}_{\mathrm{mse}}\) and \(\mathcal{L}_{\mathrm{sil}}\) supervise the rendered human appearance and silhouette using pixel-wise MSE and binary cross entropy, respectively, while \(\mathcal{L}_{\mathrm{reg}}\) penalizes degenerate Gaussian shapes as in the Scene Gaussian Decoder. We additionally use a partial LPIPS loss to improve fine-grained human appearance: \[\mathcal{L}_{\mathrm{part}} = \mathrm{LPIPS}(\hat{\boldsymbol{I}}, \boldsymbol{I}) + \sum_{r \in \{\mathrm{upper}, \mathrm{face}\}} \mathrm{LPIPS}\big(\mathrm{crop}_r(\hat{\boldsymbol{I}}), \mathrm{crop}_r(\boldsymbol{I})\big),\] where \(\hat{\boldsymbol{I}}\) and \(\boldsymbol{I}\) denote the rendered and input images, and \(\mathrm{crop}_r(\cdot)\) extracts the upper-body or face region.

We train the Human Gaussian Decoder using BEDLAM [66] following Human3R [37], which provides diverse human motions and SMPL-X supervision. To improve generalization to real-world settings, we additionally use Motion-X++ [67] with high-quality human motions and various appearances.

4 Experiments↩︎

4.1 Experimental Setups↩︎

We evaluate our method on dynamic human-scene reconstruction, focusing on the photorealistic rendering quality for both dynamic humans and static scenes. Since no existing feed-forward method directly addresses our setting, we compare against both an optimization-based human-scene method and decomposition-based feed-forward baselines. For the optimization-based baseline, we use HSR [11], which provides publicly available code. For the decomposition-based baselines, we reconstruct the static background with AnySplat [1], reconstruct humans with LHM [68], and compose them in a common coordinate frame. We report three variants: AnySplat, AnySplat+LHM+Human3R, and AnySplat+LHM+GT. AnySplat does not explicitly model humans, while the latter two use SMPL-X poses from Human3R and ground truth, respectively. Further details of the baselines and training are provided in Appendix 7.2 and Appendix 7.1. We first evaluate novel view synthesis on single-human scenes in Sec. 4.2, followed by multi-human scenes in Sec. 4.3 and ablation studies in Sec. 4.4.

4.2 Single-Human Scene Reconstruction↩︎

We evaluate the quality of novel view synthesis on NeuMan [69] and EMDB [70], both of which contain monocular videos of dynamic single-human scenes.

As shown in Fig. 3, decomposition-based baselines often leave visible artifacts around humans and scene boundaries, reflecting the difficulty of aligning separately reconstructed humans and scenes. In contrast, our method produces coherent human-scene renderings while maintaining comparable visual quality for both the human body and the surrounding scene.

We then quantitatively evaluate novel view synthesis on both NeuMan and EMDB. For each target frame, we render the target view by applying the camera parameters and SMPL-X [38] parameters at the target frame to Gaussians built from the input frames. We use peak signal-to-noise ratio (PSNR), structural similarity index measure (SSIM) [71], learned perceptual image patch similarity (LPIPS) [64], and frames per second (FPS) as evaluation metrics.

3pt

As shown in Table 2, HSR achieves higher PSNR and SSIM due to per-scene optimization, whereas our method obtains better LPIPS while being orders of magnitude faster. Compared with the decomposition-based baselines, our method improves both rendering quality and FPS, suggesting that joint human-scene reconstruction is more effective than post-hoc composition.

4.3 Multi-Human Scene Reconstruction↩︎

We evaluate our method on multi-human scenes, which require reconstructing multiple dynamic humans and their interactions with the surrounding scene. We begin with qualitative comparisons of novel view synthesis on the BEDLAM [66] test split against the decomposition-based baselines. As shown in Fig. 4, the decomposition-based baselines can reconstruct the background to some extent and humans in good quality, but often suffer from inaccurate human-scene alignment and visible composition artifacts. In contrast, our method reconstructs the static environment and multiple dynamic humans within a unified 3D Gaussian representation.

We then quantify these observations about novel view synthesis on the BEDLAM test split.

As shown in Table 3, our method achieves the best performance on the full human-scene evaluation, demonstrating the benefit of avoiding post-hoc composition. On human regions, our method remains competitive with the human-specific LHM baseline, despite not using ground-truth SMPL-X poses at test time. Most importantly, our method is substantially faster than the decomposition-based baselines, since it works in a feed-forward manner by leveraging shared geometric priors.

Finally, Fig. 5 shows additional qualitative results beyond the BEDLAM test split.

The examples cover diverse scenes [69], [70], [72] with varying numbers of people, poses, camera motions, and layouts, demonstrating generalization to different multi-human configurations.

4.4 Ablation Study↩︎

We conduct ablation studies to analyze the contributions of key components in our model. We separately evaluate the Scene Gaussian Decoder and the Human Gaussian Decoder under the various input settings on the NeuMan [69] dataset shown in Table 4.

Scene Gaussian Decoder. GT depth supervision consistently improves PSNR, SSIM, and LPIPS, indicating that explicit geometric guidance helps regularize Gaussian placement and improves scene reconstruction. In contrast, adding DL3DV [23] provides only marginal gains, suggesting that stable geometry is more critical than additional appearance diversity in this setting.

Human Gaussian Decoder. Removing cross-attention leads to the largest degradation, highlighting its importance for transferring image appearance to canonical human Gaussians. Motion-X++ [67] and memory tokens further improve perceptual quality and temporal consistency, respectively. Partial LPIPS has limited impact on PSNR/SSIM, as these metrics underrepresent perceptual improvements.

5 Conclusion↩︎

In this paper, we presented a novel feed-forward framework for dynamic human-scene reconstruction from a monocular video. Our method bridges the gap between geometric reconstruction and photorealistic rendering by extending the geometric and human priors of a pretrained human-scene foundation model. We represent the human-scene environment using a unified 3DGS formulation, with a Scene Gaussian Decoder for the static scene and a Human Gaussian Decoder for dynamic humans. Through experiments, we demonstrated competitive novel view synthesis performance on dynamic human scenes against optimization-based and decomposition-based baselines, while significantly improving inference efficiency. Overall, our results suggest that combining foundation models with structured representations such as 3DGS is a promising direction for scalable and photorealistic dynamic human-scene reconstruction.

6 Acknowledgements↩︎

This work was partially supported by JST ASPIRE Program, Japan, Grant Number JPMJAP2303; JST ACT-X (JPMJAX25C5); JST SPRING, Grant Number JPMJSP2108; and JSPS KAKENHI Grant Number 26K21245.

7 Implementation Details↩︎

7.1 Training details↩︎

We freeze the pretrained Human3R [37] backbone and train only the Gaussian prediction modules. For the Scene Gaussian Decoder, we train the DPT-style feature decoder, CNN feature decoder and Gaussian MLP head. For the Human Gaussian Decoder, we train the cross-attention transformer and Gaussian MLP head. All input images are resized such that the longer side is 512 pixels while preserving the aspect ratio. We train both decoders using AdamW [73] with a learning rate of \(1\times10^{-4}\), weight decay of \(1\times10^{-4}\), and a batch size of 2 for Scene Gaussian Decoder and 1 for Human Gaussian Decoder. The Scene Gaussian Decoder is trained for 100k iterations on one NVIDIA A100 80GB GPU, taking approximately 1 day, while the Human Gaussian Decoder is trained for 150k iterations on one NVIDIA A100 40GB GPU, taking approximately two days. As for the loss function, we set \(\lambda_{\mathrm{mse}} = 1.0\), \(\lambda_{\mathrm{lpips}} = 0.2\), \(\lambda_{\mathrm{reg}} = 0.05\), \(\lambda_{\mathrm{dep}} = 0.1\) for \(\mathcal{L}_{\mathrm{scene}}\), and \(\lambda_{\mathrm{mse}} = 1.0\), \(\lambda_{\mathrm{part}} = 0.5\), \(\lambda_{\mathrm{reg}} = 100.0\), \(\lambda_{\mathrm{sil}} = 1.0\) for \(\mathcal{L}_{\mathrm{human}}\).

7.2 Decomposition-based baselines↩︎

Since no existing feed-forward method reconstructs photorealistic, renderable 3D Gaussian human-scene representations with multiple dynamic humans, we construct decomposition-based baselines by combining existing feed-forward reconstruction methods. These baselines separately reconstruct the static scenes and each dynamic human, and then compose them in a common coordinate frame using Umeyama’s alignment [74].

AnySplat. As a scene-only baseline, we directly apply AnySplat [1] to the input frames. This baseline reconstructs the scene as static 3D Gaussians and does not explicitly model dynamic humans.

AnySplat+LHM+Human3R. We first use Human3R [37] to estimate human masks, background masks, and SMPL-X [38] parameters. The background is reconstructed by applying AnySplat only to the masked background regions. For each detected human, we crop the human region from the input image and reconstruct a canonical human representation using LHM [68]. We then animate and place the reconstructed human using the SMPL-X parameters estimated by Human3R, and compose it with the reconstructed background.

AnySplat+LHM+GT. We additionally report an oracle variant that replaces the Human3R-estimated masks and SMPL-X parameters with ground-truth annotations. This baseline evaluates the upper-bound performance of the decomposition pipeline when human segmentation and pose alignment are given.

8 Analysis↩︎

8.1 Video Depth Estimation↩︎

r0.45

We evaluate geometric accuracy by comparing predicted depth with ground truth and prior methods using AbsRel and threshold accuracy (\(\delta < 1.25\)), as shown in Table 5. Following a video depth estimation setting, we estimate a single global scale per sequence and apply it to all frames, instead of performing per-frame scale alignment. This protocol better reflects dynamic scene reconstruction, as it requires temporally consistent geometry across the video. Our method achieves the best performance on both metrics, outperforming DepthAnything3 [18] and the underlying Human3R [37]. While Human3R provides strong geometric priors, its point-based depth can be sparse and less regularized. DepthAnything3 relies on monocular depth estimation and therefore lacks explicit cross-frame consistency. In contrast, our unified Gaussian representation produces more coherent and temporally stable geometry.

8.2 Static scene reconstruction↩︎

3pt

We additionally evaluate novel view synthesis on static background regions to isolate the scene reconstruction capability of each method. As shown in Table 6, HSR [11] achieves higher PSNR and SSIM in several settings, benefiting from per-scene optimization over the target sequence. However, our method obtains better LPIPS, suggesting that it preserves perceptual scene quality despite operating in a feed-forward manner. Compared with feed-forward scene reconstruction methods such as AnySplat [1] and DepthAnything3 [18], our method is less favorable on background-only metrics. This is partly because these methods process all input frames in a batch, making it easier to enforce multi-view consistency for static scene reconstruction, whereas our method performs streaming reconstruction while jointly modeling dynamic humans. These results highlight a batch–streaming trade-off: batch-based feed-forward methods better exploit multi-view consistency for static background reconstruction, whereas our streaming method reconstructs dynamic human-scene representations frame by frame.

9 Limitations↩︎

Although our method enables feed-forward photorealistic reconstruction of dynamic human-scene representations, several limitations remain. First, it relies on geometric and human priors from the underlying foundation model [37], so errors in camera estimation, scene point maps, human detection, or SMPL-X fitting can propagate to the final reconstruction. Second, severe human-human occlusions and complex interactions remain challenging, as monocular videos provide limited cues for reliable identity association and complete human geometry. Finally, fine-scale appearance details, such as faces, hands, and clothing textures, may remain imperfect under motion blur, large pose changes, or limited observations.

References↩︎

---

1. Project lead.↩︎