DeSeG: Decoupling Semantic Intent and Geometric Constraints for Physically Plausible Human-Scene Interaction
July 07, 2026
Abstract
Synthesizing physically plausible human–scene interactions (HSI) remains a critical challenge in computer vision and the development of human avatars. Although recent generative models enable diverse motion synthesis, they suffer from an inductive bias referred to as semantic-geometric entanglement. Because spatial constraints often strongly correlate with specific actions in training data, monolithic models will learn the shortcut bias, aggressively overriding the semantic intent when faced with strict geometric cues. Furthermore, this entanglement exacerbates physical hallucinations, such as body-scene penetrations. To address these limitations, we propose DeSeG, a hierarchical framework that explicitly decouples semantic intent from geometric constraints. First, we introduce a Residual Semantic Planner that encodes textual instructions and canonicalized goal voxels into a compact latent space, enabling fine-grained semantic control independent of spatial trajectories. Second, we propose a physics regularized diffusion executor that incorporates differentiable repulsive potential fields directly into the diffusion objective, enforcing collision-aware motion generation. Extensive experiments on the Lingo dataset demonstrate that DeSeG achieves state-of-the-art performance, reducing mean scene penetration by 47% and improving semantic alignment by 29% over the sota baselines.
1 Introduction↩︎
Ensuring digital humans move in a physically consistent way within 3D scenes is crucial for immersive experiences, virtual reality, and embodied intelligence [1], [2]. Recent years have seen significant progress in conditional human-scene interaction (HSI) modeling that matches high-level intent with the physical environment [3]–[6]. In particular, powerful generative models such as variational autoencoders and diffusion models synthesize long-term, realistic actions from language prompts and scene context [5], [7], marking an important step toward flexible action synthesis for human avatars in complex scenes.
Despite this progress, current methods still exhibit two critical limitations that impede robust deployment in complex and unstructured scenes:
Semantic-Geometric Entanglement. Existing frameworks typically conflate high-level interaction intent with low-level spatial trajectories. In diffusion-based models such as Lingo [8], the text-conditioned motion generator implicitly concatenates semantic intent to geometric targets provided as goal locations. For example, a goal on the bed often results in a lying motion. We argue that this is not merely a rare artifact, but a pervasive instance of shortcut learning in multimodal generative models. This entanglement arises because there is no explicit mechanism to decouple ‘how to perform’ from ‘where to go’, leading to incorrect behaviors when geometric cues conflict with the underlying textual instruction.
Physical Hallucinations. Generative motion models fundamentally learn data distributions without explicit physical laws, which can result in unrealistic outcomes, such as body-scene penetrations or implausible contacts. While some methods attempt to mitigate such violations using post-processing or heuristic guidance, these approaches are computationally expensive and often produce unnatural motion artifacts. Moreover, they do not guarantee consistent avoidance of collisions throughout the generation process.
Based on the above insights, we introduce DeSeG, a hierarchical framework that explicitly decouples semantic intent from geometric feasibility for HSI synthesis. The proposed framework comprises two novel components. First, a conditional latent module encodes textual instructions and canonicalized goal voxels into a compact intent code, so the latent space captures pure semantic affordance and conditions the subsequent motion generator. Second, an autoregressive diffusion module embeds differentiable repulsive potential fields, derived from scene geometry, directly into the denoising objective. Penalizing predicted poses against penetration during training teaches the model to perceive and avoid obstacles intrinsically, yielding physically constrained motions without test-time optimization.
By integrating decoupling mechanisms, our framework maintains the character’s motion within realistic, physically plausible long-term sequences. Experiments on the Lingo and TRUMANS datasets demonstrate significant improvements in both physical plausibility and semantic consistency compared to current baselines. Our contributions are summarized as follows:
We introduce a novel latent learning mechanism encoding textual instructions and local goal voxels into a latent space. This enables precise semantic control and robust generalization, even under conflicting geometric cues.
We formulate a physics-regularized objective by integrating differentiable repulsive potential fields directly into the denoising loss. This endows the executor with a collision-avoidance reflex, yielding physically plausible motions without the computational overhead of test-time optimization.
We propose DeSeG, a hierarchical framework that explicitly decouples high-level semantic planning from low-level geometric execution. This resolves the conflict between semantic and geometric controllability in HSI synthesis.
2 Related Work↩︎
2.1 Human-Scene Interaction Synthesis↩︎
Human motion synthesis has evolved from isolated character animation to modeling complex interactions within 3D environments. Early works focused on single-person motion generation conditioned on text or action labels [7], [9]–[13], using VAEs or diffusion models to produce high-fidelity motions but largely neglecting environmental context. To address object manipulation, research expanded to Human-Object Interaction (HOI), synthesizing motions conditioned on specific objects [14], [15]. However, these approaches are typically limited to short, atomic interactions (e.g., grasping, sitting) and lack the capability to navigate complex layouts.
Recently, Human-Scene Interaction (HSI) has emerged to tackle long-term, multi-stage behaviors in large-scale scenes. HUMANISE [16] establishes the foundational paradigm of language-conditioned HSI by aligning textual commands with captured motions in 3D scans, and Move-as-You-Say [17] further grounds language-guided motion in scene affordances. PROX [18] enforces scene-aware static pose optimization, while SceneDiffuser [19] and Lingo [8] formulate interaction synthesis as scene-conditioned diffusion, enabling multi-stage tasks in cluttered environments. More recent efforts pursue hierarchical or geometry-grounded decompositions: SceMoS [20] disentangles a text-conditioned global planner from local execution using lightweight 2D scene cues (bird’s-eye-view maps and local heightmaps), reporting state-of-the-art realism on TRUMANS, while HOSIG [21] couples a scene-aware grasp generator, a collision-aware navigation planner, and a trajectory-controlled synthesizer to compose long-horizon full-body human–object–scene interactions. Despite this progress, current methods rely on goal locations as strong conditioning, and even multi-stage pipelines such as Lingo couple textual intent and geometric targets in one backbone; when goal and instruction conflict, the geometric prior dominates. We instead introduce a hierarchical decomposition of high-level semantic planning and low-level geometric execution, retaining the goal but structurally separating its influence.
2.2 Latent Representations for Human Motion↩︎
Latent representations serve as the bridge between high-level semantics and low-level kinematics. Previous approaches leverage VAE, VQ-VAE, and transformer-based architectures [22]–[24] to compress motion into compact latent codes. To achieve semantic control, MotionCLIP [25] aligns the motion latent space with the CLIP text embedding, enabling zero-shot generation. Most latent spaces target free-space motion and ignore spatial constraints; scene-conditioned models inject goal voxels directly into the diffusion model, entangling instruction and geometry in one backbone. We instead introduce a Residual Semantic Planner that produces a compact latent code before geometric diffusion, preventing spatial signals from overwhelming semantic intent.
The most closely related effort is SemGeoMo [26], which likewise pursues a two-stage decoupling for HOI. The key distinction lies in how the two factors are separated. SemGeoMo predicts an explicit intermediate contact geometry that guides the subsequent motion, which is informative but tightly tied to a predefined contact parameterization and offers no built-in penetration guarantee. In contrast, DeSeG learns an implicit disentangled affordance latent through a residual CVAE and, crucially, internalizes physical plausibility directly into the diffusion training objective via a differentiable repulsive potential field. This removes the need for explicit contact annotation and for test-time optimization, and, unlike prior decoupling work, we explicitly stress-test the separation under conflicting semantic–geometric cues through our NC-Bench.
2.3 Physics-Aware Human Motion Synthesis↩︎
For physical plausibility, one paradigm uses physics simulators [27] or reinforcement learning [4], [28], but simulators are often non-differentiable and hard to integrate into end-to-end generative pipelines. Another prevailing approach is Test-Time Guidance [3], [5], [29], where SDF-based collision losses or contact optimizations are used to guide the sampling process during inference. While this improves quality, it significantly increases computational latency, hindering real-time applications. To achieve both physical plausibility and inference efficiency, recent works integrate geometric losses directly into training. For instance, SceneDiffuser [19] applies scene-penetration penalties during optimization. In contrast, we embed the Potential Field [30] into the diffusion training objective, so the model intrinsically generates collision-free trajectories without expensive test-time guidance, balancing physical constraints with diversity.
3 Method↩︎
In this section, we present the mathematical formulation and detailed architectural implementation of DeSeG. As illustrated in 2, Our method factorizes the complex human motion generation into two processes: a Residual Semantic Planner that infers a latent interaction affordance via conditional multimodal inputs, and a Physics-Regularized Diffusion Executor that grounds latent code into physically plausible motion sequences through a potential field diffusion training strategy.
3.1 Problem Formulation↩︎
We define the HSI synthesis task as learning a conditional generative model:
\[p(\mathbf{M} \mid \mathcal{T}, \mathcal{S}, \mathbf{g}).\]
Given a 3D scene \(\mathcal{S}\) and a target interaction position or object \(\mathbf{g} \in \mathbb{R}^3\), our goal is to generate a plausible human-scene interaction, where the motion style can be controlled by a user-specified natural language instruction \(\mathcal{T} = \{w_1, \dots, w_L\}\). Instructions are provided at the action-segment level, a single command describes one interaction segment, and a long-horizon sequence is composed of consecutive segments. The output is a motion sequence \(\mathbf{M} \in \mathbb{R}^{T \times J \times D}\) over \(T\) timesteps, where the model operates directly in 3D joint-position space with \(J{=}28\) body joints and \(D{=}3\) (in a \(y\)-up frame). The corresponding SMPL-X [31], [32] body parameters are recovered from the predicted joints through a lightweight learned joints-to-SMPL-X regressor, rather than a classical inverse-kinematics optimization.
Previous diffusion-based approaches model this distribution with a monolithic backbone \(\epsilon_\varphi(\mathbf{M}_t, t, \mathcal{C})\), where \(\mathcal{C}\) concatenates conditioning signals such as text and geometry. This architecture suffers from a severe inductive bias: in typical HSI datasets the geometric goal \(\mathbf{g}\) is a dense, high-frequency signal highly predictive of the action category, so optimization latches onto \(\mathbf{g}\) while ignoring the subtle linguistic signal \(\mathcal{T}\), causing failures on negative constraints.
This is an architectural pathology that loss-level regularization cannot remove: sharing one conditioning pathway lets the gradient of \(\mathbf{g}\) structurally dominate that of \(\mathcal{T}\). Our decomposition instead imposes an information bottleneck—routing semantics through a compact latent \(\mathbf{z}\) computed from a canonicalized, trajectory-free view of the geometry prevents absolute spatial configuration from leaking into the semantic factor. We test this decoupling advantage directly in our ablations (1).
We therefore propose a two-stage decoupled formulation via a latent code \(\mathbf{z}\) that encodes interaction affordance from multimodal inputs, emphasizing the semantic “what” and “how”. We decompose the joint distribution as:
\[{!}{\displaystyle p(\mathbf{M} \mid \mathcal{T}, \mathcal{S}, \mathbf{g}) = \int p_\varphi(\mathbf{M} \mid \mathbf{z}, \mathcal{S}_{local}, \mathbf{g}) \cdot p_\phi(\mathbf{z} \mid \mathcal{T}, \Psi(\mathcal{S}, \mathbf{g})) \, d\mathbf{z}. }\]
Here \(\Psi(\cdot)\) is the Canonicalization Operator, and \(p_\phi\), \(p_\varphi\) are our Planner and Executor. This structure forces \(\mathbf{z}\) to capture the interaction affordance.
3.2 Residual Semantic Planner↩︎
Inspired by hierarchical vision-language-action methods [33], the Semantic Planner is the high-level reasoning module. It maps the linguistic instruction and scene geometry into a compact latent space using a residual Conditional Variational Autoencoder (CVAE) [34], [35], focusing on semantic intent while offloading trajectory-specific details to the motion encoder. The sampled latent code conditions a decoder that predicts future poses \(\hat{S}_{t+1}, \dots, \hat{S}_{t+M}\) from the current state \(S_t\).
3.2.0.1 Kinematics Encoder.
Since the Semantic Planner only observes the current conditional inputs during inference, we introduce a kinematics encoder during training to capture additional temporal information from future states. The encoder is a Multi-Layer Perceptron (MLP) that takes the flattened ground-truth future states \(\mathbf{S}_{t+1:t+M}\) as input and predicts the residual mean \(\Delta\mu\) and the posterior standard deviation \(\sigma_{post}\) of the latent distribution.
3.2.0.2 Canonicalized Geometric Perception.
We represent the scene as a full 3D occupancy grid at \(2\,\mathrm{cm}\) voxel resolution for both training and inference. To prevent absolute-orientation leakage into the latent space, we apply the canonicalization operator \(\Psi\): we crop a local voxel grid centered at the goal \(\mathbf{g}\) and express it in an egocentric canonical frame via the rigid transform \(R^{-1}\) that aligns the agent’s heading to a fixed axis—the same frame in which motion is generated—so the goal voxel no longer depends on absolute scene orientation. The resulting \(V_{can} = \Psi(\mathcal{S}, \mathbf{g})\) exposes the intrinsic shape of the interaction region (seat height, table width) in an orientation-invariant manner.
We then align language with concrete geometry via cross-attention. The instruction \(\mathcal{T}\) is encoded by a frozen CLIP model [36] into text tokens and \(V_{can}\) by a 3D-CNN [37] into geometric tokens; the text embeddings act as queries that selectively attend to the voxel features, grounding prompts to task-relevant geometric affordances. The fused features are pooled and projected through MLPs to parameterize the prior mean \(\mu_\phi\) and standard deviation \(\sigma_\phi\).
3.2.0.3 Residual Latent Space.
To effectively bridge high-level semantic planning and low-level kinematic execution, we formulate the latent space using a residual connection. Specifically, we define the generative prior distribution as
\[p_\phi(z_t \mid \mathcal{T}, V_{can}) = \mathcal{N}(\mu_\phi, \sigma_\phi^2).\]
During training, the posterior \(q_\theta(z_t)\) combines the kinematics encoder and the prior network. Rather than predicting the absolute mean, the kinematics encoder predicts a residual shift \(\Delta \mu_\theta\), so the posterior mean is a residual addition while the variance is taken from the encoder (\(\sigma = \sigma_\theta\)).
This residual design explicitly forces the kinematics encoder to capture only the fine-grained residual details that are not already inferable from the scene-language inputs, thereby allowing the semantic planner to focus entirely on the abstract intent. The network is optimized using the Evidence Lower Bound (ELBO), where a KL divergence loss \(D_{KL}(q_\theta \parallel p_\phi)\) is applied to regularize the posterior towards the prior.
3.3 Physics-Regularized Diffusion Executor↩︎
The Executor translates the abstract semantic latent \(\mathbf{z}\) into a dense, physically valid motion sequence \(\mathbf{M}\) conditioned on the local scene geometry. We employ a Transformer-based Diffusion Model (DiT) [38], [39] and introduce a training-time physics-aware strategy to address the physical hallucination problem common in probabilistic generative models.
3.3.0.1 Conditional Diffusion Backbone.
We model the motion generation as a reverse diffusion process \(p_\varphi(\mathbf{M}_{t-1} \mid \mathbf{M}_t, C)\). To bridge the gap between high-level intent and low-level execution, the core conditioning context \(C\) comprises two primary components: (1) The semantic affordance latent \(\mathbf{z}\) sampled from the Planner; (2) The local scene voxel \(V_{local}\) , providing regional obstacle awareness.
To re-introduce the spatial guidance abstracted away by the Planner, we run A* [40] on the occupancy map to produce collision-free waypoints toward the goal; this path acts as a macro-level directional prompt that guides the autoregressive diffusion toward the target.
3.3.0.2 Differentiable Repulsive Potential Field.
To enforce physical plausibility, we internalize a collision-avoidance mechanism inspired by the Humanoid Potential Field [30] directly into the diffusion training objective. Since the network predicts noise \(\epsilon_\varphi\), we employ Tweedie’s formula [41], [42] to analytically estimate the clean motion \(\hat{\mathbf{M}}_0\) at any timestep \(t\): \[\hat{\mathbf{M}}_0 = \frac{\mathbf{M}_t - \sqrt{1-\bar{\alpha}_t} \cdot \epsilon_\varphi(\mathbf{M}_t, t, C)}{\sqrt{\bar{\alpha}_t}}.\]
We extract the absolute world coordinates \(o_{j,k} \in \mathbb{R}^3\) for each joint from \(\hat{\mathbf{M}}_0\). To efficiently obtain the shortest Euclidean distance to environmental obstacles, we query a pre-computed high-resolution Signed Distance Field (SDF) [43], denoted as \(\mathcal{D}(o)\), and explicitly define a pure repulsive potential energy function \(U_{rep}\) for any spatial point \(o\): \[U_{rep}(o) = \frac{1}{2} \eta \cdot \text{ReLU}\big(d_{safe} - \mathcal{D}(o)\big)^2,\] where \(d_{safe}\) is a pre-defined safe margin, and \(\eta\) is a scaling factor controlling the stiffness of the potential field. The ReLU activation strictly ensures that the penalty is only triggered when a joint breaches the safety threshold.
Building upon this continuous potential function, we formulate the overall physical regularization loss across all frames and skeletal joints: \[\mathcal{L}_{phy}(\hat{\mathbf{M}}_0) = \sum_{k=1}^T \sum_{j=1}^J w_j \cdot U_{rep}(o_{j,k}).\]
Treating all joints uniformly can trap humanoids in local minima, as symmetric penalties on opposing limbs cancel. We therefore use joint-specific weights \(w_j\): higher penalties on the torso and head prevent catastrophic penetrations, while contact-prone extremities get softer margins. Since \(\hat{\mathbf{M}}_0\) is differentiable in the weights \(\varphi\), the field’s spatial derivative acts as a gradient-based repulsive signal; backpropagating it teaches the network to predict noise that steers kinematics away from body-scene penetrations.
3.3.0.3 Optimization Strategy.
We adopt a decoupled two-stage training paradigm to preserve semantic disentanglement. First, the Semantic Planner is trained to convergence using the ELBO objective to establish a stable latent affordance space. Second, the Planner’s weights are frozen, and the Diffusion Executor is trained conditioned on the sampled \(z \sim p_\phi(\cdot)\).
To simultaneously capture the data distribution and adhere to physical constraints, the Executor is optimized using a joint training objective: \[\mathcal{L}_{overall} = \mathcal{L}_{ddpm} + \lambda \cdot \mathcal{L}_{phy}(\hat{\mathbf{M}}_0),\] where \(\lambda\) balances the two terms. To keep the physics penalty from destabilizing training or collapsing diversity, we (i) warm up \(\lambda\) from 0 to \(\lambda_{\max}\) over the first 20 epochs; (ii) apply the penalty only at small timesteps \(t \le 5\), where the Tweedie estimate \(\hat{\mathbf{M}}_0\) is close to the clean motion and yields reliable gradients, whereas at larger \(t\) it is dominated by noise and would inject instability; and (iii) use the margin \(d_{\text{safe}}\) and per-joint weights \(w_j\) to avoid over-penalizing fingertips. The Executor thus satisfies both constraints: it executes the behavior commanded by \(\mathbf{z}\) while carving out collision-free, physically plausible trajectories.
4 Experiments↩︎
To rigorously evaluate the efficacy of DeSeG, we conduct extensive experiments to assess whether the proposed hierarchical framework improves semantic grounding and whether physics regularization eliminates physical inconsistency. All analyses are specifically designed to validate the structural disentanglement assumptions introduced in 3.
| Method | FID\(\downarrow\) | Diversity\(\rightarrow\) | M-modality\(\rightarrow\) | R-Precision\(\uparrow\) | Pene\(_{\text{mean}}\)\(\downarrow\) | Pene\(_{\text{max}}\)\(\downarrow\) |
|---|---|---|---|---|---|---|
| Comparison with baselines | ||||||
| TRUMANS [44] | 3.286 | 4.984 | 3.036 | 0.413 | 0.404 | 1.825 |
| TeSMo [45] | 3.558 | 5.328 | 2.956 | 0.392 | 0.278 | 0.915 |
| Lingo [8] | 3.093 | 5.832 | 2.239 | 0.437 | 0.392 | 1.229 |
| Ablation of DeSeG components | ||||||
| w/o Hierarchical Latent | 3.279 | 5.461 | 2.439 | 0.430 | 0.241 | 0.938 |
| w/o Cross-Attention | 3.113 | 5.636 | 2.721 | 0.457 | 0.239 | 0.946 |
| w/o Physical Regularization | 2.983 | 5.822 | 2.524 | 0.514 | 0.367 | 1.393 |
| DeSeG (full) | 2.882 | 5.865 | 2.541 | 0.562 | 0.207 | 0.921 |
| Comparison on TRUMANS | FID\(\downarrow\) | Diversity\(\rightarrow\) | PFC\(\downarrow\) | Cont\(\uparrow\) | Pene\(_{\text{mean}}\)\(\downarrow\) | Pene\(_{\text{max}}\)\(\downarrow\) |
| Lingo [8] | 0.493 | 2.932 | 0.657 | 0.941 | 2.279 | 13.376 |
| DeSeG | 0.338 | 2.853 | 0.621 | 0.986 | 1.183 | 9.187 |
![Figure 3: Qualitative comparison of 3D human–scene interaction in indoor scenes. Rows (top to bottom): “walk around the table”, “stand up and creep”, “pick up the book on cabinet”. Columns: DeSeG, Lingo [8], TRUMANS [44]. Red boxes mark scene penetration; red curves show temporal trajectories. DeSeG gives the most reasonable result in all three cases.](/article/2607.05787/imgs/exp.png "fig:")
4.1 Experimental Setup↩︎
4.1.0.1 Dataset and Preprocessing.
We train and evaluate on the large-scale Lingo dataset, which provides 16 hours of high-fidelity, text-annotated HSI motion capture across 120 indoor scenes. Following [8], we use a 4:1 scene-based train/eval split, ensuring evaluation in geometrically unseen environments. For physical regularization, we precompute high-resolution SDFs [46] at \(2\,\mathrm{cm}\) voxel resolution for all training scenes; SDF values are normalized per scene and joint positions are trilinearly interpolated from the grid to compute \(\mathcal{D}(\mathbf{x})\). Full preprocessing details are in the appendix.
4.1.0.2 Evaluation Metrics.
To provide a comprehensive assessment, we align our evaluation protocol with existing state-of-the-art HSI frameworks [9], [44]. For generation quality and semantic alignment, we report the Fréchet Inception Distance (FID), Diversity, Multi-modality (M-modality) and R-Precision [9]. These metrics quantify how accurately the generated motion distribution matches the linguistic intent and motion quality. For physical plausibility, we employ standard scene penetration metrics, including \(Pene_{mean}\), and \(Pene_{max}\).
4.1.0.3 Baseline Setup.
We compare against three HSI models: Lingo [8], TRUMANS [44], and TeSMo [45], all retrained on the Lingo dataset. All models use a 3D occupancy grid except TeSMo, which uses a 2D floor map. For fairness, every model is trained on identical data splits on a single NVIDIA RTX A6000 GPU from its official public configuration, with no extra tuning advantage given to our method; full protocols are in the appendix. All quantitative results are averaged over 3 random seeds (mean and standard deviation).
4.1.0.4 Implementation Details.
The planner pairs a frozen CLIP ViT-L/14 encoder with a 3D-CNN scene encoder (16-head cross-attention, 256-d latent); the executor is an 8-block DiT. We train the planner (50K iters), freeze it, then the executor (100K iters, batch 128, AdamW, lr \(1\!\times\!10^{-4}\)); the physics loss uses \(d_{safe}=0.05\) m, \(\eta=0.5\). Full architecture and per-joint weights are in the appendix.
4.1.0.5 Negative Constraint Benchmark.
Standard metrics often miss semantic-geometric entanglement, as test distributions mirror training biases. We therefore introduce NC-Bench, 30 scenarios in which the instruction semantically contradicts the target object’s strong geometric prior; we contextualize standalone AMASS [47] motions within unseen TRUMANS [44] scenes. Performance is Semantic-Geometric Consistency (SGC), from a perceptual user study: 10 annotators give a binary judgment of whether the character performs the specified action without being captured by the object’s canonical geometry, averaged into SGC. We use a perceptual protocol because NC-Bench targets the conflict cases that automated, distribution-matched metrics (FID) systematically mis-score, inheriting the same shortcut bias as the models they assess. Inter-annotator agreement is high (Fleiss’ \(\kappa=0.818\)); the study collects only anonymous binary judgments on synthetic clips.
| Method | FID\(\downarrow\) | R-Precision\(\uparrow\) | SGC (%)\(\uparrow\) |
|---|---|---|---|
| TRUMANS [44] | 4.102 | 0.427 | 38.6 |
| TeSMo [45] | 3.845 | 0.393 | 41.4 |
| Lingo [8] | 3.120 | 0.459 | 58.2 |
| Ours | 3.155 | 0.584 | 72.3 |
4.2 Quantitative Results↩︎
4.2.0.1 Semantic Alignment and Motion Quality.
1 compares DeSeG with three baselines. DeSeG attains the best FID (2.882), indicating that the Semantic Planner suppresses geometrically induced shortcuts, and the highest Diversity (5.865)—unlike constrained methods that sacrifice richness for compliance. Its Multi-modality (2.541) stays comparable to the baselines (Lingo at 2.239), indicating that decoupling and physical regularization preserve per-prompt diversity rather than collapsing generation onto a single dominant mode.
4.2.0.2 Physical Plausibility.
DeSeG substantially reduces scene penetration, lowering mean penetration from 0.392 (Lingo) to 0.207 and maximum penetration from 1.229 to 0.921. It also matches TeSMo, which applies SDF-based guidance during sampling, despite a fundamental difference: TeSMo enforces physics through post-hoc test-time optimization, whereas DeSeG internalizes collision awareness directly into the diffusion objective. The suppression of catastrophic penetrations indicates that the backbone has learned a robust collision-aware prior.
4.2.0.3 Inference Efficiency.
Internalizing physics during training removes any test-time SDF optimization or post-processing, yielding roughly a \(20\%\) inference speed-up over TeSMo while attaining lower penetration. The autoregressive executor also has a fixed per-segment cost, so runtime grows linearly with the number of action segments.
4.2.0.4 Cross-Dataset Generalization.
A central concern of HSI (Human-Scene Interaction) methods is their generalization capability across different training distributions. We retrain our model on TRUMANS [44] dataset, also transferring the Lingo baseline identically and following the TRUMANS protocol—reporting foot-sliding (PFC\(\downarrow\)) and contact (Cont\(\uparrow\)) in place of M-modality and R-Precision (1, bottom). Despite the domain shift, DeSeG transfers more robustly—lower FID (\(0.338\) vs.\(0.493\)) and roughly half the penetration (\(1.183\) vs.\(2.279\))—showing the decoupled latent and physics-regularized executor are effective for data with various distributions.
4.2.0.5 Results on NC-Bench.
As shown in 2, FID masks the limitations of existing methods out of distribution: Lingo keeps a competitive FID (3.120) yet its Semantic-Geometric Consistency (SGC) drops sharply. This confirms that previous methods suffer from severe entanglement, triggering canonical interactions dictated by the target position while overriding the conflicting instruction.
In contrast, by decoupling semantic intent from geometric priors, DeSeG reaches an SGC of 72.3%, exceeding the strongest baseline by over 14% absolute without sacrificing motion quality (comparable FID and R-Precision)—validating that it generates semantically faithful, physically plausible interactions even when instructions diverge from default affordances.
4.3 Ablation Studies↩︎
To understand the contribution of each component of DeSeG, we perform an ablation study by removing key modules and observing the impact on performance in 1. Different variants were trained on the same dataset using a single NVIDIA A6000 GPU.
4.3.0.1 Ablating the Semantic Planner.
w/o Hierarchical Latent: We remove the CVAE Semantic Planner and inject the raw CLIP text embeddings and global scene voxels directly into the Diffusion Executor. This reduces our framework to a strong monolithic architecture that keeps the same backbone, conditioning, and physics regularization—only the decoupling is removed—the controlled monolithic baseline called for in 3. R-Precision drops to 0.430 and FID to 3.279: without \(\mathbf{z}\), dense spatial tokens overwhelm the sparse textual signals and the network regresses to shortcut behavior, confirming that loss-level conditioning over a shared pathway does not overcome the entanglement. w/o Cross-Attention: Replacing geometry-aware cross-attention with simple feature concatenation drops R-Precision to 0.457 and raises FID to 3.113, showing that explicit attention routing is necessary to ground language into spatial affordances.
4.3.0.2 Ablating the Physics-Regularized Executor.
w/o Physical Regularization: Removing \(\mathcal{L}_{phy}\) from the training objective abolishes physical awareness: \(Pene_{mean}\) rises to 0.367 and \(Pene_{max}\) to 1.393. The unregularized model generates semantically correct yet physically implausible motions.
![Figure 4: Visualization in a TRUMANS [44] scenario for the instruction sit down on sofa and then sleep: Lingo freezes in a sitting posture, whereas DeSeG transitions into a lying pose.](/article/2607.05787/imgs/truman.png "fig:")
4.4 Qualitative Analysis and Limitations↩︎
4.4.0.1 Visualization.
3 compares DeSeG with baselines on three multi-action instructions across Lingo and TRUMANS scenes: our avatars navigate around obstacles and rest hands on armrests without mesh penetration (red cues), showing that the executor internalizes the humanoid potential field. 4 further shows a sofa\(\rightarrow\)sleep case in which Lingo freezes in a sitting posture—the sofa’s geometric prior overwhelms the later textual condition—whereas DeSeG transitions smoothly into a lying pose, confirming successful decoupling.
4.4.0.2 Limitations.
Failure cases occur in cluttered environments or with geometrically abstract objects lacking clear affordances, exposing the limits of voxel-based geometry encoding; finer-grained representations are a natural next step. A second limitation is the decoupling trade-off: for affordance-critical tasks demanding tight semantic–geometric coupling (facing the screen when “watching television”), routing intent through a compact, trajectory-free latent may discard fine-grained alignment cues—modulating the decoupling strength per interaction type is a promising direction.
5 Conclusion↩︎
We presented DeSeG, a hierarchical framework that resolves semantic-geometric entanglement and physical hallucination in HSI. By structurally decoupling generation, the Residual Semantic Planner extracts interaction affordances via latent codes, while the Physics-Regularized Diffusion Executor gains an intrinsic collision-avoidance reflex without test-time optimization. Experiments show state-of-the-art quality and physical plausibility, especially in negative-constraint scenarios where monolithic baselines fail.
6 Additional Details↩︎
This appendix provides details omitted from the main text due to space constraints. Section 7 gives implementation details (data preprocessing, network architectures, optimization). Section 8 reports an A* path-guidance ablation and statistical-significance details, and Section 9 lists the Negative Constraint Benchmark scenarios.
7 Implementation Details↩︎
7.1 Data Preprocessing and SDF Generation↩︎
We precompute a 3D Signed Distance Field (SDF) for each Lingo scene from a 3D occupancy grid at \(2\,\mathrm{cm}\) resolution. Voxel grids from Blender meshes often have hollow interiors; rather than closing them with a
2.5D elevation filling—which marks free space under furniture ( beneath a table) as solid, inconsistent with under-object interactions such as NC-Bench cases 14–17—we resolve the interiors directly, so only genuine surfaces and solids are occupied
while traversable space beneath objects is preserved. The signed grid is obtained via distance_transform_edt() (CuPy) on the occupied and free regions; the repulsive field then reads SDF values at each joint position.
7.2 Network Architecture Details↩︎
Residual Semantic Planner: A frozen CLIP ViT-L/14 text encoder yields an \(L_2\)-normalized \(768\)-d embedding, concatenated with pelvis/hand goal embeddings and fused by a linear layer to form the attention query. The local goal-centric scene voxel is encoded by a \(2\)-layer 3D-CNN, flattened, and projected to the model dimension. A \(16\)-head cross-attention uses the fused text-goal embedding as Queries and the 3D-CNN features as Keys/Values. The prior network, kinematics encoder, and motion decoder are MLPs with hidden dimension \([512,512]\); the affordance latent \(z\) has dimension \(256\).
Physics-Regularized Diffusion Executor: A DiT backbone of \(8\) Transformer blocks (\(16\) heads, hidden dimension \(512\)). The latent \(z\) is injected into each block via MLP layers, and the local scene voxel is encoded by a lightweight ViT and injected via cross-attention.
7.3 Optimization and Physics Hyperparameters↩︎
The diffusion network is optimized using the AdamW optimizer with a learning rate of \(1 \times 10^{-4}\) and a cosine annealing schedule. We adopt a decoupled two-stage training paradigm. The Semantic Planner is trained for \(50\)K iterations. Subsequently, its weights are frozen, and the Diffusion Executor is trained for \(100\)K iterations with a batch size of \(128\). For the physical regularization loss \(\mathcal{L}_{phy}\), we define the safe distance margin \(d_{\text{safe}}=0.05\) m and the stiffness scaling factor \(\eta=0.5\). To avoid destabilizing the early stages of the reverse diffusion process, we selectively apply the physical loss only at lower noise levels. The joint-specific penalty weights \(w_j\) are assigned as shown in Table 3.
| Index | Name | Weight | Index | Name | Weight |
|---|---|---|---|---|---|
| 0 | pelvis | 1.00 | 14 | right collar | 0.55 |
| 1 | left hip | 0.85 | 15 | head | 0.60 |
| 2 | right hip | 0.85 | 16 | left shoulder | 0.55 |
| 3 | spine1 | 0.84 | 17 | right shoulder | 0.55 |
| 4 | left knee | 0.80 | 18 | left elbow | 0.20 |
| 5 | right knee | 0.80 | 19 | right elbow | 0.20 |
| 6 | spine2 | 0.70 | 20 | left wrist | 0.15 |
| 7 | left ankle | 0.60 | 21 | right wrist | 0.15 |
| 8 | right ankle | 0.60 | 23 | left eye smplhf | 0.10 |
| 9 | spine3 | 0.61 | 24 | right eye smplhf | 0.05 |
| 10 | left foot | 0.55 | 25 | left index1 | 0.01 |
| 11 | right foot | 0.55 | 34 | left ring1 | 0.10 |
| 12 | neck | 0.50 | 40 | right index1 | 0.05 |
| 13 | left collar | 0.55 | 49 | right ring1 | 0.01 |
3pt
8 Additional Experiments and Analysis↩︎
8.1 Ablation of A* Path Guidance↩︎
At inference, the Executor consumes coarse A* waypoints as a macro-level directional prompt (main paper, Sec. 3). To isolate how much of the improvement stems from the hierarchical latent decoupling rather than from this directional guidance, we re-run both Lingo and DeSeG with A* replaced by simple linear interpolation between sub-goals. As reported in Table 4, A* mainly refines navigation-related quality (FID and penetration), whereas DeSeG’s R-Precision remains high even without it (\(0.548\) vs.\(0.562\)). This confirms that the semantic gain originates from the latent decoupling itself, while A* contributes complementary refinement to geometric and physical quality.
| Variant | FID\(\downarrow\) | R-Prec\(\uparrow\) | \(Pene_{m}\downarrow\) | \(Pene_{x}\downarrow\) |
|---|---|---|---|---|
| Lingo w/o A* | 3.244\(\pm\).07 | 0.395\(\pm\).005 | 0.436\(\pm\).01 | 1.427\(\pm\).05 |
| Lingo | 3.093\(\pm\).05 | 0.437\(\pm\).004 | 0.392\(\pm\).03 | 1.229\(\pm\).08 |
| Ours w/o A* | 3.047\(\pm\).04 | 0.548\(\pm\).007 | 0.243\(\pm\).01 | 1.182\(\pm\).03 |
| Ours | 2.882\(\pm\).07 | 0.562\(\pm\).006 | 0.207\(\pm\).01 | 0.921\(\pm\).04 |
3pt
8.2 Statistical Significance and Reliability↩︎
All primary metrics in the main comparison are averaged over \(3\) random seeds. For the headline result, \(\text{FID}=2.882\pm0.07\), \(\text{R-Precision}=0.562\pm0.006\), and \(Pene_{mean}=0.207\pm0.01\); every primary-metric gap between DeSeG and the strongest baseline exceeds \(2\sigma\), so the reported improvements reflect a genuine effect rather than seed noise. For NC-Bench, each of the \(30\) conflict scenarios is rated by \(10\) annotators, and the inter-annotator agreement is high (Fleiss’ \(\kappa=0.818\)), supporting the reliability of the reported SGC scores despite the deliberately compact benchmark size.
9 Negative Constraint Benchmark↩︎
Standard generative metrics fail to rigorously evaluate a model’s robustness against geometric shortcut bias. In highly structured indoor scenes, certain objects ( beds, chairs, sofas) possess overwhelming geometric affordances, and monolithic models tend to ignore the text prompt and trigger these affordances. To evaluate semantic-geometric disentanglement, we construct NC-Bench. Table 5 details \(30\) challenging scenarios where the linguistic instruction explicitly or implicitly contradicts the object’s canonical prior.
| ID | Target | Prior | NC-Bench Text Prompt |
|---|---|---|---|
| 1–5 | Bed | Lie / Sleep | Walk around the edge of the bed. |
| 6–9 | Sofa | Sit down | Walk behind the sofa and stand still. |
| 10–13 | Chair | Sit down | Squat down beside the chair. |
| 14–17 | Table | Pick / Reach | Sit on the floor under the table. |
| 18–20 | Cabinet | Open / Reach | Stand still facing away from the cabinet. |
| 21–23 | TV | Watch / Look | Walk past the TV and look at it. |
| 24–26 | Window | Look outside | Stand with your back to the window. |
| 27–30 | Toilet | Sit down | Sit on the floor leaning against the toilet. |
3pt