1 Introduction↩︎

Automatically generating high-fidelity and logical 3D indoor scenes holds immense application potential in virtual reality, game development, interior design, and embodied AI training. These downstream applications require generated environments with rational spatial layouts and physically reliable interactions. Moreover, populating large-scale scenes necessitates an abundant and diverse supply of 3D assets. Driven by the rapid development of MLLMs and generative artificial intelligence, natural language guided 3D scene generation [1]–[4] draws widespread attention. While recent pioneering works [5]–[9] design autonomous agents that significantly improve generation efficiency and facilitate large-scale scene construction through multi-turn reasoning, they still face a massive gap when processing highly complex real-world indoor environments.

Most mainstream indoor scene generative methods [10], [11] are limited to macroscopic assumptions of single rooms and typically predefine room boundary as simple rectangular contours. Buildings in the real world, however, are inherently holistic, featuring multi-story structures with diverse exterior footprints and strict vertical constraints. Naively stacking single-story layouts generated by existing methods inevitably fails, as it ignores crucial inter-floor dependencies such as precise staircase alignments and coherent exterior boundaries. Directly mapping such building-level layouts from highly abstract text descriptions remains an unresolved challenge. Furthermore, regarding room-level micro-arrangement, existing feed-forward generation models [12], [13] generally lack a deep grounding of 3D continuous physical space. This deficiency frequently leads to floating objects, severe geometric collisions, or irrational placements. Such physical blind spots severely undermine the physical realism and usability of the generated scenes.

Moreover, in practical digital scene development, synthesizing these complex architectures requires bridging the gap between massive 3D asset acquisition and rigorous physical-semantic verification. Traditional pipelines struggle to efficiently source appropriate models and lack a continuous feedback mechanism to correct microscopic placement errors, which severely undermines the physical realism and usability of the generated scenes. To address these bottlenecks and emulate the top-down, physically and functionally bounded human design process, we propose Text2Villa, a hierarchical generation framework that decouples macro-level architectural structure derivation from micro-level scene instantiation.

First, we construct and validate a dataset specifically for multi-story building generation which precisely parameterizes spatial boundaries using geometric polygon primitives and injects connectivity and usability priors. Through autoregressive fine-tuning on this dataset, our architectural layout generator learns implicit architectural rules and robustly outputs globally consistent 3D multi-story building foundations. Second, we introduce a nested graph to represent the scene, in which each room node in the spatial layout graph is recursively expanded into our proposed A-PSSG for asset arrangement. It goes beyond traditional spatial scene graphs by explicitly abstracting the physical affordances of objects as multiple attributes of graph nodes and strict geometric intersection rules as edge constraints. Based on A-PSSG, we formulate the scene instantiation problem as a constrained closed-loop iterative optimization process following the analysis-by-synthesis concept. Our heuristic solver combines a physical engine with the visual semantic feedback of MLLMs to dynamically perform physics-aware actions within a continuous observe-evaluate-modify closed-loop search, which can not only automatically corrects the semantic positional relationships among furniture but also successfully resolves fine-grained mesh penetrations, support failures, and cavity containment problems within restricted parameter spaces. The contributions of this work are summarized as follows:

• We propose a hierarchical generation framework, Text2Villa, capable of directly parsing text into multi-story 3D building structures. This macro-level capability is powered by an autoregressive layout generator fine-tuned on a custom-built dataset with architectural usability constraints.

• We propose a nested graph to hierarchically model 3D scenes, involving a layout graph for macro-level spatial configuration and an A-PSSG for micro-level asset placement. The A-PSSG abstracts multiple functional affordances like support surfaces and containment cavities into node attributes and further constructs dual physical-semantic edge constraints accordingly, providing a robust theoretical framework for the physically realistic generation of 3D scenes.

• Based on the A-PSSG, we design an analysis-by-synthesis instantiation solver to ensure that objects acquired from diverse asset sources are physically plausible within the spaces. By integrating geometric hard constraint verification and MLLM heuristic evaluation, this solver effectively addresses mesh collisions, floating, and semantic errors in fine-grained asset configuration, thereby achieving high-fidelity indoor scene synthesis.

2 Related work↩︎

2.0.0.1 Text-driven 3D Scene Generation

Previous text-driven 3D scene generation primarily relies on predefined rules [14], [15] or graph-based retrieval [16]. Recently, autoregressive [17], [18] and diffusion models [19]–[22] drive significant progress in this field. Diffusion-based methods typically generate panoramas or meshes through depth estimation and image inpainting. While producing visually coherent results, these approaches lack independent 3D asset entities and thus hinder subsequent physical interactions. Other approaches based on asset retrieval and arrangement [8], [11], [23], [24] generate scenes comprising independent 3D models. Most of these methods restrict their scope to a single room with preset rectangular boundaries. In contrast, Text2Villa directly parses text to generate complex villa-level architectural structures featuring polygonal boundaries and multi-story connectivity.

2.0.0.2 Scene Graph Representation for Layouts

Scene graphs serve as a structured representation widely applied in 3D scene understanding and synthesis. In existing layout generation research [25]–[28], scene graphs primarily capture the relative semantic spatial relationships among nodes. Real interior design follows strict physical laws where spatial semantics alone fail to accurately guide micro-level arrangements, such as placing a book inside a specific shelving unit. While the concept of affordance [29], [30] is crucial for resolving these physical interactions, it has not yet been integrated into scene graph representations. The breakthrough of this work lies in the proposal of A-PSSG. It explicitly abstracts physical attributes like support surfaces and containment cavities into internal features of graph nodes, and establishes dual physical and semantic constraints for the edges accordingly, providing a mathematical foundation for precise geometric intersection and placement.

2.0.0.3 Layout Generation and Optimization in Spatial Reasoning

3D scene layout generation constitutes a combinatorial optimization problem requiring spatial reasoning and geometric constraints. Existing methods combine decomposition-recomposition [31]–[33], depth/point cloud alignment [34]–[38], and multi-instance attention [39], [40] to update multi-asset poses and restore scene layouts. Meanwhile, large language models (LLMs) demonstrate remarkable capabilities in commonsense reasoning and zero-shot task planning, prompting several works [41]–[43] to employ them for layout generation. Inherently trained on text corpora, LLMs lack a genuine perception of continuous 3D physical spaces. Consequently, layouts generated in a feed-forward manner appear semantically reasonable at a macroscopic level while containing extensive physical hallucinations at a microscopic level. Overcoming this limitation, we integrate semantic spatial reasoning with physical optimization and propose an analysis-by-synthesis closed-loop iterative instantiation solver. We utilize a physics engine to calculate support and collision penalties, and an MLLM to assess scene semantics. Within continuous iterations of observation, evaluation, and modification, our solver guides pose search, optimizes layouts, and provides semantic feedback.

3 Text2Villa↩︎

Text2Villa take unstructured natural language describing complex buildings as input and outputs fully instantiated, physically plausible 3D indoor scenes. We formulate the scene generation as a nested generative process, as illustrated in Fig. 1. At the macro level, we model the building as a structural layout graph for the parametric representation of floor (Subsec. 3.1). At the micro level, each room node within this macro graph acts as a parent container, recursively expanding into our proposed A-PSSG (Subsec. 3.2) to define fine-grained object relations. Following this nested graph construction, the pipeline proceeds with hybrid 3D asset acquisition (Subsec. 3.3) and a physics-aware closed-loop iterative instantiation (Subsec. 3.4).

3.1 Multi-Story Architectural Layout Generation↩︎

Mapping text into complex 3D architectural structures is the initial step of the scene generation pipeline. We model the architectural layout as a hierarchical data composition formally represented as \(\mathcal{H} = \{\mathcal{F}_1, \dots, \mathcal{F}_N\}\), where \(\mathcal{F}_i\) represents the \(i\)-th floor. Each floor \(\mathcal{F}_i\) is viewed as a spatial layout graph \((\mathcal{R}, \mathcal{E})\). The nodes \(\mathcal{R}\) represent various room instances covering ten core functional areas (such as living rooms, bedrooms, and kitchens) with polygonal boundaries. The edges \(\mathcal{E}\) explicitly encode the connectivity between rooms. These relationships include restricted connections with physical partitions such as entrances, doors, and windows, open connections, and multi-story vertical connections such as staircases located in utility rooms. This macro-level spatial graph serves as a basis for the micro-level graph expansion.

Based on the above definition, we construct and validate a fine-tuning dataset dedicated to multi-story building generation comprising 1,000 high-quality text-to-architecture JSON pairs (additional details are provided in Appendix 7). At the data representation level, we parameterize spatial boundaries and wall segments using geometric primitives. This allows us to define various exterior architectural contours including H-shape, L-shape, T-shape, U-shape, and irregular polygons while ensuring the internal room layout supports arbitrary polygonal shapes. We introduce rationality prior constraints based on real-world architectural design during dataset construction. For instance, all rooms must be accessible, bathrooms and kitchens require windows, and balconies can feature panoramic windows. Using this dataset, we fine-tune an LLM-based [44] autoregressive layout generator. By parsing the input text, the generator outputs structured JSON sequences, and its geometric parsing module converts the parametric information into multi-story building volume containing 3D base walls, floors, and door or window meshes. This process provides a precise physical space for subsequent fine-grained 3D asset population.

3.2 Affordance-driven Physical-Semantic Scene Graph↩︎

Our Text2Villa represents the scene as a nested graph, capturing the macro-level space configuration via a layout graph and encoding the object arrangement with an A-PSSG, as illustrated in Fig. 2. Formally defined as \(\mathcal{G} = (\mathcal{V}, \mathcal{E}_{sem}, \mathcal{E}_{phys})\), the A-PSSG introduces explicit physical affordances and strict geometric constraints to resolve physical ambiguities. The node and edge representations are defined as follows:

3.2.0.1 Node physical affordance

Unlike previous hard partition methods based on object size or a single semantic category, we abstract the physical affordance of objects as multi-assignable graph node attributes. A node in the graph dynamically assumes three core interactive roles.

• Support providers \(\mathcal{V}_{pro}\) such as floors, tables, or cabinets expose valid support surfaces \(\mathcal{S}\) outwards or provide collision-free containment cavities \(\mathcal{C}\) inwards.

• Support consumers \(\mathcal{V}_{con}\) such as desk lamps, books, or murals are constrained by gravity or attachment dependencies and must establish a physical connection with the support surfaces \(\mathcal{S}\) or cavities \(\mathcal{C}\) provided by \(\mathcal{V}_{pro}\).

• Functional interactors \(\mathcal{V}_{int}\) such as sofas or wardrobes define an interaction reservation zone along their functional orientation to allocate space for human activities or part movements.

3.2.0.2 Edge dual constraints

In A-PSSG, edges describe relative positions and introduce geometric and physical constraints.

• Semantic relation edges \(\mathcal{E}_{sem}\) define the relative orientation between objects. For instance, \((O_i, \text{face\_to}, O_j)\) requires the angle between their functional orientation normal vectors to fall within a specific threshold.

• Physical interaction edges \(\mathcal{E}_{phys}\) include support relations \((O_i, \text{ontop}, O_j)\) and containment relations \((O_i, \text{inside}, O_j)\). Specifically, the former requires the bottom bounding box of \(O_i\) to be coplanar with the support surface \(\mathcal{S}\) of \(O_j\) and projected inside it. The latter further requires the geometry of \(O_i\) to have no collision with the solid part of \(O_j\) and to be located within an independent cavity \(\mathcal{C}\) of \(O_j\), provided the support condition is met.

In order to construct the A-PSSG efficiently, we conduct hierarchical instructions so that our workflow can utilize the common sense reasoning capability of MLLMs [45]. Specifically, our A-PSSG representation is constructed with a node set \(\mathcal{V}\) initialized by the 3D asset categories \(C = \{c_1, c_2, \ldots, c_n\}\) inferred for the current room \(\mathcal{R}\). The nodes are assigned the aforementioned physical affordance attributes based on spatial logic parsed from the text and implicit real-world object associations. Consequently, the graph establishes semantic relation edges \(\mathcal{E}_{sem}\), as well as physical interaction edges \(\mathcal{E}_{phys}\) between attribute ports \(\mathcal{V}_{pro}\) and \(\mathcal{V}_{con}\) based on affordance matching. This process successfully translates unstructured text into physical and semantic contracts that guide subsequent 3D spatial layout solving.

3.3 Hybrid 3D Asset Acquisition↩︎

To balance scene diversity, semantic alignment, and the precise geometric requirements of specific furniture, we adopt a hybrid asset acquisition. This strategy encompasses procedural generation, cross-modal retrieval, and style-aware extraction, ensuring both high-quality geometric interactions and open-vocabulary richness.

For structural categories \(c_i \in C_{struc}\) (e.g., shelving units, cabinets), geometric accuracy is paramount for determining affordances. Here, we prioritize procedural generation frameworks like Infinigen [14]. Beyond merely generating the solid mesh, Text2Vill explicitly computes and exports a set of available support surfaces \(\mathcal{S}\) and a set of internal free-space bounding boxes \(\mathcal{C}\) during generation. For other objects, we involve a retrieval method that can query any large-scale 3D dataset, as detailed in our acquisition pipeline (see Algorithm 9 in Appendix 8). We use the text category of the graph node as a query condition and extract the text embedding and compute the cosine similarity against the 3D assets within a unified shape latent space [46]. Then, we filters out unusable or geometrically erroneous assets, sorts the remaining candidates by their similarity scores, and extracts the best-matching mesh. Furthermore, by integrating metadata tags and asset texture editing methods [38], our hybrid asset acquisition also exhibits potential in stylized scene generation when user prompts specify distinct stylistic preferences (e.g., a rustic medieval tavern or a vintage detective’s study, as shown in Fig. 3).

Semantic relation edges \(\mathcal{E}_{sem}\) in the A-PSSG depend on exact knowledge of canonical object orientations. Thus, we introduce a preprocessing step before asset storage. We render four orthogonal views for each retrieved 3D model around the \(Z\)-axis at intervals of \(90^\circ\) and input them into the MLLM alongside specific prompts, requiring it to identify the image representing the functional front of the object. The extracted forward offset angle is recorded in the asset metadata, laying the foundation for the closed-loop solver in Subsec 3.4 to execute precise pose optimization.

3.4 Physics-Aware Closed-Loop Solver↩︎

We formulate the A-PSSG instantiation process as an optimization problem with geometric constraints to determine the exact six-degree-of-freedom pose \((T, R)\) and scale \(D\) for each 3D asset in the scene. Fig. 4 illustrates this iterative resolution process through a concrete scene instantiation example. We adopt an analysis-by-synthesis paradigm, which leverages an MLLM and a physics engine collaboratively to minimize the overall energy of the generated scene \(E(S_t)\), defined as: \[\label{eq1} {!}{ \displaystyle E(S_t) = \lambda_{col} E_{col}(S_t) + \lambda_{sup} E_{sup}(S_t) + \lambda_{sem} E_{sem}(S_t) }\tag{1}\] where \(S_t\) denotes the scene state at the \(t\)-th iteration of our optimization process. Based on the A-PSSG, we compute the following energy terms: the semantic penalty \(E_{sem}\) is evaluated by the MLLM. Taking multi-view rendered images of the current scene as input, it evaluates whether the relative object relationships defined by \(\mathcal{E}_{sem}\) are satisfied, whether appropriate interaction zones are reserved for nodes with the \(\mathcal{V}_{int}\) attribute, and assesses the overall richness of the scene. Meanwhile, \(E_{col}\) and \(E_{sup}\) are calculated via the physics engine. Specifically, \(E_{col} = \sum_{i \neq j} \text{Vol}(O_i \cap O_j)\) is the collision penalty that calculates the intersection volume between any pair of independent objects \(O_i\) and \(O_j\) in the scene. This term drops to zero if and only if the two objects do not overlap. Similarly, the support penalty is defined as the sum of the distance error and the projection out-of-bounds error: \[\label{eq2} {!}{ \displaystyle E_{sup} = \sum_{(i,j)\in \mathcal{E}_{phys}} \big( \| p_{bottom}^i - p_{proj}^i \|_2 + \mathcal{I} (p_{proj}^i \notin \mathcal{S}_j) \cdot \text{Penalty} \big) }\tag{2}\] where \(p_{bottom}^i\) represents the bottom center point of object \(O_i\), and \(p_{proj}^{i}\) denotes its projection onto the plane containing the valid support surface \(\mathcal{S}_j\) of the provider object \(O_j\), cast along the negative normal direction of \(\mathcal{S}_j\). \(\mathcal{I}\) is an indicator function. This penalty is evaluated only for nodes with the \(\mathcal{V}_{con}\) attribute to ensure gravity and attachment constraints.

3.4.0.1 Analyzing-Updating Loop

The instantiation process begins at the initial state \(S_0\), where we roughly initialize the base asset set \(\mathcal{A}\) into the architectural container (generated in Sec. 3.1) based on the text prompts and the constraints defined in the A-PSSG. Because the scene at this step typically involves geometric conflicts and layout blanks, the solver continuously generates new A-PSSG and states \(S_{t+1}\) through iteration. During each update step \(t\), the solver computes the aforementioned penalties to determine the optimization direction and executes discrete atomic actions: it utilizes add_entity and remove_entity to improve scene completeness and resolve irreconcilable collisions; and relies on update_pose (outputting a translation \(\Delta T\) and rotation \(\Delta R\)) alongside update_scale to adjust the placements and dimensions of the assets. The solver continuously executes these operations until \(E(S_t)\) converges or drops below a predefined safety threshold.

3.4.0.2 Fine-grained resolving for inside constraints

Handling the constraint \((O_i, \text{inside}, O_j)\) requires specific treatment during the update loop. Traditional collision detection triggers false alarms here since the bounding box of the parent object inevitably intersects with the child. Therefore, the optimization target specifically enforces that the bounding box \(B_{child}\) of \(O_i\) must be completely contained within a candidate cavity \(\mathcal{C}_k \in \mathcal{C}\) of \(O_j\), while their solid meshes must not intersect. Crucially, an internal relationship inherently dictates a valid support dependency; the child object cannot float and must rest firmly on the bottom surface of the cavity \(\mathcal{C}_k\). As shown in Fig. 5, the solver enters a local search mode by triggering the add_entity inside action. Guided by boundary detection and support verification from the physics engine, it continuously utilizes update_pose to search the valid parameter space. Once a pose satisfying cavity containment, collision-free, and resting conditions is found, both the \(E_{col}\) and \(E_{sup}\) penalties for this inside constraint immediately drop to zero, thereby successfully locking in the precise microscopic asset placement.

4 Experiments↩︎

4.1 Experimental Setup↩︎

4.1.0.1 Implementation Details

Our autoregressive layout generator builds upon Qwen3.5-9B [44] and undergoes fine-tuning with the LoRA strategy. We trained the model for 15 epochs on a single NVIDIA L20 GPU using the AdamW optimizer with a batch size of 4, and a learning rate of 1e-4. For hybrid 3D asset acquisition, we integrated the Objaverse [47] and 3D-FUTURE [48] datasets to support cross-modal retrieval and utilized the geometry nodes of Infinigen [14] to generate structural assets. The MLLM utilized throughout the pipeline is GPT-4o [45]. In Eq. 1 , the penalty weights are empirically set to \(\lambda_{sem} = 1.0\), \(\lambda_{col} = 5.0\), and \(\lambda_{sup} = 2.0\).

2pt

4.1.0.2 Baselines

We selected state-of-the-art text-to-3D indoor scene generation methods based on LLMs or agents as primary baselines. The evaluation comprises two groups according to the task hierarchy. For macro-level architectural layout generation, we compare with Holodeck [1] and SceneFoundry [49]. Because these methods inherently lack support for multiple floors, we relax the constraints and compare their multi-room generation capabilities with our single-story degraded results. For micro-level asset instantiation, we evaluate against Holodeck, LayoutVLM [10], and SceneWeaver [8]. These approaches focus on single-room generation and typically restrict room outlines to rectangles.

4.1.0.3 Evaluation Metrics

At the macro-layout level, we measure Room Reachability (RR) to detect rooms lacking door connections or completely enclosed in the generated structure. We also use Gemini 3.1 Pro [50] as a judge to provide a Gemini-Score (GS) based on real-world interior design standards. To evaluate the unique multi-story generation capability of Text2Villa, we introduce the Format Success Rate (FSR) to quantify the proportion of outputs that can be parsed into 3D buildings without errors. Additionally, we use the Valid Connection Rate (VCR) to verify whether stairs are correctly generated and floor slabs are properly connected between levels. At the micro-level, we report the Collision Rate (CR), Floating Rate (FR), and Out of Boundary Rate (OBR) to assess physical compliance. We specifically define the Containment Success Rate (CSR) to accurately measure fine-grained constraints such as books placed inside bookshelf compartments. For visual and semantic correctness, we employ the CLIP-Score (CS) to measure the semantic alignment between rendered multi-view images and texts. We also use GS to evaluate overall scene completeness and the rationality of interaction reserved areas, while counting the Number of Objects (NO) in the scene. All metrics represent the calculated average of three scenes for each architectural layout and room type.

4.2 Quantitative Results↩︎

4.2.0.1 Macro-Level

As shown in Table 1, our method comprehensively outperforms Holodeck and SceneFoundry in both RR and GS even in the single-story generation subtask. More importantly, when processing complex multi-story text descriptions, existing baselines fail to handle such tasks while Text2Villa robustly outputs buildings with valid multi-story connectivity. This demonstrates that our layout generator learns and follows real-world architectural logic.

4.2.0.2 Micro-Level

Table ¿tbl:quantitative2? presents the fine-grained asset layout performance across nine different room types. LayoutVLM and SceneWeaver are highly prone to geometric collisions or floating issues when handling high-frequency physical interactions. In contrast, Text2Villa achieves physically compliant layouts with zero collisions and zero floating across all room types. When dealing with highly challenging object containment relationships, Holodeck and LayoutVLM fail to respond effectively and only manage simple top-surface support cases. SceneWeaver shows unstable performance despite having some processing capability. Benefiting from the A-PSSG and internal constraint solving, our method achieves an average CSR of 97.6%. Furthermore, our scenes attain the best performance in semantic alignment and scene rationality even while generating a significantly larger number of assets than the baselines.

4.2.0.3 Human Preference

Table 2 reports the preference scores obtained from a blind test conducted by 20 human volunteers. Our method achieves an overwhelming advantage in both physical plausibility and semantic alignment, with an overall preference rate superior to all baselines. The implementation can be found in Appendix 9.

4.3 Qualitative Results↩︎

4.3.0.1 Macro-Level

As illustrated in Fig. 6, Holodeck is limited to rectangular boundaries, generating fragmented room layouts that lack realistic architectural connectivity. SceneFoundry attempts to convert text into procedural parameters of Infinigen to drive layout generation. Nevertheless, its parameter mapping process suffers from deviations, which frequently result in imbalanced room area distribution and tend to produce unreasonable redundant windows. Conversely, Text2Villa successfully maps complex spatial descriptions such as open-plan kitchens into a globally connected layout with irregular polygonal boundaries. The generated 3D views exhibit extremely high villa-level architectural realism.

4.3.0.2 Micro-Level

Fig. 7 provides a qualitative comparison of micro-level instantiation. Baselines exhibit obvious spatial and physical perception defects. The differentiable optimization process of LayoutVLM fails to resolve all physical conflicts, leading to frequent intersections among furniture pieces. Its self-consistent decoding mechanism also struggles to correct orientation errors completely. SceneWeaver relies on agents that lack strict physical constraints, causing objects to scatter frequently on the floor away from supporting surfaces. This approach also bypasses collision issues through deletion operations, generating overly sparse scenes. While Holodeck produces physically reasonable global layouts, it cannot resolve fine-grained containment relationships and lacks visual realism in its rendering. In contrast, Text2Villa strictly enforces semantic orientation constraints such as placing nightstands on both sides of a double bed. It also achieves the precise placement of small objects inside display racks and storage cabinets, yielding highly organized, natural, and functional visual performances.

4.4 Ablation Study↩︎

We verify the necessity of the key designs in Text2Villa, with quantitative results summarized in Table 3.

4.4.0.1 Effectiveness of Fine-tuning

Regarding macro-level layout generation, removing the LoRA fine-tuning to rely solely on zero-shot LLMs causes a severe degradation in system performance. The FSR, which measures the successful parsing of output JSON sequences into 3D building meshes, plummets drastically from 98.2% to 26.8%. Furthermore, even when syntactically correct coordinate parameters are generated, the zero-shot model completely lacks spatial geometric reasoning, reflected by the drops in both RR and VCR. As qualitatively illustrated in Fig. 8, the model fails to translate textual instructions into irregular polygonal boundaries, generates unreachable enclosed rooms, and cannot establish valid multi-story staircase connections. This proves the necessity of constructing domain-specific datasets for autoregressive fine-tuning to learn implicit architectural rules.

4.4.0.2 Impact of A-PSSG

We revert the A-PSSG to a traditional pure semantic scene graph. Under this configuration, the CR of the scenes surges to 36.3%, the FR reaches 22.7%, and the CSR completely drops to zero. This indicates that pure semantic relationships cannot guide precise geometric intersections in 3D space without explicit physical affordance abstraction.

4.4.0.3 Analysis of the Physics-Aware Closed-Loop Solver

We design three variants to evaluate the closed-loop instantiation mechanism. 1) W/o Closed-loop (Init-Only): Objects are coarsely placed only once based on the initial A-PSSG, without subsequent iterative optimization. The resulting scenes are filled with collisions and empty spaces, proving that feed-forward single pose estimation is highly unreliable in dense 3D spaces. 2) W/o Physics (MLLM-Only): The underlying physical check is removed, and pose updates are performed solely based on visual feedback from the MLLM. Because the MLLM lacks precise depth perception, it can move objects to semantically correct general areas but fails to handle fine-grained object interactions. 3) W/o MLLM (Physics-Only): The MLLM semantic misalignment penalty is discarded (by setting \(\lambda_{sem}=0\)) , forcing the solver to search solely for the elimination of physical conflicts. Although the generated scenes are physically compliant, this setup triggers severe semantic confusion, such as televisions facing walls or beds blocking doorways. Reflected in the data, the GS evaluating semantic rationality reaches only 5.6.

5 Discussion↩︎

Text2Villa achieves remarkable progress in generating multi-story buildings and opens up promising avenues for future exploration. Regarding asset acquisition, although we currently rely on finite databases, achieving fully open-vocabulary expansion can be readily addressed by integrating emerging native 3D generation models on the fly. Furthermore, real-world indoor scenes encompass a myriad of deformable (e.g., cloth) and articulated objects (e.g., cabinet doors). Although our proposed A-PSSG primarily focuses on rigid bodies, its graph structure is inherently extensible. Mapping kinematic parameters into node attributes and integrating into closed-loop solver remains a valuable open problem. From a computational perspective, the multiple rounds of heuristic search executed by the solver incur substantial inference latency (see Appendix 10 for runtime analysis), particularly when resolving complex interactions among dozens of microscopic assets. We plan to implement parallel optimization across rooms and explore knowledge distillation to train a lightweight, dedicated 3D spatial reasoning network, thereby drastically accelerating the convergence of scene instantiation.

6 Conclusion↩︎

In this paper, we present Text2Villa, a text-driven multi-story 3D indoor scene generation framework. To bridge the gap between highly abstract natural language and strict physical reality, we bypass the limitations of forward generation paradigms and single-room assumptions by proposing a hierarchical generation pipeline. At the macro level, we fine-tune an autoregressive layout generator that parses unstructured text to generate architectural foundations containing polygonal boundaries and multi-story connectivity. At the micro level, we expand the macro nodes and propose A-PSSG, which extends traditional pure semantic relations into physical interaction graphs encompassing geometric support and containment cavity constraints. Building upon this representation, we design a closed-loop iterative solver rooted in the analysis-by-synthesis paradigm. By integrating semantic feedback from MLLM with the hard constraints of an underlying physics engine, this solver adopts a dynamic observation-evaluation-modification mechanism, effectively resolving common issues such as collision, floating, and semantic error during fine-grained asset instantiation within a continuous parameter space. Experiments demonstrate that Text2Villa achieves high-quality generation from text to villa-scale architectures and provides a robust algorithmic framework for the physically compliant placement of assets. Text2Villa offers a reliable pathway for generating coherent and physically usable 3D scenes for applications in virtual reality, digital content generation, and embodied artificial intelligence.

7 Dataset Construction Details↩︎

As mentioned in Subsec. 3.1, we construct a fine-tuning dataset containing 1,000 text-JSON pairs to train the autoregressive layout generator. To this end, we develop a procedural spatial partitioner that synthesizes geometrically precise building foundations and floor plans.

7.0.0.1 Procedural Spatial Partitioner

The spatial partitioner relies on 2D Boolean operations and the binary space partitioning (BSP) algorithm. The process operates as follows:

1. Exterior Contour Initialization: We start with a rectangular bounding box. To create diverse architectural footprints (e.g., L-shape, U-shape, T-shape, H-shape, and irregular polygons), we subtract randomized rectangular chunks from the corners or edges using Boolean difference operations via the shapely library.

2. Internal BSP Partitioning: The remaining polygon on each floor is recursively subdivided using a BSP tree. Splitting hyperplanes are randomly selected along the \(X\) or \(Y\) axes, terminating when the target number of rooms is reached or the area falls below a usability threshold.

3. Layout Connectivity and 1D Elements: Based on the partitioned 2D rooms, we extract shared line segments between adjacent rooms as interior walls, and segments without neighbors as exterior walls. Doors and opens (represented as 1D LineString) are placed on interior walls to establish accessibility based on functional priors. Windows are sampled on exterior walls, and a main entrance is assigned to a exterior wall on the ground floor.

4. Vertical Alignment: For multi-story buildings, a fixed \(4\times4\) staircase is rigidly anchored in the identical room across all floors, thereby guaranteeing valid vertical multi-story connectivity.

7.0.0.2 Human-in-the-loop Annotation

The raw outputs generated by the spatial partitioner are mathematically precise but lack semantic information. We adopt a human-in-the-loop approach to assign specific room categories to the layout, such as labeling large polygonal areas as living rooms, and to filter out unreasonable configurations. To enhance the generalization capability of the model to diverse user instructions, we annotate each layout with either coarse-grained or fine-grained natural language descriptions.

7.0.0.3 Dataset Statistics and Setup

The dataset contains 1,000 samples in total, with a ratio of \(2:4:4\) for single-story, two-story, and three-story buildings. Although the procedural generation theoretically supports buildings with unlimited floors, we restrict the maximum number of floors to three to balance the computational overhead of micro-asset instantiation. Each floor has a standard height of 2.8 meters and contains an average of five rooms. Table 10 presents an example of a typical text-JSON data pair. The system prompts used to fine-tune the autoregressive layout generator are provided in Table 11. Moreover, Fig. 12 visualizes diverse 3D building foundations generated by Text2Villa, which are obtained by processing the JSON data from the layout generator through its geometric parsing module built based on Infinigen [14]. The generated layouts overcome the limitations of traditional single-room bounding boxes and smoothly scale from single-story apartments to multi-story villas.

8 Hybrid Asset Acquisition Algorithm↩︎

9 Human Preference↩︎

We conduct a user study to comprehensively evaluate the quality of micro-level scene instantiation from the perspective of human perception. We invite twenty volunteers to evaluate ten distinct indoor scenes through a custom web-based interface shown in Fig. 13. For each case, the interface presents the input text description at the top and displays the 3D scenes generated by three baselines and Text2Villa side-by-side at the bottom. To eliminate potential bias caused by the presentation order, these four rendering results is randomized for each trial. Participants are asked to carefully observe the scenes and select the optimal result according to three criteria, namely physical plausibility, semantic consistency, and overall preference. The interface provides clear definitions for each criterion to ensure that volunteers apply consistent evaluation standards. We ultimately collect 600 independent and valid votes. The detailed comparisons are summarized in Table 2.

10 Runtime Analysis↩︎

Generating fully instantiated, physically and semantically plausible 3D scenes is a complex combinatorial optimization problem. As detailed in Subsec. 3.4, Text2Villa employs an analysis-by-synthesis paradigm driven by a heuristic closed-loop solver. While this design eliminates physical conflicts like collision and floating prevalent in feed-forward methods, it introduces higher inference latency.

10.0.0.1 Time-Quality Trade-off

Experiments show that the energy function \(E(S_t)\) for micro-level asset instantiation typically converges within 20 iterations. Under the predefined safety threshold \(E(S_t) \leq 1.0\), physical conflicts are completely eliminated, leaving only slight and acceptable semantic deviations. The average instantiation time per room is approximately 15 minutes, resulting in a total time of about 225 minutes for a standard three-story villa comprising approximately 15 rooms. The major computational bottleneck arises at each optimization step \(t\): the solver must query the MLLM via API and wait for its response to compute \(E_{sem}(S_t)\) and determine the next action. This frequent network communication overhead significantly prolongs the total running time. We consider this a necessary and worthwhile trade-off: exchanging inference time for absolute physical correctness, effectively elevating the usability of generated scenes from visually plausible to physically interactive for downstream embodied AI tasks.

10.0.0.2 Mitigation and Scalability

The computational cost primarily stems from continuous MLLM API calls, which cost approximately $0.80 per room, and the massive search space of 6-DoF poses. We provide the following strategies to enhance runtime flexibility:

• Search Space Reduction: Layout generation for small rooms takes significantly less time than for open areas, because spatial boundaries heavily restrict the number of valid heuristic actions.

• Hyperparameter tuning: Users can accelerate the generation process by reducing the maximum number of iterations or increasing the energy safety threshold \(E(S_t)\). Although this adjustment might affect the results, the underlying physics engine ensures that any degradation in layout quality remains marginal and controllable.

11 Application↩︎

As illustrated in Fig. 14, Text2Villa can generate villa-scale 3D environments from text, demonstrating significant potential for various downstream applications. For game development, Text2Villa lowers the barrier to user-generated content. The generated architectural assets and furniture are represented as explicit meshes that seamlessly integrate into mainstream game engines such as Unity and Unreal Engine for rapid prototyping. This integration effectively reduces the manual modeling costs typically required to build large-scale virtual communities. Driven by the dual physical and semantic constraints of A-PSSG alongside a physics-aware closed-loop solver, Text2Villa produces highly realistic scenes with zero collisions and accurate affordances. This capability establishes a reliable and interactive real-to-sim workflow, providing critical support for training embodied agents in complex multi-room tasks.

References↩︎