1 Introduction↩︎

Unified 3D models [1]–[3] aim to bridge 3D understanding and generation within one backbone, where a single model can reconstruct a 3D object from an image, generate one from language, describe its structure in text, and support downstream reasoning over geometry. This unification is appealing because generation and understanding can support each other within a shared representation, but it also imposes stronger requirements on multimodal reasoning: a unified 3D model should balance global structure and local detail, translate open-ended language into concrete geometric decisions, and allocate compute dynamically when semantic-geometric alignment requires finer reasoning.

Current systems fall short on each of these requirements because text-3D interaction remains largely implicit. Previous works [1], [2] concatenate text and 3D tokens into a monolithic sequence and rely on self-attention to discover cross-modal correspondences. Recent 3D advances [4]–[7] make the geometric representation multiscale, but the reasoning architecture has not evolved accordingly. What is missing, therefore, is not merely a stronger hierarchical 3D representation, but a unified design that structures language reasoning and geometric reasoning jointly, enabling structured interaction between semantic cues and geometric content.

To address this gap, we introduce , a unified 3D model built around elastic semantic anchoring. ELSA3D first represents 3D shapes with an octree VQ-VAE in which every content token carries an explicit deterministic scale tag, exposing multiple geometric resolutions to the model. Then, the model organizes language into a semantic trace spanning Global, Structure, and Appearance cues, decomposing text descriptions into finer semantic granularity. To connect both semantic and geometric abstractions, ELSA3D introduces Anchor Tokens. Each anchor is a transient cross-modal unit instantiated from a selected semantic token, routed to the most relevant 3D scale, fused with scale-specific geometric evidence, and written back into the unified sequence. Anchors keep cross-modal interaction sparse and explicit at once, avoiding the cost of dense text-geometry attention while preserving precise binding where it matters.

Since not all semantic tokens contribute equally to 3D grounding (e.g., function words, generic modifiers, etc.), ELSA3D equips each transformer block with a lightweight elastic router. The router selects which text tokens instantiate anchors and which 3D scale they query, while also deciding whether the block should execute and how much MLP width to allocate. This makes both computation and reasoning elastic by focusing on cross-modal interaction where targeted reasoning is most needed.

Extensive experiments demonstrate that elastic semantic anchoring improves unified 3D models along generation fidelity, 3D-language understanding, and inference efficiency. ELSA3D achieves state-of-the-art results across image-to-3D generation, text-to-3D generation, and 3D captioning, improving over the strongest unified baseline by \(+2.74\) CLIP and \(-2.03\) FD on image-to-3D, \(+1.35\) CLIP and \(-4.05\) FD on text-to-3D, and \(+3.56\) Sentence-BERT on 3D captioning. Ablations further show that sparse anchor routing outperforms dense cross-modal fusion while reducing FLOPs from 1081G to 632G and latency from 29.8s to 17.2s.

Our contributions are summarized as follows:

• We introduce , a unified 3D understanding-and-generation model that structures language reasoning and geometric reasoning along matched abstraction scales.

• We propose Anchor Tokens, a sparse and dynamic cross-modal interface that grounds selected semantic tokens in scale-specific 3D evidence and writes the fused language-geometry signal back into the unified representation.

• We design an elastic routing mechanism that jointly controls block execution, MLP width, anchor selection, and geometric scale assignment, enabling elastic computation and elastic semantic-geometric reasoning under a single routing scheme.

• ELSA3D establishes new state-of-the-art across image-to-3D generation, text-to-3D generation, and 3D captioning, outperforming the strongest unified baseline on every reported metric while reducing FLOPs and inference latency by roughly half relative to a non-elastic variant.

2 Related Work↩︎

Our work is closely related to unified 3D understanding and generation models [1]–[3], hierarchical 3D representation and generation [4], [5], [8]–[10], adaptive token selection and elastic computation [11]–[15], and multimodal token fusion and alignment [16]–[22]. A comprehensive discussion is provided in Appendix 6. Prior hierarchical 3D models demonstrate that multiscale structure is effective for scalable representation and generation, but they use hierarchy primarily to organize geometry, whereas prior routing and token-selection methods mainly use routing to reduce computation, prune redundant tokens, or allocate tokens across experts. ELSA3D instead couples hierarchy and routing for scale-aware language–geometry grounding: the hierarchy provides explicit geometric levels of detail, and routing determines which scale each semantic anchor should query before writing fused cross-modal evidence back into the unified understanding-and-generation model.

3 Method↩︎

A core challenge in unified 3D foundation models is translating open-ended language into structured reasoning signals that preserve compositional semantics and physical constraints [1]–[3]. Directly mapping text prompts to latent 3D tokens is inherently under-specified: language descriptions typically omit precise geometric details, relations, and material cues, yielding shapes that match coarse appearance but break in structure and texture consistency. Yet exhaustively coupling every text token to all 3D tokens is both computationally wasteful and semantically noisy, since many tokens lack precise geometric counterparts. ELSA3D addresses this mismatch with elastic semantic anchoring (1). First, we represent each 3D object with a multiscale octree VQ-VAE whose tokens carry explicit scale tags, making geometric resolution available to the transformer (3.1). Second, we introduce anchor tokens, transient cross-modal units that bind selected semantic tokens to geometric evidence at a specific scale and write the fused signal back into the unified sequence (3.2). Third, we use a lightweight per-block router to make both computation and grounding adaptive (3.3). ELSA3D is trained in two stages: the octree VQ-VAE is first trained on 3D data, then the unified LLM over interleaved text and 3D sequences with auxiliary budget losses that shape the router’s depth, width, and anchor-sparsity decisions ([sec:training]).

3.1 Semantic and Geometric Representations↩︎

Given an input condition \(x\) (e.g., a text prompt, an image, or both), ELSA3D generates an output sequence \(y\) that represents either a 3D object or a language response grounded in 3D geometry. For 3D generation, \(y\) consists of structural and content tokens that define an octree-based 3D representation, which is decoded into a textured 3D shape. For 3D understanding, \(y\) is a language sequence conditioned on 3D geometry. We model both tasks with a unified autoregressive transformer over semantic tokens and geometric tokens. At transformer block \(i\), we denote the hidden states of the unified sequence by \(\mathbf{H}^{i}=\{\mathbf{h}^{i}_{j}\}_{j=1}^{N_{\mathrm{uni}}}\), where \(N_{\mathrm{uni}}\) is the total sequence length. The semantic-token subset is \(\mathbf{T}^{i}=\{\mathbf{t}^{i}_{m}\}_{m=1}^{M}\). The geometric tokens are organized by octree scale, where \(\mathbf{G}^{i,s}\) denotes the hidden states of geometric tokens at scale \(s\), with \(s\!=\!1\) being the coarsest scale and \(s\!=\!S\) the finest scale. This unified formulation requires representations that make both semantic abstraction and geometric resolution explicit. We therefore first define the semantic and geometric traces used by the model, then describe how anchor tokens dynamically couple them.

Semantic Representation. Language specifies a 3D object through cues at different abstraction levels, including category and global shape, part structure and proportions, and surface appearance. To expose this structure to the model, we organize semantic tokens into a semantic trace with three aspects. The Global aspect captures the object’s category, overall silhouette, and dominant orientation, the Structure aspect describes mass distribution, proportions, and part composition, and the Appearance aspect captures material, color, texture, and local detail. This trace acts as an interpretable, language-level scaffold that anchors the subsequent geometric reasoning.

Geometric Representation. We represent each 3D object as a multiscale octree constructed from a canonicalized \(128^{3}\) voxel grid. Starting from the unit cube, space is recursively subdivided to a maximum depth \(S\). To provide a stable global scaffold, we fully populate the first three octree levels: every node at these depths is present regardless of surface occupancy. Beyond this coarse scaffold, the octree becomes sparse, and nodes are added only in regions that intersect surface geometry. As illustrated in 2, within each depth, nodes are serialized in Morton (Z-order), preserving spatial locality in the token sequence [10], [23]. Formally, each octree node \(v\) is represented by a structural bit \(o_v \in \{0,1\}\) indicating whether \(v\) is subdivided, and a content token \(\mathbf{g}_v\) encoding its local geometry. Structure bits are used to reconstruct the octree topology during decoding. We encode node geometry using scale-specific vector quantization. Inspired by previous octree-based works [23], [24], for each node \(v\) at scale \(s\), we extract a feature vector \(\mathbf{f}_v \in \mathbb{R}^{C}\) and quantize it against a scale-specific codebook \(\mathcal{C}^{(s)}\). The discrete code index and corresponding codebook embedding are \[\setlength{\abovedisplayskip}{3pt} \setlength{\belowdisplayskip}{3pt} \setlength{\abovedisplayshortskip}{2pt} \setlength{\belowdisplayshortskip}{2pt} k_v = \arg\min_c \|\mathbf{f}_v - \mathbf{c}\|_2, \qquad \mathbf{g}_v = \mathcal{C}^{(s)}[k_v].\]

Using separate codebooks per scale allows each vocabulary to specialize in geometric primitives at its resolution. However, quantization collapses geometrically similar regions onto the same codebook entry, erasing their spatial identity. We therefore augment each content token with a learned positional embedding \(\mathbf{E}_{\text{pos}}^{(s)}[v]\) indexed by the node’s Morton position at scale \(s\), and a deterministic scale tag \(\mathbf{s}^{\text{det}}_s\). The scale tag is a fixed, non-trainable vector appended to each token, so a token’s scale is recoverable from its embedding at initialization. This explicit scale signal is also needed by the alignment module (3.2). The augmented content token is \(\tilde{\mathbf{g}}_v\!=\![\,\mathbf{g}_v + \mathbf{E}_{\text{pos}}^{(s)}[v]\,;\,\mathbf{s}^{\text{det}}_s\,].\) We denote by \(\mathbf{G}^{s}\) the Morton-ordered sequence of augmented structural-content pairs \((o_v,\tilde{\mathbf{g}}_v)\) at scale \(s\), and define the full geometric trace as \(\mathcal{G}_{\text{geo}}\!=\![\mathbf{G}^{1}; \mathbf{G}^{2}; \dots; \mathbf{G}^{S}].\) At transformer block \(i\), we write \(\mathbf{G}^{i,s}\) for the hidden states corresponding to the geometric tokens at scale \(s\), initialized from \(\mathbf{G}^{s}\) at the input layer. While the semantic trace provides the conceptual plan in language space, this representation organizes the geometric trace into a natural coarse-to-fine hierarchy with an explicit scale signal on every token.

3.2 Anchor Tokens↩︎

Dense interaction between all semantic tokens and all 3D tokens is costly and often unnecessary, as many words provide global or contextual constraints, while only a subset requires precise geometric grounding. To improve reasoning between language and 3D, we therefore introduce Anchor Tokens, a sparse semantic-geometric interface that creates cross-modal interaction only where it is useful. An anchor is a block-local fusion unit formed from a selected semantic token and the geometric evidence it retrieves from a routed 3D scale. Unlike fixed alignments or dense text–3D attention, anchors are constructed dynamically for each input and transformer block, allowing the model to allocate cross-modal capacity to the semantic cues that most require geometric evidence.

Anchor construction. At block \(i\), the router introduced in 3.3 emits anchor-routing signals \((\beta_m^i, \boldsymbol{\pi}_m^i, \alpha_m^i)\) for each semantic token \(\mathbf{t}_m^i\). Here, \(\beta_m^i\in[0,1]\) is an anchor gate that indicates whether the \(m\)-th text token should participate in anchor construction, \(\boldsymbol{\pi}_m^i\in\Delta^{S-1}\) is a distribution over geometric scales, and \(\alpha_m^i = \arg\max_{s\in\{1,\dots,S\}} \pi_{m,s}^i\) is the selected geometric scale. Let \(\mathcal{M}^i=\{m:\beta_m^i\ge\tau_a\}\) denote the selected semantic-token indices at block \(i\). For each selected token \(m\in\mathcal{M}^i\), we gather scale-specific geometric evidence by cross-attention: \[\setlength{\abovedisplayskip}{2pt} \setlength{\belowdisplayskip}{2pt} \setlength{\abovedisplayshortskip}{2pt} \setlength{\belowdisplayshortskip}{2pt} \mathbf{e}_m^{i} = \mathrm{CrossAttn}\!\left(\mathbf{t}_m^{i},\, \mathbf{G}^{i,\alpha_m^i}\right),\] where the text token \(\mathbf{t}_m^{i}\) serves as the query and the 3D tokens at the selected scale \(\alpha_m^i\) serve as keys and values. The anchor token is then formed by fusing the semantic token with the aligned geometric evidence: \(\mathbf{a}_m^{i} = \mathrm{MLP}_{\mathrm{anchor}}\!\left([\mathbf{t}_m^{i};\,\mathbf{e}_m^{i}]\right).\) Each anchor is therefore a compact semantic-geometric interaction unit grounded in both the current text state and a specific level of 3D context. The number of anchor tokens is dynamic: at block \(i\), only the subset of text tokens selected by \(\beta_m^i\) instantiates anchors, so \(|\mathbf{A}^i|\) varies across layers and inputs.

Write-back. The resulting anchor set \(\mathbf{A}^{i} = \{\mathbf{a}_m^i \mid m\in\mathcal{M}^i\}\) acts as a transient block-local cross-modal workspace. To make anchors influence the persistent unified sequence, every token in the main sequence cross-attends to the current anchor set: \[\mathbf{d}_j^{i} = \mathrm{CrossAttn}_{\mathrm{wb}}\!\big(\mathbf{h}_j^{i},\, \mathbf{A}^{i}\big), \qquad j = 1,\dots,N_{\mathrm{uni}},\] where \(\mathbf{h}_j^{i}\) is the \(j\)-th hidden state of the persistent sequence entering block \(i\). This lets both text and 3D tokens absorb the fused semantic-geometric signal. A learned per-token gate controls the injection magnitude, preventing anchors from dominating early in training: \[\gamma_j^{i} = \sigma\!\big(\mathrm{Linear}([\mathbf{h}_j^{i};\, \mathbf{d}_j^{i}])\big), \qquad \mathbf{h}_j^{i,\text{+a}} = \mathbf{h}_j^{i} + \gamma_j^{i} \odot \mathbf{d}_j^{i}.\] The gate bias is initialized to \(-2\) so that \(\gamma_j^{i}\!\approx\!0\) at the start of training, letting anchors gradually earn influence. The anchor-augmented sequence \(\mathbf{H}^{i,\text{+a}}\) then replaces \(\mathbf{H}^{i}\) as the input to block \(i\)’s self-attention and feed-forward sub-layers (see 3.3). After the block update, anchors are discarded, and only the persistent sequence continues to the next layer. If no anchors are selected, the write-back is skipped and \(\mathbf{H}^{i,\mathrm{+a}}=\mathbf{H}^{i}\). Since \(|\mathbf{A}^i|\!\ll\!N_{\mathrm{uni}}\), the write-back cross-attention adds only \(O(N_{\mathrm{uni}}\!\cdot\!|\mathbf{A}^i|)\) cost per block, which is negligible compared to the main self-attention.

3.3 Dynamic Routing and Elastic Reasoning↩︎

Existing unified 3D models [1]–[3], [25] typically use a fixed transformer computation pattern and rely on implicit token mixing for cross-modal exchange. This treats all blocks as equally necessary and leaves semantic-geometric grounding to emerge without explicitly deciding which language tokens should query which geometric scale. In practice, both assumptions are limiting: input difficulty varies across examples, and only a subset of semantic tokens requires precise geometric grounding [12], [26]. We therefore introduce a lightweight router that makes reasoning elastic in both computation and grounding. Inspired by dynamic transformer designs [13], [14], [27], the router adaptively controls computation by deciding block execution and MLP width. More importantly, it also controls the cross-modal communication pattern by selecting which semantic tokens instantiate anchor tokens and at which geometric scale each should ground.

Router Architecture. Given a unified transformer backbone with \(n\) blocks \(\{\mathbf{B}^{i}\}_{i=1}^{n}\), we attach a lightweight three-headed router \(\mathbf{R}^{i}\) to each block. Let \(\mathbf{H}^{i}\in\mathbb{R}^{N_{\mathrm{uni}}\times D}\) denote the hidden states of the persistent unified sequence entering block \(i\), where \(N_{\mathrm{uni}}\) is the total number of semantic and geometric tokens, and \(D\) is the hidden dimension. Anchor tokens are auxiliary block-local units and are not included in this persistent sequence length. The router first projects each token into a low-dimensional routing space \(\mathbf{r}_l^{i} = \mathbf{h}_l^{i}\mathbf{W}_r, \quad l = 1,\dots,N_{\mathrm{uni}},\) where \(\mathbf{W}_r \in \mathbb{R}^{D\times d_r}\) and \(\mathbf{r}_l^i\in\mathbb{R}^{d_r}\). These per-token routing features are mean-pooled to obtain a block-level context \(\setlength{\abovedisplayskip}{3pt} \setlength{\belowdisplayskip}{3pt} \setlength{\abovedisplayshortskip}{2pt} \setlength{\belowdisplayshortskip}{2pt} \mathbf{b}^{i} = \frac{1}{N_{\mathrm{uni}}}\sum_{l=1}^{N_{\mathrm{uni}}}\mathbf{r}_l^{i}.\) The first two router heads use \(\mathbf{b}^i\) for block-level computation decisions: (1) a scalar block-gating logit \(\ell^{i}=\mathbf{b}^{i}\mathbf{w}_{\ell}\), which determines whether block \(i\) is executed, and (2) a width logit vector \(\mathbf{u}^{i}=\mathbf{b}^{i}\mathbf{W}_{u}\), which determines the MLP width used by the block. The third head operates at the token level over the semantic-token subset. For each semantic token \(\mathbf{t}_m^i\), it uses the token routing feature \(\mathbf{r}_m^i\) together with the block context \(\mathbf{b}^i\) to predict anchor-routing signals \((\beta_m^i,\boldsymbol{\pi}_m^i)\), where \(\beta_m^i\) gates whether semantic token \(m\) instantiates an anchor, and \(\boldsymbol{\pi}_m^i\) is a distribution over geometric scales.

Adaptive Block Skipping. From the block-gating logit, we compute an execution probability \[p^{i} = \sigma(\ell^{i}) \in [0,1].\] Block \(\mathbf{B}^i\) is executed when \(p^i\ge\tau\) and skipped otherwise; we use \(\tau=0.5\) throughout. When a block is skipped, anchor construction and write-back are also skipped, so \(\mathbf{H}^{i,+a}=\mathbf{H}^{i}\). Each block residual update is modulated by gate \(\delta^i=\mathbf{1}[p^i\ge\tau]\), yielding \[\mathbf{H}^{i+1} = \mathbf{H}^{i,+a} + \delta^{i} \big( \mathbf{B}^{i}(\mathbf{H}^{i,+a})-\mathbf{H}^{i,+a} \big).\] Because \(\delta^{i}\) is discrete, we use a Straight-Through Estimator[28] during training and replace \(\delta^{i}\) with \(\mathbf{1}[p^{i}\ge\tau] + p^{i} - \mathrm{sg}(p^{i})\) in the backward pass, where \(\mathrm{sg}(\cdot)\) denotes stop-gradient.

Adaptive MLP Width. Skipping a block is a coarse decision. For executed blocks, the router additionally selects the feed-forward capacity. Let \(H\) be the full hidden width of the MLP. We discretize the width choice into four levels \(\mathcal{W} = \{\tfrac{1}{4},\tfrac{1}{2},\tfrac{3}{4},1\}\), so the router can trade capacity for compute in meaningful increments. From the width logits \(\mathbf{u}^{i}\), we obtain a distribution over levels and select the most probable one, i.e., \(\mathbf{q}^{i} = \mathrm{softmax}(\mathbf{u}^{i})\), \(\hat{w}^{i} = \mathcal{W}\big[\arg\max_{j}\, q^{i}_{j}\big].\) Such a hard weight slice would block gradients from flowing to the unused columns during training. We therefore apply a binary channel mask \(\mathbf{m}(\hat{w}^{i})\!\in\!\{0,1\}^{H}\) that keeps the first \(\hat{w}^{i}\!\cdot\!H\) intermediate channels: \[\mathrm{MLP}_{\text{adapt}}(\mathbf{z}) = \Big(\sigma(\mathbf{z}\mathbf{W}_{1})\odot \mathbf{m}(\hat{w}^{i})\Big)\,\mathbf{W}_{2}.\] This preserves a shared parameterization across all width choices during training, while producing the same forward computation as a narrowed MLP.

Anchor Routing Head. The anchor-routing head determines the structure of cross-modal grounding. Unlike the block-gating and width heads, which make block-level decisions from \(\mathbf{b}^i\), anchor routing is token-specific. For each semantic token \(\mathbf{t}_m^i\), we concatenate its routing feature with the block context: \(\mathbf{z}_m^i = [\mathbf{r}_m^i;\mathbf{b}^i],\) and predict an anchor gate and a soft distribution over 3D scales: \[\beta_m^{i}=\sigma(\mathbf{z}_m^{i}\mathbf{W}_{\beta}), \qquad \boldsymbol{\pi}_m^{i}=\mathrm{softmax}(\mathbf{z}_m^{i}\mathbf{W}_{\pi}).\] Here \(\beta_m^i\in[0,1]\) controls whether token \(m\) instantiates an anchor, while \(\boldsymbol{\pi}_m^i=[\pi_{m,1}^i,\dots,\pi_{m,S}^i]\) defines its preference over the \(S\) geometric scales. The hard scale assignment is \(\alpha_m^i\!=\!\arg\max_{s\in\{1,\dots,S\}}\pi_{m,s}^{i}.\) A semantic token is selected when \(\beta_m^i\ge\tau_a\), where \(\tau_a=0.5\) in all experiments. As with block skipping, \(\arg\max\) is non-differentiable; we use a straight-through estimator so that gradients flow back through \(\boldsymbol{\pi}_m^{i}\) during training.

For each selected semantic token, the anchor module in 3.2 cross-attends to 3D tokens at scale \(\alpha_m^i\), retrieves scale-specific geometric evidence, and fuses this evidence with the semantic state to form an anchor. By first selecting a geometric scale and then attending within that scale, this process induces a coarse-to-fine search, allowing each text token to quickly converge on its most relevant geometric correspondence.

Training and Inference. Training proceeds in two stages. We first train the scale-aware octree VQ-VAE on 3D data to obtain discrete multiscale geometry tokens. We then freeze the tokenizer and train the unified autoregressive model over interleaved semantic tokens, structural bits, and 3D content codes, together with auxiliary losses that shape the router’s compute and grounding decisions. Additional details, including VQ-VAE training, unified autoregressive training, and router regularization, are provided in Appendix 7.

13pt

4 Experiments↩︎

We evaluate ELSA3D across four capabilities: image-conditioned 3D generation, text-conditioned 3D generation, 3D object captioning, and general conversational reasoning. For training, we use the publicly available 3D-Alpaca dataset [1] and additional 3D assets from Trellis-500K [29], curated from ObjaverseXL [30], ABO [31], 3DFUTURE [32], and HSSD [33]. We also include UltraChat [34] to preserve general language capability. For evaluation, we use Toys4K assets [35] and 200 in-the-wild images, with no overlap with the training set. Full implementation details are provided in Appendix 8.

Baselines. For 3D generation, we compare with representative open-source methods, including Direct3D [36], InstantMesh [37], SparseFlex [38], Shape-E [39], LN3Diff [40], XCube [41], SAR3D [6], 3DTopia-XL [42], GaussianAnything [43], Trellis [29], ShapeLLM-Omni [1], and CoRe3D [2]. For 3D object captioning, we compare with general VLMs LLaVA-13B [44] and Qwen2.5-VL-7B [45], 3D-specialized understanding models 3D-LLM [46], LEO [47], and PointLLM-13B [48], and unified 3D understanding-generation models ShapeLLM-Omni [1] and CoRe3D [2].

Evaluation Metrics and Benchmarks. For 3D generation, we report CLIP similarity [49] to measure consistency between the generated results and the input, FD [50] and KD [51] with Inception-v3 [52] as the feature extractor to assess overall generation quality, PSNR and LPIPS to evaluate visual reconstruction quality, and COV and MMD [53] to measure distribution-level fidelity. We also employ Q-Align [54], a learned visual quality score that aligns with human perceptual judgments of generated 3D renderings. For 3D captioning, we follow PointLLM [48] and report BLEU-1 [55], ROUGE-L [56], METEOR [57], Sentence-BERT [58] and SimCSE [59]. For general language ability, we evaluate on MMLU [60], PIQA [61], GSM8K [62], and SIQA [63].

Image-to-3D generation. 1 shows that ELSA3D achieves the best result on five of seven metrics and the second-best result on the remaining two, indicating consistently strong performance. Compared with the strongest unified baseline, our method improves by +2.74 CLIP, \(-\)​2.03 FD, and \(-\)​0.54 MMD, indicating that elastic semantic anchoring improves cross-modal grounding without sacrificing geometric quality. Although SparseFlex remains slightly stronger on PSNR and LPIPS, it is a reconstruction-specialized image-conditioned method, while ELSA3D maintains unified image, text, and language capabilities.

15pt

3D object understanding. A key promise of unified 3D foundation models is that generative training also strengthens understanding. 2 shows that ELSA3D sets a new state-of-the-art across all five metrics, improving over CoRe3D by +2.20 BLEU-1, +1.70 ROUGE-L, +0.80 METEOR, +3.56 Sentence-BERT, and +1.32 SimCSE. The larger gains on semantic metrics suggest that routing language cues to scale-specific geometric evidence helps the model produce captions that are more semantically aligned with the underlying 3D object.

r0.35 4.0pt

Text-to-3D generation. 3 shows ELSA3D leading all metrics, improving over CoRe3D by +1.35 CLIP, -4.05 FD, -0.01 KD, and +0.14 Q-Align. The large improvement in CLIP over Trellis (+9.58) validates that anchor tokens succeed at binding linguistic intent to fine geometric structure. Results also show that unified semantic-geometric reasoning is most valuable when the input is underspecified, and the model must infer object structure from language rather than copy visible image evidence.

Reasoning-based 3D generation. We further evaluate challenging indirect descriptive prompts that do not explicitly name the target object but instead provide a description through indirect cues. 3 shows that ELSA3D infers the intended object identity and generates the corresponding structure and appearance, including a nestable wooden female figure, a moon-associated baked pastry, a French metal tower, and a folded paper symbol of peace with recognizable silhouettes. In contrast, CoRe3D often captures only a coarse or generic shape of the referenced concept. These examples illustrate the value of semantic anchoring under language ambiguity, where the model must recover the latent concept from indirect cues and bind it to geometric evidence at the appropriate scale.

Qualitative Examples. 4 compares image-to-3D and text-to-3D generation outputs on visually complex inputs spanning toys, food, vehicles, and stylized characters, showing that ELSA3D consistently produces shapes that retain both the global shape and local appearance cues of the input images, including thin structures, part layout, and distinctive textures. For text-conditioned generation, ELSA3D better satisfies category-level intent and fine-grained prompt constraints, such as object parts, support structures (e.g., basket holds fruit), material cues, and surface appearance (e.g., the coffee cup is clean and unbranded as specified).

4.1 Ablations↩︎

4.2pt

Anchor tokens. We first ablate the mechanism used to connect semantic and geometric tokens. 4 compares four designs: removing anchors entirely (No Anchors), replacing anchors with full bidirectional cross-attention between all text and 3D tokens (Direct Cross-Attn), disabling routing and instantiating dense anchors for every text token (Dense Anchors), and the proposed elastic anchors. Removing anchors degrades every task, confirming that implicit self-attention over a flat sequence is insufficient for reliable text–3D alignment. Direct cross-attention recovers quality but requires 1081G FLOPs and 29.8s latency, while dense anchors still cost 865G FLOPs and 23.6s. ELSA3D obtains the best quality across generation and captioning while using only 632G FLOPs and 17.2s latency, showing that sparse anchor selection is not just cheaper but also less noisy than dense fusion.

15pt

Scale-aware anchor routing. We next test whether anchors must be routed to a learned geometric scale. 5 shows that random scale assignment is the weakest variant, indicating that scale diversity alone is not enough. Coarse-only and fine-only routing also underperform because they force all semantic cues into a single resolution: category-level information benefits from coarse geometry, while part and appearance cues often require finer scales. All-scale attention is closer in quality but increases latency from 17.2s to 22.4s because every anchor attends to all octree scales. The learned router achieves the best quality with near coarse-only latency, validating scale-aware anchoring as a useful inductive bias for coarse-to-fine 3D grounding.

Elastic computation. The elastic depth and width decisions are essential for the quality–efficiency trade-off. The full-compute model in Appendix Table 9 slightly improves some generation metrics but costs 1284 GFLOPs and 34.6s latency. ELSA3D retains near-full-compute quality while reducing GFLOPs to 632 and latency to 17.2s, a roughly 2\(\times\) reduction. Depth-only and width-only variants are less effective, indicating that efficient unified 3D modeling benefits from adapting both which blocks execute and how much capacity each active block uses.

5 Conclusion↩︎

We propose ELSA3D, a unified 3D understanding-and-generation model that makes language–geometry interaction sparse, adaptive, and scale-aware. ELSA3D introduces semantic anchor tokens to route selected language cues to the most relevant level of a multiscale 3D hierarchy and write the fused evidence back into the shared representation. Across image-to-3D, text-to-3D, reasoning-based generation, and 3D captioning, ELSA3D consistently outperforms strong 3D and unified baselines.

6 Related Work↩︎

Hierarchical 3D tokenization and generation. Recent 3D generative models increasingly replace flat dense grids with compact structured representations that expose sparsity, hierarchy, or progressive detail [4], [5], [10], [64], [65]. Orthogonal to native 3D tokenizers, optimization-based text- and image-to-3D methods distill pretrained 2D diffusion priors into 3D representations [66]–[77]. These approaches provide strong specialized baselines for text- or image-conditioned 3D generation, but typically require per-instance optimization and do not form a single autoregressive backbone for both 3D understanding and generation. Single-image 3D generation has also been advanced by feed-forward reconstruction and multiview-diffusion pipelines [36], [42], [78]–[89]. Octree-based models exploit spatial subdivision to scale 3D perception, reconstruction, and generation [23], [24], [65], [90], [91], with extensions to textured assets [92]–[94]. Other approaches pursue complementary compact representations, including set-based neural-field latents [95], multi-resolution latent diffusion [64], mesh autoregressive detailization [96], coarse-to-fine visual autoregression [10], semantic token ordering [4], and continuous levels of detail [5]. These works show that structured tokenization is crucial for scalable 3D generation. However, their hierarchy primarily defines how geometry is encoded, decoded, or progressively refined. In contrast, ELSA3D uses hierarchy as a cross-modal interface: language-derived semantic anchors are routed to explicit geometric scales, enabling sparse language–geometry binding across global object semantics, part structure, and local appearance.

3D-language models and unified 3D understanding-generation. Recent 3D multimodal models connect language models with 3D representations for object- or scene-level understanding, including 3D question answering, captioning, grounding, dialogue, and scene-level reasoning [46], [48], [97]–[101]. More recent unified systems extend this direction toward joint 3D understanding and generation [1]–[3], [6], [102], [103]. These methods demonstrate the promise of language-driven 3D reasoning, but most rely on global point clouds, scenes, or latent representations rather than explicitly routing semantic tokens to different geometric resolutions. ELSA3D, on the other hand, exposes the hierarchical structure of 3D geometry to the language model and grounds selected semantic tokens through scale-specific anchor units, enabling global, part-level, and local evidence to be fused within a unified autoregressive representation.

Adaptive Cross-Modal Routing. Another line of work improves transformer efficiency by dynamically selecting, merging, or reorganizing tokens [11]–[19], [27]. Token-level methods such as ToMe [11] and EViT [12] reduce redundant visual computation, while multimodal extensions adapt token selection, fusion, pruning, or cache compression across modalities [16]–[19]. Query-based connectors such as BLIP-2 and Flamingo bridge visual features and language models through learned query or resampler tokens [104], [105], and related alignment methods learn to group or associate tokens across modalities [20]–[22]. Dynamic and elastic transformer variants further adapt computation across inputs, layers, timesteps, or contexts [13]–[15], [27]. These methods motivate efficient token processing and query-based cross-modal interfaces, but they do not decide which language tokens should interact with which geometric scale. ELSA3D instead makes adaptive computation part of the grounding mechanism: selected semantic tokens instantiate anchor units that query scale-specific 3D features and write fused language–geometry evidence back into the unified representation.

7 Training and Inference Details↩︎

Stage 1: Octree-based 3D VQ-VAE Training. The octree-based VQ-VAE consists of an encoder, scale-specific codebooks, and a decoder. It is trained on 3D data alone, without text supervision. The encoder maps each \(128^3\) voxel grid to the multiscale octree representation described in 3.1, whereas the decoder reconstructs the voxel occupancy from the predicted octree topology and quantized content embeddings. Training minimizes a reconstruction loss together with the standard VQ commitment and codebook losses [106]: \[\setlength{\abovedisplayskip}{3pt} \setlength{\belowdisplayskip}{3pt} \setlength{\abovedisplayshortskip}{2pt} \setlength{\belowdisplayshortskip}{2pt} \mathcal{L}_{\mathrm{VQ}} = \mathcal{L}_{\mathrm{recon}} + \lambda_{\mathrm{commit}} \sum_{v}\|\mathbf{f}_v - \mathrm{sg}(\mathbf{c}_v)\|_2^2 + \lambda_{\mathrm{code}} \sum_{v}\|\mathrm{sg}(\mathbf{f}_v) - \mathbf{c}_v\|_2^2,\] where \(\mathbf{f}_v\) is the encoder feature at node \(v\), \(\mathbf{c}_v\) is its assigned codebook embedding, and \(\mathrm{sg}(\cdot)\) denotes stop-gradient. After this stage, the VQ-VAE encoder and codebooks are frozen.

At inference time for 3D generation, the structural bits generated by the unified model are consumed deterministically to rebuild the octree topology. For each existing node, the predicted content code \(k_v\) is mapped to its codebook embedding \(\mathbf{g}_v=\mathcal{C}^{(s)}[k_v]\) and placed at the corresponding octree location. The VQ-VAE decoder then reconstructs a dense \(128^3\) voxel grid, from which we obtain a textured mesh using a texture transformer [29] and mesh decoder.

Stage 2: Unified Autoregressive Training. We train the unified model with autoregressive next-token prediction over a combined vocabulary spanning text tokens, structural-bit tokens, and 3D tokens. Following ShapeLLM-Omni [1], we extend the base model’s vocabulary with the VQ-VAE codebook entries and two structural-bit tokens. Let \(\mathbf{z}=(z_1,\dots,z_T)\) denote the resulting training sequence. The autoregressive objective is \[\setlength{\abovedisplayskip}{3pt} \setlength{\belowdisplayskip}{3pt} \setlength{\abovedisplayshortskip}{2pt} \setlength{\belowdisplayshortskip}{2pt} \mathcal{L}_{\mathrm{AR}} = -\sum_{t=1}^{T} \log p_\theta(z_t \mid z_{<t}),\] where \(z_t\) may be a text token, a structural bit \(o_v\in\{0,1\}\), or a discrete 3D content code \(k_v\). The corresponding codebook embedding \(\mathbf{g}_v=\mathcal{C}^{(s)}[k_v]\) is then augmented with positional and scale signals before entering the transformer.

Depth and width budgets. Without explicit pressure, the router learns to execute every block at full width. We introduce two budget losses. Let \(\bar{p}=\tfrac{1}{n}\sum_{i=1}^{n} p^{i}\) denote the mean execution probability and let \(r^i\!=\!\sum_{j=1}^{4} q_j^i \mathcal{W}_j\) denote the expected width of block \(i\), averaged only over non-skipped blocks: \[\bar{r} = \frac{\sum_{i=1}^{n} \mathbf{1}[p^{i}\ge\tau]\, r^{i}}{\sum_{i=1}^{n}\mathbf{1}[p^{i}\ge\tau]}.\] Given target compute budgets \(\rho_d,\rho_w\in(0,1)\), we penalize deviations between the router’s realized average depth and width usage and their desired budgets: \[\mathcal{L}_{\text{depth}} = (\bar{p} - \rho_d)^2, \qquad \mathcal{L}_{\text{width}} = (\bar{r} - \rho_w)^2.\] These terms prevent the router from collapsing to full computation while allowing the realized computation to remain input-dependent.

Anchor sparsity and scale diversity. To keep anchor construction compact, we \(\ell_1\)-regularize the anchor gates, whereas to prevent scale routing from collapsing to a single resolution, we minimize the negative entropy of the scale distribution: \[\mathcal{L}_{\mathrm{sparse}} = \frac{1}{nM}\sum_{i=1}^{n}\sum_{m=1}^{M} |\beta_m^i|, \qquad \mathcal{L}_{\mathrm{scale}} = \frac{1}{nM}\sum_{i=1}^{n}\sum_{m=1}^{M} \sum_{s=1}^{S} \pi_{m,s}^{i}\log \pi_{m,s}^{i}.\]

All auxiliary losses are computed from quantities already produced by the router and add negligible overhead. Gradient flow through the three discrete router decisions (block gate, width selection, scale choice) is handled by straight-through estimators [28] in all cases. Crucially, the write-back mechanism (3.2) allows \(\mathcal{L}_{\text{AR}}\) to back-propagate through anchor construction into the router’s anchor-selection and scale-selection heads, so these heads receive task-level supervision in addition to the auxiliary budget losses. The final training objective is \[\setlength{\abovedisplayskip}{3pt} \setlength{\belowdisplayskip}{2pt} \setlength{\abovedisplayshortskip}{2pt} \setlength{\belowdisplayshortskip}{2pt} \mathcal{L} = \mathcal{L}_{\mathrm{AR}} + \lambda_d \mathcal{L}_{\mathrm{depth}} + \lambda_w \mathcal{L}_{\mathrm{width}} + \lambda_s \mathcal{L}_{\mathrm{sparse}} + \lambda_c \mathcal{L}_{\mathrm{scale}}.\]

Inference. At inference time, blocks with \(p^i<\tau\) are skipped entirely, and no anchors or write-back are computed for those blocks. For executed blocks, the selected MLP width is materialized by weight slicing: \[\mathrm{MLP}_{\hat{w}^{i}}(\mathbf{z}) = \sigma\!\left( \mathbf{z}\mathbf{W}_{1}[:,:\hat{w}^{i}H] \right) \mathbf{W}_{2}[:\hat{w}^{i}H,:],\] and anchors are instantiated only for semantic tokens with \(\beta_m^i\ge\tau_a\), drawing geometric evidence from scale \(\alpha_m^i\!=\!\arg\max_{s}\pi_{m,s}^i.\) Because block execution, width, anchor selection, and scale assignment are all input-dependent, the realized computation and cross-modal interaction adapt to each sample.

8 Implementation Details.↩︎

We implement the 3D tokenizer as an OctGPT-style octree VQ-VAE [24] with maximum depth \(S=7\), 8192-entry scale-specific codebooks, and embedding dimension \(C=256\). It is trained for 100 epochs with AdamW, learning rate \(1\times10^{-4}\), batch size 32, on 8 NVIDIA A100 GPUs, after which the encoder and codebooks are frozen. The unified model is initialized from Qwen-2.5-VL-Instruct-7B (\(n=28\), \(D=3584\)) [45], following ShapeLLM-Omni [1]. We extend the vocabulary with \(8192\times S\) 3D tokens and two structural-bit tokens, use a scale tag of dimension 32, and keep the visual encoder frozen. The router uses \(d_r=128\), the anchor MLP is a two-layer GELU network with hidden size 512, and anchor write-back uses 4-head cross-attention. We set \(\rho_d=0.7\), \(\rho_w=0.75\), \(\lambda_d=1.0\), \(\lambda_w=1.0\), \(\lambda_s=0.01\), and \(\lambda_c=0.01\). For training data, we follow the 3D-Alpaca construction pipeline of ShapeLLM-Omni [1], which covers text-to-3D, image-to-3D, 3D captioning, and 3D editing tasks. We train on the publicly available 3D-Alpaca dataset [1] and supplement it with additional 3D assets from Trellis-500K [29]. We also include UltraChat [34] to preserve general language capabilities. The unified model is trained for 200k steps with AdamW, learning rate decayed from \(5\times10^{-5}\) to \(5\times10^{-6}\), per-GPU batch size 2 with 4-step gradient accumulation, on 8 NVIDIA A100 GPUs. At inference, we use top-\(k=8192\), top-\(p=0.7\), temperature \(0.7\), and set both routing thresholds to 0.5.

9 Additional Experiments and Ablations↩︎

General conversational capabilities. Adding 3D generation and understanding should not come at the cost of the model’s original reasoning ability. For language and multimodal reasoning, we compare against general-purpose VLMs, including Qwen2.5-VL-7B [45] and LLaMA3.2-Vision-11B [107], and 3D-focused multimodal models, including LLaMA-Mesh-8B [102], CoRe3D [2], and ShapeLLM-Omni-7B [1]. 6 shows that ELSA3D remains competitive with general-purpose VLMs while improving over 3D-focused baselines on MMLU, GSM8K, and SIQA. It achieves the best score on MMLU, GSM8K, and SIQA, and the second-best score on PIQA behind Qwen2.5-VL-7B. These results suggest that UltraChat mixing and elastic routing preserve broad language competence while adding 3D-specific capabilities.

Semantic Trace Decomposition. We test the contribution of decomposing the semantic trace into Global (G), Structure (S), and Appearance (A) aspects (Table 7). Removing the trace entirely (No Trace) is the weakest configuration (Text-to-3D CLIP 37.18). Replacing the structured decomposition with a single free-form reasoning paragraph of comparable length (Monolithic) recovers part of the gain (CLIP 38.32) but still trails the structured variant, indicating that the gain comes from the decomposition itself rather than from added tokens. Among subset combinations, the full G+S+A trace is best across all metrics. The pattern matches the scale-routing inductive bias: global semantics, structural semantics, and appearance semantics naturally bind to corresponding geometric scales, respectively, and removing any aspect deprives anchors at the corresponding scale of clean input.

10pt

Number of Octree Scales. We sweep the maximum octree depth \(S \in \{4, 5, 6, 7, 8\}\), varying the finest voxel resolution from \(16^3\) to \(256^3\). Reconstruction quality improves monotonically with depth (IoU 0.672 \(\to\) 0.902, CD 0.049 \(\to\) 0.011), but downstream generation saturates at \(S{=}7\): pushing to \(S{=}8\) marginally regresses Text-to-3D quality while inflating latency by 50% (25.8s vs.17.2s). The gap between reconstruction and generation curves reflects a known tension in autoregressive 3D modeling [24]: deeper octrees produce sequences too long for the unified backbone to model coherently, so tokenizer fidelity outruns the AR model’s ability to exploit it. We adopt \(S{=}7\) as the best quality-cost trade-off. Training with \(S \geq 9\) also becomes prohibitively expensive.

12pt

7pt

10pt

Elastic Computation Designs. We evaluate whether dynamic block skipping and adaptive MLP width preserve quality while reducing compute (Table 9). Full-Compute runs every block at full width and serves as an upper bound (Text-to-3D CLIP 39.18, Image-to-3D FD 8.92, but at 1284 GFLOPs and 34.6s latency). Depth-only and width-only elastic variants reduce cost partially, but each loses noticeable quality (Text-to-3D CLIP drops to 38.52 and 38.21, respectively). ELSA3D combines both axes and recovers near-upper-bound quality (CLIP 39.01, FD 16.50) while halving FLOPs (632G) and cutting latency by more than 2\(\times\) (17.2s). This confirms the effectiveness of our elastic compute design.

Scale-Specific Codebooks. We evaluate whether each octree scale requires its own VQ codebook by replacing the scale-specific codebooks \(\{\mathcal{C}^{(s)}\}_{s=1}^{S}\) with a single shared codebook of equivalent total capacity. Sharing degrades the tokenizer itself (IoU drops from 0.864 to 0.832, CD rises from 0.013 to 0.017), and the loss propagates cleanly to every downstream task. Text-to-3D CLIP falls by 0.74, and FD rises by 1.93, while Image-to-3D CLIP falls by 1.25 and FD rises by 2.51. These results indicate that scale-specific codebooks allow each vocabulary to specialize to the geometric primitives of its resolution, providing a cleaner basis for the scale-routing head to make coarse-to-fine anchor decisions.

10 Additional Qualitative Results.↩︎

Qualitative 3D Captioning. Figure 5 provides qualitative examples of 3D object captioning. Given the same input shape, ShapeLLM-Omni produces short category-level descriptions (“An astronaut cat in a spacesuit”). CoRe3D adds attribute detail but stops at coarse description, whereas ELSA3D produces more compositional captions that describe object identity, part structure, and local appearance, such as clustered stone masses, attached side wings, layered pitched roofs, orange-white fur, and flat studded surfaces. These examples demonstrate that scale-aware semantic anchoring helps the model ground language in both global shape and fine geometric detail.

Additional image-to-3D comparisons. Figure 6 provides the full image-to-3D qualitative comparison. The examples span diverse object categories with different levels of geometric complexity, including smooth objects, thin structures, articulated parts, stylized shapes, and distinctive surface patterns. Across these cases, ELSA3D more consistently preserves global silhouette, part layout, and local appearance cues from the input image.

Additional text-to-3D comparisons. Figure 7 provides the full text-to-3D qualitative comparison. The prompts cover category-level generation, part-level structure, support relations, material cues, and appearance constraints. ELSA3D more faithfully follows both the intended object category and fine-grained prompt details, producing objects with more coherent part layouts and fewer generic or underspecified shapes. These examples show that scale-aware grounding is especially useful when the input is language only, and the model must infer geometry from semantic constraints.

Additional in-the-wild image-to-3D results. 8 provides additional in-the-wild image-to-3D generation results. These examples cover diverse object categories and visual appearances, including cases with complex surface details. The results further show that ELSA3D can recover coherent 3D structure from a single image while preserving salient shape cues and appearance patterns. This demonstrates the robustness of our unified 3D representation beyond the main quantitative benchmark.

11 Limitations↩︎

This work focuses on object-level unified 3D understanding and generation. Extending ELSA3D to large multi-object scenes, dynamic 3D content, and interactive editing remains future work. In addition, our octree tokenizer uses a fixed maximum depth, which balances quality and efficiency but may not capture all extremely fine surface details. Finally, like other generative 3D models, ELSA3D may produce plausible but incorrect geometry when prompts are ambiguous or when input images contain occluded object parts.

12 Broader Impacts↩︎

Unified 3D generation and understanding can support creative design, simulation, education, accessibility, robotics, and scientific visualization by making 3D content easier to generate and reason about from language or images. At the same time, such models may be misused to create deceptive or copyrighted 3D assets, generate unsafe objects, or amplify biases present in web-scale 3D and image datasets. We encourage responsible deployment with dataset documentation, provenance tracking, content filtering, and human review for high-stakes or public-facing applications. Our work focuses on object-level 3D modeling and does not claim reliability for safety-critical physical deployment without additional validation.

References↩︎