1 Introduction↩︎

Multimodal large language models (MLLMs) [1]–[3] have been increasingly adapted to the medical domain [4]–[6] by leveraging large-scale 2D image-text pairs [7], [8]. While these models demonstrate competitive performance on standard 2D tasks [9], [10], clinical practice is fundamentally grounded in 3D modalities (e.g., CT and MRI) [11], [12]. Recent attempts to extend MLLMs to 3D imaging [13]–[15] have primarily focused on representation learning and cross-modal alignment. Consequently, complex, multi-step reasoning over volumetric data remains largely unaddressed [16], [17].

Chain-of-thought (CoT) prompting [18] has proven highly effective in structuring intermediate reasoning steps for complex medical tasks [19]–[21]. However, current medical CoT methodologies are predominantly confined to 2D or text-based scenarios [22]. While recent efforts have explored shared grounded reasoning interfaces to facilitate 2D-to-3D transfer [23], directly generalizing these approaches to native 3D volumes remains non-trivial. Real-world clinical volumetric diagnosis inherently relies on a sequential, slice-by-slice reading workflow to track lesion evolution and integrate partial cross-sectional views into a coherent 3D anatomical understanding. Existing datasets fail to capture this slice-wise cognitive process, frequently reducing 3D volumes to collapsed 2D projections or treating them as unstructured black boxes. This critical absence of spatially-grounded CoT supervision leaves volumetric reasoning models without clinically aligned, interpretable guidance.

To address these challenges, we propose a novel slice-wise data synthesis paradigm that explicitly models the visual cognition of expert radiologists. Instead of relying on holistic volume-level summaries, we employ a dual-agent pipeline to simulate the clinical diagnostic workflow: an Observer agent first extracts fine-grained, per-slice findings from global clinical reports, and a Synthesizer agent then integrates these discrete observations into a structured, spatially-grounded CoT. Crucially, every generated reasoning chain is rigorously constrained to enforce three core clinical principles: sequential spatial tracking, true 3D spatial awareness, and differential exclusion. Driven by this paradigm, we introduce Hounsfield-CoT, a large-scale 3D medical reasoning dataset providing explicit supervision for volumetric understanding. Derived exclusively from the CT-RATE dataset, Hounsfield-CoT comprises 11.2k rigorously curated instances targeting foundational spatial reasoning. As illustrated in Fig. 1, this data-centric approach successfully replaces opaque reasoning shortcuts with highly interpretable clinical logic.

By instruction-tuning a 2D-pretrained MLLM on Hounsfield-CoT, we bridge the semantic gap between 2D pre-training and complex volumetric understanding. At inference time, the adapted model systematically leverages the learned slice-wise cognitive pattern to observe, track, and synthesize findings, yielding significantly stronger and more transparent diagnostic performance.

In summary, our main contributions are three-fold:

1. We propose a novel slice-wise data synthesis paradigm for 3D medical reasoning. Simulating the diagnostic workflow of expert radiologists, we introduce a dual-agent (Observer-Synthesizer) framework. This paradigm explicitly translates global volume-level priors into fine-grained, per-slice observations, rigorously enforcing sequential spatial tracking, true 3D spatial awareness, and differential exclusion.

2. We introduce Hounsfield-CoT, a large-scale, spatially-grounded reasoning dataset. Driven by our synthesis paradigm, we curate 11.2k high-quality 3D reasoning instances. This dataset bridges the critical gap in explicit volumetric supervision, replacing opaque reasoning shortcuts with transparent, step-by-step clinical logic.

3. We establish a cost-effective 2D-to-3D reasoning transfer framework and demonstrate superior diagnostic performance. By instruction-tuning a standard 2D-pretrained MLLM using Hounsfield-CoT, we effectively unlock its latent 3D spatial reasoning capabilities without computationally prohibitive 3D-specific pre-training. Extensive evaluations reveal substantial performance gains alongside highly interpretable diagnostic rationales, successfully mitigating partial-volume artifacts and spatial hallucinations.

2 Related Work↩︎

2.1 Traditional Paradigms in Medical MLLMs↩︎

While Medical MLLMs have achieved remarkable success in 2D imaging, extending these capabilities to 3D volumetric data introduces significant computational and architectural challenges. Current research predominantly diverges into two paradigms: native 3D modeling and 2D-to-3D adaptation. Pioneering explicit 3D models—such as M3D [13] for unified multi-task learning, LLaVA-3D [24] for visual grounding, and Med3DVLM [25] for efficient vision-language alignment—directly process entire volumes to capture holistic anatomical contexts. However, these native approaches demand massive computational resources and operate as opaque “black boxes” lacking diagnostic transparency.

Alternatively, 2.5D feature fusion strategies bypass these bottlenecks by leveraging mature 2D backbones. Med-2E3 [15] aggregates 2D slice embeddings to reduce training costs, while BrainGPT [8] optimizes sequence-based slice processing for neuroimaging. Though efficient, these methods inherently risk losing vital inter-slice continuity. To mitigate the opacity of 3D models and the spatial reasoning gaps of 2.5D methods, recent works have explored CoT reasoning. Methods like MC-CoT [20], MedCoT [26], and the guideline-aligned ClinCoT [19] successfully elicit step-by-step rationales to enhance transparency. Unfortunately, these CoT approaches remain almost entirely confined to 2D modalities or rely on rigid, static templates. Although nascent efforts attempt to bridge this divide via shared grounded reasoning interfaces for 2D-to-3D transfer [23], they do not fully address the complexities of native volumetric logic. Furthermore, as diagnostic reasoning chains extend across 3D space, maintaining dynamic cross-modal coordination becomes imperative to prevent attention collapse and visual-textual misalignment [27]. Consequently, there remains a critical gap in adaptive, transparent reasoning capable of explicitly unrolling diagnostic logic across complex 3D volumes.

2.2 Training Datasets for Medical MLLMs↩︎

Data synthesis has emerged as a critical strategy to overcome the severe scarcity of high-quality medical multi-modal training data. For standard 2D imaging, scalable instruction-following datasets have been successfully constructed through diverse pipelines. For instance, PMC-VQA [28] leverages massive biomedical literature mining to extract extensive figure-caption pairs, while MedTrinity-25M [29] utilizes retrieval-augmented generation to provide multigranular spatial annotations. Furthermore, works like BioMed-VITAL [30] and HuatuoGPT-Vision [31] refine these datasets by aligning model outputs directly with human clinician preferences and real-world diagnostic behaviors.

Transitioning to the 3D domain, pioneering efforts have begun scaling volumetric corpora. M3D [13] constructs extensive 3D VQA pairs for unified multi-task learning, and RadGenome-Chest CT [32] advances the field by linking diagnostic text to specific spatial anatomical regions to generate grounded reports.

Nevertheless, a critical limitation persists across these foundational efforts: existing 3D datasets predominantly exist as flat, end-to-end question-answer pairs or static final reports. This macroscopic format structurally conceals the intermediate, slice-by-slice analytical processes and differential exclusion strategies naturally employed by physicians. Consequently, models trained on such data are prone to shortcut learning rather than genuine spatial understanding. This underscores the urgent need for a novel data synthesis paradigm that explicitly unrolls and embeds dynamic, multi-step reasoning trajectories into 3D clinical datasets.

3 Dataset Construction↩︎

In this section, we detail the construction of the Hounsfield-CoT dataset, specifically engineered to bridge the semantic gap between raw 3D medical volumes and expert-level diagnostic reasoning. To address core challenges in 3D clinical data—namely the opacity of end-to-end models and the absence of explicit, spatially grounded reasoning chains—we implemented a systematic, data-centric curation pipeline (illustrated in Fig. 2). Moving away from static multi-source aggregation, we focused exclusively on the large-scale CT-RATE dataset to construct high-quality, task-specific reasoning trajectories for 3D-RadVQA. Central to our approach is a novel Observer-Synthesizer Dual-Agent Framework, which explicitly mimics the clinical slice-by-slice reading workflow. This systematic approach effectively transforms traditionally opaque visual question-answer pairs into a transparent, clinically interpretable corpus comprising 11.2k strictly curated instances.

3.1 Raw Data Preparation↩︎

The objective of this initial phase is to establish a robust factual anchor from the raw CT-RATE dataset and align it with standardized evaluation formats prior to LLM-based reasoning synthesis.

Unlike conventional 2D VQA datasets where a single image maps to a brief answer, 3D volumetric data contains dense anatomical structures. To construct a reliable factual basis, we extract the corresponding volume-level clinical reports for each scan. These reports, written by expert radiologists, encapsulate the ground-truth pathological findings of the entire 3D volume and serve as an indispensable global prior for our synthesis pipeline.

Simultaneously, we restrict our task formulation to the core competencies evaluated in the 3D-RadVQA benchmark: Anomaly Detection (T1), Image Observation (T2), Medical Computation (T3), and Existence Detection (T4). For each scan, we structure the ground-truth metadata to automatically instantiate corresponding queries in both Open-ended and Multiple-Choice Question (MCQ) formats. This rigorous formulation ensures that the generated dataset directly targets foundational 3D spatial reasoning capabilities.

3.2 Slice-wise CoT Synthesis via Dual-Agent Framework↩︎

Following data preparation, this phase focuses on generating explicit, spatially grounded reasoning trajectories. Previously, vision-language models tended to act as “end-to-end” black boxes, outputting final diagnostic results without transparent reasoning processes. Directly applying standard CoT to 3D images often fails, as volumetric data introduces complex spatial variations (e.g., partial-volume artifacts) that language priors alone cannot resolve.

To overcome this, we draw inspiration from the systematic diagnostic patterns of professional radiologists and propose a Dual-Agent Synthesis Framework powered by a Large Language Model (e.g., GPT-5.4). To ensure the synthesized CoT is a clinically valid deduction rather than a linguistic hallucination, the generation process is meticulously controlled and mandated to manifest three core cognitive elements: (1) Sequential Spatial Tracking: Documenting the appearance, peak manifestation, and fading of the target structure across consecutive Z-axis slices. (2) True 3D Spatial Awareness: Integrating discrete 2D observations to explicitly differentiate genuine 3D anatomical abnormalities from localized partial-volume artifacts. (3) Differential Exclusion: Conducting a comparative analysis against standard morphology to exclude clinically similar competing diagnoses. Concrete qualitative examples of these elements are provided in Fig. 3.

We operationalize this cognitive framework through a cohesive pipeline that explicitly decouples observation from synthesis. During Per-slice Perception, an Observer agent utilizes the volume-level clinical report as a global prior to sequentially examine the scan and extract discrete, localized findings. Subsequently, in the CoT Synthesis phase, a Synthesizer agent integrates these discrete observations into a continuous reasoning chain strictly governed by the three aforementioned clinical elements.

To ensure structural consistency, the generated reasoning trajectories are verified and decoupled, with the reasoning process confined within <think> tags and the final diagnosis in <answer> tags. Ultimately, this streamlined process yields the final Hounsfield-CoT dataset of 11.2k high-quality instances. This dense, highly interpretable cognitive foundation provides a distinct advantage for subsequent instruction tuning, enabling MLLMs to acquire robust volumetric understanding without the prohibitive cost of 3D-specific pre-training. Such a paradigm of utilizing explicitly grounded reasoning trajectories to unlock domain-specific capabilities closely mirrors recent successful dataset innovations in other complex spatial modalities, such as pixel-grounded reasoning for earth observation [33].

3.3 Dataset Statistics and Characteristics↩︎

The finalized Hounsfield-CoT dataset is systematically structured to cover the four core competencies of 3D-RadVQA. As detailed in Table 1, the dataset maintains a balanced format distribution: 51.2% of the data (T1–T3) consists of open-ended generative queries, while the remaining 48.8% (T4) is dedicated to yes-no multiple-choice verification, supporting comprehensive evaluation protocols.

A defining characteristic of our dataset is its dense, highly interpretable reasoning structure. When processing the volumetric scans (typically featuring a mode of 8 slices), the synthesized <think> blocks explicitly unroll complex spatial tracking and differential diagnostic processes. This yields a striking average token length ratio of 92.9:1 between the detailed reasoning chains and the highly concise final <answer> outputs. Such a stark contrast underscores our primary objective: shifting MLLMs from superficial vision-to-text mapping toward verifiable, deep volumetric reasoning, thereby unlocking genuine 3D spatial awareness without the prohibitive cost of explicit 3D pre-training.

4 Model↩︎

Our proposed approach, Hounsfield3D, aims to unlock multi-step spatial reasoning for 3D medical imaging by combining lightweight volumetric processing with our structured CoT supervision. Rather than designing a computationally expensive 3D-native architecture from scratch, we adapt a standard 2D-pretrained multimodal baseline to volumetric tasks, demonstrating the efficacy of our data-centric paradigm.

4.1 Model Architecture and Spatial Serialization↩︎

We build upon a robust, standard Vision-Language framework comprising a vision encoder (e.g., InternViT) coupled with a decoder-only LLM (e.g., Qwen2) via an MLP projector. To effectively handle high-dimensional clinical CT data without the massive computational overhead of dense 3D convolutions—and strictly avoiding heuristic pre-processing steps like bounding box (BBox) extraction or image cropping—we employ a straightforward 2.5D spatial serialization strategy.

Specifically, for each CT volume, we extract a sequence of contiguous slices along the z-axis, comprising eight slices. These slices are independently processed by the 2D vision encoder and sequentially embedded into a continuous visual token sequence. This slice-by-slice representation perfectly mirrors the clinical reading workflow and directly aligns with the “per-slice perception” logic generated by our Observer agent. It enables the language model to ground its sequential spatial tracking in the visual tokens, effectively capturing global structural coherence and true 3D spatial awareness from discrete 2D inputs.

4.2 Instruction Tuning and CoT Alignment↩︎

To validate our synthesized reasoning data, we fine-tune a 2D-pretrained baseline on the Hounsfield-CoT dataset (11.2k instances) using Low-Rank Adaptation (LoRA).

The model is trained to decouple its internal analytical process from the final clinical answer. By generating reasoning steps within <think> tags prior to the <answer> block, the model explicitly learns to map clinical findings to structured diagnostic logic—specifically sequential slice tracking, artifact mitigation, and differential exclusion. This alignment strategy demonstrates that high-quality, structured CoT supervision effectively transforms standard 2D backbones into robust 3D volumetric reasoners without requiring architectural modifications.

6pt

5 Experiments↩︎

This section empirically evaluates the Hounsfield-CoT paradigm. Our evaluation features rigorous cross-model and cross-benchmark analyses to validate robustness and generalizability, alongside a data scaling ablation to uncover the sample efficiency and spatial learning dynamics of MLLMs under structured reasoning.

5.1 Experimental Settings↩︎

We fine-tuned models via LoRA (bfloat16) on 11.2k curated CT-RATE instances, extracting consecutive Z-axis slices to preserve spatial tracking without bounding boxes. Evaluation spans four core 3D-RadVQA tasks: Anomaly Detection (T1), Image Observation (T2), Medical Computation (T3), and Existence Detection (T4). To assess cross-benchmark generalization, we further incorporate DeepchestVQA’s Recognition, Visual Reasoning, and Medical Reasoning tasks. We report standard Accuracy (%) for T4 and DeepchestVQA. For open-ended generative tasks (T1–T3), we employ an LLM-as-a-judge to semantically parse predictions against ground-truth coordinates, computing a task-specific Accuracy (%) that reflects genuine 3D spatial awareness rather than rigid string matching.

5.2 Main Results and Generalization↩︎

Table 2 summarizes performance across the 3D-RadVQA and DeepchestVQA benchmarks. Initially, 2D-pretrained baselines exhibit limited clinical reasoning (e.g., Hounsfield2D (Base) achieves only 7.2% in T1). Conversely, fine-tuning on Hounsfield-CoT unlocks massive gains: Hounsfield2D (Ours, FT) surges to 33.5% in T1, significantly outperforming its Lingshu counterpart. Notably, while Accuracy spikes, the stagnant semantic overlap (BERT-F1) confirms that structured CoT supervision shifts models from surface-level textual mimicry to genuine multi-step spatial reasoning. While native 3D models like M3D establish a performance upper bound (46.0% in T1) via dense convolutions, our data-centric approach dramatically closes this gap using purely 2D initializations, offering a sample-efficient and transparent alternative.

To rigorously evaluate the generalizability and model-agnostic impact of our paradigm, we extend our evaluation across two additional dimensions:

• Cross-Benchmark Generalization (DeepchestVQA): Evaluated zero-shot, Hounsfield2D (Ours, FT) (39.6%) outperforms robust foundation baselines like Huatuo (38.0%) and Med3DVLM (37.9%). While we observe minor degradation in basic Recognition tasks—attributable to the disruption of heuristic shortcut matching—this analytical rigor translates into decisive advantages in complex categories; specifically, Medical Reasoning improves from 39.7% to 45.1%.

• Cross-Architecture Robustness (Model-Agnostic Validation): To isolate the impact of our data paradigm from the underlying architecture, we fine-tuned the Lingshu baseline on our 11.2k dataset. The consistent performance surges—e.g., Organ Localization (T1) accuracy nearly doubling from 9.9% to 19.4%—confirm that our slice-wise data synthesis serves as a universal, model-agnostic catalyst for unlocking volumetric understanding.

5.3 Data Scaling Ablation↩︎

To quantitatively evaluate the sample efficiency of our structured CoT supervision, we partitioned the 11.2k dataset into four stratified subsets (25%, 50%, 75%, and 100%) while maintaining exact task ratios. We specifically selected Tasks 1, 2, and 4 from the 3D-RadVQA benchmark for this ablation, as they most directly evaluate the model’s core competencies in 3D spatial grounding and evidence-based verification. As visualized in Fig. 5, the scaling reveals distinct dynamics:

• Continuous Growth in Spatial Tasks (T1 & T2): For Anomaly Detection (T1) and Image Observation (T2), performance scales robustly with data volume. For instance, T1 accuracy surges from a 7.2% baseline to 33.5% at 100% scaling. This confirms that explicitly supervising sequential slice tracking yields direct and substantial gains in spatial awareness.

• Shortcut Disruption in Binary Detection (T4): Conversely, Existence Detection initially drops at 25% data (77.8% \(\rightarrow\) 74.3%). Unlike open-ended tasks, T4 reflects natural clinical distributions, allowing base models to inflate baselines via prior-based guessing. The limited 25% CoT data disrupts these brittle shortcuts before genuine 3D reasoning is fully established. However, at 100% scaling, accuracy recovers and peaks at 80.2%, confirming that reliable visual verification has successfully replaced heuristic guessing.

Ultimately, this ablation demonstrates that a modestly sized, high-quality CoT dataset effectively unlocks genuine 3D medical intelligence without requiring massive data scaling.

5.4 Case Study and Interpretability Analysis↩︎

To qualitatively demonstrate the superior reasoning capabilities of our proposed paradigm, we present a successful case of 3D abnormality localization (Figure 4). Identifying atelectatic changes requires rigorous volumetric integration, a task where standard visual-language models often fail by hallucinating spatial coordinates or prematurely truncating their analysis to a single slice.

Conversely, Hounsfield-CoT explicitly unrolls the reasoning trajectory. As illustrated in Figure 4, the model initiates an exhaustive volumetric traversal, scanning from the “apex to base”, observing that the “upper slices” are fully aerated, and detecting “linear streaky opacities” exclusively in the lower lobes. Crucially, the model leverages its 3D spatial awareness to perform differential exclusion: by verifying that the opacities “persist across multiple contiguous slices” and reside “within the lung parenchyma itself”, the model logically rules out single-slice partial-volume artifacts and pleural interface scarring. This evidence-based reasoning chain achieves a perfectly accurate diagnosis, demonstrating that Hounsfield-CoT effectively mirrors the systemic diagnostic workflow of radiologists, shifting MLLMs from superficial vision-to-text mapping toward genuine 3D spatial cognition.

6 Conclusion↩︎

In this work, we introduce Hounsfield-CoT, a data-centric paradigm that addresses the opaque nature of 3D medical MLLMs. By employing an Observer-Synthesizer framework, we encode the clinical slice-by-slice reading workflow into 11.2k structured reasoning instances. This approach bridges the spatial reasoning gap by synthesizing the Hounsfield-CoT data paradigm and constructing the Hounsfield3D model, which enables 2D-pretrained backbones to achieve robust volumetric reasoning and clinical interpretability. By enforcing sequential spatial tracking and differential diagnostic logic, our method effectively bypasses the need for computationally prohibitive 3D pre-training while maintaining high diagnostic fidelity.

While our paradigm yields significant gains, limitations persist. Current synthesis relies on global reports, which may occasionally lead to subtle spatial hallucinations in reasoning steps. Future work will extend this framework to diverse modalities like MRI and refine evaluation metrics to strictly verify the clinical correctness of individual thought steps. Ultimately, this research provides a transparent, scalable pathway for adapting 2D-pretrained models toward robust, high-fidelity 3D medical image understanding.

References↩︎

---

1. \(^{\dagger}\) These authors contributed equally to this work.↩︎

2. * Corresponding author: liyan@uir.edu.cn. This work was supported in part by the Fundamental Research Funds for the Central Universities under Grant 3262025T82, the Beijing Natural Science Foundation under Grant 4262075, and the Research Funds for NSD Construction, University of International Relations, under Grant 3262026T23.↩︎