1 Introduction↩︎

Radiology report generation (RRG) aims to automatically generate free-text descriptions from radiological images such as X-rays [1] and CT scans [2]. Current RRG approaches can be categorized into single-view and multi-view methods. Single-view models [3]–[8] focus on generating reports from individual radiographic projections but suffer from inherent limitations in capturing comprehensive anatomical information. Multi-view approaches [9]–[11] address these limitations by incorporating information from multiple perspectives - such as frontal, lateral, posteroanterior (PA), and anteroposterior (AP) views - demonstrating empirical improvements over their single-view counterparts. This multi-view paradigm aligns with clinical practice, where radiologists routinely examine each imaging view individually and then integrate these per-view observations into a comprehensive report.

Multi-view medical imaging is clinically essential for identifying posterior abnormalities, localizing lesions, and detecting conditions obscured in single projections [12]–[15]. However, existing multi-view RRG methods [9]–[11], [16] primarily employ fusion strategies to enhance feature representation, overlooking a critical issue: different views exhibit intrinsic variations that challenge automated systems. Such variations may lead to inconsistent interpretations across views, compromising diagnostic reliability and report quality. Current systems like InternVL-DA-Medical-4B [17] generate reports from frontal views that generally align with GT reports, while lateral view outputs often contain contradictory findings. Such inconsistencies pose significant clinical risks, particularly when frontal images are unavailable due to patient positioning constraints, trauma conditions, or equipment limitations, making lateral views the primary diagnostic input. In settings where frontal and lateral views are provided separately, inconsistent outputs may further lead to confusion, misinterpretation, and unreliable clinical decision-making. This raises a fundamental question: what underlying mechanisms cause these view-specific discrepancies, and how can we systematically address them? Without understanding the root causes at the model’s internal level, solutions may only address symptoms rather than the core problem. Inspired by [18], [19], this motivates neuron-level analysis to identify which computational components are responsible for view-dependent variations, enabling targeted interventions to systematically improve cross-view consistency.

In this paper, we present a comprehensive neuron-level analysis of RRG models, namely View-specific Pattern Neuron Detection and Fine-tuning (View-PNDF). Rather than retraining the entire model, we first introduce a View-specific Neuron Detection (VND) module to identify a compact set of pattern neurons responsible for view-specific inconsistency. We then apply a View-specific Neuron Verification (VNV) module to quantitatively evaluate the contribution of these detected neurons by selectively deactivating them and measuring the impact on view-specific report generation. Building on these insights, we further propose a neuron-level adaptation strategy, View-specific Neuron Fine-tuning (VNF), which focuses training exclusively on the identified pattern neurons. The targeted fine-tuning enables the model to generate consistent view-specific findings from any available view, while findings from multiple views can be further consolidated into an integrated radiology report using LLMs. For evaluation, we use standard Natural Language Generation (NLG) metrics to assess the quality of the integrated reports. However, evaluating view-specific reports presents unique challenges as conventional NLG metrics fail to capture clinical validity: reports describing different anatomical views may receive similar scores even when they express contradictory clinical findings since NLG metrics measure textual overlap rather than medical semantics. To address this limitation, we employ a LLM–based evaluation method (e.g., GPT-4o) to assess the semantic fidelity and diagnostic accuracy of view-specific reports. By integrating view-aware methodology and appropriate evaluation strategies with RRG, our proposed framework offers a targeted and efficient solution for enhancing multi-view RRG performance.

2 Method↩︎

In this section, we introduce View-specific Pattern Neuron Detection and Fine-tuning (View-PNDF), a framework designed to enhance Radiology Report Generation (RRG) by identifying and optimizing neurons responsible for processing specific X-ray views (e.g., frontal vs. lateral). As illustrated in Fig. 1, our approach consists of three key components: View-specific Neuron Detection (VND), Neuron Verification (VNV), and Neuron Fine-tuning (VNF).

2.1 View-specific Pattern Neuron Detector (VND)↩︎

We hypothesize that within the RRG model - specifically in the linear and self-attention layers of the Transformer decoder - distinct subsets of neurons are specialized for handling different X-ray views. We define these as View-specific Pattern Neurons.

Neuron Influence via Perturbation. Ideally, the importance of a neuron can be measured by the impact of its deactivation on the model’s output. Formally, let \(\mathcal{RRG}(x, \theta)\) denote the generated embedding for a view-specific input sample \(x\) (comprising visual features and prompt tokens) given model parameters \(\theta\). The influence \(I\) of the \(k\)-th neuron at the \(l\)-th layer, denoted as \(n_l^k\), on a specific view pattern \(v\) is defined as the perturbation effect: \[I(n_l^k | x^v) = \left\| \mathcal{RRG}(x^v, \theta) - \mathcal{RRG}(x^v, \theta \setminus w_l^k) \right\|_2, \label{equ:influence}\tag{1}\] where \(\theta \setminus w_l^k\) represents the model weights with the specific neuron \(n_l^k\) deactivated (zeroed out).

Efficient Detection via Attribution Scores. Since calculating Eq. 1 for every neuron across the entire dataset is computationally prohibitive, we follow prior work [20], which suggests that neurons with high activation values are strong indicators of pattern importance. We therefore employ an efficient Attribution Score as a proxy for detection. The attribution score for neuron \(n_l^k\) given input \(x^v\) is defined as the magnitude of its activation output: \[\text{Attr}(n_l^k | x^v) = \left\| \sum_{t} f(x^v, w_l^k)_t \right\|_1, \label{equ:attr}\tag{2}\] where \(f(\cdot)\) computes the activation values (e.g., the output of a specific linear projection or attention head) across the sequence dimension \(t\).

Detection Strategy. To robustly identify neurons associated with a view \(v\) (e.g., lateral), we utilize a set of detection samples \(\mathcal{D}_{detect} = \{x^v_1, \dots, x^v_B\}\). A neuron is classified as a View-specific Pattern Neuron if it consistently ranks among the most active neurons across all samples in the detection set. The set of pattern neurons \(\mathcal{N}^v\) is formally defined as: \[\mathcal{N}^v = \bigcap_{b=1}^{B} \left\{ n_l^k \mid \text{rank}(\text{Attr}(n_l^k | x^v_b)) \leq \gamma N \right\}, \label{equ:detection}\tag{3}\] where \(\text{rank}(\cdot)\) sorts neurons in descending order of attribution score, \(N\) is the total number of neurons in the layer, and \(\gamma\) is a selection ratio (e.g., top 5%). This intersection strategy ensures we select neurons that are consistently salient for the specific view, filtering out noise from instance-specific variations.

2.2 Verification and Fine-tuning↩︎

View-specific Neuron Verification (VNV). To validate that the detected set \(\mathcal{N}^v\) genuinely controls view-specific generation, we employ the perturbation metric from Eq. 1 on a held-out verification set. We compare the performance drop when deactivating the detected pattern neurons versus deactivating a random set of neurons of the same size. The existence of view-specific neurons is confirmed if: \[\sum_{d=1}^{D} \mathcal{L}_{\Delta}(\mathcal{N}^v_{deact}) \gg \sum_{d=1}^{D} \mathcal{L}_{\Delta}(\mathcal{N}^{rand}_{deact}),\] where \(\mathcal{L}_{\Delta}\) represents the degradation in generation quality (e.g., increase in loss or decrease in clinical accuracy metrics) when neurons are deactivated.

View-specific Neuron Fine-tuning (VNF). Once identified and verified, we employ a targeted fine-tuning mechanism. Unlike standard fine-tuning methods which updates all parameters, VNF freezes the majority of the model and optimizes only the identified pattern neurons \(\mathcal{N}^v\). This allows the model to enhance its representation of specific views without catastrophically forgetting general language generation capabilities. The optimization objective is the standard cross-entropy loss over the view-specific fine-tuning set \(\mathcal{D}_{ft}\): \[\mathcal{L}_{VNF} = -\sum_{(x, y) \in \mathcal{D}_{ft}} \sum_{t=1}^{T} \log P(y_t | y_{<t}, x; \theta_{\mathcal{N}^v}),\] where \(\theta_{\mathcal{N}^v}\) denotes that only the parameters of the pattern neurons are updated.

0.5pt 9pt

0.5pt 2.0pt

3 Experiments and Results↩︎

3.1 Dataset and Evaluation Metrics↩︎

Datasets. We evaluate on IU-XRay [24] and MIMIC-CXR [25], both containing frontal and lateral views. For IU-XRay (\(2,955\) samples), we adopt the standard 70%/10%/20% split [26]. For MIMIC-CXR, to ensure balanced view representation, we curate a subset of \(10,000\) training samples and \(1,000\) validation/test samples, instead of the standard split [3], which is dominated by single-view reports.
Evaluation Metrics. Integrated reports are evaluated using NLG metrics, including BLEU [27], METEOR [28], and ROUGE-L [29]. For view-specific reports, given the limited ability of n-gram metrics to assess clinical validity, we employ GPT-4o [30] to score semantic fidelity on a scale of 1–10, following recent LLM-based evaluation practices [31].

4pt

Performance comparison of medical VLMs with and without View-PNDF. Scores are evaluated by GPT-4o.

Before FT	BL-1	BL-4	MTR	RG-L
Frontal	0.398	0.156	0.163	0.265
Lateral	0.397	0.154	0.162	0.266
After FT	BL-1	BL-4	MTR	RG-L
Frontal	0.397	0.155	0.164	0.265
Lateral	0.397	0.154	0.163	0.264

3.2 Implementation↩︎

The model is trained using a PyTorch implementation on two NVIDIA A6000 GPUs. The initial learning rate is set as \(4\times 10^{-6}\) with a cosine learning rate schedule. We use AdamW [32] optimizer to optimize the network with a weight decay of \(0.5\). We train the model for 1 epoch with a batch size of 8. The warm up ratio is set as \(0.3\).

3.3 Comparative Experiment↩︎

We evaluated our method on IU-XRay [24] and MIMIC-CXR [25] against state-of-the-art methods. As shown in Tab. ¿tbl:tab:iuxray95results?, applying our method to four VLM backbones (MedGemma, InternVL, LLaVA, Hulu) on IU-XRay consistently outperformed baselines across all metrics, with View-PNDF\(_{\mathrm{Hulu}}\) achieving the best results (e.g., \(0.510\) BLEU-1). On the MIMIC-CXR dataset (Tab. ¿tbl:tab:mimic95results?), our approach maintained this advantage, surpassing the leading baseline MLRG in both linguistic quality (ROUGE-L \(0.291\) vs \(0.278\)) and Clinical Efficacy (F1 \(0.414\) vs \(0.401\)). These results confirm that fine-tuning view-specific neurons significantly enhances both the linguistic fluency and clinical accuracy of generated reports.

3.4 Ablation Study↩︎

To verify the effectiveness of each module, we conduct ablation studies on the IU-XRay [24] and MIMIC-CXR [25] datasets using the InternVL medical model [17].

The Effectiveness of View-PNDF. Tab. ¿tbl:tab:comparison? details the GPT-4o evaluation of four VLMs with and without View-PNDF. The method consistently improved performance across all models and datasets. Notably, lateral view scores saw significant gains (e.g., +4.2 points on MIMIC-CXR for InternVL), demonstrating that View-PNDF effectively captures view-specific patterns to generate more accurate descriptions for both views.

The Effectiveness of VND and VNV. To validate our detection mechanism, we compared view-specific versus random neuron deactivation (Fig. 2 (a)). View-specific deactivation caused drastic performance drops (e.g., lateral score dropping from 2.9 to 0.3 on MIMIC-CXR), whereas random deactivation had minimal impact. This significant disparity confirms that VND successfully identifies neurons critical for specific views.

The Effectiveness of VNF. Fig. 2 (b) shows that view-specific fine-tuning significantly boosts lateral view performance (e.g., reaching 7.3 vs. 3.7 for random tuning on MIMIC-CXR) while preserving or enhancing frontal view quality. This proves VNF effectively bridges the performance gap between views without compromising general capabilities.

LLM Evaluators. Tab. [tbl:FT] shows that standard NLG metrics yield nearly identical scores before and after fine-tuning, failing to capture clinical nuances. In contrast, evaluations using GPT-4o [30], Qwen [33], and DeepSeek [34] (Tab. [tbl:tab:eval]) reveal clear performance differences, demonstrating that LLM-based evaluation aligns better with clinical validity for this task.

Qualitative Results. Fig. 2 (c) illustrates the qualitative impact of View-PNDF. While the baseline model misses critical findings (e.g., sternal wires, cardiomegaly) and lacks inter-view consistency, the fine-tuned model successfully captures these pathologies across both frontal and lateral views. This confirms that our method enhances the detection of view-specific pathological patterns, yielding more comprehensive reports.

4 Conclusion↩︎

We proposed View-PNDF to detect and fine-tune view-specific pattern neurons in RRG models. Experiments confirm the existence of these neurons and show that optimizing a small fraction (\(<5.0\%\)) significantly enhances generation consistency and accuracy across views. While demonstrated on chest X-rays, our parameter-efficient method holds potential for other modalities (e.g., CT, MRI). Future work will focus on extending this approach to multi-modal data and developing model-agnostic implementations.

References↩︎