1 Introduction↩︎

Human visual attention is capacity-limited and shifts over time. During whole-slide image (WSI) examination, pathologists scan the slide sequentially, selectively focusing on task-relevant regions. These attention trajectories reflect the visual evidence a pathologist encodes during cancer diagnosis. Clinically, modeling pathologist attention can support trainee education by revealing how specialists examine slides and can also facilitate the assessment of pathologists’ expertise levels [1]–[4], and understand decision-making [5]. Computationally, expert attention has increasingly been used as auxiliary supervision in medical imaging. Across pathology and radiology, attention signals have guided fully supervised and weakly supervised classification [6]–[8], segmentation [9], and text-to-image generation [10]. Beyond medical imaging, evidence from cognitive science and attention-aware image description [11]–[13] suggests that where people look shapes how they interpret and describe visual content. Together, these findings motivate leveraging pathologists’ attention as an auxiliary cue for pathology report generation, which grounds generated text in plausible visual evidence, and improves both interpretability and report quality.
Despite rapid recent progress in pathology report generation and visual question answering [14]–[18], only a few works incorporate auxiliary supervision such as model-predicted semantic information, and none so far leverage human attention in pathology report generation. Kim et al. [19] introduces semantic cues via a multi-label classifier over key diagnostic elements, but the generated reports are limited to predefined categories and depend on predicted semantics rather than ground-truth signals. Agent-based approaches such as CPathAgent [20] and PathChat+ [21] supervise the training by predicting the Region Of Interest(ROI)-based visual support with step-by-step reasoning; however, these ROIs are not derived from the attention of actual pathologists, and inducing such visual support typically requires large-scale instruction tuning.
In this paper, we align the visual attention of report generation models with pathologist attention. In vision–language transformers, model attention weights indicate how much each image patch contributes to token generation, providing a natural mechanism for grounding diagnostic statements in visual evidence. To supervise this alignment, we first collect a prostate cancer pathologist-attention dataset that captures multi-scale WSI examination behavior for five clinically critical components: Gleason patterns, perineural invasion, intraductal carcinoma, extraprostatic extension, and surgical margin. While the pathologists read the slides, we record viewport trajectories across magnification levels and synchronized, real-time verbal descriptions of observed findings, which provide soft labels for all the five components across the corresponding viewports.
Using this rich supervision, we introduce a plug-and-play module for report generation in the form of an auxiliary attention-alignment loss that regularizes model attention toward the corresponding pathologist attention distribution over image tokens (1). Pathologist attention is required only during training, with no additional inputs at inference, and the loss adds negligible overhead. We evaluate this design on both a report generation model and a WSI visual question answering model, covering two common vision–language fusion designs: cross-attention and self-attention. Experiments show 5.5%–12% average improvements on NLP-based metrics, and 8%–34% gains in accuracy on the five clinically related components across the two models. The model attention in the inference-time is aligned more closely with pathologist attention, providing stronger visual evidence for the generated reports. Our contributions are:

• To our knowledge, we are the first to incorporate pathologist-attention supervision into pathology report generation, showing consistent report-quality gains across architectures.

• We collect a multimodal pathologist-attention dataset of prostate WSIs with multi-scale viewport trajectories synchronized with spoken descriptions of observed findings.

• We improve interpretability by producing attention-based visual evidence that better matches how pathologists attend during cancer diagnosis.

As we show, fine-tuning a pathology report generation model with human attention improves performance with relatively small pathologist annotation effort (\(\sim7\) hours in total for the reported experiments). This suggests that our plug-and-play module can easily become a part of model training for pathology report generation.

2 Pathologist Attention Dataset↩︎

Dataset collection. We record pathologists’ attention using QuIP caMicroscope [22]. Six pathologists reviewed 121 TCGA-PRAD prostate WSIs [23] and annotated seven elements: Gleason score, intraductal carcinoma (IDC), perineural invasion (PNI), extraprostatic extension (EPE), surgical margin, lymphovascular invasion, and seminal vesicle invasion. As shown in 2, slides are viewed in a 1912\(\times\)​930 viewport with iterative zooming and panning, yielding multi-magnification viewports. We sample viewports at 20,Hz and convert each to its center point to form a magnification-aware scanpath; synchronized verbal descriptions provide soft component labels. We remove duplicates and keep only positively labeled points per component. Due to sparsity, we exclude lymphovascular and seminal vesicle invasion and focus on five components. On average, scanpaths contain 59 points with 221,s per slide (\(\sim7\) hours total).
Human attention map construction. We construct Gaussian-blurred saliency maps [1], [2], [24]. For each magnification (\(2\times\), \(4\times\), \(10\times\), \(20\times\)), we project scanpath points onto the corresponding grid, apply Gaussian smoothing, then resize and merge maps into a single multi-scale saliency map per slide.

3 Method↩︎

3.1 Preliminaries↩︎

Report generation models adopt transformers structure [25]. Given queries \(Q\), keys \(K\), and values \(V\), scaled dot-product attention computes \[A=\mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d}}\right), \qquad \mathrm{Attn}(Q,K,V)=AV,\] where \(A\) is the model attention weights, and \(\mathrm{Attn}(Q,K,V)\) is the output feature of a transformer attention layer.
Multimodal report generation models typically use two image-text fusion designs. Self-attention fusion concatenates image and text tokens \(Z=[\mathbf{X};\mathbf{Y}_{<t}]\), where \(\mathbf{X}\in\mathbb{R}^{I\times d}\) are image tokens and \(\mathbf{Y}_{<t}\in\mathbb{R}^{t\times d}\) are previously generated text tokens, and projects \(Q,K,V\) from \(Z\). Cross-attention fusion projects \(Q\) from the current text query, while \(K,V\) come only from \(\mathbf{X}\). These fusion determine how slide features influence next-token prediction: WSI-LLaVA [14] uses self-attention fusion, whereas HistGen [15], Bi-Gen [17], and HistoGPT [18] adopt cross-attention.

3.2 Pathologist Attention Alignment↩︎

Pathologist attention encoding in token space. Pathology report generation models first encode each WSI into a sequence of image tokens \(\mathbf{F}\in\mathbb{R}^{I\times d}\). Given a full-resolution human saliency map, we assign each token a saliency value by querying the map at the token’s recorded patch coordinates, yielding \(\mathbf{a}^{\text{human}}\in\mathbb{R}^{I}\). To encourage model attention consistent with pathologists’ attention, we fine-tune WSI-LLaVA and HistGen with the standard autoregressive next-token objective augmented by an attention-alignment loss.
Human attention alignment in WSI-LLaVA [14]. We use the last-layer self-attention weights \(\mathbf{A}\in\mathbb{R}^{(I+P+T)\times(I+P+T)}\), where \(I\) is the number of image tokens, \(P\) is the prompt length, and \(T\) is the number of answer tokens. We extract the answer-to-image attention block \[\mathbf{A}_{\text{ans}\rightarrow \text{img}} =\mathbf{A}\big[\mathcal{T}_{\text{ans}},\,\mathcal{I}\big]\in\mathbb{R}^{T\times I},\] where \(\mathcal{I}\) indexes image-token positions and \(\mathcal{T}_{\text{ans}}\) indexes answer-token query positions. We then aggregate token-wise attention into an image-level model attention vector by averaging over answer tokens: \[\mathbf{a}^{\text{model}} =\frac{1}{T}\sum_{t\in\mathcal{T}_{\text{ans}}}\mathbf{A}_{\text{ans}\rightarrow \text{img}}[t,:]\in\mathbb{R}^{I}.\] Given the corresponding human attention vector \(\mathbf{a}^{\text{human}}\in\mathbb{R}^{I}\), we define the alignment loss as the KL divergence between the attention distributions following the design of GazeVLM [26], \[\mathcal{L}_{\mathrm{KL}} = D_{\mathrm{KL}}\!\left( \mathrm{softmax}(\mathbf{a}^{\text{human}}) \;\Big\|\; \mathrm{softmax}(\mathbf{a}^{\text{model}}) \right).\]

Training objective. Given the ground-truth tokens \(\{t_1,\dots,t_T\}\), the base objective is the next-token negative log-likelihood \(\mathcal{L}_{\mathrm{CE}} = -\sum_{t=1}^{T} \log p_{\theta}(t_t \mid t_{<t}, \mathbf{X})\). The overall loss is \(\mathcal{L} = \mathcal{L}_{\mathrm{CE}} + \lambda\,\mathcal{L}_{\mathrm{KL}}\), where \(\lambda\) controls the strength of alignment, and is set to 1.0 in the experiment.
Human attention alignment in HistGen. Given the different attention mechanism of HistGen (cross-attention) from WSI-LLaVA (self-attention), we directly extract the model attention \(\mathbf{A}^{\text{model}}\in\mathbb{R}^{T\times I}\) and then follow the same way to compute alignment loss.

3.3 Inference-Time Visual Support Generation↩︎

During inference, we generate reports and extract model attention to image tokens in each step, then average across steps to obtain a slide-level vector \(\mathbf{a}^{\text{model}}\), then back-project it to a pixel-level heatmap by assigning each patch its token value, followed by normalization and Gaussian smoothing ( 3.2).

4 Experiments↩︎

4.1 Setups↩︎

Baselines. We use HistGen [15] and WSI-LLaVA [14] as base models, which differ in multimodal fusion attention mechanisms and are designed for different tasks (report generation vs.visual question answering). Zero-shot denotes directly evaluating each pretrained model on its test set. w/o alignment denotes LoRA fine-tuning [27] using only the next-token prediction loss, without any pathologist-attention alignment.
Report Dataset. We use the official datasets and splits of HistGen and WSI-LLaVA. For HistGen, attention alignment is trained only on the 81 slides with our attention annotations, while evaluation follows the official val/test sets (86/76). Visual-support evaluation (requiring GT comparison) is performed on 20 test slides. For WSI-LLaVA, we use 107 training examples and 11 test examples for report-quality evaluation, and 15 test samples for visual-support evaluation.
Metrics. We evaluate both report quality and visual support. For report generation, we report standard NLP metrics: BLEU (B1–B4) [28], METEOR (M) [29], and ROUGE-L (R) [30]. Since our attention annotations target five pathological components, we use ChatGPT-5.2 to extract them and compute exact-match scores between generated and ground-truth reports: Gleason requires matching both primary and secondary patterns, while the remaining components are binary scored. We report ACC-G, ACC-PNI, ACC-EPE, and ACC-S for Gleason score, perineural invasion, extraprostatic extension, and surgical margin, respectively; we omit intraductal carcinoma because it does not appear in the ground-truth reports. For visual support, we measure attention heatmap similarity with Normalized Scanpath Saliency (NSS) and KL divergence (KL) [31], and quantify agreement with ground-truth Gleason regions using segmentation labels from [32], reporting ROC-AUC (the probability that a pixel inside the binary segmentation mask receives a higher heatmap score than a pixel outside).
Implementation details. We use LoRA finetuning for both HistGen and WSI-LLaVA with the same split and training setting as the original papers. The experiment runs on a single RTX 6000 GPU. We set \(\lambda=1.0\) for both models.

3.5pt

\begin{table}[t] 3.5pt

Method	HistGen			WSI-LLaVA
	B4	R	ACC-G	B4	R	ACC-G
all layers	0.079	0.199	34.2	0.124	0.325	45.5
last layer (Ours)	0.077	0.203	35.7	0.136	0.338	63.6

\vskip

-0.2in
\end{table}

4.2 Results↩︎

Quantitative results. In ¿tbl:tab:main-result?, aligning pathologist attention achieves the best performance on both HistGen and WSI-LLaVA. In particular, our method yields higher accuracy on all five clinical components than the baselines, suggesting that human attention guides the model toward more informative regions. We also observe that fine-tuning WSI-LLaVA without attention alignment leads to a performance drop, likely due to overfitting in this low-data setting, as WSI-LLaVA is built on a large language model (Vicuna-7b-v1.5 [33]). By providing human attention as a visual constraint, this alignment discourages the model from memorizing report text.
Qualitative results of generated reports. In 3, our model captures the clinically critical components targeted in our dataset collection more consistently than the baselines, suggesting it learns to leverage pathologist attention cues effectively.
Visual support. We report qualitative and quantitative results on visual support in 4 and 1, evaluating (1) alignment with pathologist attention and (2) coverage of Gleason patterns. These results suggest our alignment loss encourages the model to focus on more informative, human-preferred regions, improving report quality. The resulting visual support also enhances interpretability by indicating which regions support the model’s diagnostic statements.
Ablation. We ablate which transformer layer to use for extracting model attention in human-attention alignment. Results in ¿tbl:tab:layer-ablation? show that supervising only the last layer is sufficient to steer the model toward pathologist-attended regions, likely because last-layer attention most directly influences next-token prediction. In contrast, aligning multiple layers in WSI-LLaVA degrades performance below the zero-shot baseline, likely because it is much deeper than HistGen due to its large language model backbone, where different layers encode distinct information and enforcing a uniform attention target across all layers is overly restrictive.

\begin{table}[t] 3.5pt

Method	HistGen			WSI-LLaVA
	B4	R	ACC-G	B4	R	ACC-G
all layers	0.079	0.199	34.2	0.124	0.325	45.5
last layer (Ours)	0.077	0.203	35.7	0.136	0.338	63.6

\vskip

-0.2in
\end{table}

5 Conclusion and Discussion↩︎

We propose a plug-and-play pathologist attention–alignment framework that uses multi-scale expert attention as auxiliary supervision for explainable pathology report generation. Across HistGen and WSI-LLaVA, it improves report quality and clinically relevant component accuracy while producing attention maps that better match pathologists’ attention with minimal overhead. Our results highlight the value of auxiliary signals for report generation and suggest scalability to additional cancer types due to the modest data-collection burden. Future work will expand attention collection to more cancer types and develop models that generalize to new cancer types by leveraging shared examination behaviors of pathologists among different cancer types.

6 Acknowledgement↩︎

This research is supported by US National Science Foundation (NSF) grants IIS-2123920, IIS-2212046 and 2442053, National Institutes of Health (NIH), National Cancer Institute (NCI) grants 1R21CA25849301A1, 1R01CA297843-01, 3R21CA25849302S1, 1R03DE033489-01A1 and the Health Data Hub as part of the second edition of the France-Québec call for projects Intelligence Artificielle en santé. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

Disclosure of Interests. The authors have no competing interests to declare that are relevant to the content of this article.

References↩︎