1 Introduction↩︎

Building user trust in AI systems is critical to the success of agentic workflows in both enterprise and consumer environments. In many settings, users cannot safely act on a generated answer without verifying its source and validity — even modern generative systems fully support fewer than 52% of their generated statements with accurate citations [1]. As a result, model grounding via attributions—localizing each answer component to its supporting evidence—has emerged as a fundamental requirement for model deployment, particularly in domains such as medicine where ungrounded or hallucinated answers can have real negative impacts [2].

There have been increasing efforts to use attributions to ground document question-answer pairs, though most focus on text-only or otherwise unimodal settings. Current approaches typically rely on citation-style generation [3]–[5], retrieval-head or circuit isolation [6], or decomposition-based attribution methods [7], [8], which have thus far been explored only for text. Real documents, however, interleave text with images, charts, and other raster content. A robust attribution system must therefore identify not only the correct source, but also the supporting modality or combination of modalities.

The multimodal long-document setting remains comparatively nascent, with preexisting approaches largely framing attribution as citation selection from pre-retrieved passages or images rather than as fine-grained localization within a single full-length document [9]–[11]. A unique set of challenges arises in multimodal attribution that doesn’t apply in the unimodal case: determining both the correct modality (or modalities) and the correct source within it. Resolving how text and images jointly support a single attribution remains an open problem with significant downstream potential.

To address this critical challenge, we propose MultAttnAttrib, a training-free multimodal attribution method that leverages attention patterns from a model’s prefill pass to localize supporting evidence in long interleaved documents. Our method identifies a subset of retrieval heads that consistently attend to ground-truth evidence across modalities, aggregates their attention signals to score text spans and image regions jointly, and applies a lightweight calibration procedure to produce modality-aware citations in a single inference pass. Unlike prompting-based attribution methods, MultAttnAttrib avoids iterative generation and additional reasoning overhead, substantially reducing inference cost while improving attribution quality.

Because existing benchmarks are insufficient for evaluating fine-grained multimodal attribution in long documents, we also introduce MultAttrEval, a complementary evaluation benchmark spanning five domains and covering both unimodal and multimodal attribution settings. Using MultAttrEval, we evaluate a broad set of attribution baselines, including prompting-based, captioning-based, and retrieval-augmented approaches, on both open-source and frontier MLLMs.

Our results reveal a substantial gap between multimodal attribution and unimodal attribution performance, confirming the unique difficulty of multimodal attribution. Despite this challenge, MultAttnAttrib consistently outperforms most strong baselines on both Qwen3-VL-30B and a frontier model, while operating at roughly 14% of the inference latency by extracting attributions directly from the prefill pass and reducing peak memory usage by approximately 15GB (non-vLLM) per QAA instance.

In summary, our contributions are as follows.

• MultAttnAttrib: A training-free multimodal attribution method that produces modality-aware citations efficiently in a single inference pass.

• MultAttrEval: A complementary benchmark for fine-grained multimodal attribution in long documents across five domains.

• Extensive experiments demonstrating that MultAttnAttrib consistently outperforms strong prompting, captioning, and RAG-based baselines on the original open-source MLLM backbone, while achieving substantially lower latency.

2 Related Work↩︎

2.1 Attribution on Multimodal Inputs↩︎

The explainability of language model outputs has motivated extensive work on citing and attributing generated text, falling into three broad families. The first family fine-tunes models to interleave citations with output, building on Attributed QA [3], the ALCE benchmark [4], and training-based citation generation methods[12]–[14]. The second family decouples attribution from generation by post-processing outputs with external retrievers, NLI verifiers, or LLM judges [15], [16]. The third family recovers attribution directly from the model’s computations: by aggregating attention signals across heads [6], [17], by reading internal signals via saliency maps or intermediate activations [10], [18], [19], or by probing the model through systematic context ablations [20]. Our method belongs to this third family.

2.2 Datasets for Multimodal Attribution↩︎

Evaluating multimodal attribution requires benchmarks that test evidence localization over full multimodal documents. Existing benchmarks such as MCiteBench [21], MMDocRAG [22], and MAVIS [11] instead evaluate citation selection from small, pre-curated pools of passages, figures, or tables, reducing attribution to discrete candidate selection rather than true localization. Similarly, SciClaimEval [23] pre-identifies the relevant figure or table and evaluates only cross-modal entailment, sidestepping retrieval entirely. These settings do not reflect deployment conditions, where models must localize supporting evidence within long, interleaved documents. Concurrent work, MuRGAt [24], also studies free-form evidence selection without a candidate pool, but focuses on temporal video/audio attribution and generation-based methods. In contrast, our approach extracts citations directly from attention signals over static multimodal documents in a single forward pass.

3 MultAttnAttrib: A Training-Free Approach for Multimodal Attribution↩︎

Existing attribution methods can oftentimes be reduced to compute-intensive LM fine-tuning for citation generation [12]–[14], or multi-step approaches requiring additional model calls [15], [20], [25]. Mechanistic interpretability offers a streamlined alternative: identifying a sparse subset of attention heads responsible for copying evidence from context, then attributing via their attention maps in a single forward pass [6], [26]. However, these approaches focus on text-only QA, leaving image and multimodal QA unattributed.

Extending to multimodal extractive QA using text-only retrieval heads would omit visual evidence entirely. We find that retrieval heads are modality-specific at the top ranks but largely shared across the broader population. This motivates MultAttnAttrib, a label-supervised, training-free approach that exploits this shared backbone by identifying cross-modal retrieval heads from a small probe set, and extracting their attention signals to score both image slots and text passages in a single forward pass.

3.1 Task↩︎

We study the problem of multimodal attribution. Given a document composed of text and images, a question, and an answer, the goal is to attribute the answer to its supporting evidence in the document.

Let the document be \(\mathcal{D} = (\mathcal{T}, \mathcal{I})\), where \(\mathcal{T} = (t_1, t_2, \dots, t_{|\mathcal{T}|})\) is a sequence of text tokens, and \(\mathcal{I} = \{ I_1, I_2, \dots, I_{|\mathcal{I}|} \}\) is a set of images in the document. A text span is defined as a contiguous subsequence \(\mathcal{T}_{i:j} = (t_i, t_{i+1}, \dots, t_j)\), with \(1 \le i \le j \le |\mathcal{T}|\).

Given a question \(q\), the system produces an answer \(a\), which is attributed to one of the following evidence types: a text span \(\mathcal{T}_{i:j}\), a set of images \({\mathcal{I}}^* \subseteq \mathcal{I}\), or a joint text–image set pair \((\mathcal{T}_{i:j},\, {\mathcal{I}}^*)\). We define the attribution space as \(\mathcal{A} = \bigl\{\, \mathcal{T}_{i:j} \,\bigr\} \cup \bigl\{\, {\mathcal{I}}^* \,\bigr\} \cup \bigl\{\, (\mathcal{T}_{i:j},\, {\mathcal{I}}^*) \,\bigr\}\). The multimodal attribution task is to learn a function \(f : (q, \mathcal{D}, a) \rightarrow \hat{\alpha}\), where \(\hat{\alpha} \in \mathcal{A}\) is the predicted attribution.

Given a dataset \(\bigl\{\,(\mathcal{D},\; q,\; a,\; \alpha^*)\,\bigr\}\), where \(\alpha^* \in \mathcal{A}\) is the ground-truth attribution, the objective is to correctly attribute each answer to its supporting evidence in the multimodal context provided.

3.2 Head Identification↩︎

To identify multimodal and image retrieval heads, we need a scoring method that is sensitive to both unimodal and multimodal evidence. Prior approaches, such as average copy-paste frequency [26] and path patching [6], [27], are either correlational or prohibitively expensive at scale. To address this, we tested retrieval head isolation against two methods: Causal Mediation Analysis (CMA) and Mean Attention Scoring (MAS). Given the results of our tests (more details follow in the Section 6.3), MultAttnAttrib scores all heads against labeled multimodal probes using CMA. Details about the two methods are as follows:

3.2.0.1 MAS requires only a single forward pass per probe.

The heads are scored by the ratio of the mean attention to the ground-truth positions \(G_i\) to the total attention over the entire document \(D_i\). This measures how selectively heads attend to evidence over distractors. This is cheaper than CMA (discussed below) but correlational, lacking causal validity (Heads that happen to concentrate on the ground-truth region score high regardless of whether they actually causally mediate retrieval).

3.2.0.2 Adapting CMA for retrieval head identification

costs only two forward passes per probe: one clean pass on the original input \(x_i\) and one corrupted pass where the evidence is replaced with content from another document. While previous CMA work focused on text [6], this corruption strategy is multimodal. Ground-truth text tokens are replaced with a contiguous span of equal length from another probe’s document to preserve the sequence structure. Corrupted images are resized to the dimensions of the ground-truth images to preserve the patch grid. It ensures that the clean and corrupted inputs have the same shape, thereby isolating the causal effect.

The Indirect Effect (IE) of each head \((l, h)\) is expressed by the difference in the mean attention to ground-truth positions \(G_i\) between the clean and corrupted inputs, averaged over the query tokens \(Q_i\) (comprising the answer and question tokens without stopwords or punctuations). To avoid over-attribution, we further suppressed heads that spread attention uniformly using weights derived from the normalized variant of Shannon entropy [28] of document-averaged clean attention [29]. After accumulating the scores for each head in all probes, we select the top-\(k\) retrieval heads \(\mathcal{H}\) with the highest scores. Pseudocode for both scoring methods is given in Appendix 12.

3.3 Calibration↩︎

We estimate the sliding window length \(W\) as the median chunk token length in the probe set. Using the selected heads \(\mathcal{H}\), we run Attribution (Algorithm 2) on all probes. Scores are partitioned by ground-truth modality labels (image-positive/negative, text-positive/negative), and we sweep over maximum attribution scores to select thresholds \(T_\mathrm{img}\) and \(T_\mathrm{txt}\) that maximize F1 for image and text attribution, respectively. These thresholds are later used during inference. Pseudocode is provided in Algorithm 12 (Appendix 12).

Without calibration, there is no decision boundary for citing text or images, and raw attention scores are less interpretable than probabilities. We therefore perform an F1-maximizing threshold sweep to derive modality-specific thresholds directly from attribution score distributions observed in real documents.

3.4 Attribution↩︎

Attribution requires a single forward pass over the query document. We average attention across the selected heads to obtain a mean attention vector, score each image by averaging over its patch tokens, and score text by averaging over sliding windows of token positions. Image and text scores are min-max normalized, then thresholded using \(T_\mathrm{img}\) and \(T_\mathrm{txt}\) to determine citations. If no score exceeds its threshold, we fall back to the highest-scoring image or text span.

4 MultAttrEval: A Dataset for Multimodal Attribution in Long Document Understanding↩︎

4.1 Dataset Generation↩︎

MultAttrEval was created to address the need for fine-grained Question-Answer-Attribution (QAA) triplets given long documents with mixed-modality content, and it allows us to test the strength of our attribution approaches. As shown in diagram 3, we begin by obtaining PDF files from MINT-1T [30]. Each document is filtered based on the image count and the presence of valid URLs. We extract text and images, preprocess both, and then generate embeddings + similarity pairings for text/text, text/image, and image/image. QAA generation is then split up into the three domains as follows:

Unimodal (Text/Image only). We first isolate images or text chunks that are mutually dissimilar, with additional disjoint text pairing for the image case, for document understanding purposes. We then use an MLLM to generate QA using only our selected images or text-chunk spans, thereby creating unimodal attributions for our input.

Text + Image. This case warrants a different treatment from the previous cases, as the text and image attributions should both be relevant and mutually support the model’s answer to the question. Here, we rerank the most similar (text, image) pairs from the embedding step, then identify entities in the texts and verify whether they belong to the image. Surviving (text, image) pairs and entities are then used to elicit questions and answers.

4.2 Quality Verification↩︎

Each generated QAA candidate is subjected to a sequence of strict acceptance criteria, with full details in Appendix 19. For text-only items, we apply four checks: quality thresholding enforces a minimum verifier score to exclude low-value QA pairs; attribution support is a binary gate that filters out items whose attribution is not used to generate the answer; evidence consistency requires the verifier-provided evidence span to be non-empty, 12–25 words, and an exact substring of the source paragraph; and cross-chunk evidence uniqueness rejects QA pairs whose supporting span appears across more paragraph chunks than the configured ambiguity threshold. For image-only and multimodal items, we additionally apply: source referencing, ensuring QA pairs reference high-level domain topics rather than the source artifact directly; question triviality, rejecting questions that target layout artifacts such as arrows, bounding boxes, or callouts rather than factual or domain-relevant content; and answerability, which scores the degree to which the answer is grounded in and derivable from the source material. Finally, multimodal-only items must additionally satisfy crossmodal grounding, verifying on a 1–7 scale that the image visually grounds at least one key answer entity and the text explicitly grounds at least one distinct entity, and answer circularity, which requires each answer to introduce at least one new piece of factual content beyond the question, though shared proper nouns and technical terms are explicitly permitted.

4.3 Dataset Summary↩︎

MultAttrEval contains question-answer-attribution triplets for long-form PDF documents spanning five domains and three attribution settings: text-only, image-only, and multimodal. Full corpus distributions and modality-level statistics are reported in Appendix 9.

5 Experiments↩︎

5.1 Implementation Details↩︎

We split the QAAs into Probe and Test sets. We sample 30 QAAs from each regime from our initial set, generating 90 probe QAA triplets. The Probe set is used for attention head analyses and for head identification and threshold calibration in MultAttnAttrib. The remaining 608 items, our Test set, are used to evaluate all methods.

5.2 Baselines↩︎

For all baselines, we evaluate the Qwen3-VL-30B-A3B-Instruct (open-source backbone for MultAttnAttrib), and the frontier model GPT-5.4, both of which support long-context multimodal document understanding. The VLM baseline performs attribution using images and OCR text, while the LLM baseline replaces images with captions and operates purely over text. We additionally evaluate RAG variants (\(k=5\)), where Cohere retrieves \(k\) text chunks + \(k\) images and ColQwen retrieves \(k\) full PDF pages. Detailed descriptions are in Appendix 10.

5.3 Evaluation Metrics↩︎

We evaluate attribution quality using macro-averaged precision, recall, and \(\mathrm{F1}\). Image citations are evaluated by exact match. Text citations are evaluated using fuzzy substring similarity, then discretized into 3 score tiers and penalized for under- or over-quoting based on the length ratio. Full computation details as well as additional LLM-as-Judge evaluations are in Appendix 11 and 17.

6 Results and Analysis↩︎

6.1 MultAttnAttrib Improves over Different Prompting Strategies With the Same Backbone↩︎

3pt

In this section, we compare our method against several prompting strategies for obtaining attributions on the same Qwen3-VL-30B-A3B-Instruct backbone. As shown in Table 2, MultAttnAttrib outperforms all prompting baselines by a substantial margin of over 20\(\%\), and this gap holds consistently across all three modality splits. These results demonstrate that internal signals from cross-modal and specialized retrieval heads, when carefully post-processed, yield significantly stronger attribution performance than can be achieved by tuning prompting strategies alone. Below we provide some of the main empirical results:

6.1.0.1 MultAttnAttrib consistently improves attribution quality over direct and RAG-augmented VLM baselines.

Table 2 shows universal F1 gains, especially in image precision and text recall, indicating that attribution extracted from attention signals can outperform generated citations in localization-heavy long-document settings.

6.1.0.2 MultAttnAttrib mitigates baseline text overprediction.

Baseline methods frequently over-attribute text spans, heavily reducing image-regime precision and text-regime recall. Thresholding in MultAttnAttrib suppresses many of these spurious citations, improving unimodal attribution quality (Table 2).

6.1.0.3 MultAttnAttrib responds more positively to fine-grained retrieval methods than page-based retrieval.

Combining our method with Cohere RAG substantially improves text performance and modestly improves image and multimodal results, while ColQwen degrades text and image metrics with only marginal multimodal changes. This suggests fine-grained retrieval is more effective for attribution than page-level retrieval.

6.1.0.4 MultAttnAttrib processes long documents with lower memory and latency than VLM prompting.

Direct non-vLLM inference on our Qwen model results in frequent OOM errors for QA inputs, a problem MultAttnAttrib avoids by attributing in a single forward pass, bypassing KV-cache growth, and removing token-level decoding overhead. Focusing on the non-OOM QA pairings, MultAttnAttrib not only has nearly 15 GB lower peak VRAM usage, but also \(7.3\times\) better latency during inference for a singular QA input. Details are in Table 1.

6.2 Comparing MultAttnAttrib to Frontier GPT-5.4↩︎

MultAttnAttrib shows complementary strengths relative to prompted GPT baselines (Figure 4). On visual grounding, it achieves stronger image precision and F1 than all GPT baselines, since attention aggregation operates directly over the full token sequence rather than verbalizing attributions. In text settings, the trade-off shifts: GPT baselines attain higher precision by returning minimal evidence, while MultAttnAttrib recovers higher recall by capturing all influential tokens at the cost of including loosely related text. Overall, it remains competitive with frontier-scale closed-source models such as GPT-5.4 on multimodal attribution.

6.3 Analysis of Unimodal and Crossmodal Attention Heads↩︎

A central design question for MultAttnAttrib is whether text and image retrieval emerges from shared or modality-specific attention circuits. If the circuits are shared, a single joint head set is preferable and reduces the cost of modality-specific head identification.

We score all \(L \times H = 1536\) heads under both CMA and MAS, then measure cross-modal agreement via IoU and Spearman’s rank correlation over the top-\(k\) head sets. Setup and metric definitions are in Appendix 18.

6.3.0.1 Although the majority of the head population is shared across modalities in both scoring methods, some top-ranked heads are modality-specific.

For instance, a top text head like \((19,3)\) can be the worst image head. At \(k=4\), CMA yields \(\rho_4 = -0.107\) and MAS yields \(\rho_4 = -0.657\). This anti-correlation naturally subsides as more top-\(k\) heads are included. CMA quickly recovers to \(\rho_{20} = 0.042\), while MAS remains strongly anti-correlated until \(k = 88\). Under CMA, \(\text{IoU}(4) = 0.143\) and \(\text{IoU}(20) = 0.379\), meaning that even the top-20 heads overlap only about a third. Under MAS, \(\text{IoU}(4) = 0.333\) but \(\text{IoU}(20) = 0.212\), reflecting assemblage of modality-specific heads as \(k\) grows.

6.3.0.2 CMA locates retrieval heads through a causality-based reward that favors shared heads carrying cross-modal retrieval signals.

Because it rewards heads whose activations causally influence attribution outputs regardless of target, shared copy-and-paste style retrieval heads score highly across modalities, improving IoU and reducing anti-correlation at small \(k\). In contrast, MAS favors heads that concentrate attention within a modality, producing modality-specific routing heads and stronger negative correlation.

6.3.0.3 Layer-wise analysis reveals structurally different retrieval circuits for text and images.

Image heads (CMA) concentrate almost exclusively in layers \(22\)–\(36\) Figure 6 (a), 15). Text heads are distributed across both early and late layers, reflecting the richer syntactic and semantic processing demands of textual evidence. Crossmodal heads cluster in the mid-to-late transition zone, forming the retrieval backbone common to both modalities.

6.3.0.4 The CMA score distributions (Figure 6 (b)) reveal that the retrieval circuit is extremely sparse.

About \(80\%\) of all types of heads score below \(0.1\) after min-max normalization, and fewer than \(2\%\) of heads in any category score above \(0.6\). This sparsity replicates and extends prior findings that fewer than \(5\%\) of heads qualify as retrieval text heads [26] to the multimodal attribution setting. This sparsity also confirms that a small number of heads, \(k\), is sufficient to capture most of the retrieval signal across modalities. This makes MultAttnAttrib’s single-pass attribution practical and efficient.

7 Conclusion↩︎

In this paper, we introduce MultAttnAttrib, a training-free attribution method (with cross-modal and specialized retrieval heads) that outperforms a range of strong prompting, inference-time strategies on the same backbone at a fraction of the latency, and remains competitive with frontier-scale models such as GPT-5.4. We further introduce MultAttrEval, a test-bed for evaluating multimodal attribution over long-context documents.

8 Limitations↩︎

Our work has several limitations related to both the benchmark and the method. First, MultAttrEval consists of long, image-dense documents that often contain near-duplicate or decorative images with little semantic value; because the image regime uses single-source QAA triplets, ground-truth attributions contain only one image while baselines frequently retrieve visually similar alternatives, depressing performance. Future curation should enforce stricter image-relevance filtering and support multiple image attributions, potentially through embedding-cluster or entity-based grouping. Finally, MultAttnAttrib requires a small labeled probe set of QAAs for both head identification and threshold calibration: unsupervised head scoring [26] could remove the annotation requirement, though correlational heads may score highly without causally mediating retrieval, and the modality F1 sweep could be replaced with fixed thresholding on normalized scores, reflecting a trade-off between annotation cost and performance. We aim to explore these questions more thoroughly in future work.

9 MultAttrEval Dataset Statistics and Analysis↩︎

10 Baseline Design↩︎

For all baselines, we experimented with an open-source model, Qwen3-VL-30B-A3B-Instruct (apache 2.0 license), and a closed-source frontier model, GPT-5.4. Both models can ingest long documents with interleaved text and images and have been shown to perform well across multiple VQA benchmarks. Our high-level goal was to compare our method against a diverse set of attribution-generation methods, yielding a basic VLM prompting baseline for generating attributions.

Additional baselines are organized around two central questions: whether providing full document context (raster + OCR text) gives a VLM an advantage for attribution, and whether replacing visual content with text captions (effectively reducing the task to a text-only problem) is competitive. We further test each setting with and without retrieval augmentation to isolate the contribution of context compression. This yields four baselines and their subsequent variants:

10.0.0.1 VLM

We provide raster image data and document text, along with a batch of QAA’s from the document, and prompt a Visual-Language Model to identify where the provided answer could be sourced.

10.0.0.2 LLM

We provide captions for each image and document text along with a batch of QAAs from the document, and prompt a Language Model to identify where the provided answer could be sourced.

10.0.0.3 RAG

Both RAG variants operate similarly: sources are embedded into a shared space, and the top sources are retrieved against the QA embedding (\(k=5\)). Cohere retrieves the top-5 text chunks and top-5 images independently, while ColQwen retrieves the top-5 full PDF pages.

11 Evaluation Metrics↩︎

We evaluate attribution quality using precision, recall, and \(\mathrm{F1}\) score.

For each item, let \(G\) and \(C\) denote the ground-truth and predicted citation sets, \[G = \mathcal{I}^* \cup \{\mathcal{T}_{i:j}\}, \qquad C = \hat{\mathcal{I}} \cup \{\mathcal{T}_{k:\ell}\},\] where \(\mathcal{I}^*,\, \hat{\mathcal{I}} \subseteq \mathcal{I}\) are the ground-truth and predicted image sets, \(\mathcal{T}_{i:j}\) and \(\mathcal{T}_{k:\ell}\) is the ground-truth and predicted text spans.

11.0.0.1 Citation scoring.

Image citations are scored by exact match: \[\begin{align} \sigma_p(I_m,\, G) &= \mathbb{1}[I_m \in \mathcal{I}^*], \\ \sigma_r(I_m,\, C) &= \mathbb{1}[I_m \in \hat{\mathcal{I}}]. \end{align}\]

Text citations are scored by fuzzy substring matching. Let \(s^* = \texttt{partial\_ratio}(\mathcal{T}_{k:\ell},\, \mathcal{T}_{i:j}) \in [0,1]\).

The match score is discretized to reduce sensitivity to trivial differences: \[d(s^*) = \begin{cases} 1.0, & \text{if } s^* \geq 0.9 \\ 0.5, & \text{if } 0.6 \leq s^* < 0.9 \\ 0.0, & \text{otherwise} \end{cases} \label{eq:discrete}\tag{1}\]

Length ratios penalize precision for over-quoting and recall for under-quoting: \[\begin{align} \sigma_p(\mathcal{T}_{k:\ell},\, G) &= d(s^*)\cdot\min\!\left(1,\frac{|\mathcal{T}_{i:j}|}{|\mathcal{T}_{k:\ell}|}\right),\\ \sigma_r(\mathcal{T}_{i:j},\, C) &= d(s^*)\cdot\min\!\left(1,\frac{|\mathcal{T}_{k:\ell}|}{|\mathcal{T}_{i:j}|}\right). \end{align}\]

11.0.0.2 Precision and recall.

Per-item precision and recall are the mean scores over predicted and ground-truth citations, respectively: \[\begin{gather} \mathrm{P} = \frac{1}{|C|}\sum_{c \in C}\sigma_p(c, G), \qquad \mathrm{R} = \frac{1}{|G|}\sum_{g \in G}\sigma_r(g, C) \\ \mathrm{F1} = \frac{2PR}{P + R} \end{gather}\] Macro-averaged \(\mathrm{P}\), \(\mathrm{R}\), and \(\mathrm{F1}\) are reported over the dataset.

12 MultAttnAttrib↩︎

13 Comparing GPT-5.4 to Qwen3-VL-30B↩︎

14 Comparing GPT-5.4 to MultAttnAttrib↩︎

15 Domain Difficulty Analysis↩︎

15.0.0.1 Domains have no tangible performance impact on intra-baseline relationships.

Generally, the VLM and LLM baselines perform the worst, with Cohere + VLM and Cohere + LLM being similarly better, and MultAttnAttrib, along with its Cohere variant, generally outperforming other methods. Changing the document type does not affect the relationships among the baselines, indicating that our baseline implementations are robust and impartial across the domains tested in our experiments.

15.0.0.2 Academic documents are consistently more difficult to generate attributions for.

Academic documents have a regime-wide unified F1 score of 0.54, with marketing, the second worst performing domain, seeing an approximate 8% increase in F1 scores. This is the highest domain-to-domain jump for unified F1 scores. The reason for this disparity is likely due to the structure of academic/research documents. Redundancy in academic documents is common, as reference material is sparsely distributed, cross-referential, and frequently restated or paraphrased. As a result, we often see over-attribution in a QA pairing, leading to inaccuracies when comparing against the more lightweight ground-truth attributions and to poor F1 results, as seen in 13.

15.0.0.3 Legal documents tend to receive more accurate attributions in comparison to other domains.

We see that there is a 6% domain-to-domain jump in F1 metrics between Legal (with an F1 score of 0.74) and Business. This suggests that the legal domain is relatively easier to generate accurate attributions for. The reason is that references are densely structured within specific clauses, claims, laws, or cases. This allows for fine-grained attributions for QA pairings (attributions that our baseline can locate with more ease) and creates opportunities for better fine-grained attributions.

16 Baseline Findings↩︎

16.0.0.1 Switching to image captions improves performance in GPT baselines, with mixed results in the Qwen baselines.

In the GPT case (13), we observed improvements particularly in precision for the text and multimodal regimes and somewhat in the image-only regime. In the Qwen case (Table 2 and Appendix 13), we see a slight boost in text-only regimes, but degradation in image-only and text + image QAAs. This asymmetry suggests that the open-source model leans more heavily on fine-grained visual representations and is less able to perform attribution reasoning over abstracted textual descriptions of images. This observation directly motivates MultAttnAttrib’s design: rather than mediating images through captions, we read attribution signals off attention over image patches, where the fine-grained visual evidence is already encoded.

16.0.0.2 RAG generally improves metrics in comparison to direct inference, with gains being dependent on regime and model used.

For GPT-5.4, layering Cohere-based RAG on top of LLM nearly closes the gap on text-only attribution, but for Qwen, the same intervention yields only modest gains, even harming performance in the image-only regime. We hypothesize this asymmetry arises because retrieval preselects evidence into a smaller candidate pool, which a stronger generator can exploit but a weaker one cannot. Replacing Cohere with ColQwen as the retriever further degrades performance across all methods and splits, indicating that retrieval quality, not just its addition, drives the gains we observed.

16.0.0.3 Multimodal attribution is challenging and resists frontier gains.

Taking the strongest baseline within each split, GPT-5.4 (Appendix 13) outperforms Qwen3-VL-30B by 35.7 F1 points on text-only and 14.6 points on image-only, but only 10.7 points on combined attribution. The pattern persists under direct baseline comparisons: with VLM, the GPT–Qwen F1 gap is 11.1 points on text-only and 11.5 on image-only, but collapses to just 1.8 on combined attribution Table 2. This suggests that combined attribution exposes a difficulty distinct from those addressed by scale alone—arbitrating between modalities and aggregating partial evidence from each—which current frontier-generation methods do not resolve on their own.

17 LLM Judge Analysis↩︎

17.1 Judge Setup↩︎

To complement token-overlap metrics, we additionally evaluate attribution quality using a multi-judge LLM panel. Each (question, answer, answer_part, citation) tuple is scored by a panel of three GPT-4o judges, each assigned a distinct deliberation persona: a balanced evaluator, a detail-focused critic, and a consensus mediator. Judges share a discussion history and deliberate for up to two rounds, with early termination upon unanimous consensus; the final decision is determined by majority vote. A citation is judged as supportive if it grounds at least one fact in the answer component, and as non-supportive if it contradicts or is entirely unrelated to the attributed claim. We report Relevance and Support as the proportions of citations judged supportive for each method across attribution regimes.

17.2 Judge Results↩︎

18 Head Analysis↩︎

18.0.0.1 Setup.

We score each of the \(N = L \times H = 48 \times 32 = 1536\) attention heads under two methods: Mean Attention Scoring and CMA Scoring (discussed in Section 3.2). Both produce score matrices \(\mathbf{S}^{\text{img}}, \mathbf{S}^{\text{txt}} \in \mathbb{R}^{L \times H}\), which we use to measure cross-modal agreement via IoU and Spearman’s rank correlation (Figures 5–6).

18.0.0.2 Metrics.

Let \(\mathcal{H}^{\text{img}}_k\) and \(\mathcal{H}^{\text{txt}}_k\) denote the top-\(k\) image and text head sets, respectively. \[\text{IoU}(k) \;=\; \frac{|\mathcal{H}^{\text{img}}_k \cap \mathcal{H}^{\text{txt}}_k|}{|\mathcal{H}^{\text{img}}_k \cup \mathcal{H}^{\text{txt}}_k|}.\] Let \(\mathcal{U}_k = \mathcal{H}^{\text{img}}_k \cup \mathcal{H}^{\text{txt}}_k\) and \(r_i^{\text{img}}, r_i^{\text{txt}}\) be the rank of the head \(i\)’s score within \(\mathcal{U}_k\) under specified modality. Spearman’s rank correlation over this union measures how similar text and image modalities order these heads: \[\rho_k \;=\; 1 - \frac{6\displaystyle\sum_{i \in \mathcal{U}_k} \!\bigl(r_i^{\text{img}} - r_i^{\text{txt}}\bigr)^2}{|\mathcal{U}_k|\bigl(|\mathcal{U}_k|^2 - 1\bigr)}.\]

18.0.0.3 Score heatmaps.

19 QAA Rubrics↩︎

1: Not answerable from channel
- Answer is unsupported, contradictory, or mostly hallucinated.

2: Severely weak support
- Only a tiny fragment is grounded; core claim remains unsupported.

3: Partially answerable
- Some grounded signal exists but major claim elements are missing or uncertain.

4: Moderately answerable
- Core claim is plausible and partly supported, but specificity/precision is limited.

5: Strongly answerable
- Main claim is supported with minor uncertainty or missing detail.

6: Very strongly answerable
- Precise and well-supported with clear channel evidence.

7: Near-certain answerability
- Exact, unambiguous, and fully supported by clear, legible evidence (rare).

1: Poor
- Unsupported or weakly grounded answer; evidence is missing, contradictory,
  or largely hallucinated.

2: Acceptable
- Some support is present, but grounding is limited in precision,
  completeness, or clarity.

3: Good
- Answer is clearly supported by the paragraph with reasonably specific
  and relevant evidence.

4: Excellent
- Fully supported, precise, and unambiguous; evidence directly and
  convincingly grounds the answer.

1: Absent
- No visual evidence of the entity in the image.

2: Unlikely
- Faint or ambiguous trace; probably refers to something else.

3: Possible
- Entity may be present, but evidence is weak or unclear.

4: Probable
- Entity appears to be present with only minor uncertainty.

5: Clear
- Entity is unambiguously and prominently visible.

1: Redundant
- One modality alone contains everything needed to answer the question;
  the other adds nothing essential.

2: Weak synergy
- One modality provides most of the answer; the other contributes only
  minor corroborating detail.

3: Good synergy
- Both modalities contribute meaningfully and neither alone is sufficient,
  but the split is somewhat uneven.

4: Strong synergy
- Each modality provides essential, non-overlapping information; the answer
  can only be constructed by combining both.

20 Prompts↩︎

20.1 QAA Generation↩︎

20.1.1 Image-only↩︎

You are an expert technical Q\&A generator for an **image-only** benchmark.

**Contract** You receive non-text rasters plus grounding text chunks selected by image-text similarity from the same document. Use grounding chunks **only** to infer document domain, terminology, and what is in-scope. **Do not** treat grounding text as evidence for answers: every **specific factual claim** in each answer (numbers, units, labels, named entities, relationships) must be **legibly visible or unambiguously readable in the rasters**. Do **not** copy long phrases or whole sentences from grounding chunks into answers---paraphrase minimally and anchor claims in what the raster shows.

**General-topic questions**  
Questions must sound like ordinary domain questions (e.g. clinical, engineering, policy) a reader would ask without knowing a figure exists---see Q1. They may name domain constructs (pathways, metrics, components) but must **not** point at layout, panels, or the carrier medium.

**Generation goals**
- Generate high-value QA coverage across distinct labels, values, structures, mechanisms, relationships, comparisons, and findings.
- Prefer reasoning-first questions (mechanism, causality, comparison, procedure, quantity, trade-off, failure mode, subsystem links).
- Use only entities/readouts that are reliably legible; omit blurry or ambiguous anchors.

**QUESTION RULES**
Q1 --- SOURCE-AGNOSTIC \& STANDALONE  
The user does NOT know that this specific document, page, figure, or paragraph exists.  
The question must make sense on its own outside of this document and must NOT assume or mention access to a particular figure, page, section, table, report, image, text, passage, or document.  
Forbidden phrasing includes (non-exhaustive):  
"in the image", "in this image", "from the image", "in the figure", "in this figure", "from the figure", "in the diagram", "in this diagram", "from the diagram", "in the table", "in this infographic", "in the infographic", "from this infographic", "shown in", "depicted", "illustrated", "pictured", "visual representation", "in the text", "in this text", "in the passage", "according to", "mentioned in", "described in", "as seen", "as shown", "the figure shows", "the image shows", "the diagram shows", "the infographic shows", "the document states", "the report says", "in this document", "in this report", "in this section", "on this page", "on the page", "in the screenshot", "as displayed".  
GOOD examples (general topic, no carrier):
- "Which operating mode corresponds to the highest throughput value?"
- "How does the reported failure rate change after the calibration step?"
- "Under the stated inclusion criteria, which comorbidity category is excluded from randomization?"
BAD examples:
- "What does the graph show about yield?" -> drop.
- "What label appears in the lower panel?" -> drop.
- "What anatomical landmark does the measurement line terminate at?" -> drop.
- "According to this infographic, which region had the highest demand?" -> drop.
- "What trend is shown in this screenshot for month-over-month growth?" -> drop.
- "From the figure above, what is the operating temperature?" -> drop.

Q2 --- DOMAIN GROUNDED  
Questions must target verifiable facts (measurements, labels, values, relationships, mechanisms, comparisons, classifications, findings). Ask as if the reader already knows there is source material and wants scientific/technical content.
GOOD examples:
- "What was the peak activation observed under condition X?"
- "Which region showed the greatest fold-change between groups A and B?"

Q3 --- NO PERCEPTUAL QUESTIONS  
Do not ask about colors, spatial layout, positions, background, lighting, shadows, aesthetics, textures, or appearance-only size judgments.
GOOD examples:
- "What pressure range is specified for safe operation?"
- "Which subsystem is identified as the bottleneck in the described workflow?"
BAD examples:
- "What color is the highlighted region?" -> drop.
- "What two objects appear together on the left?" -> drop.
- "What color are the tiles on the structure?" -> drop.

Q4 --- NO CO-OCCURRENCE QUESTIONS  
Do not ask questions where the only answer is that two things appear together.
GOOD examples:
- "What functional dependency is described between the valve setting and outlet flow?"
- "Which component failure would most directly explain the observed pressure drop?"
BAD examples:
- "What concept is the researcher shown alongside?" -> drop.
- "What is the relationship between the tractor and Food?" -> drop.

Q5 --- NO HALLUCINATION  
Do not assert facts not supported by legible raster content.

Q6 --- MAXIMIZE DIVERSITY  
Cover as many distinct supported facts as possible; avoid near-duplicate rewordings.
GOOD examples:
- Ask one quantitative question, one mechanism question, and one comparison question when support exists.
- Prefer new facts over paraphrasing an earlier question about the same value or label.
Hard uniqueness requirement:
- Do not emit duplicate or near-duplicate questions (including paraphrases with the same answer target).
- If two candidates ask essentially the same thing, keep only the more specific one.

**ANSWER RULES**
A0 --- RASTER-ANCHORED SPECIFICS  
Concrete claims in the answer must be justified by **legible raster content** (printed text, axis tick values, table cells, diagram labels). If grounding text suggests a fact but the raster does not clearly show it, **omit** that pair.

A1 --- FACTUAL AND PRECISE  
State exact value/label/name/relationship using domain terminology.
GOOD examples:
- "Peak torque is 245 N$\cdot$m at 1800 rpm."
- "The limiting stage is the heat-exchanger loop, which caps flow to 3.2 L/min."

A2 --- SELF-CONTAINED  
Answer must be informative to a domain expert without seeing the source raster.
GOOD examples:
- "The alarm state indicates over-temperature protection triggered by sustained inlet values above 90 C."
- "The procedure requires depressurization before seal replacement to prevent cavitation damage."

**Process**
1) Produce `domain_grounding` (2--4 sentences) summarizing subject matter and terminology from **raster-legible** content, aligned with grounding chunks for domain only.  
2) Set `is_relevant` false only for blank/decorative/unusably degraded content.  
3) If relevant, emit all strong non-redundant `qa_pairs`. Each pair **must** include:
- `question`, `answer`, `type` in \{relational, inferential, procedural, quantitative\}
- `answer_evidence`: one of `"visual"` (specific values/labels/readouts in the raster are **essential** to justify the answer) or `"visual_plus_general"` (answer combines one raster-specific fact with a short domain-general clause that is still consistent with the raster)
- optional `evidence_anchor`

**Output** Raw JSON only. Relevant: `domain_grounding`, `is_relevant` true, `relevance_rationale`, `qa_pairs`. Else `is_relevant` false, `qa_pairs` [].

20.1.2 Text-only↩︎

You are a high-quality QA data generator.

Given a single paragraph of text, you must generate question-answer pairs for reading-comprehension style evaluation.

Each triplet must satisfy ALL of these rules:
1. The answer MAY be paraphrased (it does not need to be copied verbatim).
2. The answer MUST be fully supported by the paragraph. Do NOT add facts not present in the paragraph.
3. The answer MUST be between 12 and 25 words long (inclusive).
4. The question MUST require reading comprehension of the paragraph, not just simple word or name lookup.
5. Each question MUST be answerable solely from the given paragraph, without any external knowledge.
6. Triplets must be diverse: do NOT ask multiple questions that can be answered with nearly the same statement.
7. NEVER refer to 'the paragraph', 'this paragraph', 'the text', 'the document', or similar meta wording in the question.

You MUST output valid JSON only, with a top-level key 'triplets' containing a list of objects with keys: 'question' and 'answer'.

20.1.3 Multimodal↩︎

You are an expert at creating challenging, non-trivial questions about scientific and technical content.

Your questions will be used for MULTIMODAL RETRIEVAL evaluation: a user has a genuine information need, submits their question, and the system must find the right document.
For this to work, the question must be something a user would ACTUALLY ASK -- not a pure blank-fill where the answer adds nothing the question did not already contain.

**QUESTION RULES**

RULE 1 -- SOURCE-AGNOSTIC & STANDALONE
The user does NOT know that this specific document, page, figure, or paragraph exists.
They only have an information need in the world. The question MUST make sense on its own,
outside of this document, and MUST NOT assume or mention that the user has access to a
particular figure, page, section, table, or report.

Forbidden behaviour:
- Do NOT reference the image, text, figure, diagram, passage, page, or document.
- Do NOT write questions that would only make sense if the user could see "this page", "this figure", "this document", or "this section".

Forbidden phrases (non-exhaustive, always rewrite if they appear):
- "in the image", "in this image", "in the figure", "in this figure", "in the table", "shown in", "depicted", "illustrated", "pictured", "visual representation"
- "in the text", "in this text", "in the passage", "according to", "mentioned in", "described in", "as seen", "as shown", "the figure shows", "the document states", "in this document", "in this report", "in this section", "on this page".

RULE 2 -- ANSWER MUST ADD NEW FACTUAL CONTENT
The answer must introduce at least one piece of information the question did not already contain: a specific number, measurement, date, named entity, mechanism, comparison, or qualifying detail that cannot be read directly out of the question.

Shared entity names, technical terms, and proper nouns between question and answer are FINE - these are what retrieval systems use to find the right document and image.
What is forbidden is an answer that is a pure blank-filling completion with nothing new.

JEOPARDY (WRONG) - answer adds nothing new:
  Q: "Which VLT drive controls the high-pressure pump on the ROV?"
  A: "The VLT drive controls the high-pressure pump on the ROV."
  Why wrong: the answer merely echoes the question with no new fact added.

GENUINE (RIGHT) - answer adds a new fact using shared terminology:
  Q: "What does the VLT drive control on the ROV?"
  A: "The VLT drive controls the high-pressure pump mounted directly on the ROV, delivering jetting water to the sword at up to 200 bar."
  Why right: "mounted directly on the ROV", "delivering jetting water", "200 bar" are
  all new facts not present in the question. Shared terms like "VLT drive" and "ROV" are expected and help retrieval.

RULE 3 -- REQUIRES BOTH MODALITIES
The answer must be impossible to construct from either the image OR the text alone.
Ask for things that only exist at the intersection: a number visible in a diagram but explained in the text; a species identified by visual features but located by the text; a mechanism shown in a schematic but described in prose; a comparison between what is labelled and what is measured.

RULE 3B -- NO ANNOTATION-DEPENDENT QUESTIONS
Do not generate questions that are only answerable because of a specific graphical annotation - an arrow, measurement line, bounding box, bracket, callout, or pointer - and that ask what the annotation points to, originates from, or terminates at.
The question must be independently answerable from domain knowledge, not from knowing where a graphical mark happens to appear.

Forbidden examples:
  - "What anatomical landmark does the measurement line terminate at?" -> DROP
  - "What does the arrow on the left indicate?" -> DROP
  - "What component is the callout pointing to?" -> DROP

RULE 4 -- ASK FOR SPECIFIC FACTS, NOT ENTITY NAMES
Prefer questions that ask HOW, HOW MANY, WHY, WHAT DOES X DO, WHAT DISTINGUISHES X FROM Y, UNDER WHAT CONDITIONS, rather than WHICH X / WHAT IS THE NAME OF X.
If you must ask "what is X", make sure the question does not already describe X so fully that only one possible answer exists.

RULE 5 -- ANSWER MUST ADD NEW FACTUAL CONTENT
The answer must introduce at least one new fact not already stated in the question: a number, measurement, mechanism, named entity not in the question, or qualifying detail.
Shared proper nouns and technical terms between question and answer are EXPECTED and FINE -- they are the vocabulary retrieval systems use to find the right document. Rewrite only when the answer is a pure blank-fill that adds nothing new at all

Generate exactly {n_questions} question-answer pairs.
Output ONLY raw JSON -- no markdown fences, no preamble.

20.2 QAA Filtering↩︎

20.2.1 Image-only↩︎

You certify **image-only** Q\&A using the bundled non-text raster and the QUESTION + REFERENCE ANSWER (no separate document text).

**GLOBAL --- rationales** No image/figure/page/diagram/photo/chart/graph/
table/slide/panel/process/map/
screenshot/infographic/; no shown/depicted/visible/here/this/that/left/
right/above/below/look at. Use ``question text'', ``reference answer'', ``supporting channel'', ``pair''.

**Ranking** Answerability 1--7 is the primary rank key; 7 is rare; typical acceptable pairs 4--6. Optional per-document **retention cap** is configured outside this prompt (0 = keep all certified rows).

**What answerability means (operational definition)**  
Answerability is the extent to which the QUESTION can be answered correctly, specifically, and unambiguously from the bundled raster channel, with the REFERENCE ANSWER aligned to what that channel supports.  
- Judge support from legible technical content only (readable labels, values, structures, and explicit relationships).  
- Penalize when the answer relies on speculation, unstated assumptions, weak visual impressions, or information not recoverable from the raster.  
- Penalize when question scope and answer scope do not match (overclaiming, added details, wrong granularity).  
- Penalize when the REFERENCE ANSWER could be written as **generic domain boilerplate** without checking **specific marks, numbers, or labels** in the raster (no image-tied specifics).  
- This is not fluency scoring; a well-written but unsupported answer should still score low.

**Answerability score rubric (1-7)**  
- 1: Not answerable from channel; answer is unsupported, contradictory, or mostly hallucinated.  
- 2: Severely weak support; only tiny fragment is grounded, core claim remains unsupported.  
- 3: Partially answerable; some grounded signal exists but major claim elements are missing or uncertain.  
- 4: Moderately answerable; core claim is plausible and partly supported, but specificity/precision is limited.  
- 5: Strongly answerable; main claim is supported with minor uncertainty or missing detail.  
- 6: Very strongly answerable; precise and well-supported with clear channel evidence.  
- 7: Near-certain answerability; exact, unambiguous, and fully supported by clear legible evidence (rare).

**Hard floors** If triggered: `answerability`=1; align passes; cite rule id (QR1--QR8) in a rationale.  
QR1 question violates SOURCE-AGNOSTIC \& STANDALONE rule (references/assumes a specific image/text/figure/document/page/section/
table/report or uses forbidden source phrases) \textperiodcentered{} QR2 circular answer \textperiodcentered{} QR3 annotation-dependent/layout-dependent question \textperiodcentered{} QR4 trivial/low-value fact \textperiodcentered{} QR5 instance facts not in image channel \textperiodcentered{} QR6 perceptual or structure-appearance question \textperiodcentered{} QR7 co-occurrence-only question \textperiodcentered{} QR8 panel-specific reference.

**Axes**  
1 **Answerability** --- channel vs Q+A; downgrade unsupported claims.  
2 **source_ref_pass** --- evaluate the QUESTION only using SOURCE-AGNOSTIC \& STANDALONE.  
Fail if the question references the attributed source/carrier (image, figure, diagram, infographic, screenshot, text, passage, document, page, section, table, report), assumes access to ``this'' material, or uses forbidden source phrases such as ``in the image'', ``from the image'', ``in this diagram'', ``the diagram shows'', ``in this infographic'', ``the infographic shows'', ``shown in'', ``according to'', ``as shown'', ``the document states'', ``the report says'', or ``on this page'' (fail $\rightarrow$ QR1).  
Do not fail merely because the question mentions domain entities; fail only when the wording depends on or points to a specific source artifact.
3 **image_quality_pass** --- fail if channel lacks readable domain signal where the pair needs it.  
4 **triviality_pass** --- fail shallow naming / label echo without reasoning.

**Hard-floor examples (pairs that should trigger floors / very low answerability)**
- QR1: ``What does the graph show about yield?'', ``What is the label in the lower panel?''
- QR1: ``According to this infographic, what category dominates?'', ``From the screenshot, what value is displayed in the top-right widget?''
- QR3: ``What does the arrow on the left indicate?'', ``What anatomical landmark is the measurement line originating from?''
- QR6: ``What color is the highlighted region?'', ``What color are the tiles on the structure?''
- QR7: ``What concept is the person shown in relation to?'', ``What is the relationship between the tractor and Food?''
- QR8: ``What is the significance of label IVV in the lower panel?'', ``What structure is labeled in panel B?''
- Visual-layout answer penalty: ``The cell is shown in the context of apoptosis.'' -> score in 1--2 range severity.

**GOOD certification examples** (question text avoids carrier/deictic/source phrasing; answer matches legible technical content; expect answerability \textasciitilde{}5--6+, passes aligned with content)
- Q: ``What is the maximum rated load in kilonewtons?'' \textperiodcentered{} REF: ``Maximum rated load is 120 kN.'' $\rightarrow$ Strong if the value appears clearly on a label/spec table in the raster.
- Q: ``Which intervention arm achieved the higher median survival at 24 months?'' \textperiodcentered{} REF: ``Arm B had higher median survival at 24 months than Arm A.'' $\rightarrow$ Strong if survival or a chart encodes the comparison without guessing.
- Q: ``What step immediately precedes the pressure-relief sequence in the flow diagram?'' \textperiodcentered{} REF: ``The purge cycle completes immediately before the pressure-relief sequence.'' $\rightarrow$ Strong if the workflow order is unambiguous in the figure.

**BAD certification examples** (expect floor QR codes, `source_ref_pass` false, or answerability 1--3 as appropriate)
- Q: ``What trend does the figure illustrate for cohort 2?'' \textperiodcentered{} REF: ``Cohort 2 declines after week 6.'' $\rightarrow$ BAD (QR1 + weak grounding if `figure' is the only anchor).
- Q: ``What is written on the sticker in the photo?'' \textperiodcentered{} REF: ``The sticker says `authorized personnel only'.'' $\rightarrow$ BAD (entity/carrier wording in question; fail source_ref / QR1-class).
- Q: ``According to this infographic, which segment has the highest share?'' \textperiodcentered{} REF: ``The enterprise segment has the highest share.'' $\rightarrow$ BAD (QR1 source-reference phrasing).
- Q: ``From the screenshot, what error code is shown?'' \textperiodcentered{} REF: ``Error code E-417 is shown.'' $\rightarrow$ BAD (QR1 source-reference phrasing).
- Q: ``What does the image above say about the pressure threshold?'' \textperiodcentered{} REF: ``It says the threshold is 2.5 bar.'' $\rightarrow$ BAD (QR1 source-reference + deictic wording).
- Q: ``Is the arrow pointing up or down?'' \textperiodcentered{} REF: ``The arrow points up.'' $\rightarrow$ BAD (layout/deictic; QR3/QR6-class).
- Q: ``What color is the safety railing?'' \textperiodcentered{} REF: ``The railing is yellow.'' $\rightarrow$ BAD (perceptual; QR6).
- Q: ``What two logos appear side by side in the header?'' \textperiodcentered{} REF: ``Company A and Company B.'' $\rightarrow$ BAD (co-occurrence / appearance; QR7).
- REF alone: ``The measurement is taken at the inlet port shown on the right.'' $\rightarrow$ BAD answer pattern (spatial/deictic in the reference answer; penalize answerability and align rationales).

**Output** Raw JSON: answerability (int), answerability_rationale, source_ref_pass, source_ref_rationale, image_quality_pass, image_quality_rationale, triviality_pass, triviality_rationale.

20.2.2 Text-only↩︎

You are a strict verifier for QA data.

Given a paragraph, a question, and an answer, decide if the answer is fully supported by the paragraph.
Also identify the exact verbatim evidence span inside the paragraph that supports the answer.

Rules:
- Output JSON ONLY with keys: supported (boolean), quality_score (int), extractive_answer (string).
- Do NOT rewrite the question or answer. Judge only.
- extractive_answer MUST be a verbatim substring of the paragraph and MUST be 12-25 words.
- If supported=true, extractive_answer is the evidence span supporting the answer.
- If supported=false, extractive_answer should still be the best contiguous verbatim span
  that relates to the question (or empty string if none).
- quality_score is on a 1-4 scale: 1 poor, 2 acceptable, 3 good, 4 excellent.

20.2.3 Multimodal↩︎

You are a QA quality evaluator for multimodal scientific document comprehension.
You will rate question-answer pairs on a 1-7 scale based purely on QA quality: how well-formed, specific, accurate, and non-trivial the pair is.
Do NOT factor in whether both modalities are required -- that is assessed separately.
Use the FULL range of scores. Most well-formed pairs should score 4-6; reserve 7 for truly exemplary pairs and 1-2 for pairs that should be excluded.

-- HARD FLOOR RULES (immediately set score = 1, stop evaluation) --

QR1 - SOURCE-REFERENCE VIOLATION
The question MUST NOT reference the source document, image, figure, table, page, or section.
Banned phrases: "in this image", "in the figure", "shown in", "according to this document", "on this page", "in the text", "the diagram shows", "as depicted", "based on", "as per", and any equivalent phrasing that assumes the reader can see a specific source.
-> ANY occurrence of source-referencing language: HARD FLOOR score = 1.

QR2 - CIRCULAR ANSWER (no new factual content)
The answer must introduce at least one new fact not already stated in the question: a new number, measurement, named entity, mechanism, or qualifying detail.
Shared proper nouns and technical terms between question and answer are PERMITTED - they are useful for retrieval and expected in domain-specific QA.
The floor triggers only when the answer is a pure blank-filling completion that adds no new information whatsoever. -> HARD FLOOR score = 1.

QR3 - ANNOTATION-DEPENDENT QUESTION
A question whose answer is only knowable from a graphical annotation (arrow, callout,
measurement line, bounding box) rather than domain knowledge. -> HARD FLOOR score = 1.

You MUST provide 'Rule:' (the violated rule ID or "PASS"), 'Evaluation:', and 'Total rating:' in your answer.

You are a strict multimodal grounding evaluator for document QA pairs.
Your task is to verify whether each modality (image and text) individually grounds the specific entities and claims in the answer -- NOT just whether they are topically related.

Critical distinction:
- An image of product packaging does NOT ground an answer about a biological process, even if both relate to the same general topic.
- An image must VISUALLY DEPICT, LABEL, or MEASURE the specific entities named in the answer to count as meaningful image grounding.
- The score reflects the WEAKEST modality: if the image does not ground any key answer entity, the overall score is low even if the text is perfect.

You MUST provide 'Image grounding:', 'Text grounding:', 'Evaluation:', 'Reason:', and 'Total rating:' in your answer.
The 'Reason:' field must be exactly one sentence, max 30 words, starting with the weakest modality or "PASS".

20.3 Baselines↩︎

20.3.1 General Baseline Prompt↩︎

Given the custom preambles for VLM/LLM approaches, our baseline system prompt is as follows:

Do not force use of a source type. Include text evidence only when supportive text is present; include image indices only when supportive visual evidence is present.

## Evidence extraction

- `text_quote`: copy the **complete paragraph** from the provided text that most directly supports the reference answer. Use the paragraph exactly as it appears --- no truncation, no trimming to sub-paragraph fragments. If the paragraph exceeds 150 words, copy the most relevant contiguous sentences within it, but include at least the full sentence containing the key fact plus one sentence of context on each side. Must be an exact substring of the provided text. If no text evidence is used, return `""`.
- `image_indices`: list of integer indices that refer to the provided image labels `[Image k]` only. Use unique integers with at most 2 entries. If no image evidence is used, return `[]`.

## Constraints
- `image_indices`: sorted, unique integers. At most 2 entries.
- If text evidence is used, `text_quote` must be non-empty.
- If image evidence is used, `image_indices` must be non-empty.
- It is valid to use text only (`image_indices = []`), images only (`text_quote = ""`), or both.

## Output format (strict)

Reply with **only** a single JSON object (no markdown fences, no commentary, no extra keys).

The user message will repeat the exact question and reference answer; echo them back in your JSON as `question` and `reference_answer` for traceability.

The JSON object must contain **exactly** these keys:

- `question` (string)
- `reference_answer` (string)
- `text_quote` (string)
- `image_indices` (array of integers): unique integers, at most 2 entries
- `rationale` (string): One sentence describing which selected source material supports the answer.

20.3.2 VLM Preamble↩︎

When using raw raster content, we can apply the modification below, followed by the general prompt.

You are given **document images** (labels `[Image 0]`, `[Image 1]`, ... in order) and **document text**. Use only this provided context as evidence for attribution.

The user message ends with a **benchmark question** and a **reference answer**.
Your job is **not** to re-answer the question. Your job is to select the source material from the provided images and/or text that supports the reference answer, then attribute that evidence.

## Source material selection
Review all provided evidence and pick what directly supports the reference answer:

1. **Scan document images** and identify indices whose visual content (figures, tables, charts, diagrams, schematics, photos, labeled components, layouts) is relevant and supportive of the reference answer.
2. **Scan document text** and find the passage that most directly states or supports the key fact(s) in the reference answer.
3. Use whichever evidence is actually supportive:
   - text only,
   - images only, or
   - both text and images.

20.3.3 LLM Preamble↩︎

When using captions instead of raw raster content, we can apply the modification below, followed by the general prompt.

You are given **document image captions** (for labels `[Image 0]`, `[Image 1]`, ... in order) and **document text**. Use only this provided context as evidence for attribution.

The user message ends with a **benchmark question** and a **reference answer**.
Your job is **not** to re-answer the question. Your job is to select the source material from the provided captions and/or text that supports the reference answer, then attribute that evidence.

## Source material selection
Review all provided evidence and pick what directly supports the reference answer:

1. **Scan image captions** tied to `[Image k]` and identify indices whose described visual content is relevant and supportive of the reference answer.
2. **Scan document text** and find the passage that most directly states or supports the key fact(s) in the reference answer.
3. Use whichever evidence is actually supportive:
   - text only,
   - images only, or
   - both text and images.

References↩︎

---

1.   Equal contribution.↩︎