1 Introduction↩︎

Vision language models (VLMs) exhibit impressive abilities in visual perception [1]–[3], grounding [4]–[6], and reasoning [7]–[9], by adapting powerful Large Language Models (LLMs) [10]–[12]. Despite these advances, it remains unclear whether such "remarkable capabilities" can consistently transfer to realistic, real-world scenarios [13], [14]. A significant gap lies in the stark misalignment between existing benchmarks—typically static and conducted in controlled, laboratory settings—and the realistic usage of MLLMs, which is inherently dynamic, multi-turn, and contextualized. To address this misalignment, we revisit MLLM evaluation as a tripartite interplay of (i) the evaluatee model, (ii) an examiner, and (iii) a grader. As shown in Figure 1, the examiner interacts with the evaluatee through iterative questioning, and the grader scores the resulting dialogue to produce a final performance measure.

Building on this framing, we propose CEDI (Contextualized Evaluations of MLLMs through Dynamic multi-round Interactions). CEDI draws inspiration from evaluative interviews in human assessment: rather than relying on static, paper-based tests, a semi-structured interview allows the examiner to adapt question type and difficulty based on the model’s responses, eliciting more faithful evidence of capability. To better approximate real-world usage, CEDI introduces context modeling to operationalize the conversation over user scenarios. To perform effective semi-structured interviews, CEDI leverages a graph-based representation to provide a manipulable state space for both the examiner and the grader, enabling strategies ranging from clarification requests to adversarial probes.

We apply CEDI to visual hallucination [15]. Closely analogous to the hallucination problem in LLMs [16], [17], visual hallucination refers to cases where models generate outputs that are not faithful to the image content [18], such as describing non-existent objects, hallucinating attributes, or fabricating incorrect relations [19]–[21]. Existing evaluation approaches for visual hallucination often adopt static captioning tasks in closed settings [15] or rely on simple binary questions to assess object presence [22]–[24]. Other works validate the correctness of object descriptions [25] or generate open-ended image captions as a proxy for hallucination detection [26], [27]. While informative, these methods struggle to capture the nuanced, contextualized, and multi-turn nature of real-world usage where visual hallucinations arise as critical problems.

Empirical results across diverse settings, domains, and datasets show that CEDI consistently exposes more hallucinations, achieves broader visual coverage, and produces interactions and failures that are more relevant to realistic user contexts than captioning, binary-question, and prompt-only baselines. For instance, on the Synthetic Visual Genome (SVG) dataset [28] spanning three sources (VG/COCO/ADE20K), CEDI elicits substantially more hallucinations than caption- and prompt-only baselines: on every evaluatee, Coverage rises by \(3\text{--}19\) pp, the VALOR [27] per-turn hallucination rate by \(5\text{--}23\) pp, and the scene graph (SG)-based Graph Edit Distance (GED) is \(1.2\text{--}2.9\times\) higher; against the caption baseline, CHAIR\(_I\) further rises by \(8\text{--}18\) pp. The Pearson correlation between the hallucination component of our SG-based grader (GED\(_{\mathrm{hal}}\)) and human-annotated hallucination counts reaches \(\rho\!=\!0.22\text{--}0.33\) consistently across all four annotated evaluatees, whereas MMHal [29] fluctuates between \(-0.02\) and \(0.33\) and CHAIR\(_I\) turns negative for three of the four models. Our analyses further reveal contextualization, dialogue history, and multi-turn interaction as key drivers for eliciting more hallucinations: models hallucinate more frequently under long contexts, reinforce earlier unsupported claims across turns, and struggle most when required to reject false premises or abstain from answering.

Looking forward, we envision CEDI as a foundation for moving beyond static, single-turn benchmarks toward evaluations realistically grounded in dynamic interactions, offering a more faithful basis for diagnosing model failures and guiding the development of VLMs that remain reliable under context-dependent, multi-turn realistic usage.

2 Related Work↩︎

2.1 Evaluation and Benchmarking MLLMs↩︎

Static Evaluation. Static benchmarks remain the dominant paradigm for MLLM evaluation, offering standardized comparisons across tasks. A wide range of benchmarks has been proposed to evaluate the growing capabilities of MLLMs, spanning vision question answering (VQA) [30], [31], image captioning [32], [33], and OCR/text understanding [34]. More recently, benchmarks have shifted toward higher-level reasoning, such as MathVerse [35] and ScienceQA [36], which emphasize multimodal reasoning in scientific domains. Other benchmarks, including GQA [37], OK-VQA [31], OwlEval [38], MMBench [39], and MMMU [40], evaluate visual perception and reasoning through large collections of question–answer pairs. To improve coverage, SEED-Bench [41] and SEED-Bench-2-Plus [42] extend evaluation to video and text modalities. Despite their breadth, static benchmarks increasingly suffer from performance saturation and data contamination, limiting their ability to expose model failures. More fundamentally, they ignore the dynamic, interactive, and contextualized nature of real-world MLLM use, which further undermines their reliability as measures of practical effectiveness.

Dynamic Evaluation. While most prior work relies on single-turn benchmarks, recent studies have begun to evaluate MLLMs in interactive, conversational settings [43]. MMDU [44] targets multi-turn VLM dialogues involving multiple images and open-ended question answering. ConvBench [45] organizes interaction sequences to progressively probe perception, reasoning, and creativity, aiming to mimic how these cognitive processes unfold during human conversation. MMRC [46] further assesses open-ended conversational capabilities. While these efforts move evaluation closer to real-world use by incorporating multi-image inputs and multi-turn interactions, they remain constrained by static rubrics and largely pre-scripted conversation structures [43], [44]. Moreover, they offer limited mechanisms for explicit context modeling, which restricts their ability to reflect the situated, goal-driven nature of practical MLLM deployments.

2.2 Visual Hallucination↩︎

Visual hallucination refers to outputs that are not faithful to the given image [15]. VLMs may generate plausible but non-existent objects, attributes, or relations, often driven by spurious co-occurrence patterns learned during training rather than visual evidence [47]. This undermines the reliability of MLLMs in real-world deployment, especially in critical applications such as autonomous driving.

Hallucination Evaluation. Therefore, evaluating visual hallucinations has attracted increasing attention, spanning evaluation frameworks [23], [48], [49], benchmarks [22], [24], [25], [29], [50], [51], grading methods [15], [52]–[54], and datasets for training hallucination detectors [26]. However, most approaches are single-turn (e.g., caption-based [15]) and rely on coarse binary labels (e.g., yes/no questions on object presence [22]), which poorly reflect multi-turn, context-dependent interactions. Moreover, isolated prompts make it difficult to assess whether models maintain consistency and grounding over a dialogue. Although some work considers instruction following [48] and informativeness [29] in LLM-based evaluation, it rarely differentiates hallucination severity under varying contexts.

3 Methodology↩︎

Three-party Interplay. To enable dynamic and contextualized evaluation, CEDI recasts evaluation as a three-party interaction among an evaluatee model, an examiner, and a grader (Figure  1, left). The examiner elicits capability evidence from VLMs through multi-turn interaction with the evaluatee, while the grader scores the model responses, flags potential hallucinations, and produces a final performance score. Building upon this framing, CEDI develops a contextualized, interactive examiner (Section 3.1) and an SG-based grader (Section 3.2).

3.1 Dynamic and Contextualized Examiner↩︎

Drawing inspiration from interactive human assessment, we move beyond static, paper-based tests and adopt a semi-structured evaluative interview in which the examiner adapts question type and difficulty based on the model’s responses, eliciting more faithful evidence of capability. Given ground-truth (GT) visual annotations (and optionally the image), the evaluator models real-world context and conducts a multi-turn interview, dynamically adjusting its questioning strategy according to the model’s prior answers. An overview is demonstrated in Figure 2.

Structure Representation of Image. To enable an adaptive semi-structured interview systematically, the examiner leverages the structured representation of visual content. We propose to adopt Scene Graphs [55] to store visual information and guide the examiner’s traversal over visual evidence across multiple rounds of interaction. Formally, for an image \(I\), we define its scene graph as \[\small G = (V, E, \mathcal{A}),\] where \(V=\{v_1,\dots,v_{|V|}\}\) is a set of nodes corresponding to object instances, \(E \subseteq V \times \mathcal{R} \times V\) is a set of directed, typed edges capturing relations, and \(\mathcal{A}\) is an attribute mapping. Each node \(v \in V\) is associated with an object category \(c(v) \in \mathcal{C}\) and a (possibly empty) set of attribute–value pairs \[\small \mathcal{A}(v) = \{(a_k, x_k)\}_{k=1}^{m_v}, \quad a_k \in \mathcal{K}, x_k \in \mathcal{X}_{a_k},\] where \(\mathcal{K}\) denotes the attribute vocabulary (e.g., color, size) and \(\mathcal{X}_{a_k}\) is the value space for attribute \(a_k\). Each edge is a triple \((v_i, r, v_j) \in E\) with relation type \(r \in \mathcal{R}\), representing a predicate \(r(v_i, v_j)\) (e.g., on, holding, next to). This representation yields a manipulable state space for both the examiner and the grader. It supports question generation that is explicitly grounded in faithful visual content—rather than driven solely by language priors—and enables diverse interviewing strategies, ranging from clarification requests to adversarial probes (See Question Types part).

Context Modeling. Recall how humans practically utilize MLLMs. As illustrated in Figure 2, users typically interact with MLLMs from a first-person scenario that combines context (visual or linguistic) with an explicit objective (e.g., “I am visually impaired and want to locate the restroom in this restaurant,” or “I am driving in a park and need to find a parking lot.”). To reflect this situated, goal-driven aspect, our examiner incorporates context modeling that embeds each image in a plausible user-perspective scenario with a clear objective. Concretely, given an image \(I\) and its scene graph \(G=(V,E,\mathcal{A})\), the examiner treats \(I\) as the user’s immediate environment and constructs a scenario \(s\) with goal \(g\) that is grounded in objects, attributes, and relations encoded in \(G\). The goal is designed to require reasoning over multiple elements (e.g., querying a subset of \(V\) and/or edges in \(E\)), rather than relying on single-object recognition.

In parallel, the examiner selects a context-relevant node subset \(V_s \subseteq V\) that are highly relevant to the generated scenario and objective, forming a sub-graph \(G_s = (V_s, E_s, \mathcal{A}_s),\) where the edge set is restricted to relations among selected nodes \(E_s = {(v_i, r, v_j) \in E \;: \;v_i \in V_s, \;v_j \in V_s},\) and the attribute mapping is the restriction of \(\mathcal{A}\) to \(V_s\), \[\small \mathcal{A}_s = \mathcal{A}\big|_{V_s}, \quad \text{i.e., } \mathcal{A}_s(v)=\mathcal{A}(v)\;\;\forall \;v\in V_s.\] During the multi-round interview, question generation is constrained to a visual traversal over \(V_s\): at turn \(t\), the examiner chooses a node \(v_t \in V_s\) and generates questions anchored to \(v_t\) and its local neighborhood in \(G\) (i.e., its attributes \(\mathcal{A}(v_t)\) and incident relations in \(E\)). This constraint keeps the dialogue explicitly grounded in the simulated context.

Multi-turn Conversation as Traversal over the Scene Graph. The goal \(g\) together with the contextual sub-graph \(G_s=(V_s,E_s,\mathcal{A}_s)\) provides a structured basis for conducting multi-round interactions with the evaluatee, as shown in the upper examiner part of Figure 2. Formally, we model the interview as a traversal process over \(G_s\). At turn \(t\), the examiner randomly selects a target node \(v_t \in V_s\) (or retains \(v_{t-1}\) for a follow-up question) and generates a question grounded in \(v_t\) and its local structure, i.e., \(\mathcal{A}_s(v_t)\) and incident relations in \(E_s\). Three properties of this design are worth emphasizing. (i) Grounded traversal: every turn is anchored to a concrete node \(v_t\in V_s\) and its local structure, so questions stay verifiable against \(G\) rather than drifting into open-ended elaboration. (ii) Goal-driven interaction: the scenario \(s\) and goal \(g\) persist across turns, making the dialogue purposeful rather than a sequence of independent queries. (iii) Contextual continuity, follow-up consistency, and conversational dependency: retaining the target node (\(v_t\!=\!v_{t-1}\)) for follow-up questions and conditioning every turn on the prior conversation lets the examiner stress-test whether the model’s later answers stay consistent with its earlier claims. From this scaffold, the examiner actively selects one of four question types to elicit different hallucination behaviors — adversarial questions are only one component of this repertoire:

• Regular questions are grounded in \(v_t\) and the goal \(g\), querying observable objects, attributes, or relations in \(G_s\). They establish a factual baseline under realistic user objectives.

• Follow-up questions condition on the dialogue history and the model’s prior claims, aiming to confirm, clarify, or interrogate earlier responses. If a follow-up is issued, the examiner continues to use the same target node (i.e., \(v_t = v_{t-1}\)) to test consistency and expose hallucinations and uncertainty that may not be exposed in single-turn interactions.

• Adversarial questions ask an existential question about a highly plausible but absent object \(v \notin V_s\), an attribute \((a,x) \notin \mathcal{A}_s(v_t)\), or a relation \((v_i,r,v_j)\notin E_s\) that is not in the image, but commonly co-occurs with the visual content. These questions are designed to exploit statistical co-occurrence biases and test whether the model relies on visual evidence or language priors.

• Unanswerable questions introduce a false premise that is not entailed by the visual evidence in \(G\). Concretely, the examiner first injects a premise \(p\) by fabricating an unsupported element in \(G\), i.e., either a non-existent object \(v^{(0)}\notin V_s\), an absent attribute \((a^{(0)},x^{(0)})\notin \mathcal{A}_s(v_t)\), or an absent relation \((v_i^{(0)},r^{(0)},v_j^{(0)})\notin E_s\). Conditioned on this false premise, the examiner then composes a follow-on query by introducing an additional fabricated element (object/attribute/relation) that depends on \(p\), yielding a compound question \(q(p)\) whose answer is undefined under \(G\): \[\small G \not\models p \quad \Rightarrow \quad G \not\models q(p).\] where \(G \not\models p\) refers to that \(p\) is not entailed by \(G\). For example, given a scene graph encoding a cake on a table and a fork nearby, the examiner may fabricate a node \(v_{\text{man}}\notin V_s\) and a relation \((v_{\text{man}},\texttt{holding},v_{\text{fork}})\notin E_s\), and then ask: “What is the man using to eat the cake?” A well-grounded model should detect that the premise is unsupported (i.e., \(G \not\models p\)) and refuse to answer or correct the premise (e.g., "There is no man in the image") while hallucinating models instead tend to produce fabricated details (e.g., "the man is holding the fork to eat the cake").

Type selection is conditioned on the conversation history and previously used types to avoid repetitive questions and support a coherent dialogue flow. This form of examiner evaluates not only descriptive accuracy, but also robustness under confirmation pressure, ambiguity, misleading premises, and adversarial distractors, which are failure modes that static, single-turn protocols often miss.

Examiner Backbone. To enable structured yet adaptive evaluation, we combine the flexibility of MLLMs with the graph-based workflow described above. The examiner is empowered by an MLLM such as GPT-4o [10]. Concretely, we prompt an MLLM to generate the scenario and objectives, actively select the question type for each turn, and generate diverse yet contextually and visually grounded questions.

Scaling Contexts for Coverage. In addition to precision over visual elements, recall (i.e., coverage of image content) is also critical for visual hallucination assessment [27]. Otherwise, an MLLM can minimize risk by producing overly sparse descriptions, remaining correct simply by saying little. Prior work accounts for this by grading captions on both hallucination and coverage [27], encouraging outputs that are not only accurate but also broadly descriptive. However, enforcing exhaustive coverage is misaligned with the notion of a brief caption (which typically focuses on salient foreground content) and is incompatible with single-turn VQA formats.

We argue that coverage should be primarily driven by the examiner, not imposed by the grader. Rather than requiring a single response (e.g., in a brief caption) to cover all details, our examiner induces broad coverage through multi-round interaction: it generates diverse, goal-driven contexts, each selecting a different sub-graph \(G_s\), and asks questions that target distinct objects, attributes, and relations. As we increase the number of sampled contexts, the union of queried sub-graphs expands, yielding higher coverage in the resulting transcripts while preserving naturalistic interaction.

3.2 Scene Graph-Based Grader↩︎

As shown in the lower grader part of Figure 2, the grader evaluates response faithfulness by parsing the MLLM’s transcript into a predicted scene graph and comparing it against the ground-truth one. By computing a graph similarity (or distance), the grader quantifies alignment with the underlying visual content and coverage of all possible visual elements. Discrepancies correspond to either hallucinated (less faithful) or missing (less covered) objects, attributes, and relations.

Measuring Performance with SG. Let the ground-truth scene graph be \(G^{\star}=(V^{\star},E^{\star},\mathcal{A}^{\star})\), and let the MLLM transcript over \(T\) turns be \(\mathcal{T}=y_{t=1}^{T}\). Using an LLM-based parser, we extract semantic tuples from \(\mathcal{T}\) at the sentence level (i.e., object instances, attributes, and predicates) and aggregate them into a predicted graph \(\hat{G}=(\hat{V},\hat{E},\hat{\mathcal{A}})\). To improve instance-level precision, we filter non-physical nouns by restricting candidate entities to WordNet physical-entity classes [56]. The final evaluation score is computed by comparing \(\hat{G}\) to \(G^{\star}\) using standard scene-graph matching metrics.

Graph Edit Distance (GED). We adopt graph edit distance to measure disagreement between \(\hat{G}\) and \(G^{\star}\). GED computes the minimum total cost of edit operations that transform one graph into another: \[\small \mathrm{GED}(\hat{G},G^{\star})= \min_{\pi \in \Pi(\hat{G},G^{\star})} \sum_{e \in \pi} c(e),\] where \(\Pi(\hat{G},G^{\star})\) is the set of valid edit paths converting \(\hat{G}\) into a graph isomorphic to \(G^{\star}\), and each edit operation \(e\) (node/edge insertion, deletion, or substitution) incurs cost \(c(e)\). Under this formulation, insertions penalize missing visual content (present in \(G^{\star}\) but absent in \(\hat{G}\)), whereas deletions penalize hallucinated entities or relations (present in \(\hat{G}\) but absent in \(G^{\star}\)). Therefore, \(\mathrm{GED}(\hat{G},G^{\star})\) jointly captures hallucination and coverage: lower GED indicates closer agreement with the annotated scene graph, while higher GED reflects greater divergence due to fabricated, incorrect, or uncovered objects, attributes, and relations.

SG \(\Delta\)Con [57] measures graph similarity by comparing node-affinity matrices computed with fast belief propagation. Let \(S_{\hat{G}}\) and \(S_{G^{\star}}\) denote the affinities of the predicted and ground-truth graphs. The structural distance is \[\small d_{\Delta}(\hat{G},G^{\star})=\sqrt{\sum_{i,j}\left(\sqrt{S_{\hat{G},ij}}-\sqrt{S_{G^{\star},ij}}\right)^2},\] which corresponds to the Matusita distance between propagated influence patterns. We further score node attributes with the normalized symmetric difference \(d_{\mathcal{A}}=\frac{|\hat{\mathcal{A}}\triangle\mathcal{A}^{\star}|}{|\hat{\mathcal{A}}\cup\mathcal{A}^{\star}|}\), yielding the final distance \(d=d_{\Delta}+\lambda d_{\mathcal{A}}\). This metric captures global structural agreement while penalizing semantic mismatches.

4 Experiments↩︎

4.1 Experiment Setup↩︎

Datasets. We evaluate across two complementary evaluation sets. The first is human-annotated Visual Genome (VG) [58], from which we sample 100 images. VG provides dense object, attribute, relationship, and region annotations that are converted into ground-truth scene graphs \(G^\star\). The second is the Synthetic Visual Genome (SVG) dataset [28], a large-scale collection drawn from diverse image sources, from which we sample 500 images (100 from VG, 200 from ADE-20K [59], and 200 from COCO [33]) to assess cross-domain generalization.

Evaluatees. We evaluate widely used VLMs, spanning open-source and proprietary models. The complete list is provided in Appendix 12. Notably, we also include LLaVA-1.5 with OPERA decoding [60], a decoding strategy specifically designed to mitigate hallucinations.

Baseline Examiners. We compare against three baseline examiners: (i) the widely used caption-based examiner, which prompts the model to generate a brief description of the image [15], (ii) the binary-question examiner, which asks existential questions about objects and relations (e.g., POPE [22]), and (iii) our prompt-only baseline, which follows pre-defined conversations as in-context-learning examples to elicit hallucinations via contextualized, multi-turn interactions with the evaluatees.

Baseline Graders. To assess hallucination elicitation, we use several automatic graders alongside our SG-based grader. CHAIR [15] reports sentence-level (CHAIR\(_S\)) and instance-level (CHAIR\(_I\)) object hallucination rates. Building upon CHAIR, we also report object coverage, which measures the proportion of GT objects mentioned. We further include MMHal [29], a GPT-4-based grader that scores hallucination severity on a 0-6 scale, and HaELM [49], an LLaMA-based detector that evaluates hallucination and answer informativeness. Finally, we adopt VALOR, an LLM-based two-stage grader that generalizes the CHAIR metric and incorporates both faithfulness and coverage into the evaluation [27]: VALOR faith\(_i\) measures the fraction of mentioned objects that exist in GT, and VALOR cov\(_i\) measures the fraction of GT objects the model mentions. See Appendix 9 (Table 7) for the full list of grader variants with formulas, native directions, and value ranges.

Human Annotation. Beyond automatic graders, we recruit crowdsourced annotators to annotate phrase-level hallucinations. Annotators assess object existence and counts, attributes, relationships, and scene text. These annotations are aggregated into sample- and sentence-level hallucination scores, which serve as human reference labels for evaluating both hallucination elicitation by examiners and hallucination detection by graders. The annotation guidelines are provided in Appendix [appendix:hallucination95annotation].

4.2 Research Questions↩︎

We organize our experiments around three research questions (RQs): 1⃝ Do contextualized, multi-turn, and dynamic interactions reveal more VLM hallucinations than conventional static evaluation? 2⃝ Does the SG-based CEDI grader provide a more consistent and human-aligned assessment of visual hallucinations? 3⃝ What behavioral insights does CEDI offer for evaluating the hallucinations of VLMs?

5 Results↩︎

5.1 Effectiveness of Examiner (RQ1⃝)↩︎

r0.35 6pt

Comparison with Baseline Examiners. To answer whether contextualized, multi-turn, and dynamic interactions reveal more VLM hallucinations than conventional static evaluation, we compare our CEDI examiner with several well-known baseline examiners, such as the caption-based [15] and our own prompt-only baseline. Results on the SVG dataset are summarized in Figure 4. Across all graders, CEDI examiner consistently elicits more hallucinations on the SVG dataset across multiple hallucination metrics.

We further compare with single-turn binary-question examiner (e.g., POPE [22]). For a fair comparison, we first use Qwen3-30B-A3B-Instruct-2507 to parse each evaluatee’s CEDI transcript into yes/no questions, keeping only questions whose ground truth is available in annotations; we then prompt the evaluatee to answer these parsed yes/no questions and compare the accuracy with CEDI’s implicit prediction (in the transcript). Table 1 shows that CEDI elicits more hallucinations than yes/no probing. Together, these results show that static benchmarks miss a non-trivial slice of hallucination behavior that surfaces only in contextualized, dynamic, interactive use.

Context Scaling. As discussed in Section 3.1, CEDI introduces Context Scaling to induce broad coverage. Figure 5 sweeps the number of contexts per image (1, 2, 5). All metrics improve monotonically across three evaluatees (LLaVA-1.5-7B, Qwen2.5-VL-7B, Qwen3-VL-8B), indicating that contextual diversity, rather than raw conversation length, is the key factor driving hallucination elicitation and broader coverage in hallucination evaluation.

r0.5 2pt

Human Annotation. Human-annotated hallucination rates for the same models under all three examiners are reported in Table 2, further confirming the previous finding that CEDI consistently elicits more hallucinations than other examiners. Across all six statistics, CEDI is the highest-scoring examiner on every evaluatee (bolded values), reflecting significantly more hallucinations.

Ablation Study on Contextualization. Figure 6 compares three conditions: CEDI full, CEDI w/o node selection, and CEDI with the entire context generator removed. Both ablations regress on every metric across every evaluatee; dropping the context generator entirely causes consistently larger degradation than dropping only the node selector.

Ablation Study on Question-type. CEDI mixes four question types: regular, follow-up, adversarial (adv.), and unanswerable (unans.). To test the contribution of the two refusal-demanding question types (adv. and unans.), we ablate them, keeping only regular and follow-up, and compare against the two examiner baselines. Removing the two types does reduce CEDI’s hallucination exposure across most graders (GED \(160.1\!\to\!156.7\), SG \(\Delta\)Con \(4.76\!\to\!4.68\), MMHal \(39.4\!\to\!34.0\), CHAIR\(_I\) \(48.6\!\to\!45.2\)), while redirecting the evaluative conversation toward more coverage (Cov.\(52.7\!\to\!56.7\), VALOR cov\(_i\) \(63.1\!\to\!70.8\)). Nevertheless, the ablated variant still exposes more hallucination than either baseline on all graders except a tie with the prompt baseline on MMHal (\(34.0\) vs \(34.9\)). Thus the two “less-natural” types add hallucination pressure on top of what regular and follow-up already provide, but they are not the sole driver of CEDI’s effect.

4pt

Multi-round Interaction. Restricting CEDI to a single-round question per image (Figure 7) drops exposure by 15 – 40 % on CHAIR\(_I\), 60 – 75 % on Coverage, and 24 – 35 % on GED across evaluatees. We also test a multimodal examiner variant that attaches the image to the examiner backbone; it does not improve elicitation (Appendix 11).

Conversational History. We ablate whether the evaluatee sees the running dialogue as context on its next turn (w/ hist.) versus receiving each question in isolation (w/o hist.) on five evaluatees (InternVL2-8B, InternVL2.5-8B, Qwen2.5-VL-7B, gemma-3-12B, LLaVA-1.5-7B). Figure 8 reports unweighted per-metric means (w/ hist relative to w/o hist), and Figure [fig:history-valor-box] shows the per-model spread by question type. Exposing the model to its own history amplifies whole-turn hallucination judges (MMHal \(+11.7\) pp, VALOR faith\(_i\) \(-5.7\) pp, GED \(+31.7\), SG \(\Delta\)Con \(+0.37\)) and reduces both aggregate coverage (Cov.\(-5.7\) pp) and per-image VALOR cov\(_i\) (\(-6.8\) pp): with history the model tends to repeat and extend its own earlier claims rather than probe new visual content.

Human Evaluation of Interaction Quality. We complement with human evaluation of both interaction realism and context relevance. The annotation guidelines are provided in Appendix [appendix:realism95annotation]. For realism, annotators rank complete conversations on 50 sampled images by how closely they resemble natural VLM usage (Figure [fig:realism-ranking]). CEDI is consistently preferred over Caption and POPE, with a statistically significant improvement over POPE (\(p=.011^*\)). We further evaluate question and hallucination relevance using 100 question–answer pairs sampled from the examiners and evaluated models. Annotators judge whether each generated question or elicited hallucination is directly relevant to the user’s stated objective and context. For a fair comparison, we re-evaluate Caption by giving the VLMs the same context description as CEDI, and we re-generate POPE questions by querying an LLM with the GT annotation and the context description (see Appendix 10 for the exact prompt and POPE-question construction pipeline). As shown in Figure 9, CEDI achieves higher relevance for both question generation and hallucination elicitation, while the baselines more often surface errors involving objects or relations peripheral to the context.

5.2 Analysis of CEDI Grader (RQ2⃝)↩︎

Scene Graph Parsing Accuracy. We validate the scene-graph parser used by the SG-based grader against human annotations on 50 model responses. Across 238 predicted triplets, the parser achieves 210 TP, 21 FP, and 29 FN, corresponding to 0.882 precision, 0.879 recall, and 0.881 F1, confirming that downstream graph-distance scores rest on a reliable parse.

Alignment with Human Annotation. We next evaluate each grader by measuring its agreement with human annotations. Notably, human annotators are tasked to flag hallucination (precision) but not coverage (recall). Therefore, the overall GED score is not directly comparable to human judgments, because it conflates fabricated content (precision) with missing ground-truth content (coverage). To disentangle the two failure modes, we define two variants of GED anchored on the synset-aligned intersection \(G^{\cap}=\hat{G}\cap G^{\star}\): \[\small \mathrm{GED}_{\text{hal}}=\mathrm{GED}(G^{\cap},\hat{G}), \qquad \mathrm{GED}_{\text{cov}}=\mathrm{GED}(G^{\cap},\hat{G}\cup G^{\star}).\] \(\mathrm{GED}_{\text{hal}}\) measures the edit cost of reducing \(\hat{G}\) to its grounded subgraph, thereby isolating prediction-only content (precision, aligned with human annotations); \(\mathrm{GED}_{\text{cov}}\) measures the cost of expanding \(G^{\cap}\) to the full union, capturing both hallucinated content and uncovered ground-truth content. Thus, only \(\mathrm{GED}_{\text{hal}}\) is directly comparable to the human hallucination annotations.

We then evaluate how closely each grader aligns with human annotators. For the three evaluatees with sentence-level annotations, we compute the Pearson correlation \(\rho\) between each grader score and the human-annotated hallucination count across 50 images. Including both CHAIR variants helps disentangle two distinct limitations: normalization by the total number of mentioned objects and CHAIR’s restriction to object-level hallucinations. Replacing CHAIR\(_I\) with the unnormalized CHAIR \(n_{\mathrm{hall}}\) raises the correlation for LLaVA-1.5 from \(\rho=0.05\) to \(0.32\), suggesting that length normalization dilutes the hallucination signal for verbose evaluatees. However, for Opera and Qwen3-VL, the raw count remains near zero or negative (\(-0.03\) and \(-0.24\), respectively), indicating that CHAIR can still fail when hallucinations concern relations or attributes outside its object-centric vocabulary. In contrast, GED\(_{\mathrm{hal}}\) provides the most consistent signal across evaluatees, achieving correlations between \(\rho=0.22\) and \(0.33\) in all cases, including Opera-LLaVA-1.5, where both CHAIR variants collapse.

r0.5 4pt

Stability. We evaluate the stability of each grader by measuring the variability of its outputs across repeated runs. Table 4 reports the mean and standard deviation of each metric over three repeated runs on a fixed 20-sample subset, aggregated over four evaluatees (LLaVA-1.5-7B, InternVL3-8B, Opera-LLaVA-1.5, and Qwen3-VL-8B). Both metrics reported by our SG-based grader show a std below 1 % of the metric mean, demonstrating they are highly stable across repeated runs.

5.3 Evaluation Result (RQ3⃝)↩︎

Having established the effectiveness of both the CEDI examiner and the SG-based grader, we next investigate what behavioral insights CEDI reveals about hallucination patterns in VLMs.

We first report the results of numerous prior and state-of-the-art VLMs in Table 5, including open-source VLMs such as LLaVA-1.5 [61], InternVL2/2.5 [62], [63], Qwen2.5-VL [64], and Gemma-3 [11], and proprietary VLMs such as GPT-4o [10] and GPT-5.4 mini [65], with our CEDI examiner and SG-based grader.

Table 5 reveals substantial variation in hallucination behavior across both models and image sources. GPT-4o consistently achieves the lowest or near-lowest GED and SG \(\Delta\)Con scores, indicating relatively fewer hallucinations, while Gemma-3-12B exhibits the highest hallucination scores on all three sources; the broadest coverage is attained by Qwen2.5-VL-7B and GPT-5.4 mini. Performance also varies across datasets, with ADE20K generally eliciting higher hallucination scores than COCO and VG, likely due to its denser and more diverse scene content.

We examine CEDI’s turn-level hallucination dynamics using VALOR, with \(1{-}\mathrm{faith}_i\), i.e., the fraction of object mentions at turn \(i\) that do not match any ground-truth object, as the per-turn hallucination signal. We use VALOR rather than the SG-based grader here because the latter is designed to evaluate the accumulated, transcript-level response of a VLM, making it unsuitable or costly for turn-level hallucination assessment (see Section 7).

Four Types of Questions. Figure 10 (a) shows that regular, follow-up, and adversarial questions account for most hallucinated turns. Although unanswerable questions occur less frequently, they yield the highest per-turn hallucination rate (see Figure [fig:history-valor-box]). Overall, refusal-demanding questions (adversarial and unanswerable) are more error-prone than fact-seeking questions, particularly when the model can condition on its prior responses.

Hallucination as Conversation Progresses. As shown in Figure 10 (b), the hallucination rate remains low and nearly flat without conversation history (\(21\text{--}25\%\) after the opening turns). With history, it begins near \(26\%\), rises to roughly \(30\text{--}32\%\) within the first third of the conversation, and then plateaus around \(30\%\). This suggests that long contexts increase VLMs’ susceptibility to visual hallucination, potentially because attention to the visual input gradually weakens as the dialogue progresses.

Qualitative Error Types. Human-annotated errors in Table 6 are dominated by hallucinated objects (\(62\%\)), particularly in unanswerable and adversarial turns. Wrong attributes account for \(20\%\) of errors and frequently arise in follow-up turns, where models revise previously correct attributes under conversational pressure. Wrong relations contribute another \(15\%\), typically asserting an incorrect relation for an existing object, such as describing a telephone that sits directly on the desk as placed on a stack of papers.

4pt

6 Conclusion↩︎

We introduced CEDI, a contextualized evaluation framework that recasts VLM assessment as a three-party interaction among an evaluatee, a dynamic examiner, and a grader. Guided by scene graphs, the examiner conducts context-driven, multi-turn interaction using regular, follow-up, adversarial, and unanswerable questions, while the SG-based grader evaluates hallucinations beyond object-level errors by capturing incorrect objects, attributes, and relations, together with coverage. Across multiple models, datasets, automatic metrics, and human annotations, CEDI consistently exposes more hallucinations, achieves broader visual coverage, and produces interactions and failures that are more relevant to realistic user contexts than captioning, binary-question, and prompt-only baselines. Our analyses further show that contextualization, conversational history, and multi-round interaction are central to this effect: hallucinations often accumulate through self-reinforcing dialogue history, and models are particularly vulnerable to questions requiring premise rejection or refusal. Although the current instantiation focuses on visual hallucination, we argue that the underlying formulation and evaluation framework are general and can be extended to broader tasks and applications. We hope CEDI motivates a broader shift from static benchmark testing toward adaptive, context-dependent evaluation of multimodal systems under realistic interaction.

7 Limitations↩︎

Dependence on ground-truth scene graphs. To ground the examiner in faithful visual content and enable structured scoring, we primarily use annotated scene graphs to prevent the examiner from relying on language priors alone, which is critical for exposing hallucinations. To reduce this dependency, we validate CEDI on SVG, a cross-domain corpus whose scene graphs are automatically constructed from diverse sources rather than hand-labeled.

Examiner backbone accessibility. We use GPT-4o as the default examiner backbone for its strong instruction-following and context-modeling capabilities. More broadly, both the examiner and the grader are instantiated with LLM/MLLM components, so their quality is bounded by the underlying backbone models.

Transcript-level grading. The SG-based grader is designed to assess the accumulated, transcript-level response of an evaluatee; it is therefore ill-suited (or prohibitively costly) for grading hallucinations at the level of individual turns, for which we instead rely on per-turn signals such as VALOR (Section  5.3).

8 Ethics Statement↩︎

This paper presents work whose goal is to advance the evaluation of VLMs by introducing a contextualized and interactive framework for diagnosing hallucinations. Improved hallucination evaluations have the potential to positively impact the deployment of multimodal systems in real-world applications, such as assistive tools, where incorrect or fabricated visual information may lead to user confusion or safety risks. By enabling more fine-grained assessments, this work contributes to the development of more reliable and trustworthy models. We note, however, that any automated evaluator can itself be imperfect and should not be the sole basis for deployment decisions.

All images used in this work come from publicly available datasets (VG, COCO, ADE20K, and the SVG collection) under their respective licenses, and no personally identifiable information was collected. Human annotations were obtained from crowdsourced workers who were compensated for their time; all model-generated content was reviewed to confirm the absence of sensitive or harmful material before being shown to annotators.

9 Baseline Grader Definitions↩︎

For completeness, Table 7 enumerates every non-SG grader variant we report in the main paper and appendix, with its formula, what it captures, native direction, and value range. Native \(\uparrow\) means the raw metric increases when the evaluatee hallucinates more (or, for coverage-style metrics, when the examiner probes more of GT). Native \(\downarrow\) metrics are inverted where a shared convention is needed; in particular, the radar in Figure 4 orients every axis so that greater radius = more hallucination, normalizing percentage metrics by \(\text{score}/100\), GED by \(\text{score}/150\), SG \(\Delta\)Con by \(\text{score}/5\), and VALOR faith\(_i\) by \((100-\text{faith}_i)/50\).

4pt

Beyond these baselines, our SG-based graders GED, GED\(_{\mathrm{hal}}\), GED\(_{\mathrm{cov}}\), and SG \(\Delta\)Con are defined in Section 3.2.

10 Contextualized Caption and POPE Baselines↩︎

For the human relevance study (Section 5.1), we contextualize both the Caption baseline [15] and the POPE baseline [22] using the same scenario description (a background + goal pair) that CEDI conditions its examiner on. This appendix records the exact prompt used for Caption and the two-step pipeline used to build contextualized POPE questions.

Caption. We prepend the scenario as a plain instruction:

Considering this context: {context}, please provide a detailed description.

where {context} is the same scenario string CEDI receives (concatenation of background and goal). The evaluatee’s single response is graded exactly as in the vanilla Caption baseline.

POPE. Vanilla POPE asks binary object-existence questions drawn from a fixed sampling scheme over MSCOCO categories. To make it comparable to CEDI on our SVG images, we regenerate the yes/no pool from the scenario and the image’s scene graph in two LLM calls per image, and match the total number of questions to the number of CEDI rounds (\(n_{\text{rounds}}\)) on that image.

1. Candidate pool (one LLM call). Given {background, goal}, prompt an LLM (Qwen3-30B-A3B-Instruct-2507) to list \(N\!=\!50\) concrete visible objects that could appear in the scenario, mixing plausible-present and plausible-absent.

2. Existence + co-occurrence check (one LLM call). Given the image’s ground-truth object list (from the scene graph) and the 50 candidates, the LLM (i) labels each candidate yes/no using synonym and parent-class matching (e.g.”vehicle” counts as yes if the scene graph has “car”), and (ii) flags which of the no-candidates are frequent real-world co-occurrences of the ground-truth objects — these serve as plausible distractors.

3. Composition. From the labelled pool, draw exactly \(n_{\text{rounds}}\) questions per image in the ratio \(\tfrac{1}{2}\) yes : \(\tfrac{1}{4}\) random no : \(\tfrac{1}{4}\) co-occurrence no, then shuffle. Each question is formatted as “is there a \(\langle\text{object}\rangle\) in the image?” with the corresponding yes/no ground-truth answer.

This pipeline preserves POPE’s binary-question format and its balanced yes/no ratio, but replaces its fixed vocabulary with a per-image, scenario-conditioned pool so that the questions probe the same visual surface CEDI does.

11 Multimodal Examiner↩︎

We test whether attaching the image to the (otherwise text-only) examiner LLM improves hallucination exposure. Table 8 compares the two variants on two evaluatees.

4pt

12 Evaluated Models↩︎

We evaluate widely used VLMs spanning open-source and proprietary models, and we include a model designed to mitigate hallucinations (LLaVA-1.5 with OPERA decoding [60]). The full list is as follows.

Open-source: LLaVA-1.5-7B [61] (and its OPERA-decoding variant [60]), InternVL2-8B [62], InternVL2.5-8B [63], InternVL3-8B-Instruct [66], Qwen2.5-VL-7B-Instruct [64], Qwen3-VL-8B-Instruct [67], and Gemma-3-12B-it [11].

Proprietary API models: GPT-4o [10] and GPT-5.4 mini [65].

13 Human Annotation Guidelines↩︎

Hallucination Annotation Guidelines. Annotators were asked to identify segments of a model response that conflict with the accompanying image, given its ground-truth captions and object locations.

Conversation Realism Ranking Guidelines. Annotators were shown three candidate conversations generated for the same image and asked to rank them by how realistically they reflected genuine human–AI interactions, independent of whether the AI’s responses were correct.

Conversation Relevance Annotation Guidelines. Annotators were given the background and stated goal of each conversation and asked to identify which questions and flagged response errors were relevant to that goal, i.e., which aspects a person pursuing the stated goal would actually care about.

References↩︎