1 Introduction↩︎

Vision–language models (VLMs) often operate under partial observability [1]–[6]. Occlusion, low resolution, and motion blur frequently lead them to produce speculative semantic completions [7]–[9]. According to community standards, such unverifiable content is considered hallucination. However, hallucinations arise naturally from incomplete visual evidence rather than isolated system failures [10]–[14]. Consequently, most existing studies focus on detecting [2], [15]–[20] or mitigating hallucinations [21]–[25] at the moment they appear. Far less attention has been paid to what happens after hallucinated semantics enter the reasoning process of a VLM, leaving this post-hallucination stage largely unexplored.

Meanwhile, studies of LLMs show that chain-of-thought reasoning [23], [26]–[31] does not strictly follow human causal logic: incorrect intermediate steps can still lead to correct conclusions. This observation suggests that reasoning may continue to evolve in latent space even when intermediate representations are imperfect or partially incorrect. By analogy, hallucinations in VLMs may also interact with downstream reasoning processes. However, whether such speculative cues help, hinder, or simply perturb reasoning remains unclear.

We refer to this phenomenon as Post-Hallucination Reasoning (PHR), which describes the reasoning behavior that emerges after hallucinated semantics become part of the model’s inference context and influence subsequent predictions. Despite the prevalence of hallucinations in multimodal systems, the structure and impact of PHR remain poorly understood. As illustrated in Fig. 1, once hallucinated semantics enter the model’s inference context, they may reshape the observation and reasoning process, ultimately influencing downstream predictions. Across diverse benchmarks, we observe a consistent pattern: hallucinated semantic cues can sometimes improve downstream reasoning in vision-language tasks. This suggests that hallucinations may occasionally provide speculative semantic anchors that guide inference rather than merely degrading predictions. However, this effect is neither universal nor well understood. These observations raise two fundamental questions: (1) When does PHR improve downstream reasoning? (2) Why does PHR lead to improved reasoning?

To systematically study these questions, we introduce HIVE (Hallucination Inference and Verification Engine), an evaluation infrastructure for studying PHR. HIVE constructs paired faithful and hallucinated captions, verifies their validity, and enables controlled measurement of their influence on downstream predictions. By comparing raw inputs, faithful, and hallucinated captions under matched conditions, HIVE allows us to isolate the reasoning effects induced by hallucinated semantics and analyze the structure of PHR. Across 9 tasks and 9 models, we observe structured patterns: hallucinated semantic cues often improve performance on vision-language benchmarks, with gains varying systematically across model families and decoding settings.

To understand the mechanisms underlying PHR, we analyze hallucination-induced reasoning dynamics at multiple levels. (I) Hallucinated captions broaden semantic coverage and widen embedding distributions. (II) They modulate reasoning entropy, with correct predictions correlating with higher entropy.

These findings reveal PHR as an overlooked stage of multimodal inference, where hallucinated semantics can reshape and sometimes enhance downstream reasoning rather than merely introducing noise. Our contributions are:

• Post-Hallucination Reasoning Phenomenon. We identify and characterize PHR, a previously overlooked stage in which hallucinated semantics influence downstream reasoning in VLMs across diverse task settings. We observe structured patterns showing that hallucinated semantic cues can systematically reshape model predictions rather than merely introducing noise.

• Evaluation Infrastructure for PHR. We present HIVE, an evaluation infrastructure designed to study PHR through controlled semantic comparisons. HIVE organizes caption generation, hallucination discrimination, and downstream evaluation under matched configurations, enabling paired comparisons between faithful and hallucinated captions and allowing us to isolate and quantify the downstream impact across multiple models and task domains.

• Mechanistic Analysis of PHR. We analyze PHR across the input, reasoning-process, and output levels. Our results show that hallucinated captions broaden semantic coverage, induce distinctive distributional shifts, and modulate reasoning dynamics associated with successful predictions. These findings reveal how hallucinated semantics reshape reasoning trajectories in multimodal inference.

2 Related Work↩︎

Hallucinations in VLMs: Detection and Mitigation. Hallucination denotes content inconsistent with the given input. Hallucinations are widely observed in LLM and VLM outputs, appearing in tasks such as summarization [32] and open-domain QA [33]. Their presence undermines reliability in high-stakes domains, including healthcare [34], [35] and legal decision support [36], [37]. Two major threads dominate: detection and mitigation. For detection, studies propose factuality metrics and benchmarks such as FactCC [38], QAGS [39], TruthfulQA [40], and Q2 [41], along with agreement- or internals-based judging [15], [19], [42]. For mitigation, approaches include (I) instruction-level tuning [43]–[45], (II) constrained decoding [19], [46]–[48], and (III) training or retrieval augmentation [49]–[51]. These lines of work largely treat hallucinations as undesirable errors and focus on detecting or suppressing them, leaving the role of hallucinated semantics in subsequent reasoning largely unexplored.

Intermediate Reasoning and Latent Dynamics in LLMs. Recent studies on language-only models reveal that a model’s reasoning process does not necessarily align with its explicit chain-of-thought text. Multiple works show that LLMs may generate incorrect, inconsistent, or partially fabricated intermediate steps while still producing correct final answers [52], [53]. This phenomenon suggests that reasoning continues to evolve in the latent space and is not strictly determined by the faithfulness of intermediate text. Further analyses indicate that LLMs can maintain stable internal inference even when intermediate steps contain hallucinated elements [54]–[56]. These findings imply that unfaithful or speculative content does not terminate reasoning; instead, the model proceeds to a subsequent reasoning stage that may still yield coherent conclusions.

Hallucinations in Multimodal Reasoning Models. Recent work has begun to study hallucination effects in multimodal reasoning, showing that imperfect perceptual cues can influence reasoning outcomes in VLMs. Studies such as More Thinking, Less Seeing [57], More Thought, Less Accuracy [58], MIRAGE [59], and multimodal CoT analyses [60] indicate that hallucinations interact with reasoning dynamics beyond perception-level errors. A recent survey further summarizes evaluation and detection efforts in this area [61]. However, these methods mainly analyze either intermediate reasoning traces or benchmark-level behavior, without explicitly studying how hallucinated semantics, once used as input context, affect downstream reasoning.

Different from existing methods, we focus on a largely overlooked stage in VLM inference: Post-Hallucination Reasoning (PHR), where hallucinated semantics, once generated and adopted by the model, may influence subsequent reasoning and task predictions. Prior work on VLM hallucinations focuses almost entirely on whether hallucinations occur, why they arise, and how to reduce them. Yet little attention has been paid to how hallucinated semantics affect the reasoning processes that follow. Consequently, the structure and impact of PHR remain poorly understood, and the field lacks frameworks for controlled semantic interventions to study it.

3 Evaluation Pipeline for PHR↩︎

3.1 Problem Setup↩︎

HIVE is designed as an evaluation infrastructure for studying PHR in VLMs. It examines how hallucinated semantics, once generated and adopted by the model, influence subsequent reasoning and task predictions compared with faithful semantics. Rather than introducing a new model or training method, HIVE follows established community practices, including prompting-based caption generation and hallucination detection, and organizes them into a controlled semantic intervention framework for analyzing the causal effects of hallucinated semantics. To examine this question, we organize HIVE around three principles.

Fair comparison. We must ensure that the only difference between faithful and hallucinatory captions lies in the presence of hallucination itself. Thus, captions are generated from a unified source with identical prompts, temperature, and token budget. This design rules out confounds and allows us to focus on the difference in downstream performance across tasks and models, measured as the accuracy gap \(\Delta(H-F)\) between hallucinated and faithful augmentations.

Task-agnostic hallucination generation. Hallucinations naturally arise when an LLM is asked to produce a free-form caption of any input, regardless of modality. Some captions remain faithful, while others introduce unverifiable or speculative elements. This inherent property enables us to adapt the paradigm seamlessly across diverse textual and multimodal tasks.

Reliable discrimination. Hallucination detection is inherently imperfect, and even human annotators may disagree. To enhance robustness, we adopt an ensemble of detectors that independently judge each caption, and apply majority voting to obtain the final label. This ensemble strategy ensures that the framework remains reliable under noisy classifiers. Formally, let \(x \in \mathcal{X}\) denote an input with label \(y \in \mathcal{Y}\), and \(f: \mathcal{X} \times \mathcal{C} \to \mathcal{Y}\) a task-specific model. Each input can be paired with a faithful caption \(\boldsymbol{C}_F\) or a hallucinatory caption \(\boldsymbol{C}_H\). Such a triplet design ensures that hallucination is the only varying factor across conditions, thus enabling a controlled and interpretable evaluation. Formally, we define these conditions: \[y_{\mathrm{\small raw}} = \underbrace{\boldsymbol{f}(x)}_{\text{Raw}},\quad y_{F} = \underbrace{\boldsymbol{f}(x \mathbin{\|}\boldsymbol{C}_{F})}_{\text{+ Faithful}},\quad y_{H} = \underbrace{\boldsymbol{f}(x \mathbin{\|}\boldsymbol{C}_{H})}_{\text{+ Hallucinatory}},\] where \(\mathbin{\|}\) denotes concatenation with the task instruction. Given an evaluation metric \(\mathcal{L}(\hat{y}, y)\) (instantiated as accuracy in our experiments), we quantify the hallucination effect as \[\boldsymbol{\Delta}(H{-}F) \;=\; \mathbb{E}_{(x,y)\sim \boldsymbol{\mathcal{D}}} \Big[\, \mathcal{L}\!\big(\boldsymbol{f}(x \mathbin{\|}\boldsymbol{C}_H),\, y\big) - \mathcal{L}\!\big(\boldsymbol{f}(x \mathbin{\|}\boldsymbol{C}_F),\, y\big) \,\Big].\] This paired comparison isolates hallucination as a controlled experimental variable and enables apples-to-apples analysis of PHR across models and tasks.

3.2 HIVE Evaluation Pipeline↩︎

The HIVE evaluation pipeline consists of three modules: (I) Caption Generator, (II) Caption Discriminator, and (III) Task Solver. Given an input instance, the pipeline proceeds through these components to ensure consistent and controlled evaluation across tasks. The Caption Generator first takes the raw input (text, image, or structured record) and produces a set of candidate captions under a unified, task-agnostic prompt, which may include both faithful and hallucinatory semantics. The Caption Discriminator then evaluates these candidates and classifies each as either faithful (\(\boldsymbol{C}_F\)) or hallucinatory (\(\boldsymbol{C}_H\)); only contrasted pairs \(\langle \boldsymbol{C}_F, \boldsymbol{C}_H \rangle\) with majority agreement among detectors are retained. Finally, the Task Solver integrates the original input \(x\), one of the paired captions, and a task-specific instruction to produce the final prediction \(y\). This design yields three controlled conditions Raw (\(y_{\mathrm{\small raw}}=\boldsymbol{f}(x)\)), +Faithful (\(y_F=\boldsymbol{f}(x \mathbin{\|}\boldsymbol{C}_F)\)), and +Hallucinatory (\(y_H=\boldsymbol{f}(x \mathbin{\|}\boldsymbol{C}_H)\)).

Caption Generator. The Caption Generator aims to produce diverse semantic candidates that may include both faithful and hallucinatory variants. All captions are generated from a unified source using the same prompt, temperature, and token budget, ensuring that decoding hyper-parameters cannot confound attribution. Given an input \(x\), the generator outputs \(N\) natural-language captions describing it. Due to the inherent stochasticity of LLMs, some captions remain faithful while others introduce speculative content, which later enables the construction of contrasted F/H pairs by the discriminator. This design guarantees that both F and H captions originate from an identical generation process, providing a controlled entry point for subsequent evaluation.

Caption Discriminator. Given the candidate captions, the Caption Discriminator determines whether each caption is faithful (\(C_F\)) or hallucinatory (\(C_H\)). Since hallucination detection is inherently noisy, we adopt an ensemble of multiple detectors, each providing an independent judgment. Final labels are decided via majority voting, which significantly improves robustness under noisy or imperfect classifiers. We further verify the reliability of the hallucination discriminator through dedicated control experiments (Section 4.2). Detailed implementation specifics of the individual detectors and ensemble configuration are provided in Appendix S7 for full clarity and reproducibility.

Task Solver. The solver is not a novel model but a controlled interface to isolate the effect of captions on downstream predictions under identical model and decoding settings. We measure the isolated impact of caption faithfulness by contrasting predictions under three conditions \(C \in \{\mathrm{\small raw}, \boldsymbol{C}_{F}, \boldsymbol{C}_{H}\}\). We use a unified prompt builder that concatenates three parts: \[{\small \Phi(\mathcal{I}_{\text{task}},x,C)= \mathrm{\small Instruction}\,\mathcal{I}_{\text{task}} \mathbin{\|} \mathrm{\small Serialized Input}\,\sigma(x) \mathbin{\|} \mathrm{\small Seed}\,s(C) }\] HIVE frames hallucination as a controlled semantic intervention, enabling systematic study of how speculative cues influence downstream reasoning in multimodal models. Prompt templates are listed in Appendix S2.

4 Experiments↩︎

4.1 When Does PHR Help?↩︎

We begin by characterizing how hallucinated semantics affect downstream predictions across a diverse set of models and tasks. Using the HIVE evaluation infrastructure, we systematically analyze PHR across 9 tasks and 9 models spanning both textual and multimodal scenarios. Each model is evaluated under two input conditions: faithful and hallucinated. 1 reports results on text-only benchmarks, while 2 summarizes vision–language tasks. Further dataset and model details are provided in Appendix S6.

****LLMs on text-only tasks: limited or unstable benefits.**** On textual tasks, hallucinations provide little or no benefit. Across multiple LLM families, performance under hallucinated inputs is often similar to or lower than the faithful setting, with only occasional isolated improvements (see 1). The overall trends are small and inconsistent, indicating that hallucinated semantics are hard to exploit in rule-based or symbolically constrained tasks.

****VLMs on vision–language tasks: strong and consistent gains.**** On vision–language benchmarks, hallucinated inputs lead to clear and substantial improvements across models (see 2). GPT-4o, Gemini-2.0-Flash, and Qwen-VL-Max generally show gains, with occasional drops on specific datasets such as GQA in certain settings. Unlike the text-only setting, where hallucination effects are small and unstable, VLMs reliably benefit from hallucinated semantics across multiple tasks and model families.

These results suggest that hallucinated captions can enrich visual grounding by introducing additional perceptual cues that support downstream reasoning. Vision–language tasks often involve partial observability, where important cues may be missing or ambiguous. In such settings, hallucinated captions can serve as speculative hypotheses that expand the model’s hypothesis space and provide additional semantic anchors for reasoning.

Model-wise variation. While this overall pattern holds across VLMs, the magnitude of PHR gains varies by model. For example, GPT-4o exhibits moderate but stable improvements, whereas Gemini-2.0-Flash and Qwen-VL-Max often show larger gains on perception-heavy tasks. This variation likely reflects differences in visual grounding strength and reasoning strategies across models: models with stronger perceptual grounding can better exploit speculative semantic cues introduced by hallucinated captions, while others benefit less.

0pt

2.2pt

4.2 Validating the Hallucination Discriminator↩︎

Reliability on hallucination benchmarks. To assess the reliability of HIVE’s hallucination discriminator, we evaluate it in two complementary settings. First, we test the discriminator on the TruthfulQA benchmark, which is widely used to probe hallucination behavior in language models. The discriminator achieves an accuracy of 81.76% on this benchmark. To approximate real-world cross-domain usage, we construct a curated dataset of 180 captions by sampling 20 captions from each of nine tasks. These captions are manually annotated as either hallucinated or faithful. Evaluating the discriminator on this human-labeled dataset yields an accuracy of 83.72%. These results indicate that the discriminator generalizes beyond a single benchmark and can reliably separate hallucinated from faithful captions, providing a stable basis for the paired comparisons used throughout our experiments.

Random-checker control. To rule out improvements arising from arbitrary filtering, we replace the hallucination discriminator with a random checker that accepts captions without factual assessment. All decoding controls remain fixed to ensure a fair comparison. As shown in 3, random filtering yields only negligible performance changes (0.17–1.23%) and no statistically significant gains across datasets (all \(p>0.14\)). This result indicates that the improvements reported in Section 4.1 are unlikely to arise from chance filtering effects.

4pt

Prompt robustness. To rule out the possibility that the observed gains arise from prompt sensitivity, we repeat the experiments using semantically equivalent prompt paraphrases. Specifically, we evaluate three prompts: the original prompt (P0) and two paraphrased variants (P1 and P2). The results are summarized in 4. Across datasets and models, hallucination-augmented inputs consistently outperform faithful inputs under all prompt variants. Although absolute accuracies vary slightly due to known prompt sensitivity in LLMs, the direction of the effect remains stable. This suggests that the observed improvements are not tied to a specific prompt formulation.

3pt

3pt

Token-level ablation. We conduct a token-level ablation study to test whether hallucinated content is necessary for successful predictions. Specifically, we first identify the subset of samples that are solved only when hallucinated captions are provided, where the hallucinated path succeeds while the faithful path fails. Within these captions, we locate hallucinated tokens that serve as core evidence in the model’s reasoning process. We then mask these tokens using a neutral placeholder and re-evaluate the model on the same samples. As reported in 5, we evaluate a subset of samples where the hallucinated path succeeds while the faithful path fails. For this subset, H-before is near-perfect by construction, and post-ablation accuracy (H-after) drops substantially across all four datasets after masking hallucinated evidence tokens. This result indicates that key hallucinated tokens act as informative cues rather than redundant noise, and that the model relies on them to reach correct predictions.

6pt

4.3 Reasoning Convergence Analysis↩︎

To assess whether the semantic shifts introduced by hallucinated captions affect the stability of model inference, we analyze reasoning convergence at two complementary levels: (I) Intra-chain convergence, which measures whether intermediate reasoning steps progressively align with the final conclusion under hallucinated inputs (3 left); and (II) Inter-chain consistency, which evaluates whether multiple reasoning paths sampled from the same input converge to semantically similar trajectories across different sampling seeds (3 right). These analyses provide a fine-grained view of how hallucinations affect convergence both within and across reasoning chains, enabling us to determine whether they destabilize or preserve the model’s inference process. Further implementation details are provided in Appendix S9.

Hallucinations do not disrupt intra-chain convergence. From 3 (left), step-to-final semantic similarity steadily increases, with intermediate steps progressively aligning with the final conclusion. This trajectory indicates that reasoning chains naturally converge as inference unfolds rather than drifting from the target answer. Moreover, the narrowing variance band suggests that this convergence pattern remains stable across runs and datasets.

Reasoning chains exhibit strong inter-chain consistency. From 3 (right), we observe that reasoning paths generated under hallucinated (H) and faithful (F) captions both achieve very high pairwise similarity across multiple sampling runs (means \(\approx 0.97\)). The two distributions nearly overlap, and statistical tests confirm no significant difference between them (\(p > 0.6\)). This indicates that, regardless of whether captions contain hallucinations, the model converges to consistent reasoning trajectories across chains. Rather than diverging into unstable alternatives, multiple sampled paths remain semantically aligned, underscoring the robustness of the model’s inference.

4.4 Why PHR Helps↩︎

To explain the strong and consistent improvements observed on vision–language tasks (see Section 4.1), we analyze why hallucinated captions provide useful semantic signals for VLMs. Our analysis focuses on the beneficial case and examines hallucination effects at three complementary levels: (I) Input-level shifts. Hallucinated captions substantially reshape the semantic inputs provided to the model, exhibiting broader distributional spread and lower similarity to faithful captions (4). This indicates that hallucinations introduce meaningful semantic variation rather than redundant noise, enriching the model’s visual grounding. (II) Process-level modulation. Reasoning-chain entropy analysis (2 left) shows that hallucinations modulate inference dynamics in a task-dependent manner. They reduce trajectory entropy on reasoning-heavy tasks (promoting convergence and stability), while increasing entropy on structurally open-ended tasks (supporting exploration), indicating active reshaping of the model’s reasoning process. (III) Output-level diversity. Correct predictions consistently exhibit higher caption entropy than incorrect ones (2 right), suggesting that broader semantic coverage induced by hallucinations is positively associated with successful reasoning. Rather than mere lexical variety, this diversity reflects expanded semantic grounding that supports downstream task performance. Further implementation details of similarity computation, entropy estimation, and statistical testing are provided in Appendix S8. We give the following observations:

Hallucinations reshape semantic inputs. From 4, we observe that hallucinated captions differ systematically from faithful ones in both mean similarity and distributional spread. Hallucinated inputs exhibit wider variance and heavier tails in the embedding space, a difference that is statistically significant under paired t-tests (\(p<0.01\)). This confirms that they introduce genuine input-level shifts rather than acting as redundant noise, providing models with additional anchors to explore alternative reasoning paths. Importantly, this shift is consistently observed across multiple datasets, underscoring that input-level semantic reshaping is a general property of hallucinations rather than a dataset-specific artifact, and holds robustly across diverse modalities and domains.

****Hallucinations modulate reasoning dynamics.**** As shown in 2 (left), hallucinated prompts alter the entropy of reasoning trajectories in a task-dependent manner. On reasoning-heavy tasks such as BBBP, AntiCP2, and ProofWriter, hallucinations reduce movement entropy, suggesting that they promote more convergent and stable inference. Conversely, on structurally open-ended tasks such as SARA v3, hallucinations increase entropy, enabling the model to explore a broader range of reasoning paths. These differences are statistically significant (paired \(t\)-tests, \(p<0.05\)), indicating that hallucinations do not merely inject noise but actively reshape inference dynamics in ways that can either encourage convergence or support exploration, depending on the task.

****Correct predictions align with higher caption entropy.**** As shown in 2 (right), hallucinated captions that lead to correct predictions consistently exhibit higher semantic entropy than those leading to incorrect predictions, across datasets such as GQA, Dex-Net, ISIC, and PlantVillage. These differences are statistically significant (\(p<0.05\)), and the effect holds consistently across all evaluated datasets, underscoring that higher semantic diversity is a general marker of successful reasoning rather than a dataset-specific artifact. Rather than mere lexical variety, this result highlights that semantic diversity in the latent space is a useful signal that supports accurate task performance.

We further analyze hallucinations that arise within the reasoning chain itself. By detecting the position of the first hallucinated token (early, middle, late), we observe consistent positional patterns: early hallucinations tend to correlate with stronger downstream improvements. This suggests that speculative cues introduced early in the reasoning process may shape the evolving inference trajectory. Detailed experimental results are provided in Appendix S4.

Intuitively, hallucinated captions expand the semantic hypothesis space available to the model. Under partial observability, faithful captions often describe only visible attributes, which may provide insufficient cues for reasoning. Hallucinated cues introduce speculative but task-relevant semantic anchors that guide the model toward plausible interpretations of the input. When these anchors align with the latent structure of the task, they help the model explore useful reasoning trajectories rather than remaining confined to incomplete observations.

4.5 Case Study↩︎

Beyond aggregate results, we present a case study to illustrate how hallucinated captions reshape reasoning chains in practice. We select a sample from the ISIC dataset, where the task is to determine whether a skin lesion is benign or malignant. Given the same input, the faithful (F) caption focuses on superficial attributes (e.g., asymmetry, irregular borders), constraining the reasoning to melanoma-like criteria and resulting in an incorrect malignant prediction. In contrast, the hallucinated (H) caption introduces a vascular cue suggestive of a seborrheic keratosis (SK) frame. Although this cue is not strictly faithful to the image, it provides a task-aligned semantic trigger that reshapes the reasoning trajectory: intermediate steps increasingly anchor the SK frame, ultimately leading to the correct benign diagnosis. This example demonstrates how hallucinations can supply additional semantic anchors that steer the reasoning process toward more effective diagnostic paths, rather than merely injecting noise. As illustrated in 6, hallucinated captions can introduce auxiliary but task-relevant cues that guide the reasoning chain toward the correct outcome. More case studies across additional datasets are provided in Appendix S5.

4.6 Ablation Study↩︎

To identify the conditions under which hallucinations provide the strongest utility, we conduct a set of ablation studies examining three key generation factors: sampling temperature, token budget, and hallucination intensity. These analyses reveal how different generation configurations shape the semantic expansion introduced by hallucinated captions and how such expansion translates into downstream performance gains.

Temperature. We analyze how sampling temperature influences the usefulness of hallucinated captions. As shown in 6, all four datasets peak at \(T=0.6\), yielding the strongest and most consistent gains (e.g., +11.76% on ISIC, +14.68% on PlantVillage, +2.51% on Dex-Net, +3.76% on AntiCP2). Lower temperatures (\(T=0.0/0.3\)) generate conservative captions with limited semantic expansion (e.g., \(-4.27\)% on AntiCP2, \(-5.05\)% on ISIC), while higher temperature (\(T=0.9\)) increases variance and reduces controllability (e.g., \(-5.00\)% on AntiCP2). These observations indicate that a moderate temperature around \(T=0.6\) achieves the most effective balance, providing useful semantic enrichment without introducing excessive variability.

6pt

Maximum token budget. We examine the impact of token length using temperature \(T=0.6\) (6). Short generations (128 tokens) limit semantic coverage and result in weak or inconsistent gains. A budget of 256 tokens produces strong and stable improvements across all datasets. Larger budgets (512 or 1024 tokens) can yield higher peaks but also introduce greater variance. Ablating key hallucinated tokens results in a substantial accuracy drop, confirming the benefits of hallucinations rely on sufficient and well-structured semantic cues.

Hallucination intensity. To assess how hallucination strength affects downstream performance, we generate captions with different hallucination levels using GPT-4o and group them into strong and weak categories. We interpolate between faithful and hallucinated captions, and between strong and weak hallucinations, then re-project them into SBERT space for alignment. 5 shows smooth semantic transitions (right) and an inverted U-shaped pattern (left), where moderate hallucination intensity provides the most reliable accuracy improvements, while excessive intensity reduces controllability.

5 Discussion and Conclusion↩︎

Our findings point to a dual characterization of post-hallucination reasoning. The faithful path encourages exploitation, grounding the model in verified evidence and producing precise but narrow predictions. The hallucinated path promotes exploration, expanding the hypothesis space through speculative but task-relevant cues that occasionally unlock shortcuts unavailable to faithful inputs. Hallucinations are thus not merely errors but alternative signals that broaden the model’s inference landscape. Their benefits, however, depend critically on control: moderate hallucination intensity enriches semantics without destabilizing inference, whereas excessive or misaligned hallucinations degrade reliability. Temperature, token budget, and interpolation strength provide practical levers for tuning this balance across datasets, models, and decoding settings, and fallback mechanisms to the faithful path help manage risk.

Limitations. Although our study reveals consistent patterns of PHR across tasks and models, several limitations remain. First, our evaluation is conducted on a fixed set of benchmarks and may not fully capture the diversity of real-world multimodal reasoning scenarios. Second, our interventions operate primarily at the caption level, whereas hallucinations may also arise within intermediate reasoning steps or latent representations. Third, while hallucinated cues can sometimes assist reasoning under partial observability, they may also introduce spurious signals in other settings. Future work should investigate broader task domains and develop principled mechanisms to better control hallucination induced semantic expansion in multimodal reasoning systems.

Broader Impact. This work contributes to a deeper understanding of hallucination behavior in multimodal models by examining how hallucinated semantics interact with downstream reasoning. Importantly, our findings should not be interpreted as advocating hallucination as a deliberate inference strategy. Rather, PHR appears to arise as a by-product of reasoning under incomplete evidence. Understanding this phenomenon may inform future methods that balance exploratory semantic cues with reliable grounding and verification.

We hope this work encourages the community to further investigate the role of hallucinated semantics in model reasoning and develop principled methods to understand and control PHR.

Acknowledgements↩︎

We thank the reviewers and area chair for their constructive feedback.

This supplementary contains additional details for ECCV 2026, titled “HIVE: Understanding Post Hallucination Reasoning in Vision Language Models”. The supplementary is organized as follows:

• §6 reports the significance and robustness analysis. It includes mean\(\pm\)std, \(\Delta\)(H–F), and \(p\)-values across datasets.

• §[app:Prompt95Templates] lists the prompt templates. Each dataset has a role prompt, a generation prompt, and an evaluation prompt.

• §8 studies the effect of model scale, showing that hallucination gains are non-monotonic across Qwen2.5-VL sizes.

• §9 analyzes the positional effects of hallucinations in reasoning chains. It studies how the position of hallucinated tokens (early, middle, late) influences downstream task performance across datasets and models.

• §10 presents the case studies. It provides qualitative examples on DexNet, BBBP, and PlantVillage, showing how hallucinated captions act as anchors that guide reasoning toward correct outcomes

• §11 summarizes the datasets, models, and evaluation protocols.

• §12 explains the caption discriminator. Three complementary factuality verifiers are described in detail.

• §[app:analysis95setup] provides the analysis setup. Input-level, process-level, and output-level analysis pipelines are described separately.

• §14 provides the convergence and similarity analysis. Both intra-chain and inter-chain convergence are reported.

• §15 reports the additional robustness experiments on MMStar and MMBench, showing consistent gains across perception and reasoning subsets.

• §16 lists the dataset and model licenses.

• §17 states that GPT-5 was used only for grammar checking.

6 Significance and Robustness↩︎

All reported results are averaged over five independent runs with different random seeds and presented as mean\(\pm\)std. We conduct two-sided paired \(t\)-tests to compare faithful (F) and hallucinated (H) inputs. 7 reports the full results across all nine datasets, including mean\(\pm\)std, relative gain \(\Delta\)(H–F), and \(p\)-values. The majority of tasks exhibit statistically significant gains (\(p{<}0.05\) or \(p{<}0.01\)). For example, hallucinations yield large and consistent improvements on perception-heavy datasets such as ISIC (\(+11.8\%\), \(p=0.0015\)) and PlantVillage (\(+14.7\%\), \(p{<}10^{-8}\)), while rule-driven tasks such as CodeXGLUE and SARA show negligible or non-significant differences (\(p{>}0.1\)). These results confirm that the reported improvements are statistically reliable rather than random variation.

7 Prompt Templates↩︎

4pt

To ensure consistency across benchmarks, we design unified prompt templates that follow a three-part structure: a role prompt, a generation prompt, and an evaluation prompt. The role prompt assigns the model an expert identity tailored to the domain (e.g., drug discovery, legal reasoning, medical diagnosis). The generation prompt asks the model to verbalize the raw input ({Sign}) into natural language, thereby producing either a faithful or a hallucinated caption. Finally, the evaluation prompt specifies the downstream task, which always requires a binary decision (yes/no) together with a step-by-step reasoning chain. This design ensures that the only experimental variable is the type of caption (faithful vs.hallucinated), while all other aspects of the prompt remain controlled.

8 lists the complete templates used for all nine datasets. These include both text-based tasks (AntiCP2, BBBP, CodeXGLUE, SARA, ProofWriter) and multimodal tasks (GQA, DexNet, ISIC, PlantVillage). The templates were fixed across all models and experiments, so that observed differences can be attributed solely to the presence or absence of hallucinated semantics.

8 Effect of model scale↩︎

We evaluate Qwen2.5-VL models at three different scales (7B, 32B, 72B). As shown in 9, the impact of hallucinations is not monotonic with scale. The 32B model benefits substantially (+9.4%), while the 7B and 72B models show slight drops. This suggests that scale alone does not determine hallucination effectiveness: intermediate-scale models may gain from additional semantic cues, whereas very large models may already saturate on faithful inputs, making further hallucinations redundant or even distracting.

9 Positional Effects of Hallucinations in Reasoning Chains↩︎

While the main paper focuses on hallucinations introduced through caption-based interventions, hallucinations may also emerge within the model’s reasoning chain itself. To better understand their potential impact on downstream predictions, we conduct an additional analysis examining the positional effects of hallucinated tokens within generated reasoning trajectories.

Specifically, we detect hallucinated tokens in the reasoning chains produced by the models and group samples according to the position of the first hallucination in the sequence, categorized as early, middle, late, or none. For each category, we measure the resulting task accuracy and compare it with the baseline condition where no hallucination occurs. Experiments are conducted on two representative datasets (BBBP and ISIC) using two models (GPT-4o and Claude-3 Sonnet). The results are summarized in Table 10, which reports the accuracy differences relative to the none condition.

10 Qualitative Case Studies↩︎

DexNet: Robotic Grasping. 8 shows a robotic grasping case from DexNet. The hallucinated (H) caption mistakenly interprets the depth map as a wheeled robot silhouette, but this spurious cue provides a concrete object anchor that enables correct reasoning for graspability. In contrast, the faithful (F) caption only describes gray gradients and claw-like shapes, failing to establish object identity and thus leading to the wrong “No” prediction. This case illustrates how hallucinations, though factually incorrect, can enrich the reasoning space and support the correct decision.

BBBP: Molecular Permeability. 9 presents a molecular classification example from BBBP. The hallucinated (H) caption misidentifies the scaffold as naphthalene, but this cue anchors reasoning toward favorable hydrophobicity, guiding the model to correctly predict blood–brain barrier penetration. Meanwhile, the faithful (F) caption emphasizes a biphenyl scaffold with protonated amine, anchoring reasoning on size/charge constraints and resulting in the wrong “No” prediction. This case demonstrates that even erroneous aromatic anchors can serve as constructive signals for correct permeability classification.

PlantVillage: Crop Disease Recognition. 7 illustrates a crop disease recognition task. The hallucinated (H) caption highlights darkened tips and edges, anchoring reasoning toward late blight and producing the correct diagnosis. In contrast, the faithful (F) caption notes similar edge darkening but does not explicitly anchor late blight, leading to the incorrect early-blight decision. This case underscores that hallucinations, even when based on spurious cues, can act as decisive anchors that steer reasoning toward the correct outcome.

Summary of Case Study. Across DexNet, BBBP, and PlantVillage, a consistent pattern emerges: hallucinated captions often introduce spurious or factually mistaken cues (e.g., a robot silhouette, a naphthalene core, or darkened leaf edges). Yet these cues act as decisive anchors that expand the reasoning space, providing additional structure that guides the model toward the correct outcome. By contrast, faithful captions though factually accurate may lack sufficient anchoring, causing reasoning to remain shallow and sometimes incorrect. These case studies highlight hallucination’s constructive potential: even when imperfect, hallucinations can inject inductive signals that improve decision quality.

11 Experimental setup↩︎

Datasets. We conduct experiments on 9 datasets spanning both textual and multimodal domains. Text (5): AntiCP2 [62] (antimicrobial peptide classification), BBBP [63] (blood–brain barrier penetration), CodeXGLUE [64] (C++ exception prediction), SARA [65] (legal reasoning), ProofWriter [66] (logic-based natural language inference). Multimodal (4): GQA [67] (visual question answering), DexNet [68] (depth-based robotic grasping), ISIC [69] (skin-lesion classification), PlantVillage [70] (plant-disease recognition from RGB images).

Models. We evaluate 9 large language models, covering both proprietary and open-source systems: Closed-source: GPT-4o, GPT-3.5-turbo, Claude 3 Sonnet, Gemini 2.0 Flash, O3.
Open-source: DeepSeek-V3, DeepSeek-R1, Mistral Large, Qwen-VL

Evaluation. For binary classification tasks, we report Accuracy as the primary metric, ensuring consistency across datasets and model families.

Statistics. Unless otherwise noted, we report mean\(\pm\)std over 5 independent runs. For each dataset, we conduct two-sided paired t-tests to compare faithful vs.hallucinated inputs. Statistical significance is reported at conventional thresholds (\(p{<}0.01\)); full results and additional details are provided in Appendix 6. Token budget is implemented as a maximum generation length, though generations may terminate earlier.

12 Implementation of Caption Discriminator↩︎

We implement three complementary modules to assess the factual plausibility of hallucinated captions. Each module is motivated by prior work on self-consistency, fine-grained fact-checking, and paraphrase-based semantic validation.

Fine-Grained Factuality Verifier. Motivated by the fine-grained evaluation perspective of Factcheck-Bench [71], this module decomposes each caption into individual factual claims using sentence segmentation. Each claim is independently verified by a large language model with a structured prompt that returns a binary verdict (True/False), a confidence score (0–1), and a short justification. The final score averages the confidence of verified True claims, with a penalty for detected False claims. This design enables auditing hallucinations at the individual claim level, rather than only at the aggregated whole-caption level used previously in prior evaluations.

Self-Evaluation Factuality Verifier. Inspired by self-consistency approaches such as SelfCheckGPT [15], this module prompts the model to directly self-assess the factual correctness of an answer (with optional question context) in multiple practical scenarios. The model outputs a binary verdict with confidence and explanation. Compared to the fine-grained verifier, this method is lightweight and evaluates factuality at the whole-answer level. We also support a multimodal variant that incorporates image inputs when available across diverse evaluation settings.

Paraphrase-Consistency Verifier. Following the idea of leveraging paraphrasing and question generation for semantic consistency [72], this module generates two paraphrases of the original caption while strictly preserving meaning. The paraphrases serve only as auxiliary evidence to clarify intent, while the factuality decision always prioritizes the original caption. A fact-adjudication prompt then produces a binary verdict, confidence, and concise reasoning. This consistency check reduces prompt variance and stabilizes factuality judgments overall accuracy.

Caption Discriminator. Together, these three discriminators provide complementary perspectives: (I) fine-grained claim verification, (II) holistic self-evaluation, and (III) paraphrase-assisted consistency. In our experiments, we include a random checker baseline (accept/reject uniformly at random) and ensemble variants combining multiple verifiers.

13 Implementation Details of Analysis Setup↩︎

Input-level. To quantify the differences between faithful and hallucinated captions, and between correct and incorrect predictions, we adopt the following analysis pipeline. For each dataset, we collect hallucinated caption embeddings produced by the generation model. When multiple runs are available, we resolve embeddings by searching run-specific directories or aggregated to ensure consistent coverage. Captions and embeddings are aligned with prediction labels, with samples truncated if necessary to guarantee matching length. High-dimensional embeddings are projected to a three-dimensional latent space using principal component analysis (PCA) with full SVD. This preserves the dominant semantic directions while removing redundant variance, facilitating density estimation. We estimate local distributional entropy of the embeddings by fitting a Gaussian kernel density estimator (KDE) with fixed bandwidth. For each sample, the negative log-density serves as its entropy value, reflecting whether it lies in a dense or sparse region of the semantic space. We report mean and standard deviation of entropy separately for correct and incorrect predictions. To assess the significance of differences between correct and incorrect groups, we conduct two-sided independent-sample t-tests without assuming equal variance. We report the test statistic and \(p\)-value for each dataset. Results are aggregated across all nine benchmarks and summarized in Appendix §6. This procedure provides a principled way to examine how hallucinations reshape semantic distributions at the input level, modulate reasoning trajectories, and correlate with prediction accuracy through entropy-based analysis.

Process-level. To examine how hallucinations modulate inference dynamics, we quantify the entropy of reasoning-chain embeddings. For each input, we record the hidden-state representations of step-wise reasoning trajectories under both faithful (F) and hallucinated (H) captions. We then project these embeddings into a lower-dimensional space via principal component analysis (PCA) and estimate their density distribution using kernel density estimation (KDE). The negative log-likelihood of KDE outputs serves as an entropy measure, capturing the dispersion of reasoning movements across steps. Paired two-sided \(t\)-tests are applied to assess statistical significance (\(p<0.05\)). This measurement allows us to characterize whether hallucinations encourage more convergent reasoning trajectories (lower entropy) or diversify inference paths (higher entropy), depending on the task structure.

Output-level. To analyze the semantic effect of hallucinations, we estimate the entropy of caption embeddings under the hallucinated (H) condition and compare between correct and incorrect predictions. For each dataset, we collect the OpenCLIP embeddings of hallucinated captions (\(C_H\)). Predictions and gold labels are aligned with these embeddings by matching the number of instances. To improve stability and reduce noise in density estimation, embeddings are projected into a 3-dimensional latent space using Principal Component Analysis (PCA). This preserves the dominant variance directions while mitigating the curse of dimensionality. We adopt Kernel Density Estimation (KDE) with a Gaussian kernel (bandwidth = 0.5) to approximate the underlying semantic distribution. For each sample, we compute the negative log-likelihood under the KDE as a proxy for semantic entropy. We split samples into two groups based on prediction correctness and compute mean \(\pm\) standard deviation of entropy for each group. Statistical differences are assessed using two-sided \(t\)-tests under unequal variance assumptions. This procedure yields a robust measure of semantic diversity in hallucinated captions, allowing us to test whether correct predictions are associated with higher entropy than incorrect ones. 11 summarizes caption entropy under hallucinated (H) inputs, split by correct vs.incorrect predictions. We find that correct predictions generally align with higher entropy, with significant differences on four multimodal datasets (GQA, DexNet, ISIC, PlantVillage). These results confirm that semantic diversity is a reliable marker of successful reasoning rather than a dataset-specific artifact.

14 Implementation Details of Convergence and Similarity Analysis↩︎

Intra-chain convergence. To further understand the internal dynamics of hallucinated reasoning, we analyze whether intermediate steps in a reasoning chain progressively converge toward the final conclusion. Specifically, we extract step-wise reasoning traces from hallucinated captions and compute semantic embeddings using OpenCLIP (ViT-L/14, OpenAI weights). Each intermediate step is compared to the final step via cosine similarity, yielding a step-to-final similarity curve averaged across reasoning chains. Similarity consistently increases as the chain progresses, while variance bands narrow, indicating that hallucinated reasoning exhibits stable intra-chain convergence. This suggests that intermediate steps are not drifting away but instead steadily aligning with the final conclusion.

Inter-chain convergence. To further evaluate the stability of reasoning trajectories, we computed the average path similarity across multiple sampled chains. For each dataset, we first extracted hallucinated (H) and non-hallucinated (NH) reasoning paths, then embedded all intermediate steps using OpenCLIP (ViT-L/14, OpenAI weights). The cosine similarity between different runs was averaged to yield an overall path-level similarity score. We then compared the distribution of average similarities between H and NH conditions. Kernel density estimation (KDE) was applied to visualize the distributions. Results indicate that both H and NH paths consistently achieve very high similarity (means \(\approx 0.97\)), with nearly overlapping distributions. This confirms that hallucinations do not compromise inter-chain stability, and that multiple reasoning paths remain semantically aligned across runs.

15 Additional Robustness Experiments↩︎

To test whether the observed PHR effect is limited to our original domain-specific datasets, we further evaluate GPT-4o on two general multimodal benchmarks, MMStar and MMBench. We focus on perception and reasoning subsets that align with the scope of our study. As shown in Table 12, hallucinated captions yield directionally consistent gains across all evaluated subsets.

5pt

16 License↩︎

Datasets. All datasets used in this study are publicly available benchmarks. Their license terms are as follows: AntiCP2 is released under GPL-3.0; BBBP under the MIT License; CodeXGLUE under the Computational Use of Data Agreement (C-UDA); SARA_V3 under CC BY 4.0; ProofWriter under CC BY 4.0; GQA annotations under CC BY 4.0; Dex-Net code under BSD-3-Clause while its HDF5 databases are restricted to research-only (non-commercial) use; ISIC under CC BY-NC (non-commercial); and PlantVillage under CC0. We emphasize that our use of these datasets is strictly for academic research purposes.

Models. All models used in this study are publicly available APIs or checkpoints released by their respective providers. Specifically, Qwen2.5-VL-3B and Qwen2.5-VL-72B are released under the Qwen Research License, while Qwen2.5-VL-7B and Qwen2.5-VL-32B adopt the Apache 2.0 License. For commercial API models, including GPT-4o, GPT-3.5-turbo (OpenAI), Claude 3 Sonnet (Anthropic), Gemini 2.0 Flash (Google DeepMind), O3 (OpenAI), DeepSeek-V3 and DeepSeek-R1 (DeepSeek), Mistral Large (Mistral), and Qwen-VL (Alibaba), usage is governed by their providers’ service terms and API agreements. We emphasize that our use of these models is strictly for academic research purposes in accordance with their public availability and license terms.

17 AI Disclosure↩︎

We acknowledge the use of GPT-5 for grammar checking only. The model was employed to correct grammatical errors while ensuring the original meaning and intent of the text remained unchanged.

References↩︎