1 Introduction↩︎

Large Vision-Language Models (LVLMs) [1]–[4] have achieved remarkable successes in a plethora of applications in the past few years [5]–[7]. However, the phenomenon of hallucination can cause severe consequences in some practical scenarios, such as medical analyzes [8], [9], autonomous driving [10], [11], and human-computer interaction [12]. These issues raise pressing security concerns and may cause unintended harm to the public, significantly hindering the real-world deployment of LVLMs.

To mitigate hallucinations, existing approaches can be broadly categorized into two groups: instruction tuning methods [13], [14] and decoding-based methods [15], [16]. The former requires retraining LVLMs using curated datasets and tuning strategies, which effectively reduce hallucinations while introducing expensive computational costs. The latter employs post-hoc correction mechanisms, adjusting logit scores at each generative step to encourage LVLMs to focus more on the corresponding visual content. While these methods have shown promising results for photographic images, they suffer from significant performance drops when applied to stylized images (e.g., game and sketch).

Stylized images play a critical role in various applications, including art education, medical analysis, and criminal forensics. For instance, LVLMs can interpret forensic sketches to retrieve matches from photographic databases or generate suspect profiles. Minimizing hallucinations in such scenarios is essential to enhance the reliability of these models. However, a comprehensive benchmark for assessing LVLM performance on stylized images remains unavailable. To fill this gap, we construct a dataset comprising paired captions and both photographic and style-transferred images generated using state-of-the-art methods across five stylistic domains. We evaluate 13 advanced LVLMs and observe that stylized images significantly increase hallucination rates compared to their original photographic counterparts.

To investigate the underlying causes, we conduct an empirical analysis by measuring the correlations between stylized image representations and the generated tokens. Fig. 1 visualizes the representational patterns of real and hallucinated objects in stylized images. We observe that real object representations exhibit concentrated, high-confidence activation regions, while the hallucinated object displays sparse, low-confidence distributions. This insight naturally inspires the design of a decoding strategy that encourages the model to focus more precisely on the relevant visual regions.

We propose SAVER: Style-Aware Visual Early Revision, a training-free hallucination mitigation strategy designed to address hallucination issues in stylized images. Due to the strong knowledge priors of language models, LVLMs tend to progressively suppress visual information in later model layers [16], [17]. To counteract this, SAVER dynamically searches the optimal preceding layers with highly concentrated representational patterns to refine the output token logits. As a plug-and-play decoding strategy, SAVER can be seamlessly integrated into various LVLMs and significantly reduces hallucination rates for stylized images without requiring additional training.

Our key contributions are: (1) To the best of our knowledge, we are the first to construct a captioning dataset specifically for stylized images. We further establish a benchmark using 13 advanced LVLMs and find that stylized images tend to produce more hallucinations; (2) Compared to hallucinated tokens, we demonstrate that the correlation patterns between correct tokens and input visual contents exhibit denser visual activations in early layers. Based on this insight, we propose SAVER, a training-free decoding strategy that dynamically corrects generated tokens by leveraging high-confidence activations from earlier layers; (3) Extensive experimental results show that SAVER consistently outperforms previous methods, effectively mitigating hallucinations across a variety of models, datasets, and tasks.

2 Related Work↩︎

Large Vision-Language Models (LVLMs). Large Language Models (LLMs) such as LLaMA [18] and Vicuna [19] have achieved remarkable advancements. The rapid development of LVLMs has significantly enhanced the ability of foundation models to interpret and reason about visual content. Early LVLMs, including BLIP [5] and LLaVA [6], extended LLMs to handle image understanding and reasoning tasks. To bridge the modality gap between vision and language, various approaches have been adopted, such as linear projection layers (e.g., LLaVA [6], MiniGPT-4 [7]), Q-former modules (e.g., BLIP-2 [20], InstructBLIP [21]), and cross-attention mechanisms (e.g., Flamingo [22], OpenFlamingo [23]). More recent models, such as GPT-4o [1] and Gemini [24], have demonstrated exceptional capabilities in visual reasoning and understanding. Nevertheless, despite these advancements, hallucination remains a significant challenge.

Visual Hallucination in LVLMs typically refers to cases where the generated text is inconsistent with the input image at the instance level [25]–[27] or the faithfulness of the generated free-form answer [28]. There are various potential factors that can cause hallucinations, including modality gap, training data bias, error accumulation, etc. [26]. To evaluate LVLM hallucination levels, CHAIR [25] proposes measuring object hallucination rates in output captions. The POPE benchmark [27] assesses object hallucinations using binary “Yes/No” questions. Additionally, MME  [29] provides a more challenging dataset for hallucination evaluation, which encompasses various hallucination types, such as object, attribute, counting, etc. AMBER [30] supports both generative and discriminative tasks, covering existence, attribute, and relation hallucinations. While prior works have primarily focused on natural photographic images, this study constructs a new dataset and comprehensive benchmark to investigate hallucination in LVLMs when processing stylized images.

Hallucination Mitigation. Existing methods fall into tuning-based and decoding-based categories. Tuning-based approaches curate specialized datasets [31]–[33] and apply alignment training [13], [14], [34], but are costly in annotation and computation. In contrast, training-free methods [35] and decoding-based techniques [36], [37] are more efficient. Recent contrastive decoding methods [15], [38], [39] further enhance performance by leveraging visual comparisons. Deco [17] and Attention Lens [16] emphasize layer selection to mitigate hallucination. Our method, SAVER, differs from these two approaches by dynamically selecting non-hallucinated tokens without the need to train a detector. And SAVER leverages richer visual signals during layer selection for final output revision, thereby achieving better hallucination mitigation performance.

3 Benchmarking Stylized Image Hallucination↩︎

In this section, we introduce our proposed style dataset by outlining its motivation, construction process, and image generation pipeline. We then present the benchmark along with a detailed description of the procedures. An example style image from the dataset is shown in Fig. 2.

Motivation. We propose this dataset to examine object hallucination in cross-style scenarios, evaluating LVLMs’ ability to analyze images from diverse domains. This section describes the construction of our style-diverse dataset and benchmark. Existing LVLM benchmarks mainly use photographic (“Original”) images, reflecting their dominance in real-world applications. However, images also come in styles like Cartoon and Sketch, where objects may appear visually distinct (e.g., a cat with unusual colors). As shown in Fig. 2, such variations pose challenges for LVLMs, which may lack exposure to these styles during training. To address this, we introduce a framework to assess hallucination behaviors across a range of artistic styles.

Dataset. Follow previous hallucination works [25], [27], we sampled images from the COCO dataset to construct a high-quality stylized dataset. Each selected image contains at least five annotated objects to create a challenging evaluation setting. We applied SOTA InstantStyle [40] model to generate style-transferred images in five styles: Cartoon, Game, Graffiti, Painting, and Sketch, ending up with 1,800 images. We carefully checked each pair of images and manually filter out the low-quality one, ensuring that each original image and its stylized versions share identical annotations, including object and caption labels.

Benchmark. We construct a benchmark using two metrics:

• CHAIR: Caption Hallucination Assessment with Image Relevance (CHAIR) [25] measures the proportion of hallucinated objects—those mentioned in the caption but absent in the image. We use two variants: CHAIRi (instance-level) and CHAIRs (sentence-level), defined as: \[\begin{align} \text{CHAIR}_i &= \frac{\{\text{hallucinated instances}\}}{\{\text{all mentioned instances}\}}, \nonumber \\ \text{CHAIR}_s &= \frac{\{\text{captions with hallucinations}\}}{\{\text{all captions}\}}. \end{align}\]

• POPE: Polling-based Object Probing Evaluation (POPE) [27] evaluates hallucination by asking LVLMs “Yes/No” questions about object presence, using three negative sampling strategies: random, popular (top-\(k\) frequent absent objects), and adversarial (top-\(k\) absent objects ranked by co-occurrence with ground-truth). We follow the original POPE settings and generate 6 questions for each image, resulting in 1800 \(\times\) 6 \(\times\) 3 question image pairs.

4 Style Aware Visual Early Revision↩︎

We conduct a comprehensive benchmark and study to uncover the underlying mechanisms that lead to object hallucination when processing stylized images. Guided by these findings, we propose Style-Aware Visual Early Revision (SAVER), a novel inference-time strategy aimed at reducing hallucinations caused by style-transferred content. The overall design and workflow of SAVER are illustrated in Fig. 3.

4.1 Preliminaries↩︎

LVLMs for text generation typically consist of three key components: a vision encoder, a projection module, and an autoregressive language model. The vision encoder first converts an input image into a sequence of visual tokens \(X_V = \{x_{v_1}, x_{v_2}, \dots, x_{v_P}\}\). In parallel, a text prompt is tokenized into \(Q\) textual tokens \(X_C = \{x_{c_1}, x_{c_2}, \dots, x_{c_Q}\}\). Here \(P\) and \(Q\) are the lengths of the visual and textual tokens. The visual and textual embeddings are concatenated to form the model input \(X\), which is passed through an autoregressive language model composed of \(N\) stacked transformer decoder layers. At each layer \(i\), the model produces hidden states \(h^i = \{ h^i_0, h^i_1, \dots, h^i_{T-1} \}\), where \(T = P + Q\), \(P\) is the length of the visual tokens. During generation, the hidden state at the final position \(h^N_{T-1}\) is projected via an affine transformation \(\phi(\cdot)\), typically using an unembedding matrix \(W_U \in \mathbb{R}^{|V| \times d}\), to produce a logit distribution over the vocabulary \(V\).

4.2 Stylization Amplifies Object Hallucination↩︎

We investigate the impact of image stylization on object hallucination in LVLMs, specifically in the context of image captioning. Our benchmark includes various LVLMs, covering both open-source models (e.g., LLaVA [6], MiniGPT-4 [7], InstructBLIP [21], TinyLLaVA [41], Phi-3-V [42], Fuyu [43], Idefics2 [44], mPLUG-Owl2 [45],Qwen-VL [46]) and closed-source platforms (e.g., GPT-4o [1], Gemini-1.5-Pro [24]).

To elicit diverse responses, we employ two prompt templates: a detailed prompt (“Please describe this image in detail”) and a concise prompt (“Provide a one-sentence caption for the provided image.”). The generated captions are evaluated using the CHAIR metric, with results summarized in Tab. 1 and Tab. 7 (Appendix). In addition, we assess hallucination across three splits of the Style-POPE benchmark, with detailed results visualized in Fig. 6 in the Appendix.

Our experiments reveal a consistent increase in object hallucination when models process stylized images, such as those rendered in Cartoon, Game, Graffiti, Painting, and Sketch styles, compared to original photographs. This performance degradation highlights the models’ reduced grounding capabilities under stylistic shifts.

4.3 Visual Sensitivity with Style-Aware Score (SAS)↩︎

To quantify the influence of visual style on token generation, we propose Style-Aware Score (SAS), which measures the alignment between intermediate visual representations and the final output tokens. Specifically, SAS captures the contribution of early-layer visual embeddings to token prediction by analyzing the intermediate hidden states within the transformer decoder. Given a set of candidate tokens selected from the final decoder layer’s logits, we compute the SAS by aggregating token logits across visual token positions and selected intermediate layers. Let \(\mathcal{L} \subset \{1, 2, \dots, N-1\}\) denote a predefined set of candidate transformer layers (e.g., early or deep layers). For each selected layers \(l \in \mathcal{L}\), we extract logits \(\mathbf{Z}_l \in \mathbb{R}^{T \times |V|}\) by projecting the hidden states \(h^l\) via the output embedding matrix \(W_U\), i.e., \(\mathbf{Z}_l = W_U h^l\). We then isolate the logits corresponding to the visual token positions, yielding \(\mathbf{Z}_l^{(v)} \in \mathbb{R}^{P \times |V|}\). The Style-Aware Score for a candidate token \(c\) at layer \(l\) is defined as: \[\text{SAS}_{l}(c) = \sum_{p=1}^{P} softmax (\mathbf{Z}_{l}^{(v)}( c))\] Where \(P\) is the visual token length. This score quantifies the model’s reliance on visual features during token generation. A higher SAS implies stronger alignment between visual input and the generated token, indicating stronger grounding. SAS provides a way to quantitatively assess a model’s sensitivity to visual inputs.

4.4 Reduced Style Awareness Produces Hallucination↩︎

Prior studies have shown that language model priors often dominate visual inputs, with attention skewed toward object-like tokens [16], [17]. Motivated by these findings, we hypothesize that intermediate representations in early decoder layers encode varying degrees of visual awareness, which directly influences hallucination. Fig. 1 illustrates this hypothesis: tokens grounded in real visual objects (e.g., “tree”) exhibit higher visual awareness, while hallucinated tokens show weaker spatial alignment.

To empirically validate this observation, we conduct a layer-wise analysis of SAS trajectories throughout the transformer stack using the prompt “Please describe this image in detail". In the Style-POPE benchmark, each image is paired with six object existence questions (e.g.,”Is there a bottle in the image?“). A”no” label indicates the object is not present. For each question extract object names and their existence, and compute the average SAS values based on the predicted answer and the queried object. As shown in Fig. 4, tokens corresponding to real objects consistently exhibit significantly higher SAS scores than those associated with non-existent (hallucinated) objects, especially in the early layers. We further observe that SAS distributions are style-dependent, with peaks often occurring in earlier layers of the model.

4.5 Our Method↩︎

As discussed in the previous section, object hallucination in stylized images arises from the model’s neglect of the corresponding visual patterns during decoding. To mitigate this issue, we propose Style-Aware Visual Early Revision (SAVER), a test-time strategy that adjusts final predictions using early-layer representations. Notably, SAVER introduces no additional learnable parameters and can be seamlessly integrated into existing LVLMs and mitigate hallucination problems. SAVER operates during decoding and consists of three key steps: (1) identifying a candidate set of plausible tokens using top-\(p\) filtering; (2) selecting the most visually grounded layer via the Style-Aware Score (SAS); and (3) revising the logits to better reflect visual evidence from that layer. The algorithm pipeline is presented in Algorithm 5, which outlines the step-by-step implementation of SAVER at test time.

Style-Aware Layer Selection. Our previous analysis shows that correct tokens are highly correlated with the activation regions in early layers where visual features dominate. In this vein, we define a candidate set of transformer layers \(\mathcal{L} \subset \{1, \dots, N{-}1\}\) and identify a candidate token set \(\mathcal{C}_t\) by applying top-\(p\) filtering to the final-layer logits \(\mathbf{z}_t^N \in \mathbb{R}^{|V|}\) and top-\(k\) for the tokens. For each layer \(l \in \mathcal{L}\), we compute a style-confidence score based on the maximum Style-Aware Score (SAS; see Eq. (2)) among the candidate tokens: \[l^\star = \arg\max_{l \in \mathcal{L}} \sigma_l, \quad \gamma = \sigma_{l^\star}, \quad \text{where} \quad \sigma_l = \max_{c \in \mathcal{C}_t} \operatorname{SAS}_l(c).\] Here, \(\gamma \in [0,1]\) quantifies the influence of stylistic features at the selected layer \(l^\star\) and adaptively modulates visually relevant outputs. This mechanism allows SAVER to dynamically identify which layer provides the most relevant visual grounding at each decoding step, balancing between overfitting to style and ignoring visual context. The use of maximum SAS ensures that even if a single token strongly activates visual features at a certain layer, that signal is preserved in layer selection. In practice, we find that this adaptive grounding depth is critical for generalization across diverse styles and model architectures.

Logit Revision. SAVER refines the final-layer logits \(\mathbf{z}_t^N\) by incorporating evidence from the selected style-sensitive layer \(l^\star\). Let \(\mathbf{m}_t \in \{0,1\}^{|V|}\) denote a binary mask that activates only the candidate tokens in \(\mathcal{C}_t\), which suppresses noise and stabilizes the logit revision. The revised logits \(\hat{\mathbf{z}}_t\) are computed as: \[\label{eq:revision} \hat{\mathbf{z}}_t = \mathbf{z}_t^N + \alpha \cdot \gamma \cdot \left( \mathbf{z}_t^{l^\star} \odot \mathbf{m}_t \right),\tag{1}\] where \(\alpha\) is a scalar hyperparameter and \(\odot\) is element-wise multiplication. This formulation selectively amplifies predictions that are grounded in style-aware visual evidence, while suppressing tokens that may arise solely due to stylistic noise. By scaling with both \(\alpha\) and the layer-specific confidence \(\gamma\), the method modulates its correction strength based on how strongly the model attends to visual input at \(l^\star\). Since SAVER operates per decoding step, it enables fine-grained correction without modifying the backbone model or compromising generation fluency. This makes it applicable in real-world scenarios with diverse data distributions and effective even under domain shifts.

5 Experiments↩︎

5.1 Implementation Details↩︎

Evaluation Models and Settings. We evaluate our proposed method and baseline approaches on four representative LVLMs: InstructBLIP [21], MiniGPT-4 [7], and LLaVA-1.5 [6], LLaVA-1.6 [47]. The maximum length of the generated sequence is set to 64 tokens, with a repetition penalty of 1.0. Unless otherwise specified, decoding is performed using a fixed temperature of 0 and top-\(k{=}1\). For beam search, we adopt a beam width of 3 while maintaining the same temperature setting.

Baselines. We compare SAVER with two baseline decoding strategies (greedy decoding and beam search) as well as three SOTA hallucination mitigation methods, detailed as follows: Dola [15] is specifically designed for alleviating hallucinations in factual tasks for LLMs by reducing shallow semantic influences to improve the factuality of the final layer’s output. OPERA [36] dynamically penalizes overconfident tokens based on the emergence of aggregation patterns, while proposing a retrospective allocation strategy to avoid cases where hallucinations have already occurred. Deco [17] adaptively chooses relevant layers and integrates their knowledge into the final layer to adjust outputs. AGLA [37] uses an ensemble of global features for response generation and local features to mitigate hallucination. For all baseline methods, we use their official implementations and follow the recommended hyperparameter settings from the released source code to ensure fair comparisons.

Benchmark and Metrics. We evaluate the effectiveness, generalizability, and captioning quality of our method across five challenging benchmarks in both stylized and real-world scenarios: \(\bullet\) CHAIR. Using the prompt “Please describe the image in detail,” we assess hallucination rates with CHAIR\(\mathrm{i}\) and CHAIR\(\mathrm{s}\) on our constructed dataset. Additionally, we evaluate captioning quality using BLEU-1/2/3/4 [48], METEOR [49], and ROUGE-L [50] which can be found in Appendix. \(\bullet\) POPE. Following the official POPE protocol, we report F1 scores as the primary metric. In the Appendix, we further incorporate ACC, Precision, and Recall to comprehensively evaluate our method. \(\bullet\) sec:MME [29] is a practical benchmark encompassing 14 sub-tasks, including OCR, visual knowledge, object recognition, and relational reasoning. \(\bullet\) Real-World Cases. To evaluate performance beyond stylized images, we construct a dataset containing depth, thermal, medical, and RGB images with carefully designed query prompts. Additional details are available in the Appendix. \(\bullet\) AMBER. [30] To evaluate hallucination in more dimisions such as attribute, relation, and existence, we conduct experiments on the AMBER benchmark. More details are provided in the Appendix.

5.2 Experimental Results↩︎

Experiments on Style-CHAIR. As shown in Tab. 2, our method consistently reduces hallucinations across all styles, achieving the lowest average CHAIR across all evaluated models. Notably, SAVER achieves the best overall performance on MiniGPT-4 and InstructBLIP, clearly outperforming the SOTA method Deco. On LLaVA-1.5, SAVER achieves competitive results, closely matching the best scores. These results demonstrate the effectiveness of SAVER in mitigating object hallucinations for stylized images across diverse models and styles. Since captioning quality is critical for real-world applications, we further report the performance of different mitigation methods in terms of captioning quality in Tabs. 10, 11, and 12 in the Appendix. Compared to existing baselines, SAVER consistently exhibits outstanding text captioning performance.

Experiments on Style-POPE and MME Benchmarks. As shown in Tab. 3, SAVER consistently outperforms prior methods under various configurations on the Style-POPE benchmark, demonstrating superior robustness in mitigating hallucinations across adversarial, popular, and random settings. The detailed results are shown in Tab. 13, Tab. 14, and Tab. 15 in the Appendix. Additionally, in Tab. 4 and Tab. 16 (Appendix), SAVER achieves the highest scores across both perception and recognition tasks on the challenging MME benchmark, significantly improving performance for LLaVA-1.5 and InstructBLIP. Beyond stylized image captioning, SAVER exhibits strong generalizability to mitigate hallucinations in various practical challenging tasks.

Experiments on Real-World Benchmarks. To further examine SAVER’s practicability, we evaluate it on real-world scenarios, including depth, thermal, and medical images. As the average results shown in Tab. 5, SAVER consistently achieves the highest performance in different modalities. Specifically, it obtains the best average F1 scores of 71.8% on LLaVA-1.5 and 68.6% on InstructBLIP. These results further validate the effectiveness of SAVER in real-world scenarios. The detailed results and discussions can be found in the Appendix.

Experiments on AMBER Benchmarks. Previous experiments comprehensively demonstrated SAVER’s effectiveness in mitigating object hallucination. Herein, we further evaluate SAVER’s generalizability on the AMBER dataset, which consists of three types of hallucinations, including existence, attribute, and relation. AMBER comprises 1,004 images, each paired with the corresponding designed questions and annotated labels. As shown in Tab. 6, SAVER consistently achieves the best or second-best Acc/F1 scores compared to the SOTA methods, exhibiting its strong generalizability to various hallucination types. This can be attributed to our visual early revision design that drives the model to pay more attention to visual signals.

5.3 Ablation Study↩︎

Tabs. 18, 19, 20 and Fig. 7 in the Appendix detail the ablation experimental results. We vary five components—scale factor \(\alpha\), confidence threshold \(p\), candidate set size \(k\), number of image‑representative tokens \(N_i\), and early‑exit depth—to quantify their contributions to hallucination mitigation and caption fluency. For scale factor, \(\alpha=0.6\) consistently balances visual grounding and language quality, attaining the lowest hallucination scores on LLaVA‑1.5. For token filtering, higher thresholds reduce low confidence tokens. And \(p=0.9\) achieves the strongest average results for MiniGPT‑4 and InstructBLIP, while lower \(p\) preserves diversity in challenging styles at the cost of stability. In the candidate size, \(k=20\) emerges as a robust optimum across models, and larger \(k\) introduces spurious evidence. Visual awareness is best supported by moderate \(N_i\) (50–100), as larger values can introduce low-confidence tokens that increase hallucination risks. Finally, the choice of early exit depth can greatly affect hallucinations. Experiments show that “Standard” provides the most stable performance.

6 Conclusion↩︎

This work studies the high hallucination risk of existing LVLMs when understanding stylized images and how to mitigate it. By constructing a stylized dataset and a comprehensive benchmark, we demonstrate that stylized inputs significantly increase hallucination rates. To explore the underlying causes, we analyze the correlation between generated tokens and image tokens, revealing that the later layers of LVLMs tend to suppress visual information and rely more heavily on language priors. To address this, we propose a training-free mitigation method, SAVER, which dynamically corrects generated tokens by retrieving optimal early layers with dense visual correlation. Extensive experiments across diverse models, datasets, and tasks validate the effectiveness of SAVER. We hope this work contributes to the development of more trustworthy LVLMs and facilitates their application in challenging scenarios.

7 Acknowledgments↩︎

This work was carried out at the Rapid-Rich Object Search (ROSE) Lab, School of Electrical & Electronic Engineering, Nanyang Technological University (NTU), Singapore. This research is supported by the National Research Foundation, Singapore and Infocomm Media Development Authority under its Trust Tech Funding Initiative. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore and Infocomm Media Development Authority.

Appendix↩︎

7.1 Style-POPE Benchmark↩︎

Here we present the Style-POPE hallucination benchmark results on existing LVLMs as shown in Fig.  6, including GPT-4o [1], Gemini-1.5-Pro [24], three variants of LLaVA (LLaVA-1.5-7B, LLaVA-1.5-13B, and LLaVA-1.6) [6], InstructBLIP-7B [21], TinyLLaVA [41], Phi3V [42], IDEFICS2-8B [44], and InternVL2 [51]. We report accuracy, precision, recall, and F1 scores across Adversarial, Popular, and Random settings to comprehensively benchmark the LVLMs’ hallucination performance in various styles. It can be observed that GPT-4o and Gemini-1.5-Pro consistently achieve superior hallucination evaluation scores, demonstrating their outstanding visual understanding performance. In addition, the evaluation results on original photographs largely outperform the other five styles across all POPE metrics, proving that LVLMs tend to produce more hallucinations with style images, aligning with our findings discussed in the main manuscript.

6pt

7.2 Style-Chair Benchmark↩︎

Here we present an extended analysis and results of our Chair benchmark using a different prompt. Tab. 7 summarizes hallucination rates on the Style-CHAIR benchmark using the prompt “Provide a one-sentence caption for the provided image.” Similarly, the benchmark results show existing LVLMs have higher CHAIR scores in five styles. Compared to Tab. 1, the benchmark results in Tab. 7 indicate that more detailed captions may produce more hallucinations, and LVLMs generate more hallucinations for style images. Moreover, variations in model architecture and training data also contribute to differences in hallucination rates.Tab. 8 reports the Style-CHAIR benchmark results for LLaVA-1.5 with a maximum of 512 tokens. The average scores, 11.3/37.5 (Deco) vs. 10.3/35.1 (Ours), demonstrate the effectiveness of our method compared with the SOTA approach, and the per-style results follow the same trend as in the 64-token setting.

Additionally, we employ a SOTA style-transfer model and convert 50 stylized images across five styles back to photographic images to investigate whether the more severe hallucinations are caused by noise introduced by style transfer or by the style itself. As shown in Tab. 9, the average scores (Ci/Cs: 14.9/46.0 vs. 14.3/34.0) indicate that stylized images indeed induce more hallucinations.

7.3 Evaluation on Captioning Quality↩︎

While SAVER is explicitly designed to reduce object hallucinations, maintaining captioning fluency and informativeness is also crucial for practical applications. Tabs. 10, 11, and 12 present standard n–gram–based metrics (BLEU-1–4, METEOR, and ROUGE-L) evaluated on the generated captions of three models: InstructBLIP, LLaVA-1.5, and MiniGPT-4. Overall, these results demonstrate that SAVER can significantly mitigate hallucination while negligibly affecting captioning quality.

7.4 Detailed Hallucination Mitigation Results on Style-POPE↩︎

Tabs. 13, 14, and 15 report the results on the Style-POPE benchmark for MiniGPT-4 [7], InstructBLIP [21], and LLaVA-1.5 [6] across three evaluation settings: Adversarial, Popular, and Random. Our proposed method, SAVER, consistently achieves superior hallucination mitigation performance across various experimental settings, evaluation metrics, and models. Specifically, SAVER enhances accuracy across every style–sampling combination for MiniGPT-4, with the most substantial improvement observed in the Painting style under Random sampling, where accuracy rises significantly from 60.2% to 67.9%. Similarly, for LLaVA-1.5 under the Random sampling setting of Original images, SAVER boosts accuracy from 82.8% to 88.2%, alongside proportional increases in precision and F1 scores. Furthermore, SAVER can still achieve performance gains on InstructBLIP. These results highlight SAVER’s significant improvements across diverse LVLM architectures and sampling strategies, underscoring its effectiveness in mitigating object hallucinations for style images.

7.5 Hallucination Evaluation Results on MME dataset↩︎

MME is a more challenging dataset for LVLM hallucination evaluation, which comprises 14 fine-grained Visual Question Answering (VQA) tasks grouped into two distinct tracks: (i) ten perception-oriented tasks evaluating fundamental visual understanding abilities such as object existence, counting, and color identification; and (ii) four recognition-oriented tasks requiring advanced reasoning abilities, including commonsense inference, numerical calculations, text translation, and code understanding. As shown in the Tabs. 4 and 16, SAVER consistently achieves the highest scores across both perception and recognition tracks, regardless of the evaluated backbone architectures. These results indicate that our method not only effectively mitigates hallucinations but also enhances general visual comprehension. This improvement can be attributed to our method leveraging more visual information aligned with the text cues.

7.6 Real-World Cases↩︎

Benchmark Details. To evaluate SAVER in more practical cases, we first collect a cross-modal dataset from three public datasets:

• Depth–RGB (100 pairs). Indoor scenes from NYU-Depth [52]; each dense depth map (\(640\times480\)) is aligned with a RGB photograph captured from the same viewpoint.

• Thermal–RGB (100 pairs). Thermal and RGB frames from the KAIST Multispectral Pedestrian Benchmark [53]; each long-wave infrared image (\(640\times512\)) has a pixel-wise registered RGB counterpart.

• Medical (50 images). Single-channel X-rays from BenchLLM [54].

Following POPE, we formulate hallucination detection as a binary VQA task. For each image, we pose a concise, domain-specific “Yes/No” query that targets an object:

Objects are selected to (i) exhibit clear visual signatures when present, (ii) occur frequently in text-only corpora to align with real-world scenarios, and (iii) target the specific practical usage. This benchmark comprises 450 images paired with corresponding “Yes/No” queries, providing a rigorous test of LVLM robustness against hallucination. Some example images are shown in Fig.  8.

Experiments on Real-World Benchmark. Tab. 17 reports results on five real-world modalities—Depth, Depth-RGB, Thermal, Thermal-RGB, and Medical—evaluated with three LVLMs: LLaVA-1.5 [6], MiniGPT-4 [7], and InstructBLIP [21]. We compare our SAVER with two baseline hallucination mitigation approaches, Dola [15] and Deco [17]. SAVER exhibits the highest F1 scores under various experimental settings. These findings further demonstrate that our method can be effectively extended to challenging real-world data for hallucination mitigation.

7.7 Ablation Study on Hyperparameters↩︎

Here, we present the detailed ablation studies on five key hyperparameters. The average results are illustrated in Fig. 7, and The detailed results can be found in Tabs. 18, 19, and 20.

Effect of the Scale Factor \(\boldsymbol{\alpha}\). Tuning the scale factor \(\alpha \in \{0.4, 0.6, 0.8, 1.0\}\) modulates the contribution of visual evidence in the fused representation. Higher values of \(\alpha\) enhance visual guidance and generally reduce hallucinations; however, large values occasionally result in overly constrained generation.

Effect of the Confidence Threshold \(\boldsymbol{p}\). The confidence threshold \(p \in \{0.6, 0.7, 0.8, 0.9\}\) controls token selection by filtering candidate tokens based on confidence. Higher thresholds more aggressively suppress hallucinations but may remove relevant content, adversely affecting fluency. Tab. 19 demonstrates this trade-off, where lower thresholds (\(p=0.6\)) yield fewer hallucinations on stylized inputs, whereas intermediate to high thresholds (\(p=0.8\)–\(0.9\)) perform optimally on Original images. This pattern is most evident with MiniGPT-4 and InstructBLIP, suggesting mid-range \(p\) provides an optimal balance between robustness and fluency.

Effect of the Candidate Set Size \(\boldsymbol{k}\). We also examine the effect of the number of candidate tokens \(k \in \{10, 15, 20, 25\}\). Low \(k\) value excludes potentially informative tokens, hindering performance on rare styles, while excessively high \(k\) introduces noisy tokens. From Tabs. 18, 19, and 20, we consistently observe optimal performance at \(k=20\), achieving the lowest average hallucination counts (e.g., LLaVA-1.5 in Tab. 20).

Effect of Candidate Image Representative Tokens \(\boldsymbol{N_i}\). We adjust the number of candidate image representations \(N_i \in \{50, 100, 150, 200\}\) to assess its influence. Moderate values (\(N_i = 50\) or \(100\)) consistently provide the best trade-off by ensuring sufficient diversity to mitigate hallucinations during the decoding process (Tab. 19, LLaVA-1.5). Higher values (\(N_i \geq 150\)) rarely yield further improvements.

Interaction with Early-Exit Depth. Following [17], we adopt the same layer configuration (20-29 layers) as the “Standard” and investigate two alternative exit strategies: Extract 10 layers at equal intervals with in the early 20 layers (low) and final 20 layers (high), to evaluate the impact of depth. The “Standard” policy consistently achieves the best overall hallucination mitigation results on stylized inputs. Notably, the higher layer exhibits better hallucination suppression.

7.8 Qualitative Results↩︎

To qualitatively assess the responses produced by different models and decoding strategies, we present qualitative examples in Figs. 9, 10, and 11. Compared to SOTA methods, SAVER generates captions that maintain more descriptive detail and length while remarkably reducing object hallucinations.

7.9 Hardware Setup↩︎

We run our benchmark and decoding method comparison experiments using one NVIDIA A40 GPU with 48 GB memory.

References↩︎

---

1. Corresponding author↩︎