1 Introduction↩︎

Large Vision-Language Models (LVLMs) [1]–[3] integrate Large Language Models (LLMs) with visual encoders, aligning the extracted visual features with the semantic space of the LLMs to enable comprehensive understanding of visual content. However, they often suffer from hallucination [4]–[6], where the models may generate content that is inconsistent with the visual evidence, thereby severely limiting their reliability in real-world scenarios.

Recent studies have explored methods to mitigate hallucinations in LVLMs. [7] and [8] attribute hallucinations to the influence of language priors, proposing contrastive decoding-based methods to suppress language priors. However, they do not explicitly enhance the model’s visual understanding capabilities. Other studies [9]–[11] argue that LVLMs exhibit biased mechanisms in visual attention distribution, and propose optimizations such as increasing the relative weight of key visual tokens. In this work, however, we point out that these approaches make suboptimal adjustments to the information flow, as they overlook the influence of information aggregation.

In fact, the unidirectional masked generation process in Transformer-based models can be regarded as a form of information flow, where information from earlier tokens in the input or generated sequence flows toward later tokens. Since LVLMs adopt this architecture, their understanding of visual content can be modeled as a process in which visual information flows into the semantic representations (since visual tokens are positioned before the textual tokens). Based on this perspective, we follow the Attention Rollout [12] to conduct an in-depth analysis of information flow within LVLMs. The visualization results are shown in Fig. 1.

Specifically, we use \(\mathbf{F}_{i,j}^{l}\) to represent the contributions of the \(i\text{-th}\) input embedding to the \(j\text{-th}\) representation in layer \(l\). Considering the residual connection in the forward propagation, we formulate the recursive relation as \(\mathbf{F}^l=(\mathbf{I}+\mathbf{W}_\mathrm{attn})/2 \cdot \mathbf{F}^{l-1}\), where \(\mathbf{W}_\mathrm{attn}\) is the attention weight after average pooling across multi-heads. The contributions of each input embedding were normalized to 1 across all representations. We perform layer-wise quantification of visual embeddings’ contributions to the visual and semantic representations, respectively.

As shown in the visualization, visual information progressively flows into semantic representations across layers. In the final layers, the majority of the visual information is integrated into the semantic representations. This indicates that during the forward pass of the LVLMs, visual information is gradually encoded into the semantic representations. While prior works largely concentrate on optimizing the attention distribution to visual features, they neglect the critical insight that essential information has already been integrated into semantic representations, where attention refinement may be more impactful.

To further validate this insight, we mask all visual representations and compare the consistency between the logits distribution of the vision-masked input and that of the regular (unmasked) input, as illustrated in Fig. 2. We observe that applying masking to visual representations starting from earlier layers leads to significant deviations in the model’s outputs, whereas masking from later layers has minimal impact on the final outputs. This finding further supports our view that visual information has already been integrated into the semantic representations in the later layers.

Building upon the above findings, as the number of layers increases, textual representations progressively integrate more visual information and thus play an increasingly important role. Consequently, LVLMs should place greater attention on these integrated representations. To investigate this, we statistically analyze the attention allocation between semantic representations and others at each layer, as illustrated in Fig. 3. The model’s attention to semantic representations is significantly lower than expected, accounting for only about one-fifth of the total attention. This phenomenon stems from the nature of supervision in LVLM training, which is primarily driven by soft supervision from end-to-end data and lacks explicit guidance on the attention distribution. We argue that the model’s attention distribution should align with the actual flow pattern of visual information. Otherwise, hallucinations may arise, as the model does not sufficiently attend to regions enriched with visual information.

To address the aforementioned challenge, we propose SEVI (Semantic-Enhanced Visual Interpretation), a novel training-free approach that augments the model’s attention to those semantic representations that have absorbed visual information. Furtherly, we encourage the model to focus on the most meaningful representations in order to reduce the occurrence of hallucinations. Several studies [13], [14] have revealed that contextual information tends to spontaneously concentrate in some tokens, forming anchor tokens that encapsulate core information. These tokens are typically characterized by receiving disproportionately high attention [13]. Therefore, our method focuses the model’s attention on these core semantic representations at higher layers where information integration and aggregation occur. The core visual information contained within these representations provides more effective guidance for the model’s understanding of visual inputs, thereby mitigating hallucination.

Specifically, we identify superior attention heads within the multi-head attention mechanism—those that attend to core semantic representations—and use their attention distribution as the target distribution to guide the optimization of the model’s overall attention. To achieve this, we design a two-stage attention optimization paradigm, incorporating a smoothing mechanism to ensure stable performance improvements.

We evaluate the effectiveness of our method in mitigating hallucinations by testing it with five LVLMs: InstructBLIP [1], LLaVA-1.5 [3], LLaVA-Next [15], Qwen2-VL-Instruct [16], and Qwen2.5-VL-Instruct [17], on three image captioning benchmarks: CHAIR [18], AMBER [19], and DetailCaps [20]. Experimental results show that our approach significantly reduces hallucinations while supporting manual adjustment of the model’s conservativeness, enabling a flexible trade-off between cautious outputs (fewer hallucinations) and comprehensive descriptions (richer details).

Our main contributions are as follows:

• We reveal that visual information is indeed integrated into semantic representations. However, the model’s attention distribution does not align with this flow pattern, leading to hallucinations.

• We enhance the visual understanding capabilities of LVLMs by leveraging semantic representations. Specifically, we guide the model’s attention toward core semantic representations through a two-stage optimization paradigm, effectively reducing hallucinations.

• We evaluate our method on the image captioning task, demonstrating its effectiveness in significantly reducing hallucinations across three benchmark datasets and five LVLMs. We observe a trade-off between reducing hallucinations and preserving detail, with our method enabling controllable conservativeness for task-specific needs.

2 Related Work↩︎

Various approaches have been proposed to mitigate hallucinations in LVLMs, such as improving training data quality [21]–[23] or designing specific data formats [24]–[26] to enhance model reliability. However, these training-based methods often suffer from limited scalability.

In recent years, the rapid evolution of LVLM backbones has brought increasing attention to training-free methods. Early approaches [27]–[29] primarily relied on post-processing techniques to correct hallucinated content after generation. However, the complexity of such pipelines and their dependence on external modules limit their practical usability. Researchers have since shifted focus toward investigating the root causes of hallucinations, exploring targeted optimizations in both the decoding strategies and the decoding processes. Some methods [7], [8], [30], [31] apply contrastive decoding algorithms to refine the logits, thereby reducing the likelihood of hallucination-related tokens. While these approaches improve the expressiveness of the model and enhance the transfer of visual information into textual form, they do not strengthen the model’s intrinsic cross-modal understanding. Other works have proposed enhancements to the attention mechanism, such as correcting visual positional bias [32], [33], directly boosting visual attention [11], [34], or redistributing attention weights [9], [10]. However, these methods primarily concentrate on visual attention and overlook the vision-integrated semantic representation.

In contrast to these approaches, we analyze the information flow between the visual and textual modalities in LVLMs and focus the model’s attention on core information within semantic representations, thereby reducing hallucinations more effectively.

3 Preliminaries↩︎

LVLMs consist of a visual encoder and an LLM backbone. High-dimensional features extracted by the visual encoder are concatenated with textual embeddings and jointly fed into the LLM. Subsequently, the visual and semantic features undergo cross-modal interaction through the Self-Attention layers within the LLM. The cross-modal interaction in layer \(l\) simplified as \(\Delta \mathbf{X}^l = \text{softmax}(\mathbf{W}) \mathbf{X}^l\), where \(\mathbf{X}^l = [\mathbf{X}^l_V: \mathbf{X}^l_T]\) denotes the hidden states of visual features \(\mathbf{X}^l_V\) and semantic features \(\mathbf{X}^l_T\). \(\mathbf{W} \in \mathbb{R}^{H \times L \times L}\) is the attention weights, \(H\) is the number of attention head and \(L\) is the model’s context length.

When generating the (\(j\)+1)-th token, the model’s attention to feature \(\mathbf{X}^l_i\) at layer \(l\) across all heads is quantified by the weight value \(\mathbf{W}_{i,j}\). A larger \(\mathbf{W}_{i,j}\) indicates that the model’s current layer relies more heavily on the information from \(\mathbf{X}^l_i\) when determining the generated content. Therefore, we can intuitively infer that more important features \(\mathbf{X}^l_i\) should generally correspond to larger attention weights \(\mathbf{W}_{i,j}\).

4 Method↩︎

We attribute the occurrence of hallucinations to a misalignment between the attention distribution of LVLMs and the visual information flow. To address this, we enhance the visual interpretation capabilities of LVLMs by leveraging semantic representations that encapsulate visual information. We first identify superior attention heads that focus on core semantic representations, and then propagate their strengths to other heads through a two-stage optimization paradigm, as illustrated in Fig. 4.

4.1 Attention Distribution Alignment↩︎

Considering that visual information is progressively integrated into semantic representations through forward propagation—and that only a few of these representations aggregate core information—an intuitive approach is to increase the model’s attention to these core semantic representations in the later layers. However, rigidly modifying the model’s attention distribution may disrupt its inherent latent features, potentially resulting in suboptimal performance.

In the multi-head attention mechanism, attention heads naturally diversify after training, forming a heterogeneous and specialized structure [35]. Accordingly, we identify superior attention heads that attend to core semantic representations and transfer their attention patterns to other heads, leading to a global refinement of the model’s attention distribution.

Specifically, at each layer, we calculate the attention weight of each head with respect to the semantic representations. We then identify those heads that allocate more than 50% of their attention to semantic representations: \[h \in \left\{ \begin{align} H_S \quad &\text{ if } \sum \mathbf{W}^h_S > \sum \mathbf{W}^h_O \\ H_O \quad &\text{ if } \sum \mathbf{W}^h_S \le \sum\mathbf{W}^h_O \end{align} \right.\] \(\mathbf{W}^h_S\) and \(\mathbf{W}^h_S\) respectively denote the attention weights of head \(h\) to semantic and other representations. \(H_S\) denotes the set of heads that focus more on semantic representations, whereas \(H_O\) denotes other heads. \(H_S\) are further categorized based on their attention distributions into Core Semantic Heads (\(H_{S_c}\)), which focus on core semantic representations, and Global Semantic Heads (\(H_{S_g}\)), which fail to effectively attend to them: \[\label{Eq95kp} h \in \left\{ \begin{align} H_{S_c} \quad &\text{if} \quad \max \mathbf{W}^h_S > \kappa \cdot \sum \mathbf{W}^h_S \\ H_{S_g} \quad &\text{if} \quad \max \mathbf{W}^h_S \le \kappa \cdot \sum \mathbf{W}^h_S \end{align} \right.\tag{1}\] where \(\kappa\) is a hyperparameter and \(H_{S_c} \cup H_{S_g} = H_S\) holds. We then take the core semantic heads to construct a target distribution to optimize the model’s attention distribution.

To ensure stable information integration within the model, we propose a two-stage feature optimization paradigm that more naturally guides the model’s understanding of visual information.

In the first stage, we use the global semantic heads to guide the other heads, aligning the model’s attention transition to semantic representations and harmonizing with the visual information flow. In the second stage, we leverage the core semantic heads to guide the global semantic heads, encouraging the model to concentrate more on critical information and thereby suppressing hallucinations.

The model’s internal attention is then optimized through a two-stage refinement process.

During this process, we incorporate a weighted smoothing mechanism to adjust the attention distribution of each head:

\[\hat{\mathbf{W}}^h = \frac{\mathbf{W}^h + \omega \mathbf{W}'}{1+\omega}\] where \(\hat{\mathbf{W}}^h\) denotes the optimized attention weights of head \(h\), \(\mathbf{W}'\) represents the target weight distribution derived from superior heads, and \(\omega\) is a hyperparameter that controls the extent of the modification.

We first use the global semantic heads to guide the other heads, aiming to align the model’s attention distribution with the flow of visual information. To this end, we apply average pooling across the global semantic heads to construct the target attention distribution: \[\hat{\mathbf{W}}^h_O = \frac{\mathbf{W}^h_O + \omega \cdot \frac{1}{|H_{S_g}|} \sum_{t \in H_{S_g}} \mathbf{W}^t_{S_g}}{1+\omega},\] Subsequently, we use the core semantic heads to guide the global semantic heads, with the goal of enhancing the model’s focus on critical information. Here, we employ max pooling across core semantic heads to form the corresponding target distribution: \[\hat{\mathbf{W}}^h_{S_g} = \frac{\mathbf{W}^h_{S_g} + \omega \cdot \max_{t \in H_{S_c}} \mathbf{W}^t_{S_c}}{1+\omega}\]

We provide the algorithm of the two-stage optimization process in Appendix 7.

4.2 Mitigating Aggravated Language Priors↩︎

We optimize the model’s attention distribution to align with the flow pattern of visual information, which ultimately results in increased attention to semantic representations. However, while semantic representations incorporate visual information, they also inherently contain their own contextual features. As a result, increasing the model’s reliance on semantic representations will exacerbate the issue of language priors [26], [38], [39].

To address this issue, we employ CICD [31] to eliminate language priors while preserving visual information. CICD uses the cross-image consistency of language priors to identify detrimental priors and mitigate them by contrastive decoding: \[\begin{align} \mathrm{logit}(y_t \mid v, x, y_{<t}) =\;& (1 + \alpha)\, \hat{\mathrm{logit}}_{\theta}(y_t \mid v, x, y_{<t}) \\ & - \alpha\, \mathrm{logit}_{\theta}(y_t \mid v', x, y_{<t}) \end{align}\] where \(\hat{\mathrm{logit}}_{\theta}\) is the logit distribution after attention distribution alignment and \(\mathrm{logit}_{\theta}\) is the regular logit distribution with a distinct image \(v'\). \(\alpha\) is defined as follows: \[\alpha = 1 - \log_{10}(\mathrm{JSD}(\hat{\mathrm{logit}}_{\theta}; \mathrm{logit}_{\theta}))\] where \(\mathrm{JSD}(\cdot; \cdot)\) represents Jensen-Shannon Divergence.

5 Experiments↩︎

5.1 Experimental Setup↩︎

Due to space constraints, we present only the key aspects of our experimental setup here. Detailed settings can be found in Appendix 8.

5.1.0.1 Implementation Details

Our method guides the model to focus on core information, thereby reducing the occurrence of hallucinations. In practice, we observe a trade-off between hallucination suppression and the richness of generated details, influenced by the model’s degree of attentional focus. To address this, we introduce hyperparameters to control the model’s focus on the core information. Based on empirical observations, we design two sets of hyperparameter configurations: (1) Focused mode that aggressively minimizes hallucinations, and (2) Balanced mode that strikes a compromise between detail retention and hallucination reduction. Specifically, Focused mode optimizes attention distribution starting from the 5th layer with \(\omega = 4\), while Balanced mode begins from the 9th layer with \(\omega = 0.5\).

In addition, we set the parameter \(\kappa\) in Eq. 1 to 0.2. Further details are discussed in Appendix 8.3. We employ sampling decoding for the next-token prediction with default settings. All experiments are performed on a single NVIDIA A800 40G GPU.

5.1.0.2 Benchmarks

We evaluate the performance of our method across three widely adopted multimodal hallucination benchmarks on the image captioning task. These include CHAIR [18], DetailCaps [20], and AMBER [19].

5.1.0.3 Evaluated LVLMs

To examine the generalizability of our approach, we apply it to five LVLMs drawn from three representative model families: InstructBLIP [1]; two models from the LLaVA family (LLaVA-1.5 [3] and LLaVA-Next [15]); and two from the Qwen series (Qwen2-VL-Instruct [16] and Qwen2.5-VL-Instruct [17]). All models are tested at the 7B parameter scale unless explicitly noted otherwise.

5.1.0.4 Baselines

We conduct a comparison between our method and several SOTA de-hallucination techniques: VCD [7], ICD [30], IBD [34], PAI [36], VAR [10], VAF [9], AD-HH [37], DeGF [8], and CICD [31].

5.2 Main Results↩︎

5.2.0.1 Results on CHAIR

CHAIR is a benchmark designed to detect object hallucinations in image captions, relying on human annotations to provide reliable ground truth. Following common practice, we conduct experiments with maximum sequence lengths set to 64 and 512, respectively. The results are shown in Fig. 1. Our method effectively reduces hallucinations by guiding the model’s attention toward core information, achieving particularly low hallucination rates under the focused mode. However, we observe that when the model concentrates on critical information, it tends to become more conservative, as reflected by a slight drop in recall. In contrast, the balanced mode allows the model to capture more details while still maintaining a low hallucination rate, resulting in the best overall performance. In addition, our method yields the most pronounced improvement on the 13B model relative to the 7B model, indicating its effectiveness in harnessing the latent capacity of larger models for visual understanding.

5.2.0.2 Results on DetailCaps

DetailCaps evaluates the correctness of captions in terms of objects, attributes, and relationships, incorporating both exact-match and soft-match metrics. The experimental results are presented in Tab. 2. Our method outperforms existing approaches in both hallucination rate and overall caption correctness. Moreover, the focused mode achieves a higher CPAURE score, indicating not only a lower incidence of hallucinations but also greater accuracy in the described content. This highlights the practical value of our approach for real-world applications.

5.2.0.3 Results on AMBER

AMBER carefully selects high-quality images to construct its benchmark and provides a more fine-grained evaluation of object hallucinations. The experimental results are shown in Tab. 3. Our method achieves a significantly lower hallucination rate while maintaining a comparable Cover score to other approaches, resulting in the best overall performance.

5.3 Visual or Semantic Representations, Which Should LVLMs Focus On?↩︎

Some works [34], [36] emphasize the importance of visual attention, whereas we encourage the model to focus more on semantic representations. To investigate whether the model should prioritize visual or semantic representations, we take PAI [36] as a contrast, which directly increases the model’s attention to visual representations.

As we analyze in Fig. 1 and Fig. 2, visual information resides in visual representations during the early layers of LVLMs and is gradually integrated into semantic representations in later layers. Therefore, we hypothesize that LVLMs should focus on visual representations in the lower layers and shift to semantic representations in the higher layers. We perform ablation studies on both lower and higher layers within the PAI to better understand the layer-wise impact. The results at the top of Tab. 5 are consistent with our hypothesis, indicating that it is more beneficial for the model to focus on visual representations in the lower layers. Furthermore, we explore combining PAI with our method by applying PAI to the lower layers (layers [2–15]) to enhance visual attention, while adopting our approach in the higher layers (layers [16–31]). The results at the bottom of Tab. 5 further demonstrate our hypothesis.

5.4 Effectiveness on more LVLMs↩︎

We evaluated our method on several state-of-the-art LVLMs, with the experimental results presented in Tab. 4. Across these models, our method continues to demonstrate strong hallucination suppression capabilities, with both modes exhibiting their expected effectiveness. These results validate the excellent generalization ability of our approach and highlight its practical potential as a training-free solution.

5.5 Ablation Study↩︎

Through a two-stage optimization paradigm, we reallocate the model’s attention toward the semantic representations, encouraging it to focus on the core visual information and thereby reducing hallucinations. In addition, we employ CICD to address the issue of language priors that aggravate when the model over-focuses on semantic representations. We conduct ablation studies on these three components, with the results presented in Tab. 6. The core semantic heads play a pivotal role in guiding the model’s focus, and removing them in the ablation study leads to a noticeable increase in hallucination rate. The global semantic heads help modulate the cross-modal attention distribution, aligning it with the flow of visual information, which also contributes significantly to hallucination reduction. The combination of both components has a synergistic effect, and incorporating the CICD method to mitigate language priors, amplified by overattending to semantic representations, further enhances the performance of the model. Additionally, the two-stage optimization paradigm also has a positive effect, making the optimization process more stable and smooth. The experimental results confirm the effectiveness and soundness of our method’s design, highlighting the value of each component.

6 Conclusion↩︎

We systematically analyze the information flow in LVLMs and verify that visual information is indeed integrated into the semantic representations. However, the model’s attention remains predominantly focused on the visual representation. This inconsistency impairs the model’s visual understanding and contributes to hallucination. To address this, we propose Semantic-Enhanced Visual Interpretation (SEVI), a method that guides the model’s attention toward the core components of semantic representations through a two-stage optimization process. Extensive experiments demonstrate that our approach significantly mitigates hallucinations.

While our method optimizes the attention distribution in LVLMs based on the underlying information flow, it does not directly enhance the flow mechanism itself. In future work, we plan to explore more efficient strategies for visual information propagation.

Limitations↩︎

Our method guides the model to focus on the most critical information, thereby reducing the occurrence of hallucinations. During our experiments, we observed a trade-off mechanism in the model’s focusing process: excessive focus leads to a more conservative image captioning. In an effort to avoid hallucinated content, the model may overlook some details, which is reflected in a slight decrease in recall. Although our method demonstrates superior overall performance—showing a significant advantage on comprehensive metrics such as CAPTURE—it still faces a trade-off between reducing hallucinations and generating more details. To address this issue, we introduce hyperparameters to control the model’s level of conservativeness, allowing users to manually adjust the behavior based on specific application scenarios. Furthermore, as a training-free approach, our method offers greater usability but is inherently limited by the performance ceiling of the model itself.

Acknowledgments↩︎

We thank all reviewers for their detailed reviews and valuable comments. This work is supported by Zhongguancun Academy Project No.20240103.

7 Two-Stage Optimization↩︎

We present the algorithm of our method in Algorithm [alg:attn95align].

8 Detailed Experiments↩︎

8.1 Benchmarks↩︎

We evaluate the performance of our method across three widely adopted multimodal hallucination benchmarks on the image captioning task. These include CHAIR, DetailCap, and AMBER.

8.1.0.1 CHAIR

[18] evaluates the proportion of hallucinated objects—those generated by the model but not present in the reference annotations. Following prior work, we randomly sample 500 images from the MSCOCO [40] dataset for evaluation. CHAIRs and CHAIRi are the main metrics to evaluate hallucination: \[\begin{align} \text{CHAIRs} = \frac{|\text{Hallucinated Objects}|}{|\text{All Objects}|}, \\ \text{CHAIRi} = \frac{|\text{Hallucinated Sentences}|}{|\text{All Sentences}|} \end{align}\]

8.1.0.2 DetailCaps

[20] is a fine-grained image captioning benchmark, accompanied by ground-truth detail captions generated by GPT-4V [41], Gemini 1.5 Pro [42], and GPT-4o [43] for evaluation purposes. It comprises 4,870 images from various datasets; we use a subset of 700 images from MSCOCO in our experiments. CAPTURE evaluates the alignment between generated and reference captions by computing F1 scores using both hard and soft matching across three semantic aspects: entities (\(F1_{\text{obj}}\)), attributes (\(F1_{\text{attr}}\)), and relations (\(F1_{\text{rel}}\)). The final score is calculated as a weighted average: \[\text{CAPTURE} = \frac{\alpha F1_{\text{obj}} + \beta F1_{\text{attr}} + \gamma F1_{\text{rel}}}{\alpha + \beta + \gamma}\] where \(\alpha = 5\), \(\beta = 5\), and \(\gamma = 2\).

8.1.0.3 AMBER

[19] contains 1,004 carefully curated high-quality images, each with manually annotated objects. AMBER contains multiple evaluation metrics: CHAIR, Cover, Hal, and Cog. Given a list of annotated objects \(A_{obj} = { obj_1^A, obj_2^A, \cdots, obj_n^A }\) and a set of generated objects \(R'_{obj}\), each metric is defined as follows: \[\begin{align} \text{CHAIR} &= 1 - \frac{\mathrm{len}(R'_{obj} \cap A_{obj})}{\mathrm{len}(R'_{obj})}, \\ \text{Cover} &= \frac{\mathrm{len}(R'_{obj} \cap A_{obj})}{\mathrm{len}(A_{obj})}, \\ \text{Hal} &= \frac{|{ \text{CHAIR} > 0 }|}{|\text{All Captions}|}, \\ \text{Cog} &= \frac{\mathrm{len}(R'_{obj} \cap H_{obj})}{\mathrm{len}(R'_{obj})} \end{align}\] where \(H_{obj}\) denotes the set of hallucinated target objects generated by LVLMs, and All Captions refers to the total number of generated captions.

8.2 Baselines↩︎

We conduct a comparison between our method and several SOTA de-hallucination techniques:

• VCD [7] introduces Gaussian noise into images to activate language priors, thereby constructing negative contexts and removing these priors through contrastive decoding. However, this approach leads to a loss of visual information. Moreover, the use of noisy images deviates from the model’s training distribution, potentially causing performance degradation.

• ICD [30] constructs negative contexts by designing adversarial instructions and applies contrastive decoding to mitigate their influence. Like VCD, it faces challenges such as visual information loss and performance bias.

• IBD [34] strengthens the model’s focus on visual information by using the original context as a negative reference, and further refines contrastive decoding via inter-layer and inter-context consistency mechanisms.

• VAR [10] reallocates attention from sink visual tokens to other visual tokens, allowing the model to capture more detailed visual information.

• VAF [9] rebalances the attention allocation between instructions and visual inputs, redirecting attention from the textual instructions toward the visual information.

• DeGF [8] generates negative contexts via cross-modal back-translation and strengthens consistency signals throughout the contrastive decoding process.

• CICD [31] leverages the consistency of language priors across images by using different images to construct negative contexts. It detects detrimental priors via consistency analysis and removes them through contrastive decoding, while retaining beneficial priors essential for accurate understanding.

8.3 Hyperparameters↩︎

We investigate the appropriate setting of hyperparameters using LLaVA-1.5-7B. The parameter \(\kappa\) serves as the threshold for distinguishing between core semantic heads and global semantic heads. We analyze the peak attention values of semantic heads, as shown in Fig. 5. Based on this analysis, we explore the impact of \(\kappa\) under the focused mode, with the results presented in Tab.7. A higher threshold encourages the model to attend to more central information, resulting in a lower hallucination rate but also a reduced recall. To balance these trade-offs, we define a heuristic metric to evaluate the model’s overall performance. Based on the experimental results, we set \(\kappa=0.2\).

The attention adjustment strength \(\omega\) and the starting layer (SL) for attention optimization are interdependent. Therefore, we perform a joint search over these two hyperparameters, with the results summarized in Tab.8. Based on two heuristic evaluation metrics, we select two representative hyperparameter configurations, corresponding to the Focused mode (\(\omega\)=4, SL=5) and Balanced modes (\(\omega\)=0.5, SL=9).

9 Case Studies↩︎

To demonstrate the effectiveness of our method in mitigating hallucinations, we provide qualitative case studies. We select one simple image (Fig. 6) and one complex image (Fig. 7) as case studies. The captions were generated by LLaVA-1.5, with hallucinated content highlighted in red and image-consistent content highlighted in green.

It can be observed that Regular Decoding produces a large amount of hallucinated content, even in the simple image. In contrast, our method effectively reduces the occurrence of such hallucinations. Both the Focused and Balanced modes significantly mitigate hallucinations; however, the Focused mode makes the model overly conservative, potentially overlooking fine-grained details. For example, in Fig. 7, the Focused mode fails to capture items such as knife and fork, whereas the Balanced mode not only reduces hallucinations but also preserves more detailed information.

References↩︎