1 Introduction↩︎

Instruction-tuning enhances Large Language Models’ (LLMs’) ability to understand and align with human intentions [1], [2]. Through this tuning, LLMs are better equipped to follow instructions across diverse tasks, reduce biased or harmful responses, and improve both flexibility and safety [3], [4]. However, this tuning may also heighten LLMs’ dependence on user inputs, making them more likely to follow external information even if it conflicts with their own parametric knowledge [5], [6]. We highlight a problematic situation in which users provide misinformation, and instruction-tuned LLMs, adhering strictly to these inputs, consequently generate hallucinations. This issue is particularly critical given that LLMs are often used to respond to contexts or documents provided by users. As shown in Figure 1, if users provide misinformation, LLMs may accept it without verification, thereby increasing the risk of hallucinations [7], [8]. This susceptibility may be exacerbated by the instruction-tuning, which can overly predispose LLMs to adhere to user inputs.

Unfortunately, studies that deeply investigate how instruction-tuning affects LLMs’ susceptibility to misinformation are scarce. Although some research has indicated that LLMs are highly receptive to external evidence that contradicts their parametric memory [6], [9], these studies have largely focused on identifying the issue rather than investigating its underlying causes. To bridge this gap, we aim to provide new insights for developing and deploying more reliable LLMs by conducting an in-depth analysis of the impact of instruction-tuning on LLMs’ susceptibility to misinformation provided by users. To the best of our knowledge, this is the first study to address this issue.

Instruction-tuned LLMs generate responses by consistently conditioning the generation process on instruction in user prompts [10]. Also, these models use chat templates that structurally distinguish between the roles of “user” and “assistant” when processing prompts. Building on these two features, it is plausible that instruction-tuned LLMs may place relatively greater emphasis on the user-role. To investigate this, we take a two-way approach. First, to compare the influence of the user and assistant roles, we present misinformation through each role and assess the model’s susceptibility to it. Second, we investigate whether presenting misinformation as a separate user-role turn amplifies the models’ focus on it, making it more prominent in the response generation process. To validate these hypotheses, we design experimental scenarios that examine how the user-role shapes LLMs’ susceptibility to misinformation. For the experiments, we used the Farm dataset [11], which contains misinformation, and conducted evaluations on two proprietary and four open-source LLMs. Building on this approach, we investigate the following research questions:
RQ1. Are instruction-tuned LLMs highly susceptible to misinformation when it is presented through the user-role? This research question explores whether instruction-tuned LLMs are more likely to accept misinformation presented in the user-role. Experimental results indicate that most models are more susceptible to misinformation when it was presented by the user-role rather than the assistant-role. Furthermore, when the misinformation was introduced as a separate user-role turn, the model’s susceptibility increased even further. These findings suggest that instruction-tuned LLMs are highly susceptible to misinformation embedded in the user-role.
RQ2. Does instruction-tuning make LLMs more susceptible to misinformation presented through the user-role? This question investigates whether the trend observed in RQ1 stems from instruction-tuning. A comparison between four open-source models and their base versions revealed that, before instruction-tuning, all base models were most susceptible to misinformation from the assistant-role. However, after instruction-tuning, three out of four models were more susceptible to misinformation from the user-role. This result suggests that instruction-tuning shifts models to be more user-focused, increasing their susceptibility to misinformation presented by the user-role.
RQ3. What other factors influence the susceptibility pattern of instruction-tuned LLMs to misinformation? This question explores other potential factors that may influence an instruction-tuned LLMs’ susceptibility pattern to misinformation.

1) Misinformation Length We conducted experiments using misinformation of three different lengths. In most cases, as the length of misinformation increased, the models’ behavior aligned more closely with that of the base models observed in RQ2. This suggests that the influence of instruction-tuning, which increases susceptibility to the user-role, diminishes as misinformation length grows.

2) Misinformation Warning In an experiment in which a simple misinformation warning was added to system prompt, the two proprietary models showed a decrease in susceptibility to misinformation, whereas the four open-source models showed no significant change. These results indicate that the effectiveness of a simple warning depends on the model’s capabilities, highlighting the need for approaches that mitigate the unintended side effects of instruction-tuning.
We examine how instruction-tuning affects an LLMs’ susceptibility to misinformation, emphasizing the need for a systematic approach to reduce hallucinations. We hope our findings, which identify key factors influencing susceptibility to misinformation, will contribute to improving the reliability and practical use of LLMs.

2 Related Work↩︎

Knowledge Conflict LLMs show diverse behavioral patterns when faced with knowledge conflicts. When presented with external information that contradicts their parametric knowledge, they tend to accept it [6]. Conversely, when given information that aligns with their parametric knowledge, they often demonstrate a strong confirmation bias [9]. Furthermore, even if they initially reject conflicting information, they may revise their beliefs when the information is repeatedly presented or when the user persistently challenges their responses [11], [12].

This tendency can lead to significant issues when misinformation is introduced. In particular, third parties may deliberately insert false information into documents [7] or manipulate LLM responses through prompt injection attacks [13], [14]. To address these challenges, various approaches have been proposed, categorized into (1) methods for detecting misinformation and (2) strategies that integrate context with parametric memory to generate reliable responses. The first approach includes techniques such as issuing warnings via system prompts [11], assessing reliability using redundant information across large corpora [15], and fine-tuning a separate model as a discriminator [16]. The second approach involves leveraging models that evaluate the consistency between generated responses and retrieved documents [17] or applying contrastive learning to select highly reliable responses [18].

However, these methods primarily focus on mitigating the issue rather than fundamentally understanding why LLMs are highly susceptible to misinformation. Therefore, this study aims to conduct a more in-depth analysis of LLMs’ high dependency on misinformation and the underlying mechanisms that drive this phenomenon.
Instruction-tuned LLMs Recently, LLMs have demonstrated enhanced instruction-following capabilities through instruction-tuning, enabling them to effectively handle a wide range of real-world tasks. Notable instruction-tuned LLMs include InstructGPT [19], ChatGPT [20], and Claude [21]. However, these models show an excessive tendency to comply with human instructions, raising concerns about potential risks. For instance, [22] reported that human-aligned LLMs are prone to sycophancy, showing an excessive inclination to conform to user opinions. Furthermore, [5] argued that this tendency becomes more pronounced as model size increases.

Our study explores how the side effects of instruction-tuning appear and analyze how instruction-tuned LLMs’ susceptibility changes when exposed to misinformation. This highlights the importance of a systematic understanding of instruction-tuning to mitigate hallucinations.

3 Experimental Design↩︎

This section provides a detailed description of the experimental design. Section 3.1 presents an overview of the dataset used in our experiments. Section 3.2 describes three key experimental scenarios that examine how instruction-tuning affects LLMs’ susceptibility to misinformation. Finally, Section 3.3 describes the evaluation metric used to assess the models’ susceptibility to misinformation.

3.1 Dataset↩︎

We used the Farm dataset [11] in our experiment. This dataset consists of a selection of questions from BoolQ [23], Natural Questions (NQ) [24], and TruthfulQA [25] that GPT-4 can easily answer in a closed-book setting. The dataset follows a multiple-choice question (MCQ) format comprising a question, answer options, and misinformation, which consist of three paragraphs. The misinformation supports one of the incorrect options. In our experiment, we used only the first paragraph of the misinformation for RQ1 and RQ2. For RQ3, we used the full misinformation to examine the relationship between misinformation length and susceptibility. Further details on the dataset are provided in Appendix 6.

3.2 Test Scenario↩︎

Instruction-tuned LLMs strongly rely on user prompts [5], [6] and consistently generate responses based on them [10]. Since these models distinguish between the “user” and “assistant” roles through chat templates, we hypothesize that they allocate relatively greater attention to the user-role. To test this, we conduct an analysis in two perspectives. First, we compare the influence of the "user" and "assistant" roles by presenting misinformation through each role and evaluating how susceptible LLMs are to misinformation. Second, we explore whether LLMs are more likely to accept misinformation when it is presented as a separate user-role turn. To verify these effects, we designed three experimental scenarios to measure how the user-role influences the responses of LLMs. To ensure consistency and fairness across all experiments, we used the same system prompt. Illustrations of each scenario is presented in Figure 2.

Single-Turn Query Single-Turn Query (STQ) is the most simplest form of document-based question answering format. In a single user-role turn, the query consists of a question, a set of options, and misinformation. This serves as a baseline for evaluating how LLMs process misinformation.
Assistant-Provided Document & User-Provided Document The Assistant-Provided Document (APD) and User-Provided Document (UPD) scenarios assess how LLMs’ susceptibility changes depending on whether the misinformation is provided by the assistant or the user-role. Unlike STQ, these scenarios present the misinformation in a separate turn. This structure minimizes the influence of non-document context, making it easier to isolate and analyze the document’s impact based on its assigned role.
In all experimental scenarios, the final user-role turn includes the question, “What is the correct answer?”. This is necessary in the APD scenario since when the conversation follows a User-Assistant structure, the model requires an additional user prompt to generate a response. To maintain consistency across conditions, we included the same question in STQ and UPD scenarios as well. This ensures a fair comparison between scenarios.

3.3 Evaluation Metric↩︎

We used the Misinformation Susceptibility Rate (MSR) metric to measure how susceptible LLMs are to misinformation. MSR is defined as follows:

\[\text{MSR(\%)}=\frac{|\mathcal{Q}_{✔} \cap \mathcal{Q}_\boldsymbol{\ding{55}}\mathtt{@m}|}{\left| \mathcal{Q}_{✔} \right|}\times100\]

Here, \(\mathcal{Q}_{✔}\) represents the set of questions from the full dataset \(\mathcal{Q}\) that LLMs correctly answer in a closed-book setting. These are considered part of the models’ parametric knowledge [26]. Meanwhile, \(\mathcal{Q}_{\boldsymbol{\ding{55}}}\mathtt{@m}\) represents the set of questions where LLMs, given misinformation, select an incorrect answer that align with the misinformation. This MSR score quantifies how often instruction-tuned LLMs disregard their parametric knowledge and instead adopt misinformation that contradicts the correct answer.

4 Experiment & Analysis↩︎

4.1 Target Models↩︎

We conducted experiments on two proprietary models and four open-source models. The proprietary models include GPT-4o [27] and GPT-4o mini [28]. For open-source models, we used Llama-3-8B-Instruct, Llama-3.1-8B-Instruct [29], Qwen2.5-7B-Instruct [30], and Mistral-7B-Instruct-v0.3 [31]. For comparison with base models in 4.3, we used Llama-3-8B, Llama-3.1-8B [29], Qwen2.5-7B [30], and Mistral-7B-v0.3 [31]. For all models, we set top-p to 1 and temperature to 0.2 for generation.

4.2 RQ1. Are instruction-tuned LLMs highly susceptible to misinformation when it is presented through the user-role?↩︎

In this section, we investigate whether instruction-tuned LLMs show high susceptibility to misinformation when it is presented through the user-role. To verify this, we conducted the experiments described in Section 3. The results are visually presented in Figure 3, with detailed numerical values available in Table 3.
Susceptibility to Misinformation by Role (APD vs. UPD) Experimental results show that, except for Qwen2.5-7B-Instruct, all models had higher MSR scores in UPD than APD across all datasets. This indicates that models are more likely to accept misinformation when presented through the user-role rather than the assistant. However, since each model undergoes a different training process, Qwen2.5-7B-Instruct may have shown the opposite trend due to these differences.
Amplifying Misinformation Influence through user-role Separation (STQ vs. UPD) Across all models and datasets, UPD consistently recorded higher MSR scores than STQ. For most models, the difference ranged between 5%p and 8%p, while Mistral-7B-Instruct-v0.3 showed a particularly large gap, averaging 22%p. This indicates that when misinformation is presented as a separate user-role turn, the models are more susceptible.

These findings suggest that LLMs are highly susceptible to misinformation when it is presented through the user-role, as they not only exhibit greater susceptibility compared to the assistant but also demonstrate increased susceptibility when misinformation is separated into an independent user-role turn.

How Models Handle the assistant-role (STQ vs. APD) The comparison between STQ and APD showed mixed results, varying by model and dataset. For example, in Llama-3-8B-Instruct, STQ had a higher MSR score than APD, whereas in Mistral-7B-Instruct-v0.3 and Qwen2.5-7B-Instruct, APD scored higher than STQ. As observed in the previous analysis, models tend to focus more on misinformation when it is presented as an independent user-role turn. However, when misinformation was presented in a separate assistant-role turn, some models showed a decrease in MSR compared to STQ. This suggests that certain models do not treat the assistant-role the same way as the user-role and tend to disregard it. On the other hand, some models showed a slight increase in MSR in APD (though not as much as in UPD), indicating that they still assign some weight to the assistant-role. These findings highlight significant differences in how models process the assistant-role compared to the user-role.

4.3 RQ2. Does instruction-tuning make LLMs more susceptible to misinformation presented through the user-role?↩︎

In Section 4.2, we found that the instruction-tuned LLMs are highly susceptible to misinformation presented in the user-role. However, it is unclear whether this tendency results from instruction-tuning itself or if it originates from characteristics developed during pre-training. While we suspect instruction-tuning plays a primary factor, a clear attribution requires comparison with base models that have not undergone instruction-tuning.

To this end, we conducted the same experiment on four open-source models using their base versions (i.e., before instruction-tuning), with the results presented in Figure 4. We also visualized the scenario-specific ranking changes before and after instruction-tuning in Figure 5. Detailed experimental results for the base models can be found in Table 4.

Base Models’ Susceptibility Pattern Experimental results show that all base models follow a consistent ranking pattern across the three datasets. This suggests that even without instruction-tuning, models can distinguish between roles and develop preferences for assigning greater weight to specific roles during pre-training. As shown in Figure 4, all base models consistently rank APD the highest. This indicates that during pre-training, models are trained to pay greater attention to the assistant-role. Conversely, in three out of the four models (excluding Qwen2.5-7B), UPD ranks higher than STQ, suggesting that, similar to instruction-tuned models, base models are also more susceptible to misinformation when it is presented in a separate turn.

The Impact of Instruction-Tuning As shown in Figure 5, instruction-tuning changes the ranking of the scenarios. In three out of the four models (excluding Qwen2.5-7B), APD dropped in ranking, while UPD recorded the highest MSR score. This suggests that instruction-tuning reduces the models’ reliance on the assistant-role while increasing the influence of the user-role. These findings indicate that the high susceptibility of instruction-tuned LLMs to misinformation from the user-role is not simply a byproduct of pre-training but rather a direct result of instruction-tuning. Since instruction-tuning aligns the model more closely with user instructions, it prioritizes the user-role, amplifying the effect of UPD. However, Qwen2.5-7B showed a slightly different trend compared to other models. This variation could stem from differences in model architecture, pre-training data, or instruction-tuning configurations.

4.4 RQ3. What other factors influence the susceptibility pattern of instruction-tuned LLMs to misinformation?↩︎

Through RQ1 and RQ2, we found that instruction-tuning is a key factor that makes LLMs more susceptible to misinformation provided by the user-role. In other words, while instruction-tuning enhances LLMs’ ability to follow user instructions, it also makes the model more susceptible to misinformation. This increased susceptibility can lead to hallucinations based on misinformation, posing a critical challenge for the safe use of LLMs. To further investigate this issue, we conducted additional experiments to identify other possible factors that may influence LLMs’ susceptibility pattern to misinformation.
Misinformation Length Figure 6 shows how the MSR score gap between UPD and APD changes as the length of the misinformation increases. As described in Section 3.1, we sequentially added the second and third paragraphs from the Farm dataset to examine how LLMs respond to longer misinformation. The results indicate that in most cases, as the misinformation length increases, the MSR score gap between UPD and APD gradually decreases. In particular, in Llama-3-8B-Instruct, this gap steadily narrowed, and in some models, APD’s MSR score even surpassed UPD’s, indicating that as misinformation length increases, the model becomes more susceptible to the assistant-role relative to the user-role. This pattern aligns with the findings in RQ2, where base models showed a preference for assistant-role. These results suggest that as misinformation become longer, gradually revert to the susceptibility pattern observed in base models, rather than prioritizing the user-role. Nevertheless, further investigation is needed to clarify why instruction-tuning’s impact diminishes with longer misinformation, leading models to behave more like their base versions. More details are provided in Appendix 8.3.
Warning on Misinformation According to [11], simply adding a misinformation warning to the system prompt can help prevent LLMs from easily falling for misinformation. Based on this, we inserted a misinformation warning into the system prompt and analyzed its impact across different scenarios. The modified system prompt is as follows:

The experimental results on the NQ dataset are presented in Figure 7. Except for Qwen-2.5-7B-Instruct, all models maintained a consistent ranking across the three scenarios before and after the warning was added. This indicates that the misinformation warning did not affect the relative susceptibility ranking across scenarios. Meanwhile, after the warning was inserted, the average MSR score across the three scenarios decreased by 69.1%p for GPT-4o and 20.1%p for GPT-4o mini. This indicates that the warning effectively reduces the influence of misinformation, as suggested by [11]. In contrast, the open-source models showed only minimal changes in MSR scores across all three scenarios, suggesting that the warning had little impact on their misinformation susceptibility. This discrepancy may be attributed to differences in how strictly models adhere to system prompts, with proprietary models generally following instructions more rigorously than open-source models. These findings further underscore the importance of our study, reinforcing the need for a deeper exploration of how instruction-tuned LLMs process and respond to misinformation. More details on the experimental results for BoolQ and TruthfulQA can be found in Appendix 8.4.

5 Conclusion↩︎

We analyze how instruction-tuning influences LLMs’ susceptibility to misinformation, particularly in knowledge conflict situations. Our findings reveal that instruction-tuning makes LLMs more user-oriented, increasing their susceptibility to misinformation provided through the user-role. Moreover, presenting misinformation as a separate user-role turn amplifies the models’ focus on it, making it more prominent in the response generation process. By comparing instruction-tuned LLMs with their base versions, we identify instruction-tuning as the underlying cause of the increased susceptibility of LLMs to misinformation when presented in the user-role. Additional factors, such as misinformation length and the presence of misinformation warnings, also affect susceptibility. As misinformation length increases, the model becomes less focused on the user-role, showing patterns similar to those of base models. While simple warnings were not universally effective, some models showed reduced susceptibility when such warnings were provided.

These findings highlight the risks of instruction-tuning, particularly when LLMs are exposed to misinformation. As instruction-tuned LLMs are increasingly integrated into real-world applications, developing effective mitigation strategies is essential to ensure their reliability. Future research should focus on techniques that balance instruction-following abilities with stronger misinformation resistance, contributing to the development of more trustworthy LLMs.

Limitations↩︎

We provide new insights into building and utilizing more reliable LLMs by conducting an in-depth analysis of how instruction-tuning influences LLMs in their susceptibility to misinformation. However, our study has three limitations.
Lack of Base Versions for Proprietary Models We compared four open-source models with their base versions to determine the direct impact of instruction-tuning. However, this comparison was limited to open-source models, as the base versions of two proprietary models were not publicly available, preventing the same analysis. Since proprietary models are widely used in real-world applications, analyzing them is crucial.
Absence of Validation for Large Open-Source LLMs Our experiments on open-source models were limited to 7B–8B models due to resource constraints, and validation for larger models was not conducted. Given that recent open-source models have been released in various sizes, further analysis is needed to understand how model size affects the susceptibility to misinformation. Future research should include models across a broader range to systematically examine the relationship between model size and misinformation susceptibility.
Limitations in Analyzing Instruction-Tuning Differences Although most models exhibited consistent patterns in our experiments, some models showed results that deviated from others or acted as outliers. Understanding the cause of these discrepancies would provide a more precise understanding of how instruction-tuning affects misinformation susceptibility in LLMs. However, the models used in this study do not disclose details about their instruction-tuning methods or training data, making it difficult to determine the fundamental reasons for these variations.

Acknowledgement↩︎

This work was supported by Institute for Information & communications Technology Planning & Evaluation(IITP) grant funded by the Korea government(MSIT) (No. RS-2022-II220369, (Part 4) Development of AI Technology to support Expert Decision-making that can Explain the Reasons/Grounds for Judgment Results based on Expert Knowledge) and (RS-2024-00398115, Research on the reliability and coherence of outcomes produced by Generative AI).

6 Dataset Description↩︎

Each sample in the Farm dataset contains all three types of misinformation: logical, credibility, and emotional. Each type consists of three paragraphs, with each paragraph independently providing sufficient evidence to support an incorrect answer. In our main experiment, we used the logical type, as [11] empirically demonstrated that it had the highest susceptibility in experiments. An example of a data sample used in our study is provided in Table 5.

To examine the generalizability of our findings across diverse types of misinformation, we also conducted the same experiments using the credibility and emotional types. The results and analysis of these additional experiments are presented in Appendix 8.2.

Within this dataset, NQ data is available in two versions: NQ1 and NQ2. NQ1 introduces misinformation by negating the original correct answer, whereas NQ2 does so by reinforcing a specific incorrect answer among the given options. Since the objective of this study is to analyze the susceptibility of misinformation by evaluating whether a model selects a specific incorrect answer when presented with supporting misinformation, we excluded NQ1. The composition and sample distribution in the Farm dataset are summarized in Table 1.

7 Experimental Details↩︎

7.1 Prompt Details↩︎

Detailed prompts for open-source LLMs across three scenarios are provided in the following tables: Table 6 (Llama3-8B-Instruct), Table 7 (Llama3.1-8B-Instruct), Table 8 (Qwen-2.5-7B-Instruct), and Table 9 (Mistral-7B-Instruct-v0.3).

Unlike instruction-tuned LLMs, base models do not have designated chat templates or role-specific tokens for distinguishing conversational roles. Therefore, we manually defined role-specific delimiters to construct a template and added a trigger to facilitate answer extraction. The detailed prompts used for the three scenarios in base models are presented in Table 10. Additionally, the prompts designed to assess the models’ parametric knowledge in a closed-book setting are labeled as “Parametric Knowledge Assessment” in both the instruction-tuned and base model tables.

7.2 Rationale for Excluding the System Prompt as a Conversational Role↩︎

The primary objective of this paper is to analyze the extent to which the user-role is strongly emphasized and reflected in the responses of instruction-tuned LLMs. To achieve this, our experiments distinguish between the roles of the user and the assistant, allowing us to isolate and evaluate the influence of each role.

In contrast, the system prompt serves to guide the model’s overall behavior by providing general context and instructions [32]. It remains consistently applied across all conversations and functions as a global control mechanism for configuring the interaction environment. If the system prompt were treated as one of the conversational roles, it could introduce ambiguity in analyzing the impact of individual roles—particularly in assessing the extent to which the user-role is emphasized. Therefore, in our experimental design, we do not consider the system prompt a separate role but rather as a global context-setting element. This approach ensures a clear distinction between the influence of the user and assistant roles, allowing for a more precise analysis.

8 Experiments Results↩︎

8.1 Result of Parametric Knowledge Assessment↩︎

To evaluate MSR, we first measured LLMs’ parametric knowledge in a closed-book setting [26], since their parametric knowledge varies across models. The parametric knowledge recall is defined as follows:

\[\text{Recall(\%)} = \frac{|\mathcal{Q}✔|}{|\mathcal{Q}|} \times 100\] Let \(\mathcal{Q}\) be the set of all questions, and let \(\mathcal{Q}_✔\) be the set of questions correctly answered by the model in a closed-book setting. The results are reported in Table 2.

8.2 Generalization to Other Types of Misinformation↩︎

As described in Appendix 6, our main experiments focused on the logical type of misinformation due to its high empirical susceptibility. To evaluate the generalizability of model behavior, we additionally conducted experiments using the credibility and emotional types of misinformation. The results of these generalization experiments are shown in Table 3 for instruction-tuned models and in Table 4 for base models.

Experimental results showed that both instruction-tuned and base models exhibited patterns across the three scenarios that were largely consistent with those observed in the logical type setting. In instruction-tuned models, all models except Qwen-2.5-7B-Instruct showed higher MSR scores for UPD compared to APD, and higher MSR scores for UPD compared to STQ—just as in the logical type experiments. However, for both the credibility and emotional types, there were some differences in the ranking between STQ and APD compared to the logical type. Similarly, the base models followed the same ranking order among the three scenarios as in the logical type setting across all experiments, with the exception of a single case.

8.3 Impact of Misinformation Length↩︎

We investigated how misinformation length affects model susceptibility across three scenarios by adjusting the length of misinformation and measuring changes in MSR. The detailed results are presented in Table 11, while Figure 8 illustrates the experimental outcomes for GPT-4o and Mistral-7B-Instruct-v0.3.

The experimental results indicate that an increase in misinformation length does not consistently lead to higher MSR scores. This suggests that a greater amount of misinformation does not necessarily heighten the model’s susceptibility to it. Nevertheless, we observed that in most experiments, as misinformation length increased, the susceptibility patterns of instruction-tuned LLMs became more similar to the trend observed in RQ2 for the base model. However, this trend did not appear in GPT-4o and Mistral-7B-Instruct-v0.3. Unlike other instruction-tuned LLMs, these two models did not exhibit a clear shift toward a base model-like pattern as misinformation length increased. The reasons behind this deviation remain unclear and require further investigation. Potential factors could include differences in instruction-tuning methodologies, long-context processing capabilities, and other architectural distinctions. Future research should explore these aspects in greater depth to determine why GPT-4o and Mistral-7B-Instruct-v0.3 exhibit different patterns despite increasing misinformation length.

8.4 Impact of Misinformation Warnings↩︎

Figure 9 and Figure 10 show how MSR changes across different scenarios in BoolQ and TruthfulQA, respectively, when a misinformation warning is added to the system prompt. The numerical values in the figures represent the percentage change in MSR. We define \(\text{MSR}_{\text{N/W}}\) as the MSR value without a warning and \(\text{MSR}_{\text{W}}\) as the MSR value after adding a warning. The percentage change in MSR is calculated as follows:

\[\Delta MSR\;Rate(\%) = \frac{MSR_{\text{W}} - MSR_{\text{N/W}}}{MSR_{\text{N/W}}} \times 100\]

This metric normalizes the MSR difference before and after adding the warning by the MSR value without a warning (\(\text{MSR}_{\text{N/W}}\)), providing a measure of the percentage change in MSR. This ensures for a fair comparison of MSR variations across different experimental conditions.

References↩︎