1 Introduction↩︎

In recent years, the evolution of large language models (LLMs) has demonstrated their immense potential across a wide range of natural language processing tasks. However, despite their impressive successes in many domains, their ability to recognize and properly handle false premises in reasoning tasks remains a significant challenge [1], [2]. When engaging with false premises, an LLM should be able to identify and reject flawed assumptions rather than proceed as if they were valid.

False premises can appear in various forms, such as misleading statements, logically inconsistent claims, or factually incorrect contextual narratives [3]–[10]. A common approach to evaluating an LLM’s ability to handle false premises is through question-answering (QA) tasks, where a question implicitly contains a misleading or incorrect assumption [1], [2], [11]. For example, a well-performing LLM should recognize that “Which year did Albert Einstein win the Nobel Prize in Chemistry?” contains a false premise—Einstein never won a Nobel Prize in Chemistry—and responds appropriately, rather than attempting to provide an incorrect or misleading answer.

Existing benchmarks focus on single-hop false premise detection, where the incorrect assumption can be identified within a single reasoning step [1], [2], [11]. However, multi-hop reasoning presents a greater challenge, as it requires models to connect multiple pieces of information across multiple inference steps before arriving at an answer [12]. Unlike single-hop questions, multi-hop questions (MHQs) require models to derive intermediate conclusions, making them a critical area of research in the evaluation and improvement of LLM reasoning abilities [12]–[16]. The complexity of MHQs arises from the necessity of bridging multiple facts, understanding implicit dependencies between reasoning steps, and maintaining logical consistency throughout the inference process [12].

While MHQs are already challenging, the problem becomes even more difficult when false premises are embedded within the reasoning chain, requiring models not only to answer questions correctly but also to detect and reject incorrect assumptions at intermediate steps. To evaluate LLMs’ ability to handle this challenge, we introduce a novel benchmark of Multi-Hop False-Premise Questions (MHFPQs), called MultiHoax, that combines the difficulty of false premise questions and multi-hop question answering. MHFPQs are designed to test whether LLMs can detect false premises that appear at one or more reasoning steps rather than simply providing an answer based on a flawed assumption.

Figure 1 shows an example of an MHFPQ from the sports category related to Iran. Answering this question requires a structured reasoning process. First, the model must determine the edition of the Olympics where Zahra Nemati served as Iran’s flag bearer for the first time, namely, the 2016 Rio Olympics. Second, it must identify the Iranian wrestler who won the gold medal in the Men’s freestyle 125 kg category at those Olympics. This introduces a false premise: no Iranian wrestler won gold in the 125 kg category at the 2016 Olympics. The only Iranian gold medalist in Men’s freestyle wrestling at those Games was Hassan Yazdani, who competed in the 74 kg weight class. To ensure realism and difficulty, we carefully designed the set of possible answer choices in the MHFPQ dataset. Each question includes believable distractors, as possible, that align with historical facts, making the task particularly challenging for LLMs. For instance, in the example from Figure 1, the distractors are real Iranian wrestlers. Moreover, Ghassem Rezaei and Komeil Ghassemi both won gold in freestyle wrestling at the 2012 London Olympics. Hassan Yazdani also won the gold at the 2016 Olympics, but was not competing in the 125 kg division. This ensures that incorrect answers remain plausible while still being contextually invalid. Without detecting the false premise, LLMs may be misled into selecting one of the plausible but incorrect answers, emphasizing the necessity for models to engage in deep contextual reasoning rather than relying on surface-level fact retrieval.

The dataset spans 7 countries and 10 distinct categories, ensuring broad diversity across historical, cultural, and geopolitical contexts. To generate the MHFPQs, we extracted relevant information from Wikipedia pages with the assistance of the Claude model ³. Each question is paired with three closely related distractors and a single “I do not know” option, which serves as the only valid response in the presence of a false premise. By structuring the benchmark in this way, we aim to assess LLMs’ ability to identify and reject incorrect assumptions embedded within multi-hop reasoning chains, rather than merely selecting the most superficially plausible response.

We conduct a comprehensive evaluation of six open-source and closed-source models using our MultiHoax dataset of 700 carefully reviewed questions. Our results reveal suboptimal performance across all models, categories, and countries, indicating that current systems struggle with detecting falsehoods in multi-hop reasoning tasks. This resource enables rigorous analysis of multi-hop reasoning under false premises and opens new research directions in question answering that require rejection of complex, embedded assumptions.

2 Related Work↩︎

2.0.0.1 False Information and False Premises.

False information is a broad and multifaceted issue, encompassing explicitly misleading claims from social networks [17], [18], blogs [19], news sources [20]–[22], and online forums [2]. However, false premises are a distinct challenge since they are not always deliberate misinformation but rather incorrect assumptions that lead to flawed logical conclusions. These premises span various domains, including historical events [23], [24], religion [25], and sports [26], making them particularly complex to detect.

2.0.0.2 False-Premise Questions.

FPQs assess a model’s ability to reason over implicit false assumptions about properties, actions, scope, existence, events, logic, or causality [1], [2]. For example, “How many eyes does the sun have?” wrongly assumes the sun has eyes [1]. Since FPQs require models to identify and reject flawed premises rather than simply answering the question, LLMs often fail to recognize implicit falsehoods, leading to misleading or nonsensical responses [2]. This inability to reject false premises poses risks for misinformation propagation and model alignment with human expectations.

Early studies on FPQs have focused on obvious falsehoods that humans can easily detect. For instance, [1] introduced a dataset of simple FPQs, such as “What is the most common color of human’s wings?”. While earlier LLMs struggled with these, recent versions of the LLMs handle them easily, raising the need to evaluate models on more subtle false premises. Similarly, [11] proposed a dataset evaluating models on questions containing false or unverifiable assumptions. However, their dataset includes unverifiable claims—statements that may become true over time. For example, “When is Steven Universe Season 5 coming to Hulu?” assumes the event has not yet occurred in order to be false, but this assumption could later become valid. This distinction makes their dataset less suitable for assessing persistent false premises. Another line of research has focused on false-presupposition questions extracted from online forums. [2] introduced a dataset of FPQs sourced from Reddit, such as “How exactly is current stored in power plants?”—a misleading assumption since electric current is not stored. However, their dataset is primarily limited to scientific and technical topics, lacking the broad factual diversity required for a general evaluation of false premises.

Beyond FPQs, unanswerable questions have also been explored in related research. [27] focus on document-grounded unanswerable questions, where a question lacks supporting information within a given document. Their contribution is in evaluating models’ ability to reformulate unanswerable questions into answerable ones, rather than directly assessing how LLMs handle false premises in an open-domain setting. Additionally, [28] study truthful question answering, focusing on questions that elicit misconceptions from humans. Relatedly, [29] and [30] examine how LLMs handle questions beyond their knowledge, where the correct response should be “I do not know”. These studies, however, focus on uncertain or conflicting information rather than inherently false premises.

2.0.0.3 Multi-Hop False-Premise Reasoning.

Most FPQ benchmarks focus on single-hop false premises, yet real-world misinformation often involves multi-step reasoning, making it significantly harder to detect. Multi-hop reasoning questions challenge LLMs by requiring them to connect multiple facts across inference steps. While extensive research has explored general MHQ understanding in LLMs [13], [31]–[33], MHQs embedding false premises remain largely unstudied. Existing benchmarks assess multi-hop factual reasoning but do not evaluate whether LLMs can detect and reject false premises within reasoning chains. This gap is critical, as misinformation is rarely an isolated error—false premises are often interwoven across reasoning steps, making them subtle, plausible, and difficult to refute. Evaluating how LLMs handle multi-hop false premises is essential for enhancing their robustness against misinformation and ensuring logical consistency in complex reasoning tasks.

[34] attempt to address this with adversarial multi-hop false premise questions, modifying HotpotQA [31]. Their approach replaces the title of a supporting document with a similar but unrelated distractor, selected based on shared Wikipedia categories. For example, Roger O. Egeberg and Steven K. Galson both fall under American public health doctors, making them interchangeable under this method. However, this technique relies on structured entity swaps rather than embedding implicit falsehoods, making it unclear whether a question contains a truly false premise or is merely unverifiable. Additionally, this approach can lead to unnatural question phrasing, limiting its applicability in assessing real-world false premise detection.

Overall, LLMs struggle with imperfect information, conflicting evidence, and questions beyond their knowledge scope [28]–[30], [35]–[40]. They are also susceptible to distraction by irrelevant details, including false premises, often leading to incorrect, misleading, or fabricated responses [1], [2], [11], [38], [41]. These findings underscore a critical gap in evaluating multi-step false premise reasoning, as prior benchmarks focus on single-hop FPQs and adversarial perturbations, failing to capture the complexity of false premises embedded within structured factual reasoning. To address this, we introduce MultiHoax, a new benchmark of multi-hop FPQs, designed to assess LLMs’ ability to detect and reject false premises within reasoning chains. By including implicit false factual claims across diverse country-related topics, our benchmark enables a more nuanced evaluation of LLMs’ ability to detect and reject false premises embedded in complex, multi-hop reasoning chains.

2.0.0.4 Multi-Regional and Multi-Cultural Resources.

Our dataset aligns with multi-cultural and multi-regional NLP research, which examines how LLMs process knowledge across different geographic and cultural contexts. While much prior work has focused on subjective tasks like norms and values [42]–[47], recent studies show that multicultural NLP also includes objective knowledge, such as region-specific facts, historical events, and socio-political contexts [48]–[52]. This highlights the need to evaluate how LLMs handle factual reasoning across diverse regions, particularly in multi-hop tasks involving false premises.

Despite growing interest in multi-regional fact verification, most work focuses on binary misinformation detection rather than multi-hop false premise reasoning [53]–[55]. A global benchmark for multi-hop false premise detection remains missing, even though factual inaccuracies vary significantly across regions. MultiHoax addresses this gap by introducing diverse, context-rich questions across seven countries and ten knowledge domains, enabling evaluation of LLMs’ reasoning over embedded falsehoods in culturally grounded settings.

3 MultiHoax Dataset↩︎

This section describes the proposed dataset, designed to evaluate multi-hop false premise reasoning across diverse countries, categories, and question types. The dataset structure enables a systematic analysis of how LLMs handle false premises within complex reasoning tasks.

3.1 Dataset Framework↩︎

Each MultiHoax’s instance consists of the following components:

• Question and Answer Choices: Each question contains at least a false premise and is paired with four answer choices, three plausible distractors, and one “I do not know”. The latter is positioned randomly to prevent models from exploiting positional bias.

• False Premise Explanation: A brief description clarifies why the assumption in the question is incorrect.

• Country and Category: The dataset spans seven countries (China, France, Germany, Iran, Italy, the United Kingdom, and the United States) and is categorized into ten knowledge domains, including food, sports, geography, education, history, entertainment, religion, science & technology, arts & literature, and holidays & leisure.

• Types of False Premises and Answer Types: Each question is annotated with the type of false premise it contains, following the taxonomy introduced by . The five main types are Property, Event, Entity, and Scope. For example, the question “Which award did Venkatraman Ramakrishnan receive first: the Shaw Prize in Life Science and Medicine or the Lasker Award?” falls under the event type, since it implies an event that never happened because he never won the mentioned awards.⁴ Furthermore, following [31], we classify answer types such as person, location, event, number, and common nouns. The answer type shows what the question is asking for.⁵

• Multi-Hop Reasoning Type: Following [12], we categorize why a question requires multi-step inference into five types: (1) Named Entity Reasoning: The question requires connecting two facts through an intermediate entity that links them logically, (2) Temporal Reasoning: An intermediate step involves identifying a specific time reference to answer the question correctly, (3) Geographical Reasoning: The reasoning process depends on understanding locations, spatial relationships, or geographic entities, (4) Intersection Reasoning: The answer is determined by an entity that satisfies multiple overlapping conditions, and (5) Comparison Reasoning: The question requires comparing attributes, facts, or values across multiple entities to arrive at the correct conclusion.⁶

• Wikipedia Grounding: Each question is linked to a relevant Wikipedia page for factual grounding.

3.2 Wikipedia Document Collection↩︎

We developed a pipeline to extract Wikipedia pages relevant to each country and category using ChatGPT-4o’s search tool. To select the set of pages, each page was evaluated based on three criteria: relevance to the category, association with the specified country, and existence. “Existence” ensures that the provided link leads to an actual Wikipedia page rather than just an important but undocumented topic. If a page was missing, the model was prompted to complete the list by suggesting alternatives. Ultimately, we collected 15 Wikipedia pages per country and category and extracted their content.

3.3 Question Generation↩︎

After collecting relevant documents, we developed a structured process for generating MHFPQs. First, we instructed Claude 3.5 Haiku to extract 15 facts per document using a fact-extraction prompt. Then, based on these facts, we prompted the model to generate MHFPQs. All prompts are detailed in Appendix ¿sec:sec:appendixa?. Due to fundamental differences between Bridge Entity-based MHQs (named, geographical, and temporal entities) and other types (intersection and comparison), we used a separate prompt for each category. We requested three questions from the former and two from the latter but did not enforce specific subtypes (e.g., one intersection, one comparison), as not all documents contained relevant supporting facts. Additionally, we avoided first generating MHQs and then falsifying them, as this could limit the variety of false information types [34] and might not ensure incorrectness.

4pt

3.4 Curated Selection and Expert Review↩︎

After generating the questions for each category and country, an expert reviewer evaluated the generated questions and selected 10 MHFPQs for each combination of category and country. The selection was guided by a structured, multi-step approach to ensure the quality, validity, and plausibility of the questions.

The first step in the selection process was to verify whether a question adhered to the MH structure. Ensuring diversity in MH question types was a key objective. If a question was not initially formatted as an MHQ but could be converted into one, it passed this initial filter. For instance, the question “What was the name of the Achaemenid ruler who appointed Cyrus as governor of the Median Empire?” is inherently a single-hop reasoning question. However, by introducing an additional reasoning step, such as asking about the ruler’s son, it could be transformed into an MHQ: “Who was the son of the Achaemenid ruler who appointed Cyrus as governor of the Median Empire?”.

Next, the question needed to contain at least one universally false piece of information—meaning the falsehood had to be global rather than merely incorrect within the context of the associated document. To ensure this criterion was met, only questions that demonstrably contained globally false statements were selected. This verification process followed a rigorous three-step approach. First, the reviewer traced each question back to its corresponding fact or set of facts. Second, these facts were cross-checked against the information found in the relevant document. Finally, the reviewer assessed whether the question was indeed incorrect relative to both the established facts and the document. If the model did not generate enough false information, the reviewer modified the question by introducing falsehoods aligned with the dataset’s predefined types of misinformation. However, if adding such falsehoods was not feasible, the question was discarded.

The third step involves explaining why the question is incorrect. To achieve this, the model’s explanation is evaluated based on the previous step’s results. If it fails to address the false information, the reviewer provides a more detailed clarification.

The final step involved verifying the plausibility of the answer choices. The reviewer ensured all options were contextually relevant by referring to the corresponding section of the document. If the model’s initial options were unrelated—either due to a change in the question’s focus or the model’s poor performance in generating relevant choices—the reviewer replaced them with more appropriate alternatives.

An example of this procedure is illustrated in Figure 1. The initial question, generated by Claude 3.5 Haiku, was: At which Olympics did Zahra Nemati carry the flag for the first time? While this was a factual question, it was not an MHQ, but it could be transformed into one, as shown in Figure 1.

Initially, the question contained no falsehoods—it simply inquired about an event that happened in reality. To introduce false information, we modified the question to ask about an event that did not happen, while adding another reasoning hop. Asking about the Iranian freestyle wrestler who won gold in the 125 kg division at the 2016 Olympics contains a falsehood, as no Iranian wrestler did so at the 2016 Olympics. After these steps, the question was transformed into an MHFPQ, such as: Which Iranian wrestler won gold in the Men’s freestyle 125 kg at the first Olympics when Zahra Nemati was the flag bearer?

To generate plausible answer choices, we included real Iranian wrestlers. Additionally, we selected the most relevant wrestlers. For example, Komeil Ghasemi won a gold medal in the men’s freestyle 120 kg event at the 2012 Summer Olympics. Moreover, Ghasem Rezaei was a former Greco-Roman wrestler and an Olympic gold medalist. Additionally, the only Iranian wrestler to win gold in the Men’s freestyle at the 2016 Rio Olympics was Hassan Yazdani in 74 kg. This set of wrestlers provides three relevant and distracting options.

8pt

3.5 Secondary False Information Verification↩︎

To ensure the accuracy of falsified content, a second round of review was conducted. In this phase, a second reviewer independently examined the description field of each question against the corresponding Wikipedia page to verify the presence of false information. The reviewer categorized each question into one of three outcomes: “There is false information”, “There is no false information”, and “I cannot tell based on the provided information”. ⁷ Questions confirmed to contain false information were directly included in the final dataset. Those labeled with the second option were double-checked and falsehood was added where required, while those detected with the last option were both improved in terms of clarity and added with false information. In the latter two cases, after the necessary modifications, the reviewer provided feedback to ensure that the question contained clear false information. The dataset was finalized upon completion of this verification process.⁸ Table 1 shows examples from our final resource from different types and countries.

4 Evaluation Setup↩︎

The MultiHoax dataset serves as the basis for our evaluation of multi-hop false premise reasoning through two primary tasks: (1) closed multiple-choice QA and (2) justification-based verification. In the first task, models were presented with a question and four answer choices, including “I do not know”, allowing them to reject the question if they identified a false premise. The prompt used for this step is shown in the first part of Table 2.

In the second task (justification), models that selected “I do not know” are prompted to explain their decision, as shown in the second part of Table 2. This step differentiates between cases where the model lacked knowledge and those where it explicitly identified a false premise. Only responses that both select “I do not know” in the first task and correctly justify it as a false premise in the second task can be considered successful detections of false premises. This justification step enhances the reliability of our evaluation, ensuring that refusal to answer stems from false premise detection rather than general uncertainty.

We additionally explore an alternative output format using structured JSON, where models are asked to produce both an answer and a textual explanation. While this format allows for more expressive reasoning, it can make it harder to unambiguously determine whether a model has identified a false premise, as rejection may be implied rather than explicit. By contrast, the multiple-choice setup includes an explicit “I do not know” option, enabling clearer detection of false-premise rejection. Despite the possibility that this option may simplify the task, it provides a more standardized and reliable evaluation framework.

4.0.0.1 Models

We tested 6 different proprietary and open-source LLMs in our experiments. The models include Claude Sonnet 3.5⁹, Gemini-2.0-pro-exp [56], GPT-4o [57], Qwen2.5-7B-Instruct [58], Llama-3.1-8B-Instruct [59], and Deepseek-llm-7b-chat [60]. All experiments used a zero temperature setting to ensure deterministic responses, with all data collected in February 2025.

5 Results↩︎

8pt

5pt

5pt

5pt

Table 3 presents model accuracy on the first group of tasks, which are the first two tasks. The first task evaluates whether models correctly reject false premises by selecting “I do not know”. While Claude, which was used during the question-generation phase, demonstrates higher accuracy compared to other models, the models generally struggle to recognize falsehoods in the first task, as they are unable to refuse to answer. Furthermore, although all models show poor performance on this task, open-source models generally exhibit lower accuracy than proprietary ones.

The second task analyzes whether models can justify their “I do not know” responses by correctly identifying the false premise. Here, Gemini slightly outperforms the generator model (Claude), but overall accuracy remains low across all models, indicating persistent difficulty in recognizing false premises.

Table 4 presents model performance across ten knowldege categories. Accuracy is highest in science and technology, while art and literature and entertainment rank lowest, reflecting variation in LLMs’ domain-specific factual competence. Notably, proprietary models consistently outperform open-source models, though performance remains low across all categories. Interestingly, Claude’s dominance is most pronounced in high-information domains like education and food, while other models exhibit irregular performance trends. These results highlight significant disparities in model strengths across different knowledge areas.

8pt

Table 5 analyzes accuracy across countries. The results indicate close accuracy levels, suggesting that models struggle to detect falsehoods regardless of the country. The lower accuracy in some countries can be a result of more challenging questions, which can suggest that the generator model was better able to design challenging questions for those countries.

Table 6 presents model accuracy across multi-hop reasoning types. While comparison-based questions show slightly higher accuracy, the differences across reasoning types are modest, and overall performance remains low. This suggests that all multi-hop reasoning categories pose significant challenges to current models.¹⁰

10pt

Finally, we evaluate models using a structured JSON output format that encourages explicit reasoning. This setting uses the prompt shown in Table 7. Table 8 presents the results of JSON output for the top-performing models from the previous task. The results indicate accuracy levels comparable to those observed in the multiple-choice setting. This suggests that models consistently struggle to detect embedded falsehoods in the MultiHoax dataset, regardless of output format or reasoning constraints.

6 Conclusion↩︎

We introduce a novel class of multi-hop false-premise questions (MHFPQs), combining the complexities of multi-hop reasoning and false-premise detection. To support this research, we present MultiHoax, the first FPQ resource that spans multiple countries and knowledge domains, enabling evaluation of LLMs’ ability to navigate multi-step falsehoods in diverse contexts. Our dataset provides a comprehensive evaluation framework, spanning seven countries and ten knowledge categories, allowing for a detailed analysis of how LLMs handle false premises across diverse topics and regions. Unlike prior FPQ datasets, MHFPQs require deeper reasoning, as falsehoods are not immediately apparent but emerge through multi-step inference. MultiHoax establishes a novel evaluation setting at the intersection of multi-hop reasoning and false-premise detection, revealing persistent limitations in current LLMs. By targeting assumption-level failures across diverse domains and regions, it provides a rigorous benchmark for advancing models that can reason robustly under implicit factual errors.

Limitations↩︎

Our dataset has some limitations. While we have aimed to include a diverse range of countries with varying levels of resource availability, there are still opportunities to incorporate additional countries from other regions worldwide.

Second, our category set, which consists of 10 categories, could be expanded as scholars explore knowledge across various areas. While we have focused on a set of categories suitable for factual, objective questions, other potential categories could be included. Additionally, subjective questions, such as “Who first imported the most popular type of ingredient to China?” could be considered. This would be an MHQ, as it requires identifying both the most popular ingredient and the first person to import it. However, determining the most popular ingredient is subjective, and the question becomes MHFP if the ingredient was never imported to China. While subjective questions are feasible, reviewing them differs significantly from the process for factual objective questions.

Furthermore, while we include questions associated with a diverse set of countries, we do not have the translation of these questions in the local languages of these countries, except from the United States and the United Kingdom. Future research can be collecting questions in the local language of such countries or translating ours to those languages.

Finally, while our current resource presents a significant challenge to different LLMs, and even the best models struggle with our tasks, the rapid progress of LLMs may make our dataset less difficult over time. As models improve, we may need to update our resources to introduce more challenging tasks that better test LLMs’ reasoning abilities.

Ethical Considerations↩︎

We aim to provide a set of factually incorrect questions requiring multiple reasoning steps to challenge various models’ ability to detect falsehoods.

To compile our dataset, we utilized LLMs to generate potential questions, which greatly facilitated the process. However, this approach may introduce biases, as LLMs are more knowledgeable about certain countries than others. For instance, the USA’s lower accuracy could be attributed to LLMs having more in-depth knowledge of U.S. facts, allowing them to craft more challenging questions. Although we used Wikipedia documents to mitigate this bias, it cannot be entirely eliminated. In future iterations, we plan to diversify the question types, incorporating topics focused on values and norms rather than just factual knowledge, and we aim to minimize reliance on LLMs for question generation.

Appendix A: Prompts↩︎

In this section, we provide the different prompts that we leveraged to collect the Wikipedia pages, and also prompts to generate the questions.

Wikipedia Document Retrieval Prompt↩︎

In this section, we first define each category. Then, we used them in order to search for the relevant documents of each category using ChatGPT-4o’s search using the prompt in Table 9.

6.0.0.1 Food

• Cuisine: Signature dishes, cooking styles, traditional meals.

• Ingredients: Locally grown spices, crops, and special ingredients.

• Drinks: Popular beverages, traditional teas, or alcoholic drinks.

6.0.0.2 Sports

• National and Popular Sports: Widely played or watched sports in the country and Official sports of a country.

• Athletes: Famous sportspeople or Olympic medalists.

• Tournaments and Sports Venues: Major leagues, championships, or cups, as well as iconic stadiums, arenas, or tracks.

6.0.0.3 Education

• Education System AND Literacy: Structure (primary, secondary, higher education) AND Efforts to promote literacy or improve access to education.

• Schools and Universities AND Curriculum: Prestigious or historic institutions AND Subjects emphasized or unique courses.

• Famous Educators: Scholars, reformers, or pioneers in education.

6.0.0.4 Holidays/Celebrations/Leisure

• National Holidays: Independence days, constitution days, or memorials.

• Festivals: Cultural, religious, or seasonal festivals.

• Others: Other topics related to Holidays/Celebrations/Leisure.

6.0.0.5 History

• Historical Figures: Leaders, revolutionaries, empires and kingdoms, or intellectuals.

• Important Events: Battles, treaties, or turning points in history.

• Landmarks: Historical monuments or UNESCO heritage sites.

6.0.0.6 Geography

• Natural Features AND Resources: Mountains, rivers, lakes, and deserts AND Natural resources, agriculture, or energy production.

• Cities AND Regions: Capitals, major cities, or urban landmarks AND Administrative divisions or cultural regions.

• Geopolitics: Borders, neighbors, or disputed territories.

6.0.0.7 Science and Technology

• Scientists: Modern renowned scientists.

• Engineering: Famous modern constructions, bridges, or technology.

• Others: Other related topics to Science and Technology like Medical Breakthroughs, Research Centers, Computing Pioneers, Green Technology, Digital Platforms, and Communications.

6.0.0.8 Arts and Literature

• Writers: Prominent authors, poets, or playwrights.

• Books: National epics, famous novels, or historical documents.

• Artists: Prominent artists.

6.0.0.9 Religion

• Religions and Religious Practices: Popular religions and Worship styles or Religious rituals.

• Holy Sites: Temples, churches, mosques, or pilgrimage locations.

• Others: Religious Leaders, Religious Festivals, or Sacred Texts.

6.0.0.10 Entertainment

• Cinema and TV: National cinema, famous directors, popular movies, or actors.

• Music: Traditional music styles, Musicians, or iconic bands.

• Others: Other topics related to entertainment like theater, gaming, festivals related to entertainment, or media.

These category definitions were used in the prompt in the Table 9 to get 15 related pages for each country.

Fact Extraction Prompt↩︎

To extract facts from each document, we leveraged a simple brief prompt. In the fact extraction prompt, we define the number of facts to be extracted, the name of the document, and the document content. The prompt is provided in the Table 10.

Question Generation Prompt↩︎

For question generation, we designed two distinct prompts corresponding to the two major types of MHQs: Entity-based and Intersection & Comparison Combination. Each prompt consists of three parts: an introduction to the task, an explanation of the specific MHQ type, and a set of generation rules. The first section, shown in the Table 13, and the last section, shown in the Table 16 are the same in both generation prompts, but the second part, the definition of the specific MHQ types, is different.

The tables 14 and 15 show the two different second sections.

Table 14: The second part of the generation prompt for the first group.

Table 15: The second part of the generation prompt for the second group.

Appendix B: Annotation Guideline↩︎

Annotation Guideline↩︎

The annotation guideline, shown in the Table 17 was used to familiarize false information reviewers with the dataset structure and their tasks. The guideline first defines what an FPQ is, then explains how questions are structured in our dataset, and finally outlines the reviewing task.

Annotators details↩︎

Following the initial round of annotation by the authors, the dataset was divided into three parts for the second review phase. We then recruited three university student annotators—one female and two male—each paid £12.21 per hour for ten hours of work.

Appendix C: Statistics↩︎

In this section, we provide further statistical information regarding the dataset.

Table 11 provides the distribution of the various types of MHQs.The majority of the resource consists of bridge entity-based questions, while the intersection type is largely included.

Table 18 shows the distribution of the various types of false information included in the questions. We include four types in our resources. The description of types is also provided in Table 18.

Moreover, Table 12 provides the details of answer type occurrence over the resource, the person type as the major type.

Regarding the annotation details, Table 19 contains the number of times the second reviewer chose each possible label for the questions.

The table 20 provides the detailed accuracy of the models across different types of answers. As shown, event, location, and number have the lowest accuracies, suggesting that the models find these types more challenging than the rest while they generally show poor performance.

Tables 6 and 21 also show the detailed accuracies for different types of MHQs and FPQs.

5pt

6pt

References↩︎