1 Introduction↩︎

Misinformation, broadly defined as false or misleading information regardless of intent [1]–[4], has long posed a major threat to the information ecosystem. The emergence of large language models (LLMs) has substantially reshaped this threat landscape. LLMs lower the cost of producing fluent, coherent, and seemingly authoritative misinformation [5]. They can generate fabricated news articles, misleading summaries, persuasive arguments, false explanations, and localized narratives within seconds. More importantly, LLMs amplify misinformation risks beyond the generation of synthetic false content. Recent studies have also shown that LLMs can be misused to rewrite existing claims to evade detectors [6], personalize narratives for different audiences [7], translate and localize false information across languages [8], simulate personas [9], generate comments [10] and social interactions [11], and assist coordinated manipulation campaigns [12]. As a result, LLMs do not merely increase the volume of misinformation; they also broaden the threat surface by enabling misinformation to be created, amplified, contextualized, and concealed in previous impossible ways.

At the same time, LLMs are increasingly used as defensive components in misinformation detection and mitigation pipelines. Recent work explores LLM-based detectors [13], retrieval-augmented generation (RAG) systems [14], multimodal fact-checkers [15], and agentic verification pipelines [16], [17] for misinformation detection and mitigation. Compared with conventional detection systems, these LLM-enabled approaches support open-ended reasoning, evidence-grounded verification, natural-language explanation, and tool-assisted fact-checking. This line of work highlights the defense role of LLMs in the misinformation ecosystem, i.e., LLMs can assist defenders by expanding the capabilities of detection and verification systems.

Although LLMs create new opportunities for scalable defense, they also introduce new vulnerabilities, making these pipelines themselves targets of adversarial manipulation. For example, a RAG-based fact-checking system may produce misleading conclusions if its retrieval corpus is poisoned or adversarial documents are ranked as supporting evidence [18], [19]. An LLM-based verifier may be misled by prompt injection [20]. An agentic detection system that browses the web, invokes tools, or coordinates multiple agents may further propagate errors across multiple reasoning and action steps [21], [22]. Thus, LLMs occupy a third role, i.e., they become vulnerable components in misinformation detection and mitigation systems.

These developments suggest that LLMs play multiple roles in the misinformation ecosystem: they can empower attackers, assist defenders, and become vulnerable components within detection and mitigation systems. However, previous related works often focus on one particular aspect of the problem, such as LLM-generated fake content, LLM-based detection, internal LLM defense, propagation modeling, or fact-checking robustness. Table 1 summarizes these works according to the roles played by LLMs in the misinformation ecosystem, including LLMs as attackers, defenders, and victims. Some studies examine misinformation in the age of LLMs and discuss the opportunities and challenges brought by LLMs for misinformation detection and mitigation [23], [24], [31]. Others focus on LLM-powered fake news detection and proactive defense against misinformation in LLMs [26], [27]. Recent robustness-oriented works further investigate adversarial attacks against automated fact-checking systems [29] or vulnerabilities of multimodal large language models [32]. Therefore, a unified view is still missing on how LLMs simultaneously empower attackers, assist defenders, and become vulnerable components across the misinformation ecosystem.

To address this gap, we introduce a role-layer framework to organize LLM-related misinformation research, as shown in Table 2. The role dimension distinguishes whether LLMs are used as attackers, defenders, or vulnerable components of detection systems. In the attacker role, LLMs are misused to generate, amplify, or strategically optimize misinformation. In the defender role, LLMs are utilized to support detection, mitigation, and explanation. In the victim role, the vulnerable object is the LLM-centered detection pipeline, including LLM-as-judge detectors, and agentic detection workflows. The layer dimension characterizes where manipulation, vulnerability, or defense emerges. We distinguish four layers: content, social contexts, evidence environments, and verification workflows. The content layer focuses on what is stated, such as claim and news articles. The social layer concerns the propagation environment in which content is shared and amplified such as comments, user interactive behaviors. The evidence layer captures the external information sources to support verification, such as knowledge bases. The verification workflows refer to the system-level processes, which are not only defensive mechanisms but also potential targets of adversarial manipulation. This role-layer framework integrates LLM-enabled attacks, system vulnerabilities, and defensive methods within a unified structure.

Following this framework, we organize around the tri-role perspective. Fig. 2 summarizes the temporal distribution of references across the three roles from 2022 to June 2026. The results show that research on LLM misuse has grown rapidly after 2023, reflecting increasing attention to LLM-enabled attacks. LLM-based defense studies also increase substantially, especially after 2024, whereas research treating LLMs as vulnerable components remains comparatively limited but has shown a clear recent rise. Section 2 summarizes how LLMs empower attackers across content, social, and evidence layers. Section 3 then discusses how detection and mitigation systems built around LLMs can be compromised by adversarial claims, poisoned evidence, prompt injection, and agentic tool-use manipulation. Section 4 turns to the defensive use of LLMs, focusing on their roles in content understanding and generation, judge-based and agentic misinformation detection, and retrieval-augmented verification. Section 5 further analyzes robustness-oriented countermeasures against LLM-enabled misinformation risks.

This paper makes the following main contributions:

1. We introduce a role-layer framework to organize LLM-related misinformation research along two complementary dimensions: the role of LLMs in the misinformation ecosystem including attacker, defender, and victim perspectives, and the layer at which manipulation, vulnerability, or defense arises including content, social contexts, evidence, and verification workflows.

2. We organize LLM-enabled threats beyond synthetic content generation, showing how LLM misuse expands misinformation risks across content, social, and evidence surfaces.

3. We systematically analyze LLM-enabled defenses and robustness-oriented countermeasures, thereby revealing how defensive uses of LLMs correspond to different attack surfaces.

4. We further discuss the underexplored vulnerability of LLM-centric misinformation detection systems, highlighting how LLM-as-judge detectors and agentic fact-checking workflows can be attacked in adversarial information environments.

5. We identify three key open challenges: transitioning from static detection accuracy to budget-aware ecosystem-level risk evaluation, hardening LLM-based detection pipelines against adversarial manipulation, and deploying auditable human-in-the-loop verification systems for trustworthy real-world misinformation defense.

2 LLMs as Attackers: Empowered Misinformation Attacks↩︎

LLMs introduce new risks to the misinformation ecosystem not simply because they can generate false text, but because their core capabilities can be misused as flexible attack engines. Natural-language generation supports low-cost fabrication and rewriting; contextual adaptation enables localized and audience-specific persuasion; persona simulation and interaction support social manipulation. These capabilities amplify the scale, automation, adaptivity, and accessibility of traditional misinformation campaigns, making content, social contexts, and evidence environments easier to manipulate.

In this section, we summarize LLM-empowered misinformation attacks from the attacker perspective. We organize the discussion around three manipulation surfaces: content-level, social-level, and evidence-level threats. As shown in Fig. 3, content-level attacks manipulate the misinformation artifact itself, including its wording, style, framing, and detectability. Social-level attacks manipulate the surrounding social context, such as comments, personas, engagements, bot behaviors, and propagation traces. Evidence-level attacks manipulate the external evidence environment on which fact-checking, retrieval-augmented generation, and verification systems rely.

2.1 Content-level Threats↩︎

Content-level threats refer to the misuse of LLMs to generate or manipulate misleading information artifacts, such as fake news articles, social media posts, misleading answers, rewritten claims, and adversarially crafted texts. This line of research focuses on how false or misleading content is generated, linguistically adapted, and rhetorically framed. Existing studies can be grouped into the following four categories.

2.1.1 Synthetic misinformation generation↩︎

Synthetic misinformation generation is the most direct form of LLM misuse studied in the literature. Many works study how adversaries prompt general-purpose LLMs to generate fluent, coherent, and news-like misinformation with minimal human effort [33]–[40]. These studies show that LLM-generated misinformation can resemble human-written content and can be adapted across topics, events, languages, and local contexts. Recent work further investigates scalable generation pipelines [41] and retrieval-augmented generation for producing more convincing or knowledge-grounded fake news [42].

2.1.2 Audience-aware persuasion↩︎

Another line of work studies audience-conditioned misinformation generation. Instead of generating a single generic false claim, these studies focus on how misleading narratives can be personalized according to user attributes, community contexts, ideological cues, or emotional states [43], [44]. Related work on emotional prompting further investigates how affective framing, such as fear, anger, urgency, or moral concern, can increase the salience and persuasive force of generated misinformation [45]. This category reflects a shift from generic fake-content generation to targeted and context-aware influence.

2.1.3 Content camouflage and defense evasion↩︎

A third group of studies investigates how misleading content can be rewritten to evade detection or moderation. These attacks preserve the deceptive meaning of a claim while changing surface style, sentiment, credibility cues, or linguistic form, making the content less recognizable to detectors that rely on textual artifacts or distributional patterns [46]–[51]. Recent studies also investigate jailbreak-based fake news generation, showing that harmful misinformation can still be elicited despite the presence of model safeguards [8]. In addition, adversarial example generation uses explanation-guided rewriting, semantic contamination, iterative mutation, or fact-checking feedback to craft texts against fake news, rumor, or fact-checking systems [44], [52]–[56]. These studies show that content-level attacks increasingly separate deceptive semantics from detectable surface cues.

2.1.4 Application-oriented content attacks↩︎

Some studies examine how LLM-generated or rewritten misinformation affects downstream information systems. In news recommendation, manipulated content may alter ranking behavior, exposure patterns, and the visibility of reliable information [44], [57]. In knowledge-intensive applications such as open-domain question answering, generated misinformation may appear as fluent answers or knowledge-like content, thereby polluting the information environment used by end users [37], [38].

2.2 Social-level Threats↩︎

Social-level threats refer to LLM misuse that targets the social context in which misinformation propagates. Unlike content-level attacks that manipulate the claim or article itself, social-level attacks manipulate the social propagation process through which information spreads such as comments, user engagements, user profiles and bot behaviors. Existing studies can be grouped into the following four categories.

2.2.1 Comment manipulation↩︎

Comment manipulation studies focus on adversarial comments, replies, or conversational traces around a news item. Some studies explore narrative-level amplification, where LLM agents or persona-conditioned generation iteratively reshape misleading claims to make them more persuasive, coherent, or ideologically aligned during propagation [58]–[61]. Some works focus on message/comment generation attacks, where adversarial textual signals are injected into the social context to mislead detectors [10], [62]–[66]. Retrieval-based attacks further show that selectively inserted real user comments can also serve as adversarial payloads, even without generating new comments [67].

2.2.2 Structure manipulation↩︎

Structure manipulation studies examine attacks on the interaction patterns through which misinformation appears to spread. User-engagement manipulation perturbs sharing, reposting, or bot-generated interaction signals to make fake news appear more credible, popular, or organically endorsed [68], [69]. Graph-structure attacks further manipulate the social interaction graphs used by graph-based fake news detectors, exposing the vulnerability of GNN-based models to propagation-level manipulation [70].

2.2.3 Strategic campaign manipulation↩︎

Some studies treat propagation manipulation as a long-horizon campaign rather than a one-shot perturbation. RL-based social bot control formulates adversarial bot behavior as a sequential decision-making problem, where bots learn when to interact and evade detection to maximize influence over time [71], [72]. LLM-powered campaign simulation further studies how malicious agents or botnets coordinate posting, commenting, and correction dynamics across multiple rounds of social interaction [73]–[75]. [76] show the abuse of LLMs can also facilitate high-level social engineering by producing personalized behaviors. [77] examine how account attributes can be modified to make automated actors appear more human-like and persistent in online communities. These studies suggest that social-level attacks are moving toward a strategic campaign-level threat rather than an isolated perturbation.

2.2.4 Propagation data poisoning↩︎

Propagation manipulation can also occur during data collection or model training. Backdoor attacks implant trigger patterns into propagation structures so that rumor detectors fail under specific conditions [78]. Robustness studies examine whether GNN-based detectors can be made provably robust against poisoning and backdoor attacks [79]. Platform-level data poisoning further shows that fabricated user-generated content can contaminate the input distribution of online platforms and downstream detection systems [80]. Collectively, these studies highlight that propagation-based models are vulnerable not only at inference time, but also during the construction of the social data on which they are trained.

2.3 Evidence-level Threats↩︎

Evidence-level threats refer to attacks on the external evidence environment. These attacks manipulate what evidence is created, stored, retrieved, ranked, or used before a verification system reaches a judgment. Existing studies can be grouped into the following three categories.

2.3.1 Evidence content manipulation↩︎

Evidence content manipulation studies how misleading or adversarial evidence can alter verification. Some studies focus on synthetic evidence generation, adversarial document insertion, or evidence rewriting, where biased claims are made to appear supported [81]–[83]. More recent work extends this to viewpoint steering, where fabricated rationales guide models toward attacker-preferred perspectives [84]. Another line of research manipulates the input claim rather than the evidence source itself. They change search queries to manipulate the evidence matching and verification trajectories via adversarially paraphrasing [20], [85]–[88]. These studies show that evidence-level failures can arise either from evidence content or from redirecting the evidence acquisition and reasoning processes via claim reformulation.

2.3.2 Evidence corpus poisoning↩︎

Evidence corpus poisoning studies examine attacks that inject malicious documents into retrieval corpora or knowledge bases used by RAG and fact-checking systems. Targeted poisoning methods craft passages for specific queries, claims, entities, or verdicts, so that the system retrieves poisoned evidence and produces attacker-desired answers or fact-checking decisions [18], [89]–[94]. Some studies further broaden this setting to black-box, transferable, competing, or stealthy poisoning, showing that poisoned evidence may remain effective across different queries, topics, or detection constraints [95]–[98]. More recent works extend poisoning objectives beyond isolated factual answers, including viewpoint manipulation, trigger-conditioned evidence stores, and reasoning-path corruption in retrieved documents [99]–[105]. These studies show that corpus poisoning can affect not only what knowledge is retrieved, but also how the model reasons over retrieved evidence.

2.3.3 Evidence retrieval manipulation↩︎

Evidence retrieval manipulation studies target the retrieval stage itself. One group of attacks increases the visibility of malicious evidence by crafting or perturbing documents so that adversarial content is more likely to enter the retrieved context [106], [107]. Related methods use typos, mutations, or imperceptible textual changes to alter retrieval behavior while preserving the apparent meaning of the document [108], [109]. Another group of attacks suppresses useful evidence by introducing jamming documents that interfere with retrieval-augmented generation and make correct supporting evidence harder to access or use [110]. These studies show that evidence integrity depends not only on whether malicious evidence exists, but also on whether reliable evidence remains retrievable, salient, and properly ranked.

2.4 Summary↩︎

Overall, existing research has shown that the misuse of LLMs enables misinformation risks from isolated content fabrication into ecosystem-level manipulation.

• Content-level threats have evolved from simple fake-text generation to adaptive misinformation production. Early attacks primarily use LLMs to produce fluent fake posts, whereas recent studies increasingly exploit LLMs for personalization, emotional framing, multilingual rewriting, detector evasion, and downstream application attacks. The key trend is that content-level attackers become more scalable, audience-aware, and defense-aware.

• Social-level threats have evolved from isolated comment manipulation to coordinated propagation manipulation. Early attacks primarily inject adversarial comments or reactions to mislead detectors, whereas recent work increasingly manipulates engagement structures, bot behaviors, user identities, and long-horizon campaign dynamics. The key trend is that attackers no longer only manipulate what users read, but also how socially credible, controversial, or organically endorsed the information appears.

• Evidence-level threats have evolved from attacking model outputs to compromising the evidential infrastructure that verification systems rely on. Early attacks primarily fabricate or rewrite supporting evidence, whereas recent work increasingly studies corpus poisoning, retrieval manipulation, viewpoint steering, universal poisoning, stealthy poisoned documents, and reasoning-path corruption. The key trend is that attackers can compromise not only what evidence exists, but also what evidence is retrieved, and used in reasoning.

The three attack layers differ in both threat assumptions and evaluation requirements. Content-level attacks assume control over the misinformation artifact itself and are typically evaluated by generation quality, semantic preservation, persuasive effect, or detector evasion. Social-level attacks manipulate interaction contexts such as comments, user profiles, engagement traces, or bot behaviors, thereby changing how credible, controversial, or organic a claim appears. Evidence-level attacks target verification systems that rely on external evidence, retrieval corpora, search engines, or knowledge bases, manipulating what evidence is created, retrieved, ranked, and used in reasoning. These differences also lead to different data needs: content attacks can often be evaluated on text or multimodal misinformation datasets, whereas social- and evidence-level attacks require propagation traces, campaign simulations, claim-evidence corpora, or RAG-style benchmarks.

3 LLMs as Victims: Vulnerabilities of LLM-based Detection Paradigms↩︎

In this section, we study LLMs as attack targets within misinformation detection systems that either directly employ LLMs as judges or organize them into agentic workflows for misinformation detection. This section differs from Sec. 4 in perspective, which aims to review how LLM-based systems are designed to detect and mitigate misinformation. In contrast, this section examines how the LLM-centric systems may fail when they are attacked adversarially. We organize this section by the primary attack surface of LLM-based misinformation detection systems. As shown in Fig. 5, for direct LLM-based detectors, the attack surface follows an input/judgment/output structure. Adversaries may manipulate the input, exploit the model’s internal judgment weaknesses, or distort downstream decisions through unreliable outputs. For agentic detectors, the attack surface expands from a single model call to a broader workflow involving planning, retrieval, tools, communication, memory, trust, and aggregation.

3.1 LLM-as-judge Vulnerabilities↩︎

Direct LLM-based detectors use an LLM as a judge or verifier. Given a claim, the LLM is prompted to produce a veracity label, factuality judgment, or hallucination assessment. This paradigm is attractive because it requires limited task-specific architectural design and can be adapted to different misinformation-related tasks through prompting. However, it also inherits the vulnerabilities of LLM-as-a-judge systems, particularly when exposed to adversarially crafted inputs, ambiguous factual contexts, or excessive downstream reliance on the LLM’s judgments.

3.1.1 Input-level Attacks on LLM Judges↩︎

The first vulnerability arises from the input interface of LLM-based detectors. Because task instructions and untrusted content are processed in the same natural-language context, adversaries can manipulate the detector before factual judgment begins. Several studies directly expose this input-level vulnerability. Some attacks directly inject malicious instructions or optimized prompt sequences into attacker-controlled inputs, thereby biasing LLM-as-a-judge systems before they perform factual assessment [111]–[113]. Others rely on short transferable adversarial phrases that can distort zero-shot LLM assessments across different judge settings [114]. The manipulation can also be more semantic and less visible via meaning-preserving perturbations [115] or adversarial persuasive attacks [20].

3.1.2 Intrinsic Judgment and Reasoning Failures↩︎

The second vulnerability stems from the LLM’s own judgment and reasoning behavior. Even in the absence of explicit prompt injection, a direct LLM-based detector may fail because it relies on linguistic cues, rhetorical fluency, or plausible yet invalid reasoning chains. In misinformation detection, this is especially problematic because false claims are often written in polished language and may be supported by fabricated explanations that appear logically coherent. Existing studies suggest that this vulnerability arises from several sources. First, LLM-based detectors may fail when synthetic misinformation becomes fluent, human-like, and difficult to distinguish from authentic content [40], [116]. This indicates that direct detectors cannot safely rely on surface-level artifacts or stylistic cues as LLM-generated misinformation becomes more natural and adaptive. Second, LLM-based detectors may be unstable when factual judgment requires resolving conflicts with the model’s parametric knowledge [117]. Third, LLMs may be biased toward reasoning that appears coherent but is logically or factually invalid [118], which can cause rationale-based detectors to overvalue fluent explanations instead of verifying their factual grounding. Broader security analyses further suggest that bias, misinformation, and prompt-based manipulation are intertwined risks in LLM-based applications [119].

3.1.3 Output-level Reliance and Evaluation Failures↩︎

The third vulnerability arises after the LLM detector produces its output. LLM-based detectors often generate not only a binary label, but also explanations, confidence estimates, factuality rationales, or safety judgments. These outputs can influence human users, content moderators, evaluation pipelines, and platform-level decision systems. Consequently, the harm of a detector is not limited to whether its label is correct; it also depends on how its output is interpreted and used downstream. Output-level failures can affect both users and evaluation pipelines. [54] show that fact-checking information generated by large language models can decrease headline discernment under certain conditions. [120] argue that attack success may sometimes reflect weaknesses of the judge rather than genuine failures of the target system. [121] show that adversarial attacks against LLM-based cyber threat intelligence systems can cause false alarms and real downstream damage.

3.2 Vulnerabilities of Agentic Misinformation Detection Systems↩︎

Agentic misinformation detectors differ from direct LLM-based detectors because they do not simply ask an LLM for a one-shot judgment. Instead, they organize LLMs into multi-step workflows that may decompose claims, retrieve evidence, invoke tools, coordinate multiple agents, store context, and aggregate intermediate results. This paradigm can emulate human fact-checking more closely and can improve transparency, adaptability, and evidence grounding. However, the same workflow structure also expands the attack surface. An adversary can target not only the final LLM judge, but also the planning process, evidence environment, tool chain, inter-agent communication, memory, trust estimation, or decision policy.

3.2.1 Workflow-based Agentic Fact-checking Detectors↩︎

This category includes agentic detectors whose primary contribution is the construction of a structured verification workflow. Their central idea is to move beyond one-shot LLM classification by approximating the behavior of human fact-checkers through claim decomposition, stepwise reasoning, tool selection, and final verdict integration [16], [122]–[124]. The vulnerability of this category lies in the workflow itself. Many works on general agentic systems show that autonomous LLM agents may suffer from unsafe planning, tool misuse, context manipulation, runtime supply-chain attacks, and uncontrolled interactions [125]–[127]. These risks are especially relevant to misinformation detection because agentic fact-checking systems typically rely on web search, external databases, browser tools, and other runtime resources that can themselves be manipulated. If the system decomposes a claim incorrectly, selects the wrong tool, overweights a misleading subclaim, or follows an unsafe reasoning path, the final verdict may fail even if the base LLM is capable.

3.2.2 Retrieval- and Evidence-centric Agentic Verification↩︎

The category includes agentic systems whose main contribution centers on external evidence. Several agentic detectors rely on retrieval and web grounding [95], [128]–[131]. External evidence can reduce hallucination and improve grounding, but it can also be poisoned, fabricated, selectively retrieved, or adversarially ranked. Fact2Fiction [93] targets agentic fact-checking systems by constructing poisoned evidence tailored to the system’s claim decomposition strategy. This demonstrates that evidence-grounded detectors can be compromised through the evidence ecosystem even when the original claim and the base LLM remain unchanged. [132] extend general security framing to multimodal agentic RAG. Its central idea is that adversarial intent may be distributed across retrieval, planning, and generation stages rather than appearing as a single malicious input. This motivates stateful trust modeling, where the system tracks potentially adversarial behavior across the workflow.

3.2.3 Multi-agent Communication and Coordination↩︎

The category includes systems whose main contribution or vulnerability comes from multiple interacting agents. Multi-agent misinformation detectors often assign different roles to specialized agents, such as claim analysis, evidence retrieval, debate, and final judgment [17], [133]–[135]. This design can improve coverage and interpretability, but it also introduces communication channels through which errors, biased intermediate conclusions, or malicious prompts can propagate. The distinctive vulnerability of this category is not merely that each agent may be wrong, but that the system depends on communication and coordination among agents. [136] reveal LLM-to-LLM prompt injection within multi-agent systems, showing that malicious prompts can spread across interconnected agents.

3.3 Summary↩︎

This section shows that vulnerabilities of LLM-based systems evolve with the detection paradigm.

• For direct LLM-based detectors, robustness centers on the input, the model’s internal judgment, and the downstream interpretation of its output. An adversary can inject malicious instructions, craft persuasive false claims, exploit reasoning bias, or cause users and evaluation systems to overtrust unreliable explanations.

• For agentic detectors, robustness becomes a workflow-level problem. The adversary can target the detector’s planning process, evidence environment, runtime tools, communication channels, stateful trust mechanisms, or coordination policies.

These observations imply that improving the underlying LLM alone is insufficient. Direct LLM-based detectors require prompt isolation, adversarial input testing, and uncertainty calibration. Agentic detectors require provenance-aware retrieval, secure tool use, authenticated inter-agent communication, memory auditing, trust modeling, and robust coordination policies. As misinformation detection systems become increasingly agentic, robustness evaluation must move from isolated model accuracy toward end-to-end security analysis of the entire verification workflow.

4 LLMs as Defenders: LLM-enabled Detection and Verification Methods↩︎

Large language models (LLMs) have been increasingly explored as defenders against misinformation. As shown in Fig. 8, we organize existing studies into three lines of work. The first uses LLMs as understanding and generation modules to enrich textual, social, and multimodal signals. The second uses LLMs as judgment modules through prompting, tuning, reinforcement learning, or explanation-aware prediction. The third integrates LLMs with retrieval-augmented generation (RAG), external evidence, and agentic workflows for more grounded verification.

4.1 Understanding and Generation↩︎

The first line of research uses LLMs as auxiliary modules for data augmentation, representation enrichment, context simulation, and multimodal understanding. Existing studies can be grouped into text augmentation, social context augmentation, and multimodal augmentation.

4.1.1 Text Augmentation↩︎

Text augmentation studies use LLMs to construct alternative textual views, semantic enrichments, or adversarial variants for misinformation detection. This is especially useful when detectors overfit to superficial writing styles or fail under LLM-enabled attacks. Some methods reduce detectors’ reliance on superficial writing patterns by exposing them to stylistic variants of the same misleading semantics [6], [137], [138]. Other works focus on enriching the latent representation via semantic relationship perception between content and supporting evidence [139], or structured semantic signals [140], [141]. These studies show that LLM-based augmentation can help detectors capture deeper semantic structures.

4.1.2 Social Context Augmentation↩︎

Social context augmentation studies use LLMs to interpret, refine, or simulate the social signals surrounding misinformation. This line of research is motivated by the fact that comments, stance labels, propagation trees, and interaction patterns are often missing, noisy, or manipulated in real-world settings. Some methods use LLMs to interpret social reactions, infer user stances, and model propagation dynamics [142]–[145]. Other works simulate propagation data to address the lack of sufficient social feedback at the early stage or low-resource detection such as emerging topics [146]–[148].

4.1.3 Multimodal Augmentation↩︎

Some studies extend LLM-based defense from text-only misinformation to settings involving visual evidence such as images and videos. The primary idea is to use LLMs or multimodal large language models (MLLMs) to enrich cross-modal representations, reason over text-image consistency, or handle missing modalities. Some methods use LLMs or MLLMs to jointly interpret textual and visual signals before making a detection decision [149]–[151]. MLLMs also offer a new perspective to address the practical problem that misinformation samples may lack complete visual, textual, or metadata information [152].

4.2 Judgment↩︎

The second line of research directly uses LLMs as judgment modules. These studies use LLMs to generate veracity labels, reliability scores, and rationales. Compared with augmentation-based methods, judgment-based methods place LLMs closer to the final decision process. Existing studies mainly explore four technical ways: direct prompting, supervised tuning, reinforcement tuning, and explanation-aware judgment.

4.2.1 Prompt-based Judgment↩︎

Some prompting-based methods reformulate misinformation detection as a natural-language judgment task via well-designed instructions [40], [142], [153], [154]. Some works further design chain-of-thought (CoT) prompts to decompose the LLM judgment process into intermediate reasoning steps [155], [156]. [157] explore the dual role of LLMs in fake news detection and show that LLMs may function as useful advisors even when they can also be misused as generators. They offer a low-cost entry point for LLM-based detection, but its reliability remains constrained by instruction sensitivity and unstable generalization.

4.2.2 Supervised Fine-tuning↩︎

Supervised fine-tuning methods adapt LLMs or LLM-enhanced detectors to misinformation-specific data, labels, domains, or rationales. Task-specific tuning is used for emergency rumors, cross-domain rumors, emergent fake news, and unseen misinformation, where models must adapt to evolving events and platforms [150], [158]–[160]. Another line of research incorporates rationales or explanations into the training process, encouraging models to produce analytical outputs rather than only labels [161]–[163]. These methods improve task alignment and stability compared with prompting, but they may introduce dataset-specific biases.

4.2.3 Reinforcement Learning-based Tuning↩︎

Reinforcement learning-based tuning methods optimize LLM-based detectors using reward signals, adversarial training, or reinforcement-guided sampling. Some studies formulate misinformation detection as an adaptive adversarial process, where detectors are trained against semantically constrained or dynamically generated fake information [164]. Others use reinforcement tuning to jointly detect stances and debunk rumors, connecting intermediate social reasoning with final veracity judgment [165]. Reinforcement-guided sampling further combines LLM generation with feedback-based instance selection, enabling detectors to benefit from informative synthetic samples [166]. These methods are useful when misinformation detection involves evolving adversaries, multiple optimization objectives, or limited labeled data.

4.2.4 Explanation-aware Judgment↩︎

Explanation-aware judgment methods treat explanations as part of the decision process. Some studies use graph-prompted or MLLM-based explanations to make decisions over text-image relations more interpretable [167], [168]. [169] move beyond label prediction toward debunking explanations, where the system identifies the misleading mechanism and communicates corrective information to users.

4.3 RAG System↩︎

The third line of research integrates LLMs with external knowledge, evidence retrieval, and agentic verification. Existing studies can be organized along the verification pipeline: knowledge grounding, evidence retrieval, evidence reasoning, and agentic fact-checking.

4.3.1 Knowledge Grounding↩︎

Knowledge grounding methods enhance LLM-based verification by incorporating external or dynamically updated knowledge. Some studies use knowledge-guided prompting or retrieval-augmented LLMs to improve rumor detection and resolve unseen rumors when parametric knowledge is insufficient [170], [171]. Other work studies dynamic knowledge updating for fake news detection, aiming to align verification with evolving factual contexts around rapidly changing events [172].

4.3.2 Evidence Retrieval↩︎

Evidence retrieval methods acquire relevant textual, visual, or cross-modal evidence before verification. Text-based RAG studies retrieve documents, passages, or facts to support fake news detection and fact-checking decisions [173], [174]. Multimodal retrieval methods further combine external evidence with visual forensics or cross-modal retrieval for multimodal fake news and video misinformation detection [175], [176].

4.3.3 Evidence Reasoning↩︎

Evidence reasoning methods investigate how verification systems determine whether retrieved evidence supports, refutes, or is irrelevant to a claim. Some works enhance semantic understanding of claim-evidence relations or incorporate event-centric and commonsense-guided reasoning to move beyond surface matching [139], [177]. Graph-prompted and out-of-context detection methods reason over entities, captions, events, and visual scenes to identify cross-modal inconsistencies [167], [168]. Another line of research focuses on trustworthy evidence reasoning, including LLM susceptibility to deceptive evidence and causal debiasing for multi-hop fact verification [178], [179].

4.3.4 Agentic Fact-checking↩︎

Agentic fact-checking methods extend RAG-based verification from a single-step pipeline to an interactive, multi-step workflow. Multi-agent systems distribute verification across roles such as claim analyzer, evidence retriever, verifier, and explainer [16], [133], [180]–[183]. Debate-driven methods use disagreement and argumentation to expose weak evidence, unsupported assumptions, and one-sided reasoning, and have been studied for misinformation detection and intervention [184]–[187]. Agentic debunking further moves from veracity prediction to user-facing correction, requiring systems to identify misleading mechanisms, collect evidence, and generate corrective explanations [169].

4.4 Summary↩︎

LLM-based misinformation defense is moving beyond conventional content classification toward more contextual, evidence-grounded, and workflow-oriented verification.

• LLMs are mainly used to enrich the inputs and representations available to downstream detectors. Text augmentation improves robustness to stylistic variation and adversarial rewriting; social context augmentation helps recover or refine missing propagation signals; and multimodal augmentation enables reasoning over text, images, videos, and cross-modal inconsistencies.

• Judgment-oriented studies place LLMs closer to the final decision process. Instruction prompting reformulates misinformation detection as reliability assessment, while supervised and reinforcement tuning adapt detectors to specific domains, evolving events, adversarial samples, or stance-aware debunking. Explanation-aware judgment further extends outputs from labels to rationales, multimodal explanations, and corrective responses.

• For RAG-based systems, knowledge grounding and evidence retrieval reduce reliance on parametric knowledge by introducing external and dynamically updated information. Evidence reasoning then determines whether retrieved evidence truly supports or refutes a claim. Agentic methods formulates fact-checking and detection system into well-designed multi-step workflows such as claim decomposition, evidence search, and debunking.

Overall, LLM-based defense broadens misinformation detection into a more comprehensive verification pipeline that combines representation enrichment, natural-language judgment, evidence grounding, and agentic reasoning.

5 Countermeasures: Adversarial Robustness against LLM-enabled Misinformation Threats↩︎

LLM-enabled misinformation attacks expand from content generation to social-context manipulation and evidence poisoning. In this section, we investigate existing countermeasures at different layers of the misinformation pipeline. This view highlights both the progress and the uneven coverage of current defenses.

5.1 Robustness against Content-level Attacks↩︎

Content-level defenses address attacks that manipulate the misinformation artifact itself, including synthetic news generation, persuasive targeting, content camouflage and evasion, and scenario-specific ecosystem attacks (see Sec. 2.1 for details). Existing studies mainly explore LLM-generated text detection, adversarial training, invariant representation learning, and multi-agent mitigation.

5.1.1 LLM-generated text detection↩︎

LLM-generated text detection provides a direct defense against synthetic misinformation by identifying whether a news article, claim, or post has been generated or rewritten by LLMs. Existing methods model prompt-induced linguistic fingerprints, generation traces, or distributional differences between human-written and LLM-generated misinformation [50], [190]. Recent work studies detection under LLM-induced distribution shifts, including mixtures of human-written, LLM-generated, and LLM-paraphrased content [154], as well as source-model attribution [191]. These defenses provide a natural first step for detecting synthetic misinformation, although their effectiveness depends on whether generation artifacts remain detectable.

5.1.2 Adversarial training↩︎

Adversarial training methods improve detector robustness by exposing models to LLM-generated or LLM-rewritten attack variants during training. For synthetic news generation threat, semantic and symbolic adversarial learning train detectors on challenging generated variants and semantically constrained fake information [140], [164]. For content camouflage and evasion attack, some works expose detectors to style-transferred or style-camouflaged misinformation, encouraging them to rely less on tone, fluency, or news-like presentation [6], [47].

5.1.3 Content-style disentanglement and invariant representation learning↩︎

Content-style disentanglement and invariant representation learning address scenarios in which misinformation is rewritten into different surface forms while preserving its misleading semantics. [188] separate factual or deceptive semantics from writing style, sentiment, credibility cues, and presentation. [189] further study invariant representation learning that aligns original and adversarially transformed versions, enabling detectors to focus on stable claim-level signals.

5.1.4 Multi-agent mitigation↩︎

Multi-agent mitigation methods use multiple LLM agents or roles to cross-check suspicious content, decompose claims, challenge weak reasoning, and aggregate different perspectives. [133] design multi-agent systems to combine claim understanding, evidence seeking, stance analysis, and final judgment for misinformation mitigation. Multi-agent designs can improve coverage and reasoning diversity, but they also introduce coordination costs, disagreement-resolution problems, and the risk of error propagation across agents.

5.1.4.1 Discussion: Gap with respect to content-level attacks

As shown in Table 3, existing content-level defenses are concentrated on synthetic misinformation generation and content camouflage or evasion. A key remaining challenge is to bridge textual robustness and downstream application robustness, especially when manipulated content is personalized, retrieved, recommended, or reused by knowledge-intensive systems.

5.2 Robustness against Social-level Attacks↩︎

Social-level defenses address attacks that manipulate the social context surrounding misinformation as shown in Sec. 2.2. Existing studies can be mainly divided into prompting-based reasoning, invariant representation learning, manipulation identification, and structure uncertainty modeling.

5.2.1 Prompting-based reasoning↩︎

Prompting-based defenses use LLM reasoning to identify manipulative, unethical, or coordinated narratives in social discussions. Chain-of-thought and ethics-oriented prompting can help detect problematic narratives in LLM-generated comments, replies, or discussion threads [192]. These methods mainly correspond to narrative-level manipulation, where misinformation is strengthened through surrounding discussion rather than by changing the original article. They provide a flexible reasoning-based defense, but their reliability depends on prompt design and may vary across personas, tones, and conversational strategies.

5.2.2 Invariant representation learning for manipulated social context↩︎

Invariant representation learning aims to reduce detector sensitivity to manipulated comments, messages, or propagation contexts. Existing studies learn stable representations under LLM-generated narratives or adversarially changed social signals [193].

5.2.3 Manipulation identification↩︎

Manipulation identification methods explicitly detect whether the social context itself has been adversarially injected or strategically manipulated. Rather than directly predicting the veracity of a news item, [137] examine whether comments, replies, or discussion patterns show abnormal or manipulated characteristics. This direction is particularly relevant to message injection and malicious comment generation, where the attacker targets the detector’s contextual input rather than the misinformation content.

5.2.4 Structure uncertainty modeling↩︎

Structure uncertainty modeling addresses attacks on propagation graphs, user-news interactions, and engagement patterns. [194] model user-news interactive edges as noisy and potentially adversarial rather than fully reliable. They can alleviate local graph or engagement manipulation but usually fail under a long-horizon campaign behavior.

5.2.4.1 Discussion: Gap with respect to social-level attacks

As shown in Table 4, current social-level defenses mainly cover narrative-level amplification, malicious comment generation, message injection, and structure manipulation. Direct countermeasures remain limited for multi-round fake narrative generation, real-comment retrieval attacks, strategic campaign manipulation, and propagation data poisoning. These attacks differ from localized comment or graph perturbations because they may involve temporal adaptation, coordinated actors, real user comments, bot control, identity manipulation, or contaminated platform data.

5.3 Robustness against Evidence-level Attacks↩︎

Evidence-level defenses protect fact-checking systems, RAG pipelines, and search-augmented LLM agents from attacks on evidence content, evidence corpora, and evidence retrieval. Existing studies mainly examine evidence reliability modeling, structured or causal evidence reasoning, evidence-grounded verification, multi-agent verification, and poisoning attribution.

5.3.1 Evidence reliability modeling↩︎

Evidence reliability modeling evaluates whether retrieved evidence is trustworthy rather than merely relevant. Existing studies model evidence reliability for misinformation detection and investigate credibility signals that help assess external sources and retrieved content [153]. Other work studies how deceptive evidence influences LLM-based fact-checking and proposes mitigation strategies against misleading evidence use [178].

5.3.2 Structured evidence reasoning and causal or multi-hop verification↩︎

Structured evidence reasoning examines whether evidence supports, contradicts, or contextualizes a claim. It shifts evidence defense from retrieval alone to verification of the relation between claim and evidence. Existing studies use LLM-powered reasoning to integrate multimodal evidence for fake-news detection [149] and causal or multi-hop verification to reduce reliance on spurious evidence associations [179]. These methods require the system to assess claim-evidence relations rather than accept surface-level semantic overlap.

5.3.3 Evidence-grounded verification and multi-agent LLMs↩︎

Evidence-grounded verification explicitly links final judgments to retrieved, checked, or refined evidence. [196] use an isolate-then-aggregate strategy to reduce the influence of poisoned evidence on the final answer. [173] improve robustness against generated evasion samples via retrieval-augmented fake-news detection with adversarial refinement. [133] explore multi-agent LLM systems to decompose verification into claim analysis, evidence retrieval, evidence critique, and final judgment for robust detection. These methods provide a bridge between evidence content defenses and workflow-based verification, although they inherit dependencies on retrieval quality, source reliability, and tool use.

5.3.4 Poisoning attribution and secure RAG analysis↩︎

Poisoning attribution addresses evidence corpus poisoning by tracing corrupted RAG outputs back to suspicious documents or knowledge sources. Prompting-based attribution methods identify retrieved or stored documents that may be responsible for poisoned generations [197], [198]. Activation-based analysis further detects poisoned RAG responses when malicious evidence is not apparent from surface text [199]. These methods mainly provide diagnostic and attribution tools for poisoned retrieval settings.

5.3.4.1 Discussion: Gap with respect to evidence-level attacks

As shown in Table 5, current evidence-level defenses primarily counter fabricated evidence, adversarial claim attacks, and targeted corpus poisoning. These defenses remain insufficient against evidence-layer manipulation, including efficient single-document or chain-of-evidence poisoning, query-agnostic universal poisoning, competing poisoning, structured knowledge poisoning, and evidence retrieval manipulation. These settings move beyond the attack on a known target claim toward cross-topic, query-agnostic, and competing evidence manipulation. Existing reliability modeling and attribution methods may provide partial diagnostic signals, but direct countermeasures are not yet systematically established.

5.4 Summary↩︎

Overall, existing countermeasures reveal a clear mismatch between the maturity of defenses and the frontier of LLM-enabled misinformation attacks. Current defenses mainly cover localized and pre-defined attacks, whereas emerging threats are becoming increasingly adaptive, knowledge-aware, long-horizon, and stealthy. This gap highlights the need for more systematic defenses against emerging attack paradigms, particularly coordinated misinformation campaigns and covert attacks targeting evidence-grounded systems.

6 Datasets and Evaluation Metrics↩︎

This section summarizes representative misinformation datasets and metrics.

6.1 Datasets↩︎

Existing datasets differ mainly in what information they provide for evaluation: content, social context, and evidence. Table ¿tbl:tab:dataset95taxonomy? follows this distinction and lists representative datasets rather than an exhaustive catalog. Traditional datasets mainly support content- and social-context-aware detection, while evidence-grounded datasets support claim verification and fact-checking. LLM-era datasets focus on machine-generated misinformation, and factuality evaluation. Most current resources still evaluate only one or two signals at a time.

6.1.1 Human-generated Datasets↩︎

Human-generated datasets are constructed from political claims, news articles, social media posts, user comments, propagation traces, or manually curated evidence. Content-centric datasets evaluate whether a model can identify false or misleading information from textual or multimodal content. Representative datasets include Weibo21, BuzzFeedNews [213], BuzzFace [214], Fakeddit [202], and NewsCLIPpings [201]. These datasets range from article-level fake-news classification to multimodal and out-of-context image–caption verification, reflecting the shift from text-only misinformation to multimodal manipulation. Social-context datasets incorporate user reactions, comments, reposts, temporal cascades, or propagation structures. Representative datasets include Twitter15 and Twitter16 [203], Weibo [204], PHEME [206], [207], PHEMEPlus [208], FakeNewsNet [209], FakeHealth [210], and CoAID [211]. Evidence-grounded datasets require systems to verify claims using external evidence rather than relying only on content features. Representative datasets include LIAR [212], FEVER [215], WICE [216], FEVEROUS [217], SciFact, and Mocheg [200]. These datasets support evidence retrieval, rationale selection, multimodal evidence reasoning, and temporally aware verification.

6.1.2 LLM-era Datasets↩︎

This category contains machine-generated misinformation. Representative datasets include LLMFake [40], Grover [219], MFC-Bench [227], and DGM4 [228]. They evaluate whether detectors can handle LLM-generated false articles, controllable fake news, multimodal fact-checking samples, and generative image–text manipulations. Compared with human-generated datasets, this category is still relatively limited, revealing a gap in realistic LLM-era misinformation benchmarks. QA and RAG benchmarks such as NQ [230], HotpotQA [231], MS-MARCO [232], SQuAD [233], BoolQ [234], MuSiQue [238], GraphRAG-Bench [237], BBQ [235], and StereoSet [236] can also serve as diagnostic benchmarks for retrieval quality, evidence reasoning, and bias under uncertainty. Some datasets evaluate factuality in LLM outputs including HaluEval [222], FELM [223], FacTool [221], FActScore [224], FactCheck-GPT [225], and BingCheck [226].

6.2 Evaluation Metrics↩︎

Metrics in LLM-era misinformation evaluation measure not only clean detection performance, but also whether the system retrieves trustworthy evidence, resists adversarial manipulation, and preserves utility after defense. We summarize representative metrics in Table 6.

6.2.1 Task Performance Metrics↩︎

Task performance metrics measure whether a system can predict correct labels under benign conditions, such as accuracy, precision, recall, macro-F1, false positive rate, false negative rate and AUC etc.

6.2.2 Evidence and Factuality Metrics↩︎

Evidence-aware metrics evaluate whether a system retrieves and uses the information required to justify its verdict. Evidence recall and precision measure whether relevant sources are successfully retrieved, whereas FEVER-style scores require both a correct label and sufficient supporting evidence. Evidence quality also include sufficiency, rationale faithfulness, factual consistency, and atomic factual precision, because a fluent explanation may appear persuasive even when it is not grounded in the retrieved evidence.

6.2.3 Attack Effectiveness Metrics↩︎

Attack metrics quantify the extent to which an adversary achieves the intended manipulation. Output-level metrics such as attack success rate and multi-attack success rate capture whether the final prediction or rationale has been altered. Retrieval-level metrics such as adversarial retrieval rate, poisoned evidence recall, and ranking boost should be reported separately, because a poisoned document may be retrieved without affecting the final verdict, or may change the verdict only through deceptive justification.

6.2.4 Defense Robustness Metrics↩︎

Defense evaluation should report both utility and robustness. Accuracy under attack and robustness gap summarize how much performance degrades under adversarial inputs or poisoned evidence. Defense success rate and utility–robustness trade-off further indicate whether a mitigation blocks attacks without sacrificing too much clean performance. This paired reporting prevents defenses from appearing effective merely because they reject many inputs, over-filter evidence, or reduce the usefulness of the underlying fact-checking system.

7 Challenges and Future Directions↩︎

We discuss the key challenges around evaluation, system robustness, and deployment.

7.1 Evaluation Gap: From Detection Accuracy to Budgeted Risk Evaluation↩︎

The evaluation gap concerns how to measure the LLM-enabled misinformation risk under realistic adversarial conditions. This gap calls for a shift from isolated detection accuracy to budgeted ecosystem-level risk assessment.

• Multi-dimensional risk measurement. Most existing studies use attack success rate as the primary metric. However, the risks posed by LLM-enabled misinformation are not limited to detector evasion. An adversarial example may preserve the misleading claim, increase its visibility, influence user beliefs, or distort retrieved evidence. Broader risk indicators are needed to cover more diverse evaluation dimensions.

• Practical threat models with limited attacker access and budget. Many attacks are evaluated under relatively strong assumptions such as white-box access, sufficient query budgets, or repeated feedback from the target system. In practice, misinformation actors often operate with partial knowledge, limited feedback, and constrained resources. Thus, black-box, low-budget, and feedback-limited settings are important for assessing whether robustness holds under practical attack constraints.

• Complex multilingual and multimodal attack evaluation. False or misleading narratives may combine generated claims with reused images, synthetic visuals, manipulated captions, or short videos. More comprehensive evaluation benchmarks are needed to assess whether detection and verification systems remain reliable when misleading cues are distributed across textual claims, image-text relationships, and video evidence.

• Long-horizon propagation risk evaluation. Current works have shown that LLMs can manipulate comments, personas, and stance signals during propagation. As recent agentic systems become capable of simulating social interaction and propagation dynamics, which may be misused to generate more adaptive campaigns, long-horizon evaluation becomes a key setting for studying self-evolving misinformation campaigns.

• Competing and multi-objective attack evaluation. Many existing attacks optimize a single objective such as evading a detector or increasing the ranking of poisoned evidence. Realistic attacks, however, often involve multiple objectives, which can conflict: aggressive rewriting may improve evasion but reduce credibility, whereas coordinated amplification may increase exposure but also make the campaign easier to flag. This makes multi-objective evaluation an urgent need for characterizing competing or collaborative attack behaviors.

7.2 System Gap: Hardening LLM-based Verification and Information Systems↩︎

The system gap concerns how to make LLM-based misinformation defenses robust when the system itself becomes part of the attack surface.

• Robustness of LLM-based detectors and agentic workflow. Research on LLM-based detectors and agentic workflow creates attack surfaces that are absent or less explicit in conventional classifiers, such as prompt injection, tool-use manipulation, and memory poisoning. Future work needs therefore analyze to the security of the full verification workflow and build more robust LLM-centered and agentic systems.

• Content-level defense against personalized misinformation attacks. Unlike earlier forms of misinformation generation, LLM-enabled attacks can adapt claims to specific audiences, rewrite them to evade generic classifiers, and use rhetorical strategies that increase perceived credibility. This makes it harder to detect with a generic classifier. Future defenses need robustness against adaptive paraphrasing and persuasion-preserving transformations.

• Defense against evolving LLM-enabled social manipulation attacks. LLMs enable social-level adversaries to coordinate engagement and manipulate propagation patterns in a strategic manner. Unlike traditional heuristic perturbations, these attacks can dynamically adapt to evade propagation-based detectors. A key future direction is to develop defenses against stealthier and more adaptive LLM-driven propagation manipulation.

• Evidence integrity and provenance modeling for robust RAG. Existing RAG-based defenses often emphasize retrieval relevance, but relevance alone does not guarantee source authenticity. Robust evidence integrity and provenance modeling are therefore important for distinguishing reliable grounding from adversarially constructed support.

• Unintentional misinformation from personalized LLM systems. Not all LLM-related misinformation comes from deliberate attacks. Personalized assistants may generate misleading information when they rely on outdated memory or over-personalized assumptions. Such errors may arise from both hallucinations and omissions of relevant facts. The factual reliability of personalized LLM systems is therefore an important part of misinformation defense, especially for time-sensitive or high-impact information needs.

7.3 Deployment Gap: Operationalizing Robust, Governed, and Human-centered Defense↩︎

The deployment gap concerns how LLM-based misinformation defenses can be used in real environments with limited resources, privacy constraints, platform-specific norms, and human oversight.

• Risk-adaptive defense under practical resource constraints. Many LLM-based defenses rely on computationally expensive verification mechanisms, which are often impractical for online deployment. Future systems should allocate verification resources in a tiered manner, so that high-risk cases receive stronger evidence verification, provenance validation, and human oversight.

• Governed use of social signals and human oversight. Social-level defenses often rely on account behavior, engagement traces, and coordination patterns. These signals help detect manipulation and misinformation but also raise concerns regarding privacy and fairness. Future deployment should distinguish manipulative coordination from legitimate collective behavior and integrate human review for ambiguous or high-impact cases.

• Auditable decision-making and post-deployment monitoring. LLM-based defenses may label, downrank, or ignore content based on model judgments and platform rules. Without auditability, it is difficult to diagnose false positives, false negatives, delayed interventions, or decisions affected by poisoned evidence. Future systems should record key decision traces and monitor long-term drift, attacker adaptation, and downstream effect.

8 Conclusion↩︎

This paper presents a tri-role perspective on the evolving relationship between large language models and misinformation. Rather than viewing LLMs solely as generators of synthetic false content, we argue that they should be understood as attackers, victims, and defenders within a broader misinformation ecosystem. We systematically analyze how LLMs empower misinformation attacks across content, social, and evidence layers; how LLM-based detection paradigms themselves become vulnerable targets; and how LLMs can support misinformation detection and mitigation, and robustness-oriented countermeasures. Through this analysis, we reveal a growing mismatch between increasingly sophisticated LLM-enabled attacks and existing defenses. Future research should move toward ecosystem-level robustness by addressing three key gaps: more realistic risk evaluation under budgeted, multilingual, multimodal, competing, and long-horizon attack settings; stronger system robustness for evidence provenance, retrieval integrity, social manipulation, and secure agentic workflows; and more deployable defenses with risk-adaptive resource allocation, auditability, and post-deployment monitoring. We hope this work provides a unified foundation for understanding, evaluating, and mitigating misinformation risks in the era of LLMs.

References↩︎