1 Introduction↩︎

Digital information ecosystems are increasingly defined by multimodal content. Videos, images, text, and synthetic media enable richer communication and expression, but also introduce new forms and avenues for large-scale deception. Beyond edited and AI-generated images, a key challenge is multimodal misinformation, in which images are paired with false claims to increase their perceived credibility [1] and wide-spread diffusion [2]. In response, researchers have been developing automated Multimodal Fact-Checking (MFC) systems to retrieve relevant information from the Web to cross-examine and verify the veracity of multimodal claims [3].

To date, developing robust MFC models for image-text verification is constrained by the scarcity of high-quality data. As a result, recent work relies on weakly annotated or synthetic datasets [4]–[9], or benchmarks sourced directly from fact-checking organizations [10]–[12]. The former introduces label noise, and synthetic data fail to capture the visual, linguistic, and cultural nuances of in-the-wild misinformation. Conversely, although datasets derived from fact-checking sources offer high-quality annotations, they provide clean, curated, and self-contained claims that do not reflect the noisy, contextual way users communicate on social media platforms.

To address this, we introduce X-POSE, a new in-the-wild, evidence-enhanced dataset of multimodal user posts from X (formerly Twitter), annotated through the Community Notes system³, enabling both training and evaluation of MFC models—unlike prior work that used it only for evaluation [13]. We leverage VLMs to refine search queries, use both text-based and reverse image search APIs to retrieve full-length news articles as potential evidence, and apply VLMs to extract the article excerpts most relevant for supporting or refuting each claim. Moreover, we integrate source credibility to enhance evidence quality and community ratings to assess consensus for more reliable evaluation.

As shown in Figure 1 (a), a post falsely claims a “Muslim congresswoman” was photographed training at an “Al Qaeda camp,” exploiting harmful religious stereotypes to incite distrust and hostility. The VLM extracts a structured claim and contextual metadata (e.g., named entities, OCR text, image descriptions, and the “5Ws”: Who, What, Where, When, Why), then generates search queries that, together with reverse image search, return relevant web articles. Finally, the VLM identifies the most pertinent evidence excerpts— revealing that the inferred individual was born in 1982, while the photo was taken in 1978—providing the key information needed for the MFC system to dispute the claim.

On the modeling side, existing methods fall into three categories: (1) Intra-modality evidence consistency models, which use separate streams to examine text-text evidence, image-image evidence, and the internal consistency of the image-text pair [14], [15]; (2) All-to-All models, which jointly attend over all modalities and evidence [16], [17]; and (3) VLM-based methods [18], [19]. However, the first ignores cross-modal evidence relations, the second can struggle to identify subtle relations, while the third can suffer from high computational complexity.

To address these limitations, we introduce TRENT, an architecture that, as illustrated in Figure 1 (b), models text-to-all evidence, image-to-all evidence, and image-to-text alignment using three parallel cross-attention streams. The resulting outputs are processed by a relational fusion mechanism designed to capture entailment and contradiction. A comparative evaluation against commercial VLMs (including Grok 4, Claude Sonnet 4.6, Gemini 3, and GPT-5.4) and state-of-the-art (SotA) specialized MFC methods trained on X-POSE shows that TRENT consistently outperforms competing methods. Extensive ablation studies demonstrate that each component—evidence source credibility, evidence reranking, VLM-assisted excerpt extraction, and especially triangulated cross-attention and relational fusion—contributes significantly to overall performance.

2 Background and Related Work↩︎

2.1 Problem Formulation↩︎

Let \(\mathcal{D} = \{(I_n, T_n, \mathcal{E}^{I}_n, \mathcal{E}^{T}_n, y_n)\}_{n=1}^{N}\) be a dataset of \(N\) multimodal instances. Each instance consists of a multimodal pair \((I_n, T_n)\), where \(I_n \in \mathcal{I}\) is an image and \(T_n \in \mathcal{T}\) is its associated text (user post). To verify the veracity of the pair, we incorporate two sets of external evidence:

• \(\mathcal{E}^{I}_n = \{E^{I}_{n,1}, E^{I}_{n,2}, \ldots, E^{I}_{n,M}\}\) represents external information retrieved via reverse-image search using \(I_n\).

• \(\mathcal{E}^{T}_n = \{E^{T}_{n,1}, E^{T}_{n,2}, \ldots, E^{T}_{n,M}\}\) represents external information retrieved using the text \(T_n\) as a query.

Each evidence set contains up to \(M\) items (e.g., full-length news articles or article excerpts). The ground-truth label is denoted by \(y_n \in \{0, 1\}\), where \(y_n = 0\) represents a truthful claim and \(y_n = 1\) indicates misinformation. The task of MFC is to learn a predictive function \(f: (\mathcal{I}, \mathcal{T}, \mathcal{E}^I, \mathcal{E}^T) \rightarrow \hat{y}\), which minimizes the discrepancy between the predicted verdict \(\hat{y}\) and the ground-truth \(y\).

2.2 Multimodal Fact-Checking Datasets↩︎

Due to the limited scale and event diversity of early MFC datasets [20]–[23], subsequent works turned to scalable synthetic data generation via decontextualization [5]–[7], named-entity manipulation [8], [9], [24], VLMs [25], or automated weak supervision [26]. To bridge the resulting distribution gap between algorithmically generated data and real-world misinformation, recent works leverage real-world evaluation benchmarks such as COSMOS [6] and VERITE [10]. However, these benchmarks comprise self-contained claims curated by fact-checkers rather than uncurated user posts, restricting their suitability for studying misinformation in the wild.

Beyond internal image-text consistency, robust multimodal fact-checking typically necessitates external information to cross-examine claims. However, existing evidence-enhanced datasets suffer from significant limitations in scale, balance, and realism. The NewsCLIPpings+ dataset [14] relies on synthetic decontextualization of news media and restricts textual evidence to article titles and brief captions. This setup frequently surfaces original or near-identical articles during evidence retrieval, causing data leakage [17], [27]. Similarly, FACTIFY [4] restricts its scope to official news handles on Twitter and relies on heuristic annotations, thus allowing models to exploit shortcuts and reach an inflated 99–100% accuracy on the Refute class [28], [29], limiting its utility and realism.

Conversely, although AVerImaTeC [11] and M4FC [30] provide curated multimodal claims from fact-checked articles, their limited scale and class imbalances (e.g., only 17 ‘Supported’ claims in AVerImaTeC, 292 in M4FC) prevent effective training. Similarly, while VeriTaS [12] expands MFC beyond image-text pairs, it is intended for evaluation and comprises curated self-contained claims from fact-checking sites rather than user-generated content.

2.3 Crowdsourced Fact-Checking↩︎

Recent work has examined the effectiveness of crowdsourced fact-checking systems such as X’s Community Notes, which append clarifying annotations to potentially misleading posts. Adding a note has been shown to significantly reduce user engagement with misleading content [32]; furthermore, posts labeled as misleading are much more likely to be voluntarily deleted by their authors [33]. Notes citing neutral, high-quality sources also receive higher helpfulness scores, suggesting that contributors actively prioritize factual accuracy [34]. Beyond human curation, recent research has explored generating notes with LLMs and predicting their subsequent helpfulness ratings [35]–[38]. Community Notes have also served as specialized benchmarks for the intent-aware classification of AI-generated images [39] and multimodal fact-checking [13]. However, they have not yet been utilized for both training and evaluating MFC models.

In contrast, as shown in Table 1, the proposed X-POSE dataset comprises uncurated, user-generated content of sufficient scale and balance to enable both the training and evaluation of MFC models under realistic, in-the-wild settings.

2.4 Evidence-Based Detection↩︎

Existing MFC methods can be classified into three paradigms. The first assesses intra-modality consistency (e.g., text-to-text and image-to-image evidence examination) alongside internal image-text consistency, leveraging memory networks, stance extraction modules, or named-entity co-occurrence features [14], [15], [40]. The second paradigm models all-to-all relationships across the claim, image, and retrieved evidence blocks using dedicated attention- and Transformer-based architectures [16], [17], [24]. However, approaches that restrict comparisons to intra-modality consistency often overlook vital cross-modal evidence relations, whereas full all-to-all architectures can capture broad salient cues but fail to isolate the subtle relations of entailment and contradiction. More recently, a third paradigm explores VLMs and LLM-based agentic frameworks to rerank retrieved evidence and generate final verifications [18], [19], [41]–[44].

In this study, we synthesize these paradigms by leveraging VLMs for claim reformulation, evidence collection, and filtering, coupled with TRENT, a specialized, lightweight MFC architecture that explicitly models the relationships between internal image-text pairs, image-to-all evidence, and text-to-all evidence.

3 Construction of X-POSE↩︎

Data Collection and Annotation. To construct X-POSE (X Posts with Evidence), we source from the X Community Notes archive. From the Notes file, we define the Truthful class using notes marked as factually correct, not satire, not outdated, and citing trustworthy sources. For the Misinformation class, we retain notes marked as missing important context or containing factual errors that still cite trustworthy sources. We exclude cases involving manipulated or AI-generated images to focus on misinformation involving authentic but miscaptioned or out-of-context images. While identifying synthetic media is increasingly relevant to multimodal misinformation [45], it often demands distinct visual forensic analysis [46], [47] that falls outside the scope of this study.

We collect the associated posts, retaining only multimodal samples that contain both text and images. The final X-POSE comprises 5,704 unique image-text pairs, balanced across factually correct (2,881) and misinformation (2,823) samples, spanning 2017–2025. Averaging the outputs of two language detection tools—Fast-LangDetect and Lingua—we estimate that the majority of posts are in English (84.8%), followed by French (3.5%), Portuguese (2.1%), Spanish (1.8%), and German (1.2%). Using the IPTC taxonomy⁴ with Gemma 3, we find X-POSE is primarily composed of political content (24.8%), followed by arts and media (19.7%), conflict and war (15.8%), crime and justice (7.6%), science and technology (6.8%), and society (5.7%), with the remaining 11 categories (e.g., health, economy, sports) each accounting for less than 5%, reflecting the dataset’s topical diversity.

3.0.0.1 Consensus-based Data Filtering.

Crowdsourced annotations can be noisy due to annotator disagreements regarding note phrasing, source credibility, or subjective interpretations. To mitigate this, we filter examples using a helpfulness score, which serves as a proxy for community consensus. For a given sample \(n\), let \(\eta_n^+\) and \(\eta_n^-\) denote the number of Helpful and Not Helpful votes, respectively, aggregated from the raw Community Notes ratings files. An item’s helpfulness score \(h_n \in [0, 100]\) is defined as: \[h_n = \begin{cases} \frac{\eta_n^+}{\eta_n^+ + \eta_n^-} \times 100, & \text{if } \eta_n^+ + \eta_n^- > 0,\\ 0, & \text{otherwise.} \end{cases}\] We retain samples where \(h_n \ge \theta\) to construct high-agreement evaluation subsets. For simplicity, we refer to these as the \(h \ge \theta\) subsets.

3.0.0.2 Evidence Collection.

Our evidence collection pipeline is grounded in the workflows of professional fact-checkers, who (i) disambiguate claims, (ii) contextualize them by identifying relevant entities and circumstances, and (iii) formulate targeted queries to search for corroborating or contradicting evidence [48]–[50]. We model these steps using a three-stage VLM-assisted pipeline⁵ (1) Claim Reformulation (Prompt 1) transforms raw, noisy user posts into clear, self-contained textual claims. (2) Multimodal Context Extraction (Prompt 2) extracts structured metadata, including named entities, image descriptions, OCR text, and the “5Ws”. (3) Search Query Generation (Prompt 3) produces concise queries for search APIs, conditioned on the reformulated claim, image, and extracted context. We utilize Gemma 3 due to its consistent performance and lack of the restrictive safety filters—such as those regarding public figures or sensitive political discourse—frequently encountered in large proprietary models. We submit the generated queries and images to the Google Search and Lens APIs via ScrapingDog, retrieving up to 10 results per query. We fetch the pages, extract clean body text and discard cookie-consent popups and JavaScript-rendering errors.

3.0.0.3 Evidence Refinement.

To reduce unreliable content and prevent information leakage (e.g., retrieving the associated Community Note), we exclude major social media platforms (X, Facebook, Instagram, YouTube, and TikTok) and use the MBFC⁶ taxonomy to assess source credibility as a proxy for evidence quality and remove low-credibility outlets. From 44,559 retrieved pages, we remove 6,366 social media items and 1,404 low-credibility articles, while 18,536 remain unrated. The final collection comprises articles from 10,296 unique domains, with the most frequent sources including Wikipedia (1,958), BBC (704), The Guardian (680), CNN (507), Al Jazeera (403), and NPR (341). Furthermore, because 49.9% of these collected articles postdate the original user post, we conduct additional experiments under a past-only evidence setting to more accurately simulate the fact-checking of novel claims without future data leakage.

3.0.0.4 Excerpt Extraction.

Information relevant to a specific claim is typically concentrated in localized segments of full-length articles [51]. Accordingly, we utilize a VLM, MiniCPM-V 2.6⁷ [52], to perform cross-modal reasoning between the multimodal claim (text and image) and each retrieved article. Using Prompt 4, the VLM extracts verbatim excerpts from the text that are directly relevant to the claim. This process yielded 22,226 evidence excerpts (median length: 61 words). Of the collected posts, 4,785 are associated with at least one relevant excerpt, while 919 yielded no relevant evidence. The number of excerpts per post ranges up to a maximum of 16, with a median of 4; 79.4% originate from text-based retrieval and 20.6% from image-based retrieval.

4 The TRENT Architecture↩︎

4.1 Multimodal Representation↩︎

We employ CLIP ViT-L/14 [53] as a pre-trained encoder to extract image embeddings \(\mathbf{i} \in \mathbb{R}^d\) and text embeddings \(\mathbf{t} \in \mathbb{R}^d\), where \(d = 768\) denotes the embedding dimension. Similarly, the top-\(M\) retrieved evidence excerpts are encoded into matrices \(\mathbf{e}^{I}, \mathbf{e}^{T} \in \mathbb{R}^{d \times M}\) representing image- and text-based evidence, respectively, or concatenated into a single matrix \(\mathbf{e} \in \mathbb{R}^{d \times 2M}\) when combined.

4.2 Evidence Reranking↩︎

Search engine APIs already include mechanisms to retrieve and rank potentially relevant content based on a given image-text pair. Nevertheless, as prior studies have shown [16], it is often beneficial to apply a reranking step. For each post \(n\), we rerank retrieved evidence items using cosine similarity in the embedding space. Specifically, we compare the claim text embedding \(\mathbf{t}_n\) against all text-retrieved evidence embeddings \(\mathbf{e}_n^T\), and the claim visual embedding \(\mathbf{i}_n\) against all image-retrieved evidence embeddings \(\mathbf{e}_n^I\). Each evidence set contains up to 10 items. We then select the top-\(M\) evidence items with the highest similarity scores. If fewer than \(M\) candidates are available, we pad with zero vectors.

4.3 Evidence Triangulation↩︎

TRENT employs three parallel cross-attention transformer blocks \(\mathrm{C(\cdot)}\) and relational fusion \(\mathcal{R(\cdot)}\) to capture interactions across images, texts, and evidence.

First, raw embeddings are projected into a shared latent space of dimension \(d_c = 512\) using modality-specific learnable linear projections: \[\mathbf{z}_i = \mathbf{W}_i \;\mathbf{i}, \quad \mathbf{z}_t = \mathbf{W}_t \;\mathbf{t}, \quad \mathbf{z}_e = \mathbf{W}_e \;\mathbf{e}\] where \(\mathbf{W}_i, \mathbf{W}_t, \mathbf{W}_e \in \mathbb{R}^{d_c\times d}\) for image, text, and evidence, respectively.

Given query \(\mathbf{z}_q\) and key-value \(\mathbf{z}_{kv}\) vectors, a \(\mathrm{C(\cdot)}\) block is defined as: \[\begin{align} \mathbf{a} &= \text{LN}(\mathbf{z}_q + \mathrm{MHA}(\mathbf{z}_q, \mathbf{z}_{kv}, \mathbf{z}_{kv})) \\ \mathrm{C}(\mathbf{z}_q, \mathbf{z}_{kv}) &= \text{LN}(\mathbf{a} + \text{FFN}(\mathbf{a})) \end{align}\] where \(\mathrm{MHA}(\cdot)\) denotes multi-head attention with \(h=8\) heads and a key padding mask to ignore zero-padded entries, \(\text{FFN}(\mathbf{z}) = \mathbf{W}_2(\text{GELU}(\mathbf{W}_1 \mathbf{z}))\) is a Feed-Forward Network with \(\mathbf{W}_1, \mathbf{W}_2\in\mathbb{R}^{d_c\times d_c}\), and Layer Normalization (LN) is applied after each residual connection.

We implement three parallel verification streams to examine the relationship between: (i) the visual content and all external evidence (\(\mathbf{c}_{i \rightarrow e}\)), (ii) the textual claim and all external evidence (\(\mathbf{c}_{t \rightarrow e}\)), and (iii) the internal cross-modal consistency between the paired text and image (\(\mathbf{c}_{t \leftrightarrow i}\)). Formally expressed: \[\begin{align} \label{eq:c} \mathbf{c}_{i \rightarrow e} = \mathrm{C}(\mathbf{z}_i, \mathbf{z}_e), \; \mathbf{c}_{t \rightarrow e} = \mathrm{C}(\mathbf{z}_t, \mathbf{z}_e), \; \mathbf{c}_{t \leftrightarrow i} &= \mathrm{C}(\mathbf{z}_t, \mathbf{z}_i) \end{align}\tag{1}\]

4.3.0.1 Relational Fusion.

To capture entailment and contradiction relations between the image-text pair and the external evidence, we employ a relational fusion function \(\mathcal{R(\cdot)}\) inspired by standard formulations in natural language inference (NLI) [54], but adapted for MFC: \[\mathcal{R}(\mathbf{z}_1, \mathbf{z}_2) = [ \mathbf{z}_1 ; \mathbf{z}_2 ; |\mathbf{z}_1 - \mathbf{z}_2| ; \mathbf{z}_1 \odot \mathbf{z}_2 ]\] where \([;]\) denotes concatenation and \(\odot\) denotes the element-wise product. Applying \(\mathcal{R}\) to each stream and concatenating yields the final representation: \[\label{eq:z} \mathbf{z}_{r} = [\mathcal{R}(\mathbf{z}_i, \mathbf{c}_{i \rightarrow e}) ; \mathcal{R}(\mathbf{z}_t, \mathbf{c}_{t \rightarrow e}) ; \mathcal{R}(\mathbf{z}_t, \mathbf{c}_{t \leftrightarrow i})]\tag{2}\] with \(\mathbf{z_r} \in\mathbb{R}^{12d_c}\); \(4d_c\) per stream.

4.3.0.2 Classification.

The final veracity score \(\hat{y}\) is obtained by a linear classification layer followed by a sigmoid activation: \(\hat{y} = \sigma(\mathbf{W}_r \;\mathbf{z}_{r} + b)\), with \(\mathbf{W}_r \in \mathbb{R}^{1 \times 12d_c}\). The model is trained using the binary cross-entropy loss.

5 Experimental Setup↩︎

Implementation Details. TRENT and all competing methods are trained and evaluated on X-POSE under two experimental setups. The primary setup uses a random split into 4,588 training, 559 validation, and 557 test samples. Performance is measured using macro F1 on the unfiltered test set and on high-agreement subsets (\(h \ge 80\%, h \ge 90\%\)). Applying thresholds \(\theta \in \{10, 20, \dots, 90\}\) on this random test set results in pruned evaluation subsets of 494, 475, 448, 402, 362, 315, 269, 200, and 120 samples, respectively. To evaluate temporal generalization, we introduce a chronological data split consisting of 4,563 training posts spanning 2017–11/2024 (with 95% in 2023–2024), 570 validation posts from 11/2024–01/2025, and 571 test posts from 01/2025–04/2025. Across both setups, the validation sets are used for hyperparameter tuning over the number of evidence items \(M \in \{1,3,5,10\}\) and learning rate \(\eta \in \{10^{-4}, 5\times10^{-5}\}\). All models are optimized with a batch size of 512 using Adam for up to 50 epochs, with early stopping after 10 epochs of validation stagnation. Random seeds are set to 0 across PyTorch, Python, and NumPy for reproducibility.

5.0.0.1 Competing Methods.

As no prior work has evaluated on X-POSE, we implement a broad set of MFC models: CCN [14], ERIC-FND [55], and ECENet [15] which leverage intra-modality evidence consistency, as well as DT-Transformer [24], RED-DOT [16], MUSE, and AITR [17] that model all-to-all interactions within a single stream. For a fair comparison, all replicated methods utilize a CLIP ViT-L/14 backbone and follow the training protocols and official repositories of the original papers. Additionally, we evaluate several open-source and commercial VLMs for zero-shot detection on X-POSE, using Prompt 6 to incorporate the image-text pair alongside the extracted evidence excerpts: MiniCPM [52], Gemma 3 and 4, GPT-5 Mini and 5.4 Mini, Gemini 2.5 Flash and 3 Flash, Grok 4 Fast, and Claude 4.6 Sonnet. Except for MiniCPM (which is run locally), all VLMs are accessed via the OpenRouter.ai API. Finally, we evaluate TRENT on NewsCLIPpings+ and VERITE, comparing it against optimized detectors RED-DOT and AITR, as well as VLM-based detectors SNIFFER [42], TRUST-VL (LLaVA-1.5) [44], E2LVLM (Qwen2-VL) [18], DEFAME (GPT-4o) [19], and MAD-Sherlock (GPT-4o) [43], reporting performance from their original papers.

6 Results↩︎

6.1 Comparative Results↩︎

Table 2 presents the performance of TRENT on X-POSE, against a wide range of MFC detectors. Our model consistently outperforms all baselines across the standard random split, achieving macro-F1 scores of 63.10%, 67.36%, and 70.83% on the full test set and high-agreement subsets (\(h \geq 80, 90\)). Under a past-only evidence setting—only allowing evidence published before the user post—TRENT’s performance drops to 62.22% on the random split and 60.14% on the chronological split. This decrease is expected given the significantly more challenging and realistic constraints of these settings; however, TRENT still consistently outperforms all competing methods.

Crucially, this competitive efficiency stems from architectural design rather than scale. TRENT comprises only 5.13M trained parameters, significantly fewer than specialized methods like RED-DOT (16.9M) or AITR (18.7M), and orders of magnitude smaller than the tens or hundreds of billions of parameters of VLMs. Additionally, excluding the shared upfront costs of evidence excerpt extraction and embedding generation (which are identical across all models and performed only once), TRENT requires a mere 6.6–32.6 MFLOPs (scaling from 2 to 20 evidence documents) and is trained in just 21–30 seconds on a single RTX 3060 GPU. Furthermore, TRENT performs inference on the test set (557 samples) in less than 1 second; in stark contrast, evaluating the same inference set took 11m for GPT-5.4 Mini, 26m for Claude Sonnet 4.6, 33m for Gemma 4, and up to 56m for Gemini 3 Flash.

Within specialized MFC methods, all-to-all attention-based models (AITR, RED-DOT, DT-Transformer) tend to outperform intra-modality consistency models (CCN, ERIC-FND, ECENet). The latter restrict examination across modalities, missing key interactions between the claim and image evidence, and the image with textual evidence. Yet, while all-to-all architectures attend to all inputs simultaneously, their single-stream design can still overlook fine-grained dependencies. In contrast, TRENT explicitly models text \(\rightarrow\) all evidence, image \(\rightarrow\) all evidence, and internal image-text interactions through three dedicated cross-attention streams, combined with relational fusion, thereby better isolating subtle supporting or contradicting signals.

Among the evaluated VLMs, Gemma 4 and Gemini 3 yield the highest performance behind TRENT. We observe improvements within these model families, with Gemma 4 significantly outperforming Gemma 3, and Gemini 3 surpassing Gemini 2.5. While these gains may stem from enhanced reasoning capabilities, they could also be partially attributed to data contamination, as these large models could have encountered some of the target posts, events, or associated source articles during pre-training. Nevertheless, this risk is mitigated on the chronological split, which features minimal overlap with the training data of these VLMs due to its post-January 2025 timeline.

Moreover, as shown in Table 3, TRENT outperforms RED-DOT, E2LVLM, TRUST-VL, SNIFFER, DEFAME, and MAD-Sherlock across NewsCLIPpings+ and VERITE, while achieving competitive performance with AITR. However, specialized MFC models like RED-DOT and AITR struggle on X-POSE because they were developed on out-of-context benchmarks prone to retrieval shortcuts (Section 2.2), where the similarity-based baseline MUSE attains high scores on NewsCLIPpings (90.0%) and VERITE (80.5%) due to leakage [17]. Similarly, when MUSE shortcuts are ablated, AITR’s performance significantly decreases on VERITE, from 81.0 to 71.0%. In contrast, MUSE drops to near-random on X-POSE (53.36% on the full test set and 49.37% on the chronological split). This sharp degradation indicates that X-POSE avoids such retrieval shortcuts.

6.2 Ablation Study↩︎

Table 4 presents ablations analyzing the training data (Ablation 1), inputs (Ablations 2–10), and architectural components (Ablations 11–16) of TRENT.

6.2.0.1 Scale vs.Data Quality.

All models are trained on the full training set and evaluated on the test set and two high-agreement subsets (\(h \ge 80\), \(h \ge 90\)). Training only on high-agreement data (Ablation 1) improves label reliability but reduces the dataset to 1,765 (\(h \ge 80\)) and 1,015 (\(h \ge 90\)) samples, causing a clear performance drop. For TRENT, scale outweighs consensus during training.

6.2.0.2 Modality and Evidence Representation.

Ablations 2-4 modify evidence representations. Using full-article embeddings from a long-context LLM (Ablation 2; 4096-dim Qwen GTE-7B⁸ with Prompt 5) substantially degrades performance, suggesting verification cues are localized and diluted in full-document embeddings. Replacing excerpts with VLM summaries (Ablation 3; MiniCPM with Prompt 5) also reduces performance, likely due to loss of fine-grained details. Substituting the CLIP-aligned text encoder with multi-E5 (Ablation 4) yields no gains, likely due to visual-textual misalignment.

6.2.0.3 Evidence Filtering and Reranking.

Allowing low-credibility and social media sources in the evidence pool reduces performance (Ablation 5), validating the role of credibility-based filtering in mitigating the impact of noisy or biased evidence. Similarly, disabling reranking (Ablation 6) results in a slight performance decline on high-agreement subsets. This indicates that while the evidence retrieval and excerpt extraction pipeline yields reasonably relevant candidates, reranking provides a beneficial refinement step.

6.2.0.4 Contribution of Modalities and External Evidence.

Removing images (Ablation 7) drastically degrades performance, confirming the contribution of the visual signals and that X-POSE is not dominated by unimodal shortcuts. Removing all external evidence (\(-\mathcal{E}\), Ablation 8) causes a significant drop, underscoring the necessity of external information. Removing image-retrieved evidence (\(-\mathcal{E}^I\), Ablation 9) is more detrimental than removing text-retrieved evidence (\(-\mathcal{E}^T\), Ablation 10), suggesting that reverse-image search often provides crucial historical context, or provenance, missing from the user post.

6.2.0.5 Evidence Triangulation and Relational Fusion.

Ablations 11-12 restrict processing to image \(\rightarrow\) image-retrieved evidence and text \(\rightarrow\) text-retrieved evidence, mirroring prior intra-modality evidence consistency architectures [14], [15], and significantly reducing performance. Replacing relational fusion \(\mathcal{R}\) (Ablations 11 and 13) with feature concatenation similarly weakens performance, underscoring its importance. Moreover, removing any individual stream from the three-stream architecture (\(\mathbf{c_{t \rightarrow e}}\), \(\mathbf{c_{i \rightarrow e}}\), \(\mathbf{c}_{t \leftrightarrow i}\)) (Ablations 14-16) significantly reduces performance. Together, these findings validate TRENT’s design.

6.3 Further Analysis↩︎

Figure 3 (a) evaluates TRENT and five of the top performing methods across varying helpfulness thresholds \(\theta \in \{10, 20, \dots, 90\}\). Performance remains static under low agreement (\(\theta \le 60\%\), slope \(\beta \approx 0.02\)), suggesting that simple majorities (50–60%) may suffer from user polarization. Conversely, beyond \(\theta = 70\%\), methods exhibit a positive trend in macro-F1 (\(\beta \approx 0.11\)), indicating that super-majority agreement yields cleaner, higher-fidelity labels. However, enforcing these higher thresholds significantly reduces available sample volume, introducing a trade-off between annotation quality and dataset scale.

Figure 3 (b) illustrates the impact of evidence quantity \(M\) on TRENT. Performance improves markedly from \(M=0\) to \(M=1\) and continues to rise, peaking at \(M=10\). This indicates that access to multiple evidence excerpts facilitates cross-source referencing, resolving ambiguities that a single excerpt (\(M=1\)) may miss. Furthermore, it demonstrates that TRENT can successfully isolate relevant signals from increasingly noisy, high-dimensional inputs.

Figure 4 presents two inference examples that illustrate the importance of consensus-based and evidence-quality filtering. In (a), the retrieved evidence is credible, but the annotation is incorrect, as indicated by the very low helpfulness score (3.6%), since the post is humorous rather than misinformation. This motivates our consensus-based filtering, which isolates higher-fidelity evaluation labels. In (b), despite a correct annotation (100% helpfulness), the evidence originates from an unreliable, highly biased source⁹ with low factual reporting, which provides misleading information that superficially supports the claim. This highlights the necessity of assessing evidence credibility (Section 3).

7 Conclusion↩︎

In this study, we introduced X-POSE, a novel in-the-wild evidence-enhanced dataset for training and evaluating MFC models, comprising user posts annotated through X’s Community Notes and augmented with VLM-refined external evidence. To address the limitations of prior MFC architectures, we proposed TRENT, which performs explicit evidence triangulation with relational fusion across text, image, and retrieved sources; it outperforms specialized MFC models and commercial VLMs, highlighting its efficiency and effectiveness.

Despite these improvements, overall performance remains limited, underscoring the difficulty of in-the-wild detection and the realism of X-POSE. This leaves ample room for further research. First, X-POSE can be used to develop and evaluate novel methods for in-the-wild MFC, while also exploring explainability [49] by comparing model explanations against community notes. Second, because nearly 50% of the retrieved articles in X-POSE currently lack credibility ratings, further research is required on evidence quality [27] and source credibility [56]. Finally, while our consensus-based filtering improves crowdsourced annotation quality by removing low-agreement entries, it introduces a quality-scale trade-off. To address this, X-POSE can be scaled up as new community notes are released, while expanding beyond X to other platforms like TikTok, Facebook, and Instagram to strengthen future research on cross-platform MFC.

Acknowledgments↩︎

This work is partially funded by Horizon Europe projects AI-CODE and ELLIOT under grant agreement no. 101135437 and 101214398, respectively.

References↩︎