MetaAdapt: Domain Adaptive Few-Shot Misinformation Detection via Meta Learning
May 22, 2023
Abstract
With emerging topics (e.g., COVID-19) on social media as a source for the spreading misinformation, overcoming the distributional shifts between the original training domain (i.e., source domain) and such target domains remains a non-trivial task for misinformation detection. This presents an elusive challenge for early-stage misinformation detection, where a good amount of data and annotations from the target domain is not available for training. To address the data scarcity issue, we propose MetaAdapt, a meta learning based approach for domain adaptive few-shot misinformation detection. MetaAdapt leverages limited target examples to provide feedback and guide the knowledge transfer from the source to the target domain (i.e., learn to adapt). In particular, we train the initial model with multiple source tasks and compute their similarity scores to the meta task. Based on the similarity scores, we rescale the meta gradients to adaptively learn from the source tasks. As such, MetaAdapt can learn how to adapt the misinformation detection model and exploit the source data for improved performance in the target domain. To demonstrate the efficiency and effectiveness of our method, we perform extensive experiments to compare MetaAdapt with state-of-the-art baselines and large language models (LLMs) such as LLaMA, where MetaAdapt achieves better performance in domain adaptive few-shot misinformation detection with substantially reduced parameters on real-world datasets.
1 Introduction↩︎
Recently, significant progress has been made in misinformation detection due to the improvements in developing machine learning-based methods [1]–[3]. Such methods include large language models (LLMs), which can be fine-tuned for detecting and responding to rumors on social media platforms [4]–[6]. However, misinformation on emerging topics remains an elusive challenge for existing approaches, as there exists a large domain gap between the training (i.e., source domain) and the target distribution (i.e., target domain) [7]. For instance, existing models often fail to detect early-stage misinformation due to the lack of domain knowledge (see 1).
With the increase of emerging topics (e.g., COVID-19) on social media as a source of misinformation, the failure to distinguish such early-stage misinformation can result in potential threats to public interest [8], [9]. To tackle the problem of cross-domain early misinformation detection, one possible solution is crowdsourcing, which collects domain knowledge from online resources [10]–[13]. Another alternative approach is to transfer knowledge from labeled source data to unlabeled target data with domain adaptive methods [7], [14]–[18]. However, the former methods use large amounts of human annotations while the latter approaches require extensive unlabeled examples. As such, existing methods are less effective for detecting cross-domain early misinformation, where neither large amounts of annotations nor target domain examples can be provided for training.
Despite the insufficiency of early misinformation data, limited target examples and annotations can often be achieved at minimal costs [19]–[21]. Nevertheless, previous approaches are not optimized to learn from the source data under the guidance of limited target examples [7], [14], [22], [23]. Such methods are often unaware of the adaptation objective and thus fail to maximize the transfer of source domain knowledge. To fully exploit existing data from different domains, we consider a cross-domain few-shot setting to adapt misinformation detection models to an unseen target domain [24]–[26]. That is, given a source data distribution and access to limited target examples, our objective is to maximize the model performance in the target domain. An example of such application can be adapting a model from fake news detection to COVID early misinformation detection, where abundant fake news from existing datasets can be used for training the model under the guidance of limited COVID misinformation examples.
In this paper, we design MetaAdapt, a few-shot domain adaptation approach based on meta learning for early misinformation detection. Specifically, we leverage the source domain examples (i.e., source task) and train a model to obtain the task gradients. Then, we evaluate the updated model on the few-shot target examples (i.e., meta task) to derive second-order meta gradients w.r.t. the original parameters. We additionally compute the similarity between the task gradients and meta gradients to select more ‘informative’ source tasks, such that the updated model adaptively learns (from the source data) to generalize even with a small number of labeled examples. In other words, the meta model learns to reweight the source tasks with the objective of optimizing the model performance in the target domain. Therefore, the resulting model can optimally adapt to the target distribution with the provided source domain knowledge. To show the efficacy of our meta learning-based adaptation method, we focus on the early-stage misinformation of COVID-19 and demonstrate the performance of MetaAdapt on real-world datasets, where MetaAdapt can consistently outperform state-of-the-art methods and large language models by demonstrating significant improvements.
We summarize our contributions as follows1:
We propose a few-shot setting for domain adaptive misinformation detection. Here, the labeled source data and limited target examples are provided for the adaptation process.
We propose MetaAdapt, a meta learning-based method for few-shot domain adaptive misinformation detection. Our MetaAdapt’learns to adapt’ to the target data distribution with limited labeled examples.
MetaAdapt can adaptively learn from the source tasks by rescaling the meta gradients. Specifically, we compute similarity scores between the source and meta tasks to optimize the learning from the source distribution.
We show the effectiveness of MetaAdapt in domain adaptive misinformation detection on multiple real-world datasets. In our experiments, MetaAdapt consistently outperforms state-of-the-art baselines and LLMs.
2 Related Work↩︎
2.1 Misinformation Detection↩︎
Existing misinformation detection methods can be categorized into the following: (1) content-based misinformation detection: such models are trained to perform misinformation classification upon input claims. For example, pretrained transformer models are used to extract semantic or syntactic properties to detect misinformation [4], [7], [27], [28]. Moreover, multimodal input is used to learn text and image features that improve detection performance [29]–[32]; (2) social-aware misinformation detection: user interactions can be used to evaluate online post credibility [33]. Similarly, patterns on propagation paths help detect misinformation on social media platforms [34], [35]. Social attributes like user dynamics enhance misinformation detection by introducing context [36]. Combined with content-based module, misinformation detection systems demonstrate improved accuracy [23], [26]; (3) knowledge-based misinformation detection: external knowledge can be leveraged as supporting features and evidence in fact verification and misinformation detection [37]–[39]. Knowledge graphs or crowdsourcing approaches can derive additional information for explainability in misinformation detection [11], [12], [20], [40]–[42]. Yet existing methods focus on improving in-domain performance or explainability, few-shot misinformation detection in a cross-domain setting is not well researched. Hence, we study domain adaptive few-shot misinformation detection using content-based language models in our work.
2.2 Domain Adaptive Learning↩︎
Domain adaptive learning aims to improve model generalization on an unseen domain given a labeled source domain. Such methods are primarily studied in image and text classification problems [43]–[45]. In image classification, existing methods minimize the representation discrepancy between source and target domains to learn domain-invariant features and transfer source knowledge to the target domain [44], [46]. Similarly, domain-adversarial and energy-based methods adopt additional critique, with which domain-specific features are regularized [45], [47]. Class-aware contrastive learning is proposed for fine-grained alignment, which regularizes the inter-class and intra-class distances to achieve domain-invariant yet class-separating features [43], [48].
In text classification, various approaches are proposed to improve the target domain performance in cross-domain settings [15], [49]–[51]. For instance, domain-adversarial training is used to learn generalizable features to detect cross-domain multimodal misinformation [17], [26], [52]. Reinforcement learning and contrastive adaptation are also adopted for fine-grained domain adaptation in misinformation detection [7], [23]. Nevertheless, domain-adaptive misinformation detection is not well studied in the few-shot learning setting. Therefore, we combine both settings and develop a method tailored for few-shot domain adaptation in misinformation detection: MetaAdapt. By leveraging knowledge transfer via the proposed meta objective, our approach shows significant improvements on out-of-domain misinformation using only a few labeled examples.
2.3 Few-Shot Learning↩︎
Few-shot learning aims to learn a new task with a few labeled examples [53]. Existing few-shot learning approaches (e.g., prototypical networks) learn class-wise features in the metric space to rapidly adapt to new tasks [54], [55]. Meta learning methods search for the optimal initial parameters for unseen few-shot tasks via second-order optimization [56]–[58]. In computer vision, few-shot domain adaptation is studied in image classification to transfer knowledge to an unseen target domain [24], [25], [59]. For language problems, meta learning is proposed to improve the few-shot performance in language modeling and misinformation detection [60]–[64].
To the best of our knowledge, few-shot domain adaptive misinformation detection via meta learning is not studied in current literature. Moreover, the mentioned few-shot setting can be helpful in real-world scenarios (e.g., detecting rumors on emerging topics). As such, we propose meta learning-based MetaAdapt for misinformation detection. MetaAdapt leverages limited target examples and adaptively exploits the source domain knowledge via task similarity, and thus improves the few-shot misinformation detection performance in the unseen target domain.
3 Preliminary↩︎
We consider the following problem setup for domain adaptive few-shot misinformation detection: labeled source data and \(k\)-shot target examples (i.e., \(k\) examples per class) are available for training. The objective of our framework is to train a misinformation detection model \(\boldsymbol{f}\) that is optimized for the target domain performance using both source and few-shot target examples.
Data: Our research is defined within the scope of single-source adaptive misinformation detection (i.e., we study the adaptation problem from a single source domain to the target domain). We denote \(\boldsymbol{\mathcal{D}}_{s}\) as the source domain and \(\boldsymbol{\mathcal{D}}_{t}\) as the (different) target domain. In our setting, labeled source data and limited target examples can be used for training. The few-shot adaptation is performed in two-fold: (1) an initial model is updated upon multiple batches of sampled source data examples (i.e., source tasks) respectively; (2) the updated models are evaluated on the few-shot target examples respectively (i.e., meta task) to compute the meta loss, followed by updating the initial parameters using the derived second-order derivatives. The input data is defined as follows:
Labeled source data: source training data \(\boldsymbol{X}_{s}\) is provided by source domain \(\boldsymbol{\mathcal{D}}_{s}\). Here, each sample \((\boldsymbol{x}_{s}^{(i)}, y_{s}^{(i)}) \in \boldsymbol{X}_{s}\) is a tuple comprising of input text \(\boldsymbol{x}_{s}^{(i)}\) and label \(y_{s}^{(i)} \in \{ 0, 1 \}\) (i.e., false or true). During training, source data batches are sampled as different ‘source tasks’ and used to optimize the initial model.
Few-shot target data: we assume limited access to the target domain \(\boldsymbol{\mathcal{D}}_{t}\). In other words, only \(k\)-shot subset \(\boldsymbol{X}'_{t}\) from \(\boldsymbol{X}_{t}\) is provided for training. Target samples are provided in the same label space with \((\boldsymbol{x}_{t}^{(i)}, y_{t}^{(i)}) \in \boldsymbol{X}'_{t}\), while the size of \(\boldsymbol{X}'_{t}\) is constrained with \(k\) examples in each label class (i.e., 10 in our experiments). \(\boldsymbol{X}'_{t}\), or ‘meta task’ is used to compute the meta loss and meta gradients w.r.t. the initial parameters.
Model & Objective: The misinformation detection model is represented by a function \(\boldsymbol{f}\) parameterized by \(\boldsymbol{\theta}\). \(\boldsymbol{f}\) takes textual statements as input and predicts the probability of input as true information, i.e., \(y^{(i)} = \arg \max (\boldsymbol{f}(\boldsymbol{\theta}, \boldsymbol{x}^{(i)}))\). For optimization, our objective is to maximize the model performance on target data \(\boldsymbol{X}_{t}\) (Note \(\boldsymbol{X}_{t} \neq \boldsymbol{X}'_{t}\)) from the target domain \(\boldsymbol{\mathcal{D}}_{t}\). Mathematically, this can be formulated as the optimization problem of minimizing the loss \(\mathcal{L}\) of \(\boldsymbol{\theta}\) over target data \(\boldsymbol{X}_{t}\): \[\min_{\substack{\boldsymbol{\theta}}} \mathcal{L}(\boldsymbol{\theta}, \boldsymbol{X}_{t}),\] where \(\mathcal{L}\) is the loss function (i.e., cross-entropy).
4 Methodology↩︎
Provided with labeled source data and \(k\)-shot target examples, we first present our meta adaptation framework for domain adaptive few-shot misinformation detection. To improve the adaptation performance, we introduce a second-order meta learning algorithm MetaAdapt based on learnable learning rate and task similarity. An illustration of MetaAdapt is provided in 2. Upon deployment, the adapted models achieve considerable improvements thanks to the adaptive optimization and similarity-guided meta adaptation.
4.1 Few-Shot Meta Adaptation↩︎
Given model \(\boldsymbol{f}\) with initial parameters \(\boldsymbol{\theta}\), source dataset \(\boldsymbol{X}_{s}\), few-shot target data \(\boldsymbol{X}'_{t}\) and the number of tasks \(n\) in each iteration, we formulate the meta adaptation framework as a bi-level optimization problem and provide a mathematical formulation:
\[\label{eq:outer95lever} \min_{\substack{\boldsymbol{\theta}}} \frac{1}{n} \sum^{n} \mathcal{L}(\mathcal{A}lg(\boldsymbol{\theta}, \texttt{Sampler}(\boldsymbol{X}_{s})),
\boldsymbol{X}'_{t}),\tag{1}\] in which Sampler stands for the source task sampler that draws source tasks of a fixed size from \(\boldsymbol{X}_{s}\), \(\mathcal{A}lg\) represents the optimization algorithm using first-order gradient descent, i.e.: \[\label{eq:inner95lever} \mathcal{A}lg(\boldsymbol{\theta},
\boldsymbol{X}) = \boldsymbol{\phi} = \boldsymbol{\theta} - \alpha \nabla_{\boldsymbol{\theta}} \mathcal{L}(\boldsymbol{\theta}, \boldsymbol{X}),\tag{2}\] with \(\alpha\) representing the task learning rate
and \(\boldsymbol{\phi}\) representing the learnt parameter set over \(\boldsymbol{X}\).
In 1 , we are interested in learning an optimal parameter set \(\boldsymbol{\theta}\) that minimizes the meta loss on the few-shot target set \(\boldsymbol{X}'_{t}\), which can be denoted as the outer-level optimization of meta adaptation. The outer-level learning is achieved by deriving gradients w.r.t. \(\boldsymbol{\theta}\)
based on the meta loss using task-specific parameters (i.e., \(\mathcal{A}lg(\boldsymbol{\theta}, \texttt{Sampler}(\boldsymbol{X}_{s}))\) or \(\boldsymbol{\phi}\)) and few-shot examples
\(\boldsymbol{X}'_{t}\). To obtain task-specific parameters, we sample a batch of source examples using Sampler and perform gradient descent steps on the original \(\boldsymbol{\theta}\) (i.e, 2 ), which is known as the inner-level optimization. The inner-level optimization only requires first-order derivatives, however, to optimize the outer-level
problem, it is necessary to differentiate through \(\mathcal{A}lg\) (i.e., \(\boldsymbol{\phi}\)), which requires using second-order gradients [56].
We now take a closer look at how to compute the derivatives with chain rule in meta adaptation: \[\begin{align} \label{eq:analysis} \frac{d \mathcal{L}(\mathcal{A}lg(\boldsymbol{\theta}, \boldsymbol{X}), \boldsymbol{X}'_{t})}{d \boldsymbol{\theta}} &= \\ \frac{d\mathcal{A}lg(\boldsymbol{\theta}, \boldsymbol{X})}{d\boldsymbol{\theta}} \nabla_{\boldsymbol{\phi}} &\mathcal{L}(\mathcal{A}lg(\boldsymbol{\theta}, \boldsymbol{X}), \boldsymbol{X}'_{t}), \end{align}\tag{3}\] Note that \(\mathcal{A}lg(\boldsymbol{\theta}, \boldsymbol{X})\) is equivalent to \(\boldsymbol{\phi}\). The right-side component \(\nabla_{\boldsymbol{\phi}}\mathcal{L}(\mathcal{A}lg(\boldsymbol{\theta}, \boldsymbol{X}), \boldsymbol{X}'_{t})\) refers to first-order derivatives by computing the meta loss using the few-shot examples \(\boldsymbol{X}'_{t}\) and task-specific parameter set \(\boldsymbol{\phi}\) (\(\mathcal{L} \rightarrow \boldsymbol{\phi}\)). This step can be computed with conventional gradient descent algorithms. The left-side component \(\frac{d\mathcal{A}lg(\boldsymbol{\theta}, \boldsymbol{X})}{d\boldsymbol{\theta}}\) is a non-trivial step as it requires second-order derivatives (i.e., Hessian matrix) to track parameter-to-parameter changes through \(\mathcal{A}lg(\boldsymbol{\theta}, \boldsymbol{X})\) to \(\boldsymbol{\theta}\). In our implementation, we compute the meta gradients w.r.t. \(\boldsymbol{\theta}\) using the meta evaluation loss similar to model agnostic meta learning [56]. We also adopt adaptive learning rate \(\alpha\) and \(\beta\) and cosine annealing to improve the convergence of source and meta tasks [65].
4.2 The Proposed MetaAdapt↩︎
While meta adaptation leverage source tasks to improve the target domain performance, it learns homogeneously from all source tasks without considering the informativeness of each individual task. To further exploit the source domain knowledge, we propose a task similarity-based MetaAdapt for domain adaptive few-shot misinformation detection. We first estimate the task-specific parameters with adaptive learning rates, followed by rescaling the meta loss using the task similarity scores. The proposed method selectively learns from source tasks, and thus, further improves the adaptation performance. We present the training details of MetaAdapt in 3.
We initialize the model and denote the parameters with \(\boldsymbol{\theta}\). For source task \(i\), the original parameters are updated with first-order derivatives as we sample source tasks from \(\boldsymbol{X}_{s}\). Specifically in each step, the task gradients can be computed with \(\nabla_{\boldsymbol{\theta}} \mathcal{L}(\boldsymbol{\theta}, \texttt{Sampler}(\boldsymbol{X}_{s}))\), as in 2 . After a few gradient descent steps, the parameters converge locally and we denote the updated parameters with \(\boldsymbol{\phi}_{i}\). As we update multiple times in each source task, we denote the task gradients with \(\boldsymbol{\phi}_{i} - \boldsymbol{\theta}\) for simplicity. Subsequently, the meta loss \(\mathcal{L}(\boldsymbol{\phi}_{i}, \boldsymbol{X}'_{t})\) is computed using \(\boldsymbol{\phi}_{i}\) and the few-shot target examples \(\boldsymbol{X}'_{t}\). To compute the meta gradients with backpropagation, we follow the chain rule and compute the derivatives w.r.t. the original parameter set \(\boldsymbol{\theta}\). Similar to 3 , we use \(\frac{d\boldsymbol{\phi}_{i}}{d\boldsymbol{\theta}} \nabla_{\boldsymbol{\phi}_{i}}\mathcal{L}(\boldsymbol{\phi}_{i}, \boldsymbol{X}'_{t})\) to denote the meta gradients.
To compute task similarity scores, we leverage task and meta gradients. The objective of computing gradient similarity is to selectively learn from the source tasks. If the task and meta gradients yield a high similarity score, the parameters are converging to the same direction in both inner- and outer-loop optimization. Thus, the source task optimization path is more ‘helpful’ to improve the meta task performance (i.e., target domain performance). Or if, on the contrary, then the source task may be less effective for improving the meta task performance. Based on this principle, we compute task similarity score \(s_{i}\) with cosine similarity: \[\label{eq:similarity} s_{i} = \texttt{CosSim}(\boldsymbol{\phi}_{i} - \boldsymbol{\theta}, \frac{d\boldsymbol{\phi}_{i}}{d\boldsymbol{\theta}} \nabla_{\boldsymbol{\phi}_{i}}\mathcal{L}(\boldsymbol{\phi}_{i}, \boldsymbol{X}'_{t})).\tag{4}\] In each iteration, we sample \(n\) source tasks and compute the similarity scores for each pair of task and meta gradients. Then, the similarity scores \([s_{1}, s_{2} , ..., s_{n}]\) are transformed to a probability distribution using tempered softmax: \[\label{eq:rescale} \boldsymbol{s} = \texttt{softmax}([\frac{s_{1}}{\tau}, \frac{s_{2}}{\tau}, ..., \frac{s_{n}}{\tau}]),\tag{5}\] where \(\tau\) is the temperature hyperparameter to be selected empirically. Finally, we update the original parameters with rescaled meta gradients: \[\label{eq:update} \boldsymbol{\theta} - \beta \sum_{i}^{n} s_{i} \cdot \frac{d\boldsymbol{\phi}_{i}}{d\boldsymbol{\theta}} \nabla_{\boldsymbol{\phi}_{i}}\mathcal{L}(\boldsymbol{\phi}_{i}, \boldsymbol{X}'_{t}).\tag{6}\]
In summary, MetaAdapt computes task and meta gradients using sampled source tasks and few-shot target examples. Then, task similarity is computed to find more ‘informative’ source tasks, followed by tempered rescaling of the meta gradients. Finally, the updated model parameters should exploit the source domain knowledge and demonstrate improved performance on the target data distribution. The overall framework of MetaAdapt is illustrated in 2. Unlike previous works [7], [24], [25], [59], we discard domain-adversarial or feature regularization methods, instead, we propose to leverage meta adaptation to guide the knowledge transfer from the source to target domain. Additionally, similarity-based gradients rescaling is designed to exploit different source tasks to achieve fine-grained adaptation performance.
5 Experiments↩︎
5.1 Settings↩︎
Model: Similar to [7], [15], we select RoBERTa as the base model to encode input examples in MetaAdapt. RoBERTa is a transformer model pretrained on a variety of NLP tasks before the COVID pandemic [66].
Evaluation: To validate the proposed method, we follow [7], [15], [20] and split the datasets into training, validation and test sets. The few-shot target examples are selected as the first \(k\) examples in the validation set and the rest validation examples are used for validating the model. For evaluation metrics, we adopt balance accuracy (BA), accuracy (Acc.) and F1 score (F1) to evaluate the performance. See evaluation details in 8.
| Domain | Dataset | BA \(\uparrow\) | Acc. \(\uparrow\) | F1 \(\uparrow\) |
|---|---|---|---|---|
| Source | FEVER | 0.796 | 0.796 | 0.817 |
| GettingReal | 0.846 | 0.959 | 0.978 | |
| GossipCop | 0.776 | 0.869 | 0.917 | |
| LIAR | 0.607 | 0.632 | 0.712 | |
| PHEME | 0.863 | 0.867 | 0.898 | |
| Target | CoAID | 0.889 | 0.972 | 0.985 |
| Constraint | 0.970 | 0.971 | 0.973 | |
| ANTiVax | 0.932 | 0.921 | 0.931 |
| Source | Target | CoAID (2020) | Constraint (2021) | ANTiVax (2022) | ||||||
|---|---|---|---|---|---|---|---|---|---|---|
| 2-11 | Metric | BA \(\uparrow\) | Acc. \(\uparrow\) | F1 \(\uparrow\) | BA \(\uparrow\) | Acc. \(\uparrow\) | F1 \(\uparrow\) | BA \(\uparrow\) | Acc. \(\uparrow\) | F1 \(\uparrow\) |
| FE | Naïve | 0.636 | 0.928 | 0.962 | 0.501 | 0.524 | 0.687 | 0.559 | 0.627 | 0.741 |
| CANMD | 0.626 | 0.918 | 0.956 | 0.684 | 0.683 | 0.686 | 0.650 | 0.679 | 0.749 | |
| ACLR | 0.721 | 0.935 | 0.965 | 0.648 | 0.651 | 0.697 | 0.739 | 0.758 | 0.805 | |
| ProtoNet | 0.751 | 0.869 | 0.925 | 0.784 | 0.788 | 0.812 | 0.748 | 0.716 | 0.718 | |
| MAML | 0.780 | 0.939 | 0.967 | 0.812 | 0.808 | 0.797 | 0.826 | 0.808 | 0.823 | |
| Ours | 0.829\(_{\small{\pm\text{0.020}}}\) | 0.875\(_{\small{\pm\text{0.049}}}\) | 0.927\(_{\small{\pm\text{0.031}}}\) | 0.828\(_{\small{\pm\text{0.001}}}\) | 0.826\(_{\small{\pm\text{0.001}}}\) | 0.829\(_{\small{\pm\text{0.004}}}\) | 0.868\(_{\small{\pm\text{0.025}}}\) | 0.880\(_{\small{\pm\text{0.036}}}\) | 0.904\(_{\small{\pm\text{0.037}}}\) | |
| GR | Naïve | 0.574 | 0.920 | 0.958 | 0.500 | 0.503 | 0.670 | 0.558 | 0.627 | 0.741 |
| CANMD | 0.669 | 0.935 | 0.965 | 0.744 | 0.742 | 0.737 | 0.582 | 0.632 | 0.729 | |
| ACLR | 0.693 | 0.928 | 0.961 | 0.683 | 0.689 | 0.736 | 0.660 | 0.695 | 0.766 | |
| ProtoNet | 0.720 | 0.639 | 0.757 | 0.672 | 0.664 | 0.608 | 0.736 | 0.756 | 0.804 | |
| MAML | 0.813 | 0.937 | 0.965 | 0.808 | 0.803 | 0.786 | 0.819 | 0.802 | 0.819 | |
| Ours | 0.830\(_{\small{\pm\text{0.062}}}\) | 0.928\(_{\small{\pm\text{0.004}}}\) | 0.960\(_{\small{\pm\text{0.003}}}\) | 0.819\(_{\small{\pm\text{0.012}}}\) | 0.819\(_{\small{\pm\text{0.010}}}\) | 0.823\(_{\small{\pm\text{0.006}}}\) | 0.886\(_{\small{\pm\text{0.035}}}\) | 0.882\(_{\small{\pm\text{0.042}}}\) | 0.902\(_{\small{\pm\text{0.043}}}\) | |
| GC | Naïve | 0.612 | 0.927 | 0.961 | 0.513 | 0.536 | 0.693 | 0.561 | 0.629 | 0.742 |
| CANMD | 0.685 | 0.931 | 0.963 | 0.802 | 0.803 | 0.817 | 0.761 | 0.777 | 0.823 | |
| ACLR | 0.687 | 0.933 | 0.964 | 0.712 | 0.715 | 0.744 | 0.811 | 0.809 | 0.835 | |
| ProtoNet | 0.708 | 0.609 | 0.731 | 0.786 | 0.782 | 0.770 | 0.730 | 0.715 | 0.736 | |
| MAML | 0.816 | 0.926 | 0.959 | 0.813 | 0.809 | 0.801 | 0.826 | 0.810 | 0.826 | |
| Ours | 0.824\(_{\small{\pm\text{0.026}}}\) | 0.918\(_{\small{\pm\text{0.004}}}\) | 0.954\(_{\small{\pm\text{0.002}}}\) | 0.826\(_{\small{\pm\text{0.023}}}\) | 0.826\(_{\small{\pm\text{0.023}}}\) | 0.833\(_{\small{\pm\text{0.023}}}\) | 0.896\(_{\small{\pm\text{0.001}}}\) | 0.907\(_{\small{\pm\text{0.000}}}\) | 0.930\(_{\small{\pm\text{0.000}}}\) | |
| LI | Naïve | 0.640 | 0.926 | 0.960 | 0.516 | 0.538 | 0.693 | 0.558 | 0.626 | 0.741 |
| CANMD | 0.770 | 0.894 | 0.940 | 0.815 | 0.814 | 0.818 | 0.755 | 0.784 | 0.834 | |
| ACLR | 0.766 | 0.938 | 0.966 | 0.756 | 0.760 | 0.786 | 0.805 | 0.793 | 0.814 | |
| ProtoNet | 0.793 | 0.910 | 0.950 | 0.738 | 0.746 | 0.788 | 0.599 | 0.576 | 0.581 | |
| MAML | 0.813 | 0.938 | 0.966 | 0.813 | 0.809 | 0.800 | 0.824 | 0.807 | 0.824 | |
| Ours | 0.815\(_{\small{\pm\text{0.031}}}\) | 0.910\(_{\small{\pm\text{0.014}}}\) | 0.949\(_{\small{\pm\text{0.008}}}\) | 0.820\(_{\small{\pm\text{0.008}}}\) | 0.820\(_{\small{\pm\text{0.006}}}\) | 0.828\(_{\small{\pm\text{0.002}}}\) | 0.873\(_{\small{\pm\text{0.026}}}\) | 0.883\(_{\small{\pm\text{0.036}}}\) | 0.906\(_{\small{\pm\text{0.038}}}\) | |
| PH | Naïve | 0.622 | 0.929 | 0.962 | 0.502 | 0.526 | 0.688 | 0.558 | 0.627 | 0.742 |
| CANMD | 0.531 | 0.938 | 0.967 | 0.559 | 0.565 | 0.624 | 0.653 | 0.676 | 0.740 | |
| ACLR | 0.709 | 0.939 | 0.967 | 0.716 | 0.719 | 0.746 | 0.733 | 0.754 | 0.804 | |
| ProtoNet | 0.721 | 0.780 | 0.867 | 0.693 | 0.686 | 0.644 | 0.628 | 0.635 | 0.687 | |
| MAML | 0.780 | 0.939 | 0.967 | 0.816 | 0.812 | 0.802 | 0.819 | 0.805 | 0.823 | |
| Ours | 0.828\(_{\small{\pm\text{0.028}}}\) | 0.909\(_{\small{\pm\text{0.009}}}\) | 0.949\(_{\small{\pm\text{0.005}}}\) | 0.818\(_{\small{\pm\text{0.012}}}\) | 0.818\(_{\small{\pm\text{0.012}}}\) | 0.828\(_{\small{\pm\text{0.013}}}\) | 0.896\(_{\small{\pm\text{0.016}}}\) | 0.880\(_{\small{\pm\text{0.032}}}\) | 0.902\(_{\small{\pm\text{0.030}}}\) | |
| Avg | Naïve | 0.617 | 0.926 | 0.961 | 0.506 | 0.525 | 0.686 | 0.559 | 0.627 | 0.741 |
| CANMD | 0.656 | 0.923 | 0.958 | 0.721 | 0.721 | 0.736 | 0.680 | 0.710 | 0.775 | |
| ACLR | 0.715 | 0.935 | 0.965 | 0.703 | 0.707 | 0.742 | 0.750 | 0.762 | 0.805 | |
| ProtoNet | 0.739 | 0.761 | 0.846 | 0.735 | 0.733 | 0.724 | 0.688 | 0.679 | 0.705 | |
| MAML | 0.800 | 0.936 | 0.965 | 0.813 | 0.808 | 0.797 | 0.823 | 0.806 | 0.823 | |
| Ours | 0.825\(_{\small{\pm\text{0.033}}}\) | 0.908\(_{\small{\pm\text{0.016}}}\) | 0.948\(_{\small{\pm\text{0.010}}}\) | 0.822\(_{\small{\pm\text{0.011}}}\) | 0.822\(_{\small{\pm\text{0.010}}}\) | 0.828\(_{\small{\pm\text{0.010}}}\) | 0.884\(_{\small{\pm\text{0.021}}}\) | 0.886\(_{\small{\pm\text{0.029}}}\) | 0.909\(_{\small{\pm\text{0.030}}}\) | |
\end{table}
Datasets and Baselines: To examine MetaAdapt performance, we adopt multiple source and target datasets. We follow [7] and adopt FEVER (FE) [67], GettingReal (GR) [68], GossipCop (GC) [69], LIAR (LI) [70] and PHEME (PH) [71] as the source datasets. For the target domain, we adopt CoAID [72], Constraint [73] and ANTiVax [74]. Our naïve baseline leverages few-shot target examples to fine-tune the source pretrained models. We also adopt state-of-the-art baselines from domain adaptation and few-shot learning methods for domain adaptive few-shot misinformation detection: CANMD, ACLR, ProtoNet and MAML [7], [26], [55], [56]. Additionally, we select two large language models LLaMA and Alpaca to evaluate few-shot in-context learning (ICL) and parameter-efficient fine-tuning (PEFT) performance in misinformation detection [6], [75].
Implementation: We follow the preprocessing pipeline as in [7]. Specifically, we translate special symbols (e.g., emojis) back into English, tokenize hashtags, mentions and URLs and remove special characters from the input. We use the 10-shot setting (i.e., \(k = 10\)), the model is trained using AdamW optimizer with \(0.01\) weight decay and no warm-up, where we sample \(3\) source tasks and perform \(3\) updates in inner-loop optimization. Then, we compute the meta loss and evaluate task similarity scores before rescaling the meta gradients with temperature \(\tau\). All our main experiments are repeated 3 times, we select the best model with the validation set for final evaluation. Hyperparameter selection (e.g., inner and outer learning rates, temperature etc.) and implementation details are provided in 8.
5.2 Main Results↩︎
Supervised results: We first report supervised results on all datasets in 1. The upper and lower parts of the table report the source and target dataset performance respectively. We observe the following: (1) overall, the performance on statements and posts achieves better performance than news articles. An example can be found on GossipCop (News) with 0.776 BA, compared to 0.863 on PHEME (Social network posts); (2) for disproportionate label distributions (e.g., CoAID), the BA metric reduces drastically compared to other metrics, indicating the difficulty of training fair models on unfair distributions. For instance, BA is circa \(10\%\) lower than accuracy and F1 on CoAID; (3) the RoBERTa model only achieves an average BA of 0.778 on source datasets, which suggests that transferring knowledge from the source to the target datasets can be a challenging task.
Few-shot adaptation results: The few-shot cross-domain experiments (10-shot) on all source-target combinations are presented in ¿tbl:tab:results?. In the table, rows represent the source datasets while the columns represent the target datasets. We include the adaptation methods in each row, while the metrics are reported in the columns under target datasets. For MetaAdapt, we report the mean results in the table and provide the standard deviation values using the \(\pm\) sign. For convenience, the best results are marked in bold and the second best results are underlined. We observe: (1) adapting to the COVID domain is a non-trivial task upon dissimilar source-target label distributions. In the example of GossipCop \(\rightarrow\) CoAID, baseline methods show BA values to be slightly higher than 0.6 (despite high accuracy and F1), suggesting that the model predicts the majority class with a much higher likelihood; (2) by learning domain-invariant feature representation, baseline methods like CANMD and ACLR can improve the adaptation results compared to the naïve baseline. For instance, CANMD achieves over \(42.3\%\) and \(21.7\%\) average improvements on BA for Constraint and ANTiVax; (3) using the meta adaptation approach, MetaAdapt significantly outperform all baselines in the BA metric. For instance, MetaAdapt outperforms the best-performing baseline MAML \(7.4\%\) in BA on ANTiVax, with similar trends to be found in accuracy and F1. Specifically for CoAID (with over \(90\%\) positive labels), improvements on BA demonstrates that MetaAdapt can learn fair features for improved detection results despite slightly worse accuracy and F1 results. In summary, the results in ¿tbl:tab:results? show that MetaAdapt is particularly effective in adapting early misinformation detection systems using limited target examples. In contrast to the baseline models, MetaAdapt can achieve significant improvements on all metrics across source-target dataset combinations. In the case of large domain discrepancy, MetaAdapt demonstrates superior performance by exploiting the few-shot target examples with second-order dynamics and similarity-based adaptive learning.
| Setting | Dataset | BA \(\uparrow\) | Acc. \(\uparrow\) | F1 \(\uparrow\) |
|---|---|---|---|---|
| LLaMA-ICL | CoAID | 0.500 | 0.906 | 0.951 |
| Constraint | 0.500 | 0.523 | 0.687 | |
| ANTiVax | 0.500 | 0.664 | 0.798 | |
| 2-5 | CoAID | 0.515 | 0.908 | 0.952 |
| Constraint | 0.537 | 0.559 | 0.704 | |
| ANTiVax | 0.528 | 0.681 | 0.806 | |
| LLaMA-FT | CoAID | 0.749 | 0.874 | 0.928 |
| Constraint | 0.724 | 0.721 | 0.718 | |
| ANTiVax | 0.742 | 0.756 | 0.811 | |
| 2-5 | CoAID | 0.766 | 0.818 | 0.892 |
| Constraint | 0.688 | 0.686 | 0.689 | |
| ANTiVax | 0.767 | 0.779 | 0.828 | |
| CoAID | 0.825 | 0.908 | 0.948 | |
| Constraint | 0.822 | 0.822 | 0.828 | |
| ANTiVax | 0.884 | 0.886 | 0.909 |
Comparison to large language models: To further demonstrate the effectiveness of MetaAdapt in domain adaptive misinformation detection, we compare our MetaAdapt with state-of-the-art large language models (LLMs). In particular, we adopt LLaMA-7B and Alpaca-7B and perform both few-shot in-context learning (i.e., ICL) and parameter-efficient fine-tuning (i.e., FT) on target datasets, with results presented in 2. We notice: (1) despite the significant increase in model parameters (from \(\sim\)0.1B to 7B), LLMs can still fail to distinguish misinformation without further tuning. For instance, LLaMA consistently scores 0.5 in BA by predicting the majority class in in-context learning. (2) Fine-tuning LLMs can significantly improve the performance in misinformation detection. With PEFT tuning, Alpaca achieves \(40.7\%\) performance improvement in BA across target datasets conditioned only on the few-shot target examples. (3) With the proposed MetaAdapt, smaller language models like RoBERTa are capable of outperforming fine-tuned large language models. On average, MetaAdapt outperforms LLaMA-PEFT and Alpaca-PEFT by \(14.3\%\) and \(14.1\%\) in the BA metric on target datasets. The results suggest that MetaAdapt is both effective and efficient in early misinformation detection by combining out-of-domain knowledge and meta adaptation.
| Setting | Dataset | BA \(\uparrow\) | Acc. \(\uparrow\) | F1 \(\uparrow\) |
|---|---|---|---|---|
| 0-shot | CoAID | 0.551 | 0.868 | 0.925 |
| Constraint | 0.567 | 0.585 | 0.705 | |
| ANTiVax | 0.528 | 0.590 | 0.689 | |
| 2-5 | CoAID | 0.594 | 0.450 | 0.588 |
| Constraint | 0.664 | 0.662 | 0.656 | |
| ANTiVax | 0.627 | 0.616 | 0.645 | |
| 2-5 | CoAID | 0.728 | 0.904 | 0.949 |
| Constraint | 0.799 | 0.796 | 0.792 | |
| ANTiVax | 0.700 | 0.721 | 0.776 | |
| 2-5 | CoAID | 0.825 | 0.908 | 0.948 |
| Constraint | 0.822 | 0.822 | 0.828 | |
| ANTiVax | 0.884 | 0.886 | 0.909 | |
| 2-5 | CoAID | 0.876 | 0.938 | 0.964 |
| Constraint | 0.844 | 0.844 | 0.870 | |
| ANTiVax | 0.865 | 0.850 | 0.872 |
| CoAID (2020) | Constraint (2021) | ANTiVax (2022) | |||||||
|---|---|---|---|---|---|---|---|---|---|
| 2-10 Metric | BA \(\uparrow\) | Acc. \(\uparrow\) | F1 \(\uparrow\) | BA \(\uparrow\) | Acc. \(\uparrow\) | F1 \(\uparrow\) | BA \(\uparrow\) | Acc. \(\uparrow\) | F1 \(\uparrow\) |
| 0.825 | 0.908 | 0.948 | 0.822 | 0.822 | 0.828 | 0.884 | 0.886 | 0.909 | |
| w/o Task Similarity | 0.811 | 0.909 | 0.948 | 0.807 | 0.805 | 0.806 | 0.862 | 0.853 | 0.880 |
| w/o Adaptive LR | 0.801 | 0.919 | 0.954 | 0.811 | 0.808 | 0.801 | 0.860 | 0.860 | 0.885 |
| w/o 2nd-order Grads | 0.789 | 0.928 | 0.960 | 0.800 | 0.794 | 0.774 | 0.843 | 0.843 | 0.871 |
Robustness study: We also evaluate the sensitivity of MetaAdapt with respect to the number of few-shot examples. In particular, we select the number from 0 to 15 and use MetaAdapt to perform the adaptation. The results are averaged across the source datasets and reported in 3. Note that for 0-shot experiments, we train the models on source datasets and directly evaluate on target data. We observe the following: (1) as expected, the adaptation performance improves rapidly with increasing number of few-shot examples. (2) Surprisingly, we observe performance drops in accuracy and F1 on CoAID when increasing the few-shot number from 0 to 1. This suggests that the large domain discrepancy between both domains may reduce the effectiveness of MetaAdapt; (3) Overall, the magnitude of improvements grows rapidly at first and then slowly plateaus as we increase the few-shot number. The largest improvements can be found from 1-shot to 5-shot setting, where the BA results improve by \(18.1\%\) on average. In sum, we observe that even limited number of target examples can improve the target domain performance with MetaAdapt. We provide additional hyperparameters sensitivity analysis in 9.
Ablation study: We evaluate the effectiveness of the proposed components in MetaAdapt. In particular, we remove the proposed similarity-based gradients rescaling (w/o Task Similarity), adaptive learning rate (w/o Adaptive LR) and second-order optimization method (w/o 2nd-order Grads) in order and observe the performance changes. Experiment results on target datasets are averaged across the source datasets and are reported in 4. For all components, we observe performance drops on BA when removed from MetaAdapt2. For example, the performance of MetaAdapt reduces by \(2.0\%\) and \(2.3\%\) in BA when we remove task similarity and adaptive LR consecutively. We further replace MetaAdapt with first-order approximation, which results in a performance drop of \(3.9\%\) in the BA metric. Overall, the results suggest that the proposed components are effective for domain adaptive few-shot misinformation detection.
6 Conclusion↩︎
In this paper, we explore meta learning in domain adaptive few-shot misinformation detection. We propose MetaAdapt, a meta learning-based approach for cross-domain few-shot adaptation. MetaAdapt is the first to leverage few-shot target examples for exploiting the source domain knowledge under the guidance of limited target examples. Experiment results demonstrate the effectiveness of our method by achieving promising results on multiple challenging source-target dataset combinations, where MetaAdapt can significant outperform the state-of-the-art baselines and large language models by a significant margin.
7 Limitations↩︎
Despite introducing meta learning for domain adaptive few-shot misinformation detection, we have not discussed the setting of cross-domain adaptation with multiple source datasets to further improve the performance for identifying early-stage misinformation. Due to the lack of early-stage misinformation data, we limit our choice of the target domain to COVID-19, which may hinder the generalization of the proposed method to other domains. Additionally, the proposed method does not leverage efficient approximation or first-order meta learning methods to reduce the computational costs in training MetaAdapt. As such, we plan to explore multi-source few-shot misinformation detection via efficient meta learning as future work.
8 Implementation↩︎
8.1 Datasets↩︎
We adopt FEVER [67], GettingReal [68], GossipCop [69], LIAR [70] and PHEME [71] as the source datasets. For the target domain, we adopt three COVID datasets: ANTiVax (2022) [74], CoAID (2020) [72] and Constraint (2021) [73].
In the following, we present the details of our source and target datasets:
FEVER is a publicly available dataset for fact verification. FEVER consists of modified claims from Wikipedia without knowledge of the claims [67].
GettingReal is a fake news dataset from Kaggle (Getting Real about Fake News). GettingReal contains text and metadata scraped from online resources [68].
GossipCop provides a fake news data from news content, social network, and dynamic iteratcions. GossipCop is adopted from the FakeNewsNet dataset [69].
LIAR is a publicly available dataset of fact verification. The provided statements are collected from politifact.com with analysis report and links to sources [70].
PHEME contains tweets rumours and non-rumours in certain breaking events (e.g., Germanwings crash). PHEME provides online interactions and structure of the tweets [71].
CoAID is a dataset with COVID-19 misinformation. CoAID provides fake news on websites and social platforms as well as user iteractions under such sources [72].
Constraint is a shared taks on COVID-19 fake news detection. It contains over 10k annotated social media posts and articles of real and fake news on COVID-19 [73].
ANTiVax is a novel dataset with over 15k COVID-19 vaccine-related tweets and annotations for vaccine misinformation detection [74].
Details of the above datasets are provided in 5, where Neg. and Pos. are the proportion of misinformation and valid information in the dataset (i.e., label distribution). Len represents the average token length of text and Content denotes the source type of the text (e.g., statement, news or social posts). Notice that CoAID is largely imbalanced with over \(90\%\) positive examples.
| Datasets | Neg. | Pos. | Len | Content |
|---|---|---|---|---|
| FEVER | 29.6% | 70.4% | 9.4 | Statement |
| GettingReal | 8.8% | 91.2% | 738.9 | News |
| GossipCop | 24.2% | 75.8% | 712.9 | News |
| LIAR | 44.2% | 55.8% | 20.2 | Statement |
| PHEME | 34.0% | 66.0% | 21.5 | Social Network |
| CoAID | 9.7% | 90.3% | 54.0 | News / Statement |
| Constraint | 47.7% | 52.3% | 32.7 | Social Network |
| ANTiVax | 38.3 | 61.7% | 26.2 | Social Network |
8.2 Baseline Methods↩︎
As a naïve baseline, we pretrained the misinformation detection model on the source dataset and fine-tune with the few-shot examples, followed by evaluation on the test set. We additionally adopt the following state-of-the-art baselines for domain adaptive few-shot misinformation detection:
Contrastive Adaptation Network for Misinformation Detection (CANMD) proposes to use label correction in the pseudo-labeling process to generate labeled target examples. Then, contrastive learning is applied to learn domain-invariant and class-separating features. Therefore, we adapt CANMD by including the few-shot target examples in the training process as in our setting, we select the best results from the original CANMD and our adaptation [7].
Adversarial Contrastive Learning for low-resource rumor detection (ACLR) leverages language alignment and contrastive learning to improve corss-domain misinformation detection performance. ACLR also introduces adversarial augmentation to enhance the robustness of few-shot rumor detection. We replace the original graph convolution networks with our base transformer model for content-based misinformation detection [26].
Prototypical Networks for Few-shot Learning (ProtoNet) designs prototypical networks for few-shot classification problems. ProtoNet projects sample features to a metric space and perform inference by computing distances to prototypes of each class. We adapt ProtoNet to meta adaptation framework by adopting the same label space and the base transformer model as encoder for domain-adaptive few-shot misinformation detection [55].
Model-Agnostic Meta-Learning (MAML) is the first to leverage second-order derivatives for few-shot learning. MAML first update model parameters upon sampled tasks, followed by computing the meta loss and derive the second-order gradients w.r.t. the original parameters. Similarly, We adapt MAML to our meta adaptation framework with the homogeneous label space acorss tasks and the transformer encoder for misinformation detection [56].
8.3 Implementation Details↩︎
For our evaluation method, we follow the previous works [7], [15], [20] and split the datasets into training, validation, and test sets with the ratio of 7:2:1. If the dataset provides a default split, we directly use the provided split sets. The validation sets are used for label correction in [7] and constructing few-shot examples and saving the best model in training. For our few-shot adaptation setting, we select the first \(k\) examples (10-shot as default) from the original validation set and used the remaining examples for validation. We use accuracy and F1 score for evaluation. The balanced accuracy (BA) is additionally introduced to evaluate the adaptation performance in both classes equally, BA is defined as the mean of sensitivity and specificity.
For the naïve baseline, we train the base model on the source dataset using AdamW optimizer (learning rate 1e-5) without warm-up. Then, the model is fine-tuned on the few-shot examples under the same training condition. For other baseline methods, we use the original implementation (if provided) and follow the original hyperparameter configuration. Otherwise we reimplement the baseline methods and adopt the identical training pipeline (AdamW with 1e-5 learning rate and no warm-up). Few-shot learning baselines (i.e., ProtoNet & MAML) are adapted to the domain adaptive few-shot learning framework by using the few-shot target examples as query set. For the large language models, in-context learning is performed using the generative and perplexity-based approach based on [76], while fine-tuning is performed using low-rank adaptation as in [77]. In the MetaAdapt experiments, we adopt \(3\) as the number of tasks \(n\) and update the task specific model \(\boldsymbol{\phi}\) for \(3\) times in inner-level optimization, notice for each task we start from the original parameter set \(\boldsymbol{\theta}\). To train MetaAdapt, we use a training batch size of \(4\) in each task and the learning rates (both \(\alpha\) and \(\beta\)) are selected from \([1e-5, 2e-5, 3e-5]\). The temperature hyperparameter is selected from \([1, 0.1, 0.01]\) (See sensitivity analysis in 9). We perform training with \(500\) or \(1000\) meta iterations and validate the model every \(50\) iterations, the best model is used for evaluation on the test sets.
9 Additional Results↩︎
Sensitivity Analysis of Initial Learning Rates: We study the sensitivity of \(\alpha\) and \(\beta\) on the CoAID dataset, the best results (averaged across source datasets) are presented in 4. Overall, we observe that the performance of MetaAdapt is insensitive to the changes of both learning rates. Interestingly, we notice that accuracy values are negatively correlated with the BA scores, suggesting that accuracy may not be an ideal metric for data distributions with disproportionate label classes.
Sensitivity Analysis of Temperature Values: We also study the sensitivity of \(\tau\) in gradient rescaling on CoAID, we present the best results (averaged across source datasets) in 5. In short, the performance of MetaAdapt first increases and then starts to reduce, with the best results ocurring at \(\tau = 0.01\), indicating the effectiveness of similarity-based gradients rescaling. Overall, the proposed MetaAdapt is robust to the hyperparameters and outperforms the baseline methods consistently.