1 Introduction↩︎

The combination of connectivity and anonymity offered by social media inadvertently also provides the perfect channel for wide-spread dissemination of malicious content [1]–[4]. By malicious content, we consider any form of content detrimental to society, and focus on two specific forms: misinformation and hate speech. Mitigating the effect of malicious content requires content moderation, but this is a labour-intensive process that exacts an immense psychological toll on moderators [5], [6]. Consequently, automated detection of malicious content has seen increased academic interest [7] and industry adoption.

Community models for malicious content detection are models that operate on social graphs, i.e., graphs of content and users. They 1) classify content nodes as malicious or not, 2) incorporate information from interacting users in the graph when doing so, and 3) leverage emergent network properties like homophily to boost detection performance [8], [9]. For community modelling on large-scale heterogeneous online communities, Graph Neural Networks (GNNs) are the architecture of choice [10].

While community models for malicious content detection perform very well on benchmark datasets [11], [12], social media platforms continue to grapple with such content. [13] find that high accuracy in malicious content detection is not indicative of trustworthiness in general, as predictions often rely on dataset-specific features. Models also become outdated quickly as online content and communities evolve [14]. [15] find detection models to be brittle to changes in domain or publication date, a finding that [16] corroborate specifically for community models. Finally, [10] conclude that “we [have] no graph benchmark data for fake news detection in the graph learning community” (p. 22), making any claims of state-of-the-art performance difficult to verify.

Evidently, there exists a mismatch in the performance of community models on research datasets and in more realistic application settings. Research datasets are static; they capture a view of the social graph weeks or months after relevant content has been introduced and spread. Current evaluation practices designed on static graphs are effectively transductive [17], i.e., they implicitly assume that no new content or users will be introduced into the social graph, which leads to performance scores that obscure the discussed deficiencies.

In realistic settings, new users and content nodes are constantly added to the social graph, and the topic or style of malicious content often radically changes. This is a property inherent to online content and communities [18], [19]. Thus, a successful community model should be able to rapidly adapt to domain shifts in content. Since labelling is prohibitively expensive, adaptation should occur in a few-shot manner. Furthermore, the community of interacting users also evolves. Initially, only a few users take note of some content, but as it gains traction, more and more users interact. To prevent harm from malicious content, detection must occur before wide-spread dissemination. This requires adaptation from a limited exploration of the social graph. Therefore, inductive evaluation is needed.

In this paper, we seek to more realistically assess the generalisation capabilities of community models for malicious content detection, specifically making the following contributions:

1. We design a novel evaluation setup based on a few-shot subgraph sampling procedure that test inductive generalisation. The subgraphs are local, contain limited context, and have only a few labels.

2. We test a state-of-the-art community model under this novel evaluation setup on unseen graphs, domains, and tasks. We find them lacking the capacity to generalise.

3. We show that graph meta-learners trained with our few-shot sampling outperform standard community models in inductive evaluation.

2 Related Work↩︎

2.1 Community Models↩︎

Community models have shown promise on static social graphs. Such models use social graphs to contextualise content by the users that interact with them, phrasing the detection task as node classification. [11], [20] and [14] find that GNNs over heterogeneous user–tweet graphs outperform models using only text or user-based features. [21] show that relational GNNs—which directly model edge relations between different types of nodes—yield significant improvements over a range of baselines. [22] argue for the inclusion of publishers as another node type, with [23] also including topics.

[24] utilise temporal replies to model the dynamic user–content interactions. Temporal representations aid in early detection of malicious content [25], [26]. However, they still assume static content. Others have focused on directly detecting actors posting the content [27], [28]; we explicitly exclude actor modelling from our methodology since it mandates different ethical considerations [29].

[12] provide a review of graph representation learning for malicious content detection. They also conclude that cross-domain generalisation remains an understudied problem for graph-based malicious content detection.

2.2 Generalisable Content-only Models↩︎

Developing malicious content detectors for cross-domain generalisation is receiving increased attention. For example, many task-aware domain adaptation approaches have been proposed [30]–[34]. These methods are either “aware” of the distribution of representations in different datasets, or use external models to correct representations post-hoc. Generating representations that are invariant to domain shifts is a related direction [35], [36].

Utilising (large) language models on unseen texts has also shown promising results [37]–[39]. [40] use multitask fine-tuning on RoBERTa [41], and show that few-shot adaptation on related, but unseen datasets improves performance over fine-tuning on individual tasks. [42] specifically train content-only misinformation detectors to rapidly adapt. We, however, focus solely on community models for malicious content on social graphs, and to the best of our knowledge, are the first to do so.

2.3 Subgraph Sampling & Meta-learning↩︎

Closely related to the idea of rapid adaptation to new tasks and domains is the field of meta-learning. Herein, models are trained to optimise themselves using a minimal amount of examples. In NLP, this has been investigated for document [43], [44], sentence [45], [46], and token-level tasks [47]. For a thorough review, we refer the reader to [48], [49].

To operationalise meta-learning from subgraphs, [50] propose G-Meta. They assume local subgraphs preserve information of a larger graph, such that training a GNN on relevant subgraphs can induce rapid adaptation from limited context. Other graph meta-learning procedures exist, however, they do not utilise episodes of local subgraphs. We refer the reader to [51] and [52] for a comprehensive review of the field.

3 Datasets & Tasks↩︎

We use three widely-adopted social graph datasets to train and evaluate community models. Social graph datasets are difficult to collect and degrade as users or content are moderated out. The first dataset, GossipCop, is used for pre-training. The other two datasets, CoAID and TwitterHateSpeech, are reserved for evaluating generalisation to unseen graphs. Table [tab:graph95stats] provides some statistics, which are further complemented by Appendix 10. All datasets were rehydrated, i.e., rebuilt through the API, using the Twitter Academic API prior to May 2023. See ‘Redistribution of Twitter Content’.

GossipCop is one of two datasets introduced by [53]. It comprises \(20\)k fact-checked celebrity rumour articles, and around \(500\)k interacting Twitter users. Labels correspond to the (now defunct) GossipCop fact-checking scores, covering a variety of (usually unreliable) publishers. Articles from a single trusted source, E!Online, were included to reduce class imbalance. Users are connected to articles and other users.

CoAID contains articles from the first months of the COVID-19 pandemic, collected by [54]. We omit the “social media” category as most contain short, poorly formatted text. Fake news articles are labelled using a variety of fact-checking websites, whereas truthful news comes from (unverified) mainstream media outlets. After rehydration, this dataset is substantially smaller than when originally devised, with most lost documents corresponding to the fake class. Users are connected to articles and other users.

TwitterHateSpeech differs in task, domain, and relational schema from the other datasets. Document nodes are tweets generated by Twitter users during a few seed events. [55] identified prolific hate speech tweeters, and include their followers and followees in the graph. They manually labelled all tweets as racist, sexist or innocuous (none). Especially racist (\(0.3\)%) users are over-represented, leading to few diverse regions in the graph. User–document connections indicate authorship (as opposed to tweet/retweet interactions). Users are also connected to other users.

4 Methodology↩︎

A social graph, \(\mathcal{G}=(\mathcal{V}, \mathcal{E})\), consists of a set of nodes \(\mathcal{V}\) and a set of edges \(\mathcal{E}\) indicating which nodes are incident to each other. A node’s \(r\)-radius neighbourhood \(\mathcal{N}_{r}(v)\) contains all other nodes that can be reached by paths of length \(r\) (also called ‘hops’, a series of incident nodes), and always includes \(v\).

All datasets considered require modelling two node-types: documents (\(\mathcal{V}^{\text{docs.}}\)) and users (\(\mathcal{V}^{\text{(users)}}\)). Hence, the social graph \(\mathcal{G}\) is heterogeneous. Document nodes \(v\) contain exogenous features \(x_{v}\) (i.e. the content representation), and have target labels \(y_{v}\in\mathcal{Y}\). Documents are only connected to users (\(\forall u\in\mathcal{N}_{1}(v), u\in \mathcal{V}^{\text{(users)}}\)), whereas users may also be connected with other users based on their interactions or relations. Users are not labelled, and have no initial representation.

4.1 Community Modelling↩︎

Community models for malicious content detection classify content nodes in a social graph, taking into account the graph context around them to make the prediction \(f_{\theta}(x_{v};\mathcal{G})\). GNNs, a common representation learning framework, perform this contextualisation using non-linear message-passing schemes.

Some node \(v\), at layer \(l\), has as hidden state an aggregation of the representations in neighbouring nodes at layer \(l-1\). We use Graph Attention Networks (GATs) [56], which employ an additive attention mechanism for the neighbourhood aggregation. Specifically: \[\label{eq:gat95layer} \mathbf{h}_{v}^{l}=\sigma({\textstyle \sum_{u\in\mathcal{N}_{1}(v)}}\alpha_{v,u}\mathbf{W}\mathbf{h}_{u}^{l-1})\tag{1}\] where \(\sigma\) is a non-linear activation function. The attention weights \(\alpha_{v,u}\) are computed as: \[\label{eq:gat95aggregation} \mathtt{softmax}(\sigma(\mathbf{a}^{\intercal}[\mathbf{W}\mathbf{h}_{v}^{l-1}\|\mathbf{W}\mathbf{h}_{u}^{l-1}]))\tag{2}\] where \(\left[\cdot\|\cdot\right]\) is concatenation, while \(\mathbf{a}\) and \(\mathbf{W}\) are the projection matrices. More expressive architectures than GATs exist and have been applied to malicious content detection [12], [21]. Such models, however, often introduce inductive biases specific to the task it seeks to solve. For example, relational attention aids performance but requires the relational schema to be consistent across datasets. Our evaluation and meta-learning setup is model agnostic.

Community modelling can be transductive or inductive. Transductive modelling assumes that the social graph remains static across training and prediction. Inductive modelling, instead, assumes the underlying social graph changes, in terms of content and users.

Herein we differ from the definition common to graph learning applications. Usually, inductive graph learning ‘only’ assumes the nodes in the evaluation graph are unseen, with those nodes coming from the same underlying graph. As argued in the introduction, this does not apply to malicious content detection; the graph has shifted between training and deployment time. True inductive generalisation, therefore, requires generalisation to entirely different graphs. Currently, no graph datasets of malicious content exist that allow testing this manner of generalisation.

Due to the social network changing, inductive community models should not rely on superficial properties, like the content of malicious posts or specific user neighbourhoods, but rather leverage universal network properties. One such property is homophily: the tendency of nodes of a similar class to cluster together. We investigate the presence of homophily (or heterophily) in Appendix 14.

4.2 Few-shot Subgraph Sampling↩︎

A successful community model for malicious content detection should be able to rapidly adapt to the constantly evolving social graph, even when presented with labelled examples.

More formally, a community model, \(f_{\theta}\), should be able to inductively learn to generalise from a limited exploration of a social graph \(\mathcal{G}^{\mathcal{S}}\subset\mathcal{G}^{\prime}\) to make accurate predictions elsewhere \(\mathcal{G}^{\mathcal{Q}}=\mathcal{G}^{\prime}-\mathcal{G}^{\mathcal{S}}\). In commonly-used meta-learning terminology, \(\mathcal{S}\) would denote the support and \(\mathcal{Q}\) the query set.

For malicious content detection specifically, the notion of ‘limited exploration’ implies the following conditions for \(\mathcal{G}^{\mathcal{S}}\):

1. Locality: all sampled document nodes come from the same graph region, due to a similar seed event, topic, or intended audience

2. Limited Context: moderation should precede wide-spread dissemination

3. Few-shot: labelling is expensive, therefore a minimal set of labels is available

Existing subgraph sampling procedures, like G-Meta, violate these conditions, especially ‘locality’. Labelled nodes are sampled independently, i.e., nodes can come from entirely different regions of the graph, which in our case, would imply entirely unrelated forms of content.

To better conform to the listed desiderata, we perform user-centred sampling for generating \(\mathcal{G}^{\mathcal{S}}\). Algorithm 2 presents pseudocode for our sampling approach, which is also depicted graphically in Figure 1. Various graph statistics are provided in Appendix 11.

To ensure locality, we first sample an anchor user and collect the smallest \(r\)-hop neighbourhood that yields \(k\) documents of each class. In Figure 1, the double-circled user represents the anchor. Then, to limit social context, we take random walks from the documents nodes into the subgraph. This process starts from a document node, and is repeated until a maximum number of nodes is reached. Bold arrows in the middle column of Figure 1 show some random walks of length \(3\).

For the training process, only \(k\) document nodes of each class have their labels unmasked in a subgraph. Other document nodes are still allowed in the subgraph, but without labels. This is depicted in the right-most column of Figure 1.

Power-law distributed degrees of nodes is a property of social media networks. This means a few, very active users and their incident document nodes will be present in the majority of subgraphs. This reduces the diversity of support episodes and thus biases generalisability estimates. To reduce the effect of these users during the training process, document nodes are labelled inversely proportional to their frequency across all created subgraphs.

4.3 Gradient-based Meta-learning↩︎

Community models learn a neighbourhood-aware mapping of a content node’s input features to target labels. Community meta-learners, instead, use an initial set of weights to produce community models only after adaptation, i.e. learning a community model from several episodes of \(\mathcal{G}^{\mathcal{S}}\) and \(\mathcal{G}^{\mathcal{Q}}\). By using our few-shot subgraph sampling method to create episodes for meta-training, the community meta-learners are better suited to inductive generalisation.

We focus on a specific subclass of meta-learners, namely, gradient-based meta-learners. Model-Agnostic Meta-Learning (MAML), introduced by [57], is the most popular such learning framework. Its optimisation objective is: \[\label{eq:maml} \underset{\theta^{\text{(meta)}}}{\text{ min }} \mathbb{E}[\mathcal{L}(\mathbf{y}_{\text{Q}}, f_{\theta^{\text{(task)}}_{T_{\text{inner}}}}(\mathbf{x}_{\text{Q}};\mathcal{G}^{\mathcal{Q}}))]\tag{3}\]

The induced update to \(\theta^{\text{(meta)}}\) is called the outer-loop update. The inner-loop occurs during adaptation to the support set, using a pre-defined number of SGD updates, \(t\in \{1, \ldots, T_{\text{inner}}\}\), with gradients \[\label{eq:maml95task95update} \nabla_{\theta^{\text{(task)}}_{t}}\mathcal{L}( \mathbf{y}_{\text{S}},f_{\theta^{\text{(task)}}_{t}}(\mathbf{x}_{\text{S}};\mathcal{G}^{\mathcal{S}}))\tag{4}\]

This bi-level objective encourages the meta-learner’s initial weights, \(\theta^{\text{(meta)}}\), to learn to adapt to new tasks, \(\theta^{\text{(task)}}\), using only \(T_{\text{inner}}\) updates.

4.3.0.1 Prototypical Initialisation

MAML implicitly assumes a new permutation of classes in each episode and re-initialises the task-specific classification head during each outer-loop iteration. Prototypical Networks (ProtoNets) [58] are a non-gradient-based meta-learning alternative that does not utilise classification heads. Instead, support samples are used to form class prototypes \(\mathbf{c}_{y}\): \[\label{eq:prototype} \mathbf{c}_{y}=\dfrac{1}{k}{\textstyle \sum_{\{\mathbf{x}_{v}|y_{v}=y\}}} f_{\theta^{\text{(meta)}}}(\mathbf{x}_{\mathcal{S}};\mathcal{G}^{\mathcal{S}})\tag{5}\] which classify query samples based on their distance to the prototypes. \[\label{eq:prototypical95classification} p(y_{v}|\mathbf{x}_{v})=\text{softmax}(-d(f_{\theta^{\text{(meta)}}}(\mathbf{x}_{v}), \mathbf{c}_{y}))\tag{6}\]

Per [59], if using the Euclidean distance as \(d\), this is equivalent to applying a linear projection \(\mathbf{W}\mathbf{h}+\mathbf{b}\) with initialisation: \[\label{eq:protomaml95initialization} \mathbf{W}=2\mathbf{c}_{y},\quad\mathbf{b}=\|\mathbf{c}_{y}\|^2\tag{7}\]

Using this reformulation, [59] propose ProtoMAML, an approach that extends MAML such that the classification head is now parameterised by Eq. 7 and fully updatable during inner loop adaptation.

4.4 Implementation Details↩︎

Support graphs for episodes are sampled using the few-shot sampling procedure detailed in Sec. 4.2. We use the lowest possible radius \(r\) that satisfies the \(k\)-shot requirement. The maximum number of nodes in the support graph is \(2048\). All classes provide an equal amount of root nodes of the random walk sub-sampling. For meta-training, \(k=4\), and the query graph is generated by sampling a set of independent document nodes along with their \(r=2\) neighbourhood. During (meta-)testing, all non-training nodes are used. Document nodes use the time-pooled averaged token representations from the penultimate RoBERTa [41] layer as initial representations. These are not trained further. Users nodes are initialised to all zeros, making them effectively anonymous and allowing for both transductive and inductive approaches. All models train end-to-end and do not include an auxiliary text-only classifier. Appendix 12 provides all additional modelling hyper-parameters.

Our GNN architecture is adapted from SAFER [21]. It consists of \(2\) ReLU activated GAT layers, each with \(3\) attention heads. These are concatenated together and linearly projected before being fed into a \(2\)-layer MLP. We use dropout node-wise on the initial representations and element-wise on the layer representation and attention weights. We reduce the computational complexity of the GAT layers by merging successive projections in the attention layers [60]. The use of \(2\) GAT layers means document nodes \(v\) have as receptive field \(\mathcal{N}_2(v)\). We optimise our models using AdamW [61], [62].

In total, we experiment with models trained under \(6\) different learning paradigms. The first two (full and subgraphs) are non-episodic baselines, trained transductively on the full graph or inductively on few-shot sampled subgraphs respectively. full mimics the current practice of training transductively without generalisation to unseen graphs in mind. subgraphs makes generalisation feasible and allows us to isolate the contribution of meta-learning.

The last four are graph meta-learners. We use two MAML variants, one with a classification head shared across episodes (maml-lh) and another where the classification head is randomly initialised at each episode (maml-rh). Appendix 13 shows the effect of classifier head resetting on adaptation speed. We also train protonet and protomaml variants to evaluate the effect of prototypical initialisation on the classification head.

MAML-based outer-loop updates (Eq. 3 ) require computing second-order gradients, which is prohibitively expensive. Instead, we use a first-order approximation (foMAML [57]) which usually does not significantly affect performance [63], [64].

5 Experiments and Results↩︎

5.1 Experimental Setup↩︎

We first assess within-dataset generalisation to unseen nodes using \(5\)-fold stratified cross-validation. The folds are strict, with no document appearing in more than one validation set. We only keep the largest connected component. User nodes can appear in multiple folds, but since they are all zero-initialised, they cannot influence nodes in other folds.

Models are trained on GossipCop and then evaluated on all three datasets. On GossipCop, we assess both the non-episodic and episodic models. On other datasets, we assess the episodic models only to ensure a few-shot generalisation setup. Within each episode, support nodes appear in the query graph but do not count towards classification performance metrics. This process is repeated \(256\) times, with summary statistics computed for each of the \(5\)-fold model checkpoints. When aggregating over checkpoints, the inverse-variance weighted mean is used to estimate a common effect size (i.e., a fixed-effect meta-analysis [65]).

To assess classification performance for each class in isolation, we use the F1-score. Matthews Correlation Coefficient (MCC) is used to assess holistically. Recent papers argue for the MCC as an informative metric, relatively robust to class imbalance [66], [67]. MCC values near 0 indicate random performance, values near 1 almost perfect performance, and negative values are worse than random. The Area Under the Precision-Recall curve (AUPR) is a multi-threshold metric, and can compare models on their ability to separate classes. We exclude it for CoAID and TwitterHateSpeech as there is no consistent way to do aggregate it in multi-class settings. Metrics are reported with 90% confidence intervals.

All hyper-parameters used were tuned on GossipCop’s validation sets. The tuning procedure, optimizer, meta-learning and evaluation hyper-parameters are described in Appendix 12.

5.2 GossipCop Results↩︎

Here, we test generalisation on unseen nodes from the same graph. Beyond the non-episodic baselines already described, we have two additional baseline methods on GossipCop. The first, text, is a \(2\) layer MLP on top of the initial document embeddings meant to test the added benefit of a graph inductive bias. The second, user id, classifies test documents based on neighbouring users’ most linked document class in the train split. The already reasonable performance indicates high homophily.

The GAT-based models leverage both text and social features. subgraphs clearly performs worse than full. maml-lh, however, outperforms full even though it is inductive, demonstrating the generalisation power of meta-learning. The lower three rows all include meta-learners which constantly re-initialise the classification head. Their performance is more in line with the non-episodic subgraphs, lagging considerably on fixed threshold metrics. This gap narrows in terms of AUPR, implying the final bias parameter may be to blame.

5.3 Generalisation to Unseen Graphs↩︎

Figure 3 shows the performance of models ported to the other two datasets.

Variance between the different model instances is large, although when aggregated their performance is relatively stable. protomaml proves to be the best model on both datasets, particularly at larger \(k\)-shot values. protonet shows competitive performance on TwitterHateSpeech (especially in lower \(k\)-shot settings), but is considerably worse on CoAID. Prototypical initialisation seems to aid generalisation, mitigating classification head learning. Regardless, meta-learning methods outperform the non-episodic subgraphs model on either dataset, indicating that training for rapid adaptation helps generalisation to new malicious content forms.

Transfer to CoAID from Gossipcop, is essentially a form of domain transfer. As such, we provide the zero-shot performance of the subgraphs model as a baseline value. Despite the similar task definition, adaptation is clearly required for generalisation, as evidenced by the near-random performance of subgraphs in the zero-shot setting, and the aggressive hyperparameters required (see Appendix 15.1.

Table 2 provides F1 scores for protomaml on each class. All-in-all, the highest achieved MCC was \(0.1709\), for protomaml at \(k=8\), corresponding to an F1-Fake of \(0.1841\). While low relative to other F1 scores reported, this should be compared to a class prevalence of  5%.

Relative to random performance, the greatest negative outlier is TwitterHateSpeech’s majority class, ‘None’. One possible explanation is the homophily pattern of TwitterHateSpeech (see Appendix 14). Whereas racist and sexist tweets are primarily homophilic in the query set, a large proportion of innocuous tweets are highly heterophilic; i.e. these are contextualised by users predominantly authoring racist or sexist. The model is therefore more likely to err on those innocuous tweets, as their author shows a proclivity towards hate speech. Here, heterophily serves as noise. This is most likely an artefact of [55]’s data collection process, with prolific racist and sexist authors serving as the anchor around which the rest of the graph is built.

In general, the results here do not correlate with those found in Table 1. Underperformers there show relatively better performance after adaptation to the other datasets. Hinting at overfitting to the GossipCop graph, this aligns with the line of argumentation presented in the introduction: performance on a single, static graph is not indicative of generalisation to emerging malicious content.

5.4 Ablating GossipCop Pre-Training↩︎

To test the effect of GossipCop pre-training on generalisation to other datasets, we ablate protomaml’s pre-trained weights, and repeat the evaluation under re-initialised weights. A comparison in terms of MCC is provided in Table 3. On CoAID, protomaml outperforms protomaml-reset at all \(k\)-shot values. On TwitterHateSpeech, this only happens at the larger \(k\)-shot values, with protomaml’s MCC performance increase outpacing its reset counterpart.

While low, comparing the performance on each class individually (Tables 2 and 13), protomaml is able to increase its performance on all classes simultaneously, whereas protomaml-reset only does so for the racist class, degrading performance on sexist and innocuous documents.

Regardless, the fact that a model trained specifically with generalisation in mind is barely able to outperform one with random weights is striking, and speaks to the inadequacy of GossipCop as an evaluation dataset. Furthermore, near perfect performance on unseen nodes of the pre-training graph does not imply inductive generalisation to new graphs.

6 Conclusion↩︎

This paper proposes a more realistic evaluation setup for community models on malicious content detection. We highlight several properties of evolving social graphs that are especially neglected: expensive labelling, limited context, and emerging content and users. Experiments verified our motivation, with performance on a single, static dataset in a transductive manner bearing little resemblance to performance during few-shot inductive generalisation.

Our proposed few-shot subgraph sampling approach presented in Section 4.2 is tailored to social media graphs and allows generalisation of community models to new networks, domains, and tasks. While standard community models performed poorly, incorporating our sampling procedure in graph meta-learners aided generalisation. Particularly promising are models with prototypical initialisation.

Ultimately, our results suggest that malicious content detection using community models is not ‘solved’, despite some models achieving near perfect evaluation scores. Current evaluation procedures neglect critical properties of malicious content, and models tested under these conditions will not prove useful in realistic deployment settings. This is a regrettable consequence, considering the high-stakes nature of malicious content detection. Much like the trend occurring in the content-only malicious content detection literature (see Section 2.2), we hope this work will lead to similar follow-up work for community models.

An open problem, warranting further investigation, is the application of meta-learning to class imbalanced datasets. Common to malicious content detection, class imbalance imposes a severe penalty on meta-learners that reset their classification heads.

7 Limitations↩︎

While we took care to increase the diversity of the training data (user-centred sampling, distributing the labels, adding dropout throughout models), ultimately, the diversity is limited by the underlying graph dataset. GossipCop is large, but it contains only a single task and a relatively uniform structure. Ideally, multiple, distinct graph datasets would be used in meta-training. However, few such datasets are available. For meta-learning, task diversity might be a critical factor in ‘learning-to-learn’, analogous to data diversity being critical in standard machine-learning setups. As such, our meta-learners are likely operating below capacity.

[10] made the conclusion that no common benchmarks for community models of malicious content are currently in use. This work, despite making a step towards more realistic evaluation of such models, does not improve this situation. The presented performance metrics are in line with related work, but meaningful comparison will only be possible with the publication and adoption of open-access benchmark datasets. This seems an unrealistic short-term aspiration at the time of writing.

8 Ethical Considerations↩︎

Following the work of [29], we take some steps to ensure that our experimental setup addresses the ethical considerations that may arise when modelling users and communities. The authors highlight the following three considerations that apply to our work:

• Personal vs. Population-level trends, i.e., are generalisations being made from personal traits to population-level trends

• Bias in datasets, i.e., is there demographic, comment distribution, or label bias in the dataset(s) being used?

• Purpose, i.e., is the purpose of the modelling being done to classify content as malicious or users and communities too?

In order to tackle the comment distribution bias whereby the majority of documents may belong to a small number of users, we remove users with more than a certain number of documents from the dataset (where possible). Furthermore, to counter the label distribution bias where we only pick documents of a particular class from a specific user, we do user-centred sampling, incorporating the entire neighbourhood of a user in the user-document graph. We initialise all users to the same zero-embedding, ensuring that we do not generalise personal traits to population-level trends. Lastly, in our work, we solely leverage the user-document graph to be able to better classify the documents, not the users themselves as malicious, hence having a clear purpose to advance malicious content detection.

9 Motivating Examples↩︎

DISCLAIMER: the chosen examples were taken verbatim from various malicious content corpora. They do not reflect the views of the authors.

As established in Section 1, malicious content and its social context evolves. This can happen quickly, and results in text that is very different from already seen forms of malicious content.

Table 5 shows such change, depicting titles from articles published by low (i.e. those that often publish severely biased or false news) and high credibility news sources, as found in the NELA-GT corpora [69]–[71].

Current models are adept at filtering out malicious content as in their training datasets, but quickly degrade when presented with novel content. For example, Table 5 shows substantial high-level semantic change across the years. Models relying on surface-level features will fail as new events spawn new content.

Currently, no existing social-network malicious content dataset captures this level of evolution. In fact, existing datasets are completely static, implicitly assuming the full network (users, content, and their connections) will be available at inference time. This paper argues that this assumption is a significant reason why malicious content continues to propagate unabated, despite the impressive classification scores reported in earlier work.

To showcase this lack of diversity, Table [tab:gossipcop95example] depicts a similar array of texts, sampled from the GossipCop graph. While a variety of topics are discussed, the overarching subject remains the same. Comparatively, relying on surface level features can already lead to strong classification performance. Simply put, in this dataset, models need not account for evolving content.

In lieu of large, temporally diverse graph datasets, we propose an evaluation framework that approximates these effects through generalisation to new graphs. Requiring adaptation from a minimal set of examples, with limited social context will serve as a much better measure of the inference time performance of community models for malicious content detection. There is little point in good within dataset performance, when unseen content forms are free to cause harm.

Put otherwise, when it comes to malicious content detection, we want models that are able to filter out tomorrow’s hate speech posts and fake news articles, not those seen yesterday.

10 Additional Details on Datasets Used↩︎

Rehydrating many years after the datasets were released, not all documents and users could be recovered. This results in empty, missing or isolated documents. The collected graph datasets are thus subgraphs of the one presented in [53], [54] and [55]. The additional preprocessing steps needed are described here:

1. Tokenization: all found documents were collected and tokenized. Any empty documents, or documents yielding only special tokens, were removed

2. Document-User Interactions: where possible, user and doc-user interactions were collected. One issue with GossipCop, as identified by [53], is the inclusion of bots. These ‘users’ tend to disproportionately interact with documents of a single class, both in terms of volume and proportion. Therefore, following the recommendation made by [21], users sharing more than 30% of the documents of any class were removed. The type of doc-user interactions in TwitterHateSpeech differs, resulting in a very small pool of racist users, so this restriction was relaxed. Documents without any user interactions, were also removed.

3. User-user Interactions: all remaining users and their interaction with other users were parsed at this point. To further reduce the number of bots, the top 1% most active users were removed on GossipCop. Then, again only on GossipCop, to further sparsify the graph, only the top 30k users were kept. Isolated documents were once again removed.

The effect of each filtering step and additional statistics of the dataset graph prior to generating episodic subgraphs, are presented in Table [tab:appendix95dataset95stats].

11 Additional Information on Few-shot Subgraph Sampling↩︎

Algorithm 2 presents the few-shot subgraph sampling pseudocode. Although the models used can only aggregate information from at most an \(r=2\) radius subgraph, the initial graph can be expanded to larger \(r\). This is to ensure at least \(k\)-shot examples of each label is present. For the used graph \(r=5\) usually contains the vast majority of the graph, and would only be needed for extremely sparse areas. In practise, however, the \(k\)-shot was achieved by \(r=3\) in all situations.

The random walk subsampling dramatically reduces the number of nodes and edges present in the subgraph. A similar strategy was employed by GraphSAINT [72], resulting in efficiency improvements for a variety of inductive graph learners. We set our random walk length to 5 for all experiments, preferring fewer document nodes (with smaller walk length requiring more roots to get to the same node budget). An approximate budget of 2048 was used during training and evaluation.

Important statistic on the produced subgraphs, for both the support and query sets, are presented in Tables 7 and [tab:ubs95query95stats].

12 Hyperparameters↩︎

12.1 GossipCop Training↩︎

The set of hyperparameters used for pre-training meta-learners on GossipCop are presented in Table 8. The model size was left constant. Input dimensions were determined by RoBERTa, and set to 768. The GAT attention heads, 3, used an internal dimensionality of 256, and all heads were concatenated afterwards. After GAT processing, representations were fed through a two layer ReLU activated MLP of dimensionality 64, before being classified.

All other hyperparameters were tuned on the validation graphs generated by inductive stratified 5-fold cross validation. Dropout was applied on the internal hidden dimensions. Dropout was applied node-wise on the initial node embeddings, stochastically setting entire nodes to 0. In our case specifically, this essentially means converting documents into users. Attention dropout was applied to the attention weights produced by the GAT layers (Equation 2 ). Alternatively, we also experimented with node masking, but we did not see a big difference in performance. [73] AdamW [61], [62] was the outer-loop optimizer, with only the learning and weight decay terms tuned. The different algorithms required substantially different numbers of gradient update before convergence. Early stopping was used, with patience equal to 10% of the maximum allowed number of steps. Most checkpoints converged well before that point. The learning rate was decayed in a step-wise manner, every 5% of the maximum number of steps, with a minimum learning rate at 0.01 of the initial value.

The inner-loop learning rate saw more variation between the different learning algorithms. maml-lh performs best under minimal adaptation, yielding a single step inner loop with a low learning rate. Resetting the head in each episode, instead, forces adaptation, reflected in a larger, more aggressive inner loop. protomaml, finally, reaches minimum validation loss only with large amounts of inner-loop adaptation.

During generalisation to unseen graphs, a more aggressive adaptation strategy was applied to most models. This was not tuned on the test set. Instead, the highest values possible were, such that no infinities appeared in the output logits.

12.2 Additional Training Details↩︎

All experiments were conducted on a Linux-based SLURM-based academic cluster. Nodes consisted of am Intel Xeon Platinum 8360Y CPU with 18 cores in user at 2.4 GHz, a single NVIDIA A100 GPU accelerator (yielding 40 GiB of HMB2 memory) and 128 GiB of DDR4 memory. The code is written exclusively using Python 3.10.6, PyTorch 1.13.0, built with CUDA 11.7. Graph modelling utilized PyTorch Geometric 2.3.0. All experiments were conducted under random seed 942. For local development we use Ubuntu 20.04.6 LTS (GNU/Linux 5.15.90.1-microsoft-standard-WSL2 x86_64).

13 Comparing maml-lh & maml-rh↩︎

As originally described in [57], MAML resets its classification head each episode in order to adapt to a new task with new labels. However, in our case, pre-training MAML uses only a single task, with a fixed label definition. Therefore, the classification head can be learned in the outer loop along with the other meta-initialized models. This setting we dubbed maml-lh (learned head), and has the benefit of requiring less inner-loop adaptation; at least the classifier head is already task-specific. The standard MAML setup we dubbed maml-rh (random head).

On GossipCop, at least, this had a dramatic effect on the degree of adaptation, and as a result, on the final performance scores. This can be seen in Figure 4, with the top row of figures giving various losses for maml-lh, and the bottom row for maml-rh. The left column of figures present the loss on the support set (prior to adaptation), along with the loss on the query set (post adaptation), on both the train and validation splits. maml-lh acts like a standard machine learnig model. After random performance initially, train loss decreases steadily on both graphs, whereas validation loss stagnates earlier on. The query loss is lower, but as we’re using foMAML with a disjoint support–query split this is to be expected (the model is never directly optimised on the support graphs). maml-rh, on the other hand, shows rapid divergence in the support loss, while the support loss decreases as usual.

These loss patterns indicate a distinction between the two operating modes of MAML trained models. maml-lh, seeing a stable learning objective, learns to initialise using representations optimal to all tasks. In the meta-learning literature this corresponds to ‘feature reuse’, and makes maml-lh similar to the ANIL [74] MAML variant. maml-rh, on the other hand, has to leverage the support set to rapdily adapt in order to achieve non-random performance on the query set. Its initial weights are not usable for representation learning, but rather for optimizing itself into a representation learner. This phenomenon is called ‘rapid learning’. This make maml-rh more reminiscent of ‘true’ MAML, or the BOIL variant [75].

This is made more clear in the right column of Figure 4. Here the support loss before and after adaptation is shown, with a green line also indicating the relative decrease. For maml-lh, there is barely any difference between the two, but the overall line is already relatively low; it simply does not need to adapt to achieve generalisble representations. maml-rh, again, is a polar opposite, with diverging initial support loss, but a much lower final loss. The relative improvement is indicative of a model that ‘learns-to-learn’.

In order to test which meta-learning property is more important for the task at hand, both were trained and applied. While maml-lh clearly showed itself superior on GossipCop, we initially thought that added bias toward rapid adaptation might aid generalisation with maml-rh.

14 Homophily↩︎

Graph homophily is the property of nodes to prefer attaching to similar nodes. In social network users, where interaction usually denotes some form of kinship, this follows naturally from people’s social relationships. In malicious content detection, this is likely a relevant feature as well. Our perception of real and fake news is influenced by our social network neighbourhood, and propagation usually occurs in homophilic settings [76].

The effect of homophily on GNN performance remains an open question. In a homophilic setting, node representations will be built from nodes of the same class, whereas in heterophilic settings, node representations contains representations of nodes of different classes. A third setting, less explored, is the case of randomness: nodes are just as likely to attach to nodes of a different class as its own. [77] define a homophily metric that measures the global propensity of links between similar nodes. They show that GNNs can fail in heterophilic settings, with an MLP being more effective, despite the graph setting. For a graph \(\mathcal{G}=(\mathcal{V}, \mathcal{E})\), they define homophily as, \[h^{\text{(edge)}}=\dfrac{|\{(u,v)| (u,v)\in\mathcal{E}\vee y_{u}=y_{v}\}|}{|\mathcal{E}|},\] i.e., the ratio of edges from nodes to similarly labelled nodes, to all edges.

Later papers dispute the claim that GNNs cannot perform under heterophily. [8] find that GNNs require certain conditions to be met for class separation (used as a proxy for classification performance). Specifically, they indicate that as long as nodes of the same label share similar neighbourhood patterns, node representations will become more similar, despite dissimilar neighbouring nodes.

[78] take issue with the definition of the homophily metric. Graphs with many node labels will naturally be less homophilic. They propose a metric that measures homophily while correcting for a randomly connected null model where nodes. Extending to neighbourhoods of radius \(r\), \[\begin{align} h^{\text{(class insen.)}}_{r}=&\frac{1}{|C|-1}\sum_{c=1}^{|C|}\lfloor h_{c}^{\text{(neigh.)}}-p_{c}\rfloor_{+}, \addtocounter{equation}{1}\label{eq:excess95edge95homophily} \\ h_{r,c}^{\text{(neigh.)}}&=\frac{\sum_{v\in \mathcal{V}_{c}}|\{u|\mathcal{N}_{r}(v)\wedge y_{u}=y_{v}\}|}{\sum_{v\in \mathcal{V}_{c}}|\{u|\mathcal{N}_{r}(v)\}|}, \\ &p_{c}=\frac{|\mathcal{V}_{c}|}{|\mathcal{V}|}. \end{align}\tag{8}\] It may be interpreted as measuring the expected excess homophily present in neighbourhoods about nodes of class \(c\).

The proposed metrics for measuring homophily summarize whole graphs, make no distinction between user or document nodes, and do differentiate between a randomly connected graph or a heterophilic graph. In the proposed method of this paper, a homophily metric must be comparable across many subgraphs. As such, inspired by the measure of assortativity, introduced by [79], we slightly modify the homophily definition as, \[\begin{align} \hat{h}^{\text{(subgraph)}}_{r,c}&=\frac{1}{|\mathcal{V}_{c}|}\sum_{v\in\mathcal{V}_{c}}\frac{h_{r,c}^{\text{(neigh.)}}(v)-p_c}{1-p_c}, \addtocounter{equation}{1}\label{eq:rel95excess95homophily}\\ h_{r,c}^{\text{(neigh.)}}(v)&=\frac{|\{u|\mathcal{N}^{\text{(docs)}}_{r}(v)\wedge y_{u}=y_{v}\}|}{|\{u|\mathcal{N}^{\text{(docs)}}_{r}(v)\}|}. \end{align}\tag{9}\] For a subgraph, it defines the homophily of class \(c\) as the expected ratio of homophilic nodes in the \(r\) radius neighbourhood of nodes \(v\in\mathcal{V}_{c}\), in excess of a random graph. The use of a neighbourhood to compute \(h_{r,c}^{\text{(neigh.)}}\) is deliberate. It can now measure the effect of other document nodes on the representation of the centre node. The division by \(1-p_{c}\) normalizes the excess: a score of 1 is achieved only if fully homophilic, 0 if random, and \(-\dfrac{p_{c}}{1-p_{c}}\) if perfectly heterophilic. This allows for interpreting homophily on a scale.

More important, Equation 9 is applicable for support graphs (which have multiple nodes of each label), and query graphs (which have a single node from a single label). For the support graph, nodes’ homophily scores are averaged, per class, to produce a single summary statistic. For the query graphs, the non-labelled nodes scores are simply omitted.

Rather than presenting a single summary metric, the distribution of homophilic nodes can be observed for sampled subgraphs. This is presented in Figure 5. The left column presents the distribution of homophily scores for support graphs, right the query graphs. The rows present the different datasets.

For GossipCop, the support graphs show a relatively broad distribution of homophily scores, with modes between the 0-0.5 range. The query graph, however, shows a significant difference between the two classes, with the real documents being highly homophilic, and fake less so. In other words, there are fake documents whose neighbourhood consists primarily of real documents. CoAID shows more consistent behaviour, with both node classes being extremely homophilic.

TwitterHateSpeech is somewhat of an outlier, with substantial differences between the classes. Again, the collection procedure used by [55] led to a small number of extremely active, racist users. Furthermore, user-document links represent authorship, not tweet/re-tweet interactions. As a result, document representations are built up entirely out of the other posts by the same user. As a result, the racist class has a narrow distribution, that indicates slight homophily. The innocuous class, ‘None’, gives a bimodal distribution in the query graphs. Many innocuous documents seem to be produced by regular Twitter users, whereas a large portion come from racist and sexist users (a large bump in the heterophilic range). Only the sexist class seems to be consistently homophilic. As a result, this dataset is relatively noisy, with representations being influenced by dissimilar documents.

15 Extended Generalisation Results↩︎

15.1 CoAID↩︎

The results presented in Figure 3 are presented in tabular form in Tables 9 and 10. Since CoAID matches, approximately, the pre-training task used in GossipCop, models were initially adaptated using the same classification head (in case of subgraphs and maml-lh) and inner-loop learning parameters. This corresponds to the low-adaptation setting, presented in Table 9. This setting estimates direct domain transfer, much like testing the Subgraphs model at \(k=0\). Performance proved disappointing for most models, with the most aggressive adapter (protomaml), clearly exceeding all other tested models.

Therefore, we conducted a second round of experiments with similarly aggressive inner-loop learning, presented in Table 10. The only models exempted, were subgraphs at \(k=0\) and the protonet, as neither adapts. All models benefited from the more aggressive inner loop, indicating that the generalisation is not trivial. All-in-all, the highest achieved MCC was \(0.1709\), for protomaml at \(k=8\), corresponding to an F1-Fake of \(0.1841\). While low relative to other F1 scores reported, this should be compared to a class prevalence of  5%.

15.2 TwitterHateSpeech↩︎

Similarly, the TwitterHateSpeech results presented in Figure 3 are presented in tabular form in Table 11. Having learnt from CoAID, only the high-adaptation hyperparameters were used. Lower \(k\)-shot values see the ‘None’ class dominate in terms of classification scores. Performance on the minority classes increased with larger \(k\)-shot values, at the cost of reduced innocuous tweet performance. Ultimately, ProtoMAML manages this trade-off best, with gradually increasing MCC scores.

15.3 Ablating GossipCop Pre-Training↩︎

Tables 12 and 13 show additional results pertaining to the ablation experiment described in Section 5.4. The largest addition, is the inclusion of the other type of GBML algorithm, MAML. The comparison model used is maml-rh.

On CoAID, maml-reset yields ‘always positive’ models, giving constant MCC scores of 0. ProtoMAML, however, proves reasonably robust, with protomaml-reset performance that exceeds trained maml-rh. The same effect also holds on TwitterHateSpeech, with protomaml only overcoming its reset counterpart in the larger \(k\)-shot settings.

15.4 Extreme k-shot↩︎

To test the capacity of the meta-learners, a limited extension of the TwitterHateSpeech experiment was conducted. Instead of limiting ourselves to \(k=16\) examples, we increased to \(k=256\). Only protonet was used. Results are depicted graphically in Figure 6 and given numerically in Table 14.

We fully expect diminishing returns. Our graph setting implies that the \(k\) labelled nodes are already present in the support graph, just with its label masked. Unmasking additional labels should provide little additional information to the model; a good graph learner can already infer the masked labels. In fact, as discussed in Section 5.3, under heterophily, one might expect the additional labels to provide additional noise for the innocuous class.

Precisely this can be observed in Table 14. While the MCC score does increase steadily, it comes at the cost of reduced F1 in the ‘None’ class, and stagnation in the ‘Racism’ class. The only class that sees improvements at very high \(k\)-shot values, is the homophilic ‘Sexist’ class.

References↩︎