1 Introduction↩︎

2 Introduction↩︎

The development of smns enables fast dissemination of critical information, large-scale discussions, and joint actions about political and social issues because smns connect people[1]. However, the capabilities of smns can be misused by ios, especially state-sponsored ones. ios are organized attempts to tamper with the regular flow of information and influence public opinion through disinformation, hate speech, and other harmful content. ios are hard to detect because they are always initiated by a small group of users[2], [3]. With targets including narrative manipulation and the fostering of division in online and real-world communities, IOs have been identified as a significant threat to democracy, and the need for robust methods to detect these operations is urgent[4].

Researchers have proposed io detection approaches to identify whether individual posts or specific users are related to an io. The following discussion focuses on the detection of io users, which is the primary interest of this study. io users are motivated or incentivized to promote ios, while legitimate organic users are called control users. io user detection approaches use patterns within user post timelines on smns. These patterns can be categorized as behavioral patterns (specifically, temporal behavioral patterns because they describe the user activities on smns over time) and language patterns, including speaking style and areas of interest.

io user detection is challenging. The biggest challenge is the generalization capability of io user detection algorithms, i.e.., their ability to detect unseen ios. In the wild, ios evolve quickly, so existing labeled IO datasets always lag. This means that existing supervised[1], [5], [6] and semi-supervised[7]–[10] io user detection methods suffer from a generalization issue, i.e.., they cannot detect new, unseen ios. Recent zero-shot llm approaches[11] provide an unsupervised alternative, but their ability to capture complex temporal behavioral dynamics is still limited.

This paper formulates io user detection as an unsupervised anomaly detection problem and proposes a novel approach, called model, to use behavioral and language patterns from user post timelines for unsupervised io user detection. Training anomaly detection models on io data is difficult because the training data, which are supposed to consist of timelines of control users, are contaminated by io users. This harms the performance of io user detection models. To mitigate this issue, we use language patterns. Specifically, first, we train a tpp on the contaminated data to identify io users based on abnormal behavioral patterns, such as coordinated behavior among different io accounts, which are commonly observed among io users but rare among control users. A tpp is a well-defined stochastic process over temporal event sequences[12]. Researchers have used the process to achieve state-of-the-art performance in unsupervised identification of abnormal event sequences from normal ones[13] and outlier events from normal events[14]. Second, we propose a novel evidence function to adjust tpp inference. This function converts llm responses, which are generated from user post timelines, into quantitative scores that refine the tpp output for better io user detection. Experimental results show that model outperforms other baselines by a significant margin on five real-world io datasets.

3 Related Work↩︎

4 Related Work↩︎

Most existing io detection studies classify io users based on pure behavioral patterns[1], [6], pure language patterns[5], [7], [15]–[17], or both[8], [10], [11]. Vargas et al.[10] classify io users based on coordination behaviors, such as tweeting the same content within a timeframe, or tweeting the same hashtag within a timeframe, extracted from users’ post timelines. Luceri et al.[11] investigate io user detection for llms under few-shot and fine-tuning settings. Minici et al.[8] propose IOHunter, an io detector built upon a graph, in which the similarity of users’ behavioral traces, including tweets, hashtags, and time of posting, justifies connections between users. Despite reported good performance on existing data, supervised and semi-supervised io user detection methods require labelled io datasets, which are limited in size and lag behind real-world io practices. This raises concerns about the generalization capabilities of supervised and semi-supervised io user-detection algorithms in real-world scenarios.

Recent zero-shot llm approaches[11] provide an unsupervised alternative, but their ability to capture complex temporal behavioral dynamics is still limited. Although not directly aimed at io user detection, other studies concerning io users verify coordination among users in the same io[18], [19]. Although such methods reveal that io users tend to form small and relatively well-separated clusters compared with legitimate organic users, our study shows that it is an oversimplification to develop unsupervised io user detection methods solely based on this observation.

Anomaly Detection using contaminated data: Anomaly detection using contaminated data is possible if the anomalies occupy a small portion of the data[20]. Most existing methods employ unsupervised identification of anomalies during training, allowing the model to either exclude these samples or leverage them to mitigate performance degradation[21]–[23]. Patra et al.[24] observed that these methods require prior knowledge about the dataset, such as the contamination ratio, which is usually unknown. To solve these issues, they proposed ephad, the first framework to adjust the prediction of an anomaly detector trained on contaminated data using an evidence function during test time. Patra et al.[24] focus on anomaly detection on visual data, but how to adapt ephad to multimodal temporal and language data remains an open question.

llms for Social Media Analysis: llm-based social media analysis has been widely investigated, including post annotation[25]–[27], misinformation detection and mitigation[28]–[31], and io detection[11]. Unlike [11], which uses llms as the sole classifier based on behavioral and language patterns, model leverages an llm to provide additional evidence to enhance the tpp-based anomaly detector.

5 Problem Definition↩︎

6 Problem Definition↩︎

We formulate io user detection as an unsupervised anomaly detection problem. A smn contains io users, i.e.., users motivated or incentivized to promote an io, and control users, i.e.., normal genuine social media users. The number of io users is significantly smaller than that of normal users, and their behavioral and language patterns are noticeably different, as reflected in their post timeline. The post timeline of user \(i\) is \(\boldsymbol{s}_i = (s_{i,1}, s_{i,2}, \cdots, s_{i,M_i})\), where \(M_i\) is the number of posts by this user. Each post event \(s_{i,k} = (t_{i,k}, c_{i,k})\) consists of a time \(t_{i,k}\) when this post is created and post content \(c_{i,k}\). Post timelines of all users, including normal and io users, are denoted as \(\boldsymbol{S}=\{\boldsymbol{s}_1, \boldsymbol{s}_2, \cdots, \boldsymbol{s}_n\}\). For later use, we denote the respective sequences of all timestamps and contents as \(\boldsymbol{T} = \{\boldsymbol{t}_1, \boldsymbol{t}_2, \dots, \boldsymbol{t}_n\}\) and \(\boldsymbol{C} = \{\boldsymbol{c}_1, \boldsymbol{c}_2, \dots, \boldsymbol{c}_n\}\), where \(\boldsymbol{t}_i = (t_{i,1}, t_{i,2}, \cdots, t_{i,M_i})\) and \(\boldsymbol{c}_i = (c_{i,1}, c_{i,2}, \cdots, c_{i,M_i})\) are sequences of timestamps and contents of user \(i\). We train a model \(\mathcal{M}\) on \(\boldsymbol{S}\) to identify io users from control users. io users are labeled 0, and control users are labeled 1, formulated as follows: \[y = \begin{cases} 0, & \mathcal{M}(\boldsymbol{t}_i, \boldsymbol{c}_i) \geqslant\epsilon \\ 1, & \text{otherwise} \end{cases} \label{eqn:definition}\tag{1}\] where \(\epsilon\) is the threshold. We show how we determine the value of \(\epsilon\) in [sec:exp].

7 Methodology↩︎

8 Methodology↩︎

In this section, we introduce model, a novel anomaly detection approach that uses behavioral and language patterns from user action sequences for unsupervised io user detection. model consists of two core modules: tpp and llm. tpp identifies io users using user action sequences, but its performance can be suboptimal because the training set is contaminated by io users. To mitigate this issue, we propose a novel evidence function to convert llm outputs into a score and use ephad[24] to adjust the output of the tpp model using the evidence function. In [sec:train95tpp] and [sec:evidence95func], we provide brief introductions to tpp and ephad. In 8.5, we present the structure and technical details of model.

8.1 tpp model↩︎

8.2 tpp model↩︎

The tpp describes a random process of an event sequence \(\boldsymbol{\tau} = (\tau_1, \tau_2, \cdots, \tau_m)\) where \(\tau_k\) is the occurrence time. This paper considers the simple tpp, which only allows at most one event at any time, thus \(\tau_k<\tau_\ell\) if \(k<\ell\). Given the history up to (exclusive) the current time \(t\), denoted as \(\boldsymbol{\mathcal{H}}\), the conditional intensity function \(\lambda^*(t)\) is the probability that an event will happen at time \(t\)[12]:² \[\label{eqn:def95tpp95intensity} \lambda^*\left(t\right) = \lim_{\Delta t \rightarrow 0}{\dfrac{P\left(\tau \in (t, t+\Delta t]\middle|\boldsymbol{\mathcal{H}}\right)}{\Delta t}}\tag{2}\] With \(\lambda^*(t)\), we can define the joint probability distribution \(p^*(t)\) of the next event at \(t\). \[\label{eqn:tpp95p} p^*\left(t\right) = \lambda^*\left(t\right)\oldexp\left(-\int_{t_l}^{t}{\lambda^*(\tau)\mathrm{d}\tau}\right)\tag{3}\] The nll loss on \(\boldsymbol{\tau}\) observed in a time interval \([t_0,T]\) is: \[\label{eqn:nll95of95tpp} L = -\log p(\boldsymbol{\tau}) = - \sum_{k = 1}^{M}{\log \lambda^*(\tau_k)} + \int_{t_0}^{T}{\lambda^*(u)\mathrm{d}u}\tag{4}\] where \(M\) is the number of events in \(\boldsymbol{\tau}\). 4 is the training loss of tpp models. Recent tpp models are based on neural networks (see [32] for a comprehensive survey). tpp has been used to detect abnormal events or sequences from normal events or event sequences[13], [14], [33].

8.3 ephad↩︎

8.4 ephad↩︎

ad algorithms need a training dataset \(\boldsymbol{D} = \{x_1, x_2, \cdots, x_n\}\) containing normal samples to train an ad classifier \(f(x)\). If \(\boldsymbol{D}\) is contaminated by abnormal data, the trained ad model \(f(x)\) will treat anomalies as normal. Patra et al.[24] observed that ad models trained on contaminated \(\boldsymbol{D}\) can be corrected at test time using a predefined evidence function \(T(x)\). The \(T(x)\) contains external knowledge about anomalies, which is not captured by \(f(x)\). The method is called ephad. Specifically, ephad revises \(f(x)\) using exponential tilting as: \[\hat{f}(x) = \frac{f(x)\oldexp\left(T(x)/\beta\right)}{Z} \label{eqn:ephad}\tag{5}\] where \(\beta > 0\) is the temperature, and \(Z=\int_{D}{f(x)\oldexp\left(T(x)/\beta\right)dx}\) is the normalizing constant. This method is theoretically grounded in Korbak et al.[34]. They treat the Post-Hoc adjustment as a KL-regularized optimization problem, where the objective is to find an adjusted distribution \(\hat{f}(x)\) that minimizes the KL divergence from the original distribution \(f(x)\) while maximizes the expected evidence \(T(x)\), as shown in the following optimization problem: \[\max_{\hat{f}} \mathbb{E}_{x\sim\hat{f}}[T(x)] - \beta D_{\textrm{KL}}(\hat{f} \| f)\] where \(D_{\textrm{KL}}(\hat{f} \| f)\) is the KL-divergence between \(\hat{f}\) and \(f\). They show that 5 is the solution to this optimization problem. Because ad only depends on the relative ordering of samples, the intractable normalizing constant \(Z\) in 5 can be omitted, which simplifies 5 into: \[\hat{f}(x) = f(x)\oldexp\left(T(x)/\beta\right) \label{eqn:ephad95sim}\tag{6}\] This is more practical for implementing ad models. In our approach elaborated in the following section, we use 6 to adjust \(f(x)\) based on tpp models trained on contaminated training data using the evidence function derived from llm outputs for better io user detection.

8.5 model↩︎

In this section, we propose model, which is sketched in 1. model checks whether a user is an io user based on their \(\boldsymbol{t}_i\) and \(\boldsymbol{c}_i\). The abstract input \(x\) in 6 is instantiated as a user \(x_i=(\boldsymbol{t}_i, \boldsymbol{c}_i)\), where \(\boldsymbol{t}_i\) and \(\boldsymbol{c}_i\) represent the sequence of timestamps and contents of user \(i\), respectively. It has two main components. The first component is a frozen tpp model trained on \(\boldsymbol{T}\), which contains both io and control users. The tpp model captures the difference between control and io users using behavior patterns, including the coordinated behavior between io accounts or an excessive number of posts compared with control users. The differences are reflected in the value \(-\frac{1}{M_i} \log p(\boldsymbol{t}_i)\). Thus, \(f(x_i)=\oldexp\left(-\frac{1}{M_i}\log p(\boldsymbol{t}_i)\right)\) is the tpp-based score computed from the timestamp component of \(x_i\). The second component is the averaged evidence function \(\bar{T}_N(\boldsymbol{t}_i, \boldsymbol{c}_i)\), backed by a frozen llm. It determines whether a user is an io user based on behavioral and language patterns in \(\boldsymbol{c}_i\) and \(\boldsymbol{t}_i\). That is, \(T(x_i)=\bar{T}_N(\boldsymbol{t}_i, \boldsymbol{c}_i)\) is the averaged evidence function over \(N\) samples derived from a frozen llm. \(-\frac{1}{M_i} \log p(\boldsymbol{t}_i)\) and \(\bar{T}_N(\boldsymbol{t}_i, \boldsymbol{c}_i)\) are later fused using ephad to mitigate the effect of tpp trained on contaminated \(\boldsymbol{T}\). The outputs of the two components are fused using 6 to mitigate the effect of data contamination in tpp.

model uses tpp models to classify io users using the behavioral patterns collected from \(\boldsymbol{t}_i\). Because there are more control users than io users, tpp models yield higher probabilities \(p(\boldsymbol{t}_i)\) for control users compared to io users. This makes tpp a valid ad classifier: \[y = \begin{cases} 0, & \text{if}\;-\frac{1}{M_i}\log p(\boldsymbol{t}_i) \geqslant \epsilon \\ 1, & \text{otherwise} \end{cases} \label{eqn:ad95mtpp}\tag{7}\] where io users are labeled \(0\), control users are labeled \(1\), \(\epsilon\) is the threshold, and \(M_i\) is the length of \(\boldsymbol{t}_i\).

However, the dataset \(\boldsymbol{S}\) is contaminated by io users, which could harm detection accuracy. To mitigate this, model adjusts \(-\frac{1}{M_i}\log p(\boldsymbol{t}_i)\) using a llm-based evidence function \(T(\boldsymbol{t}_i, \boldsymbol{c}_i)\). The llm takes in the system prompt in 2 and a structured input of \(\boldsymbol{t}_i\) and \(\boldsymbol{c}_i\). It then generates a response \(r_i = \mathrm{LLM}(\boldsymbol{t}_i, \boldsymbol{c}_i)\), indicating whether the user is an io user or a control user by answering “io account” or “Control account”. To convert \(r_i\) into \(T(\boldsymbol{t}_i, \boldsymbol{c}_i)\), we use a semantic similarity mapping strategy instead of enforcing strict text matching, which is prone to failure because llm outputs are non-deterministic. Specifically, we compute the semantic similarity between the raw llm output \(r_i\) and reference texts representing io or control users. In our case, the reference texts are “io account” for io users and “Control account” for control users, denoted as \(a_{io}\) and \(a_{cl}\). By applying a \(\mathtt{softmax}\) transformation to these similarity scores, we obtain a probability distribution over the io and control users. \(T(\boldsymbol{t}_i, \boldsymbol{c}_i)\) is defined as the probability of selecting the io user: \[\label{eqn:evidence95func} T(\boldsymbol{t}_i, \boldsymbol{c}_i) = \frac{\oldexp\left(\mathrm{sim}(a_{io}, \mathrm{LLM}(\boldsymbol{t}_i, \boldsymbol{c}_i))\right)}{\oldexp\left(\mathrm{sim}(a_{cl}, \mathrm{LLM}(\boldsymbol{t}_i, \boldsymbol{c}_i))\right) + \oldexp\left(\mathrm{sim}(a_{io}, \mathrm{LLM}(\boldsymbol{t}_i, \boldsymbol{c}_i))\right)}\tag{8}\] where \(\mathrm{sim}(x, y)\) measures the similarity between input text \(x\) and \(y\). The output of llms may vary across multiple generations with the same input. To ensure robustness, we draw \(N\) samples of \(T(\boldsymbol{t}_i, \boldsymbol{c}_i)\) and compute their mean, denoted as \(\bar{T}_N(\boldsymbol{t}_i, \boldsymbol{c}_i)\): \[\label{eqn:evidence95func95n} \bar{T}_N(\boldsymbol{t}_i, \boldsymbol{c}_i) = \frac{1}{N}\sum_{r=1}^{N}{T^{(r)}(\boldsymbol{t}_i, \boldsymbol{c}_i)}\tag{9}\] where \(T^{(r)}(\boldsymbol{t}_i, \boldsymbol{c}_i)\) refers to the \(r\)-th sample. According to 6 , the decision rule for model is: \[y = \begin{cases} 0, & \text{if}\;\oldexp\left(-\frac{1}{M_i}\log p(\boldsymbol{t}_i)\right)\oldexp\left(\bar{T}_N(\boldsymbol{t}_i, \boldsymbol{c}_i)/\beta\right) \geqslant \epsilon \\ 1, & \text{otherwise} \end{cases} \label{eqn:ad}\tag{10}\]

9 Experiments↩︎

10 Experiments↩︎

This section (i) benchmarks model against existing baselines, (ii) evaluates the impact of behavioral and language patterns on io user detection via ablation studies, (iii) compares ephad against alternative methods for integrating these patterns for io user detection, and (iv) analyzes the sensitivity of model to different llms and temperature values \(\beta\). We run each experiment on A100 and L40S GPUs three times with different random seeds, and their mean and standard deviation (1-sigma) are reported. The computational complexity of model for one user is \(O(M_i)\), because the tpp component analyzes one sequence \(\boldsymbol{s}_i\) in \(O(M_i)\), while the llm component processes one sequence in \(O(1)\)³.

model is supposed to be training-free because all its components are either frozen or deterministic computations. However, to the best of our knowledge, large tpp models pretrained on large event sequence data do not exist, so we must train a tpp model on \(\boldsymbol{T}\) beforehand. model works with any existing tpp model that provides \(p(\boldsymbol{t}_i)\). According to [13], [35], the performance gap between existing state-of-the-art tpp models is small. Without loss of generality, we choose the sahp[36] because of its relatively simple design and good performance. The loss function for training sahp on \(\boldsymbol{T}\) is 4 .

model works with all existing llms. Recently, we have seen the rise of reasoning models[37]. By enabling the llm to think during test time, reasoning models consistently improve performance across various tasks, especially for solving mathematical problems and difficult logic problems[38]. In this paper, we use an open-weight reasoning model gpt (reasoning model, 117B parameters with 5.1B active parameters)[39] with a medium reasoning effort. In [sec:exp4], we report model’s performance with llama (non-reasoning model, 70B parameters), qwen (reasoning model, 80B parameters with 3B active parameters)[40], glm (reasoning model, 106B parameters with 12B active parameters)[41], and mistral (non-reasoning model, 24B parameters)⁴. As for the similarity function in 8 , we use the bge-m3-v2 reranker[42]. The \(N\) in 9 is 5.

model has one hyperparameter: temperature \(\beta\). According to Patra et al.[24], we set \(\beta=0.5\) during the experiments. In [sec:exp4], we investigate how temperature affects the detection performance of model.

10.0.0.1 Baseline Models:

We compare model with three baselines. Although not directly aimed at io user detection, two studies concerning io users verify coordination among users in the same io[18], [19]. These studies motivate the first two baselines.

• kong[18] represents each user via a user embedding, generated by averaging the embeddings of all events for that user extracted from a model trained on \(\mathcal{T}\). Then, these user-level representations are fed into a clustering model to partition the users into two distinct groups. Users in the smaller group are io users.

• nwala[19] suggests describing user behaviors as strings of symbols using formal languages specified by rules. Next, these sequences are converted into user embeddings using a TF-IDF model. Users whose embeddings differ from the majority are considered io users.

• luceri[11] uses llms to detect in a zero-shot setting whether a user is an io user based on behavioral and language patterns. This method first checks whether a post \(x_i\) belongs to an io. Then users whose posts are mostly io posts are labelled as io users.

10.0.0.2 Datasets:

We use the io user dataset collected by Seckin et al.[2]. This dataset contains various identified ios in 26 labelled datasets across 15 countries. We evaluate model and baselines on Egypt, China_1, Iran_1, Russia_1, and UAE. These datasets have the complete activity timelines of all io users, but control users are limited to their last 100 posts on days when they are involved in ios. To mitigate this bias, we apply the same strategy to io users by limiting the number of posts from io users to 100 per day. The data statistics of curated datasets are available in ¿tbl:tab:data95statistic?. We assign 80% of data to the training set, 10% to the validation set, and 10% to the test set. Please note that although these datasets include labels indicating which users are io users, they are used solely for performance evaluations. model and baselines cannot see the labels.

10.0.0.3 Evaluation Metrics:

We evaluate model and baselines using precision, recall, F1-score, auc, and ap⁵. We only report ap for experiments in 10.3, [sec:exp2], and [sec:exp4] to simplify the comparison of results. We add ap because Davis et al.[43] suggest ap as an alternative to auc for tasks with a largely imbalanced class distribution, where auc results can be overly optimistic. The io dataset, as shown in ¿tbl:tab:data95statistic?, is highly imbalanced, with few io users and many control users, which justifies the use of ap.

Computing precision, recall, and F1-score requires a threshold \(\epsilon\) to decide the label \(y\). In this work, the threshold is adjusted by maximizing the F1 on the validation set. In practice, the scores of io and control users are often modeled as two class-conditional normal distributions. The threshold is the score that separates them, that is, most sequences on one side belong to io users and most sequences on the other side belong to control users.

10.1 Comparing model with baselines↩︎

10.2 Comparing model with baselines↩︎

This section compares model with three baselines, kong, nwala, and luceri on five io datasets. The metrics used are precision, recall, F1-score, auc and ap. The results are presented in .

The results show that model is the overall best approach for io user detection, as measured by auc and ap. One outlier is Egypt, where kong performs better on ap. The reason is the temperature \(\beta\). Later results in 10.7.0.2 show that \(\beta=0.5\) from [24] is not the best choice. By lowering the temperature, the overall performance of model across all datasets significantly improves and can outperform kong on all datasets. We do not discuss how to optimize \(\beta\) without any labels in this paper.

We also observe that the results of kong have a large standard deviation across multiple datasets. The reason is the data imbalance with many control users and few io users. As shown in 3, kong on imbalanced data is sensitive to random seeds, leading to the misclassification of control users as io users. This instability results in inconsistent performance, particularly on highly imbalanced datasets such as China_1 and Russia_1.

10.3 The impact of behavioral and language patterns on io user detection↩︎

If we only use the behavioral or language patterns of one user for io user detection, we may get suboptimal results. To demonstrate this, we compare model with the following ablation baselines: (i) model without the evidence function \(\bar{T}_N(\boldsymbol{t}_i, \boldsymbol{c}_i)\), which is described in 7 , (ii) model without tpp, detecting io users by \(\bar{T}_N(\boldsymbol{t}_i, \boldsymbol{c}_i)\), (iii) model but \(\bar{T}_N(\boldsymbol{t}_i, \boldsymbol{c}_i)\) returns random samples from the uniform distribution \(\mathcal{U}(0, 1)\), and (iv) the Random model, which assigns scores sampled from \(\mathcal{U}(0, 1)\) to users. We include Random because ap is sensitive to the ratio of control and io users, whereas the auc remains at 0.5. The results are reported in 6.

The ablation results demonstrate that jointly considering behavioral and language patterns for io user detection is better than considering only one, as the ap of model is significantly better than model without llm or tpp. The experiment results also show that a standalone llm performs poorly at zero-shot io user detection. Another observation is that model outperforms model with random \(\bar{T}_N(\boldsymbol{t}_i, \boldsymbol{c}_i)\), whose performance is even worse than model without llm. This demonstrates that llm still provides useful information from \(\boldsymbol{c}_i\) and \(\boldsymbol{t}_i\) for io user detection despite its poor performance in this task.

10.4 The benefit of llm-backed \(\bar{T}_N(\boldsymbol{t}_i, \boldsymbol{c}_i)\)↩︎

10.5 The benefit of llm-backed \(\bar{T}_N(\boldsymbol{t}_i, \boldsymbol{c}_i)\)↩︎

ephad enables model to jointly consider behavioral and language patterns for io user detection, but is it the overall best way in terms of performance? In this section, we investigate several intuitive and existing alternatives to train a io user detector based on behavioral and language patterns. Specifically, they are: (i) behavior- and language-aware tpp model. The input of tpp is \(\boldsymbol{t}_i\) and \(\boldsymbol{c}_i\). The training loss is \(-\log p(\boldsymbol{t}_i, \boldsymbol{c}_i)\). Then we use 7 for io user detection, (ii) nwala[19], (iii) luceri[11], and (iv) the Random model. The results are reported in 7.

We see that model outperforms other alternatives by a significant margin on all datasets. A detailed analysis shows that nwala performs worse than the Random baseline. A potential reason is that nwala requires more detailed information about posts, such as who the post is sent to and whether it contains a picture, which is missing from the dataset. This results in most generated nwala sequences containing only one symbol referring to a post, which significantly harms their classification capability. luceri and behavior- and language-aware tpp outperform Random but are outperformed by model. On closer inspection, we find that the results of the behavior- and language-aware tpp are basically the same as those of tpp. These results demonstrate that adding language data to the tpp model does not improve its io detection capability.

10.6 Sensitivity to different llms and \(\beta\)↩︎

10.7 Sensitivity to different llms and \(\beta\)↩︎

All reported results of model in previous sections are based on temperature \(\beta=0.5\) inherited from the ephad paper and gpt. However, model works with all existing llms and temperatures \(\beta > 0\). In this section, we evaluate the stability of model under different llms and temperatures.

10.7.0.1 Sensitivity to llms:

8 shows the performance of model with different llms. Besides gpt, we pick llama, qwen, glm, and mistral. We observe that configurations using gpt, llama, and glm consistently outperform model without llm across all datasets. Although qwen and mistral underperform on some datasets because of their smaller model or active parameter sizes, their overall average is still better than model without llm. These results show that model is stable across different llms.

10.7.0.2 Sensitivity to \(\beta\):

4 shows how temperature \(\beta\) affects the performance of model. We observe that although the optimal temperature varies across datasets, it is consistently around 0.3 for gpt. Generally, increasing the temperature from 0.05 brings a quick performance improvement to a maximum, followed by a gradual decline. For China_1, Iran_1, Russia_1, and UAE, the performance drop is relatively small, while for Egypt, the ap drops significantly from over 0.85 to below 0.70. Table 9 compares model at its optimal temperature against the default configuration and other baselines. model consistently outperforms other approaches at the optimal temperature. These results indicate that unsupervised optimization of \(\beta\) is a promising approach to further improve the performance of model. We leave this as future work.

11 Conclusion↩︎

12 Conclusion↩︎

In this study, we addressed the critical challenge of identifying io users within social networks. While traditional supervised models struggle with the evolving nature of io user behaviors, unsupervised models rely on rigid assumptions of coordination, model offers a more resilient alternative by framing detection as a multimodal anomaly problem. model shifts the focus toward the inherent friction between io campaigns and organic user behaviors. By deploying a tpp, we successfully captured the behavioral patterns that distinguish coordinated influence from genuine social interaction. A key contribution of this work is the mitigation of “training set contamination.” By integrating a novel evidence function that translates llm responses, which are generated from users’ post timelines, into quantitative scores, we effectively adjust the output of tpp to mitigate the noise caused by io users embedded in the training data. Experimental results show that model outperforms other baselines by a significant margin on five real-world io datasets.

References↩︎