1 Introduction↩︎

Due to the ubiquity of harmful content on the Internet, content moderation is key to online safety [1]–[7]. Automated content moderation using machine learning (ML) techniques has been previously studied [4], [5], [7]–[9]. The release of ChatGPT by OpenAI led on to the rapid growth of the popularity of large language models (LLMs) among the general public and the interest of potential applications within the research community. Multiple studies have reported findings of using LLMs to perform common social media content moderation tasks, such as detecting harmful content [2], [6], [8]–[10] and enforcing community rules [3], [10], [11]. Although past evaluations yielded mixed results in the case of community rule compliance checks, they were unequivocally positive for harmful content detection: Proprietary LLMs surpassed existing state-of-the-art commercial tools in terms of accuracy [2], [6], [10].

In addition to the rapid development of proprietary LLMs, open-weight LLMs (LLMs whose model weights are open, hence can be fine-tuned and deployed locally at users’ will), have also gained traction [12]. Locally-hosted LLMs are suitable in areas where there are privacy concerns [13], [14] (e.g. medical applications [15]). However, prior work had reported that, unlike proprietary LLMs, open-weight LLMs that could run on consumer-grade devices either were inadequate for harmful content detection [10], or required fine-tuning to achieve satisfiable accuracy [9]. Recently, latest open-weight models have been released with reasoning ability [16], [17], but their applicability for content moderation has not been studied. So we asked this research question: Are open-weight LLMs now as good as proprietary ones out-of-the-box at detecting harmful content on social media?

To answer the question, we started by building a ground-truth dataset. We collected real-world posts from Bluesky [18] and moderation decisions made by the Bluesky Moderation Service (BMS). We sampled three categories (namely rude, intolerant, and threat) of rule-violating posts and randomly sampled posts not flagged by BMS. For each of the posts sampled, two authors independently annotated whether or not it meets the moderation criteria set by BMS [19]. (More in Section 3.1.)

Next, for evaluation, we selected seven state-of-the-art LLMs. Three of them are open-weight reasoning LLMs that can each fit into a single consumer-grade GPU card with 24-GB VRAM; three of them are proprietary reasoning LLMs; and we pick one non-reasoning proprietary LLM as a contrast. Applying the same moderation criteria given by BMS, we prompted the LLMs to perform content moderation, i.e. annotate the posts the same way as the two author annotators did. (More in Section 3.3.)

Finally, we processed the model outputs and analyzed the data. We observe that treating the annotations made by human (BMS and the two author annotators) as ground truth, open-weight LLMs and proprietary LLMs exhibit overlapping accuracy (81%–97% and 72%–98% for sensitivity, and 91%–100% and 93%–99% for specificity). Interestingly, generally speaking, the rude test yields a higher specificity than sensitivity, while the opposite is true for the intolerant and threat tests. (More in Section 4.)

Our results show that open-weight reasoning LLMs can detect harmful content on social media as accurately as proprietary models, even without fine-tuning. We discuss the practical implications, limitations, and directions for future work in Section 5.

Key contributions↩︎

The key contributions of our work are as follows:

1. New labeled dataset of real-world social media content moderation. We contributed a new labeled dataset of real-world social media posts on Bluesky combining official moderation decisions by Bluesky Moderation Service with independent author annotations.

2. New understanding of the out-of-the-box competence of open-weight LLMs at zero-shot harmful content detection. We show that without the need to fine-tune, state-of-the-art open-weight reasoning LLMs (that can fit in the VRAM of a single consumer-grade GPU card) have comparable accuracy (both sensitivity and specificity) to larger proprietary LLMs in terms of zero-shot text-based harmful content detection, overturning the findings of previous studies [9], [10].

3. Attribute-specific performance patterns that motivate standardization. Accuracy is not uniform across harm categories. On posts unanimously labeled by humans, rudeness detection shows higher sensitivity than specificity (about 95%–98% vs. 91%–96%), whereas intolerance and threats show near-perfect specificity (about 98%–100%) but markedly lower sensitivity. These distinct error profiles arise from differences in category definitions and review criteria. This finding underscores the need for attribute-specific taxonomies, illustrative examples, and calibrated decision thresholds to reduce label ambiguity and align moderation behavior with community norms and user preferences.

2 Background and Related Work↩︎

Social media content moderation has been an active area of research. From traditional machine learning (ML) classifiers to large language models (LLMs), many studies have investigated automated content moderation systems. In this section, we provide the necessary context and summarize prior studies that inspire our research question.

Perspective API [4], [20] is a moderation service developed and provided by Google Jigsaw. Widely adopted by online platforms and extensively utilized by the research community [10], [21], Perspective API’s ML model takes text as input, and produces scores between 0 and 1 for attributes such as toxicity, identity attack, threat, etc.

[8] [8] showed that LLMs (GPT-3.5-turbo) can match and often outperform ML models such as BERT and RoBERTa in detecting hate speech, leveraging five existing benchmarks of diverse hate speech categories. [6] [6] introduced HateGuard, a framework that employs a custom chain-of-thought [22] prompting strategy to utilize LLMs for newfangled hate speech detection. In comparison to existing methods employing ML models, HateGuard makes adaptation to new hate terms faster and has higher F1 scores (90%–99% using GPT-4) in detecting various new forms of hate speech on Twitter.

[10] [10] systematically investigated content moderation using LLMs. The authors evaluated five LLMs (GPT-3, GPT-3.5, GPT-4, Gemini Pro, and LLAMA 2) on rule-based moderation and toxicity detection. For rule-based moderation, they collected 95 subreddit community rules and for each subreddit, 500 comments removed by moderators (sourced from a prior study) and 500 not removed (queried from Reddit API); whereas for toxic content detection, they sampled 5000 toxic comments and 5000 non toxic ones from a dataset with comments collected from Reddit, Twitter, and 4chan and annotated in a prior study. They found that proprietary LLMs achieve comparable performance to human moderators in rule-based moderation tasks and outperform Perspective API significantly in toxic content detection tasks (with accuracy spanning from 70% to 73% and sensitivity 73% to 86%). However, they reported that (with an accuracy of 63% and a precision of 59%,) LLAMA 2, the open-weight LLM, is incapable of performing toxic content detection.

Around the same period, [11] [11] also explored using LLMs to identify community rule violations on Reddit. The authors created a dataset with 144 rule-violating posts (36 manually selected from Reddit, 47 written by the authors, and 61 generated by GPT-3.5) and 600 (popular hence assumed to be) rule-passing posts (collected from Reddit API). However, they reported that while the GPT-3.5 has a high specificity (92.3%), it has a low sensitivity (43.1%).

In a similar theme, [3] [3] looked into automated community rule compliance checking for decentralized social networks. The authors collected community rules from 508 Mastodon [23] servers, and for each server, collected 100 posts using the API of Mastodon. They utilized six open-weight LLMs (gemma-2-9b-it, Llama-3.1-8B-Instruct, Mistral-7B-Instruct-v0.3, Mistral-Nemo-Instruct-2407, Neural-Chat-v3-3, and Qwen2-7B-Instruct) to grade community rule compliance and generate justification and suggestion for improvement. They reported positive results from qualitative and quantitative evaluations.

More recently, [9] [9] also explored the performance of open-weight LLMs for community-specific content moderation tasks. The authors collected 5000 removed and 5000 not-removed Reddit comments (, from each of 15 subreddits), sampled 8000 for finetuning and 2000 for evaluation. Using the LoRA [24] technique, they fine-tuned LLMs with fewer than 15B parameters (Llama-3.1-8b, Gemma-2-9b, and Mistral-nemo-instruct), and evaluated them against one larger open-weight LLM (Cohere’s Command R+) and two proprietary LLMs (GPT-4o and GPT-4o-mini). They found that the accuracy (72.5%–77.87%) of fine-tuned smaller LLMs consistently exceeds that of the larger LLMs (by an average of 11.5%). They also reported that larger LLMs have higher precision than recall, and smaller LLMs the opposite.

3 Methodology↩︎

3.1 Data collection↩︎

Prior studies often relied on Twitter (now X) as a major source of real-world data [2], [6], [10]. After Twitter restricted public content access in 2023, we sought an alternative data source. We selected Bluesky [18], a Twitter-like microblogging platform that remains publicly accessible and has a relatively active user base (cf. Mastodon). we decided to turn to Bluesky [18], a microblogging social media platform that resembles Twitter, allows public access, and has a relatively large active user base (cf Mastodon).

Bluesky, perhaps due to its young age, was designed with moderation in mind. Bluesky’s stackable approach to moderation¹ means that everyone can become a moderation service provider, and that everyone can subscribe to their choice of moderation services. For the research community, this means that in addition to access to content (posts and comments) and interactions (follows, likes, and reposts) on Bluesky, we also have access to (previous difficult to obtain) moderator decisions.

3.1.1 Posts on Bluesky↩︎

Unlike traditional centralized social media platforms, where user-generated content are managed and distributed by the platforms, Bluesky users keep their data (including posts, likes, follows, etc.) on Personal Data Servers (PDS) of their choosing (or host their own PDSes), and Bluesky’s indexing infrastructure (i.e. Bluesky Relay and App View) subscribes to known PDSes and distributes the content generated by users. This AT Protocol design implies that anyone can access all posts authored by known users via the authors’ PDSes.

We obtained a list of known Bluesky accounts utilizing Skyfall [25], an open-source toolkit written in Go, which sends com.atproto.sync.listRepos queries to the Bluesky Relay at https://bsky.network. To obtain a list of known Bluesky posts we referenced Skyfall and developed a specialized toolkit called Ouranos². For each unique account, Ouranos requests all posts authored by the account by sending app.bsky.feed.getAuthorFeed queries to Bluesky’s index server at https://public.api.bsky.app. Note that to minimize the impact of our data collection process, instead of querying each user’s PDS server, we chose to query Bluesky’s index server, which employs content delivery network (CDN) for caching.

We took a snapshot of posts on Bluesky from the 12th to the 24th day of August, 2025. We discovered about 1.6 × 10⁹ posts from around 2.5 × 10⁷ unique accounts³. Due to limited resources, We randomly sampled 4% of accounts (1012840 accounts), which authored 6.6 × 10⁷ posts, for analysis. Since we focus on text-based content moderation in this study, we filtered out posts with multimedia content. To obtain a more consistent distribution within the confined dataset, we limited the scope of our dataset to root-level posts(, i.e. excluding replies to other posts), and filtered out posts that include mentions or hyperlinks, and posts that are not exclusively in English. The remaining are 4339221 text-only root posts in English, which are from 141506 unique accounts.

3.1.2 Labels by Bluesky Moderation Service↩︎

In addition to posts, we collected public moderation decisions (labels) published by Bluesky Moderation Service (BMS), the official moderation service operated by Bluesky. By subscribing to BMS’s labeler endpoint at wss://mod.bsky.app/xrpc. we collected more than 22 × 10⁶ labels, which were published between April, 2023 and October, 2025.

We selected three labels most-applicable to our study: rude, intolerant, and threat. Although BMS issues 67 unique types of labels⁴, we chose the most commonly seen label values in our sampled dataset. For the 4339221 text-only root posts in English, the top-5 are intolerant (202), !hide (29), threat (33), rude (26), and !warn (5). We excluded label values !hide and !warn from our study because BMS does not provide descriptions for them⁵. We then utilized Ouranos to expand our sampled dataset by sampling posts with the labels of interest from the full dataset until we have at least 520 posts for each label.

One challenge we faced is whether the BMS labels were AI-generated or ground-truth labels created by human moderators⁶. To answer this question, we calculated the Labeler Reaction Latency [29]. Figure 1 shows the labeler reaction latency, i.e. time difference between when a post was created⁷ and when BMS labeled the post. More than 80% of posts were labeled at least one day after the posts were created. Therefore, we believe the labels are authentic human moderation decisions, since it is unlikely that the entire labeling pipeline is automated while the latency is long and inconsistent.

3.2 Data annotation↩︎

In part to inspect whether Bluesky Moderation Service (BMS) can be treated as ground truth, in part to establish a baseline of agreement between human moderators, two of the authors independently annotated two sets of posts, namely 520 posts with rude labels and 786 random posts without any label. The two annotators reviewed and labeled the posts according to BMS’s description of the rude label⁸ [19].

3.3 Model selection and configuration↩︎

3.3.1 Models↩︎

We tested seven state-of-the-art large language models (LLMs) in our experiments. Three of them (gpt-oss-20b [30], NVIDIA-Nemotron-Nano-9B-v2 [31], and Qwen3-30B-A3B-Thinking-2507 [32]) are open-weight reasoning LLMs that can each fit into our Nvidia GeForce RTX 3090 GPU card with 24-GB VRAM⁹; three of them (Gemini 2.5 Pro Preview 06-05 [34], GPT-5 [35], and Grok 4 [36]) are proprietary reasoning LLMs; and we pick one non-reasoning proprietary LLM (GPT-4o [37]) as a contrast. We chose these models because they are top on multiple LLM leaderboards [38], [39], representing the most-advanced publicly-available models.

We used Microsoft Azure as the model API provider for GPT-5, GPT-4o, and Grok 4; and Google for Gemini. As for the open-weight LLMs, we used Amazon Bedrock for gpt-oss-20b and run the other two locally (using vLLM as the inference and serving engine) as they are not available on any of the three providers.

We relied on the OpenAI API [40] across LLM providers to prevent inconsistencies between implementations. For deterministic and reproducible results, we fixed the random seed to 350 and set the temperature to 0 where supported¹⁰.

3.3.2 Prompt↩︎

Researchers have reported empirical evidence that large language model (LLM) performance can depend on how the prompts, i.e. initial model inputs, are designed [41]. Prompt engineering has since become an area of focus, with studies from the academia conducted [42] and guidelines from the industry released.

Following the GPT-5 prompting guide [43], we took label descriptions given by BMS¹¹ [19] and prompted¹² GPT-5 to generate the following template:

For each post, we put the text into the {post_text} field, and gave the prompt to the models. we used the descriptions of intolerance and threat given by BMS for the respective experiments.

4 Results↩︎

4.1 LLM Output interpretation↩︎

We describe the output from LLMs and our interpretation on different content and format of output from large language models (LLMs) in this subsection.

1. Valid outputs. Although we specified in the prompt template the expected format of responses (i.e. either yes or no), the responses from the LLMs sometimes included additional formatting or text while still implying the correct binary answer. Table 1 lists all the responses we have seen and their respective number of occurrence.

2. Outputs exceed token limit. To avoid infinite loops in LLM outputs, we limit the maximum length of output to 10000 tokens. This includes the tokens the reasoning LLMs generated in their thinking process. Table 2 lists the number of posts to which each model’s responses exceeded the limit. Note that we only observed such results for the open-weight models. We excluded these samples in the analysis in the following subsections.

3. Input Prompts flagged by LLM provider content filters. LLM API providers, including Microsoft Azure [44] and Google [45], employ content filters to detect harmful inputs and outputs. Table 3 shows the number of posts flagged by LLM providers’ content filter. Since the filtering objective aligns with the nature of the content moderation task in our experiments, we interpreted these samples as positive, i.e. LLM providers’ refusals were treated as if the LLMs outputed yes.

4.2 Inter-rater agreement↩︎

We use inter-rater agreement (Rand accuracy) matrices to show the pairwise agreement ratio between any two moderators. For example, in Figure 2 (a), the cell at the intersection of the bsky column and the gpt-5 row shows that both Bluesky Moderation Service (BMS) and GPT-5 labeled 85% of the posts the same (i.e. both labeled as rude or both labeled as not rude).

Overall accuracy. Figure 2 shows the overall Rand accuracy for rude, intolerant, and threat tests. Treating BMS as ground truth, the accuracy of LLMs (84%–98%) is significantly higher than previously reported (71%–73%) in [10]. Figure 2 (a) demonstrates that, in terms of rudeness detection, that accuracy of LLMs (84%–91%) is within the agreement range between human moderators (83%–92%). We observe that the accuracy of open-weight LLMs meets, and in some cases even exceeds, that of proprietary LLMs. We can also see that none of the LLMs has the highest accuracy across the three tests. In addition, we found that LLM accuracies for rude are statistically different from those for intolerant and threat.

Sensitivity and specificity. Figure 3 shows the inter-rater agreement matrices of the three tests on posts labeled unanimously by BMS and the two authors. The first three sub-figures look at posts regarded as rule-violating by all three human moderators, while the last three look at posts none annotated as rule-violating. We notice that for rude, LLMs have higher sensitivity (true positive rate; 95%–98%) over specificity (true negative rate; 91%–96%). While for intolerant and threat, all LLMs have almost perfect specificity (98%–100%), but besides GPT-5 and gpt-oss-20b, the sensitivity is substantially lower.

5 Conclusion↩︎

In summary, using real-world posts on Bluesky with labels from the Bluesky Moderation Service and independent human annotations, we find that modern open-weight reasoning LLMs running on a single consumer-grade GPU match the zero-shot harmful-content detection performance of leading proprietary models, with overlapping sensitivity (81%–97% vs. 72%–98%) and specificity (91%–100% vs. 93%–99%). Inter-rater analyses show LLM-human agreement on par with human-human agreement, though no single model is best across all categories. Importantly, accuracy depends on target attributes: rudeness, intolerance, and threats exhibit different balances between sensitivity and specificity, underscoring the need for clearer, attribute-specific taxonomies and guidelines. These results indicate that open-weight models can enable privacy-preserving, on-device moderation and support both platform-scale and user-personalized moderation stacks.

Limitations and Future work↩︎

We focused on text-based root posts in English in this work. Plus, due to limited resources, we relied on sampling and did not scan all the posts collected. To name a few numbers, in the 1.6 × 10⁹ Bluesky posts we have collected, only 3306 posts were analyzed. We expect future work to extend to modalities other than text (e.g. image, audio, and video) and languages beyond English. Future work should also look into contextualized moderation. For instance, when investigating multimedia content in a post, consider posts the text content; when exploring a reply, include not only the parent post, but also the the whole discussion thread; or (to be extreme) even related discussions. Ultimately, our goal is to conduct a full scan of Bluesky.

Acknowledgements↩︎

Use of Generative AI↩︎

GPT-5 was used in the preparation of the title of this manuscript. Two titles created by the authors and the abstract were provided in the prompt. The authors selected the final title out of suggestions generated.

6 Ethics↩︎

We collected posts and labels from Bluesky using the APIs and endpoints recommended by Bluesky (https://docs.bsky.app/docs/advanced-guides/api-directory). We honored the rate limits imposed by Bluesky (https://docs.bsky.app/docs/advanced-guides/rate-limits). No actions were taken that would exceed the volume that Bluesky expects of a typical user or developer.

We collected posts from Bluesky’s indexing infrastructure, which aggregates data from the PDSes (either hosted by Bluesky PBC, a third party, or the users themselves) where individual users store their data. We assume the users are aware that their data on Bluesky are publically available, and hence, we assume that the posts are not of privacy concerns. Although the data we collected are made public by the users, we acknowledge that the users might not be aware that their public data might be used for research purposes. That being said, we limited the potential impact we might make to the users. For example, we limit the amount of information transmitted to the LLM providers. Only the text field of the posts are included in the prompt sent to the LLMs. Avatars, display names, or handles are excluded. We did not reproduce, distribute, or display any of the posts collected.

7 Miscellaneous↩︎

7.1 Label descriptions given by Bluesky Moderation Service↩︎

• Rude: Rude or impolite, including crude language and disrespectful comments, without constructive purpose.

• Intolerance: Discrimination against protected groups.

• Threats: Promotes violence or harm towards others, including threats, incitement, or advocacy of harm.

7.2 Prompt used to generate the prompt template↩︎

References↩︎