July 13, 2026
Content moderation practices and governance paradigms are changing rapidly, as fewer human moderators are deployed as ‘experts’ by social media companies in a centralized manner. Instead, the companies are focusing more on community approaches, relying on volunteers to provide accurate information and make correct decisions. In decentralized moderation, communities have always relied on volunteers, updated community guidelines, and internal discussions thereof. For both content moderation paradigms, Artificial Intelligence (AI) seems like it could help ease moderation burdens of time, mental health, and accuracy. One possible way to operationalize AI in content moderation is a “policy-as-prompt” approach, where the policy is formulated as a natural-language prompt and then passed to a large language model (LLM). This model then aids in moderation tasks. In this paper, we briefly lay out the technical and governance properties of this approach, and argue that its limitations lead to specific risks and harms that have to be addressed. Towards alleviating them, we lay out multiple considerations towards more effective prompt governance, but ultimately find that writing prompts alone is not appropriate for ensuring meaningful community governance.
Artificial intelligence has had a large impact on the content moderation landscape. Machine learning classifiers are now the first line of moderation on almost all major social media platforms, such as TikTok, Facebook, Instagram, or X/Twitter, while human moderation has been scaled back or removed from many of them. In modern moderation, humans mostly act as reviewers of edge cases, appeals reviewers, and (increasingly) training data labellers. [1]–[4] This is in part due to a shift of trust and safety policies being politically contested [5], as X/Twitter and Facebook/Instagram (Meta), for example, have reduced moderation enforcement in the name of free speech [6], [7].
Large Language Models and Large Multimodal Models (LLMs & MLLMs) are increasingly proposed and trialed for tasks related to content moderation, promising to alleviate pressures on time and mental health while outperforming human moderators [8], [9]. For example, Meta’s Llama Guard [10] takes a safety taxonomy as an input and then labels posts as ‘safe’ or ‘unsafe’. More recently, OpenAI published an open-weight model gpt-oss-safeguard [11], where developers supply a written policy and the model reasons over it to produce a labelled and explained decision on a post.
Due to scaling (and monetary hurdles), biases, and performance drops in non-English text, these models are not yet widely independently deployed [9], [12]–[14]. As models get better, these problems are expected to become negligible. LLMs would then swiftly replace multiple tasks in content moderation workflows. Still, one underlying problem would persist: simply giving an LLM a policy or a set of rules as a prompt alone cannot govern its behavior absolutely [15], and will have implications on content moderation practice and governance. Therefore, in this paper, we explore the implications of setting moderation policies through natural language prompts given to AI systems on community governance.
Content moderation has never been just about the application of a set of textual rules: it is also a community governance practice [5], [16], [17]. Moderators deliberate and discuss edge cases; formalized appeals processes determine how community members can seek recourse. Additionally, mechanisms to ensure meaningful transparency of implemented rules are vital [18], [19]. The process of moderation also needs to be able to adapt to the evolution of norms in the moderated space, leaving room for changing definitions, contexts, and understandings to emerge [12]. These cannot be described by one centralized way of knowing, as the same words (can) carry different meaning across communities and contexts: “Words change depending on who speaks them; there is no cure”. [20]
These sense-making operations have historically been facilitated through human processes, but with the advent of generative AI systems paradigms in moderation, practices like deliberation by multiple moderators over specific edge cases could be outsourced. Therefore, depending on how LLM-based moderation is implemented, it risks disempowering communities by destructing their influence and recourse options on moderation practices.
The seemingly most accessible approach to integrating LLMs into moderation is called “policy-as-prompt”. The moderation policy itself is encoded as a natural-language instruction [8]. This prompt is directly supplied as a system instruction [21] to a general-purpose LLM, which is asked to apply it to a given piece of content. Policy-as-prompt is appealing, as moderation could be reconfigured by simply editing a sentence in the prompt instead of retraining a model. Thus, it promises rapid adaptation to emerging harms, new policies by organizations or jurisdictions, or a shift in platform or community norms. Yet, the swiftness of adaptation is dictated by interdisciplinary writing processes of the new prompts as well as their evaluations and trials before deployment. However, as outlined in previous research [15], system instructions are not reliable enough to give governance guarantees towards goals such as alignment, performance, or robustness of the system.
An additional limiting factor of these approaches is the presence of what are sometimes described as “prompt stacks” [21], [22]. Prompts are processed according to a hierarchical order that prioritizes those set by foundation-model developers before downstream developers, API users, or interface end-users. This prioritization means that moderation instructions (e.g., asking the LLM to apply rules to a specific piece of content) ought to only be processed if they align with the input from prompt layers before. Higher priority prompts should not be able to be overridden by lower priority ones [23]. Therefore, what looks like writing hard rules is, in practice, adding one instruction to a possibly unaligned hierarchy.
For generative AI approaches in content moderation, the mixture of technological limitations [24], [25], well-known evasion or circumvention tactics (e.g., prompt injections) [26], [27], insufficient guardrail effectiveness [28], [29], and other behavior stemming from scaffolding around the foundation model [30], [31], ultimately means that policy-as-prompt approaches on their own are not stable enough to deliver on the envisioned guarantees. As prompt governance unfolds further, prompts have to be supplemented with robust evaluations and other governance measures to have a chance at easing content moderation burdens. Still, the seeming ease of implementation and oversight incentivizes the application of policy-as-prompt.
In addition to prompt-based moderation having specific technical risk and harm possibilities, the removal of human expertise and accountability from moderation processes will be of consequence. We trace both kinds of impacts across two moderation paradigms.
On platforms where moderation is centralized, e.g., done by a governing body, expertise plays a vital role. Policy specialists draft guidelines that then get tested and adjusted with the help of moderators and domain experts [8], [32], ideally in dialogue with community members [33]. This process shapes understanding of the policy that moderators then have to apply in practice. With policy-as-prompt regimes, a deliberated policy is compressed into a single artifact translated not for a human moderator but for an AI system. As [8] put it: “Writing for machines relies on machine interpretation”. New AI policy translation is therefore not focused on translating between organizations and communities, but between organizations and AI. The AI prompt now carries all of the nuance once passed through levels of judgment and competing incentives [34]. This translation additionally might not work as intended due to limitations of language operationalization of stochastic systems in general [8], [15].
Self-governing communities rely heavily on volunteers and a contextual understanding of their community. Volunteer moderators write their own guidelines, deliberate over them in (mostly) transparent ways, and local norms evolve with the community and their discussions [35]. A prompted moderator AI could change community dynamics: the rules could still be written, revised, and understood by the community, then defined in a (possibly non-transparent) prompt set and applied by AI. Some moderators may not want to disclose instructions, because of worries about ‘easier circumvention’ [22]. This also changes rule enforcement dynamics. When a moderator from the community applies a rule, this is based on context and knowledge of the community. By no longer performing this task, the community members lose vital expertise. This would impact the feedback loop in self-governance and potentially lead to less effective and situated moderation. The rules may even be changed in their formulation to account for misinterpretations by AI [36], [37]. The community guidelines become what the generative AI system can operationalize and change in parallel to the development of LLMs [15], [38]. Change in community norms will then (in part) be governed by the affordances of technological developments.
Policy-as-prompt approaches already exist, and are likely to increase across organizations due to perceived ease of intervention and accessibility [15]. As such, some limitations and implications should be taken into consideration when designing such systems.
First, given the technical limitations and idiosyncrasies of generative AI models, with the possibility of prompt injection and (emergent) circumvention methods, they should generally be handled with caution in content moderation [25], [39], [40]. As such, if policy-as-prompt is implemented, it needs to go beyond specifying words to go into system instructions. Writing a prompt and giving it to a general-purpose model is simply not robust enough at steering system behavior [15]. Prompts need to be evaluated on their impact on model behavior and judgment, as well as their effects across the prompt stack, e.g., through sensitivity analyses.
Second, technical evaluation of policy-as-prompt will still not give governance guarantees. The approach needs to be embedded in robust governance structures that can deliver meaningful disclosure, agency, and accountability [18], [22], [41] to the governed communities.
Third, the inability of generative AI systems to take responsibility for their outputs limits the content moderation tasks they can be deployed for. Moderation actions should be contestable and/or appealable. As it stands, LLMs cannot take the required accountability for their (moderation) actions. Therefore, they should only assist in human decisions, or supplement human deliberation processes.
Although AI might have its place in content moderation, using policy-as-prompt on its own is not enough. Moderation is highly dependent on the fluidity of language, and, thus, on context, time, and intention. As such, content moderation necessitates various sense-making operations, which include deliberation, appeals, and contextual interpretation of broader community values. Outsourcing these mechanisms (even partially) to LLMs has a profound impact. Not only because they take some of these sense-making operations out of human hands, but also because LLM “decisions” are based on a warped mirror of past language and its context, instead of current use of language within the specified context. As such, looking forward requires acknowledging that AI can only look backward, which both justifies and necessitates human-driven processes in content moderation.