Learn AI /Responsible AI, Human Rights and Safety /Content moderation
Content moderation
- Starter
- Responsible AI, Human Rights and Safety
- Communications teams
- Human rights practitioners
Also called: moderation, trust and safety.
Back to the milestone you came from
In five seconds
What it is
The process of reviewing and acting on user content against rules to reduce harm, illegality and abuse while protecting lawful expression and human rights.
In plain words
It is the organised process for checking posts or uploads against rules and deciding whether to allow them, label them, limit them or remove them.
An everyday example
A city office receives 2,000 survey comments before a public meeting. Staff first group repeated comments together, then mark a smaller set that may contain threats, hate speech or personal attacks for closer review. Most comments are published as they are, some are published with names removed, and a few are withheld because they break the meeting rules. People can contact the office if they think their comment was wrongly withheld.
This analogy is simpler than real moderation. Survey comments usually arrive in one batch and under one clear set of meeting rules, while online content can appear continuously, spread quickly across borders, include images or live video, and require decisions in many languages and legal contexts. Real moderation also involves safety risks for reviewers and errors from automated systems.
Remember thisContent moderation combines rules, automated signals and human judgement. Its decisions should be contextual, proportionate, documented and open to challenge.
This term is contested
The broad idea is widely used, but the exact boundaries of what counts as moderation, especially around ranking, recommendation, demotion and labelling, can vary between organisations and legal frameworks.
Understand it
Understand it
Why it matters
Content moderation affects what people can say, what they can safely see and whether online spaces become useful, abusive or dangerous. For a non-specialist, the key issue is not only whether harmful material is removed, but how the decision is made, who can challenge it, and whether some groups are treated unfairly. Weak moderation can leave people exposed to threats, exploitation or disinformation. Overbroad moderation can wrongly silence lawful speech, including reporting, satire, political expression or testimony about abuse. Understanding moderation helps you ask better questions about rules, evidence, transparency, appeals, human oversight and the limits of automation.
How content moderation works
After viewing this diagram, a reader should be able to describe the main stages from content submission to review, action and appeal.
01
1. Content arrives
A post, image, video or comment is submitted, or an existing item is reported by a user.
Draw a box labelled 'Content or report'. Show small icons for text, image and video entering the box.
02
2. Initial checks
Software may compare the item with rules or known patterns and assign risk labels such as spam, abuse or graphic content.
Draw a second box labelled 'Automated checks'. Add small tag labels like 'spam', 'abuse', 'graphic' attached to the box.
03
3. Review
Clear cases may be handled automatically, while uncertain or high-risk cases go to trained human reviewers.
Draw a fork after the automated checks: one short path marked 'clear case' and one path to a box labelled 'Human review' marked 'uncertain/high-risk'.
04
4. Action
Possible outcomes include allow, label, reduce visibility, remove, restrict account or escalate for specialist review.
Draw an outcomes box with six vertically stacked buttons: 'allow', 'label', 'limit reach', 'remove', 'restrict account', 'specialist review'.
05
5. Notice and appeal
The user may receive a reason and, where available, a chance to ask for the decision to be checked again.
Draw a final box labelled 'Notice and appeal'. Add a loop-back arrow from this box to 'Human review'.
A left-to-right flow diagram with five boxes connected by arrows. It starts with incoming content and reports, moves through automated checks and human review, then shows possible actions and an appeal path. Read from left to right to see how the same item can be allowed, limited, removed or sent for a second look.
Hovering or tapping each step highlights the input, decision point and output for that stage; selecting an outcome reveals the appeal loop if one exists.
Describe this without the diagram
The diagram explains content moderation as a five-step flow. First, content is posted or reported. Second, software performs initial checks and may add labels such as spam, abuse or graphic content. Third, clear cases may be handled directly, but uncertain or high-risk cases go to human review. Fourth, an action is chosen: allow, label, limit reach, remove, restrict the account or send to specialist review. Fifth, the user can be notified and may have an appeal route, which sends the case back for another human check. The main point is that moderation is a process with stages and outcomes, not a single automatic block button.
With reduced motion: Show the same five boxes and arrows in a static left-to-right layout, including the fork to human review and the appeal arrow looping back to review, with no animation.
Content moderation is the process used by platforms, services or organisations to check text, images, video or audio against rules about what is allowed. The input is content people upload, plus the rules, reports from users and sometimes context such as where the content appeared.
What happens in the middle can involve people, software or both. A system may first sort or flag content into categories such as spam, harassment or graphic violence. Human reviewers may then examine difficult cases, especially when context matters. The possible actions include leaving content up, adding a warning label, reducing its reach, removing it, restricting an account or sending the case for appeal.
The output is not just a yes-or-no decision. It can also include a record of why action was taken, a notice to the user, and a route to challenge the decision. Good moderation aims to reduce harm while also protecting freedom of expression, non-discrimination and due process.
Content moderation is often confused with toxicity detection or guardrails. Toxicity detection is usually one tool that scores likely abusive language. Guardrails are broader controls on how an AI system behaves. Moderation is the wider operational process of applying rules and making or supporting decisions about content.
How “Sorting survey comments before a public meeting” maps onto the real thing
| In the story | In the AI system |
|---|---|
| survey comments sent by residents | user-generated content such as posts, images or videos |
| staff grouping and flagging comments | automated tools and reviewers sorting content into categories and risk levels |
| publishing, redacting or withholding comments | moderation actions such as allow, label, limit, remove or escalate |
| contacting the office to challenge a decision | appeals and review mechanisms |
A worked example
During an online youth health consultation, a participant reports the comment, “You people should be attacked at your next event.” The platform must apply its rule on threats while accounting for context and lawful expression.
Starting from: The reported comment, nearby replies, account history, user reports and the platform rule on threats of violence.
- An automated classifier assigns a high-risk score and places the comment in a priority review queue. It does not remove the comment solely because it contains violent language.
- A human reviewer examines the surrounding exchange to determine whether the words are a quotation, satire, counterspeech or a threat directed at the organisers.
- Finding a direct threat under the published rule, the reviewer removes the comment and applies the proportionate account action specified by the platform policy.
- The system records the evidence and reason, notifies the user and provides an appeal route. Any separate security escalation follows established procedures.
Result: The threatening content is removed through a documented process that includes contextual review, a policy basis, proportionate action and a means of challenge.
In a United Nations workplace
By close of business, a duty officer in a procurement unit must draft a note for the file responding to an audit observation on abusive posts in a supplier question portal. The note sets out the posting rules, automated flagging thresholds, human review queue, decision log and route for challenging a removal.
The control is not merely a blocked-word list. It combines defined rules, contextual signals, risk-based queueing and documented human decisions. The audit trail should show why content was retained, labelled or removed, and whether comparable cases received comparable treatment.
Before you try thisDo not enter bids, evaluation material, personal data or other procurement-sensitive information into a moderation tool unless it is approved for that material. Automated flags must not determine supplier eligibility, sanctions or other consequential action. Such decisions require authorised human review, recorded reasons and the applicable procedural safeguards.
Where you may hear it
- A vendor demo in a procurement meeting, when the supplier explains flagging, review and appeals
- An ICT governance board reviewing controls for a public consultation platform
- A side-event planning meeting before an online questions channel is opened
- A response to an audit observation on moderation logs and reviewer decisions
Avoid confusion
Avoid confusion
Often confused with
Toxicity
Toxicity means abusive, hostile or degrading output, including insults, threats or dehumanising content aimed at people or groups.
Guardrail
A guardrail is a rule, check or control that limits unsafe or unacceptable AI outputs or actions before people rely on them.
Common misconceptions
Common misconception: Content moderation just means deleting bad posts.
Correction: Removal is only one possible action. Moderation can also include warnings, reduced visibility, age gates, temporary restrictions, escalation to specialists, or deciding to leave content up because context makes it lawful or important.
Common misconception: A good moderation system can be fully automated.
Correction: Software can help sort and prioritise large volumes, but many cases depend on language, culture, satire, documentation of abuse or local context. Those cases need human judgement, review processes and appeals, especially where rights or safety are at stake.
Common misconception: If a rule is written down, applying it is neutral.
Correction: Rules still need interpretation, and both rules and tools can affect groups differently. For example, dialect, reclaimed slurs, political speech or images documenting violence may be misclassified, so fairness and non-discrimination checks matter.
Quick check
Which description best fits content moderation?
Content moderation is broader than one detection tool. It includes rules, review, actions such as label or removal, and often an appeal path. It does not guarantee truth, and it is not the same as access control.
Go deeper
Go deeper
The simple version
It is the organised process for checking posts or uploads against rules and deciding whether to allow them, label them, limit them or remove them.
The technical version
Content moderation is a socio-technical enforcement process applied to user-generated content or model outputs under defined policies, legal obligations and safety rules. Inputs can include the content itself, metadata, user reports, prior violations and contextual signals. Operationally, systems often combine automated detection or classification tools with rule-based thresholds, queueing, risk scoring and human review. Outputs may include no action, labelling, downranking, age restriction, removal, account sanctions, referral to specialist teams and an appeal record. The quality of moderation depends not only on detection accuracy but also on policy design, procedural fairness, transparency, reviewer support, human rights safeguards and ongoing auditing for unequal error patterns.
Reviewed external resources
Primary sourceRecommendation on the Ethics of Artificial Intelligence (opens in a new tab)UNESCO
This UNESCO recommendation sets out high-level ethical principles for AI governance and suits readers looking for an official normative framework.
Intermediatelink checked 2026-07-31
Primary sourceB-Tech Project (opens in a new tab)UN Human Rights Office (OHCHR)
This OHCHR B-Tech material helps readers connect platform practices with human rights responsibilities and suits policy, rights and governance audiences.
Intermediatelink checked 2026-07-31
EncyclopaediaContent moderation (opens in a new tab)Wikipedia
This Wikipedia page gives a broad introductory overview of content moderation and suits readers who want quick background before going deeper.
Intermediatelink checked 2026-07-31
VideoVideo explanations: Content moderation (opens in a new tab)YouTube search
These YouTube search results offer a range of video explanations on the topic and suit readers who prefer to compare multiple visual introductions rather than read one article.
search results, not one pagelink checked 2026-07-31
DiscussionCommunity discussion: Content moderation (opens in a new tab)Reddit search
These Reddit search results show community discussions and examples of how people talk about moderation in practice, and suit readers who want informal perspectives with caution about reliability.
search results, not one pagelink checked 2026-07-31
Where this came from
Drafted with a language model through an offline pipeline, checked against this site's schema, and every external link above fetched to see whether it answers. Treat it as a starting point and check anything that matters against the primary sources. How this site is built and what that means.
The one thing to remember
Content moderation combines rules, automated signals and human judgement. Its decisions should be contextual, proportionate, documented and open to challenge.