Learn AI /Data and Infrastructure /Data cleaning
Data cleaning
- Starter
- Data and Infrastructure
- Programme & data teams
Also called: data preparation, data wrangling.
Back to the milestone you came from
In five seconds
What it is
The process of checking data, fixing problems, and putting it into a usable form for analysis or model training.
In plain words
It is the work of making messy data usable by fixing obvious problems, making formats consistent and marking anything uncertain.
An everyday example
After changing phones, you find the same person listed twice, telephone numbers written in several formats, old numbers still marked as current and contacts with no email address. You merge entries only after checking that they refer to the same person, standardise the number format, mark obsolete numbers and leave unknown email addresses blank. Deleting every incomplete contact would make the list neater, but might remove someone you still need.
A document shelf is handled by people who can look at each item and use context. Real data cleaning is often done at large scale, sometimes across millions of records, using rules or scripts that can introduce new errors if the rules are poor. Also, some data types, such as images or sensor streams, do not fit neatly into a shelf-and-folder picture.
Remember thisData cleaning is not simply tidying. The rules must fit the intended use, preserve traceability and avoid turning missing or uncertain data into invented certainty.
Understand it
Understand it
Why it matters
Poor-quality data can produce poor analysis, weak search results and misleading model behaviour. A small format problem can split one group into several false categories. Duplicate records can make something look more common than it is. Missing values can hide important gaps. For non-specialists, the key point is this: results do not only depend on the model or chart. They also depend on how the data was prepared. Asking how data was cleaned is part of checking whether a result is trustworthy, fair for its purpose and worth acting on.
From raw data to usable data
Show the main steps between messy input data and a cleaned dataset, and explain why each step changes reliability.
01
1. Raw data arrives
The starting data may contain duplicates, missing values, inconsistent formats, irrelevant entries or obvious errors.
Show a mixed stack of table rows and document icons with warning marks: duplicate rows, blank cells, date formats like 03/04/24 and 2024-04-03, and one row marked irrelevant.
02
2. Inspect problems
People or software scan the data to find patterns of error and decide which problems matter for the task.
Show a magnifying glass over the messy data, with callouts for duplicates, blanks, spelling variation and out-of-range values.
03
3. Set cleaning rules
The team agrees what to fix, remove, standardise or flag. These choices should match the purpose of the project.
Show a checklist with four rules: standardise format, remove exact duplicates, flag uncertain records, keep a change log.
04
4. Apply fixes carefully
Rules are applied by hand, by script or both. Some values are corrected, some dropped, and some left unchanged but marked for review.
Show arrows from messy items to corrected versions: country names aligned, duplicate removed, blank cell flagged, uncertain item moved to review.
05
5. Check and document
The cleaned data is reviewed to see whether it is more consistent and still suitable for the task. Changes are recorded.
Show a tidy table and document set with a small side panel labelled change log and a final check mark.
A left-to-right step diagram with five boxes. It starts with messy raw data, then inspection, rule-setting, cleaning actions and final checks, ending with a cleaned dataset. Read in order to see how each stage reduces specific problems and records choices.
On hover or tap, each problem type highlights the matching fix in the next step.
Describe this without the diagram
This diagram shows data cleaning as a five-step process from left to right. Step 1 shows raw data with common problems: duplicate records, blank fields, inconsistent date formats and irrelevant items. Step 2 inspects the data to identify these problems. Step 3 sets rules for what will be standardised, removed or flagged. Step 4 applies those rules, for example removing exact duplicates, making formats consistent and marking uncertain records for review. Step 5 checks the result and records what changed, producing a cleaner dataset that is easier to use but not guaranteed to be perfect.
With reduced motion: A static five-step diagram with the same boxes and labels, no animation. Problem icons are connected by thin lines to the corresponding fixes and to the final cleaned dataset.
Raw data often arrives messy. Some rows may be duplicated. Some values may be missing. Dates may use different formats. Text may contain spelling variations, extra spaces or irrelevant symbols. Data cleaning prepares that material so people or software can use it more reliably.
The input can be a table, documents, images, sensor readings or a mix of these. The work usually includes spotting errors, deciding what counts as an error, correcting what can be corrected, removing what should not be kept, and making formats consistent. For example, one column might use “Kenya”, another “kenya”, and another “KE”. Cleaning may standardise those into one agreed form.
The output is a cleaner dataset, not a perfect one. Some problems cannot be safely fixed, so they may be flagged instead. Good cleaning also records what was changed and why. That matters because later results depend on these choices.
Data cleaning is often confused with data labelling. Cleaning improves the quality and consistency of existing data. Labelling adds tags or categories, such as marking whether an image contains flood damage. In practice, the two can happen in the same project, but they are not the same task.
How “Tidying a phone contact list” maps onto the real thing
| In the story | In the AI system |
|---|---|
| duplicate copies on the shelf | duplicate records in a dataset |
| titles written in different styles | inconsistent formats such as names, dates or text case |
| outdated folders mixed with current ones | irrelevant or low-quality data that may need removal |
| uncertain items set aside for review | records that cannot be safely fixed automatically and must be flagged |
A worked example
An information management officer in a regional bureau is preparing monitoring reports for upload to the portal before the weekly search index is refreshed.
Starting from: The metadata export contains exact duplicate records, inconsistent region labels, dates stored in different formats, blank publication years and one superseded draft.
- The officer defines the cleaning rules for the intended search function and preserves an unchanged copy of the export.
- Exact duplicates are identified using document identifiers and removed. Records with similar titles are retained for review rather than treated as duplicates automatically.
- Region labels are mapped to the portal's controlled list, and valid dates are converted to a common format.
- Blank publication years are marked as unknown rather than filled with a default. The superseded draft is excluded after its status is confirmed against the file.
- The officer validates the allowed categories, checks a sample against the source reports and saves a log of every removal, conversion and unresolved issue.
Result: The portal receives consistent, searchable metadata with confirmed duplicates removed, the superseded draft excluded and missing values clearly distinguished from errors.
In a United Nations workplace
A duty officer staffing an inter-agency secretariat is assembling talking points for informals the next morning. Inputs have arrived in several spreadsheets, with agencies named in different ways, repeated rows, dates in mixed formats and blank status fields.
Cleaning aligns names and date formats, removes confirmed duplicates and marks missing values without guessing. This reduces the risk that the talking points repeat one update or present a formatting difference as a substantive disagreement.
Before you try thisUncirculated agency inputs and internal comments must not be pasted into a public or otherwise unapproved AI tool. Use an approved environment, retain the source files and a change log, and ask the relevant focal point to confirm any correction that could alter the substance.
Where you may hear it
- At a vendor demo, when someone asks how the tool handles duplicate and missing records
- In an inter-agency working group before survey results are circulated
- During clearance of a dashboard whose figures do not reconcile with the source file
- At a handover before the next situation report is produced
Avoid confusion
Avoid confusion
Often confused with
Annotation and data labelling
People or software tools add task-relevant tags, notes or boundaries to data for model training, testing, analysis or monitoring.
Sampling (of data)
Choosing a smaller set of cases from a larger group so you can measure, inspect or train on it without using every case.
Metadata
Information that describes other data, such as who created it, when, where, in what format, and under what rules it can be used.
Common misconceptions
Common misconception: Cleaning means deleting anything that looks untidy.
Correction: Not every unusual value is an error. A rare event, minority language spelling or extreme measurement may be valid and important. Good cleaning uses rules linked to the task and flags uncertain cases instead of simply removing them.
Common misconception: Once data has been cleaned, it is unbiased and fully reliable.
Correction: Cleaning can reduce obvious errors, but it does not remove all bias, gaps or uncertainty. The original data may still under-represent some places or groups, and cleaning choices themselves can affect outcomes.
Common misconception: Cleaning and labelling are the same job.
Correction: They often happen in the same workflow, but they do different things. Cleaning improves data quality and consistency. Labelling adds categories or tags, such as marking an image as damaged or undamaged.
Quick check
Which example best fits data cleaning?
Data cleaning focuses on improving the usability and consistency of existing data. Removing duplicates and standardising dates are classic cleaning tasks. Adding labels is annotation, collecting new responses is data collection, and choosing a larger server is a compute or infrastructure decision.
Go deeper
Go deeper
The simple version
It is the work of making messy data usable by fixing obvious problems, making formats consistent and marking anything uncertain.
The technical version
Data cleaning is a preprocessing stage that improves data quality before analysis, search or model training. Typical operations include deduplication, schema alignment, type conversion, normalisation of formats, handling missing values, filtering irrelevant records, resolving obvious inconsistencies and validating ranges or categories. The exact rules depend on the intended use. For example, replacing missing values with defaults may help one workflow and distort another. Cleaning may be manual, scripted or mixed. It is usually paired with documentation so others can audit what was changed, dropped or flagged. It differs from annotation, which adds labels, and from sampling, which changes which records are included rather than primarily improving their consistency.
Reviewed external resources
Primary sourceAI Risk Management Framework (opens in a new tab)NIST
This official NIST framework gives risk-management context for data quality and governance choices, and suits programme managers, policy staff and technical leads.
Intermediatelink checked 2026-07-31
Primary sourceOECD AI Principles (opens in a new tab)OECD
These OECD principles provide high-level guidance on trustworthy AI, including the importance of data quality and accountability, and suit non-specialists and decision-makers.
Intermediatelink checked 2026-07-31
EncyclopaediaWikipedia search: Data cleaning (opens in a new tab)Wikipedia
This search results page helps readers find a range of Wikipedia entries related to data cleaning, and suits someone who wants a quick starting point rather than a single authoritative article.
search results, not one pagelink checked 2026-07-31
VideoVideo explanations: Data cleaning (opens in a new tab)YouTube search
This search results page helps readers browse different video explanations of data cleaning, and suits visual learners who want multiple introductory teaching styles.
search results, not one pagelink checked 2026-07-31
DiscussionCommunity discussion: Data cleaning (opens in a new tab)Reddit search
This search results page shows community discussions and practical questions about data cleaning, and suits readers who want informal examples and peer experiences.
search results, not one pagelink checked 2026-07-31
Where this came from
Drafted with a language model through an offline pipeline, checked against this site's schema, and every external link above fetched to see whether it answers. Treat it as a starting point and check anything that matters against the primary sources. How this site is built and what that means.
The one thing to remember
Data cleaning is not simply tidying. The rules must fit the intended use, preserve traceability and avoid turning missing or uncertain data into invented certainty.