Learn AI /Computer Vision and Language Technology /Corpus
Corpus
- Intermediate
- Computer Vision and Language Technology
- Technical readers
- Programme & data teams
Also called: text corpus, corpora.
Back to the milestone you came from
In five seconds
What it is
An organised collection of text or recorded speech used to study language or train and test language-processing systems.
In plain words
A corpus is an organised collection of language examples that people use to study language or build and test language tools.
An everyday example
Imagine a school event receives many messages about missing items, written in several languages. The organisers sort them by language and date, then label details such as the item, colour and last known location. They use most of this organised collection to teach a search helper which messages are similar, while keeping some aside to check whether it works on messages it has not seen. The organised messages form the corpus. The labels and coverage determine which patterns the helper can recognise.
A real corpus is usually much larger and more formally structured than a pile of traveller notes. It may include millions of words, careful sampling rules, tokenisation or other processing steps, and quality checks. Also, a corpus does not by itself “improve” a system; people and algorithms use it for analysis, training or evaluation.
Remember thisA corpus is an organised collection of language material. Its coverage, metadata and annotations shape what a language system can learn and how fairly it can be tested.
Understand it
Understand it
Why it matters
If you commission, buy, evaluate or govern an AI language system, you need to ask what corpus it was built and tested on. That affects what language varieties it handles, what biases it may repeat, and whether results fit your context. A system trained on news text may fail on field reports. A corpus with only major languages may exclude low-resource languages. Knowing about corpora helps you ask practical questions about coverage, quality, consent, labelling, and representativeness before relying on outputs.
How a corpus is built and used
Show that a corpus is language material collected, organised and often labelled so it can support analysis, training or evaluation.
01
1. Source texts
Start with language materials such as articles, reports, transcripts or message logs collected for a defined purpose.
Draw a box with document and speech-bubble icons labelled “articles”, “transcripts”, “messages”. Add a caption: “Raw language material”.
02
2. Gather into one collection
Bring the texts together and record basic information such as language, date, source or domain.
Draw arrows from the source box into a larger container labelled “Corpus”. Add small tags: “language”, “date”, “source”.
03
3. Clean and annotate
Remove obvious errors where needed and add labels or markup if the task requires them, such as sentiment labels or named entities.
Show the corpus box leading to a second corpus box with coloured highlights and tag labels like “positive” and “organisation”.
04
4. Use for a task
The organised corpus can then support language analysis, model training, or testing whether a system works on relevant text.
Draw three outgoing arrows to small boxes labelled “analyse”, “train”, and “test” with example outputs like a chart, a model icon, and an evaluation score.
A left-to-right flow with four boxes. Read from source texts, to collection, to preparation, to use. Small examples in each box show how raw language becomes an organised resource for a specific task.
Hover or tap each stage to reveal a short example, such as a raw sentence becoming a labelled sentence for named-entity recognition.
Describe this without the diagram
The diagram is a four-step flow from left to right. Step 1 shows raw language sources such as articles, transcripts and messages. Step 2 shows these gathered into a single corpus with metadata like language, date and source. Step 3 shows the corpus cleaned and, if needed, annotated with labels such as sentiment or entity tags. Step 4 shows three uses of the prepared corpus: analysing language patterns, training a system, and testing system performance. The main idea is that a corpus starts as collected language material and becomes useful after organisation and sometimes labelling.
With reduced motion: A static four-column flowchart shows the same stages without animation: source texts, corpus collection, cleaned and annotated corpus, and use for analysis, training or testing.
A corpus is a body of language data gathered for a purpose. It usually contains many documents, sentences, or transcripts collected in one place. The items may be books, news articles, call-centre transcripts, social media posts, parliamentary debates, or transcribed speech.
What goes in depends on the task. For machine translation, a corpus may contain sentence pairs in two languages. For sentiment analysis, it may contain reviews labelled positive or negative. For named-entity recognition, it may contain text where names of people, places, and organisations have been marked.
What happens next is preparation. The text is cleaned, organised, and often annotated, meaning humans or software add useful labels or structure. Researchers or developers then use the corpus to analyse patterns in language, train a model, or test whether a system works well on the kind of language it will meet in practice.
What comes out is not the corpus itself, but evidence or a system built from it. For example, a corpus can reveal which words often appear together, help train a translation system, or show that a chatbot performs poorly on a regional variety of English. A corpus is related to a dataset, but it usually refers specifically to language material rather than any kind of data.
How “Organising messages for a lost-and-found helper” maps onto the real thing
| In the story | In the AI system |
|---|---|
| traveller notes gathered in one place | texts collected into a corpus |
| sorting by route, language and topic | organising corpus documents and metadata |
| marking notes with labels like “late bus” | annotating text for a task such as classification |
| using patterns to improve the guide | analysing or training a language-processing system |
A worked example
Before talking points for a side event go for clearance, a desk officer wants to test whether a multilingual search tool can retrieve related passages across public statements in English, French and Spanish.
Starting from: A collection of publicly available statements and approved translations, organised by language and document, with selected passages marked as relevant to sample search queries.
- Collect the statements, record metadata such as language and source, and remove duplicate or incomplete files.
- Align translated passages where reliable equivalents exist, while retaining monolingual material that remains useful for testing.
- Have reviewers mark which passages are relevant to each sample query and record disagreements for resolution.
- Use one part of the corpus to configure the retrieval system and keep another part separate for evaluation.
Result: The organised corpus supports a reproducible test of cross-language retrieval. If the tool finds relevant English passages but misses comparable French or Spanish passages, the result may point to gaps in corpus coverage, alignment or annotation rather than simply a weak model.
In a United Nations workplace
At a regional bureau, an information management officer is finalising terms of reference, due for clearance before Friday, for a tool that will identify recurring issues in multilingual progress reports from country offices.
The terms of reference should describe the corpus separately from the model: which reports are included, their languages and provenance, what metadata or annotations accompany them, and which material is reserved for evaluation. Those choices determine what language and reporting patterns the tool can learn and assess.
Before you try thisProgress reports may contain personal, partner-sensitive or security-sensitive information. Such material must not be uploaded to, or used to train, a service that has not been approved for it. Access controls, retention arrangements and permitted uses must be settled before the corpus is assembled, not after the vendor demo.
Where you may hear it
- While clearing terms of reference for a multilingual search tool
- At a vendor demo reviewing what the model was trained and tested on
- In an evaluation meeting on annotation quality and language coverage
- During a technical review of translation or speech-recognition results
Avoid confusion
Avoid confusion
Often confused with
Dataset
A dataset is an organised collection of examples that a system can use for analysis, training, testing or search.
Knowledge base
An organised set of trusted documents that a search system can look through to find relevant passages for an answer.
Common misconceptions
Common misconception: A corpus is just any dataset.
Correction: A corpus is a kind of dataset, but the term usually points to a collection of language material such as text or transcripts. A dataset can contain many other things, including images, sensor readings or mixed data types.
Common misconception: A bigger corpus always means a better system.
Correction: Size helps only if the content fits the task and is of usable quality. A very large corpus with noisy text, weak labels or the wrong domain can perform worse than a smaller, carefully chosen corpus.
Common misconception: A corpus is the same as a knowledge base.
Correction: A corpus stores examples of language as people wrote or said it. A knowledge base stores structured facts or records designed for retrieval. Some systems use both, but they serve different roles.
Quick check
Which description best matches a corpus in language technology?
A corpus is a collection of language material such as texts, recordings or transcripts, organised for analysis, training or evaluation. A knowledge base holds trusted documents for passage retrieval, a single prompt document is not such a collection, and object-detection images are a different kind of dataset.
Go deeper
Go deeper
The simple version
A corpus is an organised collection of language examples that people use to study language or build and test language tools.
The technical version
In language technology, a corpus is an organised collection of textual or recorded linguistic data, often stored with metadata and sometimes with annotations such as part-of-speech tags, entity labels, sentiment labels or aligned translations. It may be monolingual, multilingual or parallel, depending on the task. A corpus can be used for descriptive analysis, feature extraction, supervised or unsupervised model development, and evaluation. The term is narrower than dataset because it usually refers specifically to language material rather than arbitrary data. It differs from a knowledge base, which organises trusted documents for searching and passage retrieval rather than treating them as language examples.
Reviewed external resources
EncyclopaediaText corpus (opens in a new tab)Wikipedia
This Wikipedia link gives a broad overview of what a text corpus is and suits readers who want a quick general introduction.
Intermediatelink checked 2026-07-31
ArticleMachine Learning Glossary (opens in a new tab)Google
This Google glossary link gives a short definition-style explanation and suits readers who want a quick refresher.
Intermediatelink checked 2026-07-31
CourseMachine Learning Crash Course (opens in a new tab)Google
This Google Machine Learning Crash Course link helps readers connect data collections to model building and suits learners who want practical ML context.
Intermediatelink checked 2026-07-31
VideoVideo explanations: Corpus (opens in a new tab)YouTube search
This YouTube search results page helps readers find different video explanations of corpus and suits people who prefer to compare teaching styles before choosing one.
search results, not one pagelink checked 2026-07-31
DiscussionCommunity discussion: Corpus (opens in a new tab)Reddit search
This Reddit search results page helps readers see community discussions and examples of how people use the term, and suits readers who want informal explanations with caution about accuracy.
search results, not one pagelink checked 2026-07-31
Where this came from
Drafted with a language model through an offline pipeline, checked against this site's schema, and every external link above fetched to see whether it answers. Treat it as a starting point and check anything that matters against the primary sources. How this site is built and what that means.
The one thing to remember
A corpus is an organised collection of language material. Its coverage, metadata and annotations shape what a language system can learn and how fairly it can be tested.