Learn AI /Search, Embeddings and RAG /Keyword search
Keyword search
- Starter
- Search, Embeddings and RAG
- Programme & data teams
Also called: lexical search, full-text search, BM25.
Back to the milestone you came from
In five seconds
What it is
A search method that finds documents by matching query words to indexed text and ranking the matches, often with scoring methods such as BM25.
In plain words
It finds documents by looking for the words you typed and then puts the strongest word matches near the top.
An everyday example
You remember that someone sent the phrase “blue registration form”, but not when. Searching those words brings messages containing all three near the top, while messages containing only “form” are less useful. An unusual word such as “registration” helps narrow the results more than a common word such as “the”. A message saying “the azure sign-up sheet” may be exactly what you need, but keyword search can miss it because the wording is different.
A librarian can use judgement about topic, context and credibility even when the words do not match. Basic keyword search does much less of that. It mainly relies on term matching and scoring rules. Also, a real librarian may know which report is authoritative; keyword search does not establish truth or policy validity by itself.
Remember thisKeyword search ranks documents by matching query words to indexed text. BM25 improves that ranking, but it does not by itself recognise the same idea expressed in different words.
Understand it
Understand it
Why it matters
Non-specialists often meet keyword search inside document portals, website search bars and AI systems that retrieve sources before drafting an answer. If you know how it works, you can judge why a result appeared, why a relevant document may have been missed, and when exact wording matters. This helps with practical choices: whether to improve the query, add synonyms, clean the document collection, or combine keyword search with semantic search. It also helps you ask better governance questions about what corpus was indexed, how ranking works, and whether staff might over-trust the first result.
How keyword search turns words into ranked results
After viewing this, a reader should be able to explain that keyword search finds matching terms through an index and then ranks the matches, often using BM25.
01
1. Query arrives
The user types a few words, such as “water quality monitoring”. The system treats these words as search terms.
Draw a small search box on the left containing the sample query. Add a right-pointing arrow labelled “terms”.
02
2. Index is consulted
The system does not scan every document from the start. It uses an index, a prepared map from terms to documents and positions.
Show a central box labelled “index” with rows like “water → doc 2, doc 7”. Connect the query arrow into this box.
03
3. Matches are scored
Documents with matching terms are given scores. BM25 is a common formula that rewards useful term matches and adjusts for document length and term rarity.
Add a scoring box to the right of the index. Show three document cards entering and getting scores such as 8.4, 5.9 and 2.1.
04
4. Results are shown
The system returns the highest-ranked documents or passages, often with highlighted query words.
Show a ranked list on the far right with result 1, result 2, result 3 and short snippets with highlighted words.
A left-to-right flow diagram with four boxes: user query, indexed documents, term matching, and ranked results. Arrows show that the query is broken into terms, checked against the index, scored, and returned as an ordered list with snippets.
Hovering over each stage reveals a one-sentence note: query terms, index lookup, BM25 scoring, and ranked output.
Describe this without the diagram
The diagram explains keyword search as a four-stage flow. First, a user enters a short query. Second, the system looks up those terms in an index, which is a prepared lookup table built from the document collection. Third, matching documents are scored, often with BM25, which considers factors like term frequency, term rarity and document length. Fourth, the system returns an ordered list of results with snippets showing where the terms appeared. The main point is that keyword search depends on matching words and ranking the matches, not on measuring meaning directly.
With reduced motion: A static left-to-right diagram with the same four labelled stages and arrows, without animation or hover effects. Short notes appear under each stage permanently.
Keyword search looks for the words in your query inside stored text. The input is usually a short query, such as “cash transfer eligibility”, and a collection of documents that have already been indexed. An index is a prepared lookup structure that helps the system find matching words quickly.
Several related terms are often grouped together. Keyword search and lexical search usually mean matching based on words or terms. Full-text search often refers to searching across the full text of documents, not only titles or tags, and may include phrase search, filters and highlighting. BM25 is not the whole search system: it is a common ranking formula that gives higher scores to documents where important query words appear in useful ways.
What happens next is usually a two-part process. First, the system finds documents containing some or all of the query terms. Then it ranks them. Ranking may consider how rare a word is across the collection, how often it appears in one document, and document length. A document that repeats a rare query term in a relevant passage may rank higher than a very long document that mentions it once.
The output is a ranked list of documents or passages, often with snippets showing the matched words. This is different from semantic search, which tries to find similar meaning even when the same words are not used. In many real systems, keyword search and semantic search are combined because exact wording matters in some cases and broader meaning matters in others.
How “Searching a long message history” maps onto the real thing
| In the story | In the AI system |
|---|---|
| card catalogue and digital archive | the search index built from the document collection |
| exact words and close variants | lexical matching on query terms, sometimes with stemming or token normalisation |
| put near the top | ranking results with a scoring method such as BM25 |
| matching lines marked | snippets or highlights showing where query words appeared |
A worked example
A partnerships officer searches an approved working group repository for agreed language to use in a delegation briefing before capitals reply.
Starting from: Query: "digital public infrastructure safeguards" Documents: A. "Background note: safeguards for digital public infrastructure partnerships." B. "Concept note: protections for inclusive shared digital systems." C. "Talking points on partnership safeguards and accountability." D. "Workplan for stakeholder consultations and side events."
- The system tokenises the query into terms such as “digital”, “public”, “infrastructure” and “safeguards”, then looks them up in the text index.
- Document A contains every query term. Document C contains “safeguards” but not the other terms, while document B expresses a similar idea without using the query wording.
- A ranking method such as BM25 is likely to place A first. It rewards matching terms, gives relatively more weight to terms that are uncommon in the collection, and adjusts for document length.
- The tool returns A at the top with matching words highlighted. B may rank poorly or not appear because lexical retrieval does not automatically recognise equivalent wording.
Result: Document A is the top result because its text closely matches the query. The officer should still inspect lower-ranked results and try alternative terms, since keyword search can miss relevant documents written in different language.
In a United Nations workplace
A partnerships officer supporting a UN Country Team working group is preparing a delegation briefing before capitals reply. She searches the working group’s approved repository for "digital public infrastructure safeguards" to recover agreed wording from cleared background notes, concept notes and talking points.
Keyword search can quickly surface documents containing the exact institutional language used in the query. The ranking will usually favour a focused note containing all four terms over a long document mentioning only "safeguards". It may miss a relevant document that uses different wording, such as "protections for shared digital systems".
Before you try thisSearch only repositories approved for the material concerned. Draft partnership documents may contain personal contact details, commercially sensitive submissions or uncleared positions. They must not be indexed in, or uploaded to, an external tool unless that tool and collection are approved for those data and access is appropriately controlled.
Where you may hear it
- A vendor demo during an ICT governance board meeting
- A working group discussion on improving document retrieval
- A procurement meeting reviewing search requirements for an internal portal
- A knowledge-management session before a repository is opened to staff
Avoid confusion
Avoid confusion
Often confused with
Semantic search
A search method that finds material by similar meaning, not just shared words, and returns the most relevant passages for review or further AI use.
Semantic similarity
A measure of how close two pieces of text are in meaning, even if they use different words.
Cosine similarity
A measure of how closely two non-zero vectors point in the same direction, commonly used to compare embeddings in search.
Nearest-neighbour search
A method for finding the items most similar to a query in a vector space; approximate methods often trade a little accuracy for much faster search at scale.
Side-by-side
Common misconceptions
Common misconception: Keyword search and semantic search are the same thing with different names.
Correction: They are related but different. Keyword or lexical search relies mainly on matching words or terms in text. Semantic search tries to retrieve by meaning, often using embeddings. Many systems combine both because each catches cases the other misses.
Common misconception: BM25 is just another name for full-text search.
Correction: BM25 is a ranking formula, not the whole search feature. Full-text search usually refers to searching across document text and may include indexing, phrase matching, filters, highlighting and ranking. BM25 is one common way to order the matching results.
Common misconception: If a document is ranked first, it must be the best or most correct source.
Correction: The top result is simply the highest-scoring match under that system’s rules. Ranking reflects the query, the index and the scoring method. It does not guarantee accuracy, authority, timeliness or policy validity, so human checking still matters.
Quick check
Which statement best distinguishes BM25 from semantic search?
BM25 is a lexical ranking method built around query-term matches and related weighting. Semantic search usually relies on similarity in a vector space to capture related meaning even when the same words are absent. Neither method guarantees factual correctness.
Go deeper
Go deeper
The simple version
It finds documents by looking for the words you typed and then puts the strongest word matches near the top.
The technical version
In lexical retrieval, documents are indexed by terms so the system can quickly identify which documents contain the query terms. Full-text search usually means this matching happens over document text, not only metadata. After retrieval, a ranking function orders candidates. BM25 is a widely used probabilistic ranking function that typically increases a document’s score when query terms appear in it, gives more weight to terms that are rarer across the collection, and applies length normalisation so very long documents are not unfairly favoured just because they contain more words. Exact implementation details vary by search engine and configuration, and many production systems add phrase matching, field weights, filters, stemming, stop-word handling and reranking.
Reviewed external resources
EncyclopaediaWikipedia search: Keyword search (opens in a new tab)Wikipedia
This search results page helps readers scan multiple Wikipedia entries related to keyword search and suits beginners who want a broad starting point rather than one fixed article.
search results, not one pagelink checked 2026-07-31
CourseEmbeddings (opens in a new tab)Google
This course resource introduces embeddings and suits readers who want to compare word-matching search with meaning-based methods at a beginner-friendly level.
Intermediatelink checked 2026-07-31
InteractiveEmbedding Projector (opens in a new tab)TensorFlow
This interactive tool lets readers explore how items can be placed by similarity in an embedding space and suits visual learners comparing semantic methods with keyword matching.
Starterlink checked 2026-07-31
VideoVideo explanations: Keyword search (opens in a new tab)YouTube search
This search results page offers many video explanations of keyword search and suits readers who prefer to compare teaching styles before choosing one video.
search results, not one pagelink checked 2026-07-31
DiscussionCommunity discussion: Keyword search (opens in a new tab)Reddit search
This search results page shows community discussions and practical questions about keyword search and suits readers who want informal examples, opinions and troubleshooting tips.
search results, not one pagelink checked 2026-07-31
Where this came from
Drafted with a language model through an offline pipeline, checked against this site's schema, and every external link above fetched to see whether it answers. Treat it as a starting point and check anything that matters against the primary sources. How this site is built and what that means.
The one thing to remember
Keyword search ranks documents by matching query words to indexed text. BM25 improves that ranking, but it does not by itself recognise the same idea expressed in different words.