Learn AI /Search, Embeddings and RAG /Indexing
Indexing
- Intermediate
- Search, Embeddings and RAG
- Technical readers
Also called: index.
Back to the milestone you came from
In five seconds
What it is
Indexing organises documents or document chunks into a searchable structure so a system can quickly find the most relevant passages for a query.
In plain words
Indexing is the preparation work that organises documents so a search system can quickly find useful passages later.
An everyday example
A community centre records each lost item by type, colour, material, where it was found and when it arrived. Someone later asks for a dark waterproof jacket left near the sports hall. The register can quickly surface entries such as “navy raincoat” even though the wording is not identical. It narrows the search to likely items, but a staff member still checks the actual coat before returning it.
A library card cabinet is usually built by people using clear categories and words. Many AI systems also store numeric representations of meaning, not just labels, and use distance calculations rather than human-readable categories. Also, a librarian can apply broad real-world judgement while an index only narrows the search space; it does not by itself verify truth, interpret nuance or replace human review.
Remember thisIndexing prepares documents, sections or other units for fast retrieval. It helps a system find likely passages, but does not judge whether they are correct, sufficient or current.
Understand it
Understand it
Why it matters
If indexing is done well, search systems can find better evidence faster. If it is done badly, users may see irrelevant, outdated or incomplete passages even when the right answer exists in the collection. For managers and practitioners, this affects trust, speed, cost and risk. Good indexing choices influence what can be found, what gets missed, how often systems need to be updated and whether a retrieval-augmented system can show usable sources instead of unsupported text.
How indexing prepares documents for later search
Show how raw documents are transformed into a search-ready structure that retrieval can query efficiently later.
01
1. Collect content
Documents enter the pipeline with source information such as title, date or document ID.
Draw a stack of document icons on the left, each with small metadata tags beneath.
02
2. Split into chunks
Long documents are divided into smaller passages so retrieval can return a focused section rather than an entire file.
Show one long document breaking into several smaller rectangles labelled chunk A, B and C.
03
3. Create search features
For each chunk, the system stores searchable features such as keywords, metadata and often an embedding vector representing meaning.
For each chunk rectangle, add side badges: 'keywords', 'metadata', and a small row of dots or numbers for 'embedding'.
04
4. Build the index
These features are organised into a structure designed for fast lookup, so later queries can find nearby or matching chunks quickly.
Merge the chunk features into a database-like box labelled 'index', with arrows from all chunks into it.
05
5. Use at query time
A user query is transformed into the same search form and compared against the index to return likely relevant passages.
Add a query bubble entering from below, connecting to the index, with top results emerging on the right as ranked passages.
A left-to-right flow with four boxes. Read from documents entering on the left to a search-ready index on the right. Each box shows one preparation stage and the output passed to the next stage.
Hovering or tapping a stage highlights what is added at that step: raw text, chunks, features, index structure, then retrieved passages.
Describe this without the diagram
The diagram is a five-stage flow. First, raw documents with source metadata enter the system. Second, long documents are split into smaller chunks. Third, each chunk gets search features such as keywords, metadata and often an embedding, which is a numeric representation of meaning. Fourth, these features are organised into an index, a structure built for fast search. Fifth, when a user sends a query, the system converts the query into comparable search features and searches the index to return likely relevant passages. The key point is that indexing happens before the user asks, so retrieval later can be fast and targeted.
With reduced motion: A static left-to-right diagram with five labelled boxes and arrows: documents, chunking, search features, index, and query-time retrieval results.
Indexing is the preparation step that makes later search fast and useful. The input is usually a set of documents, or smaller pieces of them called chunks. Each chunk may be stored with its text, source, identifier and sometimes an embedding, which is a list of numbers that represents meaning in a machine-usable form.
During indexing, the system turns the content into a structure built for search rather than for reading. In keyword search, that often means a table of which words appear in which documents. In semantic search, it often means storing embeddings in a way that allows fast nearest-neighbour search, which means finding vectors that are mathematically closest to the query vector.
The output is an index: a data structure optimised for retrieval. When a user asks a question, the system searches the index instead of scanning every document from the beginning each time. That saves time and makes large collections practical to search.
Indexing is related to retrieval, but it is not the same thing. Retrieval is the act of fetching likely relevant items at query time. Indexing happens earlier, usually when documents are added or updated, so retrieval can work efficiently later.
In retrieval-augmented generation, indexing matters because poor indexing leads to poor evidence. If chunks are too large, too small, missing metadata or embedded badly, the model may receive weak source passages even if the answer exists somewhere in the knowledge base.
How “The lost-property register” maps onto the real thing
| In the story | In the AI system |
|---|---|
| sections of long reports | document chunks prepared for search |
| cards in a searchable cabinet | the index data structure |
| labels with source details | metadata such as title, section and document ID |
| librarian checking the cabinet first | retrieval using the index instead of scanning all text |
A worked example
A reporting officer in a liaison office needs to locate evidence across applications against the experience and language requirements in a rostered profile.
Starting from: The approved recruitment environment contains the profile, application forms and supporting attachments. One criterion concerns experience drafting multilingual situation reports.
- The system parses each application and separates relevant fields and attachments into searchable sections.
- Each section is stored with metadata linking it to the application record, document type and form field.
- A lexical index maps terms to sections. An embedding-based index stores a vector for each section and arranges the vectors for efficient similarity search.
- The officer searches for evidence of drafting reporting products in more than one language. The system retrieves sections containing exact terms and passages with similar meaning.
- The officer opens each retrieved passage in its full application context, checks it against the rostered profile and records the assessment through the approved process.
Result: The index reduces the material that must be opened initially and preserves links to the source application. It supports evidence-finding, not the staffing decision.
In a United Nations workplace
A reporting officer in a liaison office must screen a long list of applications against a rostered profile before sending the file to the staffing focal point at close of business.
Indexing makes the application fields and supporting passages searchable. The officer can retrieve likely evidence for each criterion without opening every attachment in turn, but the index does not decide whether the evidence meets the requirement.
Before you try thisApplications contain personal and HR-sensitive data. They must not be uploaded to or indexed in an external or otherwise unapproved tool. Any index must remain in an approved environment with appropriate access, retention and deletion controls. Retrieval must not infer protected characteristics, rank applicants or make eligibility or selection decisions. The responsible officer must review the complete application and follow the approved process.
Where you may hear it
- A vendor demonstration to an ICT governance board
- A procurement meeting reviewing search requirements for an internal platform
- A technical review of a retrieval-augmented generation prototype
- A knowledge management meeting before documents are uploaded to the portal
Avoid confusion
Avoid confusion
Often confused with
Vector database
A specialised system stores numerical meaning-representations of content and quickly finds the closest matches to a query by semantic similarity.
Retrieval
Retrieval returns passages that a search method ranks as relevant, so a person or later model step can use source material.
Keyword search
A search method that finds documents by matching query words to indexed text and ranking the matches, often with scoring methods such as BM25.
Common misconceptions
Common misconception: Indexing and retrieval are the same step.
Correction: They are connected but different. Indexing usually happens before questions are asked, turning documents into a search-ready structure. Retrieval happens at query time, using that structure to fetch likely relevant items. A system can have a well-built index and still retrieve poorly if the query handling or ranking is weak.
Common misconception: An index is only a list of keywords.
Correction: That describes one common kind of index, but not the whole idea. Modern systems may index keywords, metadata and embeddings together. In semantic search, the index often supports fast comparison of vectors so the system can find passages with similar meaning, not just the same words.
Common misconception: If content is indexed, the system has checked that it is correct.
Correction: Indexing improves findability, not truth. The system stores and organises content so it can be searched quickly. It does not automatically verify whether a document is accurate, current, authorised or appropriate for the user's context.
Quick check
Which statement best describes indexing in a semantic search or RAG pipeline?
Indexing is the preparation stage. It organises chunks, metadata and often embeddings into a search-ready structure. It is not the same as generation, factual verification or a specific product category such as a vector database.
Go deeper
Go deeper
The simple version
Indexing is the preparation work that organises documents so a search system can quickly find useful passages later.
The technical version
Indexing is the process of transforming a corpus into data structures optimised for retrieval. In lexical systems, this often means building structures such as inverted indexes over terms and document fields. In embedding-based systems, it usually means generating vectors for chunks, storing them with metadata and arranging them for efficient nearest-neighbour search, sometimes approximately rather than exactly for speed at scale. The indexed unit may be a full document, a chunk or another segment chosen during ingestion. The quality of indexing depends on choices such as chunking strategy, metadata design, embedding model, update policy and whether filters or reranking will be applied later. It is distinct from the knowledge base itself, which is the content collection, and from the vector database, which is one possible storage and query layer that may hold an index.
Reviewed external resources
EncyclopaediaSearch engine indexing (opens in a new tab)Wikipedia
This link gives a broad, non-specialist overview of search engine indexing and suits readers who want the general idea before AI-specific details.
Intermediatelink checked 2026-07-31
CourseEmbeddings (opens in a new tab)Google
This link explains embeddings, which suits readers who need to understand the meaning-based representations often used during indexing.
Intermediatelink checked 2026-07-31
InteractiveEmbedding Projector (opens in a new tab)TensorFlow
This interactive tool lets readers explore how embeddings are arranged in space, which suits visual learners who want intuition for semantic indexing.
Starterlink checked 2026-07-31
VideoVideo explanations: Indexing (opens in a new tab)YouTube search
This is a YouTube search results page for video explanations of indexing, which suits readers who prefer to compare several teaching styles rather than rely on one source.
search results, not one pagelink checked 2026-07-31
DiscussionCommunity discussion: Indexing (opens in a new tab)Reddit search
This is a Reddit search results page for community discussion of indexing, which suits readers who want informal questions, practical examples and varied viewpoints to inspect carefully.
search results, not one pagelink checked 2026-07-31
Where this came from
Drafted with a language model through an offline pipeline, checked against this site's schema, and every external link above fetched to see whether it answers. Treat it as a starting point and check anything that matters against the primary sources. How this site is built and what that means.
The one thing to remember
Indexing prepares documents, sections or other units for fast retrieval. It helps a system find likely passages, but does not judge whether they are correct, sufficient or current.