Learn AI / Term Explorer /Transformers and Attention
Transformers and Attention
The architecture that made modern language models possible, explained without the mathematics.
Intermediate
Attention
A method that lets a model give different importance to different parts of the input while producing an output.
Embedding
A list of numbers that places a token or other item in a mathematical space so a model can compare patterns, relationships and context.
Transformer
A neural network design that processes many parts of a sequence in parallel by weighting which parts matter most to each other.
Vector and vector space
A vector is an ordered list of numbers, and a vector space is a set of same-length vectors that is closed under addition and scaling.
Deeper
Attention score
A number comparing one token’s query with a candidate token’s key during an attention step.
Autoregressive model
A model that assigns each token a probability based on earlier tokens in the same sequence, and generates by predicting and adding one new token at a time.
Encoder and decoder
In an encoder-decoder model, the encoder represents input, while the decoder uses those representations to generate output, often one token at a time.
Multi-head attention
A transformer mechanism that runs several attention patterns in parallel so the model can combine different kinds of relationships between tokens.
Positional encoding
A method that supplies sequence-position information so attention can use word order, not just token content.
Query, key and value
Three vectors produced by learned projections represent matching needs, matching features, and information that attention may pass onward.
Self-attention
A mechanism that lets each token reweight information from available tokens in the same sequence, including itself, before the next layer processes it.