Attention and transformers
The attention mechanism, the transformer architecture, and why it dominates modern NLP
Open slidesOverview
Lecture 6 ended on a cliffhanger: one vector per word means “apple” the fruit and “Apple” the firm share a representation. This lecture covers the mechanism that breaks that limit — self-attention — and the architecture built around it. Each word scores every other word for relevance (dot products), softmaxes the scores into weights, and rebuilds itself as a weighted average: context flows into the representation. Add the causal mask (no peeking at the future), multiple heads, residual connections and a feed-forward layer, and you have the transformer block; stack it deep, put a softmax on top, and you have a GPT — trained with the same next-token loss as lecture 4’s bigram model. Nothing in the math is new to you: it’s lectures 2, 4, and 6 wired together.
Topics
- The context problem: why static embeddings can’t disambiguate
- Self-attention: query/key/value, scaled dot products, softmax, weighted sums
- The causal mask, multi-head attention, and why parallelism enabled scale
- The transformer block: residuals, layer norm, feed-forward — stacked deep
- Positional encodings: without them, attention is a bag-of-words again
Readings
- J&M chapter 8
Asynchronous work
- Jay Alammar, The Illustrated Transformer and Visualizing neural machine translation — read before the lecture if you can; they make everything easier
Materials
- Slides: use the Open slides button above
- No dedicated notebook — the slides include a fully worked numeric attention example (backup slides), and pre-trained transformers get hands-on treatment on practical day 2
Before lecture 9
- Hugging Face NLP course, chapters 1–3 — the asynchronous task for the training and fine-tuning lecture