Attention and Transformers

<p class="kicker">GRA 4164 · Lecture 8 · Fall 2026</p>

## Attention and<br>Transformers

### Leif Anders Thorsrud & Vegard H. Larsen — BI Norwegian Business School

---

One vector per word was the limit. **Attention** breaks it.

"Attention Is All You Need" — Vaswani et al., 2017.

---

## You already have the pieces

- **Words as vectors** — lecture 6
- **Similarity = dot products / cosine** — lecture 2
- **Next-token prediction + softmax** — lecture 4
- Today just **wires them together**

---

## The problem

---

## Same word, different meanings

![Two contexts pulling the word bank toward water-words or credit-words](../figures/contextual-bank.svg)

---

## What a transformer does

- Builds **richer, contextualised representations** of each position, layer by layer
- Each word's vector is **updated using the other words** — nearby or not
- The mechanism that decides *which* words matter: **self-attention**

---

## Self-attention

---

## The idea, before the math

- For each word: **score** every other word for relevance — dot products
- **Softmax** the scores into weights that sum to one
- New vector = **weighted average** of the others' vectors

"bank" attends to "water" → the new "bank" vector contains water.

---

## Query, Key, Value: a database analogy

<div class="cols">
<div class="callout">
<h3>Query</h3>
<p>What I'm looking for — the word currently updating itself.</p>
</div>
<div class="callout">
<h3>Key</h3>
<p>What each word offers to be matched against.</p>
</div>
</div>

**Value** — the content actually retrieved when a key matches the query.

</div>

---

## The math

$$\boldsymbol{q}\_i = \boldsymbol{x}\_i W^Q \qquad \boldsymbol{k}\_i = \boldsymbol{x}\_i W^K \qquad \boldsymbol{v}\_i = \boldsymbol{x}\_i W^V$$

$$\text{score}(i,j) = \frac{\boldsymbol{q}\_i \cdot \boldsymbol{k}\_j}{\sqrt{d\_k}} \qquad \alpha\_{ij} = \text{softmax}\_j\big(\text{score}(i,j)\big)$$

$$\boldsymbol{\alpha}\_i = \sum\_{j \le i} \alpha\_{ij} \, \boldsymbol{v}\_j$$

Typical sizes: $d = 512$, $d\_k = d\_v = 64$.

---

## No peeking: the causal mask

![Lower-triangular attention mask: each position attends only to itself and the past](../figures/causal-mask.svg)

---

## Many heads, many relationships

- Run **several attention mechanisms in parallel** — each with its own $W^Q, W^K, W^V$
- Different heads learn different patterns: syntax, coreference, topic…
- Concatenate the heads' outputs → one vector again

---

## Why this won

- **Long range** — relevance is not limited by distance in the sequence
- **Parallel** — all positions computed at once: GPUs love it
- **Interpretable-ish** — attention weights show where the model looked

</div>

---

## The transformer block

---

## Attention, packaged

![Transformer block: multi-head attention, add and norm, feed-forward, add and norm](../figures/transformer-block.svg)

---

## The block, in equations

$$T^1 = \text{MultiHeadAttention}(X) \qquad T^2 = T^1 + X$$

$$T^3 = \text{LayerNorm}(T^2) \qquad T^4 = \text{FFN}(T^3)$$

$$H = \text{LayerNorm}(T^4 + T^3)$$

---

## One thing attention can't see: order

- Attention is a weighted **average** — shuffle the words, same result
- Without help, the transformer is a **bag-of-words all over again**
- Fix: add **positional encodings** to the input embeddings

---

## From block to GPT

- **Stack** the block 12–100+ times
- Top: a **softmax over the vocabulary** — predict the next token
- Trained with the **same loss** as lecture 4's bigram model

---

## In sum

- Context problem: one vector per word **can't disambiguate**
- Self-attention: **score (Q·K), softmax, weighted average of V**
- Causal mask keeps prediction honest; **heads** add variety
- Block = attention + residuals + norm + FFN, **stacked deep**

</div>

---

## Questions

## Attention by hand

Three tokens, $d = 2$, $W^Q = W^K = W^V = I$, $x\_1 = (1,0)$, $x\_2 = (0,1)$, $x\_3 = (1,1)$:

- Scores for position 3: $\frac{(1, 1, 2)}{\sqrt{2}} = (0.71,\, 0.71,\, 1.41)$
- Softmax → weights $(0.25,\, 0.25,\, 0.50)$
- $\boldsymbol{\alpha}\_3 = 0.25\,x\_1 + 0.25\,x\_2 + 0.50\,x\_3 = (0.75,\, 0.75)$

## Causal vs bidirectional

<div class="cols">
<div class="callout">
<h3>GPT-style (causal)</h3>
<p>Mask the future. Trained to generate; what this course's LLM block is about.</p>
</div>
<div class="callout">
<h3>BERT-style (bidirectional)</h3>
<p>See both sides; trained by masking random words. Great encoder for classification and search.</p>
</div>
</div>

## Where the pictures came from

- The intuition/figure canon: **J&M chapter 8** (free online) — read it with the slides
- **Jay Alammar, *The Illustrated Transformer*** — the asynchronous task, animated
- Vaswani et al. (2017), *Attention Is All You Need* — the source

---

## What's next

- **Lecture 9 (Mon):** training and fine-tuning — pre-training to assistant, the full pipeline
- **Before then:** Hugging Face NLP course, **chapters 1–3**
- Reading: J&M chapter 8