Introduction to LLMs and tokenization
A critical introduction to large language models, and why tokenization shapes everything they can and cannot do
Open slidesOverview
You already built a language model in lecture 4 — an LLM is the same idea at industrial scale: a transformer trained to predict the next token, with the same loss function. This lecture opens the LLM block with a critical introduction (what these models are, what they’re used for, and the open problems — reliability, bias, privacy, concentration — that should make you an informed user), then goes hands-on with the one component you can fully understand today: tokenization. From Unicode and UTF-8 through byte-pair encoding, we see exactly what a model reads — and why that explains its strangest failures, from counting the r’s in strawberry to charging more for Norwegian than for English.
Topics
- What an LLM is: two files, a transformer, next-token prediction at scale
- From raw model to assistant: the training pipeline (map only — details in lecture 9)
- Open problems: hallucinations, bias, privacy, market concentration
- Unicode, UTF-8, and stemming — the road to byte-pair encoding
- BPE in practice: real token splits, costs, and context windows
Readings
- J&M chapters 2 and 7
Asynchronous work
- Work through an introduction to the Hugging Face ecosystem
- Play with tiktokenizer.vercel.app — bring the weirdest split you find to lecture 8
Materials
- Slides: use the Open slides button above
- Notebook: llm_tokenizers.ipynb — stemming, Unicode, UTF-8, and the byte-pair-encoding walkthrough. No data files needed.
Before lecture 8
- Run the tokenizer notebook
- Read the Jay Alammar illustrated guides linked from lecture 8 — they make the attention lecture much easier