GRA 4164
<!-- .slide: class="title-slide" --> <p class="kicker">GRA 4164 · Lecture 3 · Fall 2026</p> ## Bag-of-words-based<br>Methods ### Leif Anders Thorsrud & Vegard H. Larsen — BI Norwegian Business School --- <!-- Section A — Opening --> <!-- .slide: class="big-statement" --> You can turn text into a matrix. Now the matrix starts **working**. One matrix — four method families. <!-- .element: class="muted" --> --- ## The map  --- <!-- Section 01 — Dictionaries and Booleans --> <!-- .slide: class="section-divider" data-background-color="#282828" --> <p class="kicker">01</p> ## Dictionaries & Booleans --- ## The workhorse of economics & finance <div class="cols"> <div class="callout"> <h3>Dictionary</h3> <p>Lists of positive and negative words → score = #pos − #neg, normalised. (General Inquirer; Loughran-McDonald for finance.)</p> </div> <div class="callout"> <h3>Boolean</h3> <p>Chosen terms + logic: count articles matching "uncertainty" OR "uncertainties" AND "policy".</p> </div> </div> $\hat{v}\_i = f(\boldsymbol{c}\_i)$ — you **specify** $f$, the concept $v\_i$ is latent. <!-- .element: class="muted" --> --- ## It works: three famous series - **Tetlock (2007/2008)** — WSJ pessimism predicts short-horizon returns - **Baker, Bloom & Davis (2016)** — the Economic Policy Uncertainty index: newspaper Boolean counts, now a standard macro indicator - **Engle et al. (2020)** — your reading: climate-news index → hedge portfolios Word counting, top journals. <!-- .element: class="muted" --> --- ## The catch - **Subjective** — you choose the words; someone else would choose differently - You must **know what to look for** before you look - **Polysemy** — "bank", "liability": same word, different meanings - **Synonyms** — concept spread across words your list misses --- <!-- Section 02 — Text regression --> <!-- .slide: class="section-divider" data-background-color="#282828" --> <p class="kicker">02</p> ## Text regression --- ## Let the data choose the words - Observed outcome $y\_i$: returns, ratings, unemployment… - Regress on the document-term matrix: estimate $E[y\_i \mid \boldsymbol{c}\_i]$ - One problem: $\boldsymbol{C}$ is $n \times p$ with $n \ll p$ — **OLS fails** --- ## The lasso $$\min\_{\alpha, \boldsymbol{\beta}} \;\; \sum\_{i=1}^{n}\left(y\_i - \alpha - \boldsymbol{c}\_i'\boldsymbol{\beta}\right)^2 \;+\; n\lambda \sum\_{j=1}^{p} \omega\_j |\beta\_j|$$ - Penalty pushes most $\beta\_j$ to **exactly zero** — automatic word selection - $\lambda$ chosen by **cross-validation**; $\omega\_j$ can upweight rare words --- <!-- Section 03 — Naive Bayes --> <!-- .slide: class="section-divider" data-background-color="#282828" --> <p class="kicker">03</p> ## Naive Bayes --- ## Classifying documents - Many outcomes are **categories**: spam / not spam, up / down, complaint / praise - **Naive Bayes**: decades-old, embarrassingly simple, still a strong baseline - *Naive* = words independent given the class (bag-of-words again) · *Bayes* = Bayes' rule --- ## A worked example | | Document | Words | Class | |---|---|---|---| | train | 1 | growth high demand | **a** | | train | 2 | demand supply chain | **a** | | train | 3 | demand demand demand | **a** | | train | 4 | demand inflation norway | **b** | | test | 5 | inflation norway high | **?** | --- ## The classifier in four lines $$P(v \mid d) = \frac{P(d \mid v)\,P(v)}{P(d)} \;\;\Rightarrow\;\; v\_{nb} = \arg\max\_{v}\; P(v) \prod\_{j} P(c\_j \mid v)$$ $$\hat{P}(v) = \frac{n\_v}{n} \qquad \hat{P}(c\_j \mid v) = \frac{\text{count}(c\_j, v) + 1}{\sum\_c \text{count}(c, v) + p}$$ Bayes' rule → independence → **counting**, with +1 smoothing. <!-- .element: class="muted" --> --- ## Run the numbers - Priors: $\hat{P}(a) = \frac{3}{4}$, $\hat{P}(b) = \frac{1}{4}$ - $\hat{P}(\text{inflation} \mid a) = \frac{1}{16}$, $\hat{P}(\text{norway} \mid a) = \frac{1}{16}$, $\hat{P}(\text{high} \mid a) = \frac{1}{8}$ - $\hat{P}(\text{inflation} \mid b) = \frac{1}{5}$, $\hat{P}(\text{norway} \mid b) = \frac{1}{5}$, $\hat{P}(\text{high} \mid b) = \frac{1}{10}$ - Scores: $a \propto \frac{3}{8192} \approx 0.00037$ · $b \propto \frac{1}{1000} = 0.001$ → $\;P(b \mid d\_5) = \mathbf{0.73}$ --- ## Beyond Naive Bayes - Logistic regression, SVMs, neural nets — all standard, all off-the-shelf - Same recipe everywhere: **train / validate / test**, cross-validate the tuning - Naive Bayes remains the **baseline to beat** --- <!-- Section 04 — Topic models --> <!-- .slide: class="section-divider" data-background-color="#282828" --> <p class="kicker">04</p> ## Topic models --- ## What is the corpus about? - No outcome, no dictionary — just text - Want a **parsimonious** answer: a handful of **topics**, not 10,000 words - Each document = a **mixture** of topics --- ## One factorization, two flavours  --- ## LSA: just PCA on the matrix $$\boldsymbol{C} \approx \underbrace{\boldsymbol{U}\_{1:K}\boldsymbol{S}\_{1:K}}\_{F\;(n \times K)} \; \underbrace{\boldsymbol{V}'\_{1:K}}\_{L\;(K \times p)}$$ - Truncated **SVD**: keep the $K$ components explaining most variance - Similar documents → similar **rows of $F$**; similar words → similar **columns of $L$** --- ## What the reduction buys <div class="checklist"> - **Interpretation** — $K$ topics instead of $p$ words - **Noise reduction** — sparse 10,000-dim rows become dense $K$-dim rows - **Synonyms** — "good" and "fine" load on the **same topic** - **Polysemy** — "check" loads on **several topics** </div> --- ## LDA: make it probabilistic - A **topic** = a probability distribution over words ($\boldsymbol{\theta}\_l$) - A **document** = a probability distribution over topics ($\boldsymbol{v}\_i$) - Generative recipe: draw the doc's topic mix → for each word, draw a topic, then a word from it - Blei et al. (2003) — the most-used topic model in the field --- ## One article, three topics  - A Nissan press story — volcano halts parts, production stops - LDA reads it as a mix: **Motor** + **Natural disasters** + **Outlook** --- ## In sum <div class="checklist"> - **Dictionaries** — you choose the words; simple, transparent, subjective - **Text regression** — the data chooses; lasso handles $n \ll p$ - **Naive Bayes** — counting + Bayes' rule; the classification baseline - **Topic models** — $\boldsymbol{C} \approx F \times L$; what the corpus is about </div> --- <!-- .slide: class="big-statement" --> ## Questions -- ## The wider regression family - **Lasso family**: ridge ($\beta^2$ penalty, no exact zeros), elastic net (both) - **Dimension reduction first**: principal-components regression, partial least squares - **Non-linear**: GLMs, regression trees, random forests - Reference: Hastie, Tibshirani & Friedman, *Elements of Statistical Learning* -- ## Why PCA = SVD $$\text{cov}(\boldsymbol{X}) = \frac{\boldsymbol{X}'\boldsymbol{X}}{n-1} = \boldsymbol{V}\frac{\boldsymbol{S}^2}{n-1}\boldsymbol{V}' \;\Rightarrow\; \lambda\_l = \frac{s\_l^2}{n-1}$$ $$F = \boldsymbol{X}\boldsymbol{V} = \boldsymbol{U}\boldsymbol{S}\boldsymbol{V}'\boldsymbol{V} = \boldsymbol{U}\boldsymbol{S}$$ -- ## The multinomial, quickly $$P(X\_1 = x\_1, \ldots, X\_k = x\_k) = \frac{m!}{x\_1! \cdots x\_k!} \, q\_1^{x\_1} \cdots q\_k^{x\_k}$$ - Word counts in a document: $m$ draws from an urn of words - LDA's likelihood: $\boldsymbol{c}\_i \sim \text{MN}(\boldsymbol{v}\_i\boldsymbol{\Theta},\, m\_i)$ -- ## LDA's priors and sparsity - $\boldsymbol{\theta}\_l \sim \text{Dir}(\beta)$ — small $\beta$: topics use **few words** - $\boldsymbol{v}\_i \sim \text{Dir}(\alpha)$ — small $\alpha$: documents use **few topics** - Conventional defaults: $\alpha = 50/K$, $\beta = 0.01$ (Griffiths & Steyvers 2004) - Choosing $K$: held-out likelihood, or judgement --- ## What's next - **Lecture 4 (Tue):** learning from prediction — n-gram language models, the door to LLMs - **Before then:** watch **Karpathy's "building makemore"** video - Practical day 1 (Mon 26 Oct) runs today's menu in code — bring a laptop