Bag-of-words-based methods

Overview

The document-term matrix starts working. This lecture covers the four method families built on bag-of-words counts: dictionary and Boolean methods (count the words you chose — the workhorse behind Tetlock’s sentiment series and the Economic Policy Uncertainty index), text regression (let the lasso choose the words when outcomes are observed), Naive Bayes classification (Bayes’ rule plus counting — worked through by hand), and topic models (LSA and LDA: explain the big sparse matrix with a few interpretable topics). The running thread: each method fixes a weakness of the previous one.

Topics

• Dictionary and Boolean techniques: sentiment scores and uncertainty indices
• Why $n \ll p$ breaks OLS, and how the lasso fixes it
• Naive Bayes: derivation, Laplace smoothing, and a hand-computed example
• Topic models: the $C \approx F \times L$ factorization, LSA/SVD, and LDA

One news article, three topics — LDA reading a Nissan story as a mixture of Motor, Natural disasters, and Outlook (from Larsen & Thorsrud’s newspaper work):

Readings

• Lecture notes; Bholat et al. (2015); Baker, Bloom & Davis (2016)
• Discussed at the start: Engle et al. (2020) from lecture 2

Materials

• Slides: use the Open slides button above
• No dedicated notebook — the Shakespeare notebook from lecture 2 provides the matrix these methods run on, and practical day 1 runs TF-IDF and LDA end to end

Before lecture 4

• Watch Andrej Karpathy’s The spelled-out intro to language modeling — lecture 4’s notebooks follow it closely