Assignment 1: Boolean search, topic models, and sentiment
Do State of the Union speeches predict GDP growth? Boolean search, text regression, LDA topic modelling, and sentiment scoring on two centuries of speeches
The task
Working with the State of the Union corpus (one speech per year, 1790s–today) and US GDP growth, you investigate whether what presidents say relates to how the economy moves — using the lecture 1–3 toolkit.
Part 1 — Boolean search and text regression. Describe how Boolean search-based techniques combined with text regression can test the hypothesis that speech sentiment correlates with GDP growth — then run it. Interpret the regression output, test sensitivity to your term choices with at least five specifications, and discuss correlation vs. causation.
Part 2 — Topic modelling. Decompose the corpus with LDA: justify your number of topics and preprocessing choices, visualise topics as word clouds, analyse which topics each president emphasised, and investigate the correlation between topics and GDP growth.
Part 3 — Sentiment analysis. Each group member manually scores the first paragraph of a set of speeches (−1 to 1); compare scorers and illustrate the disagreement. Then score the same paragraphs with a sentiment dictionary (VADER, AFINN, or your own) and analyse how human and machine scores correlate — and where and why they diverge.
Materials
- Notebook: assignment1.ipynb — reference code for all three parts; you adapt it, you don’t submit it
- Data: sotu.zip (4 MB — unzip to
data/sotu/) and GDP.csv (place indata/)
Deliverable
A PDF report of at most 5 pages, with answers organised by part and question (1a, 1b, …) and figures referenced in the text. Groups of 1–3 students; the report becomes part of the collected submission at the end of the semester.
Builds on
Lectures 1–3; practical day 1 is the supervised work session for this assignment.