Bag-of-words, TF-IDF, and cosine similarity
Turning text into numbers: document-term matrices, TF-IDF weighting, and comparing documents with cosine similarity
Open slidesOverview
Computers can’t read — so we count. This lecture makes the central move of classical text analysis: from a pile of documents to a document-term matrix, where every document is a vector of word counts. We weight the counts with TF-IDF so informative terms stand out, and compare documents with cosine similarity — measuring what a document is about, not how long it is. The running example asks which Shakespeare play is most similar to Hamlet; the closing case shows the same toolkit hedging climate risk in a top finance journal.
Topics
- Getting a corpus: extraction, storage, and what counts as a document
- Preprocessing: casing, punctuation, tokenization, stop words
- The document-term matrix and n-grams
- TF-IDF: definition and intuition
- Cosine similarity vs Euclidean distance — why direction beats length
Readings
- Lecture notes; Bholat et al. (2015)
- Before lecture 3: Engle et al. (2020), Hedging Climate Change News
Materials
- Slides: use the Open slides button above
- Notebook: Bow_tfidf_cosine_similarity.ipynb — the full pipeline on Shakespeare’s plays. Download shakespeare.txt (5 MB) and place it in a
data/folder next to the notebook. - Notebook: Extracting_data_from_the_web.ipynb — fetching and parsing HTML with BeautifulSoup. No data files needed.
Before lecture 3
- Run the Shakespeare notebook — the same pipeline runs on the assignment-1 corpus
- Read Engle et al. (2020) — we open lecture 3 with it