TF-IDF and Cosine Similarity, Explained Without the Hand-Waving
A plain-language walk through the two ideas behind my movie recommender — why counting words the naive way fails, and what cosine similarity is really measuring.
Two movies can share the word 'the' a hundred times and have nothing in common. TF-IDF exists because raw word counts overvalue words that are everywhere and undervalue the ones that actually distinguish a document.
Term frequency, tempered by rarity
TF-IDF weights a word up when it appears often in one document, and down when it appears across many. 'Heist' in a caper film survives that filter; 'the' does not.
Similarity as an angle, not a distance
Cosine similarity ignores how long two documents are and asks only whether they point the same direction in vector space. That's exactly what you want for recommendations: a short synopsis and a long one about the same theme should still read as alike.
- #NLP
- #Recommenders
- #Scikit-learn
Keep reading