TF-IDF
A scoring trick that highlights words that are common in a document but rare across all documents — so "urgent" in your email matters more than "the."
Think of it like
A librarian who knows which words are telling: "blood" in a medical journal is informative; "the" in any journal is noise.
Example
When you search Google, TF-IDF helps rank results: pages containing your rare search term score higher than pages stuffed with common words.
How it actually works
TF (term frequency) counts how often a word appears in a document; IDF (inverse document frequency) is a penalty for words that appear in many documents. Multiply them: a word gets high TF-IDF if it's frequent in one document but rare globally. It's a simple heuristic that works well for retrieval and classification without deep learning. Modern systems use learned embeddings instead, which capture semantic meaning beyond word statistics, but TF-IDF remains a solid baseline and is interpretable.
For product teams
A robust, interpretable baseline for search and classification; overtaken by learned methods but still useful.
For engineers
A term weighting scheme: TF-IDF(term, doc) = count(term, doc) × log(total_docs / docs_with_term).
Related
- Bag of words — An improvement on bag of words.
- Retrieval — Foundational to classic information retrieval.
Read anything AI without the jargon
Look up any term in plain English, or save terms as you read with the free Chrome extension.
Open DecoderAdd to Chrome