TF-IDF Explained: Formula, Worked Example & Python Code

TF-IDF (Term Frequency–Inverse Document Frequency) is a numerical weighting technique used in natural language processing (NLP) and information retrieval to measure how important a word is to a document relative to a collection of documents. It assigns greater weight to terms that appear frequently in a particular document but occur in fewer documents across the collection.

For example, words such as “the” or “is” may appear in almost every article, while terms such as “transformer” or “tokenisation” may help distinguish specific topics. TF-IDF helps a system differentiate between these terms based on their frequency and distribution. In this article, we’ll explore the TF-IDF formula, work through a numerical example, implement TF-IDF in Python using scikit-learn, and compare it with BM25 and word embeddings.

What is TF-IDF? Full Form, TF and IDF Explained

TF-IDF combines two signals: how often a term appears in a document and how many documents contain that term. It gives greater weight to terms that are frequent in a particular document but relatively uncommon across the collection.

What is Term Frequency (TF)?

Term frequency measures how often a word occurs in a document. One simple, normalised term frequency formula is:

If a document contains 100 words and a term appears five times, its TF is (5/100=0.05). Normalisation makes the score reflect the term’s share of the document rather than just its raw count.

What is Inverse Document Frequency (IDF)?

Document frequency in NLP refers to the number of documents containing a term, regardless of how many times it appears in each one. Inverse document frequency (IDF) gives more weight to terms that occur in fewer documents.

The basic IDF formula is:

Here, (N) is the total number of documents and (df(t)) is the number containing term (t). The natural logarithm is used in the calculations below. If a term appears in every document, its IDF is zero under this formula.

What does a TF-IDF Score Tell You?

TF-IDF multiplies the two values:

TF-IDF(t,d) = TF(t,d) × IDF(t)

A high score indicates that a term is relatively prominent in a document and uncommon across the collection. A low score suggests that the term is either infrequent in that document or widespread across the corpus. TF-IDF measures statistical importance, not the meaning of a word.

What is the TF-IDF Formula?

The basic TF-IDF formula combines normalised term frequency with inverse document frequency:

To understand how to calculate TF-IDF, follow these steps:

1. Count the term’s occurrences in the document.
2. Divide that count by the document’s total number of terms to obtain TF.
3. Count how many documents contain the term.
4. Calculate IDF using the total document count and document frequency.
5. Multiply TF by IDF.

The worked example uses the basic, unsmoothed formula and natural logarithms. Libraries such as scikit-learn use configurable implementations; by default, TfidfVectorizer smooths IDF weights and applies L2 normalisation. These differences affect the final scores.

TF-IDF Worked Example Using Three Documents

Consider this small corpus:

• Document 1: cat sat mat
• Document 2: cat ate fish
• Document 3: dog ate fish

We will calculate TF-IDF for terms in Document 1. Each document contains three terms, and the corpus contains three documents.

How do You Calculate TF-IDF for Document 1?

Term	Count in Document 1	TF	Documents containing term (DF)	IDF = ln(3/DF)	TF-IDF
cat	1	1/3 = 0.333	2	ln(3/2) = 0.405	0.135
sat	1	1/3 = 0.333	1	ln(3) = 1.099	0.366
mat	1	1/3 = 0.333	1	ln(3) = 1.099	0.366

For cat, TF is (1/3), while IDF is (\ln(3/2)). Multiplying them gives approximately (0.333 \times 0.405 = 0.135).

For sat, TF remains (1/3), but the term appears in only one document. Its IDF is (\ln(3)), producing a TF-IDF score of approximately (0.366). The same calculation applies to mat.

What does the Worked Example Demonstrate?

All three terms have identical TF values in Document 1, but sat and mat receive higher scores because they occur in fewer documents. This is the central idea behind TF-IDF: the IDF component distinguishes terms that are more specific to a document from those distributed across the corpus.

These values are illustrative, not universal. Different term-frequency definitions, logarithm conventions, smoothing and vector normalisation can produce different scores.

How to Calculate TF-IDF in Python Using scikit-learn

You can generate a TF-IDF matrix using TfidfVectorizer from scikit-learn. This approach is useful when working with larger collections of documents.

Python Code for TF-IDF

Install scikit-learn if it is not already available in your Python environment, then run:

from sklearn.feature_extraction.text import TfidfVectorizer
documents = [
    "cat sat mat",
    "cat ate fish",
    "dog ate fish"
]
vectorizer = TfidfVectorizer()
tfidf_matrix = vectorizer.fit_transform(documents)
print(vectorizer.get_feature_names_out())
print(tfidf_matrix.toarray())

The output contains a vocabulary and a numerical matrix. fit_transform() learns the vocabulary and IDF weights from the documents, then converts the text into TF-IDF features.

• Vocabulary: get_feature_names_out() returns the terms represented by the matrix columns.
• Rows: Each row corresponds to one document.
• Columns: Each column corresponds to a vocabulary term.
• Scores: Each value represents a term’s TF-IDF weight in a particular document.

The default output will not match the manual calculation exactly. Scikit-learn uses smoothed IDF by default and normalises each document vector using L2 normalisation. It also lowercases text by default and applies its tokenisation rules, which can affect the vocabulary.

TF-IDF vs BM25 vs Word Embeddings

TF-IDF, BM25 and word embeddings represent or score text differently. They are related tools, but they are not interchangeable.

Aspect	TF-IDF	BM25	Word embeddings
Representation	Weighted term features	Lexical relevance score	Dense numerical vectors
Main signal	Term frequency and document rarity	Term frequency, IDF and document-length adjustment	Semantic or contextual similarity, depending on the model
Exact keyword matching	Useful	Strong for lexical retrieval	May not reliably preserve exact terms
Semantic similarity	Limited	Limited	Often stronger
Common uses	Text features and keyword retrieval	Search ranking	Semantic search and dense retrieval

BM25 builds on related lexical weighting ideas, but adds term-frequency saturation and document-length normalisation. It is widely used in search systems; Elasticsearch, for example, documents BM25 as its default similarity method. See the official Elasticsearch reference.

Word embeddings represent text as dense vectors that can capture relationships beyond exact word overlap. For example, a semantic retrieval system may connect “car” with “automobile”, even if the query and document use different words. However, embeddings can still miss exact identifiers or technical terms. The right approach depends on the query, corpus and retrieval evaluation.

What are the Uses of TF-IDF in NLP and Search?

TF-IDF remains useful when a system needs interpretable text features or wants to identify terms that distinguish documents.

Information Retrieval and Keyword-Based Search

TF-IDF can help match documents against queries by weighting their terms. It provides a statistical relevance signal, but does not fully understand what a user means. Matching the right words is not always the same as answering the right question.

Text Classification and Keyword Extraction

TF-IDF features can be used with machine-learning models for tasks such as spam detection, topic classification and sentiment analysis. They can also help identify terms that distinguish one document or category from another. The results depend on the corpus, preprocessing and model.

TF-IDF in Modern RAG and Hybrid Retrieval

Retrieval-Augmented Generation (RAG) systems retrieve relevant information before passing it to a language model. Lexical retrieval can help find exact terms, product codes or technical identifiers, while dense retrieval can find semantically related content.

Hybrid search combines lexical and semantic retrieval to use both signals. In practical systems, BM25 is a common lexical retrieval method; TF-IDF is not the default ranking method in every RAG system. The best combination should be evaluated against the actual documents and queries.

What are the Limitations of TF-IDF?

TF-IDF captures word frequency and document-level rarity, but it does not inherently understand semantics, context or relationships between words.

Does TF-IDF Understand Meaning and Synonyms?

No. Standard TF-IDF treats words as separate features, so it does not inherently recognise that “car” and “automobile” can refer to the same concept. Preprocessing and n-gram features may improve some matching cases, but they do not provide general semantic understanding.

Does TF-IDF Consider Word Order and Context?

A standard unigram TF-IDF representation does not preserve word order or sentence meaning. Its usefulness can also be affected by rare terms, noisy text and document length, depending on the chosen weighting and normalisation. It is a useful baseline, but not a complete model of language.

Conclusion

TF-IDF is a foundational NLP technique that helps identify important terms in a document based on their frequency and distribution across a collection. By understanding the TF-IDF formula and working through a numerical example, you can interpret these scores and apply them using tools such as scikit-learn. While TF-IDF remains useful for text analysis, keyword extraction, and lexical retrieval, BM25 and word embeddings offer alternative approaches for information retrieval and semantic search.

Ready to build on these NLP foundations and explore how they power modern AI applications? Explore the Generative AI programme by IIT Delhi, delivered through Varsity by InterviewBit, to deepen your understanding of NLP and discover how these concepts connect with broader generative AI techniques.

Frequently Asked Questions

Is TF-IDF a machine learning algorithm?

No. TF-IDF is a feature-weighting technique, not a predictive model by itself. It is commonly used to prepare text features for machine-learning algorithms.

What Is the Difference Between TF-IDF and Bag of Words?

Bag of words represents documents using term counts or occurrences. TF-IDF reweights those terms according to how widely they appear across the document collection.

Can TF-IDF Scores Be Negative?

Standard TF-IDF scores are non-negative when calculated from non-negative term frequencies and conventional IDF formulas. Other transformations or downstream operations may produce negative values.

What Happens When a Word Appears in Every Document?

Its IDF is zero under the basic unsmoothed formula because (\ln(N/N)=\ln(1)=0). Its TF-IDF score is therefore also zero.

Does TF-IDF Require Stop-Word Removal?

No. Stop-word removal is optional and depends on the task. Common terms may already receive low weights, but preprocessing choices should be evaluated for the intended application.

Does TF-IDF Treat Uppercase and Lowercase Words Differently?

By default, scikit-learn’s TfidfVectorizer converts text to lowercase before tokenisation. This behaviour can be changed through its configuration.

Can TF-IDF Be Used for Short Documents?

Yes, but short texts may contain fewer overlapping terms and less context, making their scores less informative for some tasks.

What Is the Difference Between TF-IDF and Cosine Similarity?

TF-IDF creates weighted document vectors, while cosine similarity measures the angle-based similarity between vectors. Cosine similarity can be used to compare TF-IDF representations.

Does TF-IDF Need Labelled Training Data?

No. TF-IDF can learn its vocabulary and weights from an unlabelled collection of documents. Labelled data may be needed later if those features are used to train a supervised model.

Should TF-IDF Be Used Alone for a RAG Application?

Not necessarily. TF-IDF can support lexical retrieval, but the best setup depends on the corpus and query types. Evaluate lexical, dense and hybrid retrieval rather than assuming one method is always sufficient.

Resources

1. Scikit-learn – Feature Extraction: Text Feature Extraction and TF-IDF
2. Scikit-learn – TfidfVectorizer Documentation
3. Scikit-learn User Guide – Text Feature Extraction
4. Elasticsearch – Similarity and BM25 Documentation