Lemmatization in NLP: Meaning, Examples & Stemming vs Lemmatization
Key Insights
- Lemmatization in NLP converts inflected words into their dictionary base forms, known as lemmas, using lexical resources, linguistic rules and, where supported, grammatical context.
- Unlike stemming, which typically reduces words through suffix-based rules, lemmatization aims to produce meaningful dictionary forms. The appropriate technique depends on the task and the level of linguistic precision required.
- Morphology and part-of-speech (POS) tagging help lemmatizers identify the correct base form, particularly when a word’s grammatical role affects its meaning.
- Python libraries such as NLTK and spaCy offer practical lemmatization tools, but their outputs depend on POS information, lexical resources and configuration.
- Lemmatization can support text classification, information retrieval and linguistic analysis, but it does not automatically improve every NLP pipeline.
- Modern language models often use subword tokenisation and may not require a separate lemmatization step. Preprocessing choices should be evaluated against the specific task.
Lemmatization is a natural language processing (NLP) technique that converts a word into its dictionary base form, called a lemma. For example, running can become run. By grouping related word forms, lemmatization in NLP can help systems analyse text consistently.
Imagine you are building a search system that needs to recognise connect, connecting, and connected as related forms of the same word. Should the system treat them as three separate terms, or bring them together?
This is one of the problems lemmatization helps address. However, lemmatization differs from stemming, which typically removes word endings using simpler rules. This article explains what lemmatization means, how morphology and part-of-speech (POS) tagging influence its results, and how to implement it in Python using NLTK and spaCy.
What is Lemmatization in NLP?
Lemmatization maps an inflected word to its canonical dictionary form using vocabulary resources, linguistic rules and, depending on the implementation, grammatical information.
Lemma meaning with examples
A lemma is the dictionary form used to represent a word and its grammatical variants. For example:
- Cars → car
- Running → run
- Better → good, when used as a comparative adjective
These lemmatization examples show why the process involves more than removing suffixes. The word better cannot reliably become good through simple suffix stripping; a lexical resource must recognise the relationship.
Here’s something to notice: if you remove the ending from better, you will not arrive at good. Yet good is the appropriate lemma when better is used as a comparative adjective. This is why lemmatization needs linguistic knowledge, not just a rule for cutting off suffixes.
Why Lemmatization matters?
Lemmatization can reduce vocabulary variation by grouping forms such as walk, walks and walked. This may help text classification, information retrieval and text analysis by making related forms easier to match.
However, it is not necessary for every NLP pipeline. Whether it helps depends on the task, model and preprocessing strategy.
Finding a lemma requires more than shortening a word. The next section explains the linguistic information and resources that support this process.
How does Lemmatization Work?
A lemmatization algorithm uses linguistic rules, vocabulary resources and sometimes grammatical information to identify a suitable base form. The exact process varies across tools and languages.
Morphology and word forms
Morphology in NLP examines how words change form and how they are constructed. Inflection changes a word to express grammatical information without creating a separate dictionary entry. For example, walks, walked and walking can share the lemma walk.
Word formation can create a different word with a related meaning. For instance, teach and teacher are related, but they do not share the same lemma. A lemmatizer must distinguish these relationships rather than treating every shared word fragment as equivalent.
The role of part-of-speech tagging
POS tagging in NLP identifies a word’s grammatical role, such as noun, verb or adjective. This matters because the same surface form can have different lemmas depending on how it is used.
Consider meeting. As a noun, it may remain meeting. As a verb, it can map to meet. A lemmatizer that uses grammatical context can make a more appropriate choice.
Quick check: What should meeting become?
- In “The meeting starts at noon,” it is a noun, so it may remain meeting.
- In “We are meeting tomorrow,” it functions as a verb, so it can map to meet.
The word has not changed its spelling, but its grammatical role has changed. That is why context matters.
Dictionaries, lexical resources, and rules
Dictionary-based lemmatization uses stored vocabulary information to identify base forms. Resources such as WordNet contain lexical relationships and irregular forms that simple suffix rules may miss.
Rule-based systems apply morphological patterns, often alongside POS tags. Lookup-based systems consult stored mappings. Some tools combine these approaches, and their results depend on the language resources and configuration available.
These mechanisms explain why lemmatization can produce meaningful dictionary forms, while stemming follows a different, usually simpler process.
Stemming vs Lemmatization: What is the Difference?
Stemming typically strips or transforms word endings using heuristic rules. Lemmatization aims to return a recognised dictionary form using lexical and, where supported, grammatical information.
| Aspect | Stemming | Lemmatization |
|---|---|---|
| Method | Applies word-reduction rules | Uses lexical resources, rules and sometimes POS |
| Output | May not be a valid word | A recognised lemma where supported |
| Context | Usually ignores grammatical role | May use POS and morphology |
| Speed | Generally faster | Often more computationally involved |
| Example | studies → studi with Porter stemming | studies → study |
| Typical use | Lightweight indexing and retrieval | Tasks where meaningful base forms matter |
Examples of stemming and lemmatization
| Word | Porter stemming example | Lemmatization example |
|---|---|---|
| studies | studi | study |
| cars | car | car |
| running | run | run, when identified as a verb |
| better | better | good, when identified as an adjective |
Outputs can vary with the stemming algorithm, POS information and lexical resource. The table illustrates typical results rather than a universal mapping for every tool.
Think about your use case: If you are building a lightweight search index and mainly want to group similar word forms, stemming may be sufficient. If you are analysing sentence meaning or need readable base forms, lemmatization may be more appropriate. The best choice depends on what your system needs to preserve.
What is the Porter Stemmer in NLP?
The Porter stemmer is a widely used English stemming algorithm that applies a series of suffix-stripping rules. It reduces related word forms to stems, which may not be valid dictionary words.
For example, the NLTK Porter stemmer can reduce studies to studi. This can be useful in lightweight text preprocessing when grouping word variants matters more than producing linguistically meaningful base forms. Unlike lemmatization, the Porter stemmer does not aim to identify a dictionary lemma.
The Porter stemmer illustrates how stemming works. Now, let’s see how lemmatization works in Python and how the output changes depending on the tool and grammatical information provided.
How to Perform Lemmatization in Python Using NLTK and spaCy
NLTK provides the WordNetLemmatizer, while spaCy supports pipeline-based lemmatization using lookup tables or rules. The appropriate method depends on your applicati
Lemmatization using NLTK
The following NLTK lemmatization example specifies the POS for each word. The WordNetLemmatizer defaults to noun processing (pos=”n”) when no POS is supplied.
import nltk
nltk.download("wordnet")
from nltk.stem import WordNetLemmatizer
lemmatizer = WordNetLemmatizer()
print(lemmatizer.lemmatize("cars"))
print(lemmatizer.lemmatize("running", pos="v"))
print(lemmatizer.lemmatize("better", pos="a"))Expected output:
car
run
good
The POS values used here are n for noun, v for verb and a for adjective. Without the appropriate POS, a word may remain unchanged or produce a different result. NLTK returns the input unchanged when it cannot find a suitable lemma for the specified POS.
Try changing the input: Run lemmatizer.lemmatize(“running”) without specifying a POS, and then compare it with lemmatizer.lemmatize(“running”, pos=”v”). Why does the second call return run? The difference illustrates why the default POS setting matters.
Lemmatization using spaCy
spaCy’s rule-based lemmatizer relies on grammatical annotations supplied by earlier pipeline components. Install spaCy and download its English model before running this example:
pip install spacy
python -m spacy download en_core_web_sm
import spacy
nlp = spacy.load("en_core_web_sm")doc = nlp(“I was reading the paper.”)
print([token.lemma_ for token in doc])
Expected output:
[‘I’, ‘be’, ‘read’, ‘the’, ‘paper’, ‘.’]
The pipeline identifies grammatical information and assigns lemmas to tokens. Notice that spaCy returns be for was and read for reading. If you use a different language model or lemmatizer configuration, check the output rather than assuming every pipeline will behave identically.
spaCy also supports lookup-based approaches, so the exact setup depends on the selected language and lemmatizer mode.
When should You Use Stemming vs Lemmatization?
Choose based on your task rather than assuming one technique is always better.
Before deciding, ask yourself three questions: Do you need valid dictionary forms? Does grammatical context matter to your task? Have you tested whether normalising the words improves your results? Your answers can help determine whether stemming, lemmatization, or neither is appropriate.
When stemming is appropriate
- Search indexing or retrieval where aggressive word reduction is acceptable.
- Lightweight preprocessing where speed and simplicity are priorities.
- Tasks where producing a valid dictionary form is less important than grouping word variants.
When lemmatization is appropriate
- Text analysis where meaningful word forms improve interpretability.
- Text classification that benefits from consistent treatment of inflected forms.
- Information retrieval where matching grammatical variants may help.
- Tasks that need irregular forms or grammatical distinctions to be handled appropriately.
Always evaluate the effect on your actual dataset. Normalising words can help in some cases but can also remove distinctions that matter.
Is Lemmatization Still Relevant in the LLM Era?
Lemmatization remains useful in selected NLP workflows, but modern language-model applications do not automatically need a separate lemmatization step.
It is tempting to assume that more preprocessing automatically leads to better results. But consider this: if a model needs grammatical information to interpret a sentence, could converting every word to its lemma remove something useful? This is why preprocessing decisions should be tested against the task rather than applied by default.
Where Lemmatization can still help
It may help search and retrieval systems, structured linguistic analysis, and traditional machine-learning pipelines where consistent vocabulary forms are useful. For example, a retrieval system may benefit from matching different inflections of a query term.
When it may be unnecessary
Transformer models commonly use subword tokenisation, which splits text into units between whole words and individual characters. These models can process many word variants without requiring every word to be converted into a lemma first.
Preprocessing can also remove grammatical or semantic information that a task needs. Therefore, lemmatization should be tested against the actual objective rather than assumed to improve model accuracy.
Conclusion
Lemmatization helps NLP systems convert words into meaningful dictionary forms using lexical resources, morphology and, where supported, part-of-speech tagging. Unlike stemming, which reduces words through suffix-based rules, lemmatization aims to preserve linguistic meaning while identifying the appropriate base form. Although modern NLP and LLM pipelines do not always require a separate lemmatization step, it remains useful for selected applications such as text classification, information retrieval and linguistic analysis.
To explore these concepts further, the Generative AI programme by IIT Delhi, delivered through Varsity by InterviewBit, covers NLP foundations alongside broader generative AI concepts. These topics help learners connect linguistic processing techniques with the development of modern AI systems.
Frequently Asked Questions
Does lemmatization change the meaning of a sentence?
Lemmatization aims to identify a word’s base form, but incorrect POS tagging or ambiguity can produce an unsuitable result. Always check whether the normalised form preserves distinctions relevant to your task.
Can lemmatization handle irregular verbs?
Yes, when the relevant lexical resource or rules support the irregular form. For example, went can map to go when it is correctly identified as a verb.
Why does NLTK sometimes leave a word unchanged?
WordNetLemmatizer defaults to noun processing and returns the input unchanged when it cannot find a suitable lemma for the specified POS. Supplying the correct POS can change the result.
Does lemmatization work for languages other than English?
Support depends on the library, language model and available lexical resources. Languages with complex morphological systems may require language-specific tools and rules.
Can two different words have the same lemma?
Yes. Inflected forms such as write, writes and wrote can map to write when their grammatical roles are identified appropriately.
Is lemmatization supervised or unsupervised?
Lemmatization is a linguistic processing task, not one fixed learning paradigm. Implementations may use dictionaries, rules, statistical methods or trained components.
What is the difference between a lemma and a root word?
A lemma is a word’s canonical dictionary form. A root is the core morpheme from which a word is formed. The concepts are related but are not interchangeable.
Can lemmatization improve search results?
It can help match inflected forms, but its impact depends on the language, query patterns, indexing strategy and evaluation results. It should be tested against the retrieval task.
Does spaCy require POS tagging for lemmatization?
spaCy’s rule-based lemmatizer requires prior POS annotations. Lookup-based lemmatization works differently, so the requirements depend on the selected pipeline and configuration.
Should I lemmatize text before passing it to an LLM?
Not necessarily. Lemmatization may help structured analysis or retrieval, but it can also remove grammatical information a language model needs. Choose preprocessing based on the task and evaluate the results.
