reduce words to their dictionary forms
lemmatization is a text normalization technique in natural language processing. lemmatization uses vocabulary and morphological analysis to remove affixes of words. for example, “building has floors” reduces to “build have floor” upon lemmatization.
lemmatization applications
lemmatization is often used for:
- information retrieval for expanding search criteria
- reducing dimensionality of problems in text classification, sentiment analysis, or topic modeling
lemmatization vs. stemming
a related approach to lemmatization, stemming, is based on simple heuristic rules. it often results in roots or word parts that are not actual words, whereas lemmatization always returns valid dictionary words.
examples of lemmatization and stemming are shown below.
| actual word | lemmatization | stemming |
|---|---|---|
| requirement | requirement | requir |
| applied | apply | appli |
in matlab®, lemmatization can be done using “normalizewords” function with the style option of ‘lemma’. to learn more about using lemmatization and building predictive models with text data with matlab, see text analytics toolbox™.
examples and how to
software reference
see also: natural language processing, sentiment analysis, word2vec, stemming, n-gram, text mining with matlab, data science, deep learning, deep learning toolbox™, statistics and machine learning toolbox™