Lemmatisation

Lemmatisation

Lemmatisation is the process of grouping together the different inflected forms of a word so they can be analysed as a single item (cf. "Collins English Dictionary", entry for "lemmatise").

In computing, lemmatisation is the algorithmic process of determining the lemma for a given word. Since the process may involve complex tasks such as understanding context and determining the part of speech of a word in a sentence (requiring, for example, knowledge of the grammar of a language) it can be a hard task to implement a lemmatiser for a new language.

In many languages, words appear in several "inflected" forms. For example, in English, the verb 'to walk' may appear as 'walk', 'walked', 'walks', 'walking'. The base form, 'walk', that one might look up in a dictionary, is called the "lemma" for the word. The combination of the base form with the part of speech is often called the "lexeme" of the word.

Lemmatisation is closely related to stemming. The difference is that a stemmer operates on a single word without knowledge of the context, and therefore cannot discriminate between words which have different meanings depending on part of speech. However, stemmers are typically easier to implement and run faster, and the reduced accuracy may not matter for some applications.

For instance:

#The word "better" has "good" as its lemma, but this is missed in the stemming.
#The word "walk" is the base form for word "walking", and hence this is matched in both stemming and lemmatization.

Analysers like [http://lucene.apache.org/java/docs/lucene-sandbox Lucene Snowball] store the base stemmed format of the word without the knowledge of meaning, but taking into account the semantics of the word formation only. The stemmed word itself might not be valid. (see lazi below)

The following is an example of lemmatisation and stemming. Given the following sentence:

[The] [quick] [brown] [fox] [jumped] [over] [the] [lazy] [dogs]

org.apache.lucene.analysis.snowball.SnowballAnalyzer gives the following stems:

[quick] [brown] [fox] [jump] [over] [lazi] [dog]

the lemmas from the words in the sentence would be as follows:

[the] [quick] [brown] [fox] [jump] [over] [the] [lazy] [dog]

External links

* [http://ofset.sourceforge.net/freeduc/book/book_27.html "Collatinus", an open-source lemmatiser for latin language] and [http://collatinus.fltr.ucl.ac.be/collatinus.php its adaptation for Internet]


Wikimedia Foundation. 2010.

Игры ⚽ Поможем написать курсовую

Look at other dictionaries:

  • Lemmatisation — La lemmatisation désigne l analyse lexicale du contenu d un texte regroupant les mots d une même famille. Chacun des mots d un contenu se trouve ainsi réduit en une entité appelée lemme (forme canonique). La lemmatisation regroupe les différentes …   Wikipédia en Français

  • lemmatisation — lemmatiser [ lematize ] v. tr. <conjug. : 1> • v. 1970; de l angl. to lemmatise ♦ Ling. Donner à (un mot variable [accord, conjugaison, etc.] du discours) une forme canonique servant d entrée de dictionnaire. N. f. LEMMATISATION .… …   Encyclopédie Universelle

  • lemmatisation — noun The process of finding the lemma that corresponds to an inflected form of a word See Also: lemmatise, lemmatiser …   Wiktionary

  • lemmatisation — lemmatizāˈtion or lemmatisāˈtion noun • • • Main Entry: ↑lemma …   Useful english dictionary

  • Racinisation — La racinisation (ou désuffixation, ou stemming en anglais) est le nom donné au procédé qui vise à transformer les flexions en leur radical ou stemme. Il cherche à rassembler les différentes variantes flexionnelle et dérivationnelle d’un mot… …   Wikipédia en Français

  • Stemming — is the process for reducing inflected (or sometimes derived) words to their stem, base or root form ndash; generally a written word form. The stem need not be identical to the morphological root of the word; it is usually sufficient that related… …   Wikipedia

  • lemme — [ lɛm ] n. m. • 1613; lat. imp. lemma; mot gr.; → dilemme 1 ♦ Math. Résultat intermédiaire utilisé en cours de raisonnement lors d une longue démonstration. 2 ♦ Philos. Proposition accessoire, démontrée ou admise, qui permet de poursuivre le… …   Encyclopédie Universelle

  • Lemma (linguistics) — In linguistics a lemma (plural lemmas or lemmata ) has two distinct interpretations: # morphology / lexicography: the canonical form or citation form of a set of forms (headword); e.g. in English, run , runs , ran and running are forms of the… …   Wikipedia

  • Richard Goulet — est né à Montréal, au Québec, en 1943. Il est directeur de recherche émérite au Centre national de la recherche scientifique (CNRS) ― U.P.R. 76 : « Histoire des doctrines de la fin de l Antiquité Année Philologique »[1]. Il a… …   Wikipédia en Français

  • LEXICOMÉTRIE (domaine français) — LEXICOMÉTRIE, domaine français Depuis Zipf et Yule (1930 1950) et, en France, P. Guiraud (1954, 1960), l’étude quantitative du vocabulaire, qu’on la nomme «statistique lexicale» ou «lexicométrie», a beaucoup évolué. Elle est devenue plus… …   Encyclopédie Universelle

Share the article and excerpts

Direct link
Do a right-click on the link above
and select “Copy Link”