Brill tagger

The Brill tagger is a method for doing part-of-speech tagging. It was described by Eric Brill in his 1993 PhD thesis [http://www.cs.jhu.edu/~brill/dissertation.ps] . It can be summarized as an "error-driven transformation-based tagger". It is
* error-driven in the sense that it recourses to supervised learning
* transformation-based in the sense that a tag is assigned to each word and changed using a set of predefined rules. Note: If the word is known, it first assigns the most frequent tag, or if the word is unknown, it naively assigns the tag "noun" to it. Applying over and over these rules, changing the incorrect tags, a quite high accuracy is achieved.

Algorithm

The algorithm goes as follows:
* Initialisation:
** Known words (in vocabulary): assigning the most frequent tag associated to a form of the word
** Unknown words (out of vocabulary) :
*** Proper noun if capitalised and simple noun else (1992)
*** Learning or guessing rules on the same basis as contextual rules (1994)
* Learning Phase
** Iteratively compute the error score of each candidate rule (difference between the number of errors before and after applying the rule)
** Select the best (higher score) rule.
** Add it to the rule set and apply it to the text.
** Repeat until no rule has a score above a given threshold (that is, until applying new rules leaves the text in the same state, which is then supposed to be the final state of the tagging).

Rules

Lexical rules are used for the initialisation, and contextual rules are used to correct the tags.
*Lexical rules: "word" → "tag" IF "Condition" (example: identification of suffixes like "-tion")
*Contextual rules: "tag1" → "tag2" IF "Condition" (example: "preceding/following tag is "X", "preceding/following word is "w")

Code

Brill's code pages at John Hopkins University are no longer on the web. A mirror of the Brill tagger at its latest version is available at Plymouth Tech, here. [http://www.tech.plym.ac.uk/soc/staff/guidbugm/software/RULE_BASED_TAGGER_V.1.14.tar.Z]