Less is More, More or Less... Finding the Optimal Threshold for Lexicalization in Chunking

Balazs Indig


Lexicalization of the input of sequential taggers has gone a long way since it was invented by Molina and Pla [4]. In this paper we thoroughly investigate the method introduced by Indig and Endredy [2] to find out ´ the best lexicalization level for chunking and to explore the behavior of different IOB representations. Both tasks are applied to the CoNLL-2000 dataset. Our goal is to introduce a transformation method to accommodate the parameters of the development set to the training set using their frequency distributions which other tasks like POS tagging or NER could benefit too.


Phrase chunking, IOB labels, multiple IOB representations, sequential tagging, CRF

Full Text: PDF