Manual Corpus Annotation: Giving Meaning to the Evaluation MetricsReport as inadecuate




Manual Corpus Annotation: Giving Meaning to the Evaluation Metrics - Download this document for free, or read online. Document in PDF available to download.

1 Equipe Hultech - Laboratoire GREYC - UMR6072 GREYC - Groupe de Recherche en Informatique, Image, Automatique et Instrumentation de Caen 2 Equipe CODAG - Laboratoire GREYC - UMR6072 GREYC - Groupe de Recherche en Informatique, Image, Automatique et Instrumentation de Caen 3 SEMAGRAMME - Semantic Analysis of Natural Language Inria Nancy - Grand Est, LORIA - NLPKD - Department of Natural Language Processing & Knowledge Discovery 4 RCLN LIPN - Laboratoire d-Informatique de Paris-Nord 5 INIST - Institut de l-information scientifique et technique 6 LNE- INM - Laboratoire National de Métrologie et d-Essais - Institut National de Métrologie 7 LIMSI - Laboratoire d-Informatique pour la Mécanique et les Sciences de l-Ingénieur

Abstract : Computing inter-annotator agreement measures on a manually annotated corpus is necessary to evaluate the reliability of its annotation. However, the interpretation of the obtained results is recognized as highly arbitrary. We describe in this article a method and a tool that we developed which -shuffles- a reference annotation according to different error paradigms, thereby creating artificial annotations with controlled errors. Agreement measures are computed on these corpora, and the obtained results are used to model the behavior of these measures and understand their actual meaning.

Keywords : evaluation inter-annotator agreement manual corpus annotation evaluation.





Author: Yann Mathet - Antoine Widlöcher - Karën Fort - Claire François - Olivier Galibert - Cyril Grouin - Juliette Kahn - Sophie Ross

Source: https://hal.archives-ouvertes.fr/



DOWNLOAD PDF




Related documents