Quarterly (March, June, September, December)
160 pp. per issue
6 3/4 x 10
2014 Impact factor:

Computational Linguistics

Paola Merlo, Editor
March 2007, Vol. 33, No. 1, Pages 3-8
(doi: 10.1162/coli.2007.33.1.3)
© 2007 Massachusetts Institute of Technology
Identifying Sources of Disagreement: Generalizability Theory in Manual Annotation Studies
Article PDF (56.64 KB)

Many annotation projects have shown that the quality of manual annotations often is not as good as would be desirable for reliable data analysis. Identifying the main sources responsible for poor annotation quality must thus be a major concern. Generalizability theory is a valuable tool for this purpose, because it allows for the differentiation and detailed analysis of factors that influence annotation quality. In this article we will present basic concepts of Generalizability Theory and give an example for its application based on published data.