A Posteriori Agreement as a Quality Measure for Readability Prediction Systems

  • Philip van Oosten
  • Véronique Hoste
  • Dries Tanghe
Conference paper

DOI: 10.1007/978-3-642-19437-5_35

Part of the Lecture Notes in Computer Science book series (LNCS, volume 6609)
Cite this paper as:
van Oosten P., Hoste V., Tanghe D. (2011) A Posteriori Agreement as a Quality Measure for Readability Prediction Systems. In: Gelbukh A. (eds) Computational Linguistics and Intelligent Text Processing. CICLing 2011. Lecture Notes in Computer Science, vol 6609. Springer, Berlin, Heidelberg

Abstract

All readability research is ultimately concerned with the research question whether it is possible for a prediction system to automatically determine the level of readability of an unseen text. A significant problem for such a system is that readability might depend in part on the reader. If different readers assess the readability of texts in fundamentally different ways, there is insufficient a priori agreement to justify the correctness of a readability prediction system based on the texts assessed by those readers. We built a data set of readability assessments by expert readers. We clustered the experts into groups with greater a priori agreement and then measured for each group whether classifiers trained only on data from this group exhibited a classification bias. As this was found to be the case, the classification mechanism cannot be unproblematically generalized to a different user group.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Philip van Oosten
    • 1
    • 2
  • Véronique Hoste
    • 1
    • 2
  • Dries Tanghe
    • 1
    • 2
  1. 1.LT3 Language and Translation Technology TeamUniversity College GhentGhentBelgium
  2. 2.Ghent UniversityGhentBelgium

Personalised recommendations