Skip to main content

Reliability

  • Chapter
  • 6 Accesses

Part of the Synthesis Lectures on Human Language Technologies book series (SLHLT)

Abstract

Reliability is closely intertwined with validity. On the one hand, reliability is a necessary, but not a sufficient, condition for validity. On the other hand, in the pursuit of high reliability, validity tends to get lost, for example, by oversimplifying measurement procedures or by overly strict annotation instructions. More often than not, the notion of reliability is claimed as the primary criterion for the adequacy of a measurement. However, the concept of reliability is not clearly defined for predictions in NLP and data science. There are a multitude of metrics that are commonly applied to measure reliability in data annotation, and a different set of measures that address reliability of model predictions. The goal of this chapter is to provide a clear definition of reliability that applies to data annotation and model prediction alike, and that can be operationalized into a procedure to assess reliability of data annotation and model prediction in concrete applications in NLP and data science.

This is a preview of subscription content, access via your institution.

Buying options

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-031-02183-1_3
  • Chapter length: 35 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   44.99
Price excludes VAT (USA)
  • ISBN: 978-3-031-02183-1
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   59.99
Price excludes VAT (USA)

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Author information

Authors and Affiliations

Authors

Rights and permissions

Reprints and Permissions

Copyright information

© 2022 Springer Nature Switzerland AG

About this chapter

Cite this chapter

Riezler, S., Hagmann, M. (2022). Reliability. In: Validity, Reliability, and Significance. Synthesis Lectures on Human Language Technologies. Springer, Cham. https://doi.org/10.1007/978-3-031-02183-1_3

Download citation