Quality & Quantity

, Volume 48, Issue 5, pp 2703–2718

On the choice of measures of reliability and validity in the content-analysis of texts

  • Anton Oleinik
  • Irina Popova
  • Svetlana Kirdina
  • Tatyana Shatalova

DOI: 10.1007/s11135-013-9919-0

Cite this article as:
Oleinik, A., Popova, I., Kirdina, S. et al. Qual Quant (2014) 48: 2703. doi:10.1007/s11135-013-9919-0


The paper discusses several reliability measures: Scott’s pi, Krippendorff’s alpha, free marginal adjustment (Bennett, Alpert and Goldstein’s \(S\)), Cohen’s kappa, and Perreault and Leigh’s \(I\) and the assumptions on which they are based. It is suggested that correlation coefficients between, on one hand, the distribution of qualitative codes and, on the other hand, word co-occurrences and the distribution of the categories identified with the help of the dictionary based on substitution complement the other reliability measures. The paper shows that the choice of the reliability measure depends on the format of the text (stylistic versus rhetorical) and the type of reading (comprehension versus interpretation). Namely, Cohen’s kappa and Bennett, Alpert and Goldstein’s \(S\) emerge as reliability measures particularly suited for perspectival reading of rhetorical texts. Outcomes of the content analysis of 57 texts performed by four coders with the help of computer program QDA Miner inform the analysis.


Reliability measures Content analysis Correlation analysis Interpretation Comprehension Stylistic texts Rhetorical texts 

Copyright information

© Springer Science+Business Media Dordrecht 2013

Authors and Affiliations

  • Anton Oleinik
    • 1
    • 2
  • Irina Popova
    • 3
    • 4
  • Svetlana Kirdina
    • 5
  • Tatyana Shatalova
    • 6
  1. 1.Department of SociologyMemorial University of NewfoundlandSt. John’sCanada
  2. 2.Central Economics and Mathematics Institute Russian Academy of SciencesMoscowRussia
  3. 3.Institute of Sociology Russian Academy of SciencesMoscowRussia
  4. 4.Higher School of EconomicsMoscowRussia
  5. 5.Institute of Economics Russian Academy of SciencesMoscowRussia
  6. 6.Moscow State Lomonossov UniversityMoscowRussia

Personalised recommendations