Quality and Quantity

, Volume 40, Issue 4, pp 519–537 | Cite as

Assessment of Interjudge Reliability in the Open-Ended Questions Coding Process

  • Francisco Muñoz LeivaEmail author
  • Francisco Javier Montoro Ríos
  • Teodoro Luque Martínez


In the process of coding open-ended questions, the evaluation of interjudge reliability is a critical issue. In this paper, using real data, the behavior of three coefficients of reliability among coders, Cohen’s K, Krippendorff’s α and Perreault and Leigh’s I r are patterned, in terms of the number of judges involved and the categories of answer defined. The outcome underlines the importance of both variables in the valuations of interjudge reliability, as well as the higher adequacy of Perreault and Leigh’s I r and Krippendorff’s α for marketing and opinion research.


interjudge reliability open-ended questions concordance coefficients coding process 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Abernethy, A.M., Franke, G.R. 1996The information content of advertising: A meta-analysisJournal of Advertising25117Google Scholar
  2. Bardin, L. (1977). L’analyse de contenu. (Paris: PUF).Google Scholar
  3. Bennett, E.M., Goldstein, A.C. 1954Communications through limited response questioningPublic Opinion Quaterly18303308CrossRefGoogle Scholar
  4. Berelson, B. (1952). Content Analysis in Communications Research. Glencoe (Ill.): Free Press.Google Scholar
  5. Brenner, H., Kliebsch, U. 1996Dependence of weighted kappa coefficients on the number of categoriesEpidemiology7199202Google Scholar
  6. Cohen, J. 1960A coefficient of agreement for nominal scalesEducational and Psychological Measurement203746Google Scholar
  7. Denzin, N.K., Lincoln, Y.S. 1994Handbook of Qualitative ResearchSage PublicationsThousand Oaks (CA)Google Scholar
  8. ESOMAR (2003). Annual Study of the Market Research Industry 2002. (Amsterdam: ESOMAR).Google Scholar
  9. Fleiss, J.L. 1971Measuring nominal scale agreement among many ratersPsychological Bulletin76378382CrossRefGoogle Scholar
  10. Fontana, A., Frey, J.H. 1994Interviewing. The art of scienceDenzin, N.K.Licoln, Y.S. eds. Handbook of Qualitative ResearchSage PublicationsThousand Oaks (CA)Google Scholar
  11. Glaser, B.G., Strauss, A.L. 1999The Discovery of Grounded Theory: Strategies for Qualitative ResearchAldine de GruyterNew YorkGoogle Scholar
  12. Grayson, K., Rust, R. 2001Interrater reliabilityJournal of Consumer Psychology107173CrossRefGoogle Scholar
  13. Herbert, L. 1977Kappa revisitedPsychological Bulletin84289297CrossRefGoogle Scholar
  14. Holbrook, M.B. 1977More on content analysis in consumer researchJournal of Consumer Research4176177CrossRefGoogle Scholar
  15. Holsti, O.R. 1969Content Analysis for the Social Sciences and HumanitiesAddison-WesleyReading (MA)Google Scholar
  16. Hsu, , Fied,  2003Interrater agreement measures: Comments on Kappa, Cohen’s Kappa, Scott’s π and Aicking αUnderstanding statistics2205219CrossRefGoogle Scholar
  17. Hughes, M.A., Garret, D.E. 1990Intercoder reliability estimation approaches in marketing: A Generability Theory framework for quantitative dataJournal of Marketing Research2718595CrossRefGoogle Scholar
  18. Kang, N., Kara, A., Laskey, H.A., Seaton, F.B. 1993A SAS MACRO for calculating intercoder agreement in content analysisJournal of Advertising221728Google Scholar
  19. Kassarjian, H.H. 1977Content analysis in consumer researchJournal of Consumer Research4818CrossRefGoogle Scholar
  20. Kolbe, R.H., Burnett, M.S. 1991Content-analysis research: An examination of applications with directives for improving research reliability and objectivityJournal of Consumer Research18243250CrossRefGoogle Scholar
  21. Krippendorff, K. 1971Reliability of recording instructions: Multivariate agreement for nominal dataBehavioral Science16228235Google Scholar
  22. Krippendorff, K. 1980Content Analysis, an Introduction to Its MethodologySage PublicationsThousand Oaks (CA)Google Scholar
  23. Landis, J.R., Koch, G.G. 1977The measurement of observer agreement for categorical dataBiometrics33159174CrossRefGoogle Scholar
  24. Lehmann, R.L., Gupta, S., Steckel, J. H. 1998Marketing ResearchAddison-Wesley Educational Publishers Inc.Reading (MA)Google Scholar
  25. Light, R.J. 1971Measures of response agreement for qualitative data: some generalizations and alternativesPsychological Bulleting76365377CrossRefGoogle Scholar
  26. Lincoln, Y.S., Guba, E.G. 1985Naturalistic InquirySage PublicationsBeverly Hills (CA)Google Scholar
  27. Lombard, M., ZINDER-duch, J., Campanela, C. 2002Content analysis in mass communication: Assessment and reporting of intercoder reliabilityHuman Communication Research28587604CrossRefGoogle Scholar
  28. López-Aranguren (1989). El análisis de contenido. In: M. García et al. (Coord.): El Análisis de la Realidad Social: Métodos y Técnicas de Investigación. Madrid: Alianza Editorial.Google Scholar
  29. Luque, T. 1997Investigación de MarketingEd. ArielBarcelonaGoogle Scholar
  30. Mackensen, K., Wille, U. 1999Qualitative text analysis supported by conceptual data systemsQuality & Quantity33135156CrossRefGoogle Scholar
  31. Miles, M.B., Huberman, A.M. 1994Quality Data AnalysisAn expanded Sourcebook. Sage PublicationsThousand Oaks (CA)Google Scholar
  32. Miller, W.L., Crabtree, B.F. 1994Clinical researchDenzin, N.K.Licoln, Y.S. eds. Handbook of Qualitative ResearchSage PublicationsThousand Oaks (CA)Google Scholar
  33. Perreault, W.D., LEIGH, E.L. 1989Reliability of nominal data based on quantitative judgmentsJournal of Marketing Research2313043Google Scholar
  34. Richards, , Richards,  1994Using computers in qualitative researchDenzin, N.K.Licoln, Y.S. eds. Handbook of Qualitative ResearchSage PublicationsThousand Oaks (CA)Google Scholar
  35. Riffe, D., Freitag, A.A. 1997A content analysis of content analysis: Twenty-five years of Journalism QuarterlyJournalism & Mass Communication Quarterly74873882Google Scholar
  36. Roberts, C.W. 2000A conceptual framework for quantitative text analysis: On joining probabilities and substantive inferences about textQuality & Quantity34259274CrossRefGoogle Scholar
  37. Rust, R.T., Cooil, B. 1994Reliability measures for qualitative data: Theory and implicationsJournal of Marketing Research31114CrossRefGoogle Scholar
  38. Spiggle, S. 1994Analysis and interpretation of qualitative data in consumer researchJournal of Marketing Research21491503Google Scholar
  39. Spitznagel, E.L., Helzer, J.E. 1985A proposed solution to the base rate problem in the kappa statisticArchives of General Psychiatry4272528Google Scholar
  40. Strauss, A., Corbin, J. 1990Basic of qualitative research: Grounded theory procedures and techniquesSage PublicationsNewbury Park (CA)Google Scholar
  41. Weber, R.P. 1985Basic Content AnalysisEd. Sage PublicationsNewbury ParkGoogle Scholar
  42. Weitzman, E.A., Miles, M.B. 1995Computer Programs for Qualitative Data Analysis: A Software SourcebookSage PublicationsThousand Oaks (CA)Google Scholar

Copyright information

© Springer 2006

Authors and Affiliations

  • Francisco Muñoz Leiva
    • 1
    Email author
  • Francisco Javier Montoro Ríos
    • 1
  • Teodoro Luque Martínez
    • 1
  1. 1.Department of Marketing and Market ResearchFaculty of Economics and Business Sciences of the University of GranadaGranadaSpain

Personalised recommendations