Abstract
The ability to measure agreement between two independent observers is vital to any observational study. We use a unique situation, the calculation of inter-rater reliability for transcriptions of a parrot’s speech, to present a novel method of dealing with inter-rater reliability which we believe can be applied to situations in which speech from human subjects may be difficult to transcribe. Challenges encountered included (1) a sparse original agreement matrix which yielded an omnibus measure of inter-rater reliability, (2) “lopsided” \(2\times 2\) matrices (i.e. subsets) from the overall matrix and (3) categories used by the transcribers which could not be pre-determined. Our novel approach involved calculating reliability on two levels—that of the corpus and that of the above mentioned smaller subsets of data. Specifically, the technique included the “reverse engineering” of categories, the use of a “null” category when one rater observed a behavior and the other did not, and the use of Fisher’s Exact Test to calculate \(r\)-equivalent for the smaller paired subset comparisons. We hope this technique will be useful to those working in similar situations where speech may be difficult to transcribe, such as with small children.
Similar content being viewed by others
Notes
Theoretically, in a case such as this, the 57 % agreement can be dramatically inflated. See Table 1 for an example scenario.
References
Brennan, R.L., Light, R.J.: Measuring agreement when two observers classify people into categories not defined in advance. Br. J. Math. Stat. Psychol. 27, 154–163 (1974)
Cohen, J.: A coefficient of agreement for nominal scales. Educ. Psychol. Meas. 20, 37–46 (1960)
Colbert-White, E.N., Covington, M.A., Fragaszy, D.M.: Social context influences the vocalizations of a home-raised African grey parrot (Psittacus erithacus erithacus). J. Comp. Psychol. 125, 175–184 (2011). doi:10.1037/a0022097
Fisher, R.A.: Statistical methods for research workers. Oliver & Boyd, Edinburgh (1941)
Van Geert, P., Van Dijk, M.: Ambiguity in child language: the problem of interobserver reliability in ambiguous observation data. First Lang. 23, 259–284 (2003)
Hubert, L.: Nominal scale response agreement as a generalized correlation. Br. J. Math. Stat. Psychol. 30, 98–103 (1977)
Kaufman, A.B., Rosenthal, R.: Can you believe my eyes? The importance of interobserver reliability statistics in observations of animal behaviour. Anim. Behav. 78, 1487–1491 (2009)
Krippendorff, K.: Reliability of binary attribute data. Biometrics 34, 142–144 (1978)
Lindsay, J., O’Connell, D.C.: How do transcribers deal with audio recordings of spoken discourse? J. Psycholinguist. Res. 24, 101–115 (1995)
Montgomery, A.C., Crittenden, K.S.: Improving coding reliability for open-ended questions. Public Opin. Q. 41, 235–243 (1977)
Popping, R.: Traces of agreement: on the DOT-product as a coefficient of agreement. Qual. Quant. 17, 1–18 (1983)
Popping, R.: Traces of agreement: on some agreement indices for open-ended questions. Qual. Quant. 18, 147–158 (1984)
Rosenthal, R.: Conducting judgment studies: some methodological issues. In: Harrigan, J., Rosenthal, R., Scherer, K. (eds.) The new handbook of methods in nonverbal behavior research, pp. 199–236. Oxford University Press, New York (2005)
Rosenthal, R., Rubin, D.B.: A simple, general purpose display of magnitude of experimental effect. J. Educ. Psychol. 74, 166–169 (1982)
Rosenthal, R., Rubin, D.B.: r-equivalent: a simple effect size indicator. Psychol. Methods 8, 492–496 (2003)
Rosenthal, R., Rosnow, R.: Essentials of behavioral research: methods and data analysis. McGraw-Hill, New York (2008)
Scott, W.: Reliability of content analysis: the case of nominal scale coding. Public Opin. Q. 17, 321–325 (1955)
Siegel, S.: Nonparametric statistics for the behavioral sciences. McGraw-Hill, New York (1956)
Snedecor, G.W., Cochran, W.G.: Statistical methods. Iowa State University Press, Ames (1989)
Stockman, I.: Listener reliability in assigning utterance boundaries in children’s spontaneous speech. Appl. Psycholinguist. 31, 363–395 (2010)
Tinsley, H.E.A., Weiss, D.J.: Interrater reliability and agreement of subjective judgments. J. Couns. Psychol. 22, 358–376 (1975)
Author information
Authors and Affiliations
Corresponding author
Additional information
Allison B. Kaufman is now in the Department of Ecology and Evolutionary Biology at The University of Connecticut.
Rights and permissions
About this article
Cite this article
Kaufman, A.B., Colbert-White, E.N. & Rosenthal, R. A novel rater agreement methodology for language transcriptions: evidence from a nonhuman speaker. Qual Quant 48, 2329–2339 (2014). https://doi.org/10.1007/s11135-013-9894-5
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11135-013-9894-5