Advertisement

Behavior Research Methods

, Volume 41, Issue 1, pp 137–147 | Cite as

Observer agreement for timed-event sequential data: A comparison of time-based and event-based algorithms

  • Roger Bakeman
  • Vicenç Quera
  • Augusto Gnisci
Article

Abstract

Observer agreement is often regarded as the sine qua non of observational research. Cohen’s κ ?is a widely used index and is appropriate when discrete entities—such as a turn-of-talk or a demarcated time interval—are presented to pairs of observers to code. κ-like statistics and agreement matrices are also used for the timed-event sequential data produced when observers first segment and then code events detected in the stream of behavior, noting onset and offset times. Such κs are of two kinds: time-based and event-based. Available for download is a computer program (OASTES; Observer Agreement for Simulated Timed Event Sequences) that simulates the coding of observers of a stated accuracy and then computes agreement statistics for two time-based κs (with and without tolerance) and three event-based κs (one implemented in The Observer, one in INTERACT, and one in GSEQ). On the basis of simulation results presented here, and due to the somewhat different information provided by each, the reporting of both a time-based and an event-based κ?is recommended.

Keywords

Onset Time Commission Error Observer Agreement Noldus Information Technology Interact Algorithm 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Adamson, L. B., & Bakeman, R. (2006). Development of displaced speech in early mother-child conversations. Child Development, 77, 186–200. doi:10.1111/j.1467-8624.2006.00864.xCrossRefPubMedGoogle Scholar
  2. Adamson, L. B., Bakeman, R., & Deckner, D. F. (2004). The development of symbol-infused joint engagement. Child Development, 75, 1171–1187. doi:10.1111/j.1467-8624.2004.00732.xCrossRefPubMedGoogle Scholar
  3. Bakeman, R., Adamson, L. B., Konner, M., & Barr, R. G. (1990). !Kung infancy: The social context of object exploration. Child Development, 61, 794–809. doi:10.2307/1130964CrossRefPubMedGoogle Scholar
  4. Bakeman, R., & Gottman, J. M. (1997). Observing interaction: An introduction to sequential analysis (2nd ed.). New York: Cambridge University Press.CrossRefGoogle Scholar
  5. Bakeman, R., & Quera, V. (1995). Analyzing interaction: Sequential analysis with SDIS and GSEQ. New York: Cambridge University Press.Google Scholar
  6. Bakeman, R., Quera, V., McArthur, D., & Robinson, B. F. (1997). Detecting sequential patterns and determining their reliability with fallible observers. Psychological Methods, 2, 357–370. doi:10.1037/1082-989X.2.4.357CrossRefGoogle Scholar
  7. Bakeman, R., & Robinson, B. F. (1994). Understanding log-linear analysis with ILOG: An interactive approach. Hillsdale, NJ: Erlbaum.Google Scholar
  8. Cohen, J. A. (1960). A coeff icient of agreement for nominal scales. Educational & Psychological Measurement, 20, 37–46. doi:10.1177/001316446002000104CrossRefGoogle Scholar
  9. Galisson, F. (2000, August). Introduction to computational sequence analysis. Tutorial presented at the Eighth International Conference on Intelligent Systems for Molecular Biology, San Diego, CA.Google Scholar
  10. Haccou, P., & Meelis, E. (1992). Statistical analysis of behavioral data: An approach based on time-structured models. Oxford: Oxford University Press.Google Scholar
  11. Hollenbeck, A. R. (1978). Problems of reliability in observational data. In G. P. Sackett (Ed.), Observing behavior: Vol. 2. Data collection and analysis methods (pp. 79–88). Baltimore: University Park Press.Google Scholar
  12. Jansen, R. G., Wiertz, L. F., Meyer, E. S., & Noldus, L. P. J. J. (2003). Reliability analyis of observational data: Problems, solutions, and software implementation. Behavior Research Methods, Instruments, & Computers, 35, 391–399.CrossRefGoogle Scholar
  13. Konner, M. J. (1978). Maternal care, infant behavior, and development among the !Kung. In R. B. Lee & I. DeVore (Eds.), Kalahari huntergatherers: Studies of the !Kung Sari and their neighbors (pp. 218–245). Cambridge, MA: Harvard University Press.Google Scholar
  14. Mangold, P. (2006, September). Getting better results in less time: When using audio/video recordings in research applications makes sense. Paper presented at the Third Congress of the European Society on Family Relations, Darmstadt, Germany.Google Scholar
  15. Mannila, H., & Ronkainen, P. (1997, May). Similarity of event sequences. In Proceedings of the Fourth International Workshop on Temporal Representation and Reasoning (pp. 136–139). Daytona Beach, FL.Google Scholar
  16. Needleman, S. B., & Wunsch, C. D. (1970). A general method applicable to the search for similarities in the amino acid sequence of two proteins. Journal of Molecular Biology, 48, 443–453.CrossRefPubMedGoogle Scholar
  17. Noldus Information Technology (2003). The Observer: Professional system for collection, analysis, presentation and management of observational data [Reference Manual, Version 5.0]. Wageningen, The Netherlands: Author.Google Scholar
  18. Quera, V., Bakeman, R., & Gnisci, A. (2007). Observer agreement for event sequences: Methods and software for sequence alignment and reliability estimates. Behavior Research Methods, 39, 39–49.CrossRefPubMedGoogle Scholar
  19. Sankoff, D., & Kruskal, J. (Eds.) (1999). Time warps, string edits, and macromolecules: The theory and practice of sequence comparison (2nd ed.). Stanford, CA: CSLI Publications.Google Scholar
  20. Suen, H. K. (1988). Agreement, reliability, accuracy, and validity: Toward a clarification. Behavioral Assessment, 10, 343–366.Google Scholar

Copyright information

© Psychonomic Society, Inc. 2009

Authors and Affiliations

  1. 1.University of BarcelonaBarcelonaSpain
  2. 2.Second University of NaplesCasertaItaly
  3. 3.Department of PsychologyGeorgia State UniversityAtlanta

Personalised recommendations