Observer agreement for timed-event sequential data: A comparison of time-based and event-based algorithms
- 462 Downloads
Observer agreement is often regarded as the sine qua non of observational research. Cohen’s κ ?is a widely used index and is appropriate when discrete entities—such as a turn-of-talk or a demarcated time interval—are presented to pairs of observers to code. κ-like statistics and agreement matrices are also used for the timed-event sequential data produced when observers first segment and then code events detected in the stream of behavior, noting onset and offset times. Such κs are of two kinds: time-based and event-based. Available for download is a computer program (OASTES; Observer Agreement for Simulated Timed Event Sequences) that simulates the coding of observers of a stated accuracy and then computes agreement statistics for two time-based κs (with and without tolerance) and three event-based κs (one implemented in The Observer, one in INTERACT, and one in GSEQ). On the basis of simulation results presented here, and due to the somewhat different information provided by each, the reporting of both a time-based and an event-based κ?is recommended.
KeywordsOnset Time Commission Error Observer Agreement Noldus Information Technology Interact Algorithm
Unable to display preview. Download preview PDF.
- Bakeman, R., & Quera, V. (1995). Analyzing interaction: Sequential analysis with SDIS and GSEQ. New York: Cambridge University Press.Google Scholar
- Bakeman, R., & Robinson, B. F. (1994). Understanding log-linear analysis with ILOG: An interactive approach. Hillsdale, NJ: Erlbaum.Google Scholar
- Galisson, F. (2000, August). Introduction to computational sequence analysis. Tutorial presented at the Eighth International Conference on Intelligent Systems for Molecular Biology, San Diego, CA.Google Scholar
- Haccou, P., & Meelis, E. (1992). Statistical analysis of behavioral data: An approach based on time-structured models. Oxford: Oxford University Press.Google Scholar
- Hollenbeck, A. R. (1978). Problems of reliability in observational data. In G. P. Sackett (Ed.), Observing behavior: Vol. 2. Data collection and analysis methods (pp. 79–88). Baltimore: University Park Press.Google Scholar
- Konner, M. J. (1978). Maternal care, infant behavior, and development among the !Kung. In R. B. Lee & I. DeVore (Eds.), Kalahari huntergatherers: Studies of the !Kung Sari and their neighbors (pp. 218–245). Cambridge, MA: Harvard University Press.Google Scholar
- Mangold, P. (2006, September). Getting better results in less time: When using audio/video recordings in research applications makes sense. Paper presented at the Third Congress of the European Society on Family Relations, Darmstadt, Germany.Google Scholar
- Mannila, H., & Ronkainen, P. (1997, May). Similarity of event sequences. In Proceedings of the Fourth International Workshop on Temporal Representation and Reasoning (pp. 136–139). Daytona Beach, FL.Google Scholar
- Noldus Information Technology (2003). The Observer: Professional system for collection, analysis, presentation and management of observational data [Reference Manual, Version 5.0]. Wageningen, The Netherlands: Author.Google Scholar
- Sankoff, D., & Kruskal, J. (Eds.) (1999). Time warps, string edits, and macromolecules: The theory and practice of sequence comparison (2nd ed.). Stanford, CA: CSLI Publications.Google Scholar
- Suen, H. K. (1988). Agreement, reliability, accuracy, and validity: Toward a clarification. Behavioral Assessment, 10, 343–366.Google Scholar