Advertisement

Reactivity effects in video-based classroom research: an investigation using teacher and student questionnaires as well as teacher eye-tracking

  • Anna-Katharina Praetorius
  • Nora A. McIntyre
  • Robert M. Klassen
Article

Abstract

One prominent problem of conducting observational assessments of teaching quality is the possibility of reactivity effects. To date, the issue of reactivity has received limited empirical attention. The present study, therefore, investigated reactivity in 447 students from 24 classes as well as their 12 teachers. We compared reactivity during lessons that were video-recorded with those that were not: according to t‑test analyses of teacher ratings and MIMIC analyses of student ratings, no significant differences emerged in teaching quality or teaching practices. Significant differences were found in teacher and student emotions, as well as in student cognition and behavior. Supplementary eye-tracking analyses indicated reactivity depleted after 1 min 20 s. The results are discussed with respect to their relevance for future video studies on classroom instruction.

Keywords

Eye tracking Observer ratings Reactivity Student ratings Teacher ratings Video-based classroom research 

Reaktivitätseffekte in der videobasierten Unterrichtsforschung: Eine Untersuchung mittels Lehrer- und Schülereinschätzungen sowie Lehrer-Eye-Tracking

Zusammenfassung

Ein zentraler Nachteil von Beobachtereinschätzungen zur Erfassung von Unterrichtsqualität sind potentielle Reaktivitätseffekte. In welchem Ausmaß solche Effekte auftreten, wurde bislang kaum empirisch untersucht. Im Rahmen einer Videoerhebung wurden Daten von 447 Schüler(inne)n aus 24 Klassen sowie deren 12 Lehrkräften erhoben. Der Vergleich der Video- sowie der Nicht-Video-Bedingung zeigte sowohl für die Lehrereinschätzungen (analysiert mittels t‑Tests) als auch die Schülereinschätzungen (analysiert mittels MIMIC-Modellen) keine Unterschiede hinsichtlich Merkmalen des Unterrichts (Unterrichtsqualität und Unterrichtspraktiken); in Bezug auf Lehrer- und Schüleremotionen sowie Schülerkognition und -verhalten zeigten sich hingegen Unterschiede zwischen den beiden Bedingungen. Die ergänzenden Eye-Tracking-Analysen deuten darauf hin, dass reaktive Blickbewegungen von Lehrkräften nach maximal 1 min 20 s nicht mehr nachzuweisen sind. Die Ergebnisse werden im Hinblick auf ihre Bedeutung für zukünftige Videostudien diskutiert.

Schlüsselwörter

Reaktivität Beobachtereinschätzungen Unterrichtsforschung Lehrereinschätzungen Schülereinschätzungen Eye-tracking 

References

  1. Becker, T. E., & Marique, G. (2014). Observer effects without demand characteristics: an inductive investigation of video monitoring and performance. Journal of Business and Psychology, 29(4), 541–553.CrossRefGoogle Scholar
  2. Becker, E. S., Keller, M. M., Goetz, T., Frenzel, A. C., & Taxer, J. L. (2015). Antecedents of teachers’ emotions in the classroom: an intraindividual approach. Frontiers in Psychology, 6, 635.CrossRefGoogle Scholar
  3. Berliner, D. C. (2001). Learning about and learning from expert teachers. International Journal of Educational Research, 35(5), 463–482.CrossRefGoogle Scholar
  4. Blease, D. (1983). Observer effects on teachers and pupils in classroom research. Educational Review, 35(3), 213–217.CrossRefGoogle Scholar
  5. Borich, G. D. (2008). Observation skills for effective teaching. Upper Saddle River: Pearson Merrill Prentice Hall.Google Scholar
  6. Brophy, J. (2006). History of research on classroom management. In C. M. Evertson & C. S. Weinstein (Eds.), Handbook of classroom management: research, practice, and contemporary issues (pp. 17–43). Mahwah: Lawrence Erlbaum Associates.Google Scholar
  7. Campbell, D. T. (1957). Factors relevant to the validity of experiments in social settings. Psychological Bulletin, 54, 297–312.CrossRefGoogle Scholar
  8. Carter, V. (2008). Five Steps to becoming a better peer reviewer. College Teaching, 56(2), 85–88.CrossRefGoogle Scholar
  9. Casabianca, J. M., McCaffrey, D. F., Gitomer, D. H., Bell, C. A., Hamre, B. K., & Pianta, R. C. (2013). Effect of observation mode on measures of secondary mathematics teaching. Educational and Psychological Measurement, 73(5), 757–783.CrossRefGoogle Scholar
  10. Charness, N., Reingold, E. M., Pomplun, M., & Stampe, D. M. (2001). The perceptual aspect of skilled performance in chess: evidence from eye movements. Memory & cognition, 29(8), 1146–1152.CrossRefGoogle Scholar
  11. Chisholm, S., Caird, J. K., & Lockhart, J. (2008). The effects of practice with MP3 players on driving performance. Accident Analysis & Prevention, 40(2), 704–713.CrossRefGoogle Scholar
  12. Church, R. B., Kelly, S., & Holcombe, D. (2014). Temporal synchrony between speech, action and gesture during language production. Language, Cognition and Neuroscience, 29(3), 345–354.CrossRefGoogle Scholar
  13. Clare, L., Valdés, R., Pascal, J., & Steinberg, J. R. (2001). Teachers’ assignments as indicators of instructional quality in elementary schools. vol. 545. Los Angeles: National Center for Research on Evaluation, Standards, and Student Testing (CRESST).Google Scholar
  14. Clausen, M. (2002). Qualität von Unterricht – Eine Frage der Perspektive? (Quality of instruction – a matter of perspective?). Münster: Waxmann.Google Scholar
  15. Codding, R. S., Livanis, A., Pace, G. M., & Vaca, L. (2008). Using performance feedback to improve treatment integrity of classwide behavior plans: an investigation of observer reactivity. Journal of Applied Behavior Analysis, 41(3), 417–422.CrossRefGoogle Scholar
  16. Cortina, K. S., Miller, K., McKenzie, R., & Epstein, A. (2015). Where low and high inference data converge: validation of CLASS assessment of mathematics instruction using mobile eye tracking with expert and novice teachers. International Journal of Science and Mathematics Education, 13(2), 389–403.CrossRefGoogle Scholar
  17. Dalehefte, I. M., Rimmele, R., Prenzel, M., Seidel, T., Labudde, P., & Herweg, C. (2009). Observing instruction “next-door”: a video study about science teaching and learning in Germany and Switzerland. In T. Janík & T. Seidel (Eds.), The power of video studies in investigating teaching and learning in the classroom (pp. 83–101). Münster: Waxmann.Google Scholar
  18. DeAngelus, M., & Pelz, J. B. (2009). Top-down control of eye movements: Yarbus revisited. Visual Cognition, 17(6–7), 790–811.CrossRefGoogle Scholar
  19. Doyle, W. (1986). Classroom organization and management. In M. C. Wittrock (Ed.), Handbook of research on teaching (pp. 392–431). New York: Macmillan.Google Scholar
  20. Dreischenkämper, C., & Stanik, T. (2014). Die Reaktivitätsproblematik von Videographien. In J. Kade, S. Nolda, J. Dinkelaker & M. Herrle (Eds.), Videographische Kursforschung: Empirie des Lehrens und Lernens Erwachsener (Video-based course research: empirical evidence of teaching and learning adults) (pp. 41–54). Stuttgart: Kohlhammer.Google Scholar
  21. Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A. (2007). G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39, 175–191.CrossRefGoogle Scholar
  22. Feldon, D. F. (2007). Cognitive load and classroom teaching: the double-edged sword of automaticity. Educational Psychologist, 42(3), 123–137.CrossRefGoogle Scholar
  23. Foulsham, T., & Kingstone, A. (2012). Goal-driven and bottom-up gaze in an active real-world search task. Paper presented at the Proceedings of the Symposium on Eye Tracking Research and Applications.CrossRefGoogle Scholar
  24. Foulsham, T., & Underwood, G. (2008). What can saliency models predict about eye movements? Spatial and sequential aspects of fixations during encoding and recognition. Journal of Vision, 8(2), 1–17.CrossRefGoogle Scholar
  25. Glaholt, M. G., & Reingold, E. M. (2011). Eye movement monitoring as a process tracing methodology in decision making research. Journal of Neuroscience, Psychology, and Economics, 4(2), 125–146.CrossRefGoogle Scholar
  26. Glaholt, M. G., Wu, M.-C., & Reingold, E. M. (2010). Evidence for top-down control of eye movements during visual decision making. Journal of vision, 10(5), 15–15.CrossRefGoogle Scholar
  27. van Gog, T., Paas, F., van Merrienboer, J. J., & Witte, P. (2005). Uncovering the problem-solving process: cued retrospective reporting versus concurrent and retrospective reporting. Journal of Experimental Psychology, 11(4), 237–244. Applied.Google Scholar
  28. Helmke, A. (2009). Unterrichtsqualität und Lehrerprofessionalität: Diagnose, Evaluation und Verbesserung des Unterrichts (Quality of instruction and teacher professionalism: diagnosis, evaluation, and improvement of instruction). Seelze: Klett-Kallmeyer.Google Scholar
  29. Helmke, T., Helmke, A., Schrader, F.-W., Wagner, W., Nold, G., & Schröder, K. (2008). Die Videostudie des Englischunterrichts. In DESI-Konsortium (Ed.), Unterricht und Kompetenzerwerb zu Deutsch und Englisch. Ergebnisse der DESI-Studie (Instruction and competence development in German and English as a foreign language. Results of the DESI study) (pp. 345–363). Weinheim: Beltz.Google Scholar
  30. Henderson, J. M., Brockmole, J. R., Castelhano, M. S., & Mack, M. (2007). Visual saliency does not account for eye movements during visual search in real-world scenes. In R. van Gompel, M. Fischer, W. Murray & R. Hill (Eds.), Eye movements: a window on mind and brain (pp. 537–562). Oxford: Elsevier.CrossRefGoogle Scholar
  31. Heyns, R. W., & Zander, A. F. (1953). Observation of group behavior. In L. Festinger & D. Katz (Eds.), Research methods in the behavioral sciences (pp. 381–417). London: Staples Press.Google Scholar
  32. Hill, H. C., Charalambous, C. Y., & Kraft, M. A. (2012). When rater reliability is not enough: teacher observation systems and a case for the G‑study. Educational Researcher, 41(2), 56–64.CrossRefGoogle Scholar
  33. Hristova, E., Georgieva, S., & Grinberg, M. (2011). Top-down influences on eye-movements during painting perception: the effect of task and titles. In A. Esposito, A. M. Esposito, R. Martone, V. C. Müller & G. Scarpetta (Eds.), Toward autonomous, adaptive, and context-aware multimodal interfaces. Theoretical and practical issues (pp. 104–115). Heidelberg: Springer.CrossRefGoogle Scholar
  34. Jones, E. E., & Nisbett, R. E. (1972). The actor and the observer: divergent perceptions of the causes of behavior. In E. E. Jones, D. E. Kanouse, H. H. Kelley, R. E. Nisbett, S. Valins & B. Weiner (Eds.), Attribution: perceiving the causes of behavior (pp. 79–94). Morristown: General Learning Press.Google Scholar
  35. Kane, T. J., & Staiger, D. O. (2012). Gathering feedback for teachers: combining high-quality observations with student surveys and achievement gains. http://files.eric.ed.gov/fulltext/ED540960.pdf. Last access: 10.08.2016.
  36. Kazdin, A. E. (1982). Observer effects: reactivity of direct observation. New Directions for Methodology of Social & Behavioral Science, 14, 5–19.Google Scholar
  37. Kerlinger, F. N. (1973). Foundations of behavioral research. New York: Holt, Rinehart, and Winston.Google Scholar
  38. Kingstone, A. (2013). The cycle of social signaling. Keynote presented at the European conference on eye movements, Lund.Google Scholar
  39. Klieme, E. (2006). Empirische Unterrichtsforschung: aktuelle Entwicklungen, theoretische Grundlagen und fachspezifische Befunde. Einführung in den Thementeil (Empirical instructional research: current developments, theoretical basis and subject-specific findings). Zeitschrift für Pädagogik/German Journal of Pedagogy, 51(6), 765–773.Google Scholar
  40. Kohut, G. F., Burnap, C., & Yon, M. G. (2007). Peer observation of teaching. College Teaching, 55(1), 19–25.CrossRefGoogle Scholar
  41. Kunter, M. (2005). Multiple Ziele im Mathematikunterricht (Multiple goals in math instruction). Münster: Waxmann.Google Scholar
  42. Kunter, M., & Baumert, J. (2006). Who is the expert? Construct and criteria validity of student and teacher ratings of instruction. Learning Environments Research, 9(3), 231–251.CrossRefGoogle Scholar
  43. Land, M. F., & Hayhoe, M. (2001). In what ways do eye movements contribute to everyday activities? Vision research, 41(25), 3559–3565.CrossRefGoogle Scholar
  44. Liang, J. (2015). Live video classroom observation: an effective approach to reducing reactivity in collecting observational information for teacher professional development. Journal of Education for Teaching, 41(3), 235–253.CrossRefGoogle Scholar
  45. Lüdtke, O., Robitzsch, A., Trautwein, U., & Kunter, M. (2009). Assessing the impact of learning environments: How to use student ratings of classroom or school characteristics in multilevel modeling. Contemporary Educational Psychology, 34(2), 120–131.CrossRefGoogle Scholar
  46. Macdonald, R. G., & Tatler, B. W. (2015). Referent expressions and gaze: reference type influences real-world gaze cue utilization. Journal of Experimental Psychology: Human perception and performance, 41(2), 565–575.Google Scholar
  47. Mackworth, N. H., & Morandi, A. J. (1967). The gaze selects informative details within pictures. Perception & psychophysics, 2(11), 547–552.CrossRefGoogle Scholar
  48. Masling, J., & Stern, G. (1969). Effect of the observer in the classroom. Journal of Educational Psychology, 60(5), 351–354.CrossRefGoogle Scholar
  49. McNeill, D. (1985). So you think gestures are nonverbal? Psychological Review, 92(3), 350–371.CrossRefGoogle Scholar
  50. Medley, D. M., & Mitzel, H. E. (1962). Measuring classroom behavior by systematic observation. In N. L. Gage (Ed.), Handbook of Research on Teaching (pp. 247–328). Chicago: Rand McNally.Google Scholar
  51. Miner, M. H., Dowson, M., & Sterland, S. (2010). Ministry orientation and ministry outcomes: evaluation of a new multidimensional model of clergy burnout and job satisfaction. Journal of Occupational and Organizational Psychology, 83, 167–188.CrossRefGoogle Scholar
  52. Muthén, B., & Muthén, L. (1998–2012). Mplus (Version 7.11). Los Angeles, CA: StatModel.Google Scholar
  53. Newton, X. A. (2010). Developing indicators of classroom practice to evaluate the impact of district mathematics reform initiative: a generalizability analysis. Studies in Educational Evaluation, 36, 1–13.CrossRefGoogle Scholar
  54. Petko, D., Waldis, M., Pauli, C., & Reusser, K. (2003). Methodologische Überlegungen zur videogestützten Forschung in der Mathematikdedaktik. Ansätze der TIMSS 1999 Video Studie und ihrer schweizerischen Erweiterung (Methodological considerations about video-based research in mathematical didactics. Approaches of the TIMSS 1999 video study and its Swiss extension). Zentralblatt der Didaktik für Mathematik, 35(6), 265–280.CrossRefGoogle Scholar
  55. Pianta, R. C., & Hamre, B. K. (2009). Conceptualization, measurement, and improvement of classroom processes: standardized observation can leverage capacity. Educational Researcher, 38(2), 109–119.CrossRefGoogle Scholar
  56. Praetorius, A.-K., Pauli, C., Reusser, K., Rakoczy, K., & Klieme, E. (2014). One lesson is all you need? Stability of instructional quality across lessons. Learning and Instruction, 31, 2–12.CrossRefGoogle Scholar
  57. Rakoczy, K. (2008). Motivationsunterstützung im Mathematikunterricht – Unterricht aus der Perspektive von Lernenden und Beobachtern (Motivational support in math instruction – instruction from the learner and observer perspectives). Münster: Waxmann.Google Scholar
  58. Rötting, M. (2001). Parametersystematik der Augen- und Blickbewegungen fur arbeitswissenschaftliche Untersuchungen. RWTH Aachen, Unpublished doctoral dissertation.Google Scholar
  59. Samph, T. (1976). Observer effects on teacher verbal classroom behavior. Journal of Educational Psychology, 68(6), 736–741.CrossRefGoogle Scholar
  60. Schmidt, M., Perels, F., & Schmitz, B. (2010). How to Perform Idiographic and a Combination of Idiographic and Nomothetic Approaches: A Comparison of Time Series Analyses and Hierarchical Linear Modeling. Zeitschrift für Psychologie, 218(3), 166–174. doi: 10.1027/0044-3409/a000026
  61. Schumann, F., Einhäuser, W., Vockeroth, J., Bartl, K., Schneider, E., & Koenig, P. (2008). Salient features in gaze-aligned recordings of human visual input during free exploration of natural environments. Journal of Vision, 8(14), 12–12.CrossRefGoogle Scholar
  62. Schyns, P. G., & Oliva, A. (1994). From blobs to boundary edges: evidence for time-and spatial-scale-dependent scene recognition. Psychological Science, 5(4), 195–200.CrossRefGoogle Scholar
  63. Seidel, T., Prenzel, M., Duit, R., & Lehrke, M. (2003). Technischer Bericht zur Videostudie “Lehr-Lern-Prozesse im Physikunterricht” (Technical report of the video study “teaching and learning processes in Physics instruction”). Kiel: IPN.Google Scholar
  64. Seidel, T., Prenzel, M., & Kobarg, M. (2005). How to run a video study: technical report of the IPN video study. Münster: Waxmann.Google Scholar
  65. Stigler, J. (1998). Video surveys: new data for the improvement of classroom instruction. In S. G. Paris & H. M. Wellmann (Eds.), Global prospects for education. Development, culture and schooling (pp. 129–168). Washington DC: American Psychological Association.CrossRefGoogle Scholar
  66. Stigler, J. W., Gonzales, P., Kawanaka, T., Knoll, S., & Serrano, A. (1999). The TIMSS videotape classroom study: Methods and findings from an exploratory research project on eighth-grade mathematics instruction in Germany, Japan, and the United States. Washington DC: U.S. Department of Education, National Center for Education Statistics.Google Scholar
  67. Storms, M. D. (1973). Videotape and the attribution process: reversing actors’ and observers’ points of view. Journal of Personality and Social Psychology, 27(2), 165–175.CrossRefGoogle Scholar
  68. Tatler, B. W., Gilchrist, I. D., & Land, M. F. (2005). Visual memory for objects in natural scenes: from fixations to object files. The Quarterly Journal of Experimental Psychology Section A, 58(5), 931–960.CrossRefGoogle Scholar
  69. Tatler, B. W., Hirose, Y., Finnegan, S. K., Pievilainen, R., Kirtley, C., & Kennedy, A. (2013). Priorities for selection and representation in natural tasks. Philosophical Transactions of the Royal Society B: Biological Sciences, 368(1628), 20130066.CrossRefGoogle Scholar
  70. Todman, J. B., & Dugard, P. (2001). Single-case and small-n experimental designs. A practical guide to randomization tests. Mahwah: Erlbaum.Google Scholar
  71. Waldis, M., Grob, U., Pauli, C., & Reusser, K. (2010). Der schweizerische Mathematikunterricht aus der Sicht von Schülerinnen und Schülern und in der Perspektive hochinferenter Beobachterurteile. In K. Reusser, C. Pauli & M. Waldis (Eds.), Unterrichtsgestaltung und Unterrichtsqualität. Ergebnisse einer internationalen und schweizerischen Videostudie zum Mathematikunterricht [Practices and quality of instruction. Findings of an international and Swiss video study on math instruction (pp. 171–208). Münster: Waxmann.Google Scholar
  72. Wubbels, T., Brekelmans, M., & Hooymayers, H. P. (1992). Do teacher ideals distort the self-reports of their interpersonal behavior? Teaching and Teacher Education, 8(1), 47–58.CrossRefGoogle Scholar
  73. Yarbus, A. L. (1967). Eye movements and vision. New York: Plenum Press.CrossRefGoogle Scholar

Copyright information

© Springer Fachmedien Wiesbaden 2017

Authors and Affiliations

  • Anna-Katharina Praetorius
    • 1
  • Nora A. McIntyre
    • 2
  • Robert M. Klassen
    • 3
  1. 1.German Institute for International Educational Research (DIPF)FrankfurtGermany
  2. 2.Department of PsychologyUniversity of SheffieldSheffieldUK
  3. 3.Department of EducationUniversity of YorkYorkUK

Personalised recommendations