Abstract
The efficacy of collaborative learning depends on the student interaction taking place. Monitoring student interaction can help teachers ensure their lesson goals are met. Experienced teachers describe, explain, and predict student interactions by drawing on their professional knowledge about types of student interaction that are associated with learning. This paper reports an assessment tool that measures teachers’ monitoring competence, operationalized as a form of professional vision: the ability to describe student interaction. Participants rate student interactions shown in videos on standardized items. Ratings are compared to an expert solution to provide a measure of monitoring competence.
The assessment tool was evaluated with N = 193 teacher students that were in the pre-service or in-service phase of German teacher education. Classical test theory and generalizability theory were applied to evaluate the reliability of the assessment tool. Scores showed medium internal consistency. Measurement error resulting from videos was small. Reliability of scores was sufficient to detect differences in competence levels, before and after completing a training program on monitoring competence. Participants found the assessment tool appropriate for measuring monitoring competence. The findings show the feasibility of measuring professional vision of student interaction in collaborative learning. The assessment tool may be useful for adapting teacher dashboards, such as by advising beginning teachers how to intervene or, for more experienced teachers, by alerting when student interaction is not optimal for learning.
Zusammenfassung
Die Effektivität kooperativen Lernens hängt wesentlich von der Interaktion der Lernenden ab. Das Überwachen oder Monitoring von Schülerinteraktionen kann Lehrkräften dabei helfen, zu überprüfen, ob ihre Unterrichtsziele erreicht werden. Erfahrene Lehrkräfte können Schülerinteraktionen beschreiben, erklären und vorhersagen, indem sie auf ihr professionelles Wissen über lernförderliche Schülerinteraktionen zurückgreifen. Wir präsentieren und evaluieren in diesem Artikel ein Assessment Tool, das die Monitoringkompetenz angehender Lehrkräfte misst, operationalisiert als die Fähigkeit, Schülerinteraktionen zu beschreiben. Dazu bewerten Lehrkräfte auf standardisierten Items Schülerinteraktionen, die in Videos gezeigt werden. Die Ratings werden mit einer Expertenlösung verglichen, um ein Maß der Monitoringkompetenz zu liefern.
Das Assessment Tool wurde mit N = 193 angehenden Lehrkräften evaluiert, die sich in der ersten oder zweiten Phase der Lehrerbildung befanden. Zur Beurteilung der Reliabilität des Assessment Tools wurden klassische Testtheorie und Generalisierbarkeitstheorie angewendet. Die Ergebnisse zeigten eine mittlere interne Konsistenz. Der Messfehler durch Videos war gering. Die Reliabilität der Ergebnisse war ausreichend, um Unterschiede im Kompetenzniveau vor und nach Abschluss eines Trainingsprogramms zur Monitoringkompetenz zu erkennen. Die Teilnehmer fanden das Assessment Tool inhaltlich valide. Die Ergebnisse zeigen, dass professionelle Wahrnehmung von Schülerinteraktionen in kooperativen Lernsettings gemessen werden kann. Das Assessment Tool kann für die Anpassung von Lehrer-Dashboards nützlich sein, zum Beispiel um angehende Lehrer zu beraten, wann und wie sie eingreifen sollten, oder, für erfahrenere Lehrer, um zu warnen, wenn die Interaktion der Schüler nicht optimal für das Lernen ist.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Artzt, A. F., & Armour-Thomas, E. (1992). Development of a cognitive-metacognitive framework for protocol analysis of mathematical problem solving in small groups. Cognition and Instruction, 9(2), 137–175. https://doi.org/10.1207/s1532690xci0902_3.
Baker, M., Andriessen, J., Lund, K., Amelsvoort, M., & Quignard, M. (2007). Rainbow: A framework for analysing computer-mediated pedagogical debates. Computer Supported Learning, 2(2–3), 315–357. https://doi.org/10.1007/s11412-007-9022-4.
Bannert, M. (2009). Promoting self-regulated learning through prompts. Zeitschrift für Pädagogische Psychologie, 23(2), 139–145. https://doi.org/10.1024/1010-0652.23.2.139.
Barron, B. (2003). When smart groups fail. Journal of the Learning Sciences, 12(3), 307–359. https://doi.org/10.1207/S15327809JLS1203_1.
Bereiter, C. (2013). Principled practical knowledge: Not a bridge but a ladder. Journal of the Learning Sciences, 23(1), 4–17. https://doi.org/10.1080/10508406.2013.812533.
Blömeke, S., Gustafsson, J.-E., & Shavelson, R. J. (2015). Beyond dichotomies. Competence viewed as continuum. Zeitschrift für Psychologie, 223(1), 3–13. https://doi.org/10.1027/2151-2604/a000194.
Borko, H., Jacobs, J., Eiteljorg, E., & Pittman, M. E. (2008). Video as a tool for fostering productive discussions in mathematics professional development. Teaching and Teacher Education, 24(2), 417–436. https://doi.org/10.1016/j.tate.2006.11.012.
van Boxtel, C., van der Linden, J., & Kanselaar, G. (2000). Collaborative learning tasks and the elaboration of conceptual knowledge. Learning and Instruction, 10(4), 311–330. https://doi.org/10.1016/S0959-4752(00)00002-5.
Brennan, R. L. (2000). Performance assessments from the perspective of generalizability theory. Applied Psychological Measurement, 24(4), 339–353. https://doi.org/10.1177/01466210022031796.
Brennan, R. L. (2010). Generalizability theory and classical test theory. Applied Measurement in Education, 24(1), 1–21. https://doi.org/10.1080/08957347.2011.532417.
Briesch, A. M., Swaminathan, H., Welsh, M., & Chafouleas, S. M. (2014). Generalizability theory: A practical guide to study design, implementation, and interpretation. Journal of School Psychology, 52(1), 13–35. https://doi.org/10.1016/j.jsp.2013.11.008.
Chi, M. T. H., Bassok, M., Lewis, M. W., Reimann, P., & Glaser, R. (1989). Self-explanations: How students study and use examples in learning to solve problems. Cognitive Science, 13(2), 145–182. https://doi.org/10.1016/0364-0213(89)90002-5.
Chiu, M. M. (2004). Adapting teacher interventions to student needs during cooperative learning: How to improve student problem solving and time on-task. American Educational Research Journal, 41(2), 365–399. https://doi.org/10.3102/00028312041002365.
Christophel, E., Gaschler, R., & Schnotz, W. (2014). Teachers’ expertise in feedback application adapted to the phases of the learning process. Frontiers in Psychology, 5 https://doi.org/10.3389/fpsyg.2014.00858.
Clark, H. H. (1996). Using language. Cambridge: Cambridge University Press.
Clark, H. H., & Brennan, S. E. (1991). Grounding in communication. In L. B. Resnick, J. M. Levine & S. D. Teasley (Eds.), Perspectives on socially shared cognition (1st edn., pp. 127–148). Washington, D.C: American Psychological Association.
Cronbach, L. J., Linn, R. L., Brennan, R. L., & Haertel, E. H. (1997). Generalizability analysis for performance assessments of student achievement or school effectiveness. Educational and Psychological Measurement, 57(3), 373–399. https://doi.org/10.1177/0013164497057003001.
Cronbach, L. J., Rajaratnam, N., & Gleser, G. C. (1963). Theory of generalizability: A liberalization of reliability theory. British Journal of Statistical Psychology, 16(2), 137–163. https://doi.org/10.1111/j.2044-8317.1963.tb00206.x.
Dillenbourg, P., & Tchounikine, P. (2007). Flexibility in macro-scripts for computer-supported collaborative learning. Journal of Computer Assisted Learning, 23(1), 1–13. https://doi.org/10.1111/j.1365-2729.2007.00191.x.
Dillenbourg, P., Baker, M., Blaye, A., & O’Malley, C. (1996). The evolution of research on collaborative learning. In P. Reimann & H. Spada (Eds.), Learning in humans and machines. Towards an interdisciplinary learning science (1st edn., pp. 189–211). Oxford, New York: Pergamon.
van Es, E. A., & Sherin, M. G. (2002). Learning to notice: Scaffolding new teachers’ interpretations of classroom interactions. Journal of Technology and Teacher Education, 10(4), 571–596.
van Es, E. A., & Sherin, M. G. (2008). Mathematics teachers’ “learning to notice” in the context of a video club. Teaching and Teacher Education, 24(2), 244–276. https://doi.org/10.1016/j.tate.2006.11.005.
Fischer, F., & Mandl, H. (2003). Being there or being where? Videoconferencing and cooperative learning. In H. van Oostendorp (Ed.), Cognition in a digital world (pp. 205–223). Mahwah: Lawrence Erlbaum.
Freelon, D. G. (2010). ReCal: Intercoder reliability calculation as a web service. International Journal of Internet Science, 5(1), 20–33. http://www.ijis.net/ijis5_1/ijis5_1_freelon.pdf
Freelon, D. G. (2013). ReCal OIR: Ordinal, interval, and ratio intercoder reliability as a web service. International Journal of Internet Science, 8(1), 10–16. http://www.ijis.net/ijis8_1/ijis8_1_freelon.pdf
Friedrich, H. F., & Mandl, H. (1997). Analyse und Förderung selbstgesteuerten Lernens [Analysis and training of self-regulated learning]. In F. E. Weinert & H. Mandl (Eds.), Pädagogische Psychologie, Bd. 4. Enzyklopädie der Psychologie. Themenbereich D Praxisgebiete, (Vol. Serie I, pp. 237–293). Göttingen: Hogrefe.
Gold, B., & Holodynski, M. (2017). Using digital video to measure the professional vision of elementary classroom management: Test validation and methodological challenges. Computers & Education, 107, 13–30. https://doi.org/10.1016/j.compedu.2016.12.012.
Goodwin, C. (1994). Professional vision. American Anthropologist, 96, 606–633. https://doi.org/10.1525/aa.1994.96.3.02a00100.
Goos, M., Galbraith, P., & Renshaw, P. (2002). Socially mediated metacognition: Creating collaborative zones of proximal development in small group problem solving. Educ Stud Math, 49(2), 193–223. https://doi.org/10.1023/A:1016209010120.
Jeong, H., & Michelene, C. T. H. (2007). Knowledge convergence and collaborative learning. Instructional Science, 35(4), 287–315. https://doi.org/10.1007/s11251-006-9008-z.
Johnson, D. W., & Johnson, R. T. (1992). Positive interdependence: Key to effective cooperation. In R. Hertz Lazarowitz & N. Miller (Eds.), Interaction in cooperative groups. The theoretical anatomy of group learning 1st edn. Cambridge: Cambridge University Press.
Johnson, D. W., & Johnson, R. T. (2003). Training for cooperative group work. In M. A. West, D. Tjosvold & K. G. Smith (Eds.), International handbook of organizational teamwork and cooperative working (pp. 167–183). Chichester: John Wiley & Sons Ltd.
Johnson, D. W., & Johnson, R. T. (2009). An educational psychology success story: Cooperative learning and social interdependence theory. Educational Researcher, 38(5), 365–379. https://doi.org/10.3102/0013189X09339057.
de Jong, F., Kollöffel, B., van der Meijden, H., Staarman, J. K., & Janssen, J. (2005). Regulative processes in individual, 3D and computer supported cooperative learning contexts. Learning in Innovative Learning Environments, 21(4), 645–670. https://doi.org/10.1016/j.chb.2004.10.023.
Kaendler, C., Wiedmann, M., Spada, H., & Rummel, N. (2014). Teacher competencies for the implementation of collaborative learning in the classroom: A framework and research review. Educational Psychology Review, 27(3), 505–536. https://doi.org/10.1007/s10648-014-9288-9.
Kaendler, C., Wiedmann, M. M., Leuders, T., Rummel, N., & Spada, H. (2016). Monitoring student interaction during collaborative learning: Design and evaluation of a training program for pre-service teachers. Psychology Learning and Teaching, 15(1), 44–64. https://doi.org/10.1177/1475725716638010.
Karakostas, A., & Demetriadis, S. (2011). Enhancing collaborative learning through dynamic forms of support: The impact of an adaptive domain-specific support strategy. Journal of Computer Assisted Learning, 27(3), 243–258. https://doi.org/10.1111/j.1365-2729.2010.00388.x.
King, A. (2007). Scripting collaborative learning processes: A cognitive perspective. In F. Fischer, I. Kollar, H. Mandl & J. M. Haake (Eds.), Scripting computer-supported collaborative learning. Cognitive, computational and educational perspectives. Computer-supported collaborative learning (pp. 13–37). Boston: Springer.
King, A. (2008). Structuring peer interaction to promote higher-order thinking and complex learning in cooperating groups. In R. M. Gillies, A. Ashman & J. Terwel (Eds.), The teacher’s role in implementing cooperative learning in the classroom (pp. 73–92). New York: Springer. Unter Mitarbeit von Robyn M. Gillies.
Kobbe, L., Weinberger, A., Dillenbourg, P., Harrer, A., Hämäläinen, R., Häkkinen, P., & Fischer, F. (2007). Specifying computer-supported collaboration scripts. Computer Supported Learning, 2(2–3), 211–224. https://doi.org/10.1007/s11412-007-9014-4.
Lee, G., & Frisbie, D. A. (1999). Estimating reliability under a generalizability theory model for test scores composed of testlets. Applied Measurement in Education, 12(3), 237–255. https://doi.org/10.1207/S15324818AME1203_.
Lin, T.-J., Jadallah, M., Anderson, R. C., Baker, A. R., Nguyen-Jahiel, K., Kim, I.-H., Kuo, L.-J., Miller, B. W., Dong, T., & Wu, X. (2014). Less is more: Teachers’ influence during peer collaboration. Journal of Educational Psychology. https://doi.org/10.1037/a0037758.
Magnisalis, I., Demetriadis, S., & Karakostas, A. (2011). Adaptive and intelligent systems for collaborative learning support: A review of the field. IEEE Transactions on Learning Technologies, 4(1), 5–20. https://doi.org/10.1109/TLT.2011.2.
Meier, A., Spada, H., & Rummel, N. (2007). A rating scheme for assessing the quality of computer-supported collaboration processes. Computer Supported Learning, 2(1), 63–86. https://doi.org/10.1007/s11412-006-9005-x.
Modell, M. G. (2017). Instructors’ professional vision for collaborative learning groups. Journal of Applied Research in Higher Education, 9(3), 346–362. https://doi.org/10.1108/JARHE-11-2016-0087.
Molenaar, I., Chiu, M. M., Sleegers, P. J. C., & Boxtel, C. (2011). Scaffolding of small groups’ metacognitive activities with an avatar. Computer Supported Learning, 6(4), 601–624. https://doi.org/10.1007/s11412-011-9130-z.
OECD (2017). Collaborative problem solving. https://www.oecd.org/berlin/publikationen/PISA_ProblemSolving_Ergebnisse.pdf https://doi.org/10.1787/cdae6d2e-en. PISA in Focus, No. 78.
Olsen, J. K., Aleven, V., & Rummel, N. (2017). Statistically modeling individual students’ learning over successive collaborative practice opportunities. Journal of Educational Measurement, 54(1), 123–138. https://doi.org/10.1111/jedm.12137
Oser, F., Heinzer, S., & Salzmann, P. (2010). Die Messung der Qualität von professionellen Kompetenzprofilen von Lehrpersonen mit Hilfe der Einschätzung von Filmvignetten. Chancen und Grenzen des advokatorischen Ansatzes [Measuring the quality of professional teaching competency profiles by means of evaluating film-vignettes: Chances and limits of the advocatory approach]. Unterrichtswissenschaft, 38(1), 5–28. https://doi.org/10.3262/UW1001005.
Palincsar, A. S., & Brown, A. L. (1984). Reciprocal teaching of comprehension-fostering and comprehension-monitoring activities. Cognition and Instruction, 1(2), 117–175. https://doi.org/10.1207/s1532690xci0102_1.
Persico, D., Pozzi, F., & Sarti, L. (2010). Monitoring collaborative activities in computer supported collaborative learning. Distance Education, 31(1), 5–22. https://doi.org/10.1080/01587911003724603.
van de Pol, J., Volman, M., Oort, F., & Beishuizen, J. (2014). Teacher scaffolding in small-group work: An intervention study. Journal of the Learning Sciences, 23(4), 600–650. https://doi.org/10.1080/10508406.2013.805300.
Putnam, R. T., & Borko, H. (2000). What do new views of knowledge and thinking have to say about research on teacher learning? Educational Researcher, 29(1), 4–15. https://doi.org/10.2307/1176586.
Salomon, G., & Globerson, T. (1989). When teams do not function the way they ought to. International Journal of Educational Research, 13, 89–98. https://doi.org/10.1016/0883-0355(89)90018-9.
Schoenfeld, A. H. (1999). Looking toward the 21st century: Challenges of educational theory and practice. Educational Researcher, 28(7), 4–14. https://doi.org/10.3102/0013189X028007004.
Seidel, T., & Stürmer, K. (2014). Modeling and measuring the structure of professional vision in preservice teachers. American Educational Research Journal. https://doi.org/10.3102/0002831214531321.
Seidel, T., & Thiel, F. (2017). Standards und Trends der videobasierten Lehr-Lernforschung [Standards and trends in video-based teaching and learning research]. Zeitschrift für Erziehungswissenschaft, 20, 1–21. https://doi.org/10.1007/s11618-017-0726-6.
Seidel, T., Blomberg, G., & Stürmer, K. (2010). “Observer” – Validierung eines videobasierten Instruments zur Erfassung der professionellen Wahrnehmung von Unterricht. Projekt OBSERVE [“Observer” – Validation of a videobased instrument for assessing professional vision of classroom instruction]. In E. Klieme, D. Leutner & M. Kenk (Eds.), Kompetenzmodellierung. Zwischenbilanz des DFG-Schwerpunktprogramms und Perspektiven des Forschungsansatzes (pp. 296–306). Weinheim: Beltz. Themenheft.
Sherin, M. G. (2007). The development of teachers’ professional vision in video clubs. In R. Goldman, R. Pea, B. Barron & S. J. Derry (Eds.), Video research in the learning sciences (pp. 383–395). Mahwah: Erlbaum.
Sherin, M. G., & van Es, E. A. (2005). Using video to support teachers’ ability to notice classroom interactions. Journal of Technology and Teacher Education, 13(3), 475–491.
Sherin, M. G., & van Es, E. A. (2009). Effects of video club participation on teachers’ professional vision. Journal of Teacher Education, 60(1), 20–37. https://doi.org/10.1177/0022487108328155.
Sherin, M. G., & Han, S. Y. (2004). Teacher learning in the context of a video club. Teaching and Teacher Education, 20(2), 163–183. https://doi.org/10.1016/j.tate.2003.08.001.
Sherin, M. G., Linsenmeier, K. A., & van Es, E. A. (2009). Selecting video clips to promote mathematics teachers’ discussion of student thinking. Journal of Teacher Education, 60(3), 213–230. https://doi.org/10.1177/0022487109336967.
Shulman, L. S. (1986). Those who understand: Knowledge growth in teaching. Educational Researcher, 15(2), 4–14. https://doi.org/10.3102/0013189X015002004.
Soller, A., Martinez, A., Jermann, P., & Muehlenbrock, M. (2005). From mirroring to guiding: A review of state of the art technology for supporting collaborative learning. International Journal of Artificial Intelligence in Education, 15(4), 261–290.
Stasser, G., & Titus, W. (1985). Pooling of unshared information in group decision making: Biased information sampling during discussion. Journal of Personality and Social Psychology, 48(6), 1467–1478. https://doi.org/10.1037/0022-3514.48.6.1467.
Stürmer, K., & Seidel, T. (2015). Assessing professional vision in teacher candidates. Zeitschrift für Psychologie, 223(1), 54–63. https://doi.org/10.1027/2151-2604/a000200.
Teasley, S. D. (1997). Talking about reasoning: How important is the peer in peer collaboration? In L. B. Resnick, R. Säljö, C. Pontecorvo & B. Burge (Eds.), Discourse, tools and reasoning. NATO ASI Series, (Vol. 160, pp. 361–384). Berlin, Heidelberg: Springer.
Wainer, H., Dorans, N., Flaugher, R., Green, B., & Mislevy, R. (2000). Computerized adaptive testing. New York: Routledge. https://doi.org/10.4324/9781410605931.
Walker, E., Rummel, N., & Koedinger, K. (2009). CTRL: A research framework for providing adaptive collaborative learning support. User Modeling and User-Adapted Interaction. User Modeling and User-Adapted Interaction, 19(5), 387–431. https://doi.org/10.1007/s11257-009-9069-1
Webb, N. M. (1985). Student interaction and learning in small groups: A research summary. In R. E. Slavin, S. Shlomo, S. Kagan, R. Hertz Lazarowitz, C. Webb & R. Schmuck (Eds.), Learning to cooperate, cooperating to learn (pp. 147–172). New York: Plenum Press.
Webb, N. M. (1989). Peer interaction and learning in small groups. International Journal of Educational Research, 13(1), 21–39. https://doi.org/10.1016/0883-0355(89)90014-1.
Webb, N. M. (1991). Task-related verbal interaction and mathematics learning in small groups. Journal for Research in Mathematics Education, 22(5), 366–389. https://doi.org/10.2307/749186.
Webb, N. M. (1992). Testing a theoretical model of student interaction and learning in small groups. In R. Hertz Lazarowitz & N. Miller (Eds.), Interaction in cooperative groups. The theoretical anatomy of group learning (1st edn., pp. 102–119). Cambridge: Cambridge University Press.
Weinberger, A., Ertl, B., Fischer, F., & Mandl, H. (2005). Epistemic and social scripts in computer-supported collaborative learning. Instructional Science, 33(1), 1–30. https://doi.org/10.1007/s11251-004-2322-4.
Weinberger, A., Stegmann, K., Fischer, F., & Mandl, H. (2007). Scripting argumentative knowledge construction in computer-supported learning environments. In F. Fischer, I. Kollar, H. Mandl & J. M. Haake (Eds.), Scripting computer-supported collaborative learning. Cognitive, computational and educational perspectives. Computer-supported collaborative learning (pp. 191–211). Boston: Springer.
Wiedmann, M. (2015). Teacher’s professional vision of student interaction in collaborative learning settings: Development and evaluation of an assessment tool. Freiburg: Albert-Ludwigs-Universität. Dissertation. https://doi.org/10.6094/UNIFR/10089
Yen, W. M. (1993). Scaling performance assessments: Strategies for managing local item dependence. Journal of Educational Measurement, 30(3), 187–213. https://doi.org/10.2307/1435043.
Acknowledgements
This research was funded by the Graduate School Pro|Mat|Nat (Educational Professionalism in Mathematics and Natural Sciences). The Graduate School was a project of the Competence Network Empirical Research in Education and Teaching (KeBU) of the University of Freiburg and the University of Education, Freiburg. The Graduate School was funded by the state of Baden-Württemberg, Germany. Additional funding was provided by Wissenschaftliche Gesellschaft Freiburg im Breisgau to support the development and evaluation of the assessment tool.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Wiedmann, M., Kaendler, C., Leuders, T. et al. Measuring teachers’ competence to monitor student interaction in collaborative learning settings. Unterrichtswiss 47, 177–199 (2019). https://doi.org/10.1007/s42010-019-00047-6
Published:
Issue Date:
DOI: https://doi.org/10.1007/s42010-019-00047-6
Keywords
- Professional vision
- Monitoring
- Collaborative learning
- Video-based assessment
- Competence assessment
- Professional knowledge