Advertisement

ZDM

, Volume 50, Issue 3, pp 395–406 | Cite as

Assessing individual lessons using a generic teacher observation instrument: how useful is the International System for Teacher Observation and Feedback (ISTOF)?

  • Daniel Muijs
  • David Reynolds
  • Pamela Sammons
  • Leonidas Kyriakides
  • Bert P. M. Creemers
  • Charles Teddlie
Original Article
First Online:

Abstract

Teacher effectiveness, which impacts student attainment even when controlling for student characteristics, is of key importance as a factor in educational effectiveness and improvement. Improving the quality of teaching is thus the primary means by which we can enhance student learning outcomes. Thus there has long been great interest in the development of classroom observation measures in the field of educational effectiveness research (EER). The International System for Teacher Observation and Feedback (ISTOF) is a unique instrument in the field, as it was developed by a team from 20 countries using an iterative Delphi process to ensure cross-cultural relevance and validity. While previous studies have looked at psychometric properties of the instrument, they have not interrogated the extent to which ISTOF is useful for evaluating individual lessons and providing feedback to teachers. In this study, we observed three grade 4 mathematics lessons taken from the NCTE video library at Harvard University for this purpose. Findings show that ISTOF can provide a highly differentiated and fine-grained picture of individual lessons, but that the strengths of the generic approach in terms of breadth are to an extent counterbalanced by limitations such as the lack of attention to content richness.

Keywords

Classroom observation Teacher effectiveness ISTOF Mathematics Educational effectiveness 
This is a preview of subscription content, log in to check access.

References

  1. Antoniou, P., Kyriakides, L., & Creemers, B. P. M. (2015). The Dynamic Integrated Approach to teacher professional development: Rationale and main characteristics. Teacher Development, 19(4), 535–552.  https://doi.org/10.1080/13664530.2015.1079550.CrossRefGoogle Scholar
  2. Black, P., & William, D. (1998). Assessment and classroom learning. Assessment in Education, 5(1), 7–74.  https://doi.org/10.1080/0969595980050102.CrossRefGoogle Scholar
  3. Brophy, J. (1986). Teacher influences on student achievement. American Psychologist, 41(10), 1069–1077.  https://doi.org/10.1037/0003-066X.41.10.1069.CrossRefGoogle Scholar
  4. Chapman, C., Muijs, D., Reynolds, D., Sammons, P., & Teddlie, C. (2015). Routledge international handbook of educational effectiveness and improvement research. London: Taylor and Francis.Google Scholar
  5. Coe, R., Aloisi, C., Higgins, S., & Elliott Major, L. (2014). What makes great teaching? A review of the underpinning research. London: The Sutton Trust.Google Scholar
  6. Devine, D., Fahie, D., & McGillicuddy, D. (2013). What is ‘good’ teaching? Teacher beliefs and practices about their teaching. Irish Educational Studies, 32(1), 83–108.  https://doi.org/10.1080/03323315.2013.773228.CrossRefGoogle Scholar
  7. Devine, D., Fahie, E., McGillicuddy, D., MacRuairc, G. &. Harford, J. (2010). Report on the use of the ISTOF (International System of Teacher Observation and Feedback) protocol in Irish schools; Challenges, issues and teacher effect. Dublin: School of Education, University College Dublin.Google Scholar
  8. Dignath, C., & Buttner, G. (2008). Components of fostering self-regulated learning among students. A meta-analysis on intervention studies at primary and secondary school level. Metacognition and Learning, 3(2), 231–264.  https://doi.org/10.1007/s11409-008-9029-x.CrossRefGoogle Scholar
  9. Flanders, N. (1970). Analyzing teacher behavior. Reading, MA: Addison Wesley.Google Scholar
  10. Glaser, B. G., & Strauss, A. L. (1967). The discovery of grounded theory: strategies for qualitative research. New York: Aldine DeGruyter.Google Scholar
  11. Gordon, T. J., & Helmer, O. (1964). Report on a long range forecasting study. Rand Paper P-2982. Santa Monica, CA: Rand Corporation.Google Scholar
  12. Hall, J., Lindorff, A., & Sammons, P. (2016). Evaluation of the impact and implementation of Inspire Maths in Year 1 classrooms in England; Findings from a mixed-method randomised control trial. Oxford: University of Oxford.Google Scholar
  13. Halpin, P., & Kieffer, M. (2015). Describing profiles of instructional practice: a new approach to analyzing classroom observation data. Educational Researcher, 44(5), 263–277.  https://doi.org/10.3102/0013189X15590804.CrossRefGoogle Scholar
  14. Heylighen, F. (2003). Web dictionary of cybernetics and systems. Retrieved May 5, 2017 from http://pespmc1.vub.ac.be/ASC.
  15. Kington, A., Day, C., Sammons, P., Regan, E., & Brown, E. (2009). Effective classroom practise: A mixed-method study of influences and outcomes. Symposium paper presented at the British Educational Research Association Annual Conference, University of Manchester, 2–5 September 2009.Google Scholar
  16. Klieme, E., Pauli, C., & Reusser, K. (2009). The Pythagoras Study: Investigating effects of teaching and learning in Swiss and German mathematics classrooms. In T. Janik & T. Seider (Eds.), The power of video studies in investigating teaching and learning in the classroom (pp. 137–160). Munster: Waxmann.Google Scholar
  17. Ko, J. (2010). Consistency and variation in classroom practice: A mixed-method investigation based on case studies of four EFL teachers of a disadvantaged secondary school in Hong Kong. Doctoral thesis. Nottingham, UK: University of Nottingham. Retrieved from: http://etheses.nottingham.ac.uk/1363/1/CVCP_SUBMISSION_(FINAL)PB3.pdf [Accessed May2017).
  18. Ko, J., & Sammons, P. (2008). Variations in effective classroom practices: Confirmatory factor analysis results from analysis of measures from the International System for Teacher Observation and Feedback (ISTOF) Scale and Quality and Teaching Lesson Observation Indicator (GRIFT) Scale. Nottingham: University of Nottingham, School of Education.Google Scholar
  19. Kyriakides, L. (2008). Results from the ISTOF instrument. Paper presented at the Annual Meeting of the American Educational Research Association, New York, 17 April 2008.Google Scholar
  20. Marciniak, J., & Janssen, R. (2012). The International System for Teacher Observation and Feedback questionnaire in the biology assessment–theory, evaluation, utility. Paper presented at Biennial Meeting of the Special Interest Group Educational Effectiveness (SIG 18) of the European Association for Research on Learning and Instruction (EARLI), Zurich, Switzerland 29–31 August 2012.Google Scholar
  21. Meyer, J. P., Cash, A. H., & Mashburn, A. (2011). Occasions and the reliability of classroom observations: alternative conceptualizations and methods of analysis. Educational Assessment, 16(4), 227–243.  https://doi.org/10.1080/10627197.2011.638884.CrossRefGoogle Scholar
  22. Miao, Z., Reynolds, D., Harris, A., & Jones, M. (2015). Comparing performance: A cross-national investigation into the teaching of mathematics in primary classrooms in England and China. Asia Pacific Journal of Education, 35(3), 392–403.  https://doi.org/10.1080/02188791.2015.1056593.CrossRefGoogle Scholar
  23. Mortimore, P., Sammons, P., Stoll, L., Lewis, D., & Russell, E. (1988). School matters: The junior years. Wells: Open Books.Google Scholar
  24. Muijs, D., Chapman, C., Collins, A., & Armstrong, P. (2010). Maximum impact evaluation. The impact of Teach First teachers in schools. Manchester: University of Manchester, School of Education.Google Scholar
  25. Muijs, D., Kyriakides, L., van der Werf, G., Creemers, B., Timperley, H., & Earl, L. (2014). State of the art–teacher effectiveness and professional learning. School Effectiveness and School Improvement, 25(2), 231–256.  https://doi.org/10.1080/09243453.20192011.4.885451.CrossRefGoogle Scholar
  26. Muijs, D., & Reynolds, D. (2000). School effectiveness and teacher effectiveness: some preliminary findings from the evaluation of the Mathematics Enhancement Programme. School Effectiveness and School Improvement, 11(3), 247–263.  https://doi.org/10.1076/0924-3453(200009)11:3;1-G;FT273.CrossRefGoogle Scholar
  27. Reynolds, D. (2006). World class schools: Some methodological and substantive findings and implications of the International School Effectiveness Research Project (ISERP). Educational Research and Evaluation, 12(6), 535–560.  https://doi.org/10.1080/13803610600874026.CrossRefGoogle Scholar
  28. Reynolds, D., Creemers, B., Stringfield, S., Teddlie, C., & Schaffer, G. (Eds.). (2002). World class schools: International perspectives on school effectiveness. London: Routledge Falmer.Google Scholar
  29. Reynolds, D., Salom, K., Delaiglesia, B., & Ramon, R. Mallorca (2012). The ISTOF project-A preliminary report. Spain: IAQSE (Institut d’ Avaluacio del Sistema Education), Ministry of Education of the Balearic Isles.Google Scholar
  30. Sammons, P., Day, C., Kington, A., Gu, Q., Stobart, G., & Smees, R. (2007). Exploring variations in teachers’ work, lives and their effects on pupils: Key findings and implications from a longitudinal mixed methods study. British Educational Research Journal, 33(5), 681–701.  https://doi.org/10.1080/01411920701582264.CrossRefGoogle Scholar
  31. Sammons, P., Kington, A., Lindorff-Vijayendran, A., & Ortega, L. (2014). Inspiring teaching: What we can learn from exemplary practitioners. Reading: CfBT.Google Scholar
  32. Sammons, P., Lindorff, A., Ortega, L., & Kington, A. (2016). Inspiring teaching: Learning from exemplary practitioners. Journal of Professional Capital and Community, 1(2), 124–144.  https://doi.org/10.1108/JPCC-09-2015-0005.CrossRefGoogle Scholar
  33. Soderlund, G., Sorlie, K., & Syse, I. (2015). Mestringsforventninger i matematikk. Paper presented at Finnut, Lilyhammer, 29-01-2015.Google Scholar
  34. Sterbinsky, A., & Ross, S. (2003). School observation measure reliability study. Washington, DC: Centre for Research in Education Policy.Google Scholar
  35. Teddlie, C., Creemers, B., Kyriakides, L., Muijs, D., & Yu, F. (2006). The International System for Teacher Observation and Feedback: Evolution of an international study of teacher effectiveness constructs. Educational Research and Evaluation, 12(6), 561–582.  https://doi.org/10.1080/13803610600874067.CrossRefGoogle Scholar
  36. Teddlie, C., Kirby, P. C., & Stringfield, S. (1989). Effective versus ineffective schools: Observable differences in the classroom. American Journal of Education, 97(3), 221–236.CrossRefGoogle Scholar
  37. Teddlie, C., Kryiakides, L., & Yu, F. (2004). A proposal to develop an internationally valid teacher observation system: The International System for Teacher Observation and Feedback (ISTOF). Paper presented at the Annual Meeting of the International Congress for School Effectiveness and Improvement, Rotterdam, The Netherlands, 5 January 2004.Google Scholar
  38. Tsai, C. C. (2001). Relationships between student scientific epistemological beliefs and perceptions of constructivist learning environments. Educational Research, 42(2), 193–205.  https://doi.org/10.1080/001318800363836.CrossRefGoogle Scholar
  39. van de Grift, W. (2007). Quality of teaching in four European countries: A review of the literature and application of an assessment instrument. Educational Research, 49(2), 127–152.  https://doi.org/10.1080/00131880701369651.CrossRefGoogle Scholar
  40. van der Lans, R., van de Grift, W., van Veen, K., & Fokkens-Bruinsma, M. (2016). Once is not enough: Establishing reliability criteria for feedback and evaluation decisions based on classroom observations. Studies in Educational Evaluation, 50(1), 88–95.  https://doi.org/10.1016/j.stueduc.2016.08.001.CrossRefGoogle Scholar

Copyright information

© FIZ Karlsruhe 2018

Authors and Affiliations

  1. 1.OfstedLondonUK
  2. 2.University of SwanseaSwanseaUK
  3. 3.University of OxfordOxfordUK
  4. 4.University of CyprusNicosiaCyprus
  5. 5.University of GroningenGroningenThe Netherlands
  6. 6.Louisiana State UniversityBaton RougeUSA

Personalised recommendations

Cite article

Buy options