Educational Studies in Mathematics

, Volume 85, Issue 1, pp 109–128 | Cite as

Introducing an observational measure of standards-based mathematics teaching practices: Evidence of validity and score reliability

  • Temple A. WalkowiakEmail author
  • Robert Q. Berry
  • J. Patrick Meyer
  • Sara E. Rimm-Kaufman
  • Erin R. Ottmar


The purpose of this study is to introduce a measure of standards-based mathematics teaching practices, the Mathematics Scan (M-Scan), and to examine its validity and score reliability. First, we define standards-based mathematics teaching practices based on eight dimensions that have emerged in recent conceptualizations by researchers and in the context of existing observational measures. Second, we present three sources of validity evidence: content review by experts, analysis of response processes of coders, and convergent and discriminant patterns with existing observational measures. Third, we provide evidence of inter-coder (or inter-rater) reliability through analyses of variance components and calculation of reliability coefficients, using the framework of generalizability theory. Results show the M-Scan holds promise as a useful tool in mathematics education research, measuring indicators of standards-based teaching practices unique to the subject of mathematics.


Mathematics teaching practices Standards-based Validity Score reliability Observational measure 


  1. American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1999). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.Google Scholar
  2. Australian Association of Mathematics Teachers. (2006). Standards for excellence in teaching mathematics in Australian schools. Adelaide, SA: Author.Google Scholar
  3. Ball, D. L., & Rowan, B. (2004). Introduction: Measuring instruction. The Elementary School Journal, 105(1), 3–10.CrossRefGoogle Scholar
  4. Bartolini-Bussi, M. G., & Mariotti, M. A. (2008). Semiotic mediation in the mathematics classroom: Artifacts and signs after a Vygotskian perspective. In L. English, M. Bartolini-Bussi, G. Jones, R. Lesh, & D. Tirosh (Eds.), Handbook of international research in mathematics education (2nd ed., pp. 746–805). Mahwah, NJ: Lawrence Erlbaum.Google Scholar
  5. Battistich, V., Schaps, E., Watson, M., & Solomon, D. (1996). Prevention effects of the child development project: Early findings from an ongoing multisite demonstration trial. Journal of Adolescent Research, 11(1), 12–35.CrossRefGoogle Scholar
  6. Beguin, P., & Rabardel, P. (2000). Designing for instrument-mediated activity. Scandinavian Journal of Information Systems, 12, 173–190.Google Scholar
  7. Boaler, J. (2002). Learning from teaching: Exploring the relationship between reform curriculum and equity. Journal for Research in Mathematics Education, 33(4), 239–258.CrossRefGoogle Scholar
  8. Boero, P., Bazzini, L., & Garuti, R. (2001). Metaphors in teaching and learning mathematics: A case study concerning inequalities. Proceedings of the 25th PME Conference, Utrecht University, the Netherlands, vol.2, pp. 185–192.
  9. Boero, P., Douek, N., Morselli, F., & Pedemonte, B. (2010). Argumentation and proof: A contribution to theoretical perspectives and their classroom implementation. In M. M. F. Pinto & T. F. Kawasaki (Eds.), Proceedings of the 34th Conference of the International Group for the Psychology of Mathematics Education (Vol. 1, pp. 179–204). Belo Horizonte: PME.Google Scholar
  10. Borko, H., Stecher, B. M., & Alonzo, A. C. (2005). Artifact packages for characterizing classroom practice: A pilot study. Educational Assessment, 10(2), 73–104.CrossRefGoogle Scholar
  11. Bransford, J. D., Brown, A. L., & Cocking, R. R. (Eds.). (1999). How people learn: Brain, mind, experience, and school. Washington, DC: National Academy Press.Google Scholar
  12. Brennan, R. L. (1992). Generalizability theory. Educational Measurement: Issues and Practice, 11(4), 27–34.CrossRefGoogle Scholar
  13. Cameron, C. E., Connor, C. M. D., & Morrison, F. J. (2005). Effects of variation in teacher organization on classroom functioning. Journal of School Psychology, 43(1), 61–85.CrossRefGoogle Scholar
  14. Carpenter, T. P., Franke, M. L., Jacobs, V. R., Fennema, E., & Empson, S. B. (1998). A longitudinal study of invention and understanding in children’s multidigit addition and subtraction. Journal for Research in Mathematics Education, 29(1), 3–20.CrossRefGoogle Scholar
  15. Chomat–Mooney, L., Pianta, R. C., Hamre, B. K., Mashburn, A. J., Luckner, A. E., Grimm, K. J., et al. (2008). A practical guide for conducting classroom observations: A summary of issues and evidence for researchers. Unpublished report to the W. T. Grant Foundation, University of Virginia, Charlottesville.Google Scholar
  16. Civil, M. (2002). Culture and mathematics: A community approach. Journal of Intercultural Studies, 23(2), 133–148.CrossRefGoogle Scholar
  17. diSessa, A., Hammer, D., Sherin, B., & Kolpakowski, T. (1991). Inventing graphing: Children’s meta-representational expertise. The Journal of Mathematical Behavior, 10(2), 117–160.Google Scholar
  18. Duval, R. (2006). A cognitive analysis of problems of comprehension in a learning of mathematics. Educational Studies in Mathematics, 61, 103–131.CrossRefGoogle Scholar
  19. Duval, R. (2007). Cognitive functioning and the understanding of mathematical processes of proof. In P. Boero (Ed.), Theorems in school: From history, epistemology, and cognition to classroom practice (pp. 137–161). Rotterdam, the Netherlands: Sense Publishers.Google Scholar
  20. Embretson, S. E. (1983). Construct validity: Construct representation versus nomothetic span. Psychological Bulletin, 93(1), 179–197.CrossRefGoogle Scholar
  21. Emmer, E. T., & Stough, L. M. (2001). Classroom management: A critical part of educational psychology, with implications for teacher education. Educational Psychologist, 36(2), 103–112.CrossRefGoogle Scholar
  22. Eurydice. (2011). Mathematics education in Europe: Common challenges and national policies. Brussels, Belgium: Education, Audiovisual, and Culture Executive Agency.Google Scholar
  23. Gordon, R., Kane, T. J., & Staiger, D. O. (2006). Identifying effective teachers using performance on the job (No. 1). Washington, DC: The Brookings Institution.Google Scholar
  24. Hamre, B., & Pianta, R. C. (2007). Learning opportunities in preschool and early elementary classrooms. In R. C. Pianta, M. J. Cox, & K. L. Snow (Eds.), School readiness and the transition to kindergarten in the era of accountability. Baltimore, MD: Brookes Publishing.Google Scholar
  25. Hand, V. (2012). Seeing culture and power in mathematical learning: Toward a model of equitable instruction. Educational Studies in Mathematics, 80, 233–247.CrossRefGoogle Scholar
  26. Hiebert, J., & Stigler, J. W. (2000). A proposal for improving classroom teaching: Lessons from the TIMSS video study. The Elementary School Journal, 101(1), 3–20.CrossRefGoogle Scholar
  27. Hiebert, J., Stigler, J. W., Jacobs, J. K., Givvin, K. B., Garnier, H., Smith, M., et al. (2005). Mathematics teaching in the United States today (and tomorrow): Results from the TIMSS 1999 video study. Educational Evaluation and Policy Analysis, 27(2), 111–132.CrossRefGoogle Scholar
  28. Hill, H. C., Sleep, L., Lewis, J. M., & Ball, D. L. (2007). Assessing teachers’ mathematical knowledge: What knowledge matters and what evidence counts? In F. K. Lester (Ed.), Second handbook of research on mathematics teaching and learning (pp. 111–154). Charlotte, NC: National Council of Teachers of Mathematics, Information Age Publishing.Google Scholar
  29. Horizon Research. (2000). Inside the classroom observation and analytic protocol. Retrieved from
  30. Hufferd-Ackles, K., Fuson, K. C., & Sherin, M. G. (2004). Describing levels and components of a math-talk learning community. Journal for Research in Mathematics Education, 35(2), 81–116.CrossRefGoogle Scholar
  31. Kane, M. T. (2006). Validation. In R. L. Brennan (Ed.), Educational measurement (4th ed.). Westport, CT: Praeger Publishers.Google Scholar
  32. Knapp, M. S. (1997). Between systemic reforms and the mathematics and science classroom: The dynamics of innovation, implementation, and professional learning. Review of Educational Research, 67(2), 227–266.CrossRefGoogle Scholar
  33. Lampert, M. (1990). When the problem is not the question and the solution is not the answer: Mathematical knowing and teaching. American Educational Research Journal, 27(1), 29–63.CrossRefGoogle Scholar
  34. Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159–174.CrossRefGoogle Scholar
  35. Learning Mathematics for Teaching. (2006). A coding rubric for measuring the quality of mathematics in instruction, Technical Report LMT. Ann Arbor, MI: University of Michigan, School of Education.Google Scholar
  36. Lehrer, R., & Schauble, L. (2002). Symbolic communication in mathematics and science: Co-constituting inscription and thought. In E. Amsel & J. P. Byrnes (Eds.), Language, literacy, and cognitive development: The development and consequences of symbolic communication (pp. 167–192). Mahwah, NJ: Lawrence Erlbaum.Google Scholar
  37. Lesh, R. A., Post, T. R., & Behr, M. J. (1987). Representations and translations among representations in mathematics learning and problem solving. In C. Janvier (Ed.), Problems of representation in the teaching and learning of mathematics (pp. 33–40). Hillsdale, NJ: Erlbaum.Google Scholar
  38. Malara, N., & Zan, R. (2002). The problematic relationship between theory and practice. In L. D. English (Ed.), Handbook of international research in mathematics education (pp. 553–580). Mahwah, NJ: Lawrence Erlbaum.Google Scholar
  39. Messick, S. (1993). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed.). Phoenix, AZ: Oryx Press.Google Scholar
  40. Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. American Psychologist, 50, 741–741.CrossRefGoogle Scholar
  41. Moyer, P. S. (2001). Are we having fun yet? How teachers use manipulatives to teach mathematics. Educational Studies in Mathematics, 47(2), 175–197.CrossRefGoogle Scholar
  42. National Council of Teachers of Mathematics. (2000). Principles and standards for school mathematics. Reston, VA: National Council of Teachers of Mathematics.Google Scholar
  43. National Council of Teachers of Mathematics. (2007). Mathematics teaching today: Improving practice, improving student learning. Reston, VA: National Council of Teachers of Mathematics.Google Scholar
  44. North Central Regional Educational Laboratory. (2000). A study of the differences between higher and lower performing Indiana schools in reading and mathematics. Naperville, IL: North Central Regional Educational Laboratory.Google Scholar
  45. Nye, B., Konstantopoulos, S., & Hedges, L. V. (2004). How large are teacher effects? Educational Evaluation and Policy Analysis, 26(3), 237.CrossRefGoogle Scholar
  46. Pianta, R. C., Belsky, J., Vandergrift, N., Houts, R., & Morrison, F. J. (2008). Classroom effects on children’s achievement trajectories in elementary school. American Educational Research Journal, 45(2), 365–397.CrossRefGoogle Scholar
  47. Pianta, R. C., La Paro, K. M., & Hamre, B. K. (2008). Classroom assessment scoring system manual, K-3. Baltimore, MD: Brookes Publishing Co.Google Scholar
  48. Piburn, M., Sawada, D., Turley, J., Falconer, K., Benford, R., Bloom, I., et al. (2000). Reformed teaching observation protocol (RTOP): Reference manual. ACEPT Technical Report No. IN00-3). Tempe, AZ: Arizona Collaborative for Excellence in the Preparation of Teachers.Google Scholar
  49. Porter, A. C., Blank, R. K., Smithson, J. L., & Osthoff, E. (2005). Place-based randomized trials to test the effects on instructional practices of a mathematics/science professional development program for teachers. The Annals of the American Academy of Political and Social Science, 599(1), 147–175.CrossRefGoogle Scholar
  50. Radford, L. (2002). The seen, the spoken, and the written: A semiotic approach to problem of objectification of mathematical knowledge. For the Learning of Mathematics, 22(2), 14–23.Google Scholar
  51. Raphael, D., & Wahlstrom, M. (1989). The influence of instructional aids on mathematics achievement. Journal for Research in Mathematics Education, 20(2), 173–190.CrossRefGoogle Scholar
  52. Reid, D. A. (2002). Conjectures and refutations in grade 5 mathematics. Journal for Research in Mathematics Education, 33(1), 5–29.CrossRefGoogle Scholar
  53. Rivkin, S. G., Hanushek, E. A., & Kain, J. F. (2005). Teachers, schools, and academic achievement. Econometrica, 73(2), 417–458.CrossRefGoogle Scholar
  54. Rutter, M., & Maughan, B. (2002). School effectiveness findings 1979–2002. Journal of School Psychology, 40(6), 451–475.CrossRefGoogle Scholar
  55. Sanders, W. L., & Rivers, J. C. (1996). Cumulative and residual effects of teachers on future student academic achievement. Knoxville, TN: University of Tennessee Value-Added Research and Assessment Center.Google Scholar
  56. Sarama, J., & Clements, D. H. (2004). Building blocks for early childhood mathematics. Early Childhood Research Quarterly, 19(1), 181–189.CrossRefGoogle Scholar
  57. Sherin, M. G. (2002). A balancing act: Developing a discourse community in a mathematics classroom. Journal of Mathematics Teacher Education, 5(3), 205–233.CrossRefGoogle Scholar
  58. Sowell, E. J. (1989). Effects of manipulative materials in mathematics instruction. Journal for Research in Mathematics Education, 20(5), 498–505.CrossRefGoogle Scholar
  59. Spillane, J. P., & Zeuli, J. S. (1999). Reform and teaching: Exploring patterns of practice in the context of national and state mathematics reforms. Educational Evaluation and Policy Analysis, 21(1), 1–27.CrossRefGoogle Scholar
  60. Stecher, B., Hamilton, L., Ryan, G., Le, V. N., Williams, V., Robyn, A., et al. (2002). Measuring reform-oriented instructional practices in mathematics and science. Annual Meeting of the American Educational Research Association, New Orleans, LA.Google Scholar
  61. Stein, M. K., Grover, B. W., & Henningsen, M. (1996). Building student capacity for mathematical thinking and reasoning: An analysis of mathematical tasks used in reform classrooms. American Educational Research Journal, 33(2), 455–488.CrossRefGoogle Scholar
  62. Stein, M. K., Smith, M. S., Henningsen, M. A., & Silver, E. A. (2009). Implementing standards-based mathematics instruction: A casebook for professional development (2nd ed.). New York: Teachers College Press.Google Scholar
  63. Stipek, D. J., Givvin, K. B., Salmon, J. M., & MacGyvers, V. L. (2001). Teachers’ beliefs and practices related to mathematics instruction. Teaching and Teacher Education, 17(2), 213–226.CrossRefGoogle Scholar
  64. Sutton, J., & Krueger, A. (Eds.). (2002). EDThoughts: What we know about mathematics teaching and learning. Aurora, CO: Mid-continent Research for Education and Learning.Google Scholar
  65. Toulmin, S. (1969). The uses of argument. Cambridge, England: Cambridge University Press.Google Scholar
  66. Weiss, I. R., Pasley, J. D., Smith, P. S., Banilower, E. R., & Heck, D. J. (2003). Looking inside the classroom: A study of K-12 mathematics and science education in the united states. Chapel Hill, NC: Horizon Research, Inc.Google Scholar
  67. Xie, M. (2009). Mathematics education reform in mainland China in the past sixty years: Reviews and forecasts. Journal of Mathematics Education, 2(1), 121–130.Google Scholar
  68. Yackel, E., & Cobb, P. (1996). Sociomathematical norms, argumentation, and autonomy in mathematics. Journal for Research in Mathematics Education, 27(4), 458–477.CrossRefGoogle Scholar
  69. Yackel, E., Cobb, P., & Wood, T. (1991). Small-group interactions as a source of learning opportunities in second-grade mathematics. Journal for Research in Mathematics Education, 22(5), 390–408.CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media Dordrecht 2013

Authors and Affiliations

  • Temple A. Walkowiak
    • 1
    Email author
  • Robert Q. Berry
    • 2
  • J. Patrick Meyer
    • 2
  • Sara E. Rimm-Kaufman
    • 2
  • Erin R. Ottmar
    • 3
  1. 1.North Carolina State UniversityRaleighUSA
  2. 2.Curry School of EducationUniversity of VirginiaCharlottesvilleUSA
  3. 3.University of RichmondRichmondUSA

Personalised recommendations