Skip to main content
SpringerLink
Log in

Introducing an observational measure of standards-based mathematics teaching practices: Evidence of validity and score reliability

  • Published:
Educational Studies in Mathematics Aims and scope Submit manuscript

Abstract

The purpose of this study is to introduce a measure of standards-based mathematics teaching practices, the Mathematics Scan (M-Scan), and to examine its validity and score reliability. First, we define standards-based mathematics teaching practices based on eight dimensions that have emerged in recent conceptualizations by researchers and in the context of existing observational measures. Second, we present three sources of validity evidence: content review by experts, analysis of response processes of coders, and convergent and discriminant patterns with existing observational measures. Third, we provide evidence of inter-coder (or inter-rater) reliability through analyses of variance components and calculation of reliability coefficients, using the framework of generalizability theory. Results show the M-Scan holds promise as a useful tool in mathematics education research, measuring indicators of standards-based teaching practices unique to the subject of mathematics.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1999). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.

    Google Scholar 

  • Australian Association of Mathematics Teachers. (2006). Standards for excellence in teaching mathematics in Australian schools. Adelaide, SA: Author.

    Google Scholar 

  • Ball, D. L., & Rowan, B. (2004). Introduction: Measuring instruction. The Elementary School Journal, 105(1), 3–10.

    Article  Google Scholar 

  • Bartolini-Bussi, M. G., & Mariotti, M. A. (2008). Semiotic mediation in the mathematics classroom: Artifacts and signs after a Vygotskian perspective. In L. English, M. Bartolini-Bussi, G. Jones, R. Lesh, & D. Tirosh (Eds.), Handbook of international research in mathematics education (2nd ed., pp. 746–805). Mahwah, NJ: Lawrence Erlbaum.

    Google Scholar 

  • Battistich, V., Schaps, E., Watson, M., & Solomon, D. (1996). Prevention effects of the child development project: Early findings from an ongoing multisite demonstration trial. Journal of Adolescent Research, 11(1), 12–35.

    Article  Google Scholar 

  • Beguin, P., & Rabardel, P. (2000). Designing for instrument-mediated activity. Scandinavian Journal of Information Systems, 12, 173–190.

    Google Scholar 

  • Boaler, J. (2002). Learning from teaching: Exploring the relationship between reform curriculum and equity. Journal for Research in Mathematics Education, 33(4), 239–258.

    Article  Google Scholar 

  • Boero, P., Bazzini, L., & Garuti, R. (2001). Metaphors in teaching and learning mathematics: A http://www.horizon-research.com/insidetheclassroom/reports/looking/appendixa.pdf http://www.horizon-research.com/insidetheclassroom/reports/looking/appendixa.pdf case study concerning inequalities. Proceedings of the 25th PME Conference, Utrecht University, the Netherlands, vol.2, pp. 185–192.

  • Boero, P., Douek, N., Morselli, F., & Pedemonte, B. (2010). Argumentation and proof: A contribution to theoretical perspectives and their classroom implementation. In M. M. F. Pinto & T. F. Kawasaki (Eds.), Proceedings of the 34th Conference of the International Group for the Psychology of Mathematics Education (Vol. 1, pp. 179–204). Belo Horizonte: PME.

    Google Scholar 

  • Borko, H., Stecher, B. M., & Alonzo, A. C. (2005). Artifact packages for characterizing classroom practice: A pilot study. Educational Assessment, 10(2), 73–104.

    Article  Google Scholar 

  • Bransford, J. D., Brown, A. L., & Cocking, R. R. (Eds.). (1999). How people learn: Brain, mind, experience, and school. Washington, DC: National Academy Press.

    Google Scholar 

  • Brennan, R. L. (1992). Generalizability theory. Educational Measurement: Issues and Practice, 11(4), 27–34.

    Article  Google Scholar 

  • Cameron, C. E., Connor, C. M. D., & Morrison, F. J. (2005). Effects of variation in teacher organization on classroom functioning. Journal of School Psychology, 43(1), 61–85.

    Article  Google Scholar 

  • Carpenter, T. P., Franke, M. L., Jacobs, V. R., Fennema, E., & Empson, S. B. (1998). A longitudinal study of invention and understanding in children’s multidigit addition and subtraction. Journal for Research in Mathematics Education, 29(1), 3–20.

    Article  Google Scholar 

  • Chomat–Mooney, L., Pianta, R. C., Hamre, B. K., Mashburn, A. J., Luckner, A. E., Grimm, K. J., et al. (2008). A practical guide for conducting classroom observations: A summary of issues and evidence for researchers. Unpublished report to the W. T. Grant Foundation, University of Virginia, Charlottesville.

  • Civil, M. (2002). Culture and mathematics: A community approach. Journal of Intercultural Studies, 23(2), 133–148.

    Article  Google Scholar 

  • diSessa, A., Hammer, D., Sherin, B., & Kolpakowski, T. (1991). Inventing graphing: Children’s meta-representational expertise. The Journal of Mathematical Behavior, 10(2), 117–160.

    Google Scholar 

  • Duval, R. (2006). A cognitive analysis of problems of comprehension in a learning of mathematics. Educational Studies in Mathematics, 61, 103–131.

    Article  Google Scholar 

  • Duval, R. (2007). Cognitive functioning and the understanding of mathematical processes of proof. In P. Boero (Ed.), Theorems in school: From history, epistemology, and cognition to classroom practice (pp. 137–161). Rotterdam, the Netherlands: Sense Publishers.

    Google Scholar 

  • Embretson, S. E. (1983). Construct validity: Construct representation versus nomothetic span. Psychological Bulletin, 93(1), 179–197.

    Article  Google Scholar 

  • Emmer, E. T., & Stough, L. M. (2001). Classroom management: A critical part of educational psychology, with implications for teacher education. Educational Psychologist, 36(2), 103–112.

    Article  Google Scholar 

  • Eurydice. (2011). Mathematics education in Europe: Common challenges and national policies. Brussels, Belgium: Education, Audiovisual, and Culture Executive Agency.

    Google Scholar 

  • Gordon, R., Kane, T. J., & Staiger, D. O. (2006). Identifying effective teachers using performance on the job (No. 1). Washington, DC: The Brookings Institution.

    Google Scholar 

  • Hamre, B., & Pianta, R. C. (2007). Learning opportunities in preschool and early elementary classrooms. In R. C. Pianta, M. J. Cox, & K. L. Snow (Eds.), School readiness and the transition to kindergarten in the era of accountability. Baltimore, MD: Brookes Publishing.

    Google Scholar 

  • Hand, V. (2012). Seeing culture and power in mathematical learning: Toward a model of equitable instruction. Educational Studies in Mathematics, 80, 233–247.

    Article  Google Scholar 

  • Hiebert, J., & Stigler, J. W. (2000). A proposal for improving classroom teaching: Lessons from the TIMSS video study. The Elementary School Journal, 101(1), 3–20.

    Article  Google Scholar 

  • Hiebert, J., Stigler, J. W., Jacobs, J. K., Givvin, K. B., Garnier, H., Smith, M., et al. (2005). Mathematics teaching in the United States today (and tomorrow): Results from the TIMSS 1999 video study. Educational Evaluation and Policy Analysis, 27(2), 111–132.

    Article  Google Scholar 

  • Hill, H. C., Sleep, L., Lewis, J. M., & Ball, D. L. (2007). Assessing teachers’ mathematical knowledge: What knowledge matters and what evidence counts? In F. K. Lester (Ed.), Second handbook of research on mathematics teaching and learning (pp. 111–154). Charlotte, NC: National Council of Teachers of Mathematics, Information Age Publishing.

    Google Scholar 

  • Horizon Research. (2000). Inside the classroom observation and analytic protocol. Retrieved from http://www.horizon-research.com/insidetheclassroom/reports/looking/appendixa.pdf

  • Hufferd-Ackles, K., Fuson, K. C., & Sherin, M. G. (2004). Describing levels and components of a math-talk learning community. Journal for Research in Mathematics Education, 35(2), 81–116.

    Article  Google Scholar 

  • Kane, M. T. (2006). Validation. In R. L. Brennan (Ed.), Educational measurement (4th ed.). Westport, CT: Praeger Publishers.

    Google Scholar 

  • Knapp, M. S. (1997). Between systemic reforms and the mathematics and science classroom: The dynamics of innovation, implementation, and professional learning. Review of Educational Research, 67(2), 227–266.

    Article  Google Scholar 

  • Lampert, M. (1990). When the problem is not the question and the solution is not the answer: Mathematical knowing and teaching. American Educational Research Journal, 27(1), 29–63.

    Article  Google Scholar 

  • Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159–174.

    Article  Google Scholar 

  • Learning Mathematics for Teaching. (2006). A coding rubric for measuring the quality of mathematics in instruction, Technical Report LMT. Ann Arbor, MI: University of Michigan, School of Education.

    Google Scholar 

  • Lehrer, R., & Schauble, L. (2002). Symbolic communication in mathematics and science: Co-constituting inscription and thought. In E. Amsel & J. P. Byrnes (Eds.), Language, literacy, and cognitive development: The development and consequences of symbolic communication (pp. 167–192). Mahwah, NJ: Lawrence Erlbaum.

    Google Scholar 

  • Lesh, R. A., Post, T. R., & Behr, M. J. (1987). Representations and translations among representations in mathematics learning and problem solving. In C. Janvier (Ed.), Problems of representation in the teaching and learning of mathematics (pp. 33–40). Hillsdale, NJ: Erlbaum.

    Google Scholar 

  • Malara, N., & Zan, R. (2002). The problematic relationship between theory and practice. In L. D. English (Ed.), Handbook of international research in mathematics education (pp. 553–580). Mahwah, NJ: Lawrence Erlbaum.

    Google Scholar 

  • Messick, S. (1993). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed.). Phoenix, AZ: Oryx Press.

    Google Scholar 

  • Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. American Psychologist, 50, 741–741.

    Article  Google Scholar 

  • Moyer, P. S. (2001). Are we having fun yet? How teachers use manipulatives to teach mathematics. Educational Studies in Mathematics, 47(2), 175–197.

    Article  Google Scholar 

  • National Council of Teachers of Mathematics. (2000). Principles and standards for school mathematics. Reston, VA: National Council of Teachers of Mathematics.

    Google Scholar 

  • National Council of Teachers of Mathematics. (2007). Mathematics teaching today: Improving practice, improving student learning. Reston, VA: National Council of Teachers of Mathematics.

    Google Scholar 

  • North Central Regional Educational Laboratory. (2000). A study of the differences between higher and lower performing Indiana schools in reading and mathematics. Naperville, IL: North Central Regional Educational Laboratory.

    Google Scholar 

  • Nye, B., Konstantopoulos, S., & Hedges, L. V. (2004). How large are teacher effects? Educational Evaluation and Policy Analysis, 26(3), 237.

    Article  Google Scholar 

  • Pianta, R. C., Belsky, J., Vandergrift, N., Houts, R., & Morrison, F. J. (2008). Classroom effects on children’s achievement trajectories in elementary school. American Educational Research Journal, 45(2), 365–397.

    Article  Google Scholar 

  • Pianta, R. C., La Paro, K. M., & Hamre, B. K. (2008). Classroom assessment scoring system manual, K-3. Baltimore, MD: Brookes Publishing Co.

    Google Scholar 

  • Piburn, M., Sawada, D., Turley, J., Falconer, K., Benford, R., Bloom, I., et al. (2000). Reformed teaching observation protocol (RTOP): Reference manual. ACEPT Technical Report No. IN00-3). Tempe, AZ: Arizona Collaborative for Excellence in the Preparation of Teachers.

    Google Scholar 

  • Porter, A. C., Blank, R. K., Smithson, J. L., & Osthoff, E. (2005). Place-based randomized trials to test the effects on instructional practices of a mathematics/science professional development program for teachers. The Annals of the American Academy of Political and Social Science, 599(1), 147–175.

    Article  Google Scholar 

  • Radford, L. (2002). The seen, the spoken, and the written: A semiotic approach to problem of objectification of mathematical knowledge. For the Learning of Mathematics, 22(2), 14–23.

    Google Scholar 

  • Raphael, D., & Wahlstrom, M. (1989). The influence of instructional aids on mathematics achievement. Journal for Research in Mathematics Education, 20(2), 173–190.

    Article  Google Scholar 

  • Reid, D. A. (2002). Conjectures and refutations in grade 5 mathematics. Journal for Research in Mathematics Education, 33(1), 5–29.

    Article  Google Scholar 

  • Rivkin, S. G., Hanushek, E. A., & Kain, J. F. (2005). Teachers, schools, and academic achievement. Econometrica, 73(2), 417–458.

    Article  Google Scholar 

  • Rutter, M., & Maughan, B. (2002). School effectiveness findings 1979–2002. Journal of School Psychology, 40(6), 451–475.

    Article  Google Scholar 

  • Sanders, W. L., & Rivers, J. C. (1996). Cumulative and residual effects of teachers on future student academic achievement. Knoxville, TN: University of Tennessee Value-Added Research and Assessment Center.

    Google Scholar 

  • Sarama, J., & Clements, D. H. (2004). Building blocks for early childhood mathematics. Early Childhood Research Quarterly, 19(1), 181–189.

    Article  Google Scholar 

  • Sherin, M. G. (2002). A balancing act: Developing a discourse community in a mathematics classroom. Journal of Mathematics Teacher Education, 5(3), 205–233.

    Article  Google Scholar 

  • Sowell, E. J. (1989). Effects of manipulative materials in mathematics instruction. Journal for Research in Mathematics Education, 20(5), 498–505.

    Article  Google Scholar 

  • Spillane, J. P., & Zeuli, J. S. (1999). Reform and teaching: Exploring patterns of practice in the context of national and state mathematics reforms. Educational Evaluation and Policy Analysis, 21(1), 1–27.

    Article  Google Scholar 

  • Stecher, B., Hamilton, L., Ryan, G., Le, V. N., Williams, V., Robyn, A., et al. (2002). Measuring reform-oriented instructional practices in mathematics and science. Annual Meeting of the American Educational Research Association, New Orleans, LA.

  • Stein, M. K., Grover, B. W., & Henningsen, M. (1996). Building student capacity for mathematical thinking and reasoning: An analysis of mathematical tasks used in reform classrooms. American Educational Research Journal, 33(2), 455–488.

    Article  Google Scholar 

  • Stein, M. K., Smith, M. S., Henningsen, M. A., & Silver, E. A. (2009). Implementing standards-based mathematics instruction: A casebook for professional development (2nd ed.). New York: Teachers College Press.

    Google Scholar 

  • Stipek, D. J., Givvin, K. B., Salmon, J. M., & MacGyvers, V. L. (2001). Teachers’ beliefs and practices related to mathematics instruction. Teaching and Teacher Education, 17(2), 213–226.

    Article  Google Scholar 

  • Sutton, J., & Krueger, A. (Eds.). (2002). EDThoughts: What we know about mathematics teaching and learning. Aurora, CO: Mid-continent Research for Education and Learning.

    Google Scholar 

  • Toulmin, S. (1969). The uses of argument. Cambridge, England: Cambridge University Press.

    Google Scholar 

  • Weiss, I. R., Pasley, J. D., Smith, P. S., Banilower, E. R., & Heck, D. J. (2003). Looking inside the classroom: A study of K-12 mathematics and science education in the united states. Chapel Hill, NC: Horizon Research, Inc.

    Google Scholar 

  • Xie, M. (2009). Mathematics education reform in mainland China in the past sixty years: Reviews and forecasts. Journal of Mathematics Education, 2(1), 121–130.

    Google Scholar 

  • Yackel, E., & Cobb, P. (1996). Sociomathematical norms, argumentation, and autonomy in mathematics. Journal for Research in Mathematics Education, 27(4), 458–477.

    Article  Google Scholar 

  • Yackel, E., Cobb, P., & Wood, T. (1991). Small-group interactions as a source of learning opportunities in second-grade mathematics. Journal for Research in Mathematics Education, 22(5), 390–408.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. North Carolina State University, 317-D Poe Hall, Campus Box 7801, Raleigh, NC, 27695-7801, USA

    Temple A. Walkowiak

  2. Curry School of Education, University of Virginia, Charlottesville, VA, USA

    Robert Q. Berry, J. Patrick Meyer & Sara E. Rimm-Kaufman

  3. University of Richmond, Richmond, VA, USA

    Erin R. Ottmar

Authors
  1. Temple A. Walkowiak
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Robert Q. Berry
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. J. Patrick Meyer
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Sara E. Rimm-Kaufman
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Erin R. Ottmar
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Temple A. Walkowiak.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Walkowiak, T.A., Berry, R.Q., Meyer, J.P. et al. Introducing an observational measure of standards-based mathematics teaching practices: Evidence of validity and score reliability. Educ Stud Math 85, 109–128 (2014). https://doi.org/10.1007/s10649-013-9499-x

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10649-013-9499-x

Keywords

Search

Navigation