# Introducing an observational measure of standards-based mathematics teaching practices: Evidence of validity and score reliability

- 1.1k Downloads
- 15 Citations

## Abstract

The purpose of this study is to introduce a measure of standards-based mathematics teaching practices, the Mathematics Scan (M-Scan), and to examine its validity and score reliability. First, we define standards-based mathematics teaching practices based on eight dimensions that have emerged in recent conceptualizations by researchers and in the context of existing observational measures. Second, we present three sources of validity evidence: content review by experts, analysis of response processes of coders, and convergent and discriminant patterns with existing observational measures. Third, we provide evidence of inter-coder (or inter-rater) reliability through analyses of variance components and calculation of reliability coefficients, using the framework of generalizability theory. Results show the M-Scan holds promise as a useful tool in mathematics education research, measuring indicators of standards-based teaching practices unique to the subject of mathematics.

## Keywords

Mathematics teaching practices Standards-based Validity Score reliability Observational measure## References

- American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1999).
*Standards for educational and psychological testing*. Washington, DC: American Educational Research Association.Google Scholar - Australian Association of Mathematics Teachers. (2006).
*Standards for excellence in teaching mathematics in Australian schools*. Adelaide, SA: Author.Google Scholar - Ball, D. L., & Rowan, B. (2004). Introduction: Measuring instruction.
*The Elementary School Journal, 105*(1), 3–10.CrossRefGoogle Scholar - Bartolini-Bussi, M. G., & Mariotti, M. A. (2008). Semiotic mediation in the mathematics classroom: Artifacts and signs after a Vygotskian perspective. In L. English, M. Bartolini-Bussi, G. Jones, R. Lesh, & D. Tirosh (Eds.),
*Handbook of international research in mathematics education*(2nd ed., pp. 746–805). Mahwah, NJ: Lawrence Erlbaum.Google Scholar - Battistich, V., Schaps, E., Watson, M., & Solomon, D. (1996). Prevention effects of the child development project: Early findings from an ongoing multisite demonstration trial.
*Journal of Adolescent Research, 11*(1), 12–35.CrossRefGoogle Scholar - Beguin, P., & Rabardel, P. (2000). Designing for instrument-mediated activity.
*Scandinavian Journal of Information Systems, 12*, 173–190.Google Scholar - Boaler, J. (2002). Learning from teaching: Exploring the relationship between reform curriculum and equity.
*Journal for Research in Mathematics Education, 33*(4), 239–258.CrossRefGoogle Scholar - Boero, P., Bazzini, L., & Garuti, R. (2001). Metaphors in teaching and learning mathematics: A http://www.horizon-research.com/insidetheclassroom/reports/looking/appendixa.pdf http://www.horizon-research.com/insidetheclassroom/reports/looking/appendixa.pdf case study concerning inequalities.
*Proceedings of the 25th PME Conference*, Utrecht University, the Netherlands, vol.2, pp. 185–192. - Boero, P., Douek, N., Morselli, F., & Pedemonte, B. (2010). Argumentation and proof: A contribution to theoretical perspectives and their classroom implementation. In M. M. F. Pinto & T. F. Kawasaki (Eds.),
*Proceedings of the 34th Conference of the International Group for the Psychology of Mathematics Education*(Vol. 1, pp. 179–204). Belo Horizonte: PME.Google Scholar - Borko, H., Stecher, B. M., & Alonzo, A. C. (2005). Artifact packages for characterizing classroom practice: A pilot study.
*Educational Assessment, 10*(2), 73–104.CrossRefGoogle Scholar - Bransford, J. D., Brown, A. L., & Cocking, R. R. (Eds.). (1999).
*How people learn: Brain, mind, experience, and school*. Washington, DC: National Academy Press.Google Scholar - Brennan, R. L. (1992). Generalizability theory.
*Educational Measurement: Issues and Practice, 11*(4), 27–34.CrossRefGoogle Scholar - Cameron, C. E., Connor, C. M. D., & Morrison, F. J. (2005). Effects of variation in teacher organization on classroom functioning.
*Journal of School Psychology, 43*(1), 61–85.CrossRefGoogle Scholar - Carpenter, T. P., Franke, M. L., Jacobs, V. R., Fennema, E., & Empson, S. B. (1998). A longitudinal study of invention and understanding in children’s multidigit addition and subtraction.
*Journal for Research in Mathematics Education, 29*(1), 3–20.CrossRefGoogle Scholar - Chomat–Mooney, L., Pianta, R. C., Hamre, B. K., Mashburn, A. J., Luckner, A. E., Grimm, K. J., et al. (2008).
*A practical guide for conducting classroom observations: A summary of issues and evidence for researchers.*Unpublished report to the W. T. Grant Foundation, University of Virginia, Charlottesville.Google Scholar - Civil, M. (2002). Culture and mathematics: A community approach.
*Journal of Intercultural Studies, 23*(2), 133–148.CrossRefGoogle Scholar - diSessa, A., Hammer, D., Sherin, B., & Kolpakowski, T. (1991). Inventing graphing: Children’s meta-representational expertise.
*The Journal of Mathematical Behavior, 10*(2), 117–160.Google Scholar - Duval, R. (2006). A cognitive analysis of problems of comprehension in a learning of mathematics.
*Educational Studies in Mathematics, 61*, 103–131.CrossRefGoogle Scholar - Duval, R. (2007). Cognitive functioning and the understanding of mathematical processes of proof. In P. Boero (Ed.),
*Theorems in school: From history, epistemology, and cognition to classroom practice*(pp. 137–161). Rotterdam, the Netherlands: Sense Publishers.Google Scholar - Embretson, S. E. (1983). Construct validity: Construct representation versus nomothetic span.
*Psychological Bulletin, 93*(1), 179–197.CrossRefGoogle Scholar - Emmer, E. T., & Stough, L. M. (2001). Classroom management: A critical part of educational psychology, with implications for teacher education.
*Educational Psychologist, 36*(2), 103–112.CrossRefGoogle Scholar - Eurydice. (2011).
*Mathematics education in Europe: Common challenges and national policies*. Brussels, Belgium: Education, Audiovisual, and Culture Executive Agency.Google Scholar - Gordon, R., Kane, T. J., & Staiger, D. O. (2006).
*Identifying effective teachers using performance on the job (No. 1)*. Washington, DC: The Brookings Institution.Google Scholar - Hamre, B., & Pianta, R. C. (2007). Learning opportunities in preschool and early elementary classrooms. In R. C. Pianta, M. J. Cox, & K. L. Snow (Eds.),
*School readiness and the transition to kindergarten in the era of accountability*. Baltimore, MD: Brookes Publishing.Google Scholar - Hand, V. (2012). Seeing culture and power in mathematical learning: Toward a model of equitable instruction.
*Educational Studies in Mathematics, 80*, 233–247.CrossRefGoogle Scholar - Hiebert, J., & Stigler, J. W. (2000). A proposal for improving classroom teaching: Lessons from the TIMSS video study.
*The Elementary School Journal, 101*(1), 3–20.CrossRefGoogle Scholar - Hiebert, J., Stigler, J. W., Jacobs, J. K., Givvin, K. B., Garnier, H., Smith, M., et al. (2005). Mathematics teaching in the United States today (and tomorrow): Results from the TIMSS 1999 video study.
*Educational Evaluation and Policy Analysis, 27*(2), 111–132.CrossRefGoogle Scholar - Hill, H. C., Sleep, L., Lewis, J. M., & Ball, D. L. (2007). Assessing teachers’ mathematical knowledge: What knowledge matters and what evidence counts? In F. K. Lester (Ed.),
*Second handbook of research on mathematics teaching and learning*(pp. 111–154). Charlotte, NC: National Council of Teachers of Mathematics, Information Age Publishing.Google Scholar - Horizon Research. (2000).
*Inside the classroom observation and analytic protocol*. Retrieved from http://www.horizon-research.com/insidetheclassroom/reports/looking/appendixa.pdf - Hufferd-Ackles, K., Fuson, K. C., & Sherin, M. G. (2004). Describing levels and components of a math-talk learning community.
*Journal for Research in Mathematics Education, 35*(2), 81–116.CrossRefGoogle Scholar - Kane, M. T. (2006). Validation. In R. L. Brennan (Ed.),
*Educational measurement*(4th ed.). Westport, CT: Praeger Publishers.Google Scholar - Knapp, M. S. (1997). Between systemic reforms and the mathematics and science classroom: The dynamics of innovation, implementation, and professional learning.
*Review of Educational Research, 67*(2), 227–266.CrossRefGoogle Scholar - Lampert, M. (1990). When the problem is not the question and the solution is not the answer: Mathematical knowing and teaching.
*American Educational Research Journal, 27*(1), 29–63.CrossRefGoogle Scholar - Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data.
*Biometrics, 33*(1), 159–174.CrossRefGoogle Scholar - Learning Mathematics for Teaching. (2006).
*A coding rubric for measuring the quality of mathematics in instruction, Technical Report LMT*. Ann Arbor, MI: University of Michigan, School of Education.Google Scholar - Lehrer, R., & Schauble, L. (2002). Symbolic communication in mathematics and science: Co-constituting inscription and thought. In E. Amsel & J. P. Byrnes (Eds.),
*Language, literacy, and cognitive development: The development and consequences of symbolic communication*(pp. 167–192). Mahwah, NJ: Lawrence Erlbaum.Google Scholar - Lesh, R. A., Post, T. R., & Behr, M. J. (1987). Representations and translations among representations in mathematics learning and problem solving. In C. Janvier (Ed.),
*Problems of representation in the teaching and learning of mathematics*(pp. 33–40). Hillsdale, NJ: Erlbaum.Google Scholar - Malara, N., & Zan, R. (2002). The problematic relationship between theory and practice. In L. D. English (Ed.),
*Handbook of international research in mathematics education*(pp. 553–580). Mahwah, NJ: Lawrence Erlbaum.Google Scholar - Messick, S. (1993). Validity. In R. L. Linn (Ed.),
*Educational measurement*(3rd ed.). Phoenix, AZ: Oryx Press.Google Scholar - Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning.
*American Psychologist, 50*, 741–741.CrossRefGoogle Scholar - Moyer, P. S. (2001). Are we having fun yet? How teachers use manipulatives to teach mathematics.
*Educational Studies in Mathematics, 47*(2), 175–197.CrossRefGoogle Scholar - National Council of Teachers of Mathematics. (2000).
*Principles and standards for school mathematics*. Reston, VA: National Council of Teachers of Mathematics.Google Scholar - National Council of Teachers of Mathematics. (2007).
*Mathematics teaching today: Improving practice, improving student learning*. Reston, VA: National Council of Teachers of Mathematics.Google Scholar - North Central Regional Educational Laboratory. (2000).
*A study of the differences between higher and lower performing Indiana schools in reading and mathematics*. Naperville, IL: North Central Regional Educational Laboratory.Google Scholar - Nye, B., Konstantopoulos, S., & Hedges, L. V. (2004). How large are teacher effects?
*Educational Evaluation and Policy Analysis, 26*(3), 237.CrossRefGoogle Scholar - Pianta, R. C., Belsky, J., Vandergrift, N., Houts, R., & Morrison, F. J. (2008). Classroom effects on children’s achievement trajectories in elementary school.
*American Educational Research Journal, 45*(2), 365–397.CrossRefGoogle Scholar - Pianta, R. C., La Paro, K. M., & Hamre, B. K. (2008).
*Classroom assessment scoring system manual, K-3*. Baltimore, MD: Brookes Publishing Co.Google Scholar - Piburn, M., Sawada, D., Turley, J., Falconer, K., Benford, R., Bloom, I., et al. (2000).
*Reformed teaching observation protocol (RTOP): Reference manual. ACEPT Technical Report No. IN00-3)*. Tempe, AZ: Arizona Collaborative for Excellence in the Preparation of Teachers.Google Scholar - Porter, A. C., Blank, R. K., Smithson, J. L., & Osthoff, E. (2005). Place-based randomized trials to test the effects on instructional practices of a mathematics/science professional development program for teachers.
*The Annals of the American Academy of Political and Social Science, 599*(1), 147–175.CrossRefGoogle Scholar - Radford, L. (2002). The seen, the spoken, and the written: A semiotic approach to problem of objectification of mathematical knowledge.
*For the Learning of Mathematics, 22*(2), 14–23.Google Scholar - Raphael, D., & Wahlstrom, M. (1989). The influence of instructional aids on mathematics achievement.
*Journal for Research in Mathematics Education, 20*(2), 173–190.CrossRefGoogle Scholar - Reid, D. A. (2002). Conjectures and refutations in grade 5 mathematics.
*Journal for Research in Mathematics Education, 33*(1), 5–29.CrossRefGoogle Scholar - Rivkin, S. G., Hanushek, E. A., & Kain, J. F. (2005). Teachers, schools, and academic achievement.
*Econometrica, 73*(2), 417–458.CrossRefGoogle Scholar - Rutter, M., & Maughan, B. (2002). School effectiveness findings 1979–2002.
*Journal of School Psychology, 40*(6), 451–475.CrossRefGoogle Scholar - Sanders, W. L., & Rivers, J. C. (1996).
*Cumulative and residual effects of teachers on future student academic achievement*. Knoxville, TN: University of Tennessee Value-Added Research and Assessment Center.Google Scholar - Sarama, J., & Clements, D. H. (2004). Building blocks for early childhood mathematics.
*Early Childhood Research Quarterly, 19*(1), 181–189.CrossRefGoogle Scholar - Sherin, M. G. (2002). A balancing act: Developing a discourse community in a mathematics classroom.
*Journal of Mathematics Teacher Education, 5*(3), 205–233.CrossRefGoogle Scholar - Sowell, E. J. (1989). Effects of manipulative materials in mathematics instruction.
*Journal for Research in Mathematics Education, 20*(5), 498–505.CrossRefGoogle Scholar - Spillane, J. P., & Zeuli, J. S. (1999). Reform and teaching: Exploring patterns of practice in the context of national and state mathematics reforms.
*Educational Evaluation and Policy Analysis, 21*(1), 1–27.CrossRefGoogle Scholar - Stecher, B., Hamilton, L., Ryan, G., Le, V. N., Williams, V., Robyn, A., et al. (2002). Measuring reform-oriented instructional practices in mathematics and science.
*Annual Meeting of the American Educational Research Association,*New Orleans, LA.Google Scholar - Stein, M. K., Grover, B. W., & Henningsen, M. (1996). Building student capacity for mathematical thinking and reasoning: An analysis of mathematical tasks used in reform classrooms.
*American Educational Research Journal, 33*(2), 455–488.CrossRefGoogle Scholar - Stein, M. K., Smith, M. S., Henningsen, M. A., & Silver, E. A. (2009).
*Implementing standards-based mathematics instruction: A casebook for professional development*(2nd ed.). New York: Teachers College Press.Google Scholar - Stipek, D. J., Givvin, K. B., Salmon, J. M., & MacGyvers, V. L. (2001). Teachers’ beliefs and practices related to mathematics instruction.
*Teaching and Teacher Education, 17*(2), 213–226.CrossRefGoogle Scholar - Sutton, J., & Krueger, A. (Eds.). (2002).
*EDThoughts: What we know about mathematics teaching and learning*. Aurora, CO: Mid-continent Research for Education and Learning.Google Scholar - Toulmin, S. (1969).
*The uses of argument*. Cambridge, England: Cambridge University Press.Google Scholar - Weiss, I. R., Pasley, J. D., Smith, P. S., Banilower, E. R., & Heck, D. J. (2003).
*Looking inside the classroom: A study of K-12 mathematics and science education in the united states*. Chapel Hill, NC: Horizon Research, Inc.Google Scholar - Xie, M. (2009). Mathematics education reform in mainland China in the past sixty years: Reviews and forecasts.
*Journal of Mathematics Education, 2*(1), 121–130.Google Scholar - Yackel, E., & Cobb, P. (1996). Sociomathematical norms, argumentation, and autonomy in mathematics.
*Journal for Research in Mathematics Education, 27*(4), 458–477.CrossRefGoogle Scholar - Yackel, E., Cobb, P., & Wood, T. (1991). Small-group interactions as a source of learning opportunities in second-grade mathematics.
*Journal for Research in Mathematics Education, 22*(5), 390–408.CrossRefGoogle Scholar