Abstract
There is a clear need for valid and reliable instrumentation that measures teacher knowledge. However, the process of investigating and making a case for instrument validity is not a simple undertaking; rather, it is a complex endeavor. This paper presents the empirical case of one aspect of such an instrument validation effort. The particular instrument under scrutiny was developed in order to determine the effect of a teacher education program on novice science and mathematics teachers’ strategic knowledge (SK). The relationship between novice science and mathematics teachers’ SK as measured by a survey and their SK as inferred from observations of practice using a widely used observation protocol is the subject of this paper. Moderate correlations between parts of the observation-based construct and the SK construct were observed. However, the main finding of this work is that the context in which the measurement is made (in situ observations vs. ex situ survey) is an essential factor in establishing the validity of the measurement itself.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
The total possible score on the RTOP is 100 as the lowest category for rating on each item is zero.
RTOP scores were averaged across all observations (either 1, 2, or 3, depending on the individual) because teachers were purposefully observed more than once in order to account for the possible effects of observing an atypical lesson.
References
Advancement Via Individual Determination. (2015). About AVID. http://www.avid.org/about.ashx. Accessed 14 Jan 2015.
American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.
Berliner, D. C. (2001). Learning about and learning from expert teachers. International Journal of Educational Research, 35, 463–482.
Bond, L., Smith, T., Baker, W. K., & Hattie, J. A. (2000). A distinction that matters—why national teacher certification makes a difference. Greensboro, NC: National Board for Professional Teaching Standards.
Briggs, D., Geil, K., Harlow, D., & Talbot, R. M. (2007). Measuring the pedagogical sophistication of Math and Science Teachers using Scenario-based Items. Paper presented at the American Educational Research Association Annual Meeting.
Hammerness, K., Darling-Hammond, L., Bransford, J., Berliner, D. C., Cochran-Smith, M., McDonald, M., & Zeichner, K. M. (2005). How teachers learn and develop. In L. Darling-Hammond & J. Bransford (Eds.), Preparing teachers for a changing world: what teachers should learn and be able to do (pp. 358–389). San Francisco: Jossey-Bass.
Lund, T. J., Pilarz, M., Velasco, J. B., Chakraverty, D., Rosploch, K., Undersander, M., & Stains, M. (2015). The best of both worlds: building on the COPUS and RTOP observation protocols to easily and reliably measure various levels of reformed instructional practice. CBE Life Sciences Education, 14(2), ar18.
Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational and psychological measurement (3rd ed., pp. 13–103). New York: Macmillan.
Messick, S. (1995). Validity of psychological assessment: validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. American Psychologist, 50, 741–749.
National Council of Teachers of Mathematics. (2000). Principles and standards for school mathematics. Reston, VA: National Council of Teachers of Mathematics.
National Research Council. (1996). National Science Education Standards: observe, interact, change, learn. Washington, DC: National Academies Press.
Otero, V., Finkelstein, N., McCray, R., & Pollock, S. J. (2006). Who is responsible for preparing science teachers? Science, 313(5786), 445–446.
Piburn, M. D., Sawada, D., Turley, J., Falconer, K., Benford, R., Bloom, I., & Judson, E. (2000). Reformed teaching observation protocol (RTOP) reference manual. Tempe, AZ: Arizona Collaborative for Excellence in the Preparation of Teachers, Arizona State University.
Rutherford, F. J., & Ahlgren, A. (1991). Science for all Americans. Oxford: Oxford University Press.
Sawada, D., Piburn, M. D., Judson, E., Turley, J., Falconer, K., Benford, R., & Bloom, I. (2002). Measuring reform practices in science and mathematics classrooms: the reformed teaching observation protocol. School Science and Mathematics, 102(6), 245–253.
Shulman, L. S. (1986). Those who understand: knowledge growth in teaching. Educational Researcher, 15(2), 4–14.
Talbot, R. M. (2011). Embedding content into an instrument designed to measure Novice Science and Mathematics Teachers’ Strategic Knowledge: A challenge for validity. Phalaborwa, South Africa: International Conference on Mathematics, Science, and Technology Education.
Talbot, R. M., Hartley, L., Marzetta, K., & Wee, B. (2015). Transforming undergraduate science education with learning assistants: student satisfaction in large enrollment courses. Journal of College Science Teaching, 44(5), 24–30.
U.S. Department of Education, Office of Postsecondary Education, Office of Policy Planning and Innovation (2002). Meeting the highly qualified teachers challenge: the secretary’s annual report on teacher quality. Washington, DC
Van Driel, J. H., Verloop, N., & de Vos, W. (1998). Developing science teachers’ pedagogical content knowledge. Journal of Research in Science Teaching, 35(6), 673–695.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Talbot, R.M. Scrutinizing a Survey-Based Measure of Science and Mathematics Teacher Knowledge: Relationship to Observations of Teaching Practice. Res Sci Educ 47, 1255–1274 (2017). https://doi.org/10.1007/s11165-016-9544-8
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11165-016-9544-8