Using the S-STEM Survey to Evaluate a Middle School Robotics Learning Environment: Validity Evidence in a Different Context

  • Wenjing Luo
  • Hsin-Ro Wei
  • Albert D. RitzhauptEmail author
  • A. Corinne Huggins-Manley
  • Christina Gardner-McCune


Numerous studies have been undertaken to design, develop, and provide validity evidence for using instruments to measure students’ attitudes toward STEM (Science, Technology, Engineering, and Mathematics). This study presents validity evidence of scores produced from the S-STEM measurement tool and used to evaluate changes in attitudes during an educational intervention in a middle school robotics learning environment. All data were collected from middle school students who were involved in a school district-wide effort for integrating educational robotics into the classroom. Findings from this study provided not only internal structure validity evidence, but also criterion-related validity evidence of the proposed S-STEM tool use. In addition, measurement invariance results revealed that items in the S-STEM had equivalence in statistical properties of measurement across groups (e.g., grade level). The study provides further evidence that S-STEM survey is a powerful and useful tool to evaluate student attitude changes during STEM educational programs; offers suggestions for its future implementation; and presents other inspiring ideas for future STEM instrument development.


STEM Instrument design Instrument validation S-STEM Surveys 


Compliance with Ethical Standards

All procedures performed in this study were in accordance with the ethical standards of the institutional review board at the authors’ respective institution. All participants completed an informed consent process prior to supplying data for this research. There are no conflicts of interest for any of the authors on this article.


  1. American Educational Research Association (AERA), American Psychological Association (APA), & National Council on Measurement in Education (NCME). (2014). Standards for educational and psychological testing. Washington: Author.Google Scholar
  2. Adams, W. K., Perkins, K. K., Podolefsky, N. S., Dubson, M., Finkelstein, N. D., & Wieman, C. E. (2006). New instrument for measuring student beliefs about physics and learning physics: The Colorado learning attitudes about science survey. Physical Review Special Topics - Physics Education Research, 2(1), 010101.Google Scholar
  3. Ajzen, I. (2001). Nature and operation of attitudes. Annual Review of Psychology, 52(1), 27–58.Google Scholar
  4. Allen, M. J., & Yen, W. M. (1979). Introduction to measurement theory. Long Grove, IL: Waveland Press.Google Scholar
  5. Aydeniz, M., & Kotowski, M. R. (2014). Conceptual and methodological issues in the measurement of attitudes towards science. Electronic Journal of Science Education, 18(3).Google Scholar
  6. Bandura, A. (1997). Self-efficacy: the exercise of control. New York: Freeman.Google Scholar
  7. Brennan, R. L. (2006). Educational measurement. Westport, CT: Praeger.Google Scholar
  8. Cabrera, F. R. (2017). Outcomes-based education (OBE): its effect to the mathematics performance, mathematics skills and attitudes towards mathematics of the BSCS students. Advanced Science Letters, 23(2), 1081–1084.Google Scholar
  9. Camilli, G. (2006). Test fairness. In R. L. Brennan’s (Ed.), Educational measurement (pp. 221–256). Westport: Praeger.Google Scholar
  10. Crano, W. D., & Prislin, R. (2006). Attitudes and persuasion. Annual Review of Psychology, 57(1), 345–374.Google Scholar
  11. Crippen, K. J., Brown, J. C., Apraiz, K., Busi, R., Evran, D., McLaughlin, C., Peace, M., & Temurtas, A. (2016). A process model of the US federal perspective on STEM. Journal of STEM Teacher Education, 50(1), 19–47.Google Scholar
  12. Crocker, L., & Algina, J. (1986). Introduction to classical and modern test theory. Belmont, CA: Wadsworth Group.Google Scholar
  13. Eccles, J. S., & Wigfield, A. (2002). Motivational beliefs, values, and goals. Annual Review of Psychology, 53(1), 109–132.Google Scholar
  14. Erkut, S., & Marx, F. (2005). 4 schools for WIE. Evaluation Report. Wellesley Centers for Women.Google Scholar
  15. Ericson, B., & McKlin, T. (2012). Effective and sustainable computing summer camps. In Proceedings of the 43rd ACM technical symposium on Computer Science Education (pp. 289-294). ACM.Google Scholar
  16. Fishbein, M., & Ajzen, I. (1977). Belief, attitude, intention, and behavior: an introduction to theory and research. Reading, Mass: Addison-Wesley.Google Scholar
  17. Friedman, A. (Ed). (2008). Framework for evaluating impacts of informal science education projects. Retreieved from:
  18. Gorsuch, R. L. (1983). Factor analysis (2nd ed.). Hillsdale: Lawrence Erlbaum.Google Scholar
  19. Guzey, S. S., Harwell, M., & Moore, T. (2014). Development of an instrument to assess attitudes toward science, technology, engineering, and mathematics (STEM). School Science and Mathematics, 114(6), 271–279.Google Scholar
  20. Hall, G. E. (1974). The concerns-based adoption model: A developmental conceptualization of the adoption process within educational institutions. Austin: Research and Development Center for Teacher Education, University of Texas.Google Scholar
  21. Han, S., & Carpenter, D. (2014). Construct validation of student attitude toward science, technology, engineering, and mathematics project-based learning: the case of Korean middle grade students. Middle Grades Research Journal, 9(3), 27–41.Google Scholar
  22. Huggins, A. C., Ritzhaupt, A. D., & Dawson, K. (2014). Measuring information and communication technology literacy using a performance assessment: validation of the student tool for technology literacy (ST2L). Computers in Education, 77, 1–12.Google Scholar
  23. Hohlfeld, T. N., Ritzhaupt, A. D., & Barron, A. E. (2010). Development and validation of the student tool for technology literacy (ST2L). Journal of Research on Technology in Education, 42(4), 361–389.Google Scholar
  24. Holland, P. W., & Thayer, D. T. (1988). Differential item performance and the Mantel-Haenszel procedure. In H. Wainer & H. I. Braun (Eds.), Test validity (pp. 129–145). Hillsdale, NJ: Erlbaum.Google Scholar
  25. Hu, L. T., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: conventional criteria versus new alternatives. Structural Equation Modeling, 6(1), 1–55.Google Scholar
  26. Horn, J. L., McArdle, J. J., & Mason, R. (1983). When is invariance not invariant: a practical scientist’s look at the ethereal concept of factor invariance. The Southern Psychologist, 1, 179–188.Google Scholar
  27. Kier, M. W., Blanchard, M. R., Osborne, J. W., & Albert, J. L. (2014). The development of the STEM career interest survey (STEM-CIS). Research in Science Education, 44(3), 461–481.Google Scholar
  28. Koballa, T. R., Jr., & Glynn, S. M. (2013). Attitudinal and motivational constructs in science learning. In S. K. Abell & N. Lederman (Eds.), Handbook of Research on Science Education (pp. 89–116). Routledge.Google Scholar
  29. Krathwohl, D. R., Bloom, B. S., & Masia, B. B. (1964). Handbook II: affective domain. New York: David McKay.Google Scholar
  30. Kristen, E., Jennifer, C., & Holly, Y. A. (2012). Designing evaluations for k-12 robotics education programs. In Robots in k-12 education: a new technology for learning (pp. 31–53). Hershey, PA: Information Science Reference.Google Scholar
  31. Lent, R. W., Brown, S. D., & Hackett, G. (2000). Contextual supports and barriers to career choice: a social cognitive analysis. Journal of Counseling Psychology, 47(1), 36–49.Google Scholar
  32. Lent, R. W., & Brown, S. D. (2006). On conceptualizing and assessing social cognitive constructs in career research: a measurement guide. Journal of Career Assessment, 14(1), 12–35.Google Scholar
  33. Lent, R. W., Sheu, H. B., Singley, D., Schmidt, J. A., Schmidt, L. C., & Gloster, C. S. (2008). Longitudinal relations of self-efficacy to outcome expectations, interests, and major choice goals in engineering students. Journal of Vocational Behavior, 73(2), 328–335.Google Scholar
  34. Mahoney, M. P. (2010). Students’ attitudes toward STEM: development of an instrument for high school STEM-based programs. Journal of Technology Studies, 36(1) Retrieved from:
  35. Maltese, A. V., Melki, C. S., & Wiebke, H. L. (2014). The nature of experiences responsible for the generation and maintenance of interest in STEM. Science Education, 98(6), 937–962.Google Scholar
  36. Maltese, A. V., & Tai, R. H. (2011). Pipeline persistence: examining the association of educational experiences with earned degrees in STEM among US students. Science Education, 95(5), 877–907.Google Scholar
  37. Mantel, N., & Haenszel, W. (1959). Statistical aspects of the analysis of data from retrospective studies of disease. Journal of the National Cancer Institute, 22(4), 719–748.Google Scholar
  38. Meredith, W. (1993). Measurement invariance, factor analysis and factorial invariance. Psychometrika, 58(4), 525–543.Google Scholar
  39. Meredith, W., & Teresi, J. A. (2006). An essay on measurement and factorial invariance. Medical Care, 44, 69–77.Google Scholar
  40. Muthén, L. K., & Muthén, B. O. (2012). Mplus User’s Guide (7th ed.). Los Angeles: Muthén & Muthén.Google Scholar
  41. Myers, D. G. (1993). Social psychology (4th ed.). New York: McGraw-Hill.Google Scholar
  42. Nam, Y., Lee, S. J., & Paik, S. H. (2016). The impact of engineering integrated science (EIS) curricula on first-year technical high school Students’ attitudes toward science and perceptions of engineering. Eurasia Journal of Mathematics, Science & Technology Education, 12(7).Google Scholar
  43. National Commission on Excellence in Education (NCEE). (1983). A nation at risk: the imperative for educational reform. Washington: US Department of Education.Google Scholar
  44. Nunnaly, J. (1978). Psychometric theory. New York: McGraw-Hill.Google Scholar
  45. Pajares, F., & Miller, M. D. (1994). Role of self-efficacy and self-concept beliefs in mathematical problem solving: a path analysis. Journal of Education & Psychology, 86(2), 193–203.Google Scholar
  46. Penfield, R. D. (2005). DIFAS: differential item functioning analysis system. Applied Psychological Measurement, 29(2), 150–151.Google Scholar
  47. Raykov, T., & Marcoulides, G. A. (2011). Introduction to psychometric theory. New York: Taylor & Francis.Google Scholar
  48. Schwarz, N. (2007). Attitude construction: evaluation in context. Social Cognition, 25(5), 638–656.Google Scholar
  49. Stajkovic, A. D., & Luthans, F. (1998). Self-efficacy and work-related performance: a meta-analysis. Psychological Bulletin, 124(2), 240–261.Google Scholar
  50. Suprapto, N. (2016). Students’ attitudes towards STEM education: voices from Indonesian junior high schools. Journal of Turkish Science Education, 13(3), 75–87.Google Scholar
  51. Thorndike-Christ, T. (1991). Attitudes toward mathematics: relationships to mathematics achievement, gender, mathematics course-taking plans, and career interests.Google Scholar
  52. Tseng, K. H., Chang, C. C., Lou, S. J., & Chen, W. P. (2013). Attitudes towards science, technology, engineering and mathematics (STEM) in a project-based learning (PjBL) environment. International Journal of Technology and Design Education, 23(1), 87–102.Google Scholar
  53. Tyler-Wood, T., Knezek, G., & Christensen, R. (2010). Instruments for assessing interest in STEM content and careers. Journal of Technology and Teacher Education, 18(2), 345–368.Google Scholar
  54. Unfried, A., Faber, M., Stanhope, D. S., & Wiebe, E. (2015). The development and validation of a measure of student attitudes toward science, technology, engineering, and math (S-STEM). Journal of Psychoeducational Assessment, 33(7), 622–639.Google Scholar
  55. U.S. Department of Education. (2015). Retrieved from
  56. Wigfield, A., & Eccles, J. S. (2000). Expectancy–value theory of achievement motivation. Contemporary Educational Psychology, 25(1), 68–81.Google Scholar

Copyright information

© Springer Nature B.V. 2019

Authors and Affiliations

  • Wenjing Luo
    • 1
  • Hsin-Ro Wei
    • 1
  • Albert D. Ritzhaupt
    • 1
    Email author
  • A. Corinne Huggins-Manley
    • 1
  • Christina Gardner-McCune
    • 1
  1. 1.School of Teaching and Learning, College of EducationUniversity of FloridaGainesvilleUSA

Personalised recommendations