Skip to main content

Advertisement

SpringerLink
Log in

Missing in Measurement: Why Identifying Learning in Integrated Domains Is So Hard

  • Published:
Journal of Science Education and Technology Aims and scope Submit manuscript

Abstract

Integrating computational thinking (CT) and science education is complex, and assessing the resulting learning gains even more so. Arguments that assessment should match the learning (Biggs, Assessment & Evaluation in Higher Education, 21(1), 5–16. 1996; Airasian and Miranda, Theory into Practice, 41(4), 249–254. 2002; Hickey and Zuiker, Journal of the Learning Sciences, 21(4), 522–582. 2012; Pellegrino, Journal of Research in Science Teaching, 49(6), 831–841. 2012; Wiggins, Practical Assessment, Research and Evaluation, 2(2). 1990) lead to a performance-oriented approach to assessment, using tasks that mirror the integrated instruction. This approach reaps benefits but also poses challenges. Integrated CT is a new approach to learning. Movement is being made toward understanding what it means to operate successfully in this context, but consensus is neither general nor time tested (Kaput and Schorr, Research on technology and the teaching and learning of mathematics: Case and perspectives (Vol. 2, pp. 211–253) 2008). Movement is also being made toward developing methods for assessing CT. Despite the benefits of matching assessment with pedagogy, there may be intrinsic losses. One problem is that interactions between the two domains may invalidate the results, either because the gains in one may be easier to measure at certain times than the gains in the other, or because interactions between the two domains may cause measurement interference. Our examination draws upon both theoretical basis and also existing practices, particularly from our own work integrating CT and secondary science. We present a mixed-methods analysis of student assessment results and consider potential issues with moving too quickly toward relying on a rubric-based approach to evaluating this student learning. Centrally, we emphasize the importance of assessment approaches that reflect one of the most important affordances of computational environments, that is, the expression of multiple ways of knowing and doing (Turkle and Papert, Journal of Mathematical Behavior, 11(1), 3–33. 1992).

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  • Abedi, J. (2002). Standardized achievement tests and English language learners: Psychometrics issues. Educ Assess, 8(3), 231–257. Retrieved from. https://doi.org/10.1207/S15326977EA0803_02.

    Article  Google Scholar 

  • AERA, A., & NCME. (1999). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.

    Google Scholar 

  • Airasian, P. W., & Miranda, H. (2002). The role of assessment in the revised taxonomy. Theory Pract, 41(4), 249–254. Retrieved from. https://doi.org/10.1207/s15430421tip4104_8.

    Article  Google Scholar 

  • Barab, S., & Squire, K. (2004). Design-based research: Putting a stake in the ground. J Learn Sci, 13(1), 1–14. https://doi.org/10.1207/s15327809jls1301_1.

    Article  Google Scholar 

  • Basu, S., Biswas, G., & Kinnebrew, J. S. (2017). Learner modeling for adaptive scaffolding in a computational thinking-based science learning environment. User Model User-Adap Inter, 27(1), 5–53.

    Google Scholar 

  • Basu, S., McElhaney, K. W., Grover, S., Harris, C., & Biswas, G. (2018). A principled approach to designing assessments that integrate science and computational thinking. In ICLS Proceedings. London: ISLS.

    Google Scholar 

  • Bell, P., Hoadley, C. M., & Linn, M. C. (2004). Design-based research in education. Internet Environments for Science Education, 73–85.

  • Bell, T., Witten, I. H., Fellows, M., Adams, R., McKenzie, J., Powell, M., & Jarman, S. (2015). CS unplugged: Computer science without a computer. Retrieved from https://csunplugged.org/en/

  • Biggs, J. (1996). Assessing learning quality: Reconciling institutional, staff and educational demands. Assess Eval High Educ, 21(1), 5–16. Retrieved from. https://doi.org/10.1080/0260293960210101.

    Article  Google Scholar 

  • Blikstein, P., & Wilensky, U. (2009). An atom is known by the company it keeps: A constructionist learning environment for materials science using agent-based modeling. Int J Comput Math Learn, 14(2), 81–119. Retrieved from. https://doi.org/10.1007/s10758-009-9148-8.

    Article  Google Scholar 

  • Box, G. E. (1976). Science and statistics. J Am Stat Assoc, 71(356), 791–799.

    Google Scholar 

  • Box, G. E. (1979). All models are wrong, but some are useful. Robustness in Statistics, 202.

  • Brennan, K., & Resnick, M. (2012). New frameworks for studying and assessing the development of computational thinking. In In Proceedings of the 2012 annual meeting of the American Educational Research Association. Vancouver: Canada Retrieved from http://scratched.gse.harvard.edu/ct/files/AERA2012.pdf.

    Google Scholar 

  • Buffum, P. S., Lobene, E. V., Frankosky, M. H., Boyer, K. E., Wiebe, E. N., & Lester, J. C. (2015). A practical guide to developing and validating computer science knowledge assessments with application to middle school. In Proceedings of the 46th ACM technical symposium on computer science education (pp. 622–627). New York, NY: ACM. Retrieved from. https://doi.org/10.1145/2676723.2677295.

    Chapter  Google Scholar 

  • Capraro, R. M., & Corlu, M. S. (2013). Changing views on assessment for STEM project-based learning. In STEM project-based learning (pp. 109–118). Rotterdam, NL: SensePublishers.

    Google Scholar 

  • Dickes, A., & Sengupta, P. (2013). Learning natural selection in 4th grade with multi-agent-based computational models. Res Sci Educ, 43(3), 921–953. Retrieved from. https://doi.org/10.1007/s11165-012-9293-2.

    Article  Google Scholar 

  • diSessa, A. (2000). Changing Minds. Cambridge, MA: The MIT Press.

    Google Scholar 

  • Gane, B. D., Zaidi, S. Z., & Pellegrino, J. W. (2018). Measuring what matters: Using technology to assess multidimensional learning. Eur J Educ, 53, 176–187.

    Google Scholar 

  • Gautam, A., Wall Bortz, W., & Tatar, D. (2017). Case for integrating computational thinking and science in a low-resource setting. In Proceedings of the Ninth International Conference on Information and Communication Technologies and Development. Pakistan: Lahore.

    Google Scholar 

  • Giere, R. N. (2004). How Models Are Used to Represent Reality. Philosophy of Science, 71(5), 742–752. https://doi.org/10.1086/425063.

  • Goldstone, R. L., & Wilensky, U. (2008). Promoting transfer by grounding complex systems principles. J Learn Sci, 17(4), 465–516. Retrieved from. https://doi.org/10.1080/10508400802394898.

    Article  Google Scholar 

  • Grover, S. (2015). Systems of assessments for deeper learning of computational thinking in K - 12, 10. Chicago: Presented at the American Educational Research Association.

    Google Scholar 

  • Grover, S., & Pea, R. (2013). Computational Thinking in K–12: A Review of the State of the Field. Educational Researcher, 42(1), 38–43. https://doi.org/10.3102/0013189X12463051.

  • Grover, S., & Pea, R. (2018). Computational thinking: A competency whose time has come. In S. Sentance, S. Carsten, & E. Barendsen (Eds.), Computer science education: Perspectives on teaching and learning in school (pp. 19–38). London, UK: Bloomsbury.

    Google Scholar 

  • Herrmann-Abell, C. F., & DeBoer, G. E. (2011). Using distractor-driven standards-based multiple-choice assessments and Rasch modeling to investigate hierarchies of chemistry misconceptions and detect structural problems with individual items. Chemistry Education Research and Practice, 12(2), 184–192.

    Google Scholar 

  • Herrmann-Abell, C. F., Koppal, M., & Roseman, J. E. (2016). Toward high school biology: Helping middle school students understand chemical reactions and conservation of mass in nonliving and living systems. CBE-Life Sciences Education, 15(4), 1–21. Retrieved from. https://doi.org/10.1187/cbe.16-03-0112.

    Article  Google Scholar 

  • Hickey, D. T., & Zuiker, S. J. (2012). Multilevel aAssessment for dDiscourse, uUnderstanding, and aAchievement. J Learn SciJournal of the Learning Sciences, 21(4), 522–582. https://doi.org/10.1080/10508406.2011.652320.

    Article  Google Scholar 

  • Hoadley, C. M. (2003). Design-based research: An emerging paradigm for educational inquiry. Educ Res, 32(1), 5–8.

    Google Scholar 

  • Israel, M., Pearson, J. N., Tapia, T., Wherfel, Q. M., & Reese, G. (2015). Supporting all learners in school-wide computational thinking: A cross-case qualitative analysis. Comput Educ, 82, 263–279. Retrieved from. https://doi.org/10.1016/j.compedu.2014.11.022.

    Article  Google Scholar 

  • Jacobson, M. J., & Wilensky, U. (2006). Complex systems in education: Scientific and educational importance and implications for the learning sciences. J Learn Sci, 15(1), 11–34. Retrieved from. https://doi.org/10.1207/s15327809jls1501_4.

    Article  Google Scholar 

  • Järvelä, S. (1995). The cognitive apprenticeship model in a technologically rich learning environment: Interpreting the learning interaction. Learn Instr, 5(3), 237–259. Retrieved from. https://doi.org/10.1016/0959-4752(95)00007-P.

    Article  Google Scholar 

  • Kaput, J., & Schorr, R. (2008). Changing representational infrastructures changes most everything: The case of SimCalc, algebra, and calculus. In K. Heid & G. W. Blume (Eds.), Research on technology and the teaching and learning of mathematics: Case and perspectives (Vol. 2, pp. 211–253). Charlotte, NC: Information Age Publishing.

    Google Scholar 

  • Koh, K. H., Basawapatna, A., Bennett, V., & Repenning, A. (2010). Towards the automatic recognition of computational thinking for adaptive visual language learning. In 2010 IEEE Symposium on Visual Languages and Human-Centric Computing (pp. 59–66). Retrieved from). https://doi.org/10.1109/VLHCC.2010.17.

  • Koh, K. H., Basawapatna, A., Nickerson, H., & Repenning, A. (2014). Real time assessment of computational thinking. In 2014 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) (pp. 49–52). Retrieved from). https://doi.org/10.1109/VLHCC.2014.6883021.

    Chapter  Google Scholar 

  • Krajcik, J., Blumenfeld, P. C., Marx, R. W., Bass, K. M., Fredricks, J., & Soloway, E. (1998). Inquiry in project-based science classrooms: Initial attempts by middle school students. J Learn Sci, 7(3–4), 313–350.

    Google Scholar 

  • Krathwohl, D. R. (2002). A revision of Bloom’s taxonomy: An overview. Theory Pract, 41(4), 212–218.

    Google Scholar 

  • Lead States, N. G. S. S. (2003). Next generation science standards: For states, by states. Washington, DC: The National Academies Press.

    Google Scholar 

  • Lee, O., Eichinger, D. C., Anderson, C. W., Berkheimer, G. D., & Blakeslee, T. D. (1993). Changing middle school students’ conceptions of matter and molecules. J Res Sci Teach, 30(3), 249–270. Retrieved from. https://doi.org/10.1002/tea.3660300304.

    Article  Google Scholar 

  • Lee, I., Martin, F., Denner, J., Coulter, B., Allan, W., Erickson, J., et al. (2011). Computational thinking for youth in practice. ACM Inroads, 2(1), 32–37. Retrieved from. https://doi.org/10.1145/1929887.1929902.

    Article  Google Scholar 

  • Lye, S. Y., & Koh, J. H. L. (2014). Review on teaching and learning of computational thinking through programming: What is next for K - 12? Comput Hum Behav, 41, 51–61. Retrieved from. https://doi.org/10.1016/j.chb.2014.09.012.

    Article  Google Scholar 

  • MacQueen, K. M., McLellan, E., Kay, K., & Milstein, B. (1998). Codebook development for team-based qualitative analysis, codebook development for team-based qualitative analysis. CAM Journal, 10(2), 31–36. Retrieved from. https://doi.org/10.1177/1525822X980100020301.

    Article  Google Scholar 

  • Marzano, R. J., Pickering, D., & McTighe, J. (1993). Assessing student outcomes: Performance assessment using the dimensions of learning model. Alexandria, VA: Association for Supervision and Curriculum Development Retrieved from http://eric.ed.gov/?id=ED461665.

    Google Scholar 

  • Meerbaum-Salant, O., Armoni, M., & Ben-Ari, M. (2013). Learning computer science concepts with scratch. Comput Sci Educ, 23(3), 239–264.

    Google Scholar 

  • Miller, D. M., & Linn, R. L. (2000). Validation of performance-based assessments. Appl Psychol Meas, 24(4), 367–378. Retrieved from. https://doi.org/10.1177/01466210022031813.

    Article  Google Scholar 

  • Mislevy, R. J., & Haertel, G. D. (2006). Implications of evidence-centered design for educational testing. Educ Meas Issues Pract, 25(4), 6–20. Retrieved from. https://doi.org/10.1111/j.1745-3992.2006.00075.x.

    Article  Google Scholar 

  • Moreno-León, J., & Robles, G. (2015). Dr. scratch: A web tool to automatically evaluate scratch projects. In Proceedings of the workshop in primary and secondary computing education (pp. 132–133). New York, NY: ACM.

    Google Scholar 

  • Moskal, B. (2000). Recommendations for developing classroom performance assessments and scoring rubrics. Pract Assess Res Eval, 8(14), 1–8.

    Google Scholar 

  • National Research Council. (2011). Report of a workshop on the pedagogical aspects of computational thinking. Washington, DC: National Academies Press.

    Google Scholar 

  • National Research Council. (2012). A framework for K-12 science education: Practices, crosscutting concepts, and core ideas. Washington, DC: National Academies Press.

    Google Scholar 

  • Papert, S. (1980). Mindstorms: Children, computers, and powerful ideas. New York, NY: Basic Books, Inc..

    Google Scholar 

  • Pellegrino, J. W. (2012). Assessment of science learning: Living in interesting times. J Res Sci Teach, 49(6), 831–841. Retrieved from. https://doi.org/10.1002/tea.21032.

    Article  Google Scholar 

  • Pellegrino, J. W. (2013). Proficiency in science: Assessment challenges and opportunities. Science, 340(6130), 320–323. Retrieved from. https://doi.org/10.1126/science.1232065.

    Article  Google Scholar 

  • Pellegrino, J. W., & Hilton, M. L. (2013). Committee on defining deeper learning and 21st century skills, Center for Education, Division on Behavioral and Social Sciences and Education, and National Research Council. In Education for life and work: Developing transferable knowledge and skills in the 21stcentury. Washington DC: National Academies Press.

    Google Scholar 

  • Perković, L., Settle, A., Hwang, S., & Jones, J. (2010). A framework for computational thinking across the curriculum. In Proceedings of the fifteenth annual conference on innovation and technology in computer science education (pp. 123–127). New York, NY: ACM.

    Google Scholar 

  • Petkov, D., & Petkova, O. (2006). Development of scoring rubrics for IS projects as an assessment tool. Issues in Informing Science and Information Technology, 3, 499–510.

    Google Scholar 

  • Qualls, J. A., & Sherrell, L. B. (2010). Why computational thinking should be integrated into the curriculum. Journal of Computer Sciences in Colleges, 25(5), 66–71.

    Google Scholar 

  • Reed, D. A., Bajcsy, R., Fernandez, M. A., Griffiths, J.-M., Mott, R. D., Dongarra, J., et al. (2005). Computational science: Ensuring America’s competitiveness. Arlington, VA: President’s Information Technology Advisory Committee Retrieved from http://www.dtic.mil/docs/citations/ADA462840.

    Google Scholar 

  • Román-González, M. (2015). Computational thinking. Test: Design Guidelines and Content Validation. Retrieved from. https://doi.org/10.13140/RG.2.1.4203.4329.

    Book  Google Scholar 

  • Sampson, V., & Grooms, J. (2008). Science as argument-driven inquiry: The impact on students’ conceptions of the nature of scientific inquiry. In Annual International Conference of the National Association of Research in Science Teaching. MD: Baltimore.

    Google Scholar 

  • Schwarz, C. V., Reiser, B. J., Davis, E. A., Kenyon, L., Achér, A., Fortus, D., et al. (2009). Developing a learning progression for scientific modeling: Making scientific modeling accessible and meaningful for learners. J Res Sci Teach, 46(6), 632–654.

    Google Scholar 

  • Sendur, G., Toprak, M., & Pekmez, E. S. (2010). Analyzing of students’ misconceptions about chemical equilibrium. International Conference on New Trends in Education and Implications. Turkey: Antalya.

    Google Scholar 

  • Sengupta, P., Kinnebrew, J. S., Basu, S., Biswas, G., & Clark, D. (2013). Integrating computational thinking with K - 12 science education using agent-based computation: A theoretical framework. Educ Inf Technol, 18(2), 351–380. Retrieved from. https://doi.org/10.1007/s10639-012-9240-x.

    Article  Google Scholar 

  • Settle, A., & Perkovic, L. (2010). Computational thinking across the curriculum: A conceptual framework. In Technical reports Retrieved from http://via.library.depaul.edu/tr/13.

    Google Scholar 

  • Settle, A., Franke, B., Hansen, R., Spaltro, F., Jurisson, C., Rennert-May, C., & Wildeman, B. (2012). Infusing computational thinking into the middle- and high-school curriculum. In Proceedings of the 17th ACM annual conference on innovation and Technology in Computer Science Education (pp. 22–27). New York, NY, USA: ACM. Retrieved from. https://doi.org/10.1145/2325296.2325306.

    Chapter  Google Scholar 

  • Sherman, M., & Martin, F. (2015). The assessment of mobile computational thinking. Journal of Computing Sciences in Colleges, 30(6), 53–59.

    Google Scholar 

  • Simon, H. A. (1955). A behavioral model of rational choice. Q J Econ, 69, 99–118.

    Google Scholar 

  • Stavy, R. (1991). Children’s ideas about matter. Sch Sci Math, 91(6), 240–244.

    Google Scholar 

  • Texas Education Agency. (2013). Texas essential knowledge and skills for science. Austin, TX: Texas Education Agency Retrieved from http://ritter.tea.state.tx.us/rules/tac/chapter112/ch112b.html.

    Google Scholar 

  • Thomas, D. R. (2006). A general inductive approach for analyzing qualitative evaluation data, a general inductive approach for analyzing qualitative evaluation data. Am J Eval, 27(2), 237–246. Retrieved from https://doi.org/10.1177/1098214005283748

  • Turkle, S., & Papert, S. (1992). Epistemological pluralism and the revaluation of the concrete. Journal of Mathematical Behavior, 11(1), 3–33.

    Google Scholar 

  • Voogt, J., Fisser, P., Good, J., Mishra, P., & Yadav, A. (2015). Computational thinking in compulsory education: Towards an agenda for research and practice. Educ Inf Technol, 20(4), 715–728.

    Google Scholar 

  • Wall Bortz, W., Gautam, A., Tatar, D., Rivale, S., & Lipscomb, K. (2019). The availability of pedagogical responses and the integration of computational thinking. In M. Reardon & J. Leonard (Eds.), Integrating digital technology in education: School-university-community collaboration. Charlotte, NC: Information Age Publishing.

    Google Scholar 

  • Weintrop, D., Beheshti, E., Horn, M. S., Orton, K., Trouille, L., Jona, K., & Wilensky, U. (2014). Interactive assessment tools for computational thinking in high school STEM classrooms. In Intelligent Technologies for Interactive Entertainment (pp. 22–25). Chicago: IL.

    Google Scholar 

  • Weintrop, D., Beheshti, E., Horn, M., Orton, K., Jona, K., Trouille, L., & Wilensky, U. (2015). Defining computational thinking for mathematics and science classrooms. J Sci Educ Technol, 25(1), 127–147. Retrieved from. https://doi.org/10.1007/s10956-015-9581-5.

    Article  Google Scholar 

  • Werner, L., Denner, J., Campe, S., & Kawamoto, D. C. (2012). The fairy performance assessment: Measuring computational thinking in middle school. In Proceedings of the 43rd ACM technical symposium on computer science education (pp. 215–220). New York, NY: ACM.

    Google Scholar 

  • Wiggins, G. (1990). The case for authentic assessment. Pract Assess Res Eval, 2(2).

  • Wilensky, U. (1999). NetLogo. Evanston, IL: Northwestern University, Center for Connected Learning and Computer-Based Modeling Retrieved from http://ccl.northwestern.edu/netlogo/.

    Google Scholar 

  • Wilensky, U., & Reisman, K. (2006). Thinking like a wolf, a sheep, or a firefly: Learning biology through constructing and testing computational theories-an embodied modeling approach. Cogn Instr, 24(2), 171–209.

    Google Scholar 

  • Wilensky, U., & Stroup, W. (1999). Learning through participatory simulations: Network-based design for systems learning in classrooms. In Proceedings of the 1999 Conference on computer support for collaborative learning. Palo Alto, California: International Society of the Learning Sciences.

    Google Scholar 

  • Wing, J. M. (2006). Computational thinking. Commun ACM, 49(3), 33–35.

    Google Scholar 

  • Yadav, A., Mayfield, C., Zhou, N., Hambrusch, S., & Korb, J. T. (2014). Computational thinking in elementary and secondary teacher education. ACM Trans Comput Educ, 14(1), 5. Retrieved from. https://doi.org/10.1145/2576872.

    Article  Google Scholar 

  • Yaşar, O. (2018). A new perspective on computational thinking. Commun ACM, 61(7), 33–39.

    Google Scholar 

Download references

Funding

This study was funded by the National Science Foundation (grant number 1543022).

Author information

Authors and Affiliations

  1. Department of Computer Science, Virginia Polytechnic Institute and State University, 2202 Kraft Dr SW, Blacksburg, VA, 24060, USA

    Whitney Wall Bortz, Aakash Gautam & Deborah Tatar

  2. Department of STEM Education, University of Texas at Austin, Austin, USA

    Kemper Lipscomb

Authors
  1. Whitney Wall Bortz
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Aakash Gautam
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Deborah Tatar
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Kemper Lipscomb
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Whitney Wall Bortz.

Ethics declarations

Conflict of Interest

The authors declare that they have no conflicts of interest.

Informed Consent

Informed consent was obtained from all individual participants included in the study.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bortz, W.W., Gautam, A., Tatar, D. et al. Missing in Measurement: Why Identifying Learning in Integrated Domains Is So Hard. J Sci Educ Technol 29, 121–136 (2020). https://doi.org/10.1007/s10956-019-09805-8

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10956-019-09805-8

Keywords

Search

Navigation