Skip to main content
SpringerLink
Log in

A Scoping Review of Empirical Research on Recent Computational Thinking Assessments

  • Published:
Journal of Science Education and Technology Aims and scope Submit manuscript

Abstract

Computational thinking (CT) is regarded as an essential twenty-first century competency and it is already embedded in K-12 curricula across the globe. However, research on assessing CT has lagged, with few assessments being implemented and validated. Moreover, there is a lack of systematic grouping of CT assessments. This scoping review examines 39 empirical studies published within the last five years, coded by the specific competencies outlined in existing CT frameworks, to identify and classify the key features of existing CT assessments. Results show that most studies target K-12 settings, focus on interventions that promote CT concepts and practices, adopt a quasi-experimental design, use selected-response items as the dominant testing form, and mainly assess algorithmic thinking, abstraction, problem decomposition, logical thinking, and data. Finally, few CT assessments have been validated in educational settings. Implications include identifying gaps in the CT assessment literature, deepening our understanding of the nature of CT, focusing on the validation of CT assessments, and guiding researchers and practitioners in choosing developmentally appropriate CT assessments. Cognitive and educational implications for future research inquiry include the development of new assessment tools that comprehensively assess CT and its relation to learning.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  • Adams, C., Cutumisu, M., & Lu, C. (2019). Measuring K-12 computational thinking concepts, practices and perspectives: An examination of current CT assessments. In Proceedings of the Society for Information Technology & Teacher Education (SITE) (pp. 18–22). Las Vegas, NV: March.

  • American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). AERA, APA, & NCME. Standards for educational and psychological testing. Washington, DC: American Educational Research Association.

    Google Scholar 

  • Anderson, L.W., Krathwohl, D.R. (Eds.), Airasian, P.W., Cruikshank, K.A., Mayer, R.E., Pintrich, P.R., Raths, J., & Wittrock, M.C. (2001). A taxonomy for learning, teaching, and assessing: a revision of Bloom’s Taxonomy of Educational Objectives (Complete edition). New York: Longman.

    Google Scholar 

  • Arksey, H., & O'Malley, L. (2005). Scoping studies: towards a methodological framework. International Journal of Social Research Methodology, 8(1), 19–32. https://doi.org/10.1080/1364557032000119616.

    Article  Google Scholar 

  • Atmatzidou, S., & Demetriadis, S. (2016). Advancing students’ computational thinking skills through educational robotics: a study on age and gender relevant differences. Robotics and Autonomous Systems, 75, 661–670. https://doi.org/10.1016/j.robot.2015.10.008.

    Article  Google Scholar 

  • Barr, V., & Stephenson, C. (2011). Bringing computational thinking to K-12: what is involved and what is the role of the computer science education community? ACM Inroads, 2(1), 48–54. https://doi.org/10.1145/1929887.1929905.

    Article  Google Scholar 

  • Barr, D., Harrison, J., & Conery, L. (2011). Computational thinking: a digital age skill for everyone. Learning & Leading with Technology, 38(6), 20–23.

    Google Scholar 

  • Basnet, R. B., Doleck, T., Lemay, D. J., & Bazelais, P. (2018). Exploring computer science students’ continuance intentions to use Kattis. Education and Information Technologies, 23(3), 1145–1158. https://doi.org/10.1007/s10639-017-9658-2.

    Article  Google Scholar 

  • Basu, S., Biswas, G., & Kinnebrew, J. S. (2017). Learner modeling for adaptive scaffolding in a computational thinking-based science learning environment. User Modeling and User-Adapted Interaction, 27(1), 5–53. https://doi.org/10.1007/s11257-017-9187-0.

    Article  Google Scholar 

  • Bati, K., Yetişir, M. I., Çalişkan, I., Güneş, G., & Gül Saçan, E. (2018). Teaching the concept of time: a steam-based program on computational thinking in science education. Cogent Education, 5(1), 1–16.

    Article  Google Scholar 

  • Bers, M. U., Flannery, L., Kazakoff, E. R., & Sullivan, A. (2014). Computational thinking and tinkering: exploration of an early childhood robotics curriculum. Computers & Education, 72, 145–157. https://doi.org/10.1016/j.compedu.2013.10.020.

    Article  Google Scholar 

  • Bransford, J. D., Brown, A., & Cocking, R. (1999). How people learn: mind, brain, experience, and school. Washington, DC: National Research Council.

    Google Scholar 

  • Brennan, K., & Resnick, M. (2012). New frameworks for studying and assessing the development of computational thinking. Paper presented at the Proceedings of the 2012 Annual Meeting of the American Educational Research Association, Vancouver, Canada, 1-25.

  • Buffum, P. S., Lobene, E. V., Frankosky, M. H., Boyer, K. E., Wiebe, E. N., & Lester, J. C. (2015). A practical guide to developing and validating computer science knowledge assessments with application to middle school. In Proceedings of the 46th ACM Technical Symposium on Computer Science Education (pp. 622-627). ACM. doi:https://doi.org/10.1145/2676723.2677295.

  • Bureau of Labor Statistics (2018). Computer and Information Technology Occupations. Retrieved on October 15, 2019 from https://www.bls.gov/ooh/computer-and-information-technology/home.htm.

  • Chen, G., Shen, J., Barth-Cohen, L., Jiang, S., Huang, X., & Eltoukhy, M. (2017). Assessing elementary students’ computational thinking in everyday reasoning and robotics programming. Computers & Education, 109, 162–175. https://doi.org/10.1016/j.compedu.2017.03.001.

    Article  Google Scholar 

  • Cuny, J., Snyder, L., & Wing, J. M. (2010). Demystifying computational thinking for non-computer scientists. Unpublished Manuscript in Progress, Referenced in Http://Www.Cs.Cmu.Edu/~ CompThink/Resources/TheLinkWing.Pdf.

  • de Paula, B. H., Burn, A., Noss, R., & Valente, J. A. (2018). Playing Beowulf: bridging computational thinking, arts and literature through game-making. International Journal of Child-Computer Interaction, 16, 39–46. https://doi.org/10.1016/j.ijcci.2017.11.003.

    Article  Google Scholar 

  • Denner, J., Werner, L., Campe, S., & Ortiz, E. (2014). Pair programming: under what conditions is it advantageous for middle school students? Journal of Research on Technology in Education, 46(3), 277–296. https://doi.org/10.1080/15391523.2014.888272.

    Article  Google Scholar 

  • Denning, P. J. (2017). Computational thinking in science. American Scientist, 105(1), 13–17.

    Article  Google Scholar 

  • Denning, P. J., & Freeman, P. A. (2009). The profession of IT: computing’s paradigm. Communications of the ACM, 52(12), 28–30. https://doi.org/10.1145/1610252.1610265.

    Article  Google Scholar 

  • DiSessa, A. A. (2001). Changing minds: computers, learning, and literacy. MIT Press.

  • Doleck, T., Bazelais, P., Lemay, D. J., Saxena, A., & Basnet, R. B. (2017). Algorithmic thinking, cooperativity, creativity, critical thinking, and problem solving: exploring the relationship between computational thinking skills and academic performance. Journal of Computers in Education, 4(4), 355–369. https://doi.org/10.1007/s40692-017-0090-9.

    Article  Google Scholar 

  • Durak, H. Y., & Saritepeci, M. (2018). Analysis of the relation between computational thinking skills and various variables with the structural equation model. Computers and Education, 116, 191–202. https://doi.org/10.1016/j.compedu.2017.09.004.

    Article  Google Scholar 

  • ISTE (2011). Computational Thinking in K–12 Education leadership toolkit. Computer Science Teacher Association: http://csta. acm. org/Curriculum/sub/CurrFiles/471.11 CTLeadershipt Toolkit-SP-vF. pdf adresinden alındı.

  • Fessakis, G., Gouli, E., & Mavroudi, E. (2013). Problem solving by 5–6 years old kindergarten children in a computer programming environment: A case study. Computers & Education, 63, 87–97. https://doi.org/10.1016/j.compedu.2012.11.016.

    Article  Google Scholar 

  • Flanigan, A. E., Peteranetz, M. S., Shell, D. F., & Soh, L. K. (2017). Implicit intelligence beliefs of computer science students: Exploring change across the semester. Contemporary Educational Psychology, 48, 179–196.

  • Fronza, I., Ioini, N. E., & Corral, L. (2017). Teaching computational thinking using agile software engineering methods: a framework for middle schools. ACM Trans. Computers & Education, 17(4), 19:1–19:28. https://doi.org/10.1145/3055258.

    Article  Google Scholar 

  • Garneli, V., & Chorianopoulos, K. (2018). Programming video games and simulations in science education: exploring computational thinking through code analysis. Interactive Learning Environments, 26(3), 386–401. https://doi.org/10.1080/10494820.2017.1337036.

    Article  Google Scholar 

  • Goldstein, S., Princiotta, D., & Naglieri, J. A. (2015). Handbook of intelligence. Evolutionary Theory, Historical Perspective, and Current Concepts. New York, NY: Springer.

    Google Scholar 

  • Grover, S., & Pea, R. (2013). Computational thinking in K–12: A review of the state of the field. Educational researcher, 42(1), 38–43.

  • Grover, S., Pea, R., & Cooper, S. (2015). Designing for deeper learning in a blended computer science course for middle school students. Computer Science Education, 25(2), 199–237.

  • Grover, S., & Pea, R. (2018). Computational thinking: a competency whose time has come. Computer Science Education: Perspectives on teaching and learning in school (pp. 19–37). London: Bloomsbury Academic. https://doi.org/10.1080/08993408.2015.1033142.

    Book  Google Scholar 

  • Guzdial, M. (2008). Education paving the way for computational thinking. Communications of the ACM, 51(8), 25–27.

    Article  Google Scholar 

  • Heppner, P. P., & Petersen, C. H. (1982). The development and implications of a personal problem-solving inventory. Journal of Counseling Psychology, 29(1), 66–75. https://doi.org/10.1037/0022-0167.29.1.66.

    Article  Google Scholar 

  • Hsu, T. C., Chang, S. C., & Hung, Y. T. (2018). How to learn and how to teach computational thinking: suggestions based on a review of the literature. Computers & Education, 126, 296–310. https://doi.org/10.1016/j.compedu.2018.07.004.

    Article  Google Scholar 

  • Jacob, S., Nguyen, H., Tofel-Grehl, C., Richardson, D., & Warschauer, M. (2018). Teaching computational thinking to English learners. NYS TESOL Journal, 5(2).

  • Jun, S., Han, S., Kim, H., & Lee, W. (2014). Assessing the computational literacy of elementary students on a national level in Korea. Educational Assessment Evaluation and Accountability, 26(4), 319–332. https://doi.org/10.1007/s11092-013-9185-7.

    Article  Google Scholar 

  • Kafai, Y. B., & Resnick, M. (Eds.). (1996). Constructionism in practice: designing, thinking, and learning in a digital world. Hillsdale: Erlbaum.

    Google Scholar 

  • Kahn, K., Sendova, E., Sacristán, A. I., & Noss, R. (2011). Young students exploring cardinality by constructing infinite processes. Technology, Knowledge and Learning, 16(1), 3–34. https://doi.org/10.1007/s10758-011-9175-0.

    Article  Google Scholar 

  • Kay, A., & Goldberg, A. (1977). Personal dynamic media. Computer, 10(3), 31–41.

    Article  Google Scholar 

  • Knee, J. A., Hirsh-Pasek, K., Golinkoff, R. M., & Singer, D. (2006). Play = learning. New York: Oxford University Press. https://doi.org/10.1093/acprof:oso/9780195304381.001.0001.

    Book  Google Scholar 

  • Kong, S. C., Chiu, M. M., & Lai, M. (2018). A study of primary school students’ interest, collaboration attitude, and programming empowerment in computational thinking education. Computers & Education, 127, 178–189.

    Article  Google Scholar 

  • Korkmaz, Ö. (2012). A validity and reliability study of the Online Cooperative Learning Attitude Scale (OCLAS). Computers & Education, 59(4), 1162–1169. https://doi.org/10.1016/j.compedu.2012.05.021.

    Article  Google Scholar 

  • Korkmaz, Ö., Çakir, R., & Özden, M. Y. (2017). A validity and reliability study of the computational thinking scales (CTS). Computers in Human Behavior, 72, 558–569. https://doi.org/10.1016/j.chb.2017.01.005.

    Article  Google Scholar 

  • Lahtinen, E., Ala-Mutka, K., & Järvinen, H. (2005). A study of the difficulties of novice programmers. In: Proceedings of the 10th Annual SIGCSE Conference on innovation and Technology in Computer Science Education (pp. 14–18).

  • Leonard, J., Buss, A., Gamboa, R., Mitchell, M., Fashola, O. S., Hubert, T., & Almughyirah, S. (2016). Using robotics and game design to enhance children's self-efficacy, STEM attitudes, and computational thinking skills. Journal of Science Education and Technology, 25(6), 860–876. https://doi.org/10.1007/s10956-016-9628-2.

    Article  Google Scholar 

  • Looi, C. K., How, M. L., Longkai, W., Seow, P., & Liu, L. (2018). Analysis of linkages between an unplugged activity and the development of computational thinking. Computer Science Education, 28(3), 255–279.

    Article  Google Scholar 

  • Lye, S. Y., & Koh, J. H. L. (2014). Review on teaching and learning of computational thinking through programming: what is next for K-12? Computers in Human Behavior, 41, 51–61. https://doi.org/10.1016/j.chb.2014.09.012.

    Article  Google Scholar 

  • McMillan, J. H., Hellsten, L. M., & Klinger, D. A. (2011). Classroom assessment: Principles and practice for effective standards-based instruction (Canadian ed.). Toronto, ON: Pearson.

  • Moreno-León, J., Robles, G., & Román-González, M. (2015). Dr. Scratch: automatic analysis of scratch projects to assess and foster computational thinking. Revista De Educacion a Distancia, 46, 1–23.

    Google Scholar 

  • Moreno-León, J., Robles, G., & Román-González, M. (2016a). Code to learn: where does it belong in the K-12 curriculum. Journal of Information Technology Education: Research, 15, 283–303.

    Article  Google Scholar 

  • Moreno-León, J., Robles, G., & Román-González, M. (2016b). Comparing computational thinking development assessment scores with software complexity metrics. Paper presented at the IEEE Global Engineering Education Conference (EDUCON) (pp. 1040-1045). doi:https://doi.org/10.1109/EDUCON.2016.7474681.

  • Mouza, C., Marzocchi, A., Pan, Y., & Pollock, L. (2016). Development, implementation, and outcomes of an equitable computer science after-school program: Findings from middle-school students. Journal of Research on Technology in Education, 48(2), 84–104. https://doi.org/10.1080/15391523.2016.1146561.

    Article  Google Scholar 

  • Munoz, R., Villarroel, R., Barcelos, T. S., Riquelme, F., Quezada, Á., & Bustos-Valenzuela, P. (2018). Developing computational thinking skills in adolescents with autism spectrum disorder through digital game programming. IEEE Access, 6, 63880–63889.

    Article  Google Scholar 

  • Papert, S. (1980). Mindstorms: Children, computers, and powerful ideas. New York: Basic Books, Inc..

    Google Scholar 

  • Pellas, N., & Vosinakis, S. (2018). The effect of simulation games on learning computer programming: a comparative study on high school students’ learning performance by assessing computational problem-solving strategies. Education and Information Technologies, 23(6), 2423–2452. https://doi.org/10.1007/s10639-018-9724-4.

    Article  Google Scholar 

  • Pham, M. T., Rajić, A., Greig, J. D., Sargeant, J. M., Papadopoulos, A., & McEwen, S. A. (2014). A scoping review of scoping reviews: advancing the approach and enhancing the consistency. Research Synthesis Methods, 5(4), 371–385.

    Article  Google Scholar 

  • Psycharis, S., & Kallia, M. (2017). The effects of computer programming on high school students’ reasoning skills and mathematical self-efficacy and problem solving. Instructional Science, 45(5), 583–602. https://doi.org/10.1007/s11251-017-9421-5.

    Article  Google Scholar 

  • Rich, K., & Yadav, A. (2019). Infusing computational thinking instruction into elementary mathematics and science: patterns of teacher implementation. In Society for Information Technology & Teacher Education International Conference (pp. 76-80). Association for the Advancement of Computing in Education (AACE).

  • Rijke, W. J., Bollen, L., Eysink, T. H., & Tolboom, J. L. (2018). Computational thinking in primary school: an examination of abstraction and decomposition in different age groups. Informatics in Education, 17(1), 77.

    Article  Google Scholar 

  • Rojas-López, A., & García-Peñalvo, F. J. (2018). Learning scenarios for the subject methodology of programming from evaluating the computational thinking of new students. Revista Iberoamericana De Tecnologias Del Aprendizaje, 13(1), 30–36. https://doi.org/10.1109/RITA.2018.2809941.

    Article  Google Scholar 

  • Román-González, M., Pérez-González, J. C., & Jiménez-Fernández, C. (2017). Which cognitive abilities underlie computational thinking? Criterion validity of the computational thinking test. Computers in Human Behavior, 72, 678–691. https://doi.org/10.1016/j.chb.2016.08.047.

    Article  Google Scholar 

  • Román-González, M., Pérez-González, J.-C., Moreno-León, J., & Robles, G. (2018a). Extending the nomological network of computational thinking with non-cognitive factors. Computers in Human Behavior, 80, 441–459. https://doi.org/10.1016/j.chb.2017.09.030.

    Article  Google Scholar 

  • Román-González, M., Pérez-González, J. C., Moreno-León, J., & Robles, G. (2018b). Can computational talent be detected? Predictive validity of the Computational Thinking Test. International Journal of Child-Computer Interaction, 18, 47–58. https://doi.org/10.1016/j.ijcci.2018.06.004.

    Article  Google Scholar 

  • Romero, M., Lepage, A., & Lille, B. (2017). Computational thinking development through creative programming in higher education. International Journal of Educational Technology in Higher Education, 14(1), 1–15. https://doi.org/10.1186/s41239-017-0080-z.

    Article  Google Scholar 

  • Sáez-López, J., Román-González, M., & Vázquez-Cano, E. (2016). Visual programming languages integrated across the curriculum in elementary school: a two year case study using “scratch” in five schools. Computers & Education, 97, 129–141. https://doi.org/10.1016/j.compedu.2016.03.003.

    Article  Google Scholar 

  • Seehorn, D., Carey, S., Fuschetto, B., Lee, I., Moix, D., O’Grady-Cunniff, D., et al. (2011). CSTA K–12 computer science standards: Revised 2011. New York: ACM.

    Google Scholar 

  • Shute, V. J., Chen, S., & Asbell-Clark, J. (2017). Demystifying computational thinking. Educational Research Review, 22(2017), 142–158. https://doi.org/10.1016/j.edurev.2017.09.003.

    Article  Google Scholar 

  • Strawhacker, A., Lee, M., & Bers, M. U. (2018). Teaching tools, teachers’ rules: exploring the impact of teaching styles on young children’s programming knowledge in ScratchJr. International Journal of Technology and Design Education, 28(2), 347–376. https://doi.org/10.1007/s10798-017-9400-9.

    Article  Google Scholar 

  • Tsarava, K., Moeller, K., & Ninaus, M. (2018). Training computational thinking through board games: the case of crabs & turtles. International Journal of Serious Games, 5(2), 25–44.

    Article  Google Scholar 

  • von Wangenheim, C. G., Hauck, J. C. R., Demetrio, M. F., Pelle, R., da Cruz Alves, N., Barbosa, H., & Azevedo, L. F. (2018). CodeMaster - automatic assessment and grading of App Inventor and Snap! programs. Informatics in Education, 17(1), 117–150. https://doi.org/10.15388/infedu.2018.08.

    Article  Google Scholar 

  • Weintrop, D., & Wilensky, U. (2018). How block-based, text-based, and hybrid block/text modalities shape novice programming practices. International Journal of Child-Computer Interaction, 17, 83–92. https://doi.org/10.1016/j.ijcci.2018.04.005.

    Article  Google Scholar 

  • Werner, L., Denner, J., & Campe, S. (2015). Children programming games: a strategy for measuring computational learning. ACM Transactions on Computing Education (TOCE), 14(4), 24:1–24:22. https://doi.org/10.1145/2677091.

    Article  Google Scholar 

  • Whetton, D. A., & Cameron, K. S. (2002). Answers to exercises taken from developing management skills. Northwestern University.

  • Wilson, C., Sudol, L. A., Stephenson, C., & Stehlik, M. (2010). Running on empty: the failure to teach K-12 computer science in the digital age. Association for Computing Machinery, 26.

  • Wing, J. (1881). (2008). Computational thinking and thinking about computing. Philosophical Transactions of the Royal Society A: Mathematical. Physical and Engineering Sciences, 366(1881), 3717–3725. https://doi.org/10.1098/rsta.2008.0118.

    Article  Google Scholar 

  • Wing, J. (2014). Computational thinking benefits society. 40th Anniversary Blog of Social Issues in Computing. Retrieved from http://socialissues.cs.toronto.edu/2014/01/computational-thinking. Accessed 15 Oct 2019.

  • Witherspoon, E. B., Higashi, R. M., Schunn, C. D., Baehr, E. C., & Shoop, R. (2017). Developing computational thinking through a virtual robotics programming curriculum. ACM Trans. Comput. Educ., 18(1), 4:1–4:20. https://doi.org/10.1145/3104982.

    Article  Google Scholar 

  • Yadav, A., Gretter, S., Good, J., & McLean, T. (2017). Computational thinking in teacher education. In Emerging research, practice, and policy on computational thinking (pp. 205-220). Springer, Cham.

  • Yadav, A., Larimore, R., Rich, K., & Schwarz, C. (2019). Integrating computational thinking in elementary classrooms: introducing a toolkit to support teachers. In Society for Information Technology & Teacher Education International Conference (pp. 93-96). Association for the Advancement of Computing in Education (AACE).

  • Yağcı, M. (2018). A valid and reliable tool for examining computational thinking skills. Education and Information Technologies, 24(1), 1–23. https://doi.org/10.1007/s10639-018-9801-8.

    Article  Google Scholar 

  • Zhong, B., Wang, Q., Chen, J., & Li, Y. (2016). An exploration of three-dimensional integrated assessment for computational thinking. Journal of Educational Computing Research, 53(4), 562–590.

    Article  Google Scholar 

  • Zhong, B., Wang, Q., Chen, J., & Li, Y. (2017). Investigating the period of switching roles in pair programming in a primary school. Educational Technology & Society, 20(3), 220–233.

    Google Scholar 

Download references

Acknowledgements

We would like to thank the Callysto project and the Government of Canada CanCode program: Innovation, Science and Economic Development Canada, as well as our partner organizations, Cybera and the Pacific Institute for the Mathematical Sciences (PIMS).

Author information

Authors and Affiliations

  1. Faculty of Education, University of Alberta, 6-102 Education Centre North, Edmonton, T6G 2G5, Canada

    Maria Cutumisu, Cathy Adams & Chang Lu

Authors
  1. Maria Cutumisu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Cathy Adams
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Chang Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Maria Cutumisu.

Ethics declarations

This study is not based on empirical data. Instead, it is a review of several published empirical studies. Therefore, no ethics application nor informed consent was necessary to conduct this study.

Conflict of Interest

The authors declare that they have no conflict of interest.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Cutumisu, M., Adams, C. & Lu, C. A Scoping Review of Empirical Research on Recent Computational Thinking Assessments. J Sci Educ Technol 28, 651–676 (2019). https://doi.org/10.1007/s10956-019-09799-3

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10956-019-09799-3

Keywords

Search

Navigation