Abstract
The field of computational thinking (CT) is developing rapidly, reflecting its importance in the global economy. However, most empirical studies have targeted CT in K-12, thus, little attention has been paid to CT in higher education. The present scoping review identifies and summarizes existing empirical studies on CT assessments in post-secondary education, aiming to reveal the current trends of empirical research in this domain and key features of recent CT assessment instruments. It examines 33 peer-reviewed journal articles published between 2013 and 2019 from six databases. Results show that most assessment tools are designed for computing science and engineering undergraduates or pre-service and in-service teachers in these subjects. Most tools involve in-class interventions to promote CT skills. Several assessment formats were adopted in the selected studies, including selected-response questions, constructed-response questions, Likert scales, interviews, programming artefacts, observations, and interviews. Finally, most assessment instruments attempt to measure skills from a combination of dimensions including CT Concepts, Practices, and Perspectives from a hybrid-competency framework. More specifically, the skills assessed in most studies are algorithmic thinking, problem solving, data, logic and logical thinking, and abstraction. Findings may help instructors to select CT assessments for higher education and researchers to focus on less explored research areas.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Angeli, C., & Giannakos, M. (2020). Computational thinking education: Issues and challenges. Computers in Human Behavior, 105, 106185. https://doi.org/10.1016/j.chb.2019.106185
Arksey, H., & O’Malley, L. (2005). Scoping studies: Towards a methodological framework. International Journal of Social Research Methodology, 8(1), 19–32. https://doi.org/10.1080/1364557032000119616
Basnet, R. B., Doleck, T., Lemay, D. J., & Bazelais, P. (2018). Exploring computer science students’ continuance intentions to use kattis. Education and Information Technologies, 23(3), 1145–1158. https://doi.org/10.1007/s10639-017-9658-2
Berkaliev, Z., Devi, S., Fasshauer, G. E., Hickernell, F. J., Kartal, O., Li, X., ... & Zawojewski, J. S. (2014). Initiating a programmatic assessment report. Primus, 24(5), 403–420. doi:https://doi.org/10.1080/10511970.2014.893939
Biró, P., Csernoch, M., Máth, J., & Abari, K. (2015). Measuring the level of algorithmic skills at the end of secondary education in hungary. Procedia - Social and Behavioral Sciences, 176, 876–883. https://doi.org/10.1016/j.sbspro.2015.01.553
Brennan, K., & Resnick, M. (2012). New frameworks for studying and assessing the development of computational thinking. In Paper presented at the proceedings of the 2012 annual meeting of the American educational research association, Vancouver, Canada (pp. 1–25).
Bussaban, K., & Waraporn, P. (2015). Preparing undergraduate students majoring in computer science and mathematics with data science perspectives and awareness in the age of big data. Procedia - Social and Behavioral Sciences, 197, 1443–1446. https://doi.org/10.1016/j.sbspro.2015.07.092
Buteau, C., & Muller, E. (2017). Assessment in undergraduate programming-based mathematics courses. Digital Experiences in Mathematics Education, 3(2), 97–114. https://doi.org/10.1007/s40751-016-0026-4
Chao, P. (2016). Exploring students’ computational practice, design and performance of problem-solving through a visual programming environment. Computers & Education, 95, 202–215. https://doi.org/10.1016/j.compedu.2016.01.010
Csernoch, M., Biró, P., Máth, J., & Abari, K. (2015). Testing algorithmic skills in traditional and non-traditional programming environments. Informatics in Education, 14(2), 175–197.
Cutumisu, M., Adams, C., & Lu, C. (2019). A scoping review of empirical research on recent computational thinking assessments. Journal of Science Education and Technology, 28(6), 651–676. https://doi.org/10.1007/s10956-019-09799-3
Dağ, F. (2019). Prepare pre-service teachers to teach computer programming skills at K-12 level: Experiences in a course. Journal of Computers in Education, 6(2), 277–313. https://doi.org/10.1007/s40692-019-00137-5
Denning, P. J. (2017). Remaining trouble spots with computational thinking. Communications of the ACM, 60(6), 33–39.
Dichev, C., & Dicheva, D. (2017). Towards data science literacy. Procedia Computer Science, 108, 2151–2160. https://doi.org/10.1016/j.procs.2017.05.240
Doleck, T., Bazelais, P., Lemay, D. J., Saxena, A., & Basnet, R. B. (2017). Algorithmic thinking, cooperativity, creativity, critical thinking, and problem solving: Exploring the relationship between computational thinking skills and academic performance. Journal of Computers in Education, 4(4), 355–369. https://doi.org/10.1007/s40692-017-0090-9
Figl, K., & Laue, R. (2015). Influence factors for local comprehensibility of process models. International Journal of Human-Computer Studies, 82, 96–110. https://doi.org/10.1016/j.ijhcs.2015.05.007
Flanigan, A. E., Peteranetz, M. S., Shell, D. F., & Soh, L. (2017). Implicit intelligence beliefs of computer science students: Exploring change across the semester. Contemporary Educational Psychology, 48, 179–196. https://doi.org/10.1016/j.cedpsych.2016.10.003
Flores, C. (2018). Problem-based science, a constructionist approach to science literacy in middle school. International Journal of Child-Computer Interaction, 16, 25–30. https://doi.org/10.1016/j.ijcci.2017.11.001
Grover, S., Pea, R., & Cooper, S. (2015). Systems of assessments” for deeper learning of computational thinking in K-12. In Proceedings of the 2015 annual meeting of the American educational research association (pp. 15–20).
Grover, S., & Pea, R. (2018). Computational thinking: A competency whose time has come. In S. Sentance, S. Carsten, & E. Barendsen (Eds.), Computer science education: Perspectives on teaching and learning. Bloomsbury.
Hennessey, E. J., Mueller, J., Beckett, D., & Fisher, P. A. (2017). Hiding in plain sight: Identifying computational thinking in the ontario elementary school curriculum. Journal of Curriculum and Teaching, 6(1), 79–96.
Heppner, P. P., & Petersen, C. H. (1982). The development and implications of a personal problem-solving inventory. Journal of Counseling Psychology, 29(1), 66–75. https://doi.org/10.1037/0022-0167.29.1.66
Hsu, T., Chang, S., & Hung, Y. (2018). How to learn and how to teach computational thinking: Suggestions based on a review of the literature. Computers & Education, 126, 296–310. https://doi.org/10.1016/j.compedu.2018.07.004
Iñiguez-Berrozpe, T., & Boeren, E. (2019). Twenty-first century skills for all: Adults and problem solving in technology rich environments. Technology, Knowledge and Learning,. https://doi.org/10.1007/s10758-019-09403-y
Israel, M., Pearson, J. N., Tapia, T., Wherfel, Q. M., & Reese, G. (2015). Supporting all learners in school-wide computational thinking: A cross-case qualitative analysis. Computers & Education, 82, 263–279. https://doi.org/10.1016/j.compedu.2014.11.022
Jaipal-Jamani, K., & Angeli, C. (2017). Effect of robotics on elementary preservice teachers’ self-efficacy, science learning, and computational thinking. Journal of Science Education and Technology, 26(2), 175–192. https://doi.org/10.1007/s10956-016-9663-z
Kim, B., Kim, T., & Kim, J. (2013). Paper-and-pencil programming strategy toward computational thinking for non-majors: Design your solution. Journal of Educational Computing Research, 49(4), 437–459. https://doi.org/10.2190/EC.49.4.b
Knochel, A. D., & Patton, R. M. (2015). If art education then critical digital making: Computational thinking and creative code. Studies in Art Education, 57(1), 21–38. https://doi.org/10.1080/00393541.2015.11666280
Kordaki, M. (2013). High school computing teachers’ beliefs and practices: A case study. Computers & Education, 68, 141–152. https://doi.org/10.1016/j.compedu.2013.04.020
Korkmaz, Ö. (2012). A validity and reliability study of the online cooperative learning attitude scale (OCLAS). Computers & Education, 59(4), 1162–1169. https://doi.org/10.1016/j.compedu.2012.05.021
Korkmaz, Ö., Çakir, R., & Özden, M. Y. (2017). A validity and reliability study of the computational thinking scales (CTS). Computers in Human Behavior, 72, 558–569. https://doi.org/10.1016/j.chb.2017.01.005
Lazarinis, F., Karachristos, C. V., Stavropoulos, E. C., & Verykios, V. S. (2019). A blended learning course for playfully teaching programming concepts to school teachers. Education and Information Technologies, 24(2), 1237–1249. https://doi.org/10.1007/s10639-018-9823-2
Lee, Y., & Cho, J. (2020). Knowledge representation for computational thinking using knowledge discovery computing. Information Technology and Management, 21(1), 15–28. https://doi.org/10.1007/s10799-019-00299-9
Li, Y., Schoenfeld, A. H., diSessa, A. A., Graesser, A. C., Benson, L. C., English, L. D., & Duschl, R. A. (2020). Computational thinking is more about thinking than computing. Journal for STEM Education Research. https://doi.org/10.1007/s41979-020-00030-2
Libeskind-Hadas, R., & Bush, E. (2013). A first course in computing with applications to biology. Briefings in Bioinformatics, 14(5), 610–617. https://doi.org/10.1093/bib/bbt005
Lye, S. Y., & Koh, J. H. L. (2014). Review on teaching and learning of computational thinking through programming: What is next for K-12? Computers in Human Behavior, 41, 51–61. https://doi.org/10.1016/j.chb.2014.09.012
Magana, A. J. (2014). Learning strategies and multimedia techniques for scaffolding size and scale cognition. Computers & Education, 72, 367–377. https://doi.org/10.1016/j.compedu.2013.11.012
Magana, A. J., Falk, M. L., Vieira, C., & Reese, M. J. (2016). A case study of undergraduate engineering students’ computational literacy and self-beliefs about computing in the context of authentic practices. Computers in Human Behavior, 61, 427–442. https://doi.org/10.1016/j.chb.2016.03.025
Miller, L. D., Soh, L., Chiriacescu, V., Ingraham, E., Shell, D. F., Ramsay, S., & Hazley, M. P. (2013). Improving learning of computational thinking using creative thinking exercises in CS-1 computer science courses. In Paper presented at the (pp. 1426–1432). doi:https://doi.org/10.1109/FIE.2013.6685067
Mishra, S., & Iyer, S. (2015). An exploration of problem posing-based activities as an assessment tool and as an instructional strategy. Research and Practice in Technology Enhanced Learning, 10(1), 5. https://doi.org/10.1007/s41039-015-0006-0
Organisation for Economic Cooperation and Development (OECD) (2018). PISA 2021 mathematics framework (draft). Retrieved February 9, 2021, from http://www.oecd.org/pisa/publications/
Papert, S. (1980). Mindstorms: Children, computers, and powerful ideas. Basic Books Inc.
Peteranetz, M. S., Flanigan, A. E., Shell, D. F., & Soh, L. (2017). Computational creativity exercises: An avenue for promoting learning in computer science. IEEE Transactions on Education, 60(4), 305–313. https://doi.org/10.1109/TE.2017.2705152
Peteranetz, M. S., Flanigan, A. E., Shell, D. F., & Soh, L. (2018). Helping engineering students learn in introductory computer science (CS1) using computational creativity exercises (CCEs). IEEE Transactions on Education, 61(3), 195–203. https://doi.org/10.1109/TE.2018.2804350
Ricketts, A. (2014). Preservice elementary teachers’ ideas about scientific practices. Science & Education, 23(10), 2119–2135. https://doi.org/10.1007/s11191-014-9709-7
Robertson, J., Macvean, A., & Howland, K. (2013). Robust evaluation for a maturing field: The train the teacher method. International Journal of Child-Computer Interaction, 1(2), 50–60. https://doi.org/10.1016/j.ijcci.2013.05.001
Romero, M., Lepage, A., & Lille, B. (2017). Computational thinking development through creative programming in higher education. International Journal of Educational Technology in Higher Education, 14(1), 42. https://doi.org/10.1186/s41239-017-0080-z
Sentance, S., & Csizmadia, A. (2017). Computing in the curriculum: Challenges and strategies from a teacher’s perspective. Education and Information Technologies, 22(2), 469–495. https://doi.org/10.1007/s10639-016-9482-0
Sheehan, K. J., Pila, S., Lauricella, A. R., & Wartella, E. A. (2019). Parent-child interaction and children’s learning from a coding application. Computers & Education, 140, 103601. https://doi.org/10.1016/j.compedu.2019.103601
Shell, D. F., & Soh, L. (2013). Profiles of motivated self-regulation in college computer science courses: Differences in major versus required non-major courses. Journal of Science Education and Technology, 22(6), 899–913. https://doi.org/10.1007/s10956-013-9437-9
Sherman, M., & Martin, F. (2015). The assessment of mobile computational thinking. Journal of Computing Sciences in Colleges, 30(6), 53–59.
Shute, V. J., Sun, C., & Asbell-Clarke, J. (2017). Demystifying computational thinking. Educational Research Review, 22, 142–158. https://doi.org/10.1016/j.edurev.2017.09.003
Sneider, C., Stephenson, C., Schafer, B., & Flick, L. (2014). Computational thinking in high school science classrooms: Exploring the science “framework” and “NGSS.” Science Teacher, 81(5), 53–59.
Sung, W., Ahn, J., & Black, J. B. (2017). Introducing computational thinking to young learners: Practicing computational perspectives through embodiment in mathematics education. Technology, Knowledge and Learning, 22(3), 443–463. https://doi.org/10.1007/s10758-017-9328-x
Tang, K., Chou, T., & Tsai, C. (2020). A content analysis of computational thinking research: An international publication trends and research typology. The Asia-Pacific Education Researcher, 29(1), 9–19. https://doi.org/10.1007/s40299-019-00442-8
Tikva, C., & Tambouris, E. (2021). Mapping computational thinking through programming in K-12 education: A conceptual model based on a systematic literature review. Computers & Education, 162, 104083.
Topalli, D., & Cagiltay, N. E. (2018). Improving programming skills in engineering education through problem-based game projects with scratch. Computers & Education, 120, 64–74. https://doi.org/10.1016/j.compedu.2018.01.011
Tuttle, N., Kaderavek, J. N., Molitor, S., Czerniak, C. M., Johnson-Whitt, E., Bloomquist, D., ... & Wilson, G. (2016). Investigating the impact of NGSS-aligned professional development on PreK-3 teachers’ science content knowledge and pedagogy. Journal of Science Teacher Education, 27(7), 717–745. doi:https://doi.org/10.1007/s10972-016-9484-1
Voogt, J., Fisser, P., Good, J., Mishra, P., & Yadav, A. (2015). Computational thinking in compulsory education: Towards an agenda for research and practice. Education and Information Technologies, 20(4), 715–728.
Weintrop, D., Beheshti, E., Horn, M., Orton, K., Jona, K., Trouille, L., & Wilensky, U. (2016). Defining computational thinking for mathematics and science classrooms. Journal of Science Education and Technology, 25(1), 127–147. https://doi.org/10.1007/s10956-015-9581-5
Whetton, D. A., & Cameron, K. S. (2002). Answers to exercises taken from developing management skills. Northwestern University.
Wing, J. M. (2006). Computational thinking. Communications of the ACM, 49(3), 33–35.
Wing, J. M. (2008). Computational thinking and thinking about computing. Philosophical Transactions of the Royal Society a: Mathematical, Physical and Engineering Sciences, 366(1881), 3717–3725.
Yadav, A., Gretter, S., Good, J., & McLean, T. (2017). Computational thinking in teacher education. Emerging research, practice, and policy on computational thinking (pp. 205–220). Springer.
Yadav, A., Mayfield, C., Zhou, N., Hambrusch, S., & Korb, J. T. (2014). Computational thinking in elementary and secondary teacher education. ACM Transactions on Computing Education. https://doi.org/10.1145/2576872
Yesil, R., & Korkmaz, O. (2010). Reliability and validity analysis of the multiple intelligence perception scale. Education, 131(1), 8–33. Retrieved from https://go.gale.com/ps/i.do?p=AONE&sw=w&issn=00131172&v=2.1&it=r&id=GALE%7CA239813821&sid=googleScholar&linkaccess=abs
Zhang, L., & Nouri, J. (2019). A systematic review of learning computational thinking through scratch in K-9. Computers & Education, 141, 103607. https://doi.org/10.1016/j.compedu.2019.103607
Acknowledgements
We would like to thank the Social Sciences and Humanities Research Council of Canada—Insight Development Grant (SSHRC IDG) RES0034954 and Insight Grant (SSHRC IG) RES0048110, the Natural Sciences and Engineering Research Council (NSERC DG) RES0043209, and the Killam Cornerstone Operating Grant RES0043207 for supporting this research.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Lu, C., Macdonald, R., Odell, B. et al. A scoping review of computational thinking assessments in higher education. J Comput High Educ 34, 416–461 (2022). https://doi.org/10.1007/s12528-021-09305-y
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12528-021-09305-y