Abstract
Critical thinking is one of the important higher-order skills very much treasured in education, but hard to be measured using paper-pencil tests. In line with recent recommendation to measure high-order thinking skills with interactive tasks (vs. static one set of questions), in this study we developed an interactive and automated game-based assessment of critical thinking, using the Toulmin Model. In two real-life simulation stories, through interactive tasks in progressing scenes in the stories, students chose and rated evidence and conflicting reasons as supportive or non-supportive arguments in making the eventual decision. Critical thinking scores were awarded on choosing the appropriate evidence and reasons. The psychometric quality of the game was evaluated with 185 Chinese senior secondary students. Results showed that (i) reliabilities as measured by Cronbach’s α of the whole scale and subdomains were reasonable; (ii) parallel form reliability was high; (iii) its correlation and convergent validity with the popular Halpern Critical Thinking Assessment were comparable to those in other studies; (iv) it was generally not related to academic performance; and (v) the game was interesting and engaging. We also noted that students hesitated to query others and they were weak in applying critical thinking to problem-solving, which were in congruent with previous research showing students rarely used critical thinking to solve complex, real-world problems. In sum, we demonstrated successfully the use of interactive simulation tasks in measuring critical thinking. With the advancement of technology, our study suggested the possibility of assessing hard-to-measure important complex higher-order competence with dynamic games.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Ashford-Rowe, K., Herrington, J., & Brown, C. (2014). Establishing the critical elements that determine authentic assessment. Assessment & Evaluation in Higher Education, 39(2), 205–222. https://doi.org/10.1080/02602938.2013.819566
Awwal, N., Scoular, C., & Alom, M. (2017). An automated system for evaluating 21st century skills using game-based assessments. In 9th International Conference on Education and New Learning Technologies, Barcelona, Spain. 10.21125/edulearn.2017.1344.
Bartlett, J. E. (2002). Analysis of motivational orientation and learning strategies of high school business students. Business Education Forum, 56(4), 18–23. ERIC. https://eric.ed.gov/?id=EJ644416
Behar-Horenstein, L. S., & Niu, L. (2011). Teaching critical thinking skills in higher education: A review of the literature. Journal of College Teaching & Learning (TLC), 8(2), 25–41. https://doi.org/10.19030/tlc.v8i2.3554
Bensley, D. A., & Murtagh, M. P. (2012). Guidelines for a scientific approach to critical thinking assessment. Teaching of Psychology, 39(1), 5–16. https://doi.org/10.1177/0098628311430642
Brown, A. L., & Ferrara, R. A. (1985). Diagnosing zones of proximal development. In J. V. Wertsch (Ed.), Culture, communication, and cognition: Vygotskian perspectives (pp. 273–305). Cambridge University Press.
Butler, H. A. (2012). Halpern critical thinking assessment predicts real-world outcomes of critical thinking. Applied Cognitive Psychology, 26(5), 721–729. https://doi.org/10.1002/acp.2851
Byrnes, J. P., & Dunbar, K. N. (2014). The nature and development of critical-analytic thinking. Educational Psychology Review, 26(4), 477–493. https://doi.org/10.1007/s10648-014-9284-0
CAAP Program Management. (2012). ACT CAAP technical handbook 2011–2012. http://www.act.org/caap/pdf/CAAP-TechnicalHandbook.pdf
Cáceres, M., Nussbaum, M., & Ortiz, J. (2020). Integrating critical thinking into the classroom: A teacher’s perspective. Thinking Skills and Creativity, Article 100674. https://doi.org/10.1016/j.tsc.2020.100674
Charsky, D. (2010). From edutainment to serious games: A change in the use of game characteristics. Games and Culture, 5(2), 177–198. https://doi.org/10.1177/1555412009354727
Cohen, M. S., Freeman, J. T., & Thompson, B. (1998). Critical thinking skills in tactical decision making: A model and a training strategy. In J. A. Cannon-Bowers & E. Salas (Eds.), Making decisions under stress: Implications for individual and team training (pp. 155–189). American Psychological Association. https://doi.org/10.1037/10278-006
Davies, M., & Barnett, R. (2015). The Palgrave handbook of critical thinking in higher education. Springer.
Dicerbo, K. E. (2014). Game-based assessment of persistence. Journal of Educational Technology & Society, 17(1), 17–28. https://doi.org/10.1108/IJSHE-09-2012-0079
Ernst, J., & Monroe, M. (2004). The effects of environment-based education on students’ critical thinking skills and disposition toward critical thinking. Environmental Education Research, 10(4), 507–522. https://doi.org/10.1080/13504620600942998
Fahim, M., & Eslamdoost, S. (2014). Critical thinking: Frameworks and models for teaching. English Language Teaching, 7(7), 141–151. https://doi.org/10.5539/elt.v7n7p141
Funke, J. (2001). Dynamic systems as tools for analysing human judgement. Thinking & Reasoning, 7(1), 69–89. https://doi.org/10.1080/13546780042000046
Gold, J., Holman, D., & Thorpe, R. (2002). The role of argument analysis and story telling in facilitating critical thinking. Management Learning, 33(3), 371–388. https://doi.org/10.1177/1350507602333005
Goldstein, M., Crowell, A., & Kuhn, D. (2009). What constitutes skilled argumentation and how does it develop? Informal Logic, 29(4), 379–395. https://doi.org/10.22329/il.v29i4.2905
Halpern, D. F. (2003). The “how” and “why” of critical thinking assessment. In D. Fasko (Ed.), Critical thinking and reasoning: Current research, theory and practice (pp.355-366). Hampton Press.
Halpern, D. F. (2007). Halpern critical thinking assessment using everyday situations: Background and scoring standards. Claremont McKenna College.
Halpern, D. F. (2010). Halpern critical thinking assessment manual. Schuhfried GmbH.
Halpern, D. F. (2012). Halpern critical thinking assessment: Test manual. Schuhfried.
Halpern, D. F., Millis, K., Graesser, A. C., Butler, H., Forsyth, C., & Cai, Z. (2012). Operation ARA: A computerized learning game that teaches critical thinking and scientific reasoning. Thinking Skills and Creativity, 7(2), 93–100. https://doi.org/10.1016/j.tsc.2012.03.006
Hillocks, G. Jr. (2010). Teaching argument for critical thinking and writing: An introduction. English Journal, 99(6), 24–32. https://www.proquest.com/openview/74c8af977871a0b44c372d845bd5354e/1?pq-origsite=gscholar&cbl=42045.
Kantar, R., McNulty, K., Snow, E. L., Emery, M. A., Wainess, R., & Doshi, S. (2018). Constructing cognitive profiles for simulation-based hiring assessments. EDM. https://educationaldatamining.org/files/conferences/EDM2018/papers/EDM2018_paper_19.pdf
Ku, K. Y. (2009). Assessing students’ critical thinking performance: Urging for measurements using multi-response format. Thinking Skills and Creativity, 4(1), 70–76. https://doi.org/10.1016/j.tsc.2009.02.001
Ku, K. Y., Lai, E. C., & Hau, K. T. (2014). Epistemological beliefs and the effect of authority on argument–counterargument integration: An experiment. Thinking Skills and Creativity, 13, 67–79. https://doi.org/10.1016/j.tsc.2014.03.004
Kuhn, D. (2018). A role for reasoning in a dialogic approach to critical thinking. Topoi, 37, 121–128. https://doi.org/10.1007/s11245-016-9373-4
Kuhn, D. (2019). Critical thinking as discourse. Human Development, 62(3), 146–164. https://doi.org/10.1159/000500171
Kuhn, D., Zillmer, N., Crowell, A., & Zavala, J. (2013). Developing norms of argumentation: Metacognitive, epistemological, and social dimensions of developing argumentive competence. Cognition and Instruction, 31(4), 456–496. https://doi.org/10.1080/07370008.2013.830618
Larsson, K. (2017). Understanding and teaching critical thinking—A new approach. International Journal of Educational Research, 84, 32–42. https://doi.org/10.1016/j.ijer.2017.05.004
Leeman, R. W. (1987). Taking perspectives: Teaching critical thinking in the argumentation course. Annual Meeting of the Speech Communication Association, Boston, M.A. ERIC. https://files.eric.ed.gov/fulltext/ED292147.pdf
Macagno, F. (2016). Argument relevance and structure: Assessing and developing students’ uses of evidence. International Journal of Educational Research, 79, 180–194. https://doi.org/10.1016/j.ijer.2016.07.002
Mavridis, A., & Tsiatsos, T. (2017). Game-based assessment: Investigating the impact on test anxiety and exam performance. Journal of Computer Assisted Learning, 33(2), 137–150. https://doi.org/10.1111/jcal.12170
Mercier, H., & Sperber, D. (2011). Why do humans reason? Arguments for an argumentative theory. Behavioral and Brain Sciences, 34(2), 57–74. https://doi.org/10.1017/S0140525X10000968
Nunez, M., Medina, A., & Cubides, J. (2018). Enhancing critical thinking skills through political cartoons: A pedagogical implementation. Gist Education and Learning Research Journal, (16), 141–163. https://doi.org/10.26817/16925777.429
Nussbaum, E. M. (2011). Argumentation, dialogue theory, and probability modeling: Alternative frameworks for argumentation research in education. Educational Psychologist, 46(2), 84–106. https://doi.org/10.1080/00461520.2011.558816
Oaksford, M., Chater, N., & Hahn, U. (2008). Human reasoning and argumentation: The probabilistic approach. In J. E. Adler & L. J. Rips (Eds.), Reasoning: Studies of human inference and its foundations (pp. 383–413). Cambridge University Press. https://doi.org/10.1017/CBO9780511814273.022
OECD. (2021). Game-based assessment for education. In J. Buckley, L. Colosimo, R. Kantar, M. McCall, & E. Snow (Eds.), OECD digital education outlook 2021: Pushing the frontiers with artificial intelligence, blockchain and robots (pp. 195–208). OECD Publishing. https://doi.org/10.1787/589b283f-en
Oermann, M., Truesdell, S., & Ziolkowski, L. (2000). Strategy to assess, develop, and evaluate critical thinking. The Journal of Continuing Education in Nursing, 31(4), 155–160 https://www.proquest.com/openview/214273530097dd5c86e5290b28d933fe/1?pq-origsite=gscholar&cbl=35533
Pithers, R. T., & Soden, R. (2000). Critical thinking in education: A review. Educational Research, 42(3), 237–249. https://doi.org/10.1080/001318800440579
Rapanta, C., Garcia-Mila, M., & Gilabert, S. (2013). What is meant by argumentative competence? An integrative review of methods of analysis and assessment in education. Review of Educational Research, 83(4), 483–520. https://doi.org/10.3102/0034654313487606
Rippin, A., Booth, C., Bowie, S., & Jordan, J. (2002). A complex case: Using the case study method to explore uncertainty and ambiguity in undergraduate business education. Teaching in Higher Education, 7(4), 429–441. https://doi.org/10.1080/135625102760553928
Rosiek, J. (2003). Emotional scaffolding: An exploration of the teacher knowledge at the intersection of student emotion and the subject matter. Journal of Teacher Education, 54(5), 399–412. https://doi.org/10.1177/0022487103257089
Seelow, D. (2019). The art of assessment: Using game based assessments to disrupt, innovate, reform and transform testing. Journal of Applied Testing Technology, 20(S1), 1–16. JATT. http://www.jattjournal.net/index.php/atp/article/viewFile/142698/101462
Shaffer, D. W., Hatfield, D., Svarovsky, G. N., Nash, P., Nulty, A., Bagley, E., & Mislevy, R. (2009). Epistemic network analysis: A prototype for 21st-century assessment of learning. International Journal of Learning and Media, 1(2), 33–53. https://doi.org/10.1162/ijlm.2009.0013
Shakirova, D. M. (2007). Technology for the shaping of college students' and upper-grade students' critical thinking. Russian Education & Society, 49(9), 42–52. https://doi.org/10.2753/RES1060-9393490905
Shute, V. J. (2011). Stealth assessment in computer-based games to support learning. In S, Tobias & J. D. Fletcher (Eds), Computer games and instruction (pp. 503–524). Information Age Publishers. https://myweb.fsu.edu/vshute/pdf/shute%20pres_h.pdf
Shute, V. J., Wang, L., Greiff, S., Zhao, W., & Moore, G. (2016). Measuring problem solving skills via stealth assessment in an engaging video game. Computers in Human Behavior, 63, 106–117. https://doi.org/10.1016/j.chb.2016.05.047
Sonnleitner, P., Brunner, M., Greiff, S., Funke, J., Keller, U., Martin, R., et al. (2012). The genetics lab. Acceptance and psychometric characteristics of a computer-based microworld to assess complex problem solving. Psychological Test and Assessment Modeling, 54, 54–72 http://hdl.handle.net/10993/3189
Toulmin, S. E. (1958). The uses of argument. Cambridge University Press.
Verburgh, A., François, S., Elen, J., & Janssen, R. (2013). The assessment of critical thinking critically assessed in higher education: A validation study of the CCTT and the HCTA. Education Research International, Article 198920. https://doi.org/10.1155/2013/198920
Watson, G., & Glaser, E. M. (2008a). Watson–Glaser critical thinking appraisal, forms a and B manual. Pearson Education.
Watson, G., & Glaser, E. M. (2008b). Watson–Glaser critical thinking appraisal short form manual. Pearson Education.
Wiggins, G. 1993. Assessing student performance – Exploring the purpose and limits of testing. Jossey–Bass.
Zahner, D. (2013). Reliability and validity CLA+. Council for Aid to Education.
Zare, M., Barjesteh, H., & Biria, R. (2021). The effect of critical thinking-oriented dynamic assessment on Iranian EFL learners’ learning potential: A study of reading comprehension skill. Journal of Teaching Language Skills, 40(2), 193–227. https://doi.org/10.22099/jtls.2021.39475.2935
Zavala, J., & Kuhn, D. (2017). Solitary discourse is a productive activity. Psychological Science, 28(5), 578–586. https://doi.org/10.1177/0956797616689248
Funding
The work described in this paper was substantially supported by Grant 32071091 from National Natural Science Foundation of China.
Author information
Authors and Affiliations
Contributions
Dan Wang: Literature review, methods, analysis of findings.
Hongyun Liu: Methodology, resources, funding, supervision.
Kit-Tai Hau: Writing - review & editing.
Corresponding author
Ethics declarations
Statement The study was reviewed and approved by the research committee in Beijing Normal University, as well as by the committee in local government. All subjects and their parents were provided with written informed consent.
Conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Wang, D., Liu, H. & Hau, KT. Automated and interactive game-based assessment of critical thinking. Educ Inf Technol 27, 4553–4575 (2022). https://doi.org/10.1007/s10639-021-10777-9
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10639-021-10777-9