Abstract
Ill-structured problems, by definition, have multiple paths to a solution and are multifaceted making automated assessment and feedback a difficult challenge. Diagnostic reasoning about medical cases meet the criteria of ill-structured problem solving since there are multiple solution paths. The goal of this study was to develop an adaptive feedback mechanism that is capable of identifying and responding to novice physician misconceptions by mining the log trace data of user interactions in BioWorld, a computer-based learning environment designed to support medical students in regulating their own diagnostic reasoning. We applied a series of text pre-processing algorithms to extract the linguistic features that characterized symptoms identified by 30 medical students solving three endocrinology cases: diabetes mellitus (type 1), Pheochromocytoma, and Hyperthyroidism. A subgroup discovery mining algorithm was applied in two steps. First, rules were induced to predict an incorrect diagnosis submitted as the final solution for a case on the basis of symptoms highlighted by medical students as being pertinent, that were in fact not pertinent. Second, rules were induced to predict a distractor hypothesis (an incorrect hypothesis listed as the most probable) during the differential diagnosis process while solving the case. The rule set discovered through the subgroup discovery task was shown to predict both incorrect and distractor hypotheses set by novice physicians while solving the case. We discuss the implications in terms of developing an adaptive feedback mechanism that can detect physicians’ misconceptions and errors during problem-solving as a means to deliver prompts and feedback that promote the acquisition of metacognitive monitoring and control skills.
Similar content being viewed by others
References
Aleven, V., Roll, I., McLaren, B. M., & Koedinger, K. R. (2016). Help helps, but only so much: Research on help seeking with intelligent tutoring systems. International Journal of Artificial Intelligence in Education, 26(1), 205–223.
Avella, J. T., Kebritchi, M., Nunn, S. G., & Kanai, T. (2016). Learning analytics methods, benefits, and challenges in higher education: A systematic literature review. Online Learning. https://doi.org/10.24059/olj.v20i2.790.
Azevedo, R., & Aleven, V. (Eds.). (2013). International handbook of metacognition and learning technologies. Amsterdam: Springer.
Azevedo, R., Harley, J., Trevors, G., Feyzi-Behnagh, R., Duffy, M., Bouchet, F., et al. (2013). Using trace data to examine the complex roles of cognitive, metacognitive, andemotional self-regulatory processes during learning with multi-agent systems. In R. Azevedo & V. Aleven (Eds.), International handbook of metacognition and learning technologies (pp. 427–449). Amsterdam: Springer.
Baker, R. S. (2014). Educational data mining: An advance for intelligent systems in education. IEEE Intelligent Systems, 29(3), 78–82.
Baker, R., Walonoski, J., Heffernan, N., Roll, I., Corbett, A., & Koedinger, K. (2008). Why students engage in “gaming the system” behavior in interactive learning environments. Journal of Interactive Learning Research, 19(2), 185–224.
Baker, R. S. J. D., & Yacef, K. (2009). The state of educational data mining in 2009: A review and future visions. Journal of Educational Data Mining, 1(1), 3–17.
Basu, S., Biswas, G., & Kinnebrew, J. S. (2017). Learner modeling for adaptive scaffolding in a computational thinking-based science learning environment. User Modeling and User-Adapted Interaction. https://doi.org/10.1007/s11257-017-9187-0.
Berland, M., Baker, R., & Blikstein, P. (2014). Educational data mining and learning analytics: Applications to constructionist research. Technology, Knowledge and Learning, 19(1–2), 205–220.
Berner, E., & Graber, M. (2008). Overconfidence as a cause of diagnostic error in medicine. The American Journal of Medicine, 121(5), S2–S23. https://doi.org/10.1016/j.amjmed.2008.01.001.
Boekaerts, M. (1995). Self-regulated learning: Bridging the gap between metacognitive and metamotivation theories. Educational Psychologist, 30(4), 195–200. https://doi.org/10.1207/s15326985ep3004_4.
Bydžovská, H. & Popelínský, L. (2014). Course Recommendation from Social Data. In Proceedings of the 6th international conference on computer supported education, (pp. 268–275). Barcelona, Spain.
Cleary, T. J. (2011). Emergence of self-regulated learning microanalysis: Historical overview, essential features, and implications for research and practice. In B. J. Zimmerman & D. H. Schunk (Eds.), Handbook of self-regulation of learning and performance (pp. 329–345). New York: Routledge.
Collins, A. (2006). Cognitive apprenticeship. In K. Sawyer (Ed.), Cambridge handbook of the learning sciences (pp. 47–60). NY: Cambridge University Press.
Curtis, D. A., Lind, S. L., Dellinges, M., & Schroeder, K. (2012). Identifying student misconceptions in biomedical course assessments in dental education. Journal of Dental Education, 76(9), 1183–1194.
Delany, C., & Golding, C. (2014). Teaching clinical reasoning by making thinking visible: An action research project with allied health clinical educators. BMC Medical Education, 14(1), 20. https://doi.org/10.1186/1472-6920-14-20.
Doleck, T., Basnet, R. B., Poitras, E. G., & Lajoie, S. P. (2015). Mining learner-system interaction data: Implications for modeling learner behaviors and improving overlay models. Journal of Computers in Education, 2(4), 421–447. https://doi.org/10.1007/s40692-015-0040-3.
Doleck, T., Jarrell, A., Poitras, E. G., Chaouachi, M., & Lajoie, S. (2016). A tale of three cases: Examining accuracy, efficiency, and process differences in diagnosing virtual patient cases. Australasian Journal of Educational Technology, 36(5), 61–76. https://doi.org/10.14742/ajet.2759.
Ericsson, K. A., & Simon, H. A. (1993). Protocol analysis: Verbal reports as data. Cambridge, MA: MIT Press.
Eva, K. (2005). What every teacher needs to know about clinical reasoning. Medical Education, 39(1), 98–106. https://doi.org/10.1111/j.1365-2929.2004.01972.x.
Friedman, C., Gatti, G., Franz, T., Murphy, G., Wolf, F., Heckerling, P., et al. (2005). Do physicians know when their diagnoses are correct? Journal of General Internal Medicine, 20(4), 334–339. https://doi.org/10.1111/j.1525-1497.2005.30145.x.
García, Á. M., Charte, F., González, P., Carmona, C. J., & del Jesus, M. J. (2016). Subgroup discovery with evolutionary fuzzy systems in R: the SDEFSR package. The R Journal, 8(2), 307–323.
Graber, M., Franklin, N., & Gordon, R. (2005). Diagnostic error in internal medicine. Archives of Internal Medicine, 165(13), 1493. https://doi.org/10.1001/archinte.165.13.1493.
Graesser, A. C. (2016). Conversations with AutoTutor help students learn. International Journal of Artificial Intelligence in Education, 26, 124–132.
Greene, J., Bolick, C., Jackson, W., Caprino, A., Oswald, C., & McVea, M. (2015). Domain-specificity of self-regulated learning processing in science and history. Contemporary Educational Psychology, 42, 111–128. https://doi.org/10.1016/j.cedpsych.2015.06.001.
Herrera, F., Carmona, C. J., González, P., & del Jesus, M. J. (2011). An overview on subgroup discovery: Foundations and applications. Knowledge and Information Systems, 29(3), 495–525. https://doi.org/10.1007/s10115-010-0356-2.
Holzinger, A., Kickmeier-Rust, M. D., Wassertheurer, S., & Hessinger, M. (2009). Learning performance with interactive simulations in medical education: Lessons learned from results of learning complex physiological models with the HAEMOdynamics SIMulator. Computers & Education, 52(2), 292–301.
Holzinger, A., Valdez, A. C., & Ziefle, M. (2016). Towards Interactive Recommender Systems with the Doctor-in-the-Loop. Mensch und Computer 2016–Workshopband.
Klösgen, W. (2002). Types and forms of knowledge (patterns): Subgroup patterns. In W. Klösgen & J. Zytkow (Eds.), Handbook of data mining and knowledge discovery (pp. 47–51). New York: Oxford University Press.
Koedinger, K. R., D’Mello, S., McLaughlin, E. A., Pardos, Z. A., & Rosé, C. P. (2015). Data mining and education. WIREs Cognitive Science, 6, 333–353. https://doi.org/10.1002/wcs.1350.
Lajoie, S. P. (2003). Transitions and trajectories for studies of expertise. Educational Researcher, 32(8), 21–25.
Lajoie, S. P. (2005). Cognitive tools for the mind: The Promises of technology—Cognitive amplifiers or bionic prosthetics? In R. J. Sternberg & D. Preiss (Eds.), Intelligence and technology: Impact of tools on the nature and development of human skills (pp. 87–102). Mahwah, NJ: Erlbaum.
Lajoie, S. P. (2009). Developing professional expertise with a cognitive apprenticeship model: Examples from avionics and medicine. In K. A. Ericsson (Ed.), Development of professional expertise: Toward measurement of expert performance and design of optimal learning environments (pp. 61–83). Cambridge: Cambridge University Press.
Lajoie, S. P., Gauthier, G., & Lu, J. (2009). Convergence of data sources in the analysis of complex learning environments. Research and Practice in Technology Enhanced Learning, 4(3), 195–219.
Lajoie, S. P., Naismith, L., Poitras, E., Hong, Y.-J., Cruz-Panesso, I., Ranellucci, J., et al. (2013). Technology-rich tools to support self-regulated learning and performance in medicine. In R. Azevedo & V. Aleven (Eds.), International handbook of metacognition and learning technologies (pp. 229–242). New York: Springer.
Lajoie, S. P., & Poitras, E. G. (2017). Crossing disciplinary boundaries to improve technology-rich learning environments. Teachers College Record, 19(3), 1–30.
Lemmerich, F., Ifland, M., & Puppe, F. (2011). Identifying influence factors on students success by subgroup discovery. In Proceedings of the 4th international conference on educational data mining, (pp. 345–346). International Educational Data Mining Society, Eindhoven.
Luz, M., de Oliveira, G., de Sousa, C., & Da Poian, A. (2008). Glucose as the sole metabolic fuel: The possible influence of formal teaching on the establishment of a misconception about energy-yielding metabolism among students from Rio de Janeiro, Brazil. Biochemistry And Molecular Biology Education, 36(6), 407–416. https://doi.org/10.1002/bmb.20235.
Midgley, C., Maehr, M. L., Hruda, L. Z., Anderman, E., Anderman, L., Freeman, K. E., et al. (2000). Manual for the patterns of adaptive learning scales (PALS). Ann Arbor, MI: University of Michigan.
Mierswa, I., Wurst, M., Klinkenberg, R., Scholz, M., & Euler, T. (2006). YALE: Rapid prototyping for complex data mining tasks. In Proceedings of the 12th ACM SIGKDD international conference on knowledge discovery and data mining (pp. 935–940). New York: ACM Press.
Morton, J., Doran, D., & MacLaren, D. (2008). Common student misconceptions in exercise physiology and biochemistry. AJP Advances In Physiology Education, 32(2), 142–146. https://doi.org/10.1152/advan.00095.2007.
Naismith, L. (2013). Examining motivational and emotional influences on medical students' attention to feedback in a technology-rich environment for learning clinical reasoning (Doctoral Thesis). Retrieved from Digitool Library at McGill University Dissertations and Theses database.
Norman, G., & Eva, K. (2010). Diagnostic error and clinical reasoning. Medical Education, 44(1), 94–100. https://doi.org/10.1111/j.1365-2923.2009.03507.x.
Ocumpaugh, J., San Pedro, M. O., Lai, H., Baker, R. S., & Borgen, F. (2016). Middle school engagement with mathematics software and later interest and self-efficacy for STEM careers. Journal of Science Education and Technology, 25(6), 877–887. https://doi.org/10.1007/s10956-016-9637-1.
Olde Bekkink, M., Donders, A., Kooloos, J., de Waal, R., & Ruiter, D. (2016). Uncovering students’ misconceptions by assessment of their written questions. BMC Medical Education. https://doi.org/10.1186/s12909-016-0739-5.
Panadero, E., & Järvelä, S. (2017). Models of self-regulated learning: Zimmerman, Boekaerts, Winne, Pintrich, Efklides, and Hadwin & Järvelä. A review in 2017. Frontiers. https://doi.org/10.3389/fpsyg.2017.00422.
Papamitsiou, Z., & Economides, A. (2014). learning analytics and educational data mining in practice: A systematic literature review of empirical evidence. Educational Technology & Society, 17(4), 49–64.
Pekrun, R., Goetz, T., Frenzel, A., Barchfeld, P., & Perry, R. (2011). Measuring emotions in students’ learning and performance: The Achievement Emotions Questionnaire (AEQ). Contemporary Educational Psychology, 36(1), 36–48. https://doi.org/10.1016/j.cedpsych.2010.10.002.
Pelaez, N. J., Boyd, D. D., Rojas, J. B., & Hoover, M. A. (2005). Prevalence of blood circulation misconceptions among prospective elementary teachers. AJP Advances In Physiology Education, 29(3), 172–181. https://doi.org/10.1152/advan.00022.2004.
Peña-Ayala, A. (2014). Educational data mining: A survey and a data mining-based analysis of recent works. Expert Systems with Applications, 41(4), 1432–1462. https://doi.org/10.1016/j.eswa.2013.08.042.
Perry, N., & Winne, P. (2006). Learning from learning kits: gStudy traces of students’ self-regulated engagements with computerized content. Educational Psychology Review, 18(3), 211–228. https://doi.org/10.1007/s10648-006-9014-3.
Pintrich, P. (2004). A conceptual framework for assessing motivation and self-regulated learning in college students. Educational Psychology Review, 16(4), 385–407. https://doi.org/10.1007/s10648-004-0006-x.
Poitras, E., & Lajoie, S. (2013). A domain-specific account of self-regulated learning: The cognitive and metacognitive activities involved in learning through historical inquiry. Metacognition And Learning, 8(3), 213–234. https://doi.org/10.1007/s11409-013-9104-9.
Poitras, E. G., Lajoie, S. P., Doleck, T., & Jarrell, A. (2016a). Subgroup discovery with user interaction data: An empirically guided approach to improving intelligent tutoring systems. Educational Technology & Society, 19(2), 204–214.
Poitras, E. G., Lajoie, S. P., Jarrell, A., Doleck, T., & Naismith, L. (2016b). Intelligent tutoring systems in the medical domain: Fostering self-regulatory skills in problem-solving. In R. K. Atkinson (Ed.), Intelligent tutoring systems: Structure, applications, and challenges. Hauppauge, NY: Nova Publishing.
Poitras, E. G., Naismith, L. M., Doleck, T., & Lajoie, S. P. (2016c). Using learning analytics to identify medical student misconceptions in an online virtual patient environment. Online Learning Journal, 20(2), 183–194.
Puustinen, M., & Pulkkinen, L. (2001). Models of self-regulated learning: A review.Scandinavian. Journal of Educational Research, 45(3), 269–286. https://doi.org/10.1080/00313830120074206.
Richardson, D., & Speck, D. (2004). Addressing students’ misconceptions of renal clearance. AJP Advances In Physiology Education, 28(4), 210–212. https://doi.org/10.1152/advan.00021.2004.
Romero, C., González, P., Ventura, S., del Jesus, M., & Herrera, F. (2009). Evolutionary algorithms for subgroup discovery in E-learning: A practical application using moodle data. Expert Systems with Applications, 36(2), 1632–1644.
Romero, C., & Ventura, S. (2010). Educational data mining: A review of the state of the art. IEEE Transactions on Systems, Man, and Cybernetics, Part C C (Applications and Reviews), 40(6), 601–618.
Romero, C., & Ventura, S. (2012). Data mining in education. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 3(1), 12–27. https://doi.org/10.1002/widm.1075.
Scott, I. (2009). Errors in clinical reasoning: Causes and remedial strategies. BMJ, 338, 1860. https://doi.org/10.1136/bmj.b1860.
Sitzmann, T., & Ely, K. (2011). A meta-analysis of self-regulated learning in work-related training and educational attainment: What we know and where we need to go. Psychological Bulletin, 137(3), 421–442. https://doi.org/10.1037/a0022777.
Slater, S., Joksimovic, S., Kovanovic, V., Baker, R. S., & Gasevic, D. (2016). Tools for educational data mining: A review. Journal of Educational and Behavioral Statistics. https://doi.org/10.3102/1076998616666808.
Taub, M., Azevedo, R., Bouchet, F., & Khosravifar, B. (2014). Can the use of cognitive and metacognitive self-regulated learning strategies be predicted by learners’ levels of prior knowledge in hypermedia-learning environments? Computers in Human Behavior, 39, 356–367. https://doi.org/10.1016/j.chb.2014.07.018.
Winne, P. (2005). A perspective on state-of-the-art research on self-regulated learning. Instructional Science, 33(5–6), 559–565. https://doi.org/10.1007/s11251-005-1280-9.
Winne, P. H. (2010). Improving measurements of self-regulated learning. Educational Psychologist, 45(4), 267–276. https://doi.org/10.1080/00461520.2010.517150.
Winne, P. H., & Baker, R. S. J. D. (2013). The potentials of educational data mining for researching metacognition, motivation, and self-regulated learning. Journal of Educational Data Mining, 5(1), 1–8.
Wrobel, S. (1997). An Algorithm for multi-relational discovery of subgroups. In Proceedings of the first European symposium on principles of data mining and knowledge discovery (pp. 78–87). New York: Springer.
Zimmerman, B. (1990). Self-regulated learning and academic achievement: An overview. Educational Psychologist, 25(1), 3–17. https://doi.org/10.1207/s15326985ep2501_2.
Zimmerman, B. (2002). Becoming a self-regulated learner: An overview. Theory Into Practice, 41(2), 64–70. https://doi.org/10.1207/s15430421tip4102_2.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Ethical approval
Ethics approved by the McGill Research Ethics Board and informed consent was obtained from all participants included in the study.
Rights and permissions
About this article
Cite this article
Poitras, E.G., Doleck, T. & Lajoie, S.P. Towards detection of learner misconceptions in a medical learning environment: a subgroup discovery approach. Education Tech Research Dev 66, 129–145 (2018). https://doi.org/10.1007/s11423-017-9555-9
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11423-017-9555-9