Advances in Health Sciences Education

, Volume 15, Issue 1, pp 9–30 | Cite as

Factors affecting feeling-of-knowing in a medical intelligent tutoring system: the role of immediate feedback as a metacognitive scaffold

  • Gilan M. El Saadawi
  • Roger Azevedo
  • Melissa Castine
  • Velma Payne
  • Olga Medvedeva
  • Eugene Tseytlin
  • Elizabeth Legowski
  • Drazen Jukic
  • Rebecca S. Crowley


Previous studies in our laboratory have shown the benefits of immediate feedback on cognitive performance for pathology residents using an intelligent tutoring system (ITS) in pathology. In this study, we examined the effect of immediate feedback on metacognitive performance, and investigated whether other metacognitive scaffolds will support metacognitive gains when immediate feedback is faded. Twenty-three participants were randomized into intervention and control groups. For both groups, periods working with the ITS under varying conditions were alternated with independent computer-based assessments. On day 1, a within-subjects design was used to evaluate the effect of immediate feedback on cognitive and metacognitive performance. On day 2, a between-subjects design was used to compare the use of other metacognitive scaffolds (intervention group) against no metacognitive scaffolds (control group) on cognitive and metacognitive performance, as immediate feedback was faded. Measurements included learning gains (a measure of cognitive performance), as well as several measures of metacognitive performance, including Goodman–Kruskal gamma correlation (G), bias, and discrimination. For the intervention group, we also computed metacognitive measures during tutoring sessions. Results showed that immediate feedback in an intelligent tutoring system had a statistically significant positive effect on learning gains, G and discrimination. Removal of immediate feedback was associated with decreasing metacognitive performance, and this decline was not prevented when students used a version of the tutoring system that provided other metacognitive scaffolds. Results obtained directly from the ITS suggest that other metacognitive scaffolds do have a positive effect on G and discrimination, as immediate feedback is faded. We conclude that immediate feedback had a positive effect on both metacognitive and cognitive gains in a medical tutoring system. Other metacognitive scaffolds were not sufficient to replace immediate feedback in this study. However, results obtained directly from the tutoring system are not consistent with results obtained from assessments. In order to facilitate transfer to real-world tasks, further research will be needed to determine the optimum methods for supporting metacognition as immediate feedback is faded.


Intelligent tutoring systems Diagnostic reasoning Clinical competence Cognition Diagnostic errors Education, medical Educational technology Feeling-of-knowing Pathology Problem solving Metacognition 


  1. Aleven, V., & Koedinger, K. R. (2002). An effective metacognitive strategy: Learning by doing and explaining with a computer-based cognitive tutor. Cognitive Science, 26, 147–181.CrossRefGoogle Scholar
  2. Anderson, J. R. (1993). Rules of the mind. Hillsdale, NJ: Lawrence Erlbaum Associates.Google Scholar
  3. Azevedo, R., Cromley, J. G., & Seibert, D. (2004). Does adaptive scaffolding facilitate student’s ability to regulate their learning with hypermedia? Contemporary Educational Psychology, 29(3), 344–370. doi:10.1016/j.cedpsych.2003.09.002.CrossRefGoogle Scholar
  4. Azevedo, R., & Hadwin, A. F. (2005). Scaffolding self-regulated learning and metacognition-implications for the design of computer-based scaffolds. Instructional Science, 33, 367–379. doi:10.1007/s11251-005-1272-9.CrossRefGoogle Scholar
  5. Azevedo, R., & Lajoie, S. (1998). The cognitive basis for the design of a mammography interpretation tutor. International Journal of Artificial Intelligence in Education, 9, 32–44.Google Scholar
  6. Azevedo, R., Moos, D., Greene, J., Winters, F., & Cromley, J. (2008). Why is externally-facilitated regulated learning more effective than self-regulated learning with hypermedia? Educational Technology Research and Development, 56(1), 45–72. doi:10.1007/s11423-007-9067-0.CrossRefGoogle Scholar
  7. Azevedo, R., & Witherspoon, A. M. (2008). Self-regulated learning with hypermedia. In A. Graesser, J. Dunlosky, D. Hacker (Eds.), Handbook of metacognition in education (in press). Mahwah, NJ: Erlbaum.Google Scholar
  8. Azevedo, R., & Witherspoon, A. M. (2009) Self-regulated use of hypermedia. In A. Graesser, J. Dunlosky, D. Hacker (Eds.), Handbook of metacognition in education (in press). Mahwah, NJ: Erlbaum.Google Scholar
  9. Balzer, W., Doherty, M., & O’Conner, R. (1989). Effects of cognitive feedback on performance. Psychological Bulletin, 106, 410–433. doi:10.1037/0033-2909.106.3.410.CrossRefGoogle Scholar
  10. Butler, D., & Winne, P. H. (1995). Feedback and self-regulated learning: A theoretical synthesis. Review of Educational Research, 65(3), 245–281.Google Scholar
  11. Carver, C., & Scheier, M. (1990). Origins and functions of positive and negative affect: A control-process view. Psychological Review, 97, 19–35. doi:10.1037/0033-295X.97.1.19.CrossRefGoogle Scholar
  12. Choi, I., Land, S. M., & Turgeon, A. Y. (2005). Scaffolding peer-questioning strategies to facilitate metacognition during online small group discussion. Instructional Science, 33, 483–511. doi:10.1007/s11251-005-1277-4.CrossRefGoogle Scholar
  13. Clancy, W. (1983). Knowledge-based tutoring: The GUIDON program. Journal of Computer-based Instruction, 10, 8–14.Google Scholar
  14. Corbett, A. T., & Anderson, J. R. (2001). Locus of feedback control in computer-based tutoring: impact on learning rate, achievement and attitudes. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 245–252). Seattle, WA: ACM NY.Google Scholar
  15. Corbett, A. T., Koedinger, K. R., & Hadley, W. H. (2002). In Goodman, P. S. (Ed.), Cognitive tutors: From the research classroom to all classrooms—technology enhanced learning: Opportunities for change (pp. 198–224). Taylor & Francis.Google Scholar
  16. Crowley, R. S., Legowski, E., Medvedeva, O., Tseytlin, E., Roh, E., & Jukic, D. (2007). Evaluation of an intelligent tutoring system in pathology: Effects of external representation on performance gains, metacognition, and acceptance. Journal of the American Medical Informatics Association, 14(2), 182–190. doi:10.1197/jamia.M2241.CrossRefGoogle Scholar
  17. Crowley, R. S., & Medvedeva, O. (2006). An intelligent tutoring system for visual classification problem solving. Artificial Intelligence in Medicine, 36(1), 85–117.CrossRefGoogle Scholar
  18. Dabbagh, N., & Kitsantas, A. (2005). Using web-based pedagogical tools as a scaffolds for self-regulated learning. Instructional Science, 33, 513–540. doi:10.1007/s11251-005-1278-3.CrossRefGoogle Scholar
  19. Graber, M. L., Franklin, N., & Gordon, R. (2005). Diagnostic error in internal medicine. Archives of Internal Medicine, 165(13), 1493–1499. doi:10.1001/archinte.165.13.1493.CrossRefGoogle Scholar
  20. Graesser, A., McNamara, D., & VanLehn, K. (2005). Scaffolding deep comprehension strategies through Pint & Query, AuthTutor and iSTRAT. Educational Psychologist, 40(4), 225–234. doi:10.1207/s15326985ep4004_4.CrossRefGoogle Scholar
  21. Green, B. A. (2000). Project-based learning with the world wide web: A qualitative study of resource integration. Educational Technology Research and Development, 48(1), 45–66. doi:10.1007/BF02313485.CrossRefGoogle Scholar
  22. Hill, J. R., & Hannafin, M. J. (1997). Cognitive strategies and learning from the world wide web. Educational Technology Research and Development, 45(4), 37–64. doi:10.1007/BF02299682.CrossRefGoogle Scholar
  23. Kelemen, W. L., Frost, P. J., & Weaver, C. A., I. I. I. (2000). Individual differences in metacognition: Evidence against a general metacognitive ability. Memory & Cognition, 28(1), 92–107.Google Scholar
  24. Kulhavy, R. W., & Stock, W. A. (1989). Feedback in written instruction: The place of response certitude. Educational Psychology Review, 1, 279–308. doi:10.1007/BF01320096.CrossRefGoogle Scholar
  25. Kulhavy, R. W., Yekovich, F. R., & Dyer, J. W. (1979). Feedback and content review in programmed instruction. Contemporary Educational Psychology, 4, 91–98. doi:10.1016/0361-476X(79)90062-6.CrossRefGoogle Scholar
  26. Loboda, T. D., & Brusilovsky, P. (2006). WADEIn II: Adaptive explanatory visualization for expressions evaluation. Proceedings of the 2006 ACM Symposium on Software Visualization. Brighton, UK: ACM.Google Scholar
  27. Maries, A., & Kumar, A. (2008). The effect of student model on learning. Advanced learning technologies. ICALT’08, 8th IEEE International Conference on 2008 (pp. 877–881).Google Scholar
  28. Mattheos, N., Nattestad, A., Falk-Nilsson, E., & Attström, R. (2004). The interactive examination: Assessing students’ self-assessment ability. Medical Education, 38(4), 378–389. doi:10.1046/j.1365-2923.2004.01788.x.CrossRefGoogle Scholar
  29. Metcalf, J., & Dunlosky, J. (2008). Metamemory. In H. Roediger (Ed.), Cognitive psychology of memory (Vol. 2). Oxford: Elsevier.Google Scholar
  30. Mitrovic, A., & Martin, B. (2002). Evaluating the effects of open student models on learning (Vol. 2347/2002, pp. 296–305). Berlin/Heidelberg: Springer.Google Scholar
  31. Nelson, T. (1984). A comparison of current measures of the accuracy of feeling-of-knowing predictions. Psychological Bulletin, 95(1), 109–133. doi:10.1037/0033-2909.95.1.109.CrossRefGoogle Scholar
  32. Nelson, T., & Narens, L. (1990). Metamemory: A theoretical framework and some new findings. In G. Bower (Ed.), The psychology of learning and motivation. San Diego, CA: Academic Press.Google Scholar
  33. Pintrich, P. R., & De Groot, E. V. (1990). Motivational and self-regulated learning components of classroom. Journal of Educational Psychology, 82(1), 33–40. doi:10.1037/0022-0663.82.1.33.CrossRefGoogle Scholar
  34. Puntambekar, S., & Hubscher, R. (2005). Tools for scaffolding students in a complex learning environment: What have we gained and what have we missed? Educational Psychologist, 40, 1–12. doi:10.1207/s15326985ep4001_1.CrossRefGoogle Scholar
  35. Reiser, B. (2004). Scaffolding complex learning: The mechanisms of structuring and problematizing student work. Journal of the Learning Sciences, 13(3), 273–304. doi:10.1207/s15327809jls1303_2.CrossRefGoogle Scholar
  36. Saadawi, G. M., Tseytin, E., Legowski, E., Jukic, D., Castine, M., Crowley, R. S. (2008). A natural language intelligent tutoring system for training pathologists: implementation and evaluation. Advances in Health Sciences Education: Theory and Practice, 13, 709–722.CrossRefGoogle Scholar
  37. Sharples, M., Jeffery, N., du Boulay, B., Teather, B., Teather, D., & du Boulay, G. (2000). Structured computer-based training in the interpretation of neuroradiological images. International Journal of Medical Informatics, 60, 263–280. doi:10.1016/S1386-5056(00)00101-5.CrossRefGoogle Scholar
  38. Smith, P., Obradovich, J., Heintz, P., et al. (1998). Successful use of an expert system to teach diagnostic reasoning for antibody identification. Proceedings of the 4th International Conference on Intelligent Tutoring Systems (pp. 54–63). San Antonio, TX.Google Scholar
  39. VanLehn, K. (2006). The behavior of tutoring systems. International Journal of Artificial Intelligence in Education, 16(3), 227–265.Google Scholar
  40. Voytovich, A. E., Rippey, R. M., & Suffredini, A. (1985). Premature conclusions in diagnostic reasoning. Journal of Medical Education, 60, 302–307.Google Scholar
  41. Wenger, A. (1987). Artificial intelligence and tutoring systems-computational and cognitive approaches to the communication of knowledge. Los Altos, CA: Morgan Kaufmann Publishers Inc.Google Scholar
  42. White, B., & Frederiksen, J. (2005). A theoretical framework and approach for fostering metacognitive development. Educational Psychologist, 40(4), 211–223. doi:10.1207/s15326985ep4004_3.CrossRefGoogle Scholar
  43. Winne, P. H. (1982). Minimizing the black box problem to enhance the validity of theories about instructional effects. Instructional Science, 11, 13–28. doi:10.1007/BF00120978.CrossRefGoogle Scholar
  44. Winne, P. H. (2001). Self-regulated learning viewed from models of information processing. In J. Douglas, J. D. Hacker, & A. C. Graesser (Eds.), Self-regulated learning and academic achievement: Theoretical perspective (pp. 153–190). Mahwah, NJ: Lawrence Erlbaum Associates.Google Scholar
  45. Winne, P. H., & Hadwin, A. F. (1998). Studying as self-regulated learning. In J. Douglas, J. D. Hacker, & A. C. Graesser (Eds.), Metacognition in educational theory and practice (pp. 277–304). Mahwah, NJ: Lawrence Erlbaum Associates.Google Scholar
  46. Winne, P., & Hadwin, A. (Eds.). (2008). The weave of motivation and self-regulated learning (pp. 297–314). NY: Taylor & Francis.Google Scholar
  47. Winne, P. H., & Marx, R. W. (1982). Students’ and teachers’ views of thinking processes for classroom learning. The Elementary School Journal, 82, 493–518. doi:10.1086/461284.CrossRefGoogle Scholar
  48. Woo, C. W., Evens, M. W., Freedman, R., et al. (2006). An intelligent tutoring system that generates a natural language dialogue using dynamic multi-level planning. Artificial Intelligence in Medicine, 38(1), 25–46. doi:10.1016/j.artmed.2005.10.004.CrossRefGoogle Scholar
  49. Yudelson, M. V., Medvedeva, O. P., & Crowley, R. S. (2008). Multifactor approach to student model evaluation in a complex cognitive domain. User Modeling and User-Adapted Interaction, 18(4), 315–382.CrossRefGoogle Scholar
  50. Yudelson, M. V., Medvedeva, O., Legowski, E., Castine, M., Jukic, D., & Crowley, R. S. (2006). Mining student learning data to develop high lever pedagogic strategy in a medical ITS. Proceedings of AAAI Educational Data Mining 21st National Conference Educational Data Mining Workshop (pp. 82–90). Boston MA: AAAI press.Google Scholar
  51. Zimmerman, B. (2006). Development and adaptation of expertise: The role of self-regulatory processes and beliefs. In K. A. Ericsson, P. Charness, P. Feltovich, & R. Hoffman (Eds.), The Cambridge handbook of expertise and expert performance (pp. 705–722). New York: Cambridge University Press.Google Scholar

Copyright information

© Springer Science+Business Media B.V. 2009

Authors and Affiliations

  • Gilan M. El Saadawi
    • 1
    • 2
  • Roger Azevedo
    • 3
  • Melissa Castine
    • 1
  • Velma Payne
    • 1
  • Olga Medvedeva
    • 1
  • Eugene Tseytlin
    • 1
  • Elizabeth Legowski
    • 1
  • Drazen Jukic
    • 1
    • 4
    • 5
  • Rebecca S. Crowley
    • 1
    • 5
    • 6
  1. 1.Department of Biomedical InformaticsUniversity of Pittsburgh School of MedicinePittsburghUSA
  2. 2.Department of Health and Community ServicesUniversity of Pittsburgh School of NursingPittsburghUSA
  3. 3.Department of PsychologyUniversity of MemphisMemphisUSA
  4. 4.Department of DermatologyUniversity of Pittsburgh School of MedicinePittsburghUSA
  5. 5.Department of PathologyUniversity of Pittsburgh School of MedicinePittsburghUSA
  6. 6.Intelligent Systems ProgramUniversity of Pittsburgh School of Arts and SciencesPittsburghUSA

Personalised recommendations