Abstract
Introduction We developed and evaluated a Natural Language Interface (NLI) for an Intelligent Tutoring System (ITS) in Diagnostic Pathology. The system teaches residents to examine pathologic slides and write accurate pathology reports while providing immediate feedback on errors they make in their slide review and diagnostic reports. Residents can ask for help at any point in the case, and will receive context-specific feedback. Research questions We evaluated (1) the performance of our natural language system, (2) the effect of the system on learning (3) the effect of feedback timing on learning gains and (4) the effect of ReportTutor on performance to self-assessment correlations. Methods The study uses a crossover 2 × 2 factorial design. We recruited 20 subjects from 4 academic programs. Subjects were randomly assigned to one of the four conditions—two conditions for the immediate interface, and two for the delayed interface. An expert dermatopathologist created a reference standard and 2 board certified AP/CP pathology fellows manually coded the residents’ assessment reports. Subjects were given the opportunity to self grade their performance and we used a survey to determine student response to both interfaces. Results Our results show a highly significant improvement in report writing after one tutoring session with 4–fold increase in the learning gains with both interfaces but no effect of feedback timing on performance gains. Residents who used the immediate feedback interface first experienced a feature learning gain that is correlated with the number of cases they viewed. There was no correlation between performance and self-assessment in either condition.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Anderson, J. R., Corbett, A. T., Koedinger, K. R., & Pelletier, R. (1995). Cognitive tutors: Lessons learned. Journal of the Learning Sciences, 4(2), 167–207.
Aperio Technologies. From http://www.aperio.com/
Azevedo, R. (2002). Beyond intelligent tutoring systems: Using computers as META cognitive tools to enhance learning? Instructional Science, 30, 31–45.
Azevedo, R., & Lajoie, S. P. (1998). The cognitive basis for the design of a mammography interpretation tutor. International Journal of Artificial Intelligence in Education, 9, 32–44.
Bangert-Drowns, R. L., Kulik, C. C., Kulik, J. A., & Morgan, M. T. (1991). The instructional effect of feedback in test-like events. Review of Educational Research, 61(2), 213–238.
Carroll, M., & Nelson, T. O. (1993). Effect of over learning on the feeling of knowing is more detectable in within-subject than in between-subject designs. The American Journal of Psychology, 106(2), 227–235.
Chi, M., Siler, A., Jeong, H., Yamauchi, T., & Lavancher, C. (2001). Learning from human tutoring. Cognitive Science, 25, 471–553.
Cohen, P. A., Kulik, J. A., & Kulik, C. C. (1982). Educational outcomes of tutoring: A meta-analysis of findings. American Educational Research Journal, 19(2), 237–248.
College of American Pathologists, Cancer Protocols and Checklists. From http://www.cap.org/apps/docs/cancer_protocols/protocols_index.html
Corbett, A., McLaughlin, M., & Scarpinatto, K. C. (2000). Modeling student knowledge: Cognitive tutors in high school and college. User Modeling and User-Adapted Interaction, 10, 81–108.
Crowley, R. S., Legowski, E., Medvedeva, O., & Tseytlin, E. (2005). An ITS for medical classifcation problem-solving: Effects of tutoring and representations. Paper presented at the Proceedings of the 12th International Conference on Artificial Intelligence (AIED05), Amsterdam, The Netherlands.
Crowley, R. S., Legowski, E., Medvedeva, O., Tseytlin, E., Roh, E., & Jukic, D. (2007). Evaluation of an intelligent tutoring system in pathology: effects of external representation on performance gains, metacognition, and acceptance. Journal of American Medical Informatics Association, 14(2), 182–190.
Crowley, R. S., & Medvedeva, O. (2006). An intelligent tutoring system for visual classification problem solving. Artificial Intelligence in Medicine, 36, 85–117.
Crowley, R. S., Tseytlin, E., & Jukic, D. (2005). ReportTutor—an intelligent tutoring system that uses a natural language interface. Paper presented at the AMIA Symposium Proceedings, Washington, DC.
Evens, M. W., Brandle, S., Change, R. C., Freedman, R., Glass, M., Lee, Y. H., et al. (2001). CIRCSIM-Tutor: An intelligent tutoring system using natural language dialogue. Paper presented at the Twelfth Midwest AI and Cognitive Science Conference (MAICS), Oxford, OH.
Fox, B. (1991). Cognitive and interactional aspects of correction in tutoring. Paper presented at the Teaching Knowledge and Intelligent tutoring, Ablex, Norwood, NJ.
Gertner, A., & Van Lehn, K. (2000). ANDES: A coached problem solving environment for physics. In G. Gautheir, C. Frasson, & K. Van Lehn (Eds.), Intelligent tutoring systems: 5th International conference (ITS-2000) (pp. 131–142). Berlin: Springer.
Graesser, A., Van Lahn, K., Rose, C., Jordan, P., & Harter, D. (2001). Intelligent tutoring systems with conversational dialogue. AI Magazine, 22(4), 39–51.
Graesser, A. C., Lu, S., Jackson, G. T., Mitchell, H. H., Ventura, M., Olney, A., et al. (2004). AutoTutor: A tutor with dialogue in natural language. Behavior Research Methods, Instruments, & Computers, 36(2), 180–192.
Koriat, A. (1997). Monitoring one’s own knowledge during study: A cue-utilization approach to judgments of learning. Journal of Experimental Psychology, 126(4), 349–370.
Kulik, J. A., & Kulik, C. C. (1988). Timing of feedback and verbal learning. Review of Educational Research, 58(1), 79–97.
Lajoie, S., & Derry, S. J. (1993). Computers as cognitive tools. Hillsdale, NJ: Lawrence Erlbaum Associates.
Leinhardt, G., & Ohlsson, S. (1990). Tutorials on the structure of tutoring from teachers. Journal of Artificial Intelligence in Education, 2(1), 21–46.
Nelson, T. O. (1984). A comparison of current measures of the accuracy of feeling-of-knowing predictions. Psychological Bulletin, 95(1), 109–133.
Ohlsson, S., & Rees, E. (1991). The function of conceptual understanding in the learning of arithmatic procedures. Cognition and Instruction, 8, 103–179.
Protégé. Retrieved from http://protege.stanford.edu/
Ros’e , C., Litman, D., Behembe, D., Forbes, K., & Van Lehn, K. (2003). A comparison of tutor and student behavior in speech versus text based tutoring. Paper presented at the Proc. HLT/NAACL Workshop: Building Educational Applications Using NLP.
Slamecka, N. J., & Graf, P. (1978). The generation effect. Delineation of a phenomenon. Journal of experimental psychology. Human Learning and Memory, 4, 592–604.
Sleeman, D., & Brown, J. S. (1982). Intelligent tutoring systems. London: Academic Press.
Van Lehn, K., Jordan, P. W., Ros’e, C., & Wilson, R. (2002). The architecture of Why-2 atlas: A coach for qualitative physics essay writing. In Procedure of Intelligent Tutoring Systems.
Wenger, E. (1987). Artificial intelligence and tutoring systems—computational and cognitive approaches to the communication of knowledge. Los Altos, CA: Kaufmann Publishers.
Acknowledgments
Work on ReportTutor is supported by a grant from the National Cancer Institute (R25 CA101959). This work was conducted using the Protégé resource, which is supported by grant LM007885 from the United States National Library of Medicine. We gratefully acknowledge the contribution of the SPECIALIST NLP tools provided the National Library of Medicine. We thank Olga Medvedeva for her expert technical help with this project, and Lucy Cafeo and Maria Bond for editorial assistance.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
El Saadawi, G.M., Tseytlin, E., Legowski, E. et al. A natural language intelligent tutoring system for training pathologists: implementation and evaluation. Adv in Health Sci Educ 13, 709–722 (2008). https://doi.org/10.1007/s10459-007-9081-3
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10459-007-9081-3