Advertisement

Advances in Health Sciences Education

, Volume 13, Issue 5, pp 709–722 | Cite as

A natural language intelligent tutoring system for training pathologists: implementation and evaluation

  • Gilan M. El Saadawi
  • Eugene Tseytlin
  • Elizabeth Legowski
  • Drazen Jukic
  • Melissa Castine
  • Jeffrey Fine
  • Robert Gormley
  • Rebecca S. Crowley
Original Paper

Abstract

Introduction We developed and evaluated a Natural Language Interface (NLI) for an Intelligent Tutoring System (ITS) in Diagnostic Pathology. The system teaches residents to examine pathologic slides and write accurate pathology reports while providing immediate feedback on errors they make in their slide review and diagnostic reports. Residents can ask for help at any point in the case, and will receive context-specific feedback. Research questions We evaluated (1) the performance of our natural language system, (2) the effect of the system on learning (3) the effect of feedback timing on learning gains and (4) the effect of ReportTutor on performance to self-assessment correlations. Methods The study uses a crossover 2 × 2 factorial design. We recruited 20 subjects from 4 academic programs. Subjects were randomly assigned to one of the four conditions—two conditions for the immediate interface, and two for the delayed interface. An expert dermatopathologist created a reference standard and 2 board certified AP/CP pathology fellows manually coded the residents’ assessment reports. Subjects were given the opportunity to self grade their performance and we used a survey to determine student response to both interfaces. Results Our results show a highly significant improvement in report writing after one tutoring session with 4–fold increase in the learning gains with both interfaces but no effect of feedback timing on performance gains. Residents who used the immediate feedback interface first experienced a feature learning gain that is correlated with the number of cases they viewed. There was no correlation between performance and self-assessment in either condition.

Keywords

Cognitive modeling Dialogue-based interfaces Health sciences education Intelligent tutoring systems Natural language processing Pathology 

Notes

Acknowledgments

Work on ReportTutor is supported by a grant from the National Cancer Institute (R25 CA101959). This work was conducted using the Protégé resource, which is supported by grant LM007885 from the United States National Library of Medicine. We gratefully acknowledge the contribution of the SPECIALIST NLP tools provided the National Library of Medicine. We thank Olga Medvedeva for her expert technical help with this project, and Lucy Cafeo and Maria Bond for editorial assistance.

References

  1. Anderson, J. R., Corbett, A. T., Koedinger, K. R., & Pelletier, R. (1995). Cognitive tutors: Lessons learned. Journal of the Learning Sciences, 4(2), 167–207.CrossRefGoogle Scholar
  2. Aperio Technologies. From http://www.aperio.com/
  3. Azevedo, R. (2002). Beyond intelligent tutoring systems: Using computers as META cognitive tools to enhance learning? Instructional Science, 30, 31–45.CrossRefGoogle Scholar
  4. Azevedo, R., & Lajoie, S. P. (1998). The cognitive basis for the design of a mammography interpretation tutor. International Journal of Artificial Intelligence in Education, 9, 32–44.Google Scholar
  5. Bangert-Drowns, R. L., Kulik, C. C., Kulik, J. A., & Morgan, M. T. (1991). The instructional effect of feedback in test-like events. Review of Educational Research, 61(2), 213–238.Google Scholar
  6. Carroll, M., & Nelson, T. O. (1993). Effect of over learning on the feeling of knowing is more detectable in within-subject than in between-subject designs. The American Journal of Psychology, 106(2), 227–235.CrossRefGoogle Scholar
  7. Chi, M., Siler, A., Jeong, H., Yamauchi, T., & Lavancher, C. (2001). Learning from human tutoring. Cognitive Science, 25, 471–553.CrossRefGoogle Scholar
  8. Cohen, P. A., Kulik, J. A., & Kulik, C. C. (1982). Educational outcomes of tutoring: A meta-analysis of findings. American Educational Research Journal, 19(2), 237–248.Google Scholar
  9. College of American Pathologists, Cancer Protocols and Checklists. From http://www.cap.org/apps/docs/cancer_protocols/protocols_index.html
  10. Corbett, A., McLaughlin, M., & Scarpinatto, K. C. (2000). Modeling student knowledge: Cognitive tutors in high school and college. User Modeling and User-Adapted Interaction, 10, 81–108.CrossRefGoogle Scholar
  11. Crowley, R. S., Legowski, E., Medvedeva, O., & Tseytlin, E. (2005). An ITS for medical classifcation problem-solving: Effects of tutoring and representations. Paper presented at the Proceedings of the 12th International Conference on Artificial Intelligence (AIED05), Amsterdam, The Netherlands.Google Scholar
  12. Crowley, R. S., Legowski, E., Medvedeva, O., Tseytlin, E., Roh, E., & Jukic, D. (2007). Evaluation of an intelligent tutoring system in pathology: effects of external representation on performance gains, metacognition, and acceptance. Journal of American Medical Informatics Association, 14(2), 182–190.CrossRefGoogle Scholar
  13. Crowley, R. S., & Medvedeva, O. (2006). An intelligent tutoring system for visual classification problem solving. Artificial Intelligence in Medicine, 36, 85–117.CrossRefGoogle Scholar
  14. Crowley, R. S., Tseytlin, E., & Jukic, D. (2005). ReportTutor—an intelligent tutoring system that uses a natural language interface. Paper presented at the AMIA Symposium Proceedings, Washington, DC.Google Scholar
  15. Evens, M. W., Brandle, S., Change, R. C., Freedman, R., Glass, M., Lee, Y. H., et al. (2001). CIRCSIM-Tutor: An intelligent tutoring system using natural language dialogue. Paper presented at the Twelfth Midwest AI and Cognitive Science Conference (MAICS), Oxford, OH.Google Scholar
  16. Fox, B. (1991). Cognitive and interactional aspects of correction in tutoring. Paper presented at the Teaching Knowledge and Intelligent tutoring, Ablex, Norwood, NJ.Google Scholar
  17. Gertner, A., & Van Lehn, K. (2000). ANDES: A coached problem solving environment for physics. In G. Gautheir, C. Frasson, & K. Van Lehn (Eds.), Intelligent tutoring systems: 5th International conference (ITS-2000) (pp. 131–142). Berlin: Springer.Google Scholar
  18. Graesser, A., Van Lahn, K., Rose, C., Jordan, P., & Harter, D. (2001). Intelligent tutoring systems with conversational dialogue. AI Magazine, 22(4), 39–51.Google Scholar
  19. Graesser, A. C., Lu, S., Jackson, G. T., Mitchell, H. H., Ventura, M., Olney, A., et al. (2004). AutoTutor: A tutor with dialogue in natural language. Behavior Research Methods, Instruments, & Computers, 36(2), 180–192.Google Scholar
  20. Koriat, A. (1997). Monitoring one’s own knowledge during study: A cue-utilization approach to judgments of learning. Journal of Experimental Psychology, 126(4), 349–370.Google Scholar
  21. Kulik, J. A., & Kulik, C. C. (1988). Timing of feedback and verbal learning. Review of Educational Research, 58(1), 79–97.Google Scholar
  22. Lajoie, S., & Derry, S. J. (1993). Computers as cognitive tools. Hillsdale, NJ: Lawrence Erlbaum Associates.Google Scholar
  23. Leinhardt, G., & Ohlsson, S. (1990). Tutorials on the structure of tutoring from teachers. Journal of Artificial Intelligence in Education, 2(1), 21–46.CrossRefGoogle Scholar
  24. Nelson, T. O. (1984). A comparison of current measures of the accuracy of feeling-of-knowing predictions. Psychological Bulletin, 95(1), 109–133.CrossRefGoogle Scholar
  25. Ohlsson, S., & Rees, E. (1991). The function of conceptual understanding in the learning of arithmatic procedures. Cognition and Instruction, 8, 103–179.CrossRefGoogle Scholar
  26. Protégé. Retrieved from http://protege.stanford.edu/
  27. Ros’e , C., Litman, D., Behembe, D., Forbes, K., & Van Lehn, K. (2003). A comparison of tutor and student behavior in speech versus text based tutoring. Paper presented at the Proc. HLT/NAACL Workshop: Building Educational Applications Using NLP.Google Scholar
  28. Slamecka, N. J., & Graf, P. (1978). The generation effect. Delineation of a phenomenon. Journal of experimental psychology. Human Learning and Memory, 4, 592–604.CrossRefGoogle Scholar
  29. Sleeman, D., & Brown, J. S. (1982). Intelligent tutoring systems. London: Academic Press.Google Scholar
  30. Van Lehn, K., Jordan, P. W., Ros’e, C., & Wilson, R. (2002). The architecture of Why-2 atlas: A coach for qualitative physics essay writing. In Procedure of Intelligent Tutoring Systems.Google Scholar
  31. Wenger, E. (1987). Artificial intelligence and tutoring systems—computational and cognitive approaches to the communication of knowledge. Los Altos, CA: Kaufmann Publishers.Google Scholar

Copyright information

© Springer Science+Business Media B.V. 2007

Authors and Affiliations

  • Gilan M. El Saadawi
    • 1
    • 2
  • Eugene Tseytlin
    • 1
  • Elizabeth Legowski
    • 1
  • Drazen Jukic
    • 1
    • 3
    • 4
  • Melissa Castine
    • 1
  • Jeffrey Fine
    • 4
  • Robert Gormley
    • 4
  • Rebecca S. Crowley
    • 1
    • 4
    • 5
    • 6
  1. 1.Department of Biomedical InformaticsUniversity of Pittsburgh School of MedicinePittsburghUSA
  2. 2.Department of Health and Community ServicesUniversity of Pittsburgh School of NursingPittsburghUSA
  3. 3.Department of DermatologyUniversity of Pittsburgh School of MedicinePittsburghUSA
  4. 4.Department of PathologyUniversity of Pittsburgh School of MedicinePittsburghUSA
  5. 5.Intelligent Systems ProgramUniversity of Pittsburgh School of Arts and SciencesPittsburghUSA
  6. 6.UPMC Cancer PavilionPittsburghUSA

Personalised recommendations