Generalizing Detection of Gaming the System Across a Tutoring Curriculum
In recent years, a number of systems have been developed to detect differences in how students choose to use intelligent tutoring systems, and the attitudes and goals which underlie these decisions. These systems, when trained using data from human observations and questionnaires, can detect specific behaviors and attitudes with high accuracy. However, such data is time-consuming to collect, especially across an entire tutor curriculum. Therefore, to deploy a detector of behaviors or attitudes across an entire tutor curriculum, the detector must be able to transfer to a new tutor lesson without being re-trained using data from that lesson. In this paper, we present evidence that detectors of gaming the system can transfer to new lessons without re-training, and that training detectors with data from multiple lessons improves generalization, beyond just the gains from training with additional data.
KeywordsIntelligent Tutor System Cognitive Tutor Binary Prediction Test Lesson Training Lesson
Unable to display preview. Download preview PDF.
- 2.Aleven, V., Roll, I., McLaren, B.M., Ryu, E.J., Koedinger, K.: An Architecture to Combine Meta-Cognitive and Cognitive Tutoring: Pilot Testing the Help Tutor. In: Proceedings of the 12th International Conference on Artificial Intelligence in Education, pp. 17–24 (2005)Google Scholar
- 3.Arroyo, I., Woolf, B.: Inferring learning and attitudes from a Bayesian Network of log file data. In: Proceedings of the 12th International Conference on Artificial Intelligence in Education, pp. 33–40 (2005)Google Scholar
- 6.Baker, R.S., Corbett, A.T., Koedinger, K.R., Wagner, A.Z.: Off-Task Behavior in the Cognitive Tutor Classroom: When Students “Game The System”. In: Proceedings of ACM CHI 2004: Computer-Human Interaction, pp. 383–390 (2004)Google Scholar
- 9.Hanley, J.A., McNeil, B.J.: The Meaning and Use of the Area under a Receiver Operating Characteristic (ROC) Curve. Radiology 143, 29–36 (1982)Google Scholar
- 12.Rosenthal, R., Rosnow, R.: Essentials of Behavioral Research: Methods and Data Analysis. McGraw-Hill, New York (1991)Google Scholar
- 13.Yu, L., Liu, H.: Feature selection for high-dimensional data: a fast correlation-based filter solution. In: Proceedings of the International Conference on Machine Learning, pp. 856–863 (2003)Google Scholar