Advertisement

Generalizing Detection of Gaming the System Across a Tutoring Curriculum

  • Ryan S. J. d. Baker
  • Albert T. Corbett
  • Kenneth R. Koedinger
  • Ido Roll
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4053)

Abstract

In recent years, a number of systems have been developed to detect differences in how students choose to use intelligent tutoring systems, and the attitudes and goals which underlie these decisions. These systems, when trained using data from human observations and questionnaires, can detect specific behaviors and attitudes with high accuracy. However, such data is time-consuming to collect, especially across an entire tutor curriculum. Therefore, to deploy a detector of behaviors or attitudes across an entire tutor curriculum, the detector must be able to transfer to a new tutor lesson without being re-trained using data from that lesson. In this paper, we present evidence that detectors of gaming the system can transfer to new lessons without re-training, and that training detectors with data from multiple lessons improves generalization, beyond just the gains from training with additional data.

Keywords

Intelligent Tutor System Cognitive Tutor Binary Prediction Test Lesson Training Lesson 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Aleven, V., McLaren, B.M., Roll, I., Koedinger, K.R.: Toward Tutoring Help Seeking. In: Lester, J.C., Vicari, R.M., Paraguaçu, F. (eds.) ITS 2004. LNCS, vol. 3220, pp. 227–239. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  2. 2.
    Aleven, V., Roll, I., McLaren, B.M., Ryu, E.J., Koedinger, K.: An Architecture to Combine Meta-Cognitive and Cognitive Tutoring: Pilot Testing the Help Tutor. In: Proceedings of the 12th International Conference on Artificial Intelligence in Education, pp. 17–24 (2005)Google Scholar
  3. 3.
    Arroyo, I., Woolf, B.: Inferring learning and attitudes from a Bayesian Network of log file data. In: Proceedings of the 12th International Conference on Artificial Intelligence in Education, pp. 33–40 (2005)Google Scholar
  4. 4.
    Baker, R.S., Corbett, A.T., Koedinger, K.R.: Detecting student misuse of intelligent tutoring systems. In: Lester, J.C., Vicari, R.M., Paraguaçu, F. (eds.) ITS 2004. LNCS, vol. 3220, pp. 531–540. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  5. 5.
    Baker, R.S., Corbett, A.T., Koedinger, K.R., Roll, I.: Detecting when students game the system, across tutor subjects and classroom cohorts. In: Ardissono, L., Brna, P., Mitrović, A. (eds.) UM 2005. LNCS, vol. 3538, pp. 220–224. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  6. 6.
    Baker, R.S., Corbett, A.T., Koedinger, K.R., Wagner, A.Z.: Off-Task Behavior in the Cognitive Tutor Classroom: When Students “Game The System”. In: Proceedings of ACM CHI 2004: Computer-Human Interaction, pp. 383–390 (2004)Google Scholar
  7. 7.
    Conati, C., Maclaren, H.: Data-Driven Refinement of a Probabilistic Model of User Affect. In: Ardissono, L., Brna, P., Mitrović, A. (eds.) UM 2005. LNCS (LNAI), vol. 3538, pp. 40–49. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  8. 8.
    de Vicente, A., Pain, H.: Informing the detection of the students’ motivational state: An empirical study. In: Cerri, S.A., Gouardéres, G., Paraguaçu, F. (eds.) ITS 2002. LNCS, vol. 2363, pp. 933–943. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  9. 9.
    Hanley, J.A., McNeil, B.J.: The Meaning and Use of the Area under a Receiver Operating Characteristic (ROC) Curve. Radiology 143, 29–36 (1982)Google Scholar
  10. 10.
    Maris, E.: Psychometric Latent Response Models. Psychometrika 60(4), 523–547 (1995)MATHCrossRefMathSciNetGoogle Scholar
  11. 11.
    Roll, I., Baker, R.S., Aleven, V., McLaren, B.M., Koedinger, K.R.: Modeling students’ metacognitive errors in two intelligent tutoring systems. In: Ardissono, L., Brna, P., Mitrović, A. (eds.) UM 2005. LNCS, vol. 3538, pp. 367–376. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  12. 12.
    Rosenthal, R., Rosnow, R.: Essentials of Behavioral Research: Methods and Data Analysis. McGraw-Hill, New York (1991)Google Scholar
  13. 13.
    Yu, L., Liu, H.: Feature selection for high-dimensional data: a fast correlation-based filter solution. In: Proceedings of the International Conference on Machine Learning, pp. 856–863 (2003)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Ryan S. J. d. Baker
    • 1
  • Albert T. Corbett
    • 2
  • Kenneth R. Koedinger
    • 2
  • Ido Roll
    • 2
  1. 1.Learning Sciences Research InstituteUniversity of NottinghamNottinghamUK
  2. 2.Human-Computer Interaction InstituteCarnegie Mellon UniversityPittsburghUSA

Personalised recommendations