User Modeling and User-Adapted Interaction

, Volume 18, Issue 3, pp 287–314 | Cite as

Developing a generalizable detector of when students game the system

  • Ryan S. J. d. Baker
  • Albert T. Corbett
  • Ido Roll
  • Kenneth R. Koedinger
Original Paper

Abstract

Some students, when working in interactive learning environments, attempt to “game the system”, attempting to succeed in the environment by exploiting properties of the system rather than by learning the material and trying to use that knowledge to answer correctly. In this paper, we present a system that can accurately detect whether a student is gaming the system, within a Cognitive Tutor mathematics curricula. Our detector also distinguishes between two distinct types of gaming which are associated with different learning outcomes. We explore this detector’s generalizability, and find that it transfers successfully to both new students and new tutor lessons.

Keywords

Gaming the system Latent response models Cognitive tutors Behavior detection Machine learning Generalizable models Student modeling Interactive learning environments 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Aleven, V.: Helping students to become better help seekers: towards supporting metacognition in a cognitive tutor. Paper presented at German-USA Early Career Research Exchange Program: Research on Learning Technologies and Technology-Supported Education. Tubingen, Germany (2001)Google Scholar
  2. Aleven, V., McLaren, B.M., Roll, I., Koedinger, K.R.: Toward tutoring help seeking: applying cognitive modeling to meta-cognitive skills. In: Proceedings of the 7th International Conference on Intelligent Tutoring Systems (ITS 2004), pp. 227–239. Maceió, Brazil (2004)Google Scholar
  3. Anderson, J.R., Corbett A.T., Koedinger, K.R., Koedinger, K.R., Pelletier, R.: Cognitive tutors: lessons learned. J. Learn. Sci. 4(2), 167–207 (1995)CrossRefGoogle Scholar
  4. Arroyo, I., Woolf, B.: Inferring learning and attitudes from a Bayesian Network of log file data. In: Proceedings of the 12th International Conference on Artificial Intelligence in Education, pp. 33–40. Amsterdam (2005)Google Scholar
  5. Arroyo, I., Ferguson, K., Johns, J., Dragon, T., Meheranian, H., Fisher, D., Barto, A., Mahadevan, S., Woolf. B.P.: Repairing disengagement with non-invasive interventions. In: Proceedings of the 13th International Conference on Artificial Intelligence in Education, pp. 195–202. Marina del Rey, CA (2007)Google Scholar
  6. Baker, R.S., Corbett, A.T., Koedinger, K.R.: Detecting student misuse of intelligent tutoring systems. In: Proceedings of the 7th International Conference on Intelligent Tutoring Systems, pp. 531–540. Maceió, Brazil (2004)Google Scholar
  7. Baker, R.S., Corbett, A.T., Koedinger, K.R., Wagner, A.Z.: Off-Task Behavior in the Cognitive tutor classroom: when students “Game The System”. In: Proceedings of ACM CHI 2004: Computer-Human Interaction, pp. 383–390. Vienna, Austria (2004)Google Scholar
  8. Baker, R.S.J.d., Corbett, A.T., Koedinger, K.R., Evenson, S.E., Roll, I., Wagner, A.Z., Naim, M., Raspat, J., Baker, D.J., Beck, J.: Adapting to when students game an intelligent tutoring system. In: Proceedings of the 8th International Conference on Intelligent Tutoring Systems, pp. 392–401. Jhongli, Taiwan (2006)Google Scholar
  9. Baker, R.S.J.d., Walonoski, J.A., Heffernan, N.T., Roll, I., Corbett, A.T., Koedinger, K.R.: Why students engage in “Gaming the System” behavior in interactive learning environments. To appear in J. Interact. Learn. Res. (in press)Google Scholar
  10. Beal, C.R., Mitra, S., Cohen, P.R.: Modeling learning patterns of students with a tutoring system using Hidden Markov Models. In: Proceedings of the 13th International Conference on Artificial Intelligence in Education, pp. 238–245. Marina del Rey, CA (2007)Google Scholar
  11. Beal, C.R., Qu, L., Lee, H.: Classifying learner engagement through integration of multiple data sources. In: Proceedings of the 21st National Conference on Artificial Intelligence, pp. 2–8. Boston (2006)Google Scholar
  12. Beck, J.: Engagement tracing: using response times to model student disengagement. In: Proceedings of the 12th International Conference on Artificial Intelligence in Education (AIED 2005), pp. 88–95. Amsterdam (2005)Google Scholar
  13. Boyd, S., Vandenberghe, L.: Convex Optimization. Cambridge University Press, Cambridge, UK (2004)MATHGoogle Scholar
  14. Cheng, R., Vassileva, J.: Adaptive reward mechanism for sustainable online learning community. In: Proceedings of the 12th International Conference on Artificial Intelligence in Education, pp. 152–159. Amsterdam (2005)Google Scholar
  15. Conati, C., McLaren, H.: Data-driven refinement of a probabilistic model of user affect. In: Proceedings of the Tenth International Conference on User Modeling (UM2005), pp. 40–49. Edinburgh, Scotland (2005)Google Scholar
  16. Corbett, A.T., Anderson, J.R.: Knowledge tracing: modeling the acquisition of procedural knowledge. User Modeling and User-Adapted Interaction 4, 253–278 (1995)CrossRefGoogle Scholar
  17. Corbett, A.T., Koedinger, K.R., Hadley, W.H.: Cognitive tutors: from the research classroom to all classrooms. In: Goodman, P.S.(eds) Technology Enhanced Learning: Opportunities for Change, pp. 235–263.  Lawrence Erlbaum Associates, Mahwah, NJ (2001)Google Scholar
  18. D’Mello, S.K., Craig, S.D., Witherspoon, A.W., McDaniel, B.T., and Graesser, A.C.: Automatic detection of learner’s affect from conversational cues. To appear in User Modeling and User-Adapted Interaction. (in press)Google Scholar
  19. de Vicente, A. and Pain, H.: Informing the detection of the students’ motivational state: an empirical study. In: Proceedings of the Sixth International Conference on Intelligent Tutoring Systems, pp. 933–943. Biarritz, France (2002)Google Scholar
  20. Hanley, J.A., McNeil, B.J.: The meaning and use of the area under a Receiver Operating Characteristic (ROC) curve. Radiology 143, 29–36 (1982)Google Scholar
  21. Johns, J., Woolf, B.: A dynamic mixture model to detect student motivation and proficiency. In: Proceedings of the 21st National Conference on Artificial Intelligence (AAAI-06), pp. 163–168. Boston (2006)Google Scholar
  22. Koedinger, K.R.: Toward evidence for instructional design principles: Examples from Cognitive Tutor Math 6. In: Proceedings of PME-NA XXXIII (the North American Chapter of the International Group for the Psychology of Mathematics Education), pp. 21–49. Athens, GA (2002)Google Scholar
  23. Liu, H., Singh, P.L.: MAKEBELIEVE: using commonsense knowledge to generate stories. In: Proceedings of the 18th National Conference on Artificial Intelligence, AAAI 2002, pp. 957–958. Edmonton, Canada (2002)Google Scholar
  24. Magnussen, R., and Misfeldt, M.: Player transformation of educational multiplayer games. In: Proceedings of Other Players. Copenhagen, Denmark. Available at http://www.itu.dk/op/proceedings.htm (2004). Accessed 13 May 2007.
  25. Maris, E.: Psychometric latent response models. Psychometrika 60(4), 523–547 (1995)MATHCrossRefMathSciNetGoogle Scholar
  26. Mostow, J., Aist, G., Beck, J., Chalasani, R., Cuneo, A., Jia, P., Kadaru, K.: A La Recherche du Temps Perdu, or As Time Goes By: where does the time go in a Reading Tutor that listens? In: Proceedings of the Sixth International Conference on Intelligent Tutoring Systems (ITS’2002), pp. 320–329. Biarritz, France (2002)Google Scholar
  27. Murray, R.C., vanLehn, K.: Effects of dissuading unnecessary help requests while providing proactive help. In: Proceedings of the 12th International Conference on Artificial Intelligence in Education, pp. 887–889. Amsterdam (2005)Google Scholar
  28. Ramsey, F.L., Schafer, D.W.: The Statistical Sleuth: A Course in Methods of Data Analysis. Duxbury Press, Belmont, CA (1997)Google Scholar
  29. Rodrigo, M.M.T., Baker, R.S.J.d., Lagud, M.C.V., Lim, S.A.L., Macapanpan, A.F., Pascua, S.A.M.S., Santillano, J.Q., Sevilla, L.R.S., Sugay, J.O., Tep, S., Viehland, N.J.B.: Affect and usage choices in simulation problem solving environments. In: Proceedings of the 13th International Conference on Artificial Intelligence in Education, pp. 145–152. Marina del Rey, CA (2007)Google Scholar
  30. Roll, I., Baker, R.S., Aleven, V., McLaren, B.M., Koedinger, K.R.: Modeling students’ metacognitive errors in two intelligent tutoring systems. In: Proceedings of User Modeling 2005, pp. 379–388. Edinburgh, Scotland (2005)Google Scholar
  31. Roll, I., Aleven, V., McLaren, B.M., and Koedinger, K.R.: Can help seeking be tutored? Searching for the secret sauce of metacognitive tutoring. In; Proceedings of the 13th International Conference on Artificial Intelligence in Education, pp. 203–210. Marina del Rey, CA (2007)Google Scholar
  32. Rosenthal, R., Rosnow, R.: Essentials of Behavioral Research: Methods and Data Analysis. McGraw-Hill, Boston, MA (1991)Google Scholar
  33. Schofield, J.W.: Computers and Classroom Culture. Cambridge University Press, Cambridge, UK (1995)Google Scholar
  34. Strube, M.J.: Combining and comparing significance levels from nonindependent hypothesis tests. Psychol. Bull. 97, 334–341 (1985)CrossRefGoogle Scholar
  35. Tait, K., Hartley, J., Anderson, R.C.: Feedback procedures in computer-assisted arithmetic instruction. Br. J. Educ. Psychol. 43, 161–171 (1973)Google Scholar
  36. van Lehn, K., Lynch, C., Shulze, K., Shapiro, J.A., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., Wintersgill, M.: The Andes physics tutoring system: Five years of evaluations. In: Proceedings of the 12th International Conference on Artificial Intelligence in Education, pp. 678–685. Amsterdam (2005)Google Scholar
  37. Walonoski, J.A., Heffernan, N.T.: Detection and analysis of off-task gaming behavior in intelligent tutoring systems. In: Proceedings of the 8th International Conference on Intelligent Tutoring Systems, pp. 382–391. Jhongli, Taiwan (2006a)Google Scholar
  38. Walonoski, J.A., Heffernan, N.T.: Prevention of off-task gaming behavior in intelligent tutoring systems. In: Proceedings of the 8th International Conference on Intelligent Tutoring Systems, pp. 722–724. Jhongli, Taiwan (2006b)Google Scholar
  39. Wood, H., Wood, D.: Help seeking, learning, and contingent tutoring. Comp. Educ. 33, 153–169 (1999)CrossRefGoogle Scholar
  40. Yu, L., Liu, H.: Feature selection for high-dimensional data: a fast correlation-based filter solution. In: Proceedings of the International Conference on Machine Learning, pp. 856–863. Washington, DC (2003)Google Scholar

Copyright information

© Springer Science+Business Media B.V. 2008

Authors and Affiliations

  • Ryan S. J. d. Baker
    • 1
  • Albert T. Corbett
    • 1
  • Ido Roll
    • 1
  • Kenneth R. Koedinger
    • 1
  1. 1.Human-Computer Interaction InstituteCarnegie Mellon UniversityPittsburghUSA

Personalised recommendations