Advertisement

Modeling and Studying Gaming the System with Educational Data Mining

  • Ryan S. J. d. Baker
  • A. T. Corbett
  • I. Roll
  • K. R. Koedinger
  • V. Aleven
  • M. Cocea
  • A. Hershkovitz
  • A. M. J. B. de Caravalho
  • A. Mitrovic
  • M. Mathews
Chapter
Part of the Springer International Handbooks of Education book series (SIHE, volume 28)

Abstract

In this chapter, we will discuss our work to understand why students game the system. This work leverages models of student gaming, termed “detectors”, which can infer student gaming in log files of student interaction with educational software. These detectors are developed using a combination of human observation and annotation, and educational data mining. We then apply the detectors to large data sets, and analyze the detectors’ predictions, using discovery with models methods, to study the factors associated with gaming behavior. Within this chapter, we will discuss the work to develop these detectors, and what we have discovered through these analyses based on these detectors. We will discuss evidence for how gaming the system impacts learning and evidence for why students choose to game. We will also discuss attempts to address gaming the system through adaptive scaffolding.

Keywords

Tutoring System Educational Software Cognitive Tutor Interactive Learning Environment Educational Data Mining 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

References

  1. Aleven, V., McLaren, B. M., Roll, I., & Koedinger, K. R. (2006). Toward meta-cognitive tutoring: A model of help seeking with a cognitive tutor. International Journal of Artificial Intelligence in Education, 16, 101–130.Google Scholar
  2. Anderson, J. R., Corbett, A. T., Koedinger, K. R., & Pelletier, R. (1995). Cognitive tutors: Lessons learned. The Journal of the Learning Sciences, 4(2), 167–207.CrossRefGoogle Scholar
  3. Arroyo, I., Ferguson, K., Johns, J., Dragon, T., Meheranian, H., & Fisher, D. (2007). Repairing disengagement with non-invasive interventions. In J. Greer, R. Luckin, & K. Koedinger (Eds.), Proceedings of the 13th International Conference on Artificial Intelligence in Education (pp. 195–202). Amsterdam, Netherlands: Ios Press.Google Scholar
  4. Arroyo, I., & Woolf, B. (2005). Inferring learning and attitudes from a Bayesian Network of log file data. In U. Hoppe, F. Verdejo, & J. Kay (Eds.), Proceedings of the 12th International Conference on Artificial Intelligence in Education (pp. 33–40). Amsterdam, Netherlands: Ios Press.Google Scholar
  5. Baker, R. S. J. d. (2007a). Is gaming the system state-or-trait? Educational data mining through the multi-contextual application of a validated behavioral model. In R. Baker, J. Beck, B. Berendt, A. Kroner, E. Menasalvas, & S. Weibelzahl (Eds.), Complete on-line Proceedings of the Workshop on Data Mining for User Modeling at the 11th International Conference on User Modeling (pp. 76–80). Pittsburgh, PA: International Working Group on Educational Data Mining.Google Scholar
  6. Baker, R. S. J. d. (2007b). Modeling and understanding students’ off-task behavior in intelligent tutoring systems. In M. Rosson & D. Gilmore (Eds.), Proceedings of ACM CHI 2007 Conference on Human Factors in Computing Systems (pp. 1059–1068). Washington, DC: Association for Computing Machinery.Google Scholar
  7. Baker, R. S. J. d. (2010). Mining data for student models. In R. Nkmabou, R. Mizoguchi, & J. Bourdeau (Eds.), Advances in intelligent tutoring systems (pp. 341–356). Secaucus, NJ: Springer.Google Scholar
  8. Baker, R. S., Corbett, A. T., & Koedinger, K. R. (2004a). Detecting student misuse of intelligent tutoring systems. In J. Lester, R. Vicari, & F. Paraguacu (Eds.), Proceedings of the 7th International Conference on Intelligent Tutoring Systems (pp. 531–540). Heidelberg, Germany: Springer.CrossRefGoogle Scholar
  9. Baker, R. S. J. d., Corbett, A. T., Koedinger, K. R., Evenson, S. E., Roll, I., & Wagner, A. Z. (2006). Adapting to when students game an intelligent tutoring system. In M. Ikeda, K. Ashley, & T. Chan (Eds.), Proceedings of the 8th International Conference on Intelligent Tutoring Systems (pp. 392–401). Heidelberg, Germany: Springer.CrossRefGoogle Scholar
  10. Baker, R. S., Corbett, A., Koedinger, K., & Roll, I. (2005). Detecting when students game the system, across tutor subjects and classroom cohorts. In L. Ardissono, P. Brna, & A. Mitrovic (Eds.), Proceedings of the 10th International Conference on User Modeling (pp. 220–224). Heidelberg, Germany: Springer.Google Scholar
  11. Baker, R. S., Corbett, A. T., Koedinger, K. R., & Wagner, A. Z. (2004b). Off-task behavior in the cognitive tutor classroom: When students “game the system”. In E. Dykstra-Erickson & M. Tscheligi (Eds.), Proceedings of ACM CHI 2004 Conference on Human Factors in Computing Systems (pp. 383–390). Washington, DC: Association for Computing Machinery.CrossRefGoogle Scholar
  12. Baker, R. S. J. d., Corbett, A. T., Roll, I., & Koedinger, K. R. (2008). Developing a generalizable detector of when students game the system. User Modeling and User-Adapted Interaction, 18(3), 287–314.CrossRefGoogle Scholar
  13. Baker, R. S. J. d., Corbett, A. T., & Wagner, A. Z. (2006). Human classification of low-fidelity replays of student actions. In C. Heiner, R. Baker, & K. Yacef (Eds.), Proceedings of the Educational Data Mining Workshop at the 8th International Conference on Intelligent Tutoring Systems (pp. 29–36). Pittsburgh, PA: International Working Group on Educational Data Mining.Google Scholar
  14. Baker, R. S. J. d., & de Carvalho, A. M. J. A. (2008). Labeling student behavior faster and more precisely with text replays. In R. Baker, T. Barnes, & J. Beck (Eds.), Proceedings of the 1st International Conference on Educational Data Mining (pp. 38–47). Pittsburgh, PA: International Working Group on Educational Data Mining.Google Scholar
  15. Baker, R. S. J. d., de Carvalho, A. M. J. A., Raspat, J., Aleven, V., Corbett, A. T., & Koedinger, K. R. (2009). Educational software features that encourage and discourage “gaming the system”. In V. Dimitrova, R. Mizoguchi, B. du Boulay, & A. Graesser (Eds.), Proceedings of the 14th International Conference on Artificial Intelligence in Education (pp. 475–482). Amsterdam, Netherlands: Ios Press.Google Scholar
  16. Baker, R. S. J. d., D’Mello, S. K., Rodrigo, M. M. T., & Graesser, A. C. (2010). Better to be frustrated than bored: The incidence, persistence, and impact of learners’ cognitive-affective states during interactions with three different computer-based learning environments. International Journal of Human Computer Studies, 68(4), 223–241.CrossRefGoogle Scholar
  17. Baker, R. S. J. d., Mitrovic, A., & Mathews, M. (2010). Detecting gaming the system in constraint-based tutors. In P. de Bra, A. Kobsa, & D. Chin (Eds.), Proceedings of the 18th Annual Conference on User Modeling, Adaptation, and Personalization (pp. 267–278). Heidelberg, Germany: Springer.CrossRefGoogle Scholar
  18. Baker, R., Walonoski, J., Heffernan, N., Roll, I., Corbett, A., & Koedinger, K. (2008). Why students engage in “gaming the system” behavior in interactive learning environments. Journal of Interactive Learning Research, 19(2), 185–224.Google Scholar
  19. Baker, R. S. J. d., & Yacef, K. (2009). The state of educational data mining in 2009: A review and future visions. Journal of Educational Data Mining, 1(1), 3–17.Google Scholar
  20. Beal, C. R., Qu, L., & Lee, H. (2006). Classifying learner engagement through integration of multiple data sources. In Y. Gil & R. Mooney (Eds.), Proceedings of the 21st National Conference on Artificial Intelligence (pp. 2–8). Washington, DC: Association for the Advancement of Artificial Intelligence.Google Scholar
  21. Beal, C. R., Qu, L., & Lee, H. (2008). Mathematics motivation and achievement as predictors of high school students’ guessing and help-seeking with instructional software. Journal of Computer Assisted Learning, 24, 507–514.CrossRefGoogle Scholar
  22. Beck, J. (2005). Engagement tracing: Using response times to model student disengagement. In U. Hoppe, F. Verdejo, & J. Kay (Eds.), Proceedings of the 12th International Conference on Artificial Intelligence in Education (AIED 2005) (pp. 88–95). Amsterdam, Netherlands: Ios Press.Google Scholar
  23. Beck, J. E. (2006). Using learning decomposition to analyze student fluency development. In C. Heiner, R. Baker, & K. Yacef (Eds.), Proceedings of the workshop on Educational Data Mining at the 8th International Conference on Intelligent Tutoring Systems (pp. 21–28). Pittsburgh, PA: International Working Group on Educational Data Mining.Google Scholar
  24. Butler, D. L., & Winne, P. (1995). Feedback and self-regulated learning: A theoretical synthesis. Review of Educational Resarch, 65(3), 245–281.CrossRefGoogle Scholar
  25. Cheng, R., & Vassileva, J. (2005). Adaptive reward mechanism for sustainable online learning community. In U. Hoppe, F. Verdejo, & J. Kay (Eds.), Proceedings of the 12th International Conference on Artificial Intelligence in Education (pp. 152–159). Amsterdam, Netherlands: Ios Press.Google Scholar
  26. Cocea, M., Hershkovitz, A., & de Baker, R. S. J. (2009). The impact of off-task and gaming behaviors on learning: Immediate or aggregate? In V. Dimitrova, R. Mizoguchi, B. du Boulay, & A. Graesser (Eds.), Proceedings of the 14th International Conference on Artificial Intelligence in Education (pp. 507–514). Amsterdam, Netherlands: Ios Press.Google Scholar
  27. Corbett, A. T., & Anderson, J. R. (1995). Knowledge tracing: Modeling the acquisition of procedural knowledge. User Modeling and User-Adapted Interaction, 4, 253–278.CrossRefGoogle Scholar
  28. Duckworth, A. L., Peterson, C., Matthews, M. D., & Kelly, D. R. (2007). Grit: Perseverance and passion for long-term goals. Journal of Personality and Social Psychology, 92, 1087–1101.CrossRefGoogle Scholar
  29. Dweck, C. S. (2000). Self-theories: Their role in motivation, personality, and development. Philadelphia, PA: Psychology Press.Google Scholar
  30. Efron, B., & Gong, G. (1983). A leisurely look at the bootstrap, the jackknife, and cross-validation. The American Statistician, 37(1), 36–48.Google Scholar
  31. Fisher, S. L., & Ford, J. K. (1998). Differential effects of learner effort and goal orientation on two learning outcomes. Personnel Psychology, 51(2), 397–420.CrossRefGoogle Scholar
  32. Flavell, J. H. (1976). Metacognitive aspects of problem solving. In L. B. Resnick (Ed.), The nature of intelligence. Hillsdale, NJ: Lawrence Erlbaum Associates.Google Scholar
  33. Gobel, P. (2008). Student off-task behavior and motivation in the CALL classroom. International Journal of Pedagogies and Learning, 4(4), 4–18.CrossRefGoogle Scholar
  34. Gong, Y., Beck, J., Heffernan, N. T., & Forbes-Summers, E. (2010). The fine-grained impact of gaming (?) on learning. In V. Aleven, J. Kay, & J. Mostow (Eds.), Proceedings of the 10th International Conference on Intelligent Tutoring Systems (pp. 194–203). Heidelberg, Germany: Springer.CrossRefGoogle Scholar
  35. Hacker, D. J. (1999). Definitions and empirical foundations. In D. J. Hacker, J. Dunlosky, & A. C. Graesser (Eds.), Metacognition in educational theory and practice (pp. 1–24). Mahway, NJ: Erlbaum.Google Scholar
  36. Johns, J., & Woolf, B. (2006). A dynamic mixture model to detect student motivation and proficiency. In Y. Gil & R. Mooney (Eds.), Proceedings of the 21st National Conference on Artificial Intelligence (AAAI-06) (pp. 163–168). Washington, DC: Association for the Advancement of Artificial Intelligence.Google Scholar
  37. Karweit, N., & Slavin, R. E. (1982). Time-on-task: Issues of timing, sampling, and definition. Journal of Experimental Psychology, 74(6), 844–851.Google Scholar
  38. Koedinger, K. R., de Baker, R. S. J., Cunningham, K., Skogsholm, A., Leber, B., & Stamper, J. (2010). A data repository for the EDM community: The PSLC DataShop. In C. Romero, S. Ventura, M. Pechenizkiy, & R. S. J. de Baker (Eds.), Handbook of educational data mining (pp. 43–56). Boca Raton, FL: CRC Press.CrossRefGoogle Scholar
  39. Koedinger, K. R., & Corbett, A. T. (2006). Cognitive tutors: Technology bringing learning sciences to the classroom. In R. K. Sawyer (Ed.), The Cambridge handbook of the learning sciences. New York: Cambridge University Press.Google Scholar
  40. Magnussen, R., & Misfeldt, M. (2004). Player transformation of educational multiplayer games. In M. Sicart & J. Smith (Eds.), Proceedings of other players. Copenhagen, Denmark: IT University of Copenhagen.Google Scholar
  41. Maris, E. (1995). Psychometric latent response models. Psychometrika, 60(4), 523–547.CrossRefGoogle Scholar
  42. Martínez Mirón, E. A., du Boulay, B., & Luckin, R. (2004). Goal achievement orientation in the design of an ILE. In C. Frasson & K. Porayska-Pomsta (Eds.), Proceedings of the ITS2004 Workshop on Social and Emotional Intelligence in Learning Environments (pp. 72–78). Maceio, Brazil: Federal University of Alagoas.Google Scholar
  43. Midgley, C., & Urdan, T. (2002). Academic self-handicapping and achievement goals: A further examination. Contemporary Educational Psychology, 26(1), 61–75.CrossRefGoogle Scholar
  44. Mierswa, I., Wurst, M., Klinkenberg, R., Scholz, M., & Euler, T. (2006). Yale: Rapid prototyping for complex data mining tasks. In L. Ungar, M. Craven, & D. Gunopulos (Eds.), Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (pp. 935–940). Washington, DC: Association for Computing Machinery.CrossRefGoogle Scholar
  45. Muldner, K., Burleson, W., Van de Sande, B., & VanLehn, K. (2011). An analysis of students’ gaming behaviors in an intelligent tutoring system: Predictors and impact. User Modeling and User-Adapted Interaction, 21(1–2), 99–135.CrossRefGoogle Scholar
  46. Murray, R. C., & VanLehn, K. (2005). Effects of dissuading unnecessary help requests while providing proactive help. In U. Hoppe, F. Verdejo, & J. Kay (Eds.), Proceedings of the 12th International Conference on Artificial Intelligence in Education (pp. 887–889). Amsterdam, Netherlands: Ios Press.Google Scholar
  47. Nelson-Le Gall, S. (1985). Help-seeking behavior in learning. Review of Research in Education, 12, 55–90.Google Scholar
  48. Quinlan, J. R. (1993). C4.5: Programs for machine learning. San Francisco, CA: Morgan Kaufmann.Google Scholar
  49. Ramsey, F. L., & Schafer, D. W. (1997). The statistical sleuth: A course in methods of data analysis. Belmont, CA: Duxbury Press.Google Scholar
  50. Rodrigo, M. M. T., Rebolledo-Mendez, G., de Baker, R. S. J., du Boulay, B., Sugay, J. O., & Lim, S. A. L. (2008). The effects of motivational modeling on affect in an intelligent tutoring system. In Y. Yano (Ed.), Proceedings of 16th International Conference on Computers in Education. Jhongli, Taiwan: Asia-Pacific Society for Computers in Education.Google Scholar
  51. Roll, I., Aleven, V., McLaren, B. M., & Koedinger, K. R. (2007). Designing for metacognition—Applying cognitive tutor principles to the tutoring of help seeking. Metacognition and Learning, 2(2), 125–140.CrossRefGoogle Scholar
  52. Roll, I., Aleven, V., McLaren, B. M., & Koedinger, K. R. (2011). Improving students’ help-seeking skills using metacognitive feedback in an intelligent tutoring system. Learning and Instruction, 21, 267–280.CrossRefGoogle Scholar
  53. Romero, C., & Ventura, S. (2010). Educational data mining: A review of the state-of-the-art. IEEE Transactions of Systems, Man, and Cybernetics, Part C: Applications and Reviews, 40(6), 601–618.CrossRefGoogle Scholar
  54. Sao Pedro, M. A., de Baker, R. S. J., Montalvo, O., Nakama, A., & Gobert, J. D. (2010). Using text replay tagging to produce detectors of systematic experimentation behavior patterns. In R. Baker, A. Merceron, & P. Pavlik (Eds.), Proceedings of the 3rd International Conference on Educational Data Mining (pp. 181–190). Pittsburgh, PA: International Working Group on Educational Data Mining.Google Scholar
  55. Shih, B., Koedinger, K., & Scheines, R. (2008). A response time model for bottom-out hints as worked examples. In R. Baker, T. Barnes, & J. Beck (Eds.), Proceedings of the 1st International Conference on Educational Data Mining (pp. 117–126). Pittsburgh, PA: International Working Group on Educational Data Mining.Google Scholar
  56. Walonoski, J. A., & Heffernan, N. T. (2006a). Detection and analysis of off-task gaming behavior in intelligent tutoring systems. In M. Ikeda, K. Ashley, & T. Chan (Eds.), Proceedings of the 8th International Conference on Intelligent Tutoring Systems (pp. 382–391). Heidelberg, Germany: Springer.CrossRefGoogle Scholar
  57. Walonoski, J. A., & Heffernan, N. T. (2006b). Prevention of off-task gaming behavior in intelligent tutoring systems. In M. Ikeda, K. Ashley, & T. Chan (Eds.), Proceedings of the 8th International Conference on Intelligent Tutoring Systems (pp. 722–724). Heidelberg, Germany: Springer.CrossRefGoogle Scholar
  58. Winne, P., & Hadwin, P. (1998). Studying as self-regulated learning. In D. J. Hacker, J. Dunlosky, & A. C. Graesser (Eds.), Metacognition in educational theory and practice (pp. 277–304). Mahway, NJ: Erlbaum.Google Scholar
  59. Witten, I. H., & Frank, E. (2005). Data mining: Practical machine learning tools and techniques. San Francisco, CA: Morgan Kaufmann.Google Scholar
  60. Yu, L., & Liu, H. (2003). Feature selection for high-dimensional data: a fast correlation-based filter solution. In T. Fawcett & N. Mishra (Eds.), Proceedings of the 20th International Conference on Machine Learning (pp. 856–863). Washington, DC: Association for the Advancement of Artificial Intelligence.Google Scholar
  61. Zimmerman, B. J. (2000). Attaining self-regulation: A social cognitive perspective. In M. Boekarts, P. R. Pintrich, & M. Zeidner (Eds.), Handbook of regulation (pp. 13–39). Amsterdam, Netherlands: Elsevier.CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2013

Authors and Affiliations

  • Ryan S. J. d. Baker
    • 1
  • A. T. Corbett
    • 2
    • 3
  • I. Roll
    • 4
    • 5
  • K. R. Koedinger
    • 5
    • 6
  • V. Aleven
    • 7
    • 5
  • M. Cocea
    • 8
  • A. Hershkovitz
    • 1
  • A. M. J. B. de Caravalho
    • 9
  • A. Mitrovic
    • 10
  • M. Mathews
    • 10
  1. 1.Columbia University Teachers CollegeNew YorkUSA
  2. 2.Human-Computer Interaction InstituteCarnegie Mellon UniversityPittsburghUSA
  3. 3.Pittsburgh Advanced Cognitive Tutor CenterCarnegie Mellon UniversityPittsburghUSA
  4. 4.Carl Wieman Science Education InitiativeUniversity of British ColumbiaVancouverCanada
  5. 5.Pittsburgh Science of Learning CenterPittsburghUSA
  6. 6.Human-Computer Interaction and PsychologyCarnegie Mellon UniversityPittsburghUSA
  7. 7.Human-Computer Interaction InstituteCarnegie Mellon UniversityPittsburghUSA
  8. 8.School of ComputingUniversity of PortsmouthPortsmouth, HampshireUK
  9. 9.Human-Computer Interaction InstituteCMU Carnegie Mellon UniversityPittsburghUSA
  10. 10.Intelligent Computer Tutoring Group (ICTG), Department of Computer Science and Software EngineeringUniversity of CanterburyChristchurchNew Zealand

Personalised recommendations