Skip to main content
SpringerLink
Log in

Modeling and Studying Gaming the System with Educational Data Mining

  • Chapter
  • First Online:
Book cover International Handbook of Metacognition and Learning Technologies

Part of the book series: Springer International Handbooks of Education ((SIHE,volume 28))

Abstract

In this chapter, we will discuss our work to understand why students game the system. This work leverages models of student gaming, termed “detectors”, which can infer student gaming in log files of student interaction with educational software. These detectors are developed using a combination of human observation and annotation, and educational data mining. We then apply the detectors to large data sets, and analyze the detectors’ predictions, using discovery with models methods, to study the factors associated with gaming behavior. Within this chapter, we will discuss the work to develop these detectors, and what we have discovered through these analyses based on these detectors. We will discuss evidence for how gaming the system impacts learning and evidence for why students choose to game. We will also discuss attempts to address gaming the system through adaptive scaffolding.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Institutional subscriptions

References

  • Aleven, V., McLaren, B. M., Roll, I., & Koedinger, K. R. (2006). Toward meta-cognitive tutoring: A model of help seeking with a cognitive tutor. International Journal of Artificial Intelligence in Education, 16, 101–130.

    Google Scholar 

  • Anderson, J. R., Corbett, A. T., Koedinger, K. R., & Pelletier, R. (1995). Cognitive tutors: Lessons learned. The Journal of the Learning Sciences, 4(2), 167–207.

    Article  Google Scholar 

  • Arroyo, I., Ferguson, K., Johns, J., Dragon, T., Meheranian, H., & Fisher, D. (2007). Repairing disengagement with non-invasive interventions. In J. Greer, R. Luckin, & K. Koedinger (Eds.), Proceedings of the 13th International Conference on Artificial Intelligence in Education (pp. 195–202). Amsterdam, Netherlands: Ios Press.

    Google Scholar 

  • Arroyo, I., & Woolf, B. (2005). Inferring learning and attitudes from a Bayesian Network of log file data. In U. Hoppe, F. Verdejo, & J. Kay (Eds.), Proceedings of the 12th International Conference on Artificial Intelligence in Education (pp. 33–40). Amsterdam, Netherlands: Ios Press.

    Google Scholar 

  • Baker, R. S. J. d. (2007a). Is gaming the system state-or-trait? Educational data mining through the multi-contextual application of a validated behavioral model. In R. Baker, J. Beck, B. Berendt, A. Kroner, E. Menasalvas, & S. Weibelzahl (Eds.), Complete on-line Proceedings of the Workshop on Data Mining for User Modeling at the 11th International Conference on User Modeling (pp. 76–80). Pittsburgh, PA: International Working Group on Educational Data Mining.

    Google Scholar 

  • Baker, R. S. J. d. (2007b). Modeling and understanding students’ off-task behavior in intelligent tutoring systems. In M. Rosson & D. Gilmore (Eds.), Proceedings of ACM CHI 2007 Conference on Human Factors in Computing Systems (pp. 1059–1068). Washington, DC: Association for Computing Machinery.

    Google Scholar 

  • Baker, R. S. J. d. (2010). Mining data for student models. In R. Nkmabou, R. Mizoguchi, & J. Bourdeau (Eds.), Advances in intelligent tutoring systems (pp. 341–356). Secaucus, NJ: Springer.

    Google Scholar 

  • Baker, R. S., Corbett, A. T., & Koedinger, K. R. (2004a). Detecting student misuse of intelligent tutoring systems. In J. Lester, R. Vicari, & F. Paraguacu (Eds.), Proceedings of the 7th International Conference on Intelligent Tutoring Systems (pp. 531–540). Heidelberg, Germany: Springer.

    Chapter  Google Scholar 

  • Baker, R. S. J. d., Corbett, A. T., Koedinger, K. R., Evenson, S. E., Roll, I., & Wagner, A. Z. (2006). Adapting to when students game an intelligent tutoring system. In M. Ikeda, K. Ashley, & T. Chan (Eds.), Proceedings of the 8th International Conference on Intelligent Tutoring Systems (pp. 392–401). Heidelberg, Germany: Springer.

    Chapter  Google Scholar 

  • Baker, R. S., Corbett, A., Koedinger, K., & Roll, I. (2005). Detecting when students game the system, across tutor subjects and classroom cohorts. In L. Ardissono, P. Brna, & A. Mitrovic (Eds.), Proceedings of the 10th International Conference on User Modeling (pp. 220–224). Heidelberg, Germany: Springer.

    Google Scholar 

  • Baker, R. S., Corbett, A. T., Koedinger, K. R., & Wagner, A. Z. (2004b). Off-task behavior in the cognitive tutor classroom: When students “game the system”. In E. Dykstra-Erickson & M. Tscheligi (Eds.), Proceedings of ACM CHI 2004 Conference on Human Factors in Computing Systems (pp. 383–390). Washington, DC: Association for Computing Machinery.

    Chapter  Google Scholar 

  • Baker, R. S. J. d., Corbett, A. T., Roll, I., & Koedinger, K. R. (2008). Developing a generalizable detector of when students game the system. User Modeling and User-Adapted Interaction, 18(3), 287–314.

    Article  Google Scholar 

  • Baker, R. S. J. d., Corbett, A. T., & Wagner, A. Z. (2006). Human classification of low-fidelity replays of student actions. In C. Heiner, R. Baker, & K. Yacef (Eds.), Proceedings of the Educational Data Mining Workshop at the 8th International Conference on Intelligent Tutoring Systems (pp. 29–36). Pittsburgh, PA: International Working Group on Educational Data Mining.

    Google Scholar 

  • Baker, R. S. J. d., & de Carvalho, A. M. J. A. (2008). Labeling student behavior faster and more precisely with text replays. In R. Baker, T. Barnes, & J. Beck (Eds.), Proceedings of the 1st International Conference on Educational Data Mining (pp. 38–47). Pittsburgh, PA: International Working Group on Educational Data Mining.

    Google Scholar 

  • Baker, R. S. J. d., de Carvalho, A. M. J. A., Raspat, J., Aleven, V., Corbett, A. T., & Koedinger, K. R. (2009). Educational software features that encourage and discourage “gaming the system”. In V. Dimitrova, R. Mizoguchi, B. du Boulay, & A. Graesser (Eds.), Proceedings of the 14th International Conference on Artificial Intelligence in Education (pp. 475–482). Amsterdam, Netherlands: Ios Press.

    Google Scholar 

  • Baker, R. S. J. d., D’Mello, S. K., Rodrigo, M. M. T., & Graesser, A. C. (2010). Better to be frustrated than bored: The incidence, persistence, and impact of learners’ cognitive-affective states during interactions with three different computer-based learning environments. International Journal of Human Computer Studies, 68(4), 223–241.

    Article  Google Scholar 

  • Baker, R. S. J. d., Mitrovic, A., & Mathews, M. (2010). Detecting gaming the system in constraint-based tutors. In P. de Bra, A. Kobsa, & D. Chin (Eds.), Proceedings of the 18th Annual Conference on User Modeling, Adaptation, and Personalization (pp. 267–278). Heidelberg, Germany: Springer.

    Chapter  Google Scholar 

  • Baker, R., Walonoski, J., Heffernan, N., Roll, I., Corbett, A., & Koedinger, K. (2008). Why students engage in “gaming the system” behavior in interactive learning environments. Journal of Interactive Learning Research, 19(2), 185–224.

    Google Scholar 

  • Baker, R. S. J. d., & Yacef, K. (2009). The state of educational data mining in 2009: A review and future visions. Journal of Educational Data Mining, 1(1), 3–17.

    Google Scholar 

  • Beal, C. R., Qu, L., & Lee, H. (2006). Classifying learner engagement through integration of multiple data sources. In Y. Gil & R. Mooney (Eds.), Proceedings of the 21st National Conference on Artificial Intelligence (pp. 2–8). Washington, DC: Association for the Advancement of Artificial Intelligence.

    Google Scholar 

  • Beal, C. R., Qu, L., & Lee, H. (2008). Mathematics motivation and achievement as predictors of high school students’ guessing and help-seeking with instructional software. Journal of Computer Assisted Learning, 24, 507–514.

    Article  Google Scholar 

  • Beck, J. (2005). Engagement tracing: Using response times to model student disengagement. In U. Hoppe, F. Verdejo, & J. Kay (Eds.), Proceedings of the 12th International Conference on Artificial Intelligence in Education (AIED 2005) (pp. 88–95). Amsterdam, Netherlands: Ios Press.

    Google Scholar 

  • Beck, J. E. (2006). Using learning decomposition to analyze student fluency development. In C. Heiner, R. Baker, & K. Yacef (Eds.), Proceedings of the workshop on Educational Data Mining at the 8th International Conference on Intelligent Tutoring Systems (pp. 21–28). Pittsburgh, PA: International Working Group on Educational Data Mining.

    Google Scholar 

  • Butler, D. L., & Winne, P. (1995). Feedback and self-regulated learning: A theoretical synthesis. Review of Educational Resarch, 65(3), 245–281.

    Article  Google Scholar 

  • Cheng, R., & Vassileva, J. (2005). Adaptive reward mechanism for sustainable online learning community. In U. Hoppe, F. Verdejo, & J. Kay (Eds.), Proceedings of the 12th International Conference on Artificial Intelligence in Education (pp. 152–159). Amsterdam, Netherlands: Ios Press.

    Google Scholar 

  • Cocea, M., Hershkovitz, A., & de Baker, R. S. J. (2009). The impact of off-task and gaming behaviors on learning: Immediate or aggregate? In V. Dimitrova, R. Mizoguchi, B. du Boulay, & A. Graesser (Eds.), Proceedings of the 14th International Conference on Artificial Intelligence in Education (pp. 507–514). Amsterdam, Netherlands: Ios Press.

    Google Scholar 

  • Corbett, A. T., & Anderson, J. R. (1995). Knowledge tracing: Modeling the acquisition of procedural knowledge. User Modeling and User-Adapted Interaction, 4, 253–278.

    Article  Google Scholar 

  • Duckworth, A. L., Peterson, C., Matthews, M. D., & Kelly, D. R. (2007). Grit: Perseverance and passion for long-term goals. Journal of Personality and Social Psychology, 92, 1087–1101.

    Article  Google Scholar 

  • Dweck, C. S. (2000). Self-theories: Their role in motivation, personality, and development. Philadelphia, PA: Psychology Press.

    Google Scholar 

  • Efron, B., & Gong, G. (1983). A leisurely look at the bootstrap, the jackknife, and cross-validation. The American Statistician, 37(1), 36–48.

    Google Scholar 

  • Fisher, S. L., & Ford, J. K. (1998). Differential effects of learner effort and goal orientation on two learning outcomes. Personnel Psychology, 51(2), 397–420.

    Article  Google Scholar 

  • Flavell, J. H. (1976). Metacognitive aspects of problem solving. In L. B. Resnick (Ed.), The nature of intelligence. Hillsdale, NJ: Lawrence Erlbaum Associates.

    Google Scholar 

  • Gobel, P. (2008). Student off-task behavior and motivation in the CALL classroom. International Journal of Pedagogies and Learning, 4(4), 4–18.

    Article  Google Scholar 

  • Gong, Y., Beck, J., Heffernan, N. T., & Forbes-Summers, E. (2010). The fine-grained impact of gaming (?) on learning. In V. Aleven, J. Kay, & J. Mostow (Eds.), Proceedings of the 10th International Conference on Intelligent Tutoring Systems (pp. 194–203). Heidelberg, Germany: Springer.

    Chapter  Google Scholar 

  • Hacker, D. J. (1999). Definitions and empirical foundations. In D. J. Hacker, J. Dunlosky, & A. C. Graesser (Eds.), Metacognition in educational theory and practice (pp. 1–24). Mahway, NJ: Erlbaum.

    Google Scholar 

  • Johns, J., & Woolf, B. (2006). A dynamic mixture model to detect student motivation and proficiency. In Y. Gil & R. Mooney (Eds.), Proceedings of the 21st National Conference on Artificial Intelligence (AAAI-06) (pp. 163–168). Washington, DC: Association for the Advancement of Artificial Intelligence.

    Google Scholar 

  • Karweit, N., & Slavin, R. E. (1982). Time-on-task: Issues of timing, sampling, and definition. Journal of Experimental Psychology, 74(6), 844–851.

    Google Scholar 

  • Koedinger, K. R., de Baker, R. S. J., Cunningham, K., Skogsholm, A., Leber, B., & Stamper, J. (2010). A data repository for the EDM community: The PSLC DataShop. In C. Romero, S. Ventura, M. Pechenizkiy, & R. S. J. de Baker (Eds.), Handbook of educational data mining (pp. 43–56). Boca Raton, FL: CRC Press.

    Chapter  Google Scholar 

  • Koedinger, K. R., & Corbett, A. T. (2006). Cognitive tutors: Technology bringing learning sciences to the classroom. In R. K. Sawyer (Ed.), The Cambridge handbook of the learning sciences. New York: Cambridge University Press.

    Google Scholar 

  • Magnussen, R., & Misfeldt, M. (2004). Player transformation of educational multiplayer games. In M. Sicart & J. Smith (Eds.), Proceedings of other players. Copenhagen, Denmark: IT University of Copenhagen.

    Google Scholar 

  • Maris, E. (1995). Psychometric latent response models. Psychometrika, 60(4), 523–547.

    Article  Google Scholar 

  • Martínez Mirón, E. A., du Boulay, B., & Luckin, R. (2004). Goal achievement orientation in the design of an ILE. In C. Frasson & K. Porayska-Pomsta (Eds.), Proceedings of the ITS2004 Workshop on Social and Emotional Intelligence in Learning Environments (pp. 72–78). Maceio, Brazil: Federal University of Alagoas.

    Google Scholar 

  • Midgley, C., & Urdan, T. (2002). Academic self-handicapping and achievement goals: A further examination. Contemporary Educational Psychology, 26(1), 61–75.

    Article  Google Scholar 

  • Mierswa, I., Wurst, M., Klinkenberg, R., Scholz, M., & Euler, T. (2006). Yale: Rapid prototyping for complex data mining tasks. In L. Ungar, M. Craven, & D. Gunopulos (Eds.), Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (pp. 935–940). Washington, DC: Association for Computing Machinery.

    Chapter  Google Scholar 

  • Muldner, K., Burleson, W., Van de Sande, B., & VanLehn, K. (2011). An analysis of students’ gaming behaviors in an intelligent tutoring system: Predictors and impact. User Modeling and User-Adapted Interaction, 21(1–2), 99–135.

    Article  Google Scholar 

  • Murray, R. C., & VanLehn, K. (2005). Effects of dissuading unnecessary help requests while providing proactive help. In U. Hoppe, F. Verdejo, & J. Kay (Eds.), Proceedings of the 12th International Conference on Artificial Intelligence in Education (pp. 887–889). Amsterdam, Netherlands: Ios Press.

    Google Scholar 

  • Nelson-Le Gall, S. (1985). Help-seeking behavior in learning. Review of Research in Education, 12, 55–90.

    Google Scholar 

  • Quinlan, J. R. (1993). C4.5: Programs for machine learning. San Francisco, CA: Morgan Kaufmann.

    Google Scholar 

  • Ramsey, F. L., & Schafer, D. W. (1997). The statistical sleuth: A course in methods of data analysis. Belmont, CA: Duxbury Press.

    Google Scholar 

  • Rodrigo, M. M. T., Rebolledo-Mendez, G., de Baker, R. S. J., du Boulay, B., Sugay, J. O., & Lim, S. A. L. (2008). The effects of motivational modeling on affect in an intelligent tutoring system. In Y. Yano (Ed.), Proceedings of 16th International Conference on Computers in Education. Jhongli, Taiwan: Asia-Pacific Society for Computers in Education.

    Google Scholar 

  • Roll, I., Aleven, V., McLaren, B. M., & Koedinger, K. R. (2007). Designing for metacognition—Applying cognitive tutor principles to the tutoring of help seeking. Metacognition and Learning, 2(2), 125–140.

    Article  Google Scholar 

  • Roll, I., Aleven, V., McLaren, B. M., & Koedinger, K. R. (2011). Improving students’ help-seeking skills using metacognitive feedback in an intelligent tutoring system. Learning and Instruction, 21, 267–280.

    Article  Google Scholar 

  • Romero, C., & Ventura, S. (2010). Educational data mining: A review of the state-of-the-art. IEEE Transactions of Systems, Man, and Cybernetics, Part C: Applications and Reviews, 40(6), 601–618.

    Article  Google Scholar 

  • Sao Pedro, M. A., de Baker, R. S. J., Montalvo, O., Nakama, A., & Gobert, J. D. (2010). Using text replay tagging to produce detectors of systematic experimentation behavior patterns. In R. Baker, A. Merceron, & P. Pavlik (Eds.), Proceedings of the 3rd International Conference on Educational Data Mining (pp. 181–190). Pittsburgh, PA: International Working Group on Educational Data Mining.

    Google Scholar 

  • Shih, B., Koedinger, K., & Scheines, R. (2008). A response time model for bottom-out hints as worked examples. In R. Baker, T. Barnes, & J. Beck (Eds.), Proceedings of the 1st International Conference on Educational Data Mining (pp. 117–126). Pittsburgh, PA: International Working Group on Educational Data Mining.

    Google Scholar 

  • Walonoski, J. A., & Heffernan, N. T. (2006a). Detection and analysis of off-task gaming behavior in intelligent tutoring systems. In M. Ikeda, K. Ashley, & T. Chan (Eds.), Proceedings of the 8th International Conference on Intelligent Tutoring Systems (pp. 382–391). Heidelberg, Germany: Springer.

    Chapter  Google Scholar 

  • Walonoski, J. A., & Heffernan, N. T. (2006b). Prevention of off-task gaming behavior in intelligent tutoring systems. In M. Ikeda, K. Ashley, & T. Chan (Eds.), Proceedings of the 8th International Conference on Intelligent Tutoring Systems (pp. 722–724). Heidelberg, Germany: Springer.

    Chapter  Google Scholar 

  • Winne, P., & Hadwin, P. (1998). Studying as self-regulated learning. In D. J. Hacker, J. Dunlosky, & A. C. Graesser (Eds.), Metacognition in educational theory and practice (pp. 277–304). Mahway, NJ: Erlbaum.

    Google Scholar 

  • Witten, I. H., & Frank, E. (2005). Data mining: Practical machine learning tools and techniques. San Francisco, CA: Morgan Kaufmann.

    Google Scholar 

  • Yu, L., & Liu, H. (2003). Feature selection for high-dimensional data: a fast correlation-based filter solution. In T. Fawcett & N. Mishra (Eds.), Proceedings of the 20th International Conference on Machine Learning (pp. 856–863). Washington, DC: Association for the Advancement of Artificial Intelligence.

    Google Scholar 

  • Zimmerman, B. J. (2000). Attaining self-regulation: A social cognitive perspective. In M. Boekarts, P. R. Pintrich, & M. Zeidner (Eds.), Handbook of regulation (pp. 13–39). Amsterdam, Netherlands: Elsevier.

    Chapter  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Columbia University Teachers College, New York, NY, USA

    Ryan S. J. d. Baker & A. Hershkovitz

  2. Human-Computer Interaction Institute, Carnegie Mellon University, Pittsburgh, PA, USA

    A. T. Corbett

  3. Pittsburgh Advanced Cognitive Tutor Center, Carnegie Mellon University, Pittsburgh, PA, USA

    A. T. Corbett

  4. Carl Wieman Science Education Initiative, University of British Columbia, Vancouver, BC, Canada

    I. Roll

  5. Pittsburgh Science of Learning Center, Pittsburgh, PA, USA

    I. Roll, K. R. Koedinger & V. Aleven

  6. Human-Computer Interaction and Psychology, Carnegie Mellon University, Pittsburgh, PA, USA

    K. R. Koedinger

  7. Human-Computer Interaction Institute, Carnegie Mellon University, 5000 Forbes Avenue, Pittsburgh, PA, 15213, USA

    V. Aleven

  8. School of Computing, University of Portsmouth, Portsmouth, Hampshire, UK

    M. Cocea

  9. Human-Computer Interaction Institute, CMU Carnegie Mellon University, Pittsburgh, PA, 15213, USA

    A. M. J. B. de Caravalho

  10. Intelligent Computer Tutoring Group (ICTG), Department of Computer Science and Software Engineering, University of Canterbury, Christchurch, New Zealand

    A. Mitrovic & M. Mathews

Authors
  1. Ryan S. J. d. Baker
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. A. T. Corbett
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. I. Roll
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. K. R. Koedinger
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. V. Aleven
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. M. Cocea
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. A. Hershkovitz
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. A. M. J. B. de Caravalho
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. A. Mitrovic
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. M. Mathews
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ryan S. J. d. Baker .

Editor information

Editors and Affiliations

  1. Department of Educational and Counseling, McGill University Department of Educational and Counseling, Montreal, Québec, Canada

    Roger Azevedo

  2. Human-Computer Interaction Inst., Carnegie Mellon University Human-Computer Interaction Inst., Pittsburgh, Pennsylvania, USA

    Vincent Aleven

Rights and permissions

Reprints and permissions

Copyright information

© 2013 Springer Science+Business Media New York

About this chapter

Cite this chapter

Baker, R.S.J.d. et al. (2013). Modeling and Studying Gaming the System with Educational Data Mining. In: Azevedo, R., Aleven, V. (eds) International Handbook of Metacognition and Learning Technologies. Springer International Handbooks of Education, vol 28. Springer, New York, NY. https://doi.org/10.1007/978-1-4419-5546-3_7

Download citation

Publish with us

Policies and ethics

Search

Navigation