Abstract
While the richness of data from games holds promise for making inferences about players’ knowledge, skills, and attributes (KSAs), standard methods for scoring and analysis do not exist. A key to serious game analytics that measure player KSAs is the identification of player actions that can serve as evidence in scoring models. While game-based assessments may be designed with hypotheses about this evidence, the open nature of game play requires exploration of records of player actions to understand the data obtained and to generate new hypotheses. This chapter demonstrates the use of the 4R’s of Exploratory Data Analysis (EDA): revelation, resistance, re-expression, and residuals to gain close familiarity with data, avoid being fooled, and uncover unexpected patterns. The interactive and iterative nature of EDA allows for the generation of hypotheses about the processes that generated the observed data. Through this framework, possible evidence pieces emerge and the chapter concludes with an explanation of how these can be combined in a measurement model using Bayesian Networks.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Almond, R. G., & Mislevy, R. J. (1999). Graphical models and computerized adaptive testing. Applied Psychological Measurement, 23, 223–237.
Baker, R. S. J. D., & Yacef, K. (2009). The state of educational data mining in 2009: A review and future visions. Journal of Educational Data Mining, 1, 3–16.
Behrens, J. T. (1997). Principles and procedures of exploratory data analysis. Psychological Methods, 2(2), 131–160.
Behrens, J. T., DiCerbo, K. E., Yel, N., & Levy, R. (2012). Exploratory data analysis. In I. B. Weiner, J. A. Schinka, & W. F. Velicer (Eds.), Handbook of psychology: Research methods in psychology (2nd ed., pp. 34–70). New York: Wiley.
Box, G. E. (1976). Science and statistics. Journal of the American Statistical Association, 71, 791–799.
Gee, J. P. (2003). What video games have to teach us about learning and literacy. New York: Palgrave Macmillan.
Gelman, A., & Hill, J. (2007). Data analysis using regression and multilevel/ hierarchical models. New York: Cambridge University Press.
Heritage, M. (2008). Learning progressions: Supporting instruction and formative assessment. Washington, DC: Council of Chief State School Officers.
Hoaglin, D. C., Mosteller, F., & Tukey, J. W. (1983). Understanding robust and exploratory data analysis. Hoboken, NJ: Wiley.
Klopfer, E., Osterweil, S., & Salen, K. (2009). Moving learning games forward: Obstacles, opportunities, and openness. Cambridge, MA: The Education Arcade.
Koenig, A. D., Lee, J. J., Iseli, M., & Wainess, R. (2010). A conceptual framework for assessing performance in games and simulation. (CRESST Report 771). Los Angeles, CA: University of California, National Center for Research on Evaluation, Standards, and Student Testing (CRESST).
Levy, R. (2006). Posterior predictive model checking for multidimensionality in item response theory and Bayesian networks. Doctoral dissertation, University of Maryland at College Park.
Mallows, C. L. (1983). Data description. In G. E. P. Box, T. Leonard, & C.-F. Wu (Eds.), Scientific inference, data analysis, and robustness (pp. 135–151). New York: Academic Press.
Mislevy, R. J., Almond, R. G., Yan, D., & Steinberg, L. S. (1999). Bayes nets in educational assessment: Where the numbers come from. In Proceedings of the fifteenth conference on uncertainty in artificial intelligence (pp. 437–446). San Francisco: Morgan Kaufmann.
Mislevy, R. J., Behrens, J. T., DiCerbo, K. E., Frezzo, D. C., & West, P. (2012). Three things game designers need to know about assessment: Evidence-centered design for game-based assessment. In D. Ifenthaler, D. Eseryel, & X. Ge (Eds.), Assessment in game-based learning: Foundations, innovations, and perspectives (pp. 59–84). New York: Springer.
Mislevy, R. J., Corrigan, S., Oranje, A., Dicerbo, K., John, M., Bauer, M. I., et al. (2014). Psychometric considerations in game-based assessment. New York: Institute of Play.
Mislevy, R. J., & Gitomer, D. H. (1996). The role of probability-based inference in an intelligent tutoring system. User-Modeling and User-Adapted Interaction, 5, 253–282.
Mislevy, R. J., Steinberg, L. S., & Almond, R. G. (2002). Design and analysis in task-based language assessment. Language Testing, 19(4), 477–496. doi:10.1191/0265532202lt241oa.
NGSS Lead States. (2013). Next generation science standards: For states, by states. Washington, DC: The National Academies Press.
Osborne, J. W. (2002). Notes on the use of data transformations. Practical Assessment, Research & Evaluation, 8 (6). Retrieved from http://pareonline.net/getvn.asp?v=8&n=6.
Rodgers, J. L. (2010). The epistemology of mathematical and statistical modeling: A quiet methodological revolution. American Psychologist, 65, 1–12.
Schum, D. (1994). The evidential foundations of probabilistic reasoning. New York: Wiley.
Shute, V. J. (2011). Stealth assessment in computer-based games to support learning. In S. Tobias & J. D. Fletcher (Eds.), Computer games and instruction (pp. 503–524). Charlotte, NC: Information Age.
Steinkuehler, C. A. (2004). Learning in massively multiplayer online games. In Y. B. Kafai, W. A. Sandoval, N. Enyedy, A. S. Nixon, & F. Herrera (Eds.), Proceedings of the sixth international conference of the learning sciences (pp. 521–528). Mahwah, NJ: Erlbaum.
Tukey, J. W. (1977). Exploratory data analysis. Reading, MA: Addison-Wesley.
Tukey, J. W. (1986). Data analysis, computation and mathematics. In L. V. Jones (Ed.), The collected works of John W: Vol. IV. Philosophy and principles of data analysis: 1965–1986 (pp. 753–775). Pacific Grove, CA: Wadsworth. Original work published 1972.
VanLehn, K. (2008). Intelligent tutoring systems for continuous, embedded assessment. In C. Dwyer (Ed.), The future of assessment: Shaping teaching and learning (pp. 113–138). Mahwah, NJ: Erlbaum.
von Davier, M. (2005). A class of models for cognitive diagnosis. Research Report RR-05-17. Princeton, NJ: ETS.
Williamson, D., Mislevy, R. J., & Almond, R. G. (2000). Model criticism of Bayesian networks with latent variables. In C. Boutilier & M. Goldszmidt (Eds.), Uncertainty in artificial intelligence (pp. 634–643). San Francisco: Morgan Kaufmann.
Yen, W. M., & Fitzpatrick, A. R. (2006). Item response theory. In R. Brennan (Ed.), Educational measurement (3rd ed., pp. 111–153). Portsmouth, NH: Praeger/Greenwood.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2015 Springer International Publishing Switzerland
About this chapter
Cite this chapter
DiCerbo, K.E. et al. (2015). An Application of Exploratory Data Analysis in the Development of Game-Based Assessments. In: Loh, C., Sheng, Y., Ifenthaler, D. (eds) Serious Games Analytics. Advances in Game-Based Learning. Springer, Cham. https://doi.org/10.1007/978-3-319-05834-4_14
Download citation
DOI: https://doi.org/10.1007/978-3-319-05834-4_14
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-05833-7
Online ISBN: 978-3-319-05834-4
eBook Packages: Humanities, Social Sciences and LawEducation (R0)