Information Trails: In-Process Assessment of Game-Based Learning
Assessment is an issue that is important to educators because without it, there is no way of telling if learners have arrived at the intended destination. Conducting assessments with game-based learning requires new tools and research methodologies because traditional face-to-face techniques do not transfer well into the multi-user virtual environments found in game worlds. Formative assessment is more useful to instructors and learners because it provides multi-point feedback to help them self-reflect and improve on what they are doing. This chapter describes an “all-rounded” assessment system for game-based learning, which take into consideration the needs of multiple consumers (of information), namely, the Administrators, the Trainers or Educators, and the Learners. The assessment system made use of Information Trails ©—a viable assessment methodology to collect user-generated action data as the game-based learning is occurring (hence, an in-process assessment). The data collected are then remotely transmitted to an external data storage using telemetry, and displayed in real-time via a data visualization application known as Performance Tracing Report Assistant © (or PeTRA). The final online assessment report can be tailored differently according to individual needs of the Learners, Trainers, and Administrators.
KeywordsGame Play Game Development Summative Assessment Digital Game Game Engine
- Aldrich, C. (2009). The complete guide to simulations and serious games (p. 576). San Francisco, CA: Pfeiffer.Google Scholar
- BBC News. (2007). Computer game to boost key skills. Retrieved Jan 22, 2011 from http://news.bbc.co.uk/2/hi/uk_news/education/6254989.stm.
- Berger, A. (2006). “Neverwinter Nights” in the classroom. UMN News. Retrieved Sept 9, 2011 from http://www1.umn.edu/news/features/2006/UR_83484_REGION1.html.
- Botturi, L., & Loh, C. S. (2008). Once upon a game: Rediscovering the roots of games in education. In C. T. Miller (Ed.), Games: Purpose and potential in education (pp. 1–22). New York, NY: Springer Science.Google Scholar
- Chen, S., & Michael, D. (2005). Proof of learning: Assessment in serious games. Gamasutra. Retrieved Jan 12, 2011 from http://www.gamasutra.com/features/20051019/chen_01.shtml.
- Clark, R. E. (2007). Learning from serious games? Arguments, evidence and research suggestions. Educational Technology, 47, 56–59.Google Scholar
- Csikszentmihalyi, M. (1990). Flow: The Psychology of Optimum Experience. New York, NY: Harper & Row.Google Scholar
- Cuban, L. (2001). Oversold and underused: Computers in the classroom. Cambridge, MA: Harvard University Press.Google Scholar
- Dragon Age: Origins (2009). [Computer software]. Redwood City, CA: EA Games.Google Scholar
- Duffy, J. (1997). The role of virtual reality simulation in full scale joint military exercises (p. 17). Newport, RI: Naval War College.Google Scholar
- Gee, J. P. (2007). What video games have to teach us about learning and literacy (p. 256). New York, NY: Palgrave Macmillan.Google Scholar
- Gibson, D., Aldrich, C., & Prensky, M. (Eds.). (2006). Games and simulations in online learning: Research and development frameworks (p. 402). Hershey, PA: Information Science.Google Scholar
- Gorniak, P., & Roy, D. (2005). Speaking with your sidekick: Understanding situated speech in computer role playing games. In R. M. Young & J. E. Laird (Eds.), Proceedings of the first artificial intelligence and interactive digital entertainment conference (AIIDE 2005), June 1–5, 2005. Marina del Rey, CA: AAAI.Google Scholar
- Harrington, H. L., Meisels, S. J., McMahon, P., Dichtelmiller, M. L., & Jablon, J. R. (1997). Observing, documenting and assessing learning. Ann Arbor, MI: Rebus.Google Scholar
- Hays, R. T. (2005). The effectiveness of instructional games: A literature review and discussion (p. 63). Orlando, FL: Naval Air Warfare Center.Google Scholar
- Hattie, J. (1987). Identifying the salient facets of a model of student learning: A synthesis of meta analyses. International Journal of Educational Research, 11(2), 187–212.Google Scholar
- Ifenthaler, D., Eseryel, D., & Ge, X. (Eds.). (2012). Assessment in game-based learning: Foundations, innovations, and perspectives. New York, NY: Springer Science+Business Media New York.Google Scholar
- Joosten-ten Brinke, D., Gorissen, P., & Latour, I. (2005). Integrating assessment into e-learning courses. In R. Koper & C. Tattersall (Eds.), Learning design: A handbook on modelling and delivering networked education and training (pp. 185–202). Berlin: Springer.Google Scholar
- Kapp, K. M., & O’Driscoll, T. (2010). Learning in 3D: Adding a new dimension to enterprise learning and collaboration (p. 416). San Francisco, CA: Pfeiffer.Google Scholar
- Kebritchi, M. (2008). Effects of a computer game on mathematics achievement and class motivation: An experimental study. PhD Thesis. University of Central Florida, Orlando, Florida.Google Scholar
- Lee, H. (2008). Students’ perceptions of peer and self assessment in a higher education online collaborative learning environment. Unpublished Ph.D. dissertation, Graduate School of the University of Texas at Austin, Austin. Retrieved February 8, 2011 from http://www.lib.utexas.edu/etd/d/2008/leeh55399/leeh55399.pdf.
- Levin, A. (2010). Simulator training flaws tied to airline crashes. USA Today. Retrieved March 1, 2011 from http://travel.usatoday.com/flights/2010-08-31-1Acockpits31_ST_N.htm.
- Loh, C. S. (2006). Tracking an avatar: Designing data collection into online games. Paper presented at the Annual Conference of the Association for Educational Communications and Technology (AECT 2006). Dallas, TX.Google Scholar
- Loh, C. S., Anantachai, A., Byun, J. H., & Lenox, J. (2007). Assessing what players learned in serious games: In situ data collection, Information Trails, and quantitative analysis. In Q. Mehdi (Ed.), Proceedings of the computer games: AI, animation, mobile, educational & serious games conference (CGAMES 2007). Wolverhampton, UK: University of Wolverhampton.Google Scholar
- Loh, C. S., & Li, I. H. (2010, Oct 10–13). Reducing re-training cost through on-demand, ad hoc assessment. Proceedings of the MODSIM world conference and expo 2010—21st century decision-making: the art of modeling & simulation, Hampton, VA: MODSIM World Conference & Exposition.Google Scholar
- Nacke, L., Lindley, C., & Stellmach, S. (2008). Log who’s playing: Psychophysiological game analysis made easy through event logging. In P. Markopoulos et al. (Eds.), Fun and games 2008, lecture notes of computer science (Vol. 5294, pp. 150–157). Heidelberg, Germany: Springer.Google Scholar
- Nacke, L., & Lindley, C. (2010). Affective ludology, flow and immersion in a first-person shooter: Measurement of player experience. Loading, 3(5). Retrieved March 12, 2011 from http://journals.sfu.ca/loading/index.php/loading/article/view/72/71
- O’Neil, H. F., & Robertson, M. M. (1992). Simulations: Occupationally oriented. In M. C. Alkin (Ed.), Encyclopedia of Educational Research (6th ed., pp. 1216–1222). New York, NY: Macmillan.Google Scholar
- Prensky, M. (2001). Digital game-based learning. New York, NY: McGraw-Hill.Google Scholar
- Prensky, M. (2006). Don’t bother me mom—I’m learning! St. Paul, MN: Paragon House.Google Scholar
- Pruett, C. (2010, Sept). Hot failure: Tuning gameplay with simple player metrics. Game Developer Magazine. 19(9). San Francisco, CA: Think Services. Retrieved March 7, 2011 from http://gamasutra.com/view/feature/6155/hot_failure_tuning_gameplay_with_.php.
- Robert, L. G. (2003). Forewords. In G. D. Haertel & B. Means (Eds.), Evaluating educational technology: Effective research designs for improving learning (p. 290). New York, NY: Teachers College Press.Google Scholar
- Rupp, A. A., Gushta, M., Mislevy, R. J., & Shaffer, D. W. (2010). Evidence-centered design of epistemic games: Measurement principles for complex learning environments. The Journal of Technology Learning and Assessment, 8(4), 3–41.Google Scholar
- Saridaki, M., Mourlas, C., Gouscos, D., & Meimaris, M. (2007). Digital games as a learning tool for children with cognitive disabilities: Literature review and some preliminary methodological and experimental results. Paper presented at the European Conference on Games Based Learning, ScotlandGoogle Scholar
- Sasse, D. (2008). A framework for psychophysiological data acquisition in digital games. Unpublished Master thesis, Lund University, Lund. Retrieved March 12, 2011 from http://citeseerx.ist.psu.edu/viewdoc/download? doi: 10.1.1.140.5284&rep=rep1&type=pdf.
- Shute, V. J., Masduki, I., Donmez, O., Dennen, V. P., Kim, Y.-J., Jeong, A. C., et al. (2010). Modeling, assessing, and supporting key competencies within game environments. In D. Ifenthaler, P. Pirnay-Dummer, & N. M. Seel (Eds.), Computer-based diagnostics and systematic analysis of knowledge (Part 4, pp. 281–309). Boston, MA: Springer. Retrieved March 5, 2011, from http://www.springerlink.com/content/t5261055733n18h2/.
- Shute, V. J., & Spector, J. M. (2008). SCORM 2.0 White paper: Stealth assessment in virtual worlds. Retrieved March 12, 2011, from http://www.adlnet.gov/technologies/evaluation/library/additionalresources/letsiwhitepapers/shute-stealthassessmentinvirtualworlds.pdf.
- Skinner, B. F. (1938). The behavior of organisms: An experimental analysis. Cambridge, MA: B.F. Skinner Foundation.Google Scholar
- Stieger, I. (2008). Neverwinter Nights eXtender 4 (version 1.09) [software]. Starnberg, Germany: Stieger Hardware-und Softwareentwicklung. Retrieved April 12, 2012, from http://www.nwnx.org.
- Telemetry. (n.d.). The American Heritage Stedman’s Medical Dictionary. Retrieved March 9, 2011, from http://dictionary.reference.com/browse/telemetry.
- Tobias, S., & Fletcher, J. D. (2007). What research has to say about designing computer games for learning. Educational Technology, 47(5), 20–29.Google Scholar
- van Eck, R. (2006). Digital game-based learning: It’s not just the digital natives who are restless…. Educause Review, 41(2), 16–30.Google Scholar
- Weil, S. A., Hussain, T. S., Brunyé, T. T., Ferguson, W., Sidman, J. G., Spahr, L.S., & Roberts, B. (2005). Assessing the potential of massive multi-player games as tools for military training. Proceedings of the interservice/industry training, simulation, and education conference, (I/ITSEC 2005). Orlando, FL: NTSA.Google Scholar
- Warren, R., & Sutton, J. (2008, April). Using a computer game for research on culture and team adaptability: Lessons learned from a NATO experiment. Proceedings of the NATO Research and Technology Organization (RTO) Human Factors and Medicine Panel HFM, 142 Symposium on Adaptability in Coalition Teamwork. Copenhagen, Denmark.Google Scholar
- Younis, B., & Loh, C. S. (2010). Integrating serious games in higher education programs. Proceedings of the academic colloquium 2010: Building partnership in teaching excellence. Ramallah, Palestine: AMIDEAST.Google Scholar
- Zoeller, G. (2010). Development telemetry in video games projects. Paper presented at the Game Developer Conference 2010. San Francisco, CA. Retrieved March 1, 2011 from http://gdc.gulbsoft.org/talk.