Advertisement

Information Trails: In-Process Assessment of Game-Based Learning

  • Christian S. Loh
Chapter

Abstract

Assessment is an issue that is important to educators because without it, there is no way of telling if learners have arrived at the intended destination. Conducting assessments with game-based learning requires new tools and research methodologies because traditional face-to-face techniques do not transfer well into the multi-user virtual environments found in game worlds. Formative assessment is more useful to instructors and learners because it provides multi-point feedback to help them self-reflect and improve on what they are doing. This chapter describes an “all-rounded” assessment system for game-based learning, which take into consideration the needs of multiple consumers (of information), namely, the Administrators, the Trainers or Educators, and the Learners. The assessment system made use of Information Trails ©—a viable assessment methodology to collect user-generated action data as the game-based learning is occurring (hence, an in-process assessment). The data collected are then remotely transmitted to an external data storage using telemetry, and displayed in real-time via a data visualization application known as Performance Tracing Report Assistant © (or PeTRA). The final online assessment report can be tailored differently according to individual needs of the Learners, Trainers, and Administrators.

Keywords

Game Play Game Development Summative Assessment Digital Game Game Engine 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Notes

Acknowledgments

The research work in this article is made possible in part through funding from the Defence University Research Instrumentation Program (DURIP) grant from the US Army Research Office (ARO). Screenshots of PeTRA (Figs. 8.2 and 8.5) are used with permission from the author.

References

  1. Aldrich, C. (2009). The complete guide to simulations and serious games (p. 576). San Francisco, CA: Pfeiffer.Google Scholar
  2. Baker, R. S., Corbett, A. T., Roll, I., & Koedinger, K. R. (2008). Developing a generalizable detector of when students game the system. User Modeling and User-Adapted Interaction, 18(3), 287–314.CrossRefGoogle Scholar
  3. BBC News. (2007). Computer game to boost key skills. Retrieved Jan 22, 2011 from http://news.bbc.co.uk/2/hi/uk_news/education/6254989.stm.
  4. Berger, A. (2006). “Neverwinter Nights” in the classroom. UMN News. Retrieved Sept 9, 2011 from http://www1.umn.edu/news/features/2006/UR_83484_REGION1.html.
  5. Black, P. J., & Wiliam, D. (1998). Assessment and classroom learning. Assessment in Education: Principles, Policy and Practice, 5(1), 1–73.CrossRefGoogle Scholar
  6. Black, P. J., & Wiliam, D. (2009). Developing the theory of formative assessment. Educational Assessment, Evaluation and Accountability, 1(1), 5–31.CrossRefGoogle Scholar
  7. Botturi, L., & Loh, C. S. (2008). Once upon a game: Rediscovering the roots of games in education. In C. T. Miller (Ed.), Games: Purpose and potential in education (pp. 1–22). New York, NY: Springer Science.Google Scholar
  8. Chen, S., & Michael, D. (2005). Proof of learning: Assessment in serious games. Gamasutra. Retrieved Jan 12, 2011 from http://www.gamasutra.com/features/20051019/chen_01.shtml.
  9. Clark, R. E. (2007). Learning from serious games? Arguments, evidence and research suggestions. Educational Technology, 47, 56–59.Google Scholar
  10. Csikszentmihalyi, M. (1990). Flow: The Psychology of Optimum Experience. New York, NY: Harper & Row.Google Scholar
  11. Cuban, L. (2001). Oversold and underused: Computers in the classroom. Cambridge, MA: Harvard University Press.Google Scholar
  12. Dragon Age: Origins (2009). [Computer software]. Redwood City, CA: EA Games.Google Scholar
  13. Duffy, J. (1997). The role of virtual reality simulation in full scale joint military exercises (p. 17). Newport, RI: Naval War College.Google Scholar
  14. Gee, J. P. (2007). What video games have to teach us about learning and literacy (p. 256). New York, NY: Palgrave Macmillan.Google Scholar
  15. Gibson, D., Aldrich, C., & Prensky, M. (Eds.). (2006). Games and simulations in online learning: Research and development frameworks (p. 402). Hershey, PA: Information Science.Google Scholar
  16. Gorniak, P., & Roy, D. (2005). Speaking with your sidekick: Understanding situated speech in computer role playing games. In R. M. Young & J. E. Laird (Eds.), Proceedings of the first artificial intelligence and interactive digital entertainment conference (AIIDE 2005), June 1–5, 2005. Marina del Rey, CA: AAAI.Google Scholar
  17. Harrington, H. L., Meisels, S. J., McMahon, P., Dichtelmiller, M. L., & Jablon, J. R. (1997). Observing, documenting and assessing learning. Ann Arbor, MI: Rebus.Google Scholar
  18. Hays, R. T. (2005). The effectiveness of instructional games: A literature review and discussion (p. 63). Orlando, FL: Naval Air Warfare Center.Google Scholar
  19. Hattie, J. (1987). Identifying the salient facets of a model of student learning: A synthesis of meta analyses. International Journal of Educational Research, 11(2), 187–212.Google Scholar
  20. Ifenthaler, D., Eseryel, D., & Ge, X. (Eds.). (2012). Assessment in game-based learning: Foundations, innovations, and perspectives. New York, NY: Springer Science+Business Media New York.Google Scholar
  21. Joosten-ten Brinke, D., Gorissen, P., & Latour, I. (2005). Integrating assessment into e-learning courses. In R. Koper & C. Tattersall (Eds.), Learning design: A handbook on modelling and delivering networked education and training (pp. 185–202). Berlin: Springer.Google Scholar
  22. Kaplan-Rakowski, R., & Loh, C. S. (2010). Modding and rezzing in games and virtual environments for education. In Y. K. Baek (Ed.), Gaming for classroom-based learning: Digital role playing as a motivator of study (pp. 205–219). Hershey, PA: IGI-Global.CrossRefGoogle Scholar
  23. Kapp, K. M., & O’Driscoll, T. (2010). Learning in 3D: Adding a new dimension to enterprise learning and collaboration (p. 416). San Francisco, CA: Pfeiffer.Google Scholar
  24. Kebritchi, M. (2008). Effects of a computer game on mathematics achievement and class motivation: An experimental study. PhD Thesis. University of Central Florida, Orlando, Florida.Google Scholar
  25. Lee, H. (2008). Students’ perceptions of peer and self assessment in a higher education online collaborative learning environment. Unpublished Ph.D. dissertation, Graduate School of the University of Texas at Austin, Austin. Retrieved February 8, 2011 from http://www.lib.utexas.edu/etd/d/2008/leeh55399/leeh55399.pdf.
  26. Levin, A. (2010). Simulator training flaws tied to airline crashes. USA Today. Retrieved March 1, 2011 from http://travel.usatoday.com/flights/2010-08-31-1Acockpits31_ST_N.htm.
  27. Loh, C. S. (2006). Tracking an avatar: Designing data collection into online games. Paper presented at the Annual Conference of the Association for Educational Communications and Technology (AECT 2006). Dallas, TX.Google Scholar
  28. Loh, C. S. (2006b). Designing online games assessment as “Information Trails”. In D. Gibson, C. Aldrich, & M. Prensky (Eds.), Games and simulation in online learning: Research and development frameworks (pp. 323–348). Hershey, PA: Idea Group.CrossRefGoogle Scholar
  29. Loh, C. S. (2009). Researching and developing serious games as interactive learning instructions. International Journal of Gaming and Computer Mediated Simulations, 1(4), 1–19.CrossRefGoogle Scholar
  30. Loh, C. S., Anantachai, A., Byun, J. H., & Lenox, J. (2007). Assessing what players learned in serious games: In situ data collection, Information Trails, and quantitative analysis. In Q. Mehdi (Ed.), Proceedings of the computer games: AI, animation, mobile, educational & serious games conference (CGAMES 2007). Wolverhampton, UK: University of Wolverhampton.Google Scholar
  31. Loh, C. S., & Byun, J. H. (2009). Modding Neverwinter Nights into serious game. In D. Gibson & Y. K. Baek (Eds.), Digital Simulations for Improving Education: Learning Through Artificial Teaching Environments (pp. 408–426). Hershey, PA: IGI-Global.CrossRefGoogle Scholar
  32. Loh, C. S., & Li, I. H. (2010, Oct 10–13). Reducing re-training cost through on-demand, ad hoc assessment. Proceedings of the MODSIM world conference and expo 2010—21st century decision-making: the art of modeling & simulation, Hampton, VA: MODSIM World Conference & Exposition.Google Scholar
  33. Moreno-Ger, P., Burgos, D., Martínez-Ortiz, I., Sierra, J. L., & Fernández-Manjón, B. (2008). Educational game design for online education. Computers in Human Behavior, 24, 2530–2540.CrossRefGoogle Scholar
  34. Nacke, L., Grimshaw, M., & Lindley, C. (2010). More than a feeling: Measurement of sonic user experience and psychophysiology in a first-person shooter game. Interacting with Computers, 22(5), 336–343. doi: 10.1016/j.intcom.2010.04.005.CrossRefGoogle Scholar
  35. Nacke, L., Lindley, C., & Stellmach, S. (2008). Log who’s playing: Psychophysiological game analysis made easy through event logging. In P. Markopoulos et al. (Eds.), Fun and games 2008, lecture notes of computer science (Vol. 5294, pp. 150–157). Heidelberg, Germany: Springer.Google Scholar
  36. Nacke, L., & Lindley, C. (2010). Affective ludology, flow and immersion in a first-person shooter: Measurement of player experience. Loading, 3(5). Retrieved March 12, 2011 from http://journals.sfu.ca/loading/index.php/loading/article/view/72/71
  37. O’Neil, H. F., & Robertson, M. M. (1992). Simulations: Occupationally oriented. In M. C. Alkin (Ed.), Encyclopedia of Educational Research (6th ed., pp. 1216–1222). New York, NY: Macmillan.Google Scholar
  38. Prensky, M. (2001). Digital game-based learning. New York, NY: McGraw-Hill.Google Scholar
  39. Prensky, M. (2006). Don’t bother me mom—I’m learning! St. Paul, MN: Paragon House.Google Scholar
  40. Pruett, C. (2010, Sept). Hot failure: Tuning gameplay with simple player metrics. Game Developer Magazine. 19(9). San Francisco, CA: Think Services. Retrieved March 7, 2011 from http://gamasutra.com/view/feature/6155/hot_failure_tuning_gameplay_with_.php.
  41. Reese, D. D. (2010). Introducing flowometer: A CyGaMEs assessment suite tool. In R. van Eck (Ed.), Gaming and cognition: Theories and practice from the learning sciences (pp. 227–254). Hershey, PA: IGI-Global.CrossRefGoogle Scholar
  42. Robert, L. G. (2003). Forewords. In G. D. Haertel & B. Means (Eds.), Evaluating educational technology: Effective research designs for improving learning (p. 290). New York, NY: Teachers College Press.Google Scholar
  43. Robertson, J., & Good, J. (2005). Story creation in virtual game worlds. Communications of the ACM, 48(1), 61–65.CrossRefGoogle Scholar
  44. Rupp, A. A., Gushta, M., Mislevy, R. J., & Shaffer, D. W. (2010). Evidence-centered design of epistemic games: Measurement principles for complex learning environments. The Journal of Technology Learning and Assessment, 8(4), 3–41.Google Scholar
  45. Saridaki, M., Mourlas, C., Gouscos, D., & Meimaris, M. (2007). Digital games as a learning tool for children with cognitive disabilities: Literature review and some preliminary methodological and experimental results. Paper presented at the European Conference on Games Based Learning, ScotlandGoogle Scholar
  46. Sasse, D. (2008). A framework for psychophysiological data acquisition in digital games. Unpublished Master thesis, Lund University, Lund. Retrieved March 12, 2011 from http://citeseerx.ist.psu.edu/viewdoc/download? doi: 10.1.1.140.5284&rep=rep1&type=pdf.
  47. Shute, V. J., Masduki, I., Donmez, O., Dennen, V. P., Kim, Y.-J., Jeong, A. C., et al. (2010). Modeling, assessing, and supporting key competencies within game environments. In D. Ifenthaler, P. Pirnay-Dummer, & N. M. Seel (Eds.), Computer-based diagnostics and systematic analysis of knowledge (Part 4, pp. 281–309). Boston, MA: Springer. Retrieved March 5, 2011, from http://www.springerlink.com/content/t5261055733n18h2/.
  48. Shute, V. J., & Spector, J. M. (2008). SCORM 2.0 White paper: Stealth assessment in virtual worlds. Retrieved March 12, 2011, from http://www.adlnet.gov/technologies/evaluation/library/additionalresources/letsiwhitepapers/shute-stealthassessmentinvirtualworlds.pdf.
  49. Skinner, B. F. (1938). The behavior of organisms: An experimental analysis. Cambridge, MA: B.F. Skinner Foundation.Google Scholar
  50. Stieger, I. (2008). Neverwinter Nights eXtender 4 (version 1.09) [software]. Starnberg, Germany: Stieger Hardware-und Softwareentwicklung. Retrieved April 12, 2012, from http://www.nwnx.org.
  51. Telemetry. (n.d.). The American Heritage Stedman’s Medical Dictionary. Retrieved March 9, 2011, from http://dictionary.reference.com/browse/telemetry.
  52. Tobias, S., & Fletcher, J. D. (2007). What research has to say about designing computer games for learning. Educational Technology, 47(5), 20–29.Google Scholar
  53. van Eck, R. (2006). Digital game-based learning: It’s not just the digital natives who are restless…. Educause Review, 41(2), 16–30.Google Scholar
  54. Weil, S. A., Hussain, T. S., Brunyé, T. T., Ferguson, W., Sidman, J. G., Spahr, L.S., & Roberts, B. (2005). Assessing the potential of massive multi-player games as tools for military training. Proceedings of the interservice/industry training, simulation, and education conference, (I/ITSEC 2005). Orlando, FL: NTSA.Google Scholar
  55. Warren, R., & Sutton, J. (2008, April). Using a computer game for research on culture and team adaptability: Lessons learned from a NATO experiment. Proceedings of the NATO Research and Technology Organization (RTO) Human Factors and Medicine Panel HFM, 142 Symposium on Adaptability in Coalition Teamwork. Copenhagen, Denmark.Google Scholar
  56. Wilson, K. A., Bedwell, W. L., Lazzara, E. H., Salas, E., Burke, C. S., Estock, J. L., et al. (2008). Relationships between game attributes and learning outcomes: Review and research proposals. Simulation & Gaming, 40(2), 217–266.CrossRefGoogle Scholar
  57. Younis, B., & Loh, C. S. (2010). Integrating serious games in higher education programs. Proceedings of the academic colloquium 2010: Building partnership in teaching excellence. Ramallah, Palestine: AMIDEAST.Google Scholar
  58. Zoeller, G. (2010). Development telemetry in video games projects. Paper presented at the Game Developer Conference 2010. San Francisco, CA. Retrieved March 1, 2011 from http://gdc.gulbsoft.org/talk.

Copyright information

© Springer Science+Business Media New York 2012

Authors and Affiliations

  1. 1.Virtual Environment Lab (V-Lab), Department of Curriculum and InstructionSouthern Illinois University CarbondaleCarbondaleUSA

Personalised recommendations