Abstract
Is the CyGaMEs (cyberlearning through game-based, metaphor enhanced learning objects) Timed Report tool (embedded assessment for game-based instructional environments derived from analogical reasoning and instructional design theory) a sensitive measure of learning? Using gameplay assessed every 10 s from 221 volunteers (primarily 13–18-year-olds distributed within and outside the United States playing the online videogame Selene), an algorithm identified a targeted learning moment, designated Timed Reports as “before” or “at and after” learning, and calculated Pearson correlation coefficients and slopes for both learning trajectories. Three studies compared the strength and quality of the Timed Report: a multilevel model using raw Timed Report values, repeated measures using slopes, and repeated measures using Pearson correlation coefficients. The three studies supported the same conclusions and replicate, elaborate, and generalize earlier findings. Timed Report successfully distinguishes player progress before and after learning. The strong effect supports the CyGaMEs approach to instructional game design and Timed Report embedded assessment for educational data mining. Timed Report also distinguishes between players who are Always Progressing and those who experience a learning moment. Participants who watch gameplay before playing have early success; however, participants who play without first watching gameplay make the largest gains in performance, and their after-learning performance equals that of watchers. Watching instructional videos about targeted game content in addition to watching gameplay before playing did not enhance performance.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
This algorithm followed an approach developed for CyGaMEs by James Pusteovsky under the direction of Larry V. Hedges.
- 2.
We plot learning moment graphs using Python MatPlotLib and calculate running means, running standard deviations, slopes, and Pearson’s correlation coefficient r using the linregress function within the Python 2.6 SciPy stats sublibrary.
- 3.
CyGaMEs has since developed and released Selene II, providing optimized gameplay and performance, enhanced graphics with 3D effects, and new sound effects and animations. A still shot from the Selene II Solar System accretion module can be viewed here: http://selene.cet.edu/. A West Virginia Public Broadcasting feature at segment 4:45–5:09 discusses and illustrates the accretion scale 1 module and those leading to it, available here: http://www.youtube.com/watch?v=ZBPq3Hc_g1Y.
- 4.
We appreciate Matthew Petrole’s assistance in developing the Excel report template.
References
Anderson, J. R., Reder, L. M., & Simon, H. A. (1998). Radical constructivism and cognitive psychology. In D. Ravitch (Ed.), Brookings papers on educational policy: 1998 (p. 384). Washington, DC: The Brookings Institution Press.
Ausubel, D. P. (1968). Educational psychology: A cognitive view. New York, NY: Holt, Rinehart, and Winston.
Bandura, A. (1997). Self-efficacy: The exercise of control. New York, NY: W. H. Freeman.
Borgman, C. L., Abelson, H., Johnson, R., Koedinger, K. R., Linn, M. C., Lynch, C. A., et al. (2008). Fostering learning in the networked world: The cyberlearning opportunity and challenge. A 21st century agenda for the National Science Foundation. Arlington, VA: National Science Foundation. Retrieved August 1, 2008, from http://www.nsf.gov/pubs/2008/nsf08204/nsf08204.pdf?govDel=USNSF_124.
Diehl, V., & Reese, D. D. (2010). Elaborated metaphors support viable inferences about difficult science concepts. Educational Psychology, 30(7), 771–791.
Feltovich, P. J., Coulson, R. L., & Spiro, R. J. (1988). Learners’ (mis)understanding of important and difficult concepts: A challenge to smart machines in education. In K. D. Forbus & P. J. Feltovich (Eds.), Smart machines in education: The coming revolution in educational technology (pp. 349–375). Cambridge, MA: MIT Press.
Gentner, D. (1983). Structure mapping: A theoretical framework for analogy. Cognitive Science, 7, 155–170.
Gentner, D., & Markman, A. B. (1997). Structure mapping in analogy and similarity. The American Psychologist, 52(1), 45–56.
Hitt, B. A., & Reese, D. D. (2012). Knowledge discovery from Selene data. In E. Wiebe (Chair), New measurement paradigms: Psychometric methods for technology-based assessments: Structured poster session presented at the annual meeting of the American Educational Research Association. Vancouver, BC.
Holyoak, K. J., & Thagard, P. (1989). Analogical mapping with constraint satisfaction. Cognitive Science, 13, 295–355.
Holyoak, K. J., Gentner, D., & Kokinov, B. N. (2001). Introduction: The place of analogy in cognition. In D. Gentner, K. J. Holyoak, & B. N. Kokinov (Eds.), The analogical mind: Perspectives from cognitive science (pp. 1–20). Cambridge, MA: MIT Press.
National Research Council (2011). Learning science through computer games and simulations. Committee on Science Learning: Computer Games, Simulations, and Education. In M.A. Honey and M.L. Hilton (Eds.), Board on Science Education, Division of Behavioral and Social Sciences and Education. Washington, DC: The National Academies Press.
Hummel, J. E., & Holyoak, K. J. (1997). Distributed representations of structure: A theory of analogical access and mapping. Psychological Review, 104(3), 427–466.
Johnstone, A. H. (1991). Why is science difficult to learn? Things are seldom what they seem. Journal of Computer Assisted Learning, 7(2), 75–83.
Kurtz, K. J., Miao, C.-H., & Gentner, D. (2001). Learning by analogical bootstrapping. The Journal of the Learning Sciences, 10(4), 417–446.
Lakoff, G., & Johnson, M. (1980). Metaphors we live by. Chicago, IL: The University of Chicago Press.
Lakoff, G., & Johnson, M. (1999). Philosophy in the flesh: The embodied mind and its challenge to Western thought. New York, NY: Basic Books.
Merrill, M. D. (2002). First principles of instruction. Educational Technology Research and Development, 50(3), 43–59.
Piaget, J., & Inhelder, B. (2000). The psychology of the child (H. Weaver, Trans.). New York, NY: Basic Books.
Reese, D. D. (2008, November). Flowometer: Embedded measurement of learners’ flow perceptions within game-based instructional environments. Paper presented at the 2008 international conference of the Association for Educational Communications and Technology, Orlando, FL.
Reese, D. D. (2009a, October). Replication supports flowometer: Advancing cyberlearning through game-based assessment technologies. Paper presented at the 2009 international conference of the Association for Educational Communications and Technology, Louisville, KY.
Reese, D. D. (2009b). Structure mapping theory as a formalism for instructional game design and assessment. In D. Gentner, K. Holyoak, & B. Kokinov (Eds.), New frontiers in analogy research: Proceedings of the 2nd international conference on analogy (Analogy ’09) (pp. 394–403). Sofia, Bulgaria: New Bulgarian University Press.
Reese, D. D. (2010). Introducing flowometer: A CyGaMEs assessment suite tool. In R. V. Eck (Ed.), Gaming & cognition: Theories and perspectives from the learning sciences (pp. 227–254). Hershey, PA: IGI Global.
Reese, D. D., & Hitt, B. A. (2009, June). Selene knowledge discovery: The interface effect. Poster presented at Games + Learning + Society 5.0, Madison, WI.
Reese, D. D., & Tabachnick, B. G. (2010). The moment of learning: Quantitative analysis of exemplar gameplay supports CyGaMEs approach to embedded assessment. In J. Earle (Ed.), Building a knowledge base to inform educational practice in STEM: Examples from the REESE portfolio. Symposium conducted at the annual meeting of the Society for Research on Educational Effectiveness. Washington, DC. Structured abstract retrieved March 6, 2010, from http://www.sree.org/conferences/2010/program/abstracts/191.pdf.
Reese, D. D., Diehl, V. A., & Lurquin, J. L. (2009, May). Metaphor enhanced instructional videogame causes conceptual gains in lunar science knowledge. Poster presented at the annual meeting of the Association for Psychological Science, San Francisco, CA.
Reese, D. D., Diehl, V. A., & Lurquin, J. L. (2009, May). Metaphor enhanced instructional videogame causes conceptual gains in lunar science knowledge. Poster presented at the annual meeting of the Association for Psychological Science, San Francisco, CA.
Schwartz, D. L., & Bransford, J. D. (1998). A time for telling. Cognition and Instruction, 16(4), 475–522.
Schwartz, D. L., & Martin, T. (2004). Inventing to prepare for future learning: The hidden efficiency of encouraging original student production in statistics instruction. Cognition and Instruction, 22(2), 129–184.
Spellman, B. A., & Holyoak, K. J. (1996). Pragmatics in analogical mapping. Cognitive Psychology, 31(3), 307–346.
U.S. Department of Education, Office of Educational Technology. (2010). Transforming American education: Learning powered by technology. National education technology plan 2010. Washington, DC: U.S. Department of Education, Office of Educational Technology.
Acknowledgments
This material is based upon work supported by the National Science Foundation under Grant DRL-0814512 awarded to Debbie Denise Reese, Charles A. Wood, Ben Hitt, and Beverly Carter and work supported by the National Aeronautics and Space Administration under awards NCC5-451, NNX06AB09G-Basic, NNX06AB09G-Sup-1, NNX06AB09G-Sup-2, NAG-13782, and NNX08AJ71A-Basic to the NASA-sponsored Classroom of the Future. The first author refined the learning moment approach in collaboration with Larry V. Hedges. James Pusteovsky developed the precursor to this learning moment algorithm. Selene derived from Charles A. Wood’s mental model of lunar and planetary geology, and he is the project’s subject matter expert. We are indebted to our colleagues, Janis Worklan and Cassie Lightfritz, and our student researchers, Matthew Petrole and Steven Nowak. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation or the National Aeronautics and Space Administration.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2012 Springer Science+Business Media New York
About this chapter
Cite this chapter
Reese, D.D., Seward, R.J., Tabachnick, B.G., Hitt, B.A., Harrison, A., Mcfarland, L. (2012). Timed Report Measures Learning: Game-Based Embedded Assessment. In: Ifenthaler, D., Eseryel, D., Ge, X. (eds) Assessment in Game-Based Learning. Springer, New York, NY. https://doi.org/10.1007/978-1-4614-3546-4_9
Download citation
DOI: https://doi.org/10.1007/978-1-4614-3546-4_9
Published:
Publisher Name: Springer, New York, NY
Print ISBN: 978-1-4614-3545-7
Online ISBN: 978-1-4614-3546-4
eBook Packages: Humanities, Social Sciences and LawEducation (R0)