Skip to main content

Advertisement

SpringerLink
Log in

Measuring the (dis-)similarity between expert and novice behaviors as serious games analytics

  • Published:
Education and Information Technologies Aims and scope Submit manuscript

Abstract

The behavioral differences between expert and novice performance is a well-studied area in training literature. Advances in technology have made it possible to trace players’ actions and behaviors within an online gaming environment as user-generated data for performance assessment. In this study, we introduce the use of string similarity to differentiate likely-experts from a group of unknown performers (mixture of novices and experts) based on how similar their in-game actions are to that of experts. Our findings indicate that string similarity is viable as an empirical assessment method to differentiate likely-experts from novices and potentially useful as the first performance metric for Serious Games Analytics (SEGA).

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  • Bellotti, F., Kapralos, B., Lee, K., Moreno-Ger, P., & Berta, R. (2013). Assessment in and of Serious Games: an overview. Advances in Human-Computer Interaction, 2013, 11. doi:10.1155/2013/136864.

    Google Scholar 

  • Bilenko, M., & Mooney, R. J. (2003). Adaptive duplicate detection using learnable string similarity measures. Proceedings of the 9th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD-2003) (pp. 39–48). Washington DC. Retrieved from http://www.isi.edu/info-agents/courses/csci548/Papers/marlin-kdd-03.pdf

  • Boot, W. R., Kramer, A. F., Simons, D. J., Fabiani, M., & Gratton, G. (2008). The effects of video game playing on attention, memory, and executive control. Acta Psychologica, 129(3), 387–398. doi:10.1016/j.actpsy.2008.09.005. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/18929349

    Google Scholar 

  • Canossa, A., Seif El-Nasr, M., & Drachen, A. (2013) Benefits of game analytics: Stakeholders, contexts and domains. In M. Seif El-Nasr, A. Drachen, & A. Canossa (Eds.), Games Analytics: maximizing the value of player data (pp. 41–52). London, UK: Springer-Verlag. doi:10.1007/978-1-4471-4769-5.

  • Chamberlain, T. C., Catano, V. M., & Cunningham, D. P. (2005). Personality as a predictor of professional behavior in dental school: Comparisons with dental practitioners. Journal of Dental Education, 69(11), 1222–1237. Retrieved from http://www.jdentaled.org/cgi/content/abstract/69/11/1222

    Google Scholar 

  • Clark, R. E. (1985). Confounding in educational computing research. Journal of Educational Computing Research, 1(2), 137–148.

    Google Scholar 

  • Dreyfus, S. E. (2004). The five-stage model of adult skill acquisition. Bulletin of Science Technology Society, 24(3), 177–181. doi:10.1177/0270467604264992.

    Article  Google Scholar 

  • Dreyfus, S. E., & Dreyfus, H. L. (1980). A five-stage model of the mental activities involved in directed skill acquisition (p. 18). Report No. ORC-80-2. Berkeley, CA: Operation Research Center. Retrieved from http://www.dtic.mil/cgi-bin/GetTRDoc?AD=ADA084551&Location=U2&doc=GetTRDoc.pdf

  • Ericsson, K. A., Charness, N., Feltovich, P. J., & Hoffman, R. R. (2006). The Cambridge handbook of expertise and expert performance (p. 901). New York: Cambridge University Press.

  • Fletcher, G. H. (2013). New Initiatives Aim to Make School Data Interoperable. T.H.E. Journal, 40(4), 31–34. Retrieved from http://thejournal.com/articles/2013/04/23/new-initiatives-aim-to-make-school-data-interoperable.aspx

  • Hastings, N. B., & Tracey, M. W. (2004). Does media affect learning: where are we now? TechTrends, 49(2), 28–30. doi:10.1007/BF02773968.

  • Hofer, A. (2011). Exploratory comparison of expert and novice pair programmers. In Z. Huzar, R. Koci, B. Meyer, B. Walter, & J. Zendulka (Eds.), Software Engineering Techniques (vol. 4980, pp. 218–231). Berlin, Heidelberg: Springer Berlin Heidelberg. doi:10.1007/978-3-642-22386-0_17.

  • Hogarth, R. M., & Karelaia, N. (2008). Skill, luck, overconfidence, and risk taking. (Online). Rochester, NY: Social Science Electronic Publishing, Inc. doi:10.2139/ssrn.1374235.

  • Hughes, P. (2002). Can we improve on how we select medical students? Journal of the Royal Society of Medicine, 95(1), 18–22. doi:10.1258/jrsm.95.1.18.

    Article  Google Scholar 

  • Ifenthaler, D., Eseryel, D., & Ge, X. (2012). Assessment in Game-based Learning: Foundations, Innovations, and Perspectives (p. 465). New York: Springer.

  • Jonassen, D. H., Hannum, W. H., & Tessmer, M. (1989). Handbook of Task Analysis Procedures (p. 410). Westport: Praeger Publishers.

    Google Scholar 

  • Lachenbruch, P. A., & Mickey, M. R. (1968). Estimation of error rates in discriminant analysis. Technometrics, 10(1), 1–11. Retrieved from http://www.jstor.org/stable/1266219

    Google Scholar 

  • Law, B., Atkins, M. S., Kirkpatrick, A. E., & Lomax, A. J. (2004). Eye gaze patterns differentiate novice and experts in a virtual laparoscopic surgery training environment. In Proceedings of the Eye Tracking Research & Applications Symposium - ETRA 2004 (pp. 41–48). New York: ACM Press. doi:10.1145/968363.968370.

    Google Scholar 

  • Levenshtein, V. I. (1966). Binary codes capable of correcting deletions, insertions and reversals. Soviet Physics-Doklady, 10(8), 707–710.

    Google Scholar 

  • Loh, C. S. (2009). Researching and developing serious games as interactive learning instructions. International Journal of Gaming and Computer-Mediated Simulations, 1(4), 1–19. doi:10.4018/jgcms.2009091501.

    Article  MathSciNet  Google Scholar 

  • Loh, C. S. (2012). Information Trails: In-process assessment of game-based learning. In D. Ifenthaler, D. Eseryel, & X. Ge (Eds.), Assessment in Game-based Learning: Foundations, Innovations, and Perspectives (pp. 123–144). New York: Springer. doi:10.1007/978-1-4614-3546-4.

    Chapter  Google Scholar 

  • Loh, C. S. (2013). Improving the impact and return of investment of game-based learning. International Journal of Virtual and Personal Learning Environments 4(1), 1–15. doi:10.4018/jvple.2013010101.

  • Loh, C. S., Anantachai, A., Byun, J. H., & Lenox, J. (2007). Assessing what players learned in serious games: in situ data collection, Information Trails, and quantitative analysis. In Q. Mehdi (Ed.), Proceedings of the Computer Games: AI, Animation, Mobile, Educational & Serious Games Conference (CGAMES 2007). Wolverhampton: University of Wolverhampton.

    Google Scholar 

  • Louis, S. J., & Miles, C. (2005). Playing to learn: case-injected genetic algorithms for learning to play computer games. Evolutionary Computation, IEEE Transactions on, 9(6), 669–681. doi:10.1109/TEVC.2005.856209.

  • Monge, A. E., & Elkan, C. P. (1997). An efficient domain-independent algorithm for detecting approximately duplicate database records. In Proceedings of the SIGMOD 1997 Workshop on Research Issues on Data Mining and Knowledge Discovery (pp. 23–29). New York: ACM.

  • Moura, D., Seif el-Nasr, M., & Shaw, C. D. (2011). Visualizing and understanding players’ behavior in video games: Discovering patterns and supporting aggregation and comparison. In Proceedings of the 2011 ACM SIGGRAPH Symposium on Video Games (pp. 11–15). New York: ACM Press. doi:10.1145/2037692.2037695.

    Google Scholar 

  • Prensky, M., & Bowers, J. C. (2005). Serious games debate. Serious Games Summit. Washington, DC.

  • R Core Team. (2012). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. Retrieved from http://www.r-project.org

  • Rao, R. (2003). From unstructured data to actionable intelligence. IT Professional, 5(6), 29–35. doi:10.1109/MITP.2003.1254966.

    Article  Google Scholar 

  • Rockoff, J. E., Jacob, B. A., Kane, T. J., & Staiger, D. O. (2008, November 13). Can you recognize an effective teacher when you recruit one? Cambridge, MA: National Bureau of Economic Research. Retrieved from http://www.nber.org/papers/w14485.pdf

  • Rubin, J., & Watson, I. (2010). Similarity-based retrieval and solution re-use policies in the game of Texas Hold’em. In I. Bichindaritz & S. Montani (Eds.), Case-based reasoning, research and development (vol. 6176, pp. 465–479). Berlin, Heidelberg: Springer Berlin Heidelberg. doi:10.1007/978-3-642-14274-1.

  • Seif El-Nasr, M., Drachen, A., & Canossa, A. (Eds.). (2013). Game analytics: Maximizing the value of player data (p. 815). London: Springer. doi:10.1007/978-1-4471-4769-5.

    Google Scholar 

  • Shute, V. J., Masduki, I., Donmez, O., Dennen, V. P., Kim, Y.-J., Jeong, A. C., et al. (2010). Modeling, Assessing, and Supporting Key Competencies Within Game Environments. In D. Ifenthaler, P. Pirnay-Dummer, & N. M. Seel (Eds.), Computer-Based Diagnostics and Systematic Analysis of Knowledge (pp. 281–309). Boston, MA: Springer US. doi:10.1007/978-1-4419-5662-0_15. Retrieved from http://www.springerlink.com/content/t5261055733n18h2/

  • Siemens, G., Gasevic, D., Haythornthwaite, C., Dawson, S., Shum, S. B., Ferguson, R., et al. (2011). Open learning analytics: An integrated & modularized platform. Beaumont, Alberta: Society for Learning Analytics Research (SOLAR). Retrieved from http://solaresearch.org/OpenLearningAnalytics.pdf

  • Thomas, J. M., & DeRosier, M. E. (2010). Toward effective game-based social skills tutoring for children: an evaluation of a social adventure game. Proceedings of the 5th International Conference on the Foundations of Digital Games (FDG ’10). (pp. 217–223). Monterey, CA: ACM.

  • Underwood, J. (2005). Novice and expert performance with a dynamic control task: Scanpaths during a computer game. In G. Underwood (Ed.), Cognitive processes in eye guidance (pp. 303–323). Oxford: Oxford University Press. doi:10.1093/acprof:oso/9780198566816.003.0013.

    Chapter  Google Scholar 

  • Van Eck, R. (2006). Digital game-based learning: It’s not just the digital natives who are restless…. EDUCAUSE Review, 41(2), 16–30. Retrieved from http://www.educause.edu/ir/library/pdf/ERM0620.pdf

  • White, J. W., & Ruttenberg, B. I. (2007). Discriminant function analysis in marine ecology: some oversights and their solutions. Marine Ecology Progress Series, 329, 301–305. doi:10.3354/meps329301.

    Article  Google Scholar 

  • Williams, A. M., & Ford, P. R. (2008). Expertise and expert performance in sport. International Review of Sport and Exercise Psychology, 1(1), 4–18. doi:10.1080/17509840701836867.

    Article  Google Scholar 

  • Winkler, W. E. (1999). The state of record linkage and current research problems (p. 15). Washington, DC: Bureau of the Census. Retrieved from http://www.census.gov/srd/papers/pdf/rr99-04.pdf

  • Zoeller, G. (2010). Development Telemetry in Video Games Projects. Game Developer Conference 2010. San Francisco, CA. Retrieved from http://gdc.gulbsoft.org/talk

Download references

Acknowledgments

This research was made possible in part through funding from the 2009 Defense University Research Instrumentation Program (DURIP) from the U.S. Army Research Office. The authors wished to thank Professor Emerita Sharon Shrock for providing valuable feedback to an earlier draft, and our doctoral students, T. Zhou and I. H. Li, for their assistance in data collection. Screenshot of The Guardian (Fig. 2) is used with permission from the first author.

Author information

Authors and Affiliations

  1. Virtual Environment Lab (V-LAB), Department of Curriculum & Instruction, Southern Illinois University, Carbondale, IL, 62901, USA

    Christian Sebastian Loh

  2. Department of Educational Measurement & Statistics, Southern Illinois University, Carbondale, IL, 62901, USA

    Yanyan Sheng

Authors
  1. Christian Sebastian Loh
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yanyan Sheng
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Christian Sebastian Loh.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Loh, C.S., Sheng, Y. Measuring the (dis-)similarity between expert and novice behaviors as serious games analytics. Educ Inf Technol 20, 5–19 (2015). https://doi.org/10.1007/s10639-013-9263-y

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10639-013-9263-y

Keywords

Search

Navigation