Skip to main content
SpringerLink
Log in

DeepSkill: A methodology for measuring teams’ skills in massively multiplayer online games

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

The game industry is witnessing a significant trend of players toward massively multiplayer online games (MMO). Players are keen on forming teams and cooperating/competing in these games. Real-time measurement of players’ performance is one of the subjects of researchers’ attention to dynamically adjust the game difficulty and immerse players in the game. However, our extensive studies show that real-time measuring of teams’ skill levels has received much less attention. In this paper, a general real-time method called DeepSkill is proposed to measure the MMOs teams’ skills directly using players’ gameplay raw low-level data. The proposed method, which is based on the evidence-centered assessment design, was tested under six different configurations using popular machine learning techniques, including deep neural network (DNN), extreme gradient boosting (XGBoost), light gradient boosting machine (LightGBM), CatBoost, random forest (RF), and linear support vector regression (LinearSVR). According to the results, the proposed method provides accurate skill estimations and expertise level classifications. Specifically, Deepskill’s DNN-based evidence model provided the lowest mean absolute error of 0.09 in team skill estimation. Additionally, the proposed method achieved an accuracy of 0.973 in classifying the teams’ expertise level for the expert-novice classification task. Furthermore, a cost-effectiveness analysis was performed on the two top-performing evidence models. The LightGBM-based evidence model yielded the best results in both training and prediction phases in terms of low resource consumption alongside considerable accuracy.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Availability of data and materials

Gameplay data that supports the findings of this study has been deposited in Kaggle with the accession URL: https://www.kaggle.com/datasets/skihikingkevin/pubg-match-deaths.

References

  1. Anwar SM, Saeed SMU, Majid M (2016) Classification of expert-novice level of mobile game players using electroencephalography. In: 2016 International Conference on Frontiers of Information Technology (FIT). IEEE, pp. 315–318

  2. Anwar SM, Saeed SMU, Majid M, Usman S, Mehmood CA, Liu W (2018) A game player expertise level classification system using electroencephalography (eeg). Appl Sci 8(1):18

    Article  Google Scholar 

  3. Bartle RA (2004) Designing virtual worlds. New Riders

  4. Binkley M, Erstad O, Herman J, Raizen S, Ripley M, Miller-Ricci M, Rumble M (2011) Defining twenty-first century skills. pp 17–66. https://doi.org/10.1007/978-94-007-2324-52

  5. Breiman L (2001) Random forests. Mach Learn 45:5–32

    Article  Google Scholar 

  6. Chang CC, Lin CJ (2011) Libsvm: a library for support vector machines. ACM Transactions on Intelligent Systems and Technology (TIST) 2(3):1–27

    Article  ADS  Google Scholar 

  7. Chen T, He T, Benesty M, Khotilovich V, Tang Y, Cho H, et al (2015) Xgboost: extreme gradient boosting. R package version 0.4-2. 1(4):1–4

  8. Csikszentmihalyi M (1975) Beyond Boredom and Anxiety. The Jossey-Bass behavioral science series. Jossey-Bass Publishers (1975). https://books.google.com/books?id=afdGAAAAMAAJ

  9. Csikszentmihalyi M (1990) Flow: The psychology of optimal experience. vol 1990, Harper & Row New York

  10. Csikszentmihalyi M (1997) Finding flow: The psychology of engagement with everyday life. Basic Books

  11. Dargan S, Kumar M, Ayyagari MR, Kumar G (2020) A survey of deep learning and its applications: a new paradigm to machine learning. Archives of Computational Methods in Engineering 27:1071–1092

    Article  MathSciNet  Google Scholar 

  12. de Groot AD (1978) Thought and Choice in Chess. De Gruyter Mouton, Berlin, Boston. https://doi.org/10.1515/9783110800647

  13. Delalleau O, Contal E, Thibodeau-Laufer E, Ferrari RC, Bengio Y, Zhang F (2012) Beyond skill rating: Advanced matchmaking in ghost recon online. IEEE Transactions on Computational Intelligence and AI in Games 4(3):167–177

    Article  Google Scholar 

  14. Drachen A, El-Nasr MS, Canossa A (2013) Game analytics–the basics. In: Game analytics. Springer, pp 13–40

  15. Dreyfus SE (2004) The five-stage model of adult skill acquisition. Bulletin of science, technology & society 24(3):17–181

  16. Dreyfus SE (2004) The five-stage model of adult skill acquisition. Bulletin of science, technology & society 24(3):17–181

    Google Scholar 

  17. Dreyfus SE, Dreyfus HL (1980) A five-stage model of the mental activities involved in directed skill acquisition. California Univ Berkeley Operations Research Center, Tech. rep

    Book  Google Scholar 

  18. Ehatisham-ul Haq M, Arsalan A, Raheel A, Anwar SM (2021) Expert-novice classification of mobile game player using smartphone inertial sensors. Expert Systems with Applications 174:114700

    Article  Google Scholar 

  19. Fan RE, Chang KW, Hsieh CJ, Wang XR, Lin CJ (2008) Liblinear: A library for large linear classification. the Journal of Machine Learning Research 9:1871–1874

  20. Ericsson KA, Prietula MJ, Cokely ET et al (2007) The making of an expert. Harvard business review 85(7/8):114

    PubMed  Google Scholar 

  21. Glickman ME (1995) The glicko system. Boston University 16:16–17

  22. Glickman ME (2012) Example of the glicko-2 system. Boston University pp 1–6

  23. Goodfellow I, Bengio Y, Courville A (2016) Deep learning. MIT press

  24. Glickman ME (1995) The glicko system. Boston University 16:16–17

    Google Scholar 

  25. de Groot AD (1978) Thought and Choice in Chess. De Gruyter Mouton, Berlin, Boston. DOI https://doi.org/10.1515/9783110800647.https://doi.org/10.1515/9783110800647

  26. Ehatisham-ul Haq M, Arsalan A, Raheel A, Anwar SM (2021) Expert-novice classification of mobile game player using smartphone inertial sensors. Expert Systems with Applications 174:114700

  27. Haubruck P, Nickel F, Ober J, Walker T, Bergdolt C, Friedrich M, Müller-Stich BP, Forchheim F, Fischer C, Schmidmaier G, et al (2018) Evaluation of app-based serious gaming as a training method in teaching chest tube insertion to medical students: randomized controlled trial. Journal of medical Internet research 20(5) e9956

  28. Herbrich R, Minka T, Graepel T (2006) Trueskill\(^{\text{TM}}\): A bayesian skill rating system. In: Pro-ceedings of the 19th International Conference on Neural Information Processing Systems, NIPS’06. MIT Press, Cambridge, MA, USA, pp 569–576

  29. Ho CH, Lin CJ (2012) Large-scale linear support vector regression. The Journal of Machine Learning Research 13(1):3323–3348

  30. Ho CH, Lin CJ (2012) Large-scale linear support vector regression. The Journal of Machine Learning Research 13(1):3323–3348

    MathSciNet  Google Scholar 

  31. Hodge VJ, Devlin S, Sephton N, Block F, Cowling PI, Drachen A (2019) Win prediction in multiplayer esports: Live professional match prediction. IEEE Transactions on Games 13(4):368–379

    Article  Google Scholar 

  32. Indulkar Y (2021) Pubg winner ranking prediction using r interface ’h2o’ scable machine learning platform. In: 2021 International Conference on Emerging Smart Computing and Informatics (ESCI). IEEE, pp 300–305

  33. Jaccard P (1912) The distribution of the flora in the alpine zone. 1. New phytologist 11(2):37–50

  34. Johnson ES, Crawford A, Moylan LA, Zheng Y (2018) Using evidence-centered design to create a special educator observation system. Issues and Practice, Educational Measurement

    Book  Google Scholar 

  35. Kamikokuryo K, Haga T, Venture G, Hernandez V (2022) Adversarial autoencoder and multi-armed bandit for dynamic difficulty adjustment in immersive virtual reality for rehabilitation: Application to hand movement. Sensors 22(12):4499

    Article  PubMed  PubMed Central  ADS  Google Scholar 

  36. Ke G, Meng Q, Finley T, Wang T, Chen W, Ma W, Ye Q, Liu TY (2017) Lightgbm: A highly efficient gradient boosting decision tree. Advances in neural information processing systems 30:3146–3154

    Google Scholar 

  37. Kingma DP, Ba J (2014) Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980

  38. Larose DT, Larose CD (2014) Discovering knowledge in data: an introduction to data mining, John Wiley & Sons, vol 4

  39. Loh CS, Sheng Y (2013) Measuring the (dis-)similarity between expert and novice behaviors as serious games analytics. Education and Information Technologies. pp 1–15. https://www.scopus.com

  40. Loh CS, Sheng Y (2014) Maximum similarity index (msi): A metric to differentiate the performance of novices vs. multiple-experts in serious games. Computers in Human Behavior 39:322–330. https://www.scopus.com

  41. Loh CS, Sheng Y, Li I (2015) Predicting expert-novice performance as serious games analytics with objective-oriented and navigational action sequences. Computers in Human Behavior 49:147–155 https://www.scopus.com

  42. Louppe G (2014) Understanding random forests: From theory to practice. arXiv preprint arXiv:1407.7502

  43. Minka T, Cleven R, Zaykov Y (2018) Trueskill 2: An improved bayesian skill rating system. Tech. Rep. MSR-TR-2018-8, Microsoft

  44. Mishra P, Foster A (2007) The claims of games: A comprehensive review and directions for future research. In: Society for Information Technology & Teacher Education International Conference. Association for the Advancement of Computing in Education (AACE) pp 2227–2232

  45. Mislevy RJ (2013) Evidence-centered design for simulation-based assessment. Military medicine 178(10):107–114

    Article  PubMed  Google Scholar 

  46. Moschovitis P, Denisova A (2022) Keep calm and aim for the head: Biofeedback-controlled dynamic difficulty adjustment in a horror game. IEEE Transactions on Games

  47. Nery Bandeira I, Dullens VF, Machado TV, Oliveira RRA, Castanho CD, Sarmet MM, et al (2022) Dynamic difficulty adjustment in digital games: Comparative study between two algorithms using electrodermal activity data. In: International Conference on Human-Computer Interaction. Springer, pp 69–83

  48. Paraschos PD, Koulouriotis DE (2022) Game difficulty adaptation and experience personalization: A literature review. International Journal of Human-Computer Interaction pp 1–22

  49. Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, Blondel M, Prettenhofer P, Weiss R, Dubourg V, Vanderplas J, Passos A, Cournapeau D, Brucher M, Perrot M, Duchesnay E (2011) Scikit-learn: Machine learning in Python. Journal of Machine Learning Research 12:2825–2830

    MathSciNet  Google Scholar 

  50. Pei K (2018) Pubg match deaths and statistics. Kaggle. https://www.kaggle.com/datasets/skihikingkevin/pubg-match-deaths

  51. Prokhorenkova L, Gusev G, Vorobev A, Dorogush AV, Gulin A (2018) Catboost: unbiased boosting with categorical features. Advances in neural information processing systems 31

  52. Reis S, Reis LP, Lau N (2019) Player engagement enhancement with video games. In: Rocha Á, Adeli H, Reis LP, Costanzo S (eds) New Knowledge in Information Systems and Technologies. Springer International Publishing, Cham, pp 263–272

    Chapter  Google Scholar 

  53. Schell J (2008) The Art of Game Design: A Book of Lenses. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA

    Book  Google Scholar 

  54. Shaffer DW, Halverson R, Squire KR, Gee JP (2005) Video games and the future of learning. wcer working paper no. 2005-4. Wisconsin Center for Education Research (NJ1)

  55. Silva MP, do Nascimento Silva V, Chaimowicz L (2017) Dynamic difficulty adjustment on moba games. Entertainment Computing 18:103–123

    Article  Google Scholar 

  56. Sourmelis T, Ioannou A, Zaphiris P (2017) Massively multiplayer online role playing games (mmorpgs) and the 21st century skills: A comprehensive research review from 2010 to 2016. Computers in Human Behavior 67:41–48. https://doi.org/10.1016/j.chb.2016.10.020

    Article  Google Scholar 

  57. Spatharioti SE, Wylie S, Cooper S (2021) Exploring q-learning for adaptive difficulty in a tile-based image labeling game. In: 2021 IEEE Conference on Games (CoG). IEEE, pp 1–8

  58. Stein A, Yotam Y, Puzis R, Shani G, Taieb-Maimon M (2018) Eeg-triggered dynamic difficulty adjustment for multiplayer games. Entertainment Computing 25:14–25

    Article  Google Scholar 

  59. Taylor N, de Castell S, Jenson J, Hurley R, Lakkaraju K, Sukthankar G, Wigand R (2018) Management (im) material: Negotiating leadership in virtual worlds. Social Interactions in Virtual Worlds: An Interdisciplinary Perspective pp 162–190

  60. Tukey JW et al (1997) Exploratory data analysis, vol 2. Reading, MA

    Google Scholar 

  61. Wang X, Zhao Y, Pourpanah F (2020) Recent advances in deep learning. International Journal of Machine Learning and Cybernetics 11:747–750

    Article  Google Scholar 

  62. Zamani Joharestani M, Cao C, Ni X, Bashir B, Talebiesfandarani S (2019) Pm2. 5 prediction based on random forest, xgboost, and deep learning using multisource remote sensing data. Atmosphere 10(7):373

Download references

Funding

No funds, grants, or other support were received.

Author information

Authors and Affiliations

  1. Faculty of Computer Engineering, University of Isfahan, Isfahan, Iran

    Mohammad Mahdi Rezapour, Afsaneh Fatemi & Mohammad Ali Nematbakhsh

Authors
  1. Mohammad Mahdi Rezapour
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Afsaneh Fatemi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mohammad Ali Nematbakhsh
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Afsaneh Fatemi.

Ethics declarations

Conflicts of interest

The authors have no conflicts of interest to declare that are relevant to the content of this article.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Rezapour, M.M., Fatemi, A. & Nematbakhsh, M.A. DeepSkill: A methodology for measuring teams’ skills in massively multiplayer online games. Multimed Tools Appl 83, 31049–31079 (2024). https://doi.org/10.1007/s11042-023-15796-x

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-15796-x

Keywords

Search

Navigation