Skip to main content
SpringerLink
Log in

Skill Check: Some Considerations on the Evaluation of Gamemastering Models for Role-Playing Games

  • Conference paper
  • First Online:
Games and Learning Alliance (GALA 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14475))

Included in the following conference series:

Abstract

In role-playing games a Game Master (GM) is the player in charge of the game, who must design the challenges the players face and narrate the outcomes of their actions. In this work we discuss some challenges to model GMs from an Interactive Storytelling and Natural Language Processing perspective. Following those challenges we propose three test categories to evaluate such dialogue systems, and we use them to test ChatGPT, Bard and OpenAssistant as out-of-the-box GMs.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 119.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    For example “Call of Cthulhu: The Official Video Game”, an adaptation of the RPG.

  2. 2.

    An interesting example of this is “The Stanley Parable”, a novel videogame that makes the players think about free will and the impact of their actions.

  3. 3.

    https://openai.com/blog/chatgpt.

  4. 4.

    https://bard.google.com.

  5. 5.

    https://huggingface.co/chat/.

  6. 6.

    https://www.huntersentertainment.com/alice-is-missing.

  7. 7.

    “ChatGPT Mar 14 Version. Free Research Preview”.

  8. 8.

    Bard Experimental. Accessed on the 17th of July, 2023.

  9. 9.

    “Model: OpenAssistant/oasst-sft-6-llama-30b”. Accessed on the 14th of July, 2023.

  10. 10.

    https://github.com/sgongora27/skill-check-GM-tests.

  11. 11.

    This is evidenced by the massive amount of adventures published for RPGs with this theme, such as Dungeons & Dragons or Pathfinder.

References

  1. Acharya, D., Mateas, M., Wardrip-Fruin, N.: Story improvisation in tabletop roleplaying games: towards a computational assistant for game masters. In: 2021 IEEE Conference on Games (CoG), pp. 01–08 (2021). https://doi.org/10.1109/CoG52621.2021.9619006

  2. Arenas, D.L., Viduani, A., Araujo, R.B.: Therapeutic use of role-playing game (RPG) in mental health: a scoping review. Simul. Gaming 53(3), 285–311 (2022)

    Article  Google Scholar 

  3. Behnke, H., Fomicheva, M., Specia, L.: Bias mitigation in machine translation quality estimation. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1475–1487. Association for Computational Linguistics, Dublin (2022). https://doi.org/10.18653/v1/2022.acl-long.104

  4. Bender, E.M., Gebru, T., McMillan-Major, A., Shmitchell, S.: On the dangers of stochastic parrots: can language models be too big? In: Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT 2021, pp. 610–623. Association for Computing Machinery, New York (2021). https://doi.org/10.1145/3442188.3445922

  5. Bender, E.M., Koller, A.: Climbing towards NLU: on meaning, form, and understanding in the age of data. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5185–5198. Association for Computational Linguistics, Online (2020). https://doi.org/10.18653/v1/2020.acl-main.463

  6. Benotti, L.: Implicature as an interactive process. L’implicature comme un Processus Interactif. Ph.D. thesis, Henri Poincaré University, Nancy, France (2010)

    Google Scholar 

  7. Botelho, L.M.: A guided journey through non-interactive automatic story generation (2021)

    Google Scholar 

  8. Callison-Burch, C., Tomar, G.S., Martin, L., Ippolito, D., Bailis, S., Reitter, D.: Dungeons and dragons as a dialog challenge for artificial intelligence. In: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 9379–9393. Association for Computational Linguistics, Abu Dhabi (2022)

    Google Scholar 

  9. Chen, H., Liu, X., Yin, D., Tang, J.: A survey on dialogue systems: recent advances and new frontiers. SIGKDD Explor. Newsl. 19(2), 25–35 (2017). https://doi.org/10.1145/3166054.3166058

    Article  Google Scholar 

  10. Colton, S., Wiggins, G.A.: Computational creativity: the final frontier? In: European Conference on Artificial Intelligence (2012)

    Google Scholar 

  11. Davis, E., Marcus, G.: Commonsense reasoning and commonsense knowledge in artificial intelligence. Commun. ACM 58(9), 92–103 (2015)

    Article  Google Scholar 

  12. Durall, J., Perrin, S.: Basic Roleplaying: Universal Game Engine. Chaosium Inc. (2023)

    Google Scholar 

  13. Gatsakou, C., Bardis, N., Drigas, A.: Role playing vs RPGs as teaching strategies in educational procedure. Technium Soc. Sci. J. 26, 186 (2021)

    Google Scholar 

  14. Ji, Z., et al.: Survey of hallucination in natural language generation. ACM Comput. Surv. 55(12), 1–38 (2023). https://doi.org/10.1145/3571730

    Article  Google Scholar 

  15. Jordanous, A.: A standardised procedure for evaluating creative systems: computational creativity evaluation based on what it is to be creative. Cogn. Comput. 4(3), 246–279 (2012). https://doi.org/10.1007/s12559-012-9156-1

    Article  Google Scholar 

  16. Köpf, A., et al.: OpenAssistant conversations - democratizing large language model alignment (2023)

    Google Scholar 

  17. Louis, A., Sutton, C.: Deep dungeons and dragons: learning character-action interactions from role-playing game transcripts. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pp. 708–713. Association for Computational Linguistics, New Orleans (2018). https://doi.org/10.18653/v1/N18-2111

  18. Martin, L.J., Harrison, B., Riedl, M.O.: Improvisational computational storytelling in open worlds. In: Nack, F., Gordon, A.S. (eds.) ICIDS 2016. LNCS, vol. 10045, pp. 73–84. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-48279-8_7

    Chapter  Google Scholar 

  19. Martin, L.J., Sood, S., Riedl, M.O.: Dungeons and DQNs: toward reinforcement learning agents that play tabletop roleplaying games. In: INT/WICED@AIIDE (2018)

    Google Scholar 

  20. Oñate, A., Méndez, G., Gervás, P.: Emolift: elevator conversations based on emotions. In: International Conference on Innovative Computing and Cloud Computing (2019)

    Google Scholar 

  21. Peinado, F., Gervás, P.: Transferring game mastering laws to interactive digital storytelling. In: Göbel, S., et al. (eds.) TIDSE 2004. LNCS, vol. 3105, pp. 48–54. Springer, Heidelberg (2004). https://doi.org/10.1007/978-3-540-27797-2_7

    Chapter  Google Scholar 

  22. Peinado, F., Gervás, P.: Automatic direction of interactive storytelling: formalizing the game master paradigm. In: Cavazza, M., Donikian, S. (eds.) ICVS 2007. LNCS, vol. 4871, pp. 196–201. Springer, Heidelberg (2007). https://doi.org/10.1007/978-3-540-77039-8_18

    Chapter  Google Scholar 

  23. Qin, C., Zhang, A., Zhang, Z., Chen, J., Yasunaga, M., Yang, D.: Is ChatGPT a general-purpose natural language processing task solver? (2023). https://doi.org/10.48550/ARXIV.2302.06476

  24. Rameshkumar, R., Bailey, P.: Storytelling with dialogue: a critical Role Dungeons and Dragons Dataset. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5121–5134. Association for Computational Linguistics, Online (2020). https://doi.org/10.18653/v1/2020.acl-main.459

  25. Riedl, M.O., Bulitko, V.: Interactive narrative: an intelligent systems approach. AI Mag. 34(1), 67 (2012). https://doi.org/10.1609/aimag.v34i1.2449

    Article  Google Scholar 

  26. Sap, M., Shwartz, V., Bosselut, A., Choi, Y., Roth, D.: Commonsense reasoning for natural language processing. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts, pp. 27–33. Association for Computational Linguistics, Online (2020). https://doi.org/10.18653/v1/2020.acl-tutorials.7

  27. Tychsen, A., Hitchens, M.: The many faces of role-playing games. Int. J. Role-Play. 1(1), 3–21 (2009)

    Google Scholar 

  28. Zhou, P., et al.: An AI dungeon master’s guide: learning to converse and guide with intents and theory-of-mind in dungeons and dragons (2022). https://doi.org/10.48550/ARXIV.2212.10060

  29. Zhu, A., Aggarwal, K., Feng, A., Martin, L., Callison-Burch, C.: FIREBALL: a dataset of dungeons and dragons actual-play with structured game state information. In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 4171–4193. Association for Computational Linguistics, Toronto (2023)

    Google Scholar 

  30. Zhu, A., Martin, L.J., Head, A., Callison-Burch, C.: CALYPSO: LLMs as dungeon masters’ assistants. UMBC Faculty Collection (2023)

    Google Scholar 

Download references

Acknowledgements

This paper has been partially funded by ANII (Uruguayan Innovation and Research National Agency), Grant No. \(POS\_NAC\_2022\_1\_173659\) and by the project CANTOR: Automated Composition of Personal Narratives as an aid for Occupational Therapy based on Reminescence, Grant No. \(PID2019-108927RB-I00\) (Spanish Ministry of Science and Innovation).

Author information

Authors and Affiliations

  1. Universidad de la República, Montevideo, Uruguay

    Santiago Góngora & Luis Chiruzzo

  2. Universidad Complutense de Madrid, Madrid, Spain

    Gonzalo Méndez & Pablo Gervás

Authors
  1. Santiago Góngora
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Luis Chiruzzo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Gonzalo Méndez
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Pablo Gervás
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Santiago Góngora .

Editor information

Editors and Affiliations

  1. Technological University Dublin, Dublin, Ireland

    Pierpaolo Dondio

  2. Technological University Dublin, Dublin, Ireland

    Mariana Rocha

  3. University of Galway, Galway, Ireland

    Attracta Brennan

  4. Berlin School of Economics and Law, Berlin, Germany

    Avo Schönbohm

  5. CAPTRS, Austin, TX, USA

    Francesca de Rosa

  6. Tampere University, Tampere, Finland

    Antti Koskinen

  7. University of Genova, Genoa, Italy

    Francesco Bellotti

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Góngora, S., Chiruzzo, L., Méndez, G., Gervás, P. (2024). Skill Check: Some Considerations on the Evaluation of Gamemastering Models for Role-Playing Games. In: Dondio, P., et al. Games and Learning Alliance. GALA 2023. Lecture Notes in Computer Science, vol 14475. Springer, Cham. https://doi.org/10.1007/978-3-031-49065-1_27

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-49065-1_27

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-49064-4

  • Online ISBN: 978-3-031-49065-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation