How Many Times Should a Pedagogical Agent Simulation Model Be Run?

  • David Edgar Kiprop LeleiEmail author
  • Gordon McCalla
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11625)


When using simulation modeling to explore pedagogical phenomena, there are several issues a designer/practitioner should consider. One of the most important decisions has to do with determining how many runs of a simulation to perform in order to be confident in the results produced by the simulation [1]. With a deterministic model, a single simulation run is adequate. This issue becomes more challenging when part of the simulation model is based on stochastic elements. One of the solutions that has been used to address this challenge in other research communities is the use of Monte Carlo simulation [2]. Within the AIED research community, however, this question of how many times should a pedagogical simulation model be run to produce predictions in which the designer can have confidence has received surprisingly little attention. The aim of this paper is to explore this issue using a pedagogical simulation model, SimDoc, designed to explore longer term mentoring issues [3]. In particular, we demonstrate how to run this simulation model over many iterations until the accumulated results of the iteration runs reach a statistically stable level that matches real world performance but also has appropriate variability among the runs. We believe this approach generalizes beyond our simulation environment and could be applied to other pedagogical simulations and would be especially useful for medium and high fidelity simulations where each run may take a long time.


Simulation Simulated learners Longer-term mentoring Lifelong learning 



We would like to acknowledge the financial support of the Natural Sciences and Engineering Research Council of Canada for this research. We also would like to thank the University of Saskatchewan for providing (anonymized) data on its Ph.D. programs that we could use to inform the SimDoc simulation.


  1. 1.
    Bogdoll, J., Hartmanns, A., Hermanns, H.: Simulation and statistical model checking for modestly nondeterministic models. In: Schmitt, J.B. (ed.) International GI/ITG Conference on Measurement, Modelling, and Evaluation of Computing Systems and Dependability and Fault Tolerance, pp. 249–252. Springer, Heidelberg (2012). Scholar
  2. 2.
    Koehler, E., Brown, E., Haneuse, S.J.-P.A.: On the assessment of Monte Carlo error in simulation-based statistical analyses. Am. Stat. 63(2), 155–162 (2009)MathSciNetzbMATHCrossRefGoogle Scholar
  3. 3.
    Lelei, D.E.K., McCalla, G.: How to use simulation in the design and evaluation of learning environments with self-directed longer-term learners. In: Penstein Rosé, C., et al. (eds.) AIED 2018. LNCS (LNAI), vol. 10947, pp. 253–266. Springer, Cham (2018). Scholar
  4. 4.
    Lane, H.C., McCalla, G.I., Looi, C.-K., Bull, S.: The next 25 years: how advanced interactive learning technologies will change the world. Int. J. Artif. Intell. Educ. 26(1), 539–543 (2016)CrossRefGoogle Scholar
  5. 5.
    Lelei, D.E.K., McCalla, G.I.: The role of simulation in the development of mentoring technology to support longer-term learning. In: The Proceedings of 3rd International Workshop on Intelligent Mentoring Systems Held in Conjunction with the 19th International Conference on Artificial Intelligence in Education (2018)Google Scholar
  6. 6.
    VanLehn, K., Ohlsson, S., Nason, R.: Applications of simulated students: an exploration. J. Artif. Intell. Educ. 5(2), 1–42 (1994)Google Scholar
  7. 7.
    Dorça, F.: Implementation and use of simulated students for test and validation of new adaptive educational systems: a practical insight. Int. J. Artif. Intell. Educ. 25(3), 319–345 (2015)CrossRefGoogle Scholar
  8. 8.
    Laberge, S., Lin, F.: Simulated learners for testing agile teaming in social educational games. In: CEUR Workshop Proceedings, vol. 1432, pp. 65–77 (2015)Google Scholar
  9. 9.
    Frost, S., McCalla, G.: Exploring through simulation an instructional planner for dynamic open-ended learning environments. In: Conati, C., Heffernan, N., Mitrovic, A., Verdejo, M.Felisa (eds.) AIED 2015. LNCS (LNAI), vol. 9112, pp. 578–581. Springer, Cham (2015). Scholar
  10. 10.
    Lelei, D.E.K., McCalla, G.I.: Exploring the issues in simulating a semi-structured learning environment: the SimGrad doctoral program design. In: The Proceedings of the 2nd Workshop on Simulated Learners at the 17th International Conference on Artificial Intelligence in Education, vol. 5, pp. 11–20 (2015)Google Scholar
  11. 11.
    Carlson, R., Keiser, V., Matsuda, N., Koedinger, K.R., Penstein Rosé, C.: Building a conversational SimStudent. In: Cerri, S.A., Clancey, W.J., Papadourakis, G., Panourgia, K. (eds.) ITS 2012. LNCS, vol. 7315, pp. 563–569. Springer, Heidelberg (2012). Scholar
  12. 12.
    Booth, J.G., Sarkar, S.: Monte Carlo approximation of bootstrap variances. Am. Stat. 52(4), 354–357 (1998)Google Scholar
  13. 13.
    Truong, L.T., Sarvi, M., Currie, G., Garoni, T.M.: How many simulation runs are required to achieve statistically confident results: a case study of simulation-based surrogate safety measures. In: IEEE 18th International Conference on Intelligent Transportation Systems, pp. 274–278 (2015)Google Scholar
  14. 14.
    Van Joolingen, W.: Design and implementation of simulation-based discovery environments: the SMISLE solution. J. Artif. Intell. Educ. 7(4), 253–276 (1996)Google Scholar
  15. 15.
    Rosenberg-Kima, R.B., Pardos, Z.A.: Is this model for real? Simulating data to reveal the proximity of a model to reality. In: The Proceedings of the 17th International Conference on Artificial Intelligence in Education, pp. 78–87 (2015)Google Scholar
  16. 16.
    Weber, G.: Individual selection of examples in an intelligent learning environment. Int. J. Artif. Intell. Educ. 7(1), 3–31 (2015)Google Scholar
  17. 17.
    Liu, C.: A simulation-based experience in learning structures of bayesian networks to represent how students learn composite concepts. Int. J. Artif. Intell. Educ. 18(3), 237–285 (2008)Google Scholar
  18. 18.
    Dzikovska, M., Steinhauser, N., Farrow, E., Moore, J., Campbell, G.: BEETLE II: deep natural language understanding and automatic feedback generation for intelligent tutoring in basic electricity and electronics. Int. J. Artif. Intell. Educ. 24(3), 284–332 (2014)CrossRefGoogle Scholar
  19. 19.
    Desmarais, M.C., Pu, X.: A Bayesian student model without hidden nodes and its comparison with item response theory. Int. J. Artif. Intell. Educ. 15(4), 291–323 (2005)Google Scholar
  20. 20.
    Yeh, A., Corp, M.: More accurate tests for the statistical significance of result differences. In: Proceedings of the 18th International Conference on Computational Linguistics, pp. 947–953 (2000)Google Scholar
  21. 21.
    Erickson, G., Frost, S., Bateman, S., McCalla, G.: Using the ecological approach to create simulations of learning environments. In: Lane, H.C., Yacef, K., Mostow, J., Pavlik, P. (eds.) AIED 2013. LNCS (LNAI), vol. 7926, pp. 411–420. Springer, Heidelberg (2013). Scholar
  22. 22.
    Riedesel, M.A., Zimmerman, N., Baker, R., Titchener, T., Cooper, J.: Using a model for learning and memory to simulate learner response in spaced practice. In: The Proceedings of the 18th International Conference on Artificial Intelligence in Education, pp. 644–649 (2017)Google Scholar
  23. 23.
    Heath, T.: A quantitative analysis of PhD students’ views of supervision. High. Educ. Res. Dev. 21, 37–41 (2002)CrossRefGoogle Scholar
  24. 24.
    Gatfield, T.: An investigation into PhD supervisory management styles: development of a dynamic conceptual model and its managerial implications. J. High. Educ. Policy Manage. 27(3), 311–325 (2005)CrossRefGoogle Scholar
  25. 25.
    Ritter, F.E., Schoelles, M.J., Quigley, K.S., Klein, L.C.: Determining the number of simulation runs treating simulations as theories by not sampling their behavior. In: Rothrock, L., Narayanan, S. (eds.) Human-in-the-Loop Simulations: Methods and Practice, pp. 97–116. Springer, London (2011). Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.ARIES Lab., Department of Computer ScienceUniversity of SaskatchewanSaskatoonCanada

Personalised recommendations