How Many Times Should a Pedagogical Agent Simulation Model Be Run?
When using simulation modeling to explore pedagogical phenomena, there are several issues a designer/practitioner should consider. One of the most important decisions has to do with determining how many runs of a simulation to perform in order to be confident in the results produced by the simulation . With a deterministic model, a single simulation run is adequate. This issue becomes more challenging when part of the simulation model is based on stochastic elements. One of the solutions that has been used to address this challenge in other research communities is the use of Monte Carlo simulation . Within the AIED research community, however, this question of how many times should a pedagogical simulation model be run to produce predictions in which the designer can have confidence has received surprisingly little attention. The aim of this paper is to explore this issue using a pedagogical simulation model, SimDoc, designed to explore longer term mentoring issues . In particular, we demonstrate how to run this simulation model over many iterations until the accumulated results of the iteration runs reach a statistically stable level that matches real world performance but also has appropriate variability among the runs. We believe this approach generalizes beyond our simulation environment and could be applied to other pedagogical simulations and would be especially useful for medium and high fidelity simulations where each run may take a long time.
KeywordsSimulation Simulated learners Longer-term mentoring Lifelong learning
We would like to acknowledge the financial support of the Natural Sciences and Engineering Research Council of Canada for this research. We also would like to thank the University of Saskatchewan for providing (anonymized) data on its Ph.D. programs that we could use to inform the SimDoc simulation.
- 1.Bogdoll, J., Hartmanns, A., Hermanns, H.: Simulation and statistical model checking for modestly nondeterministic models. In: Schmitt, J.B. (ed.) International GI/ITG Conference on Measurement, Modelling, and Evaluation of Computing Systems and Dependability and Fault Tolerance, pp. 249–252. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-28540-0_20CrossRefGoogle Scholar
- 3.Lelei, D.E.K., McCalla, G.: How to use simulation in the design and evaluation of learning environments with self-directed longer-term learners. In: Penstein Rosé, C., et al. (eds.) AIED 2018. LNCS (LNAI), vol. 10947, pp. 253–266. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-93843-1_19CrossRefGoogle Scholar
- 5.Lelei, D.E.K., McCalla, G.I.: The role of simulation in the development of mentoring technology to support longer-term learning. In: The Proceedings of 3rd International Workshop on Intelligent Mentoring Systems Held in Conjunction with the 19th International Conference on Artificial Intelligence in Education (2018)Google Scholar
- 6.VanLehn, K., Ohlsson, S., Nason, R.: Applications of simulated students: an exploration. J. Artif. Intell. Educ. 5(2), 1–42 (1994)Google Scholar
- 8.Laberge, S., Lin, F.: Simulated learners for testing agile teaming in social educational games. In: CEUR Workshop Proceedings, vol. 1432, pp. 65–77 (2015)Google Scholar
- 9.Frost, S., McCalla, G.: Exploring through simulation an instructional planner for dynamic open-ended learning environments. In: Conati, C., Heffernan, N., Mitrovic, A., Verdejo, M.Felisa (eds.) AIED 2015. LNCS (LNAI), vol. 9112, pp. 578–581. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-19773-9_66CrossRefGoogle Scholar
- 10.Lelei, D.E.K., McCalla, G.I.: Exploring the issues in simulating a semi-structured learning environment: the SimGrad doctoral program design. In: The Proceedings of the 2nd Workshop on Simulated Learners at the 17th International Conference on Artificial Intelligence in Education, vol. 5, pp. 11–20 (2015)Google Scholar
- 11.Carlson, R., Keiser, V., Matsuda, N., Koedinger, K.R., Penstein Rosé, C.: Building a conversational SimStudent. In: Cerri, S.A., Clancey, W.J., Papadourakis, G., Panourgia, K. (eds.) ITS 2012. LNCS, vol. 7315, pp. 563–569. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-30950-2_73CrossRefGoogle Scholar
- 12.Booth, J.G., Sarkar, S.: Monte Carlo approximation of bootstrap variances. Am. Stat. 52(4), 354–357 (1998)Google Scholar
- 13.Truong, L.T., Sarvi, M., Currie, G., Garoni, T.M.: How many simulation runs are required to achieve statistically confident results: a case study of simulation-based surrogate safety measures. In: IEEE 18th International Conference on Intelligent Transportation Systems, pp. 274–278 (2015)Google Scholar
- 14.Van Joolingen, W.: Design and implementation of simulation-based discovery environments: the SMISLE solution. J. Artif. Intell. Educ. 7(4), 253–276 (1996)Google Scholar
- 15.Rosenberg-Kima, R.B., Pardos, Z.A.: Is this model for real? Simulating data to reveal the proximity of a model to reality. In: The Proceedings of the 17th International Conference on Artificial Intelligence in Education, pp. 78–87 (2015)Google Scholar
- 16.Weber, G.: Individual selection of examples in an intelligent learning environment. Int. J. Artif. Intell. Educ. 7(1), 3–31 (2015)Google Scholar
- 17.Liu, C.: A simulation-based experience in learning structures of bayesian networks to represent how students learn composite concepts. Int. J. Artif. Intell. Educ. 18(3), 237–285 (2008)Google Scholar
- 19.Desmarais, M.C., Pu, X.: A Bayesian student model without hidden nodes and its comparison with item response theory. Int. J. Artif. Intell. Educ. 15(4), 291–323 (2005)Google Scholar
- 20.Yeh, A., Corp, M.: More accurate tests for the statistical significance of result differences. In: Proceedings of the 18th International Conference on Computational Linguistics, pp. 947–953 (2000)Google Scholar
- 21.Erickson, G., Frost, S., Bateman, S., McCalla, G.: Using the ecological approach to create simulations of learning environments. In: Lane, H.C., Yacef, K., Mostow, J., Pavlik, P. (eds.) AIED 2013. LNCS (LNAI), vol. 7926, pp. 411–420. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-39112-5_42CrossRefGoogle Scholar
- 22.Riedesel, M.A., Zimmerman, N., Baker, R., Titchener, T., Cooper, J.: Using a model for learning and memory to simulate learner response in spaced practice. In: The Proceedings of the 18th International Conference on Artificial Intelligence in Education, pp. 644–649 (2017)Google Scholar
- 25.Ritter, F.E., Schoelles, M.J., Quigley, K.S., Klein, L.C.: Determining the number of simulation runs treating simulations as theories by not sampling their behavior. In: Rothrock, L., Narayanan, S. (eds.) Human-in-the-Loop Simulations: Methods and Practice, pp. 97–116. Springer, London (2011). https://doi.org/10.1007/978-0-85729-883-6_5CrossRefGoogle Scholar