Abstract
How many times should a simulation be run to generate valid predictions? With a deterministic simulation, the answer simply is just once. With a stochastic simulation, the answer is more complex. Different researchers have proposed and used different heuristics. A review of the models presented at a conference on cognitive modeling illustrates the range of solutions and problems in this area. We present the argument that because the simulation is a theory, not data, it should not so much be sampled but run enough times to provide stable predictions of performance and the variance of performance. This applies to both pure simulations as well as human-in-the-loop simulations. We demonstrate the importance of running the simulation until it has stable performance as defined by the effect size of interest. When runs are expensive we suggest a minimum number of runs based on power calculations; when runs are inexpensive we suggest a maximum necessary number of runs. We also suggest how to adjust the number of runs for different effect sizes of interest.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
The standard error of the mean is a standard statistical measure of how well known the mean is, and it is explained in more detail below.
- 2.
For example, http://acs.ist.psu.edu/nottingham/eccm98/home.html
- 3.
Papers with two studies had each study counted 0.5. Papers that were not simple, that examined complex data, e.g., language corpora, or that presented only tools or theoretical points, are not included
- 4.
The parameter is EGN in ACT-R 5, and EGS in ACT-R 6.
References
Anderson JR (2007) How can the human mind exist in the physical universe? Oxford University Press, New York, NY
Anderson JR, Lebiere C (1998) The atomic components of thought. Erlbaum, Mahwah, NJ
Ball J, Myers C, Heiberg A, Cooke NJ, Matessa M, Freiman M et al (2010) The Synthetic Teammate Project. Comput Math Organ Sci 16:271–299
Best B, Fincham J, Gluck K, Gunzelmann G, Krusmark MA (2008) Efficient use of large-scale computational resources. In: Proceedings of the seventeenth conference on behavior representation in modeling and simulation. Simulation Interoperability Standards Organization, Orlando, FL, pp 180−181
Byrne MD (2003) Cognitive architecture. In: Jacko J, Sears A (eds) The human–computer interaction handbook: fundamentals, evolving technologies and emerging applications. Erlbaum, Mahwah, NJ, pp 97–117
Cohen J (1988) Statistical power analysis for the behavioral sciences, 2nd edn. Lawrence Erlbaum, Hillsdale, NJ
Grant DA (1962) Testing the null hypothesis and the strategy and tactics of investigating theoretical models. Psychol Rev 69(1):54–61
Howell DC (1987) Statistical methods for psychology, 2nd edn. Duxbury Press, Boston, MA
Kieras DE (2003) Model-based evaluation. In: Jacko J, Sears A (eds) Handbook for human-computer interaction. Erlbaum, Mahwah, NJ, pp 1139–1151
Kieras DE, Wood SD, Meyer DE (1997) Predictive engineering models based on the EPIC architecture for a multimodel high-performance human-computer interaction task. Trans Comput Hum Interact 4(3):230–275
Kirschbaum C, Pirke K-M, Hellhammer DH (1993) The Trier social stress test: a tool for investigating psychobiological stress responses in a laboratory setting. Neuropsychobiology 28:76–81
Laird JE, Newell A, Rosenbloom PS (1987) Soar: an architecture for general intelligence. Artif Intell 33(1):1–64
Lovett MC, Daily LZ, Reder LM (2000) A source activation theory of working memory: cross-task prediction of performance in ACT-R. J Cogn Syst Res 1:99–118
Lovett MC, Schunn C, Lebiere C, Munro P (eds) (2004) Proceedings of the sixth international conference on cognitive modelling, ICCM 2004. Mahwah NJ: Erlbaum. http://www.lrdc.pitt.edu/schunn/ICCM2004/proceedings/schedule.htm
Mathews G (2001) Levels of transaction: a cognitive science framework for operator stress. In: Hancock PA, Desmond PA (eds) Stress, workload, and fatigue. Erlbaum, Mahwah, NJ
Miller CS, Laird JE (1996) Accounting for graded performance within a discrete search framework. Cogn Sci 20:499–537
Newell A (1990) Unified theories of cognition. Harvard University Press, Cambridge, MA
Quigley KS, Feldman Barrett LF, Weinstein S (2002) Cardiovascular patterns associated with threat and challenge appraisals: a within-subjects analysis. Psychophysiology 39:292–302
Ritter FE (1988) Extending the Seibel−Soar model: presented at the Soar V workshop held at CMU
Ritter FE (2003) Social processes in validation: Comments on Grant (1962) and Roberts and Pashler (2000). Comments as part of Symposium on Model Fitting and Parameter Estimation. In ACT-R Workshop, 129–130
Ritter FE, Young RM (2001) Embodied models as simulated users: introduction to this special issue on using cognitive models to improve interface design. Int J Hum-Comput Stud 55(1):1–14
Ritter FE, Reifers A, Klein LC, Quigley K, Schoelles MJ (2004) Using cognitive modeling to study behavior moderators: pre-task appraisal and anxiety. In: Proceedings of the human factors and ergonomics society. Human factors and ergonomics society, Santa monica, pp 2121−2125
Ritter FE, Van Rooy D, St Amant R, Simpson K (2006) Providing user models direct access to interfaces: an exploratory study of a simple interface with implications for HRI and HCI. IEEE Trans Sys Man Cybern Part A Sys Hum 36(3):592–601
Ritter FE, Kukreja U, St Amant R (2007a) Including a model of visual processing with a cognitive architecture to model a simple teleoperation task. J Cogn Eng Decis Mak 1(2):121–147
Ritter FE, Reifers AL, Klein LC, Schoelles MJ (2007b) Lessons from defining theories of stress for architectures. In: Gray W (ed) Integrated models of cognitive systems. Oxford University Press, New York, NY, pp 254–262
Ritter FE, Kase SE, Klein LC, Bennett J, Schoelles M (2009) Fitting a model to behavior tells us what changes cognitively when under stress and with caffeine. In: Proceedings of the biologically inspired cognitive architectures symposium at the AAAI fall symposium. Keynote presentation, Technical report FS-09-01. AAAI Press, Menlo Park, pp 109−115
Schoelles MJ, Gray WD (2001) Argus: a suite of tools for research in complex cognition. Behavior Res Methods Instrum Comput 33(2):130–140
Thiruvengada H, Rothrock L (2007) Time window-based performance measures: a framework to measure team performance in dynamic environments. Cogn Tech Work 9(2):99–108
Tomaka J, Blascovich J, Kelsey RM, Leitten CL (1993) Subjective, physiological, and behavioral effects of threat and challenge appraisal. J Pers Soc Psychol 65(2):248–260
Tomaka J, Blascovich J, Kibler J, Ernst JM (1997) Cognitive and physiological antecedents of threat and challenge appraisal. J Pers Soc Psychol 73(1):63–72
Acknowledgments
Earlier versions of this work have been presented at the US Air Force Workshop on ACT-R models of human-system interaction, and ONR workshops on cognitive architectures. Participants there provided useful comments. This project was supported by ONR award N000140310248 and DTRA HDTRA1-09-1-0054. Axel Cleeremans, Andrew Reifers, and Lael Schooler provided comments to improve this paper. The views expressed in this paper do not necessarily reflect the position or the policies of the US Government, and no official endorsement should be inferred.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2011 Springer-Verlag London Limited
About this chapter
Cite this chapter
Ritter, F.E., Schoelles, M.J., Quigley, K.S., Klein, L.C. (2011). Determining the Number of Simulation Runs: Treating Simulations as Theories by Not Sampling Their Behavior. In: Rothrock, L., Narayanan, S. (eds) Human-in-the-Loop Simulations. Springer, London. https://doi.org/10.1007/978-0-85729-883-6_5
Download citation
DOI: https://doi.org/10.1007/978-0-85729-883-6_5
Published:
Publisher Name: Springer, London
Print ISBN: 978-0-85729-882-9
Online ISBN: 978-0-85729-883-6
eBook Packages: Computer ScienceComputer Science (R0)