Determining the Number of Simulation Runs: Treating Simulations as Theories by Not Sampling Their Behavior

  • Frank E. Ritter
  • Michael J. Schoelles
  • Karen S. Quigley
  • Laura Cousino Klein


How many times should a simulation be run to generate valid predictions? With a deterministic simulation, the answer simply is just once. With a stochastic simulation, the answer is more complex. Different researchers have proposed and used different heuristics. A review of the models presented at a conference on cognitive modeling illustrates the range of solutions and problems in this area. We present the argument that because the simulation is a theory, not data, it should not so much be sampled but run enough times to provide stable predictions of performance and the variance of performance. This applies to both pure simulations as well as human-in-the-loop simulations. We demonstrate the importance of running the simulation until it has stable performance as defined by the effect size of interest. When runs are expensive we suggest a minimum number of runs based on power calculations; when runs are inexpensive we suggest a maximum necessary number of runs. We also suggest how to adjust the number of runs for different effect sizes of interest.


Cognitive Appraisal Cognitive Architecture Threat Appraisal Stable Prediction Serial Subtraction 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



Earlier versions of this work have been presented at the US Air Force Workshop on ACT-R models of human-system interaction, and ONR workshops on cognitive architectures. Participants there provided useful comments. This project was supported by ONR award N000140310248 and DTRA HDTRA1-09-1-0054. Axel Cleeremans, Andrew Reifers, and Lael Schooler provided comments to improve this paper. The views expressed in this paper do not necessarily reflect the position or the policies of the US Government, and no official endorsement should be inferred.


  1. Anderson JR (2007) How can the human mind exist in the physical universe? Oxford University Press, New York, NYCrossRefGoogle Scholar
  2. Anderson JR, Lebiere C (1998) The atomic components of thought. Erlbaum, Mahwah, NJGoogle Scholar
  3. Ball J, Myers C, Heiberg A, Cooke NJ, Matessa M, Freiman M et al (2010) The Synthetic Teammate Project. Comput Math Organ Sci 16:271–299CrossRefGoogle Scholar
  4. Best B, Fincham J, Gluck K, Gunzelmann G, Krusmark MA (2008) Efficient use of large-scale computational resources. In: Proceedings of the seventeenth conference on behavior representation in modeling and simulation. Simulation Interoperability Standards Organization, Orlando, FL, pp 180−181Google Scholar
  5. Byrne MD (2003) Cognitive architecture. In: Jacko J, Sears A (eds) The human–computer interaction handbook: fundamentals, evolving technologies and emerging applications. Erlbaum, Mahwah, NJ, pp 97–117Google Scholar
  6. Cohen J (1988) Statistical power analysis for the behavioral sciences, 2nd edn. Lawrence Erlbaum, Hillsdale, NJzbMATHGoogle Scholar
  7. Grant DA (1962) Testing the null hypothesis and the strategy and tactics of investigating theoretical models. Psychol Rev 69(1):54–61MathSciNetCrossRefGoogle Scholar
  8. Howell DC (1987) Statistical methods for psychology, 2nd edn. Duxbury Press, Boston, MAGoogle Scholar
  9. Kieras DE (2003) Model-based evaluation. In: Jacko J, Sears A (eds) Handbook for human-computer interaction. Erlbaum, Mahwah, NJ, pp 1139–1151Google Scholar
  10. Kieras DE, Wood SD, Meyer DE (1997) Predictive engineering models based on the EPIC architecture for a multimodel high-performance human-computer interaction task. Trans Comput Hum Interact 4(3):230–275CrossRefGoogle Scholar
  11. Kirschbaum C, Pirke K-M, Hellhammer DH (1993) The Trier social stress test: a tool for investigating psychobiological stress responses in a laboratory setting. Neuropsychobiology 28:76–81CrossRefGoogle Scholar
  12. Laird JE, Newell A, Rosenbloom PS (1987) Soar: an architecture for general intelligence. Artif Intell 33(1):1–64MathSciNetCrossRefGoogle Scholar
  13. Lovett MC, Daily LZ, Reder LM (2000) A source activation theory of working memory: cross-task prediction of performance in ACT-R. J Cogn Syst Res 1:99–118CrossRefGoogle Scholar
  14. Lovett MC, Schunn C, Lebiere C, Munro P (eds) (2004) Proceedings of the sixth international conference on cognitive modelling, ICCM 2004. Mahwah NJ: Erlbaum.
  15. Mathews G (2001) Levels of transaction: a cognitive science framework for operator stress. In: Hancock PA, Desmond PA (eds) Stress, workload, and fatigue. Erlbaum, Mahwah, NJGoogle Scholar
  16. Miller CS, Laird JE (1996) Accounting for graded performance within a discrete search framework. Cogn Sci 20:499–537CrossRefGoogle Scholar
  17. Newell A (1990) Unified theories of cognition. Harvard University Press, Cambridge, MAGoogle Scholar
  18. Quigley KS, Feldman Barrett LF, Weinstein S (2002) Cardiovascular patterns associated with threat and challenge appraisals: a within-subjects analysis. Psychophysiology 39:292–302CrossRefGoogle Scholar
  19. Ritter FE (1988) Extending the Seibel−Soar model: presented at the Soar V workshop held at CMUGoogle Scholar
  20. Ritter FE (2003) Social processes in validation: Comments on Grant (1962) and Roberts and Pashler (2000). Comments as part of Symposium on Model Fitting and Parameter Estimation. In ACT-R Workshop, 129–130Google Scholar
  21. Ritter FE, Young RM (2001) Embodied models as simulated users: introduction to this special issue on using cognitive models to improve interface design. Int J Hum-Comput Stud 55(1):1–14zbMATHCrossRefGoogle Scholar
  22. Ritter FE, Reifers A, Klein LC, Quigley K, Schoelles MJ (2004) Using cognitive modeling to study behavior moderators: pre-task appraisal and anxiety. In: Proceedings of the human factors and ergonomics society. Human factors and ergonomics society, Santa monica, pp 2121−2125Google Scholar
  23. Ritter FE, Van Rooy D, St Amant R, Simpson K (2006) Providing user models direct access to interfaces: an exploratory study of a simple interface with implications for HRI and HCI. IEEE Trans Sys Man Cybern Part A Sys Hum 36(3):592–601CrossRefGoogle Scholar
  24. Ritter FE, Kukreja U, St Amant R (2007a) Including a model of visual processing with a cognitive architecture to model a simple teleoperation task. J Cogn Eng Decis Mak 1(2):121–147CrossRefGoogle Scholar
  25. Ritter FE, Reifers AL, Klein LC, Schoelles MJ (2007b) Lessons from defining theories of stress for architectures. In: Gray W (ed) Integrated models of cognitive systems. Oxford University Press, New York, NY, pp 254–262Google Scholar
  26. Ritter FE, Kase SE, Klein LC, Bennett J, Schoelles M (2009) Fitting a model to behavior tells us what changes cognitively when under stress and with caffeine. In: Proceedings of the biologically inspired cognitive architectures symposium at the AAAI fall symposium. Keynote presentation, Technical report FS-09-01. AAAI Press, Menlo Park, pp 109−115Google Scholar
  27. Schoelles MJ, Gray WD (2001) Argus: a suite of tools for research in complex cognition. Behavior Res Methods Instrum Comput 33(2):130–140CrossRefGoogle Scholar
  28. Thiruvengada H, Rothrock L (2007) Time window-based performance measures: a framework to measure team performance in dynamic environments. Cogn Tech Work 9(2):99–108CrossRefGoogle Scholar
  29. Tomaka J, Blascovich J, Kelsey RM, Leitten CL (1993) Subjective, physiological, and behavioral effects of threat and challenge appraisal. J Pers Soc Psychol 65(2):248–260CrossRefGoogle Scholar
  30. Tomaka J, Blascovich J, Kibler J, Ernst JM (1997) Cognitive and physiological antecedents of threat and challenge appraisal. J Pers Soc Psychol 73(1):63–72CrossRefGoogle Scholar

Copyright information

© Springer-Verlag London Limited 2011

Authors and Affiliations

  • Frank E. Ritter
    • 1
  • Michael J. Schoelles
    • 5
  • Karen S. Quigley
    • 2
    • 3
  • Laura Cousino Klein
    • 4
  1. 1.College of Information Sciences and TechnologyThe Pennsylvania State UniversityPAUSA
  2. 2.Department of PsychologyNortheastern UniversityBostonUSA
  3. 3.Department of Veterans AffairsEdith Nourse Rogers (Bedford) Memorial VA HospitalBedfordUSA
  4. 4.Department of Biobehavioral HealthThe Pennsylvania State UniversityPAUSA
  5. 5.Department of Cognitive ScienceRensselaer Polytechnic InstituteTroyUSA

Personalised recommendations