Journal of the Operational Research Society

, Volume 61, Issue 9, pp 1389–1403 | Cite as

Automating warm-up length estimation

Theoretical Paper

Abstract

There are two key issues in assuring the accuracy of estimates of performance obtained from a simulation model. The first is the removal of any initialisation bias, the second is ensuring that enough output data is produced to obtain an accurate estimate of performance. This paper is concerned with the first issue, and more specifically warm-up estimation. Our aim is to produce an automated procedure, for inclusion into commercial simulation software, for estimating the length of warm-up and hence removing initialisation bias from simulation output data. This paper describes the extensive literature search that was carried out in order to find and assess the various existing warm-up methods, the process of short-listing and testing of candidate methods. In particular it details the extensive testing of the warm-up MSER-5 method.

Keywords

simulation warm-up period initialisation bias truncation point automation MSER-5 

Notes

Acknowledgements

This work is part of the Automating Simulation Output Analysis (AutoSimOA) project (www.wbs.ac.uk/go/autosimoa) that is funded by the UK Engineering and Physical Sciences Research Council (EP/D033640/1). The work is being carried out in collaboration with SIMUL8 Corporation, who is also providing sponsorship for the project.

References

  1. Alexopoulos C and Seila AF ( 1998 ). Output data analysis Handbook of simulation . Wiley: New York pp 225–272 .Google Scholar
  2. Banks J, Carson JS, Nelson BL and Nicol DM ( 2001 ). Discrete-event system simulation, 4th edn . Prentice-Hall: New Jersey .Google Scholar
  3. Bause F and Eickhoff M ( 2003 ). Truncation point estimation using multiple replications in parallel . In: Chick S, Sánchez PJ, Ferrin D and Morrice DJ (eds) . Proceedings of the 2003 Winter Simulation Conference . IEEE: Piscataway, NJ, pp . 414 – 421 .CrossRefGoogle Scholar
  4. Beck AD ( 2004 ). Consistency of warm up periods for a simulation model that is cyclic in nature . In: Brailsford SC, Oakshott L, Robinson S and Taylor SJE (eds) . Proceedings of the 2004 Operational Research Society Simulation Workshop (SW04) . OR Society: Birmingham, UK, pp . 105 – 108 .Google Scholar
  5. Box GE, Jenkins GM and Reinsel GC ( 1994 ). Time series analysis: Forecasting and control, 3rd edn . Prentice-Hall: New Jersey .Google Scholar
  6. Bratley P, Fox B and Schrage L ( 1987 ). A guide to simulation, 2nd edn . Springer-Verlag: New York .CrossRefGoogle Scholar
  7. Cash CR, Dippold DG, Long JM and Pollard WP ( 1992 ). Evaluation of tests for initial-condition bias . In: Cash CR, Nelson BL, Dippold DG, Long MJ and Pollard WP (eds) . Proceedings of the 1992 Winter Simulation Conference . ACM: New York, NY, USA, pp . 577 – 585 .Google Scholar
  8. Conway RW ( 1963 ). Some tactical problems in digital simulation . Mngt Sci 10 ( 1 ): 47 – 61 .CrossRefGoogle Scholar
  9. Delaney PJ ( 1995 ). Control of initialisation bias in queuing simulations using queuing approximations . MS thesis, Department of Systems Engineering, University of Virginia. .Google Scholar
  10. Fishman GS ( 1971 ). Estimating sample size in computing simulation experiments . Mngt Sci 18 : 21 – 38 .CrossRefGoogle Scholar
  11. Fishman GS ( 1973 ). Concepts and methods in discrete event digital simulation . Wiley: New York .Google Scholar
  12. Fishman GS ( 2001 ). Discrete-event simulation, modeling, programming, and analysis . Springer-Verlag: New York .CrossRefGoogle Scholar
  13. Gafarian AV, Ancker CJ Jr and Morisaku T ( 1978 ). Evaluation of commonly used rules for detecting ‘steady-state' in computer simulation . Nav Res Logist Q 25 : 511 – 529 .CrossRefGoogle Scholar
  14. Gallagher MA, Bauer KW and Maybeck PS ( 1996 ). Initial data truncation for univariate output of discrete-event simulations using the Kalman Filter . Mngt Sci 42 ( 4 ): 559 – 575 .CrossRefGoogle Scholar
  15. Glynn PW and Iglehart DL ( 1987 ). A new initial bias deletion rule . In: Thesen A, Grant H and Kelton DW (eds) . Proceedings of the 1987 Winter Simulation Conference . ACM: New York, NY, USA, pp . 318 – 319 .Google Scholar
  16. Goldsman D, Schruben LW and Swain JJ ( 1994 ). Tests for transient means in simulated time series . Nav Res Log 41 : 171 – 187 .CrossRefGoogle Scholar
  17. Gordon G ( 1969 ). System simulation . Prentice-Hall: New Jersey .Google Scholar
  18. Jackway PT and deSilva BM ( 1992 ). A methodology for initialisation bias reduction in computer simulation output . Asia Pac J Opl Res 9 : 87 – 100 .Google Scholar
  19. Kelton WD and Law AM ( 1983 ). A new approach for dealing with the startup problem in discrete event simulation . Nav Res Logist Q 30 : 641 – 658 .CrossRefGoogle Scholar
  20. Kimbler DL and Knight BD ( 1987 ). A survey of current methods for the elimination of initialisation bias in digital simulation . Annu Simul Symp 20 : 133 – 142 .Google Scholar
  21. Lada EK and Wilson JR ( 2006 ). A wavelet-based spectral procedure for steady-state simulation analysis . Eur J Opl Res 174 : 1769 – 1801 .CrossRefGoogle Scholar
  22. Lada EK, Wilson JR and Steiger NM ( 2003 ). A wavelet-based spectral method for steady-state simulation analysis . In: Chick S, Sánchez PJ, Ferrin D and Morrice DJ (eds) . Proceedings of the 2003 Winter Simulation Conference . IEEE: Piscataway, NJ, pp . 422 – 430 .CrossRefGoogle Scholar
  23. Lada EK, Wilson JR, Steiger NM and Joines JA ( 2004 ). Performance evaluation of a wavelet-based spectral method for steady-state simulation analysis . In: Lada EK, Wilson JR, Steiger NM and Joines JA (eds) . Proceedings of the 2004 Winter Simulation Conference . IEEE: Piscataway, NJ, pp . 694 – 702 .Google Scholar
  24. Law AM ( 1983 ). Statistical analysis of simulation output data . Opns Res 31 : 983 – 1029 .CrossRefGoogle Scholar
  25. Law AM ( 2007 ). Simulation modelling and analysis . McGraw-Hill: New York .Google Scholar
  26. L'Ecuyer P ( 1999 ). Good parameters and implementations for combined multiple recursive random number generators . Opns Res 47 : 159 – 164 .CrossRefGoogle Scholar
  27. Lee Y-H and Oh H-S ( 1994 ). Detecting truncation point in steady-state simulation using chaos theory . In: Tew JD, Manivannan S, Sadowski DA and Seila AF (eds) . Proceedings of the 1994 Winter Simulation Conference . Society for Computer Simulation International: San Diego, CA, USA, pp . 353 – 360 .Google Scholar
  28. Lee Y-H, Kyung K-H and Jung C-S ( 1997 ). On-line determination of steady-state in simulation outputs . Comput Ind Eng 33 ( 3 ): 805 – 808 .CrossRefGoogle Scholar
  29. Linton JR and Harmonosky CM ( 2002 ). A comparison of selective initialization bias elimination methods . In: Yücesan E, Chen C-H, Snowdon JL and Charnes JM (eds) . Proceedings of the 34th Winter Simulation Conference . IEEE: Piscataway, NJ, pp . 1951 – 1957 .CrossRefGoogle Scholar
  30. Ma X and Kochhar AK ( 1993 ). A comparison study of two tests for detecting initialization bias in simulation output . Simulation 61 ( 2 ): 94 – 101 .CrossRefGoogle Scholar
  31. Mahajan PS and Ingalls RG ( 2004 ). Evaluation of methods used to detect warm-up period in steady-state simulation . In: Lada EK, Wilson JR, Steiger NM and Joines JA (eds) . Proceedings of the 2004 Winter Simulation Conference . IEEE: Piscataway, NJ, pp . 663 – 671 .Google Scholar
  32. Nelson BL ( 1992 ). Statistical analysis of simulation results, Handbook of industrial engineering, 2nd edn . John Wiley: New York .Google Scholar
  33. Ockerman DH and Goldsman D ( 1999 ). Student t-tests and compound tests to detect transients in simulated time series . Eur J Opl Res 116 : 681 – 691 .CrossRefGoogle Scholar
  34. Pawlikowski K ( 1990 ). Steady-state simulation of queueing processes: A survey of problems and solutions . Comput Surv 122 ( 2 ): 123 – 170 .CrossRefGoogle Scholar
  35. Robinson S ( 2004 ). Simulation. The practice of model development and use . John Wiley & Sons Ltd: England .Google Scholar
  36. Robinson S ( 2005 ). A statistical process control approach to selecting a warm-up period for a discrete-event simulation . Eur J Opl Res 176 : 332 – 346 .CrossRefGoogle Scholar
  37. Rossetti MD, Li Z and Qu P ( 2005 ). Exploring exponentially weighted moving average control charts to determine the warm-up period . In: Kuhl ME, Steiger NM, Armstrong FB and Joines JA (eds) . Proceedings of the Winter Simulation Conference . IEEE: Piscataway, NJ, pp . 771 – 780 .Google Scholar
  38. Roth E ( 1994 ). The relaxation time heuristic for the initial transient problem in M/M/k queueing systems . Eur J Opl Res 72 : 376 – 386 .CrossRefGoogle Scholar
  39. Roth E and Josephy N ( 1993 ). A relaxation time heuristic for exponential-Erlang queueing systems . Comput Opns Res 20 ( 3 ): 293 – 301 .CrossRefGoogle Scholar
  40. Sandikci B and Sabuncuoglu I ( 2006 ). Analysis of the behaviour of the transient period in non-terminating simulations . Eur J Opl Res 173 : 252 – 267 .CrossRefGoogle Scholar
  41. Schruben LW ( 1982 ). Detecting initialization bias in simulation output . Opns Res 30 ( 3 ): 569 – 590 .CrossRefGoogle Scholar
  42. Schruben L, Singh H and Tierney L ( 1983 ). Optimal tests for initialization bias in simulation output . Opns Res 31 ( 6 ): 1167 – 1178 .CrossRefGoogle Scholar
  43. Sheth-Voss PA, Willemain TR and Haddock J ( 2005 ). Estimating the steady-state mean from short transient simulations . Eur J Opl Res 162 ( 2 ): 403 – 417 .CrossRefGoogle Scholar
  44. Spratt SC ( 1998 ). An evaluation of contemporary heuristics for the startup problem . MS thesis, Faculty of the School of Engineering and Applied Science, University of Virginia. .Google Scholar
  45. Vassilacopoulos G ( 1989 ). Testing for initialization bias in simulation output . Simulation 52 ( 4 ): 151 – 153 .CrossRefGoogle Scholar
  46. White KP Jr ( 1997 ). An effective truncation heuristic for bias reduction in simulation output . Simulation 69 ( 6 ): 323 – 334 .CrossRefGoogle Scholar
  47. White KP Jr, Cobb MJ and Spratt SC ( 2000 ). A comparison of five steady-state truncation heuristics for simulation . In: Joines JA, Barton RR, Kang K and Fishwick PA (eds) . Proceedings of the 2000 Winter Simulation Conference . Society for Computer Simulation International: San Diego, CA, USA, pp . 755 – 760 .CrossRefGoogle Scholar
  48. Wilson JR and Pritsker AAB ( 1978a ). A survey of research on the simulation startup problem . Simulation 31 ( 2 ): 55 – 58 .CrossRefGoogle Scholar
  49. Wilson JR and Pritsker AAB ( 1978b ). Evaluation of startup policies in simulation experiments . Simulation 31 ( 3 ): 79 – 89 .CrossRefGoogle Scholar
  50. Yücesan E ( 1993 ). Randomization tests for initialization bias in simulation output . Nav Res Log 40 : 643 – 663 .CrossRefGoogle Scholar
  51. Zobel CW and White KP Jr ( 1999 ). Determining a warm-up period for a telephone network routing simulation . In: Farrington P et al (eds.) Proceedings of the 1999 Winter Simulation Conference, pp . 662 – 665 .Google Scholar

Copyright information

© Operational Research Society 2009

Authors and Affiliations

  1. 1.Warwick Business School, The University of WarwickCoventryUK

Personalised recommendations