Skip to main content
SpringerLink
Log in

Selecting the Best Simulated System: Thinking Differently About an Old Problem

  • Conference paper
  • First Online:
Monte Carlo and Quasi-Monte Carlo Methods (MCQMC 2018)

Part of the book series: Springer Proceedings in Mathematics & Statistics ((PROMS,volume 324))

Abstract

The methods known collectively as “ranking & selection” have been a theoretical and practical success story for the optimization of simulated stochastic systems: they are widely used in practice, have been implemented in commercial simulation software, and research has made them more and more statistically efficient. However, “statistically efficient” has meant minimizing the number of simulation-generated observations required to make a selection, or maximizing the strength of the inference given a budget of observations. Exploiting high-performance computing, and specifically the capability to simulate many feasible solutions in parallel, has challenged the ranking & selection paradigm. In this paper we review the challenge and suggest an entirely different approach.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 129.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 169.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 169.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Bechhofer, R., Santner, T., Goldsman, D.: Designing Experiments for Statistical Selection, Screening, and Multiple Comparisons. Wiley, New York (1995)

    Google Scholar 

  2. Bechhofer, R.E.: A single-sample multiple decision procedure for ranking means of normal populations with known variances. Ann. Math. Stat. 25(1), 16–39 (1954)

    Article  MathSciNet  Google Scholar 

  3. Chen, E.J.: Using parallel and distributed computing to increase the capability of selection procedures. In: M.E. Kuhl, N.M. Steiger, F.B. Armstrong, J.A. Joines (eds.) Proceedings of the 2005 Winter Simulation Conference, pp. 723–731. Institute of Electrical and Electronics Engineers, Inc., Piscataway, NJ (2005)

    Google Scholar 

  4. Fan, W., Hong, L.J., Nelson, B.L.: Indifference-zone-free selection of the best. Oper. Res. 64(6), 1499–1514 (2016)

    Article  MathSciNet  Google Scholar 

  5. Frazier, P.I.: Decision-theoretic foundations of simulation optimization. Wiley Encyclopedia of Operations Research and Management Sciences. Wiley, New York (2010)

    Google Scholar 

  6. Goldsman, D.: Ranking and selection in simulation. In: Roberts, S., Banks, J., Schmeiser B. (eds.) Proceedings of the 1983 Winter Simulation Conference, pp. 387–393, http://informs-sim.org/wsc83papers/1983_0017.pdf (1983)

  7. Gupta, S.S.: On some multiple decision (selection and ranking) rules. Technometrics 7(2), 225–245 (1965)

    Article  Google Scholar 

  8. Hong, L.J.: Fully sequential indifference-zone selection procedures with variance-dependent sampling. Naval Res. Logist. 53(5), 464–476 (2006)

    Article  MathSciNet  Google Scholar 

  9. Hunter, S.R., Nelson, B.L.: Parallel ranking and selection. In: Tolk, A., Fowler, J., Shao, G., Yücesan, E. (eds.) Advances in Modeling and Simulation: Seminal Research from 50 Years of Winter Simulation Conferences, pp. 249–275. Springer International Publishing (2017). https://doi.org/10.1007/978-3-319-64182-9_12

  10. Jamieson, K., Nowak, R.: Best-arm identification algorithms for multi-armed bandits in the fixed confidence setting. In: 2014 48th Annual Conference on Information Sciences and Systems (CISS), pp. 1–6. IEEE (2014)

    Google Scholar 

  11. Kamiński, B., Szufel, P.: On parallel policies for ranking and selection problems. J. Appl. Stat. 45(9), 1690–1713 (2018)

    Article  MathSciNet  Google Scholar 

  12. Kim, S.H., Nelson, B.L.: A fully sequential procedure for indifference-zone selection in simulation. ACM Trans. Model. Comput. Simul. 11(3), 251–273 (2001)

    Article  Google Scholar 

  13. Luo, J., Hong, L.J., Nelson, B.L., Wu, Y.: Fully sequential procedures for large-scale ranking-and-selection problems in parallel computing environments. Oper. Res. 63(5), 1177–1194 (2015). https://doi.org/10.1287/opre.2015.1413

    Article  MathSciNet  MATH  Google Scholar 

  14. Luo, Y.C., Chen, C.H., Yücesan, E., Lee, I.: Distributed web-based simulation optimization. In: Joines, J.A., Barton, R.R., Kang, K., Fishwick, P.A. (eds.) Proceedings of the 2000 Winter Simulation Conference, pp. 1785–1793. Institute of Electrical and Electronics Engineers, Inc., Piscataway, NJ (2000). https://doi.org/10.1109/WSC.2000.899170

  15. Nelson, B.L., Goldsman, D.: Comparisons with a standard in simulation experiments. Manag. Sci. 47(3), 449–463 (2001)

    Article  Google Scholar 

  16. Ni, E.C., Ciocan, D.F., Henderson, S.G., Hunter, S.R.: Efficient ranking and selection in parallel computing environments. Oper. Res. 65(3), 821–836 (2017)

    Article  MathSciNet  Google Scholar 

  17. Pei, L., Hunter, S., Nelson, B.L.: A new framework for parallel ranking & selection using an adaptive standard. In: Rabe, M., Juan, A.A., Mustafee, N., Skoogh, A., Jain, S., Johansson, B. (eds.) Proceedings of the 2018 Winter Simulation Conference, pp. 2201–2212. IEEE Press (2018)

    Google Scholar 

Download references

Acknowledgements

Portions of this work were supported by NSF Grant CMMI-1537060 and SAS Institute.

Author information

Authors and Affiliations

  1. Northwestern University, Evanston, IL, USA

    Barry L. Nelson

Authors
  1. Barry L. Nelson
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Barry L. Nelson .

Editor information

Editors and Affiliations

  1. INRIA Rennes Bretagne-Atlantique, Campus Universtaire de Beaulieu, Rennes, France

    Bruno Tuffin

  2. Département d’informatique et de Recherche Opérationnelle (DIRO), Université de Montréal, Montreal, QC, Canada

    Pierre L'Ecuyer

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Nelson, B.L. (2020). Selecting the Best Simulated System: Thinking Differently About an Old Problem. In: Tuffin, B., L'Ecuyer, P. (eds) Monte Carlo and Quasi-Monte Carlo Methods. MCQMC 2018. Springer Proceedings in Mathematics & Statistics, vol 324. Springer, Cham. https://doi.org/10.1007/978-3-030-43465-6_3

Download citation

Publish with us

Policies and ethics

Search

Navigation