Skip to main content
SpringerLink
Log in

Parallel Ranking and Selection

  • Chapter
  • First Online:
Advances in Modeling and Simulation

Part of the book series: Simulation Foundations, Methods and Applications ((SFMA))

Abstract

The Winter Simulation Conference serves as the initial publication venue for many advances in ranking and selection (R&S), including the recently developed R&S procedures that exploit high-performance parallel computing. We formulate a new stylized model for representing parallel R&S procedures, and we provide an overview of existing R&S procedures under the stylized model. We also discuss why designing R&S procedures for a parallel computing platform is nontrivial and speculate on the future of parallel R&S procedures. In this chapter, “parallel computing” means multiple processors that can execute distinct simulations independently, rather than vector or array processors designed to speed up vector-matrix calculations.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 109.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    In this paper “parallel computing platform” means multiple processors that can independently execute simulation experiments and communicate with each other via message passing or shared memory. We use the term “processors” to refer to cores or threads that can complete computing tasks, so the total number of processors is cores \(\times \) (threads/core).

References

  • Andradóttir S, Kim SH (2010) Fully sequential procedures for comparing constrained systems via simulation. Nav Res Logist 57(5):403–421. doi:10.1002/nav.20408

    MathSciNet  MATH  Google Scholar 

  • Barlas G, Kaufman M (2015) Multicore and GPU programming: an integrated approach. Elsevier

    Google Scholar 

  • Bechhofer RE (1954) A single-sample multiple decision procedure for ranking means of normal populations with known variances. Ann Math Stat 25(1):16–39

    Article  MathSciNet  MATH  Google Scholar 

  • Bechhofer RE, Santner TJ, Goldsman D (1995) Design and analysis of experiments for statistical selection. Screening and multiple comparisons, Wiley, New York

    Google Scholar 

  • Boesel J, Nelson BL, Kim SH (2003) Using ranking and selection to ‘clean up’ after simulation optimization. Oper Res 51(5):814–825

    Article  MathSciNet  MATH  Google Scholar 

  • Branke J, Chick SE, Schmidt C (2007) Selecting a selection procedure. Manage Sci 53(12):1916–1932

    Article  MATH  Google Scholar 

  • Bubeck S, Cesa-Bianchi N et al (2012) Regret analysis of stochastic and nonstochastic multi-armed bandit problems. Foundations and trends\(\textregistered \). Mach Learn 5(1):1–122

    Article  MATH  Google Scholar 

  • Chen CH, Lee LH (2010) Stochastic simulation optimization: an optimal computing budget allocation. World Scientific, Hackensack, NJ

    Google Scholar 

  • Chen CH, Lin J, Yücesan E, Chick SE (2000) Simulation budget allocation for further enhancing the efficiency of ordinal optimization. Discrete Event Dyn Syst 10(3):251–270. doi:10.1023/A:1008349927281

    Article  MathSciNet  MATH  Google Scholar 

  • Chen EJ (2005) Using parallel and distributed computing to increase the capability of selection procedures. In: Kuhl ME, Steiger NM, Armstrong FB, Joines JA (eds) Proceedings of the 2005 winter simulation conference. Institute of Electrical and Electronics Engineers Inc, Piscataway, NJ, pp 723–731

    Chapter  Google Scholar 

  • Chick SE, Branke J, Schmidt C (2010) Sequential sampling to myopically maximize the expected value of information. INFORMS J Comput 22(1):71–80

    Article  MathSciNet  MATH  Google Scholar 

  • Dong J, Zhu Y (2016) Three asymptotic regimes for ranking and selection with general sample distributions. In: Roeder TMK, Frazier PI, Szechtman R, Zhou E, Huschka T, Chick SE (eds) Proceedings of the 2016 Winter Simulation Conference. IEEE, Institute of Electrical and Electronics Engineers Inc, Piscataway, NJ, pp 277–288

    Chapter  Google Scholar 

  • Dudewicz EJ (1976) Statistics in simulation: how to design for selecting the best alternative. In: Highland HJ, Schriber TJ, Sargent RG (eds) Proceedings of the 1976 bicentennial winter simulation conference, pp 67–71. http://informs-sim.org/wsc76papers/1976_0012.pdf

  • Dudewicz EJ, Dalal SR (1975) Allocation of observations in ranking and selection with unequal variances. Sankhya Indian J Stat B37:28–78

    MathSciNet  MATH  Google Scholar 

  • Fabian V (1964) Note on Anderson’s sequential procedures with triangular boundary. Ann Stat 2(1):170–176

    Article  MathSciNet  MATH  Google Scholar 

  • Fan W, Hong LJ, Nelson BL (2016) Indifference-zone-free selection of the best. Oper Res 64(6):1499–1514

    Article  MathSciNet  MATH  Google Scholar 

  • Feldman G, Hunter SR (2016) SCORE allocations for bi-objective ranking and selection. Optim Online. http://www.optimization-online.org/DB_HTML/2016/06/5469.html

  • Feldman G, Hunter SR, Pasupathy R (2015) Multi-objective simulation optimization on finite sets: optimal allocation via scalarization. In: Yilmaz L, Chan WKV, Moon I, Roeder TMK, Macal C, Rossetti MD (eds) Proceedings of the 2015 winter simulation conference. IEEE, Institute of Electrical and Electronics Engineers, Inc, Piscataway, NJ, pp 3610–3621. doi:10.1109/WSC.2015.7408520

  • Frazier PI, Powell WB, Dayanik S (2008) A knowledge-gradient policy for sequential information collection. SIAM J Control Optim 47(5):2410–2439

    Article  MathSciNet  MATH  Google Scholar 

  • Fu M (ed) (2015) Handbook of simulation optimization. In: International series in operations research & management science, vol 216. Springer, New York. doi:10.1007/978-1-4939-1384-8

  • Glynn PW, Heidelberger P (1990) Bias properties of budget constrained simulations. Oper Res 38(5):801–814

    Article  MathSciNet  MATH  Google Scholar 

  • Glynn PW, Heidelberger P (1991) Analysis of parallel replicated simulations under a completion time constraint. ACM Trans Model Comput Simul 1(1):3–23

    Article  MATH  Google Scholar 

  • Glynn PW, Juneja S (2004) A large deviations perspective on ordinal optimization. In: Ingalls RG, Rossetti MD, Smith JS, Peters BA (eds) Proceedings of the 2004 winter simulation conference. IEEE, Institute of Electrical and Electronics Engineers, Inc, Piscataway, NJ, pp 577–585. doi:10.1109/WSC.2004.1371364

  • Goldsman D (1983) Ranking and selection in simulation. In: Roberts S, Banks J, Schmeiser B (eds) Proceedings of the 1983 winter simulation conference, pp 387–393. http://informs-sim.org/wsc83papers/1983_0017.pdf

  • Goldsman D, Nelson BL (1998) Statistical screening, selection, and multiple comparison procedures in computer simulation. In: Medieros DJ, Watson EF, Carson JS, Manivannan MS (eds) Proceedings of the 1998 winter simulation conference. Institute of Electrical and Electronics Engineers, Piscataway, NJ, pp 159–166

    Chapter  Google Scholar 

  • Gupta, S.S., Panchapakesan, S.: Multiple decision procedures: theory and methodology of selecting and ranking populations. SIAM (2002)

    Google Scholar 

  • Heidelberger P (1988) Discrete event simulations and parallel processing: statistical properties. Siam J Stat Comput 9(6):1114–1132

    Article  MathSciNet  MATH  Google Scholar 

  • Henderson SG, Pasupathy R (2016) Simulation optimization library. http://www.simopt.org

  • Hong LJ, Nelson BL (2006) Discrete optimization via simulation using COMPASS. Oper Res 54(1):115–129

    Article  MATH  Google Scholar 

  • Hsu JC (1984) Constrained simultaneous confidence intervals for multiple comparisons with the best. Ann Stat 12:1136–1144

    Article  MathSciNet  MATH  Google Scholar 

  • Hunter SR, Applegate EA, Arora V, Chong B, Cooper K, Rincón-Guevara O, Vivas-Valencia C (2017) An introduction to multi-objective simulation optimization. Optim Online. http://www.optimization-online.org/DB_HTML/2017/03/5903.html

  • Hunter SR, McClosky B (2016) Maximizing quantitative traits in the mating design problem via simulation-based Pareto estimation. IIE Trans 48(6):565–578. doi:10.1080/0740817X.2015.1096430

    Article  Google Scholar 

  • Jamieson K, Nowak R (2014) Best-arm identification algorithms for multi-armed bandits in the fixed confidence setting. In: 2014 48th Annual conference on information sciences and systems (CISS). IEEE, pp 1–6

    Google Scholar 

  • Kim SH, Nelson BL (2001) A fully sequential procedure for indifference-zone selection in simulation. ACM Trans Model Comput Simul 11(3):251–273

    Article  Google Scholar 

  • Kim S, Nelson BL (2006a) On the asymptotic validity of fully sequential selection procedures for steady-state simulation. Oper Res 54(3):475–488. doi:10.1287/opre.1060.0281

  • Kim SH, Nelson BL (2006b) Selecting the best system. In: Henderson SG, Nelson BL (eds) Simulation, handbooks in operations research and management science, vol 13. Elsevier, Amsterdam, pp 501–534

    Google Scholar 

  • Lee LH, Chew EP, Teng S, Goldsman D (2010) Finding the non-dominated Pareto set for multi-objective simulation models. IIE Trans 42:656–674. doi:10.1080/07408171003705367

    Article  Google Scholar 

  • Li H (2017) Parallel OCBA. https://gitlab.com/o2des_dev/ParallelOCBA

  • Luo J, Hong LJ (2011) Large-scale ranking and selection using cloud computing. In: Jain S, Creasey RR, Himmelspach J, White KP, Fu M (eds) Proceedings of the 2011 winter simulation conference. IEEE, Institute of Electrical and Electronics Engineers, Inc, Piscataway, NJ, pp 4051–4061. doi:10.1109/WSC.2011.6148094

  • Luo J, Hong LJ, Nelson BL, Wu Y (2015) Fully sequential procedures for large-scale ranking-and-selection problems in parallel computing environments. Oper Res 63(5):1177–1194. doi:10.1287/opre.2015.1413

  • Luo YC, Chen CHYücesan E, Lee I (2000) Distributed web-based simulation optimization. In: Joines JA, Barton RR, Kang K, Fishwick PA (eds) Proceedings of the 2000 winter simulation conference. Institute of Electrical and Electronics Engineers, Inc, Piscataway, NJ, pp 1785–1793. doi:10.1109/WSC.2000.899170

  • Negoescu DM, Frazier PI, Powell WB (2011) The knowledge-gradient algorithm for sequencing experiments in drug discovery. INFORMS J Comput 23(3):346–363

    Article  MathSciNet  MATH  Google Scholar 

  • Nelson B (2013) Foundations and methods of stochastic simulation: a first course. Springer (2013)

    Google Scholar 

  • Nelson BL, Swann J, Goldsman D, Song W (2001) Simple procedures for selecting the best simulated system when the number of alternatives is large. Oper Res 49(6):950–963

    Article  Google Scholar 

  • Ni EC (2015a) MapRedRnS: parallel ranking and selection using mapreduce. https://bitbucket.org/ericni/mapredrns

  • Ni EC (2015b) mpiRnS: Parallel ranking and selection using MPI (2015). https://bitbucket.org/ericni/mpirns

  • Ni EC (2015c) SparkRnS: parallel ranking and selection using spark. https://bitbucket.org/ericni/sparkrns

  • Ni EC, Ciocan DF, Henderson SG, Hunter SR (2015) Comparing message passing interface and mapreduce for large-scale parallel ranking and selection. In: Yilmaz L, Chan WKV, Moon I, Roeder TMK, Macal C, Rossetti MD (eds) Proceedings of the 2015 winter simulation conference. IEEE, Institute of Electrical and Electronics Engineers, Inc, Piscataway, NJ, pp 3858–3867. doi:10.1109/WSC.2015.7408542

  • Ni EC, Ciocan DF, Henderson SG, Hunter SR (2017) Efficient ranking and selection in parallel computing environments. Oper Res 65(3):821–836. doi:10.1287/opre.2016.1577

    Article  MathSciNet  Google Scholar 

  • Ni EC, Henderson SG, Hunter SR (2014) A comparison of two parallel ranking and selection procedures. In: Tolk A, Diallo SD, Ryzhov IO, Yilmaz L, Buckley S, Miller JA (eds) Proceedings of the 2014 winter simulation conference. IEEE, Institute of Electrical and Electronics Engineers, Inc, Piscataway, NJ, pp 3761–3772. doi:10.1109/WSC.2014.7020204

  • Ni EC, Hunter SR, Henderson SG (2013) Ranking and selection in a high performance computing environment. In: Pasupathy R, Kim S, Tolk A, Hill R, Kuhl ME (eds) Proceedings of the 2013 winter simulation conference. IEEE, Institute of Electrical and Electronics Engineers, Inc, Piscataway, NJ, 833–845. doi:10.1109/WSC.2013.6721475

  • Pasupathy R, Ghosh S (2013) Simulation optimization: a concise overview and implementation guide. In: Topaloglu H (ed) TutORials in operations research, chap 7. INFORMS, Catonsville, MD, pp 122–150. doi:10.1287/educ.2013.0118

  • Pasupathy R, Hunter SR, Pujowidianto NA, Lee LH, Chen C (2014) Stochastically constrained ranking and selection via SCORE. ACM Trans Model Comput Simul 25(1):1–26. doi:10.1145/2630066

    Article  MathSciNet  MATH  Google Scholar 

  • Paulson E (1964) A sequential procedure for selecting the population with the largest mean from k normal populations. Ann Math Stat 35(1):174–180

    Article  MathSciNet  MATH  Google Scholar 

  • Peng Y, Chen C, Fu MC, Hu J (2016) Dynamic sampling allocation and design selection. INFORMS J Comput 28(2):195–208. doi:10.1287/ijoc.2015.0673

    Article  MathSciNet  MATH  Google Scholar 

  • Pinedo M (2015) Scheduling. Springer

    Google Scholar 

  • Rinott Y (1978) On two-stage selection procedures and related probability-inequalities. Commun Stat Theory Methods A7:799–877

    Article  MathSciNet  MATH  Google Scholar 

  • Ryzhov IO (2016) On the convergence rates of expected improvement methods. Oper Res 64(6):1515–1528. doi:10.1287/opre.2016.1494

    Article  MathSciNet  MATH  Google Scholar 

  • Schmeiser B (1982) Batch size effects in the analysis of simulation output. Oper Res 30(3):556–568

    Article  MathSciNet  MATH  Google Scholar 

  • Singham DI, Szechtman R (2016) Multiple comparisons with a standard using false discovery rates. In: Proceedings of the 2016 winter simulation conference. Institute of Electrical and Electronic Engineers Inc, Piscataway, NJ, pp 501–511

    Google Scholar 

  • Turnquist MA, Sussman JM (1976) A Bayesian approach to the design of simulation experiments. In: Highland HJ, Schriber TJ, Sargent RG (eds) Proceedings of the 1976 bicentennial winter simulation conference, pp 59–64. http://informs-sim.org/wsc76papers/1976_0011.pdf

  • Wang H, Pasupathy R, Schmeiser BW (2013) Integer-ordered simulation optimization using R-SPLINE: retrospective search using piecewise-linear interpolation and neighborhood enumeration. ACM Trans Model Comput Simul 23(3): doi:10.1145/2499913.2499916

  • Xu J, Nelson BL, Hong LJ (2010) Industrial Strength COMPASS: A comprehensive algorithm and software for optimization via simulation. ACM Trans Model Comput Simul 20:1–29

    Article  Google Scholar 

  • Yoo T, Cho H, Yücesan E (2009) Web services-based parallel replicated discrete event simulation for large-scale simulation optimization. Simulation 85(7):461–475

    Article  Google Scholar 

Download references

Acknowledgements

Hunter’s research was partially supported by the National Science Foundation under Grant Number CMMI-1554144. Nelson’s research was partially supported by the National Science Foundation under Grant Number CMMI-1537060 and GOALI co-sponsor SAS Institute.

Author information

Authors and Affiliations

  1. Purdue University, West Lafayette, IN, USA

    Susan R. Hunter

  2. Northwestern University, Evanston, IL, USA

    Barry L. Nelson

Authors
  1. Susan R. Hunter
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Barry L. Nelson
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Susan R. Hunter .

Editor information

Editors and Affiliations

  1. The MITRE Corporation, Hampton, Virginia, USA

    Andreas Tolk

  2. Department of Supply Chain Management, Arizona State University, Tempe, Arizona, USA

    John Fowler

  3. National Institute of Standards and Technology, Gaithersburg, Maryland, USA

    Guodong Shao

  4. T&OM Area, INSEAD, Fontainebleau, France

    Enver Yücesan

Rights and permissions

Reprints and permissions

Copyright information

© 2017 Springer International Publishing AG (outside the USA)

About this chapter

Cite this chapter

Hunter, S.R., Nelson, B.L. (2017). Parallel Ranking and Selection. In: Tolk, A., Fowler, J., Shao, G., Yücesan, E. (eds) Advances in Modeling and Simulation. Simulation Foundations, Methods and Applications. Springer, Cham. https://doi.org/10.1007/978-3-319-64182-9_12

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-64182-9_12

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-64181-2

  • Online ISBN: 978-3-319-64182-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation