Ranking and Selection: Efficient Simulation Budget Allocation

  • Chun-Hung Chen
  • Stephen E. Chick
  • Loo Hay Lee
  • Nugroho A. Pujowidianto
Chapter
Part of the International Series in Operations Research & Management Science book series (ISOR, volume 216)

Abstract

This chapter reviews the problem of selecting the best of a finite set of alternatives, where best is defined with respect to the highest mean performance, and where the performance is uncertain but may be estimated with simulation. This problem has been explored from several perspectives, including statistical ranking and selection, multiple comparisons, and stochastic optimization. Approaches taken in the literature include frequentist statistics, Bayesian statistics, related heuristics, and asymptotic convergence in probability. This chapter presents algorithms that are derived from Bayesian and related conceptual frameworks to provide empirically effective performance for the ranking and selection problem. In particular, we motivate the optimal computing budget allocation (OCBA) algorithm and expected value of information (EVI) approaches, give example algorithms, and provide pointers to the literature for detailed derivations and extensions of these approaches.

Keywords

Allocation Rule Correct Selection Multiple Alternative Linear Loss Equal Allocation 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Notes

Acknowledgements

This work has been supported in part by National Science Foundation under Award CMMI-1233376, Department of Energy under Award DE-SC0002223, NIH under Grant 1R21DK088368-01, National Science Council of Taiwan under Award NSC-100-2218-E-002-027-MY3, and the Program for Professor of Special Appointment (Eastern Scholar) at Shanghai Institutions of Higher Learning.

References

  1. 1.
    R. E. Bechhofer, T. J. Santner, and D. M. Goldsman. Design and Analysis of Experiments for Statistical Selection, Screening, and Multiple Comparisons. John Wiley & Sons, New York, 1995.Google Scholar
  2. 2.
    J. O. Berger and J. Deely. A Bayesian approach to ranking and selection of related means with alternatives to analysis-of-variance methodology. Journal of the American Statistical Association, 83(402):364–373, 1988.CrossRefGoogle Scholar
  3. 3.
    J. Blanchet, J. Liu, and B. Zwart. Large deviations perspective on ordinal optimization of heavy-tailed systems. In S. J. Mason, R. R. Hill, L. Mönch, O. Rose, T. Jefferson, J. Fowler, editors, Proceedings of the 2008 Winter Simulation Conference, 489–494, 2008.Google Scholar
  4. 4.
    J. Branke, S. E. Chick, and C. Schmidt. Selecting a selection procedure. Management Science, 53:1916–1932, 2007.CrossRefGoogle Scholar
  5. 5.
    M. W. Brantley, L. H. Lee, C. H. Chen, and A. Chen. Efficient simulation budget allocation with regression. IIE Transactions, 45(3):291–308, 2013.CrossRefGoogle Scholar
  6. 6.
    M. W. Brantley, L. H. Lee, C.-H. Chen, and J. Xu. An efficient simulation budget allocation method incorporating regression for partitioned domains. Automatica, 50(5):1391–1400, 2014.Google Scholar
  7. 7.
    A. Charnes, W. W. Cooper, and E. Rhodes. Measuring the efficiency of decision making units. European Journal of Operational Research, 2:429–444, 1978.CrossRefGoogle Scholar
  8. 8.
    C. H. Chen and L. H. Lee. Stochastic Simulation Optimization: An Optimal Computing Budget Allocation. World Scientific Publishing Co, 2011.Google Scholar
  9. 9.
    C. H. Chen, D. He, and M. C. Fu. Efficient dynamic simulation allocation in ordinal optimization. IEEE Transactions on Automatic Control, 51(12):2005–2009, 2006.CrossRefGoogle Scholar
  10. 10.
    C. H. Chen, D. He, M. C. Fu, and L. H. Lee. Efficient simulation budget allocation for selecting an optimal subset. INFORMS Journal of Computing, 20:579–595, 2008.CrossRefGoogle Scholar
  11. 11.
    C. H. Chen, V. Kumar, and Y.C. Luo. Motion planning of walking robots in environments with uncertainty. Journal of Robotic Systems, 16(10):527–545, 1999.CrossRefGoogle Scholar
  12. 12.
    C. H. Chen, J. Lin, E. Yücesan, and S. E. Chick. Simulation budget allocation for further enhancing the efficiency of ordinal optimization. Discrete Event Dynamic Systems: Theory and Applications, 10:251–270, 2000.CrossRefGoogle Scholar
  13. 13.
    C. H. Chen, E. Yücesan, L. Dai, and H. C. Chen. Efficient computation of optimal budget allocation for discrete event simulation experiment. IIE Transactions, 42(1):60–70, 2010.CrossRefGoogle Scholar
  14. 14.
    S. E. Chick, and P. Frazier. Sequential sampling with economics of selection procedures. Management Science, 58(3):550–569, 2012.CrossRefGoogle Scholar
  15. 15.
    S. E. Chick, and N. Gans. Economic analysis of simulation selection problems. Management Science, 55(3):421–437, 2009.CrossRefGoogle Scholar
  16. 16.
    S. E. Chick and K. Inoue. New two-stage and sequential procedures for selecting the best simulated system. Operations Research, 49(5):732–743, 2001.CrossRefGoogle Scholar
  17. 17.
    S. E. Chick and K. Inoue. New procedures to select the best simulated system using common random numbers. Management Science, 47(8):1133–1149, 2001.CrossRefGoogle Scholar
  18. 18.
    S. E. Chick, J. Branke, and C. Schmidt. Sequential sampling to myopically maximize the expected value of information. INFORMS Journal of Computing, 22:71–80, 2010.CrossRefGoogle Scholar
  19. 19.
    S. E. Chick and Y. Z. Wu. Selection procedures with frequentist expected opportunity cost bounds. Operations Research, 53(5):867–878, 2005.CrossRefGoogle Scholar
  20. 20.
    M. H. de Groot. Optimal Statistical Decisions. McGraw-Hill, New York, 1970.Google Scholar
  21. 21.
    E. J. Dudewicz and S.R. Dalal. Allocation of observations in ranking and selection with unequal variances. The Indian Journal of Statistics, 37B, 1:28–78, 1975.Google Scholar
  22. 22.
    P. Frazier, W. B. Powell, and S. Dayanik. A knowledge-gradient policy for sequential information collection. SIAM Journal on Control and Optimization, 47(5):2410–2439, 2008.CrossRefGoogle Scholar
  23. 23.
    P. Frazier, W. B. Powell, and S. Dayanik. The knowledge-gradient policy for correlated normal beliefs. INFORMS Journal of Computing, 21:599–613, 2009.CrossRefGoogle Scholar
  24. 24.
    P. Frazier, J. Xie, and S. E. Chick. Bayesian optimization via simulation with correlated sampling and correlated prior beliefs. In S. Jain, R. R. Creasey, J. Himmelspach, K. P. White, M. Fu, editors, Proceedings of the 2011 Winter Simulation Conference, 3979–3911, 2011.Google Scholar
  25. 25.
    M. C. Fu, C. H. Chen, and L. Shi. Some topics for simulation optimization. In S. J. Mason, R. R. Hill, L. Mönch, O. Rose, T. Jefferson, J. Fowler, editors, Proceedings the 2008 Winter Simulation Conference, 27–38, 2008.Google Scholar
  26. 26.
    M. C. Fu, F. W. Glover, and J. April. Simulation optimization: A review, new developments, and applications. In M. E. Kuhl, N. M. Steiger, F. B. Armstrong, J. A. Joines, editors, Proceedings the 2005 Winter Simulation Conference, 83–95, 2005.Google Scholar
  27. 27.
    M. C. Fu, J. Q. Hu, C. H. Chen, and X. Xiong. Simulation allocation for determining the best design in the presence of correlated sampling. INFORMS Journal of Computing, 19:101–111, 2007.CrossRefGoogle Scholar
  28. 28.
    P. Glynn and S. Juneja. A large deviations perspective on ordinal optimization. In R. G. Ingalls, M. D. Rosetti, J. S. Smith, B. A. Peters, editors, Proceedings of the 2004 Winter Simulation Conference, 577–585, 2004.Google Scholar
  29. 29.
    D. Goldsman and B. L. Nelson. Comparing systems via simulation. In J. Banks, editor, Handbook of Simulation: Principles, Methodology, Advances, Applications, and Practice. John Wiley & Sons, New York, 273–306, 1998.Google Scholar
  30. 30.
    S. S. Gupta and K. J. Miescke. Bayesian look ahead one stage sampling allocations for selecting the largest normal mean. Statistical Papers, 35:169–177, 1994.CrossRefGoogle Scholar
  31. 31.
    S. S. Gupta and K. J. Miescke. Bayesian look ahead one-stage sampling allocations for selecting the best population. Journal of Statistical Planning and Inference, 54(2):229–244, 1996.CrossRefGoogle Scholar
  32. 32.
    J. M. Hammersley and D.C. Hanscomb. Monte Carlo Methods. Methuen, London, 1964.CrossRefGoogle Scholar
  33. 33.
    D. He, S. E. Chick, and C. H. Chen. The opportunity cost and OCBA selection procedures in ordinal optimization. IEEE Transactions on Systems, Man, and Cybernetics—Part C (Applications and Reviews), 37(4):951–961, 2007.Google Scholar
  34. 34.
    D. He, L. H. Lee, C. H. Chen, and M. C. Fu, and S. Wasserkrug. Simulation optimization using the cross-entropy method with optimal computing budget allocation. ACM Transactions on Modeling and Computer Simulation, 20(1):Article 4, 2010.Google Scholar
  35. 35.
    L. J. Hong and B. L. Nelson. A brief introduction to optimization via simulation. In M. D. Rosetti, R. R. Hill, B. Johansson, A. Dunkin, R. Ingalls, editors, Proceedings of the 2009 Winter Simulation Conference, 75–85, 2009.Google Scholar
  36. 36.
    B. W. Hsieh, C. H. Chen, and S.C. Chang. Scheduling semiconductor wafer fabrication by using ordinal optimization-based simulation. IEEE Transactions on Robotics and Automation, 17(5):599–608, 2001.CrossRefGoogle Scholar
  37. 37.
    Q. S. Jia, E. Zhou, and C. H. Chen. Efficient computing budget allocation for finding simplest good designs. IIE Transactions, 45(7):736–750, 2013.CrossRefGoogle Scholar
  38. 38.
    S.-H. Kim and B. L. Nelson. A fully sequential procedure for indifference-zone selection in simulation. ACM Transactions on Modeling and Computer Simulation, 11:251–273, 2001.CrossRefGoogle Scholar
  39. 39.
    S.-H. Kim. and B. L. Nelson. Selecting the best system. In S. G. Henderson, B. L. Nelson, editors, Handbooks in Operations Research and Management Science: Simulation. Elsevier Science, Oxford, 501–534, 2006.Google Scholar
  40. 40.
    L. W. Koenig and A. M. Law. A procedure for selecting a subset of size m containing the one best of k independent normal populations, with applications to simulation. Communications in Statistics B14, 3:719–734, 1985.Google Scholar
  41. 41.
    A. M. Law. Simulation Modeling and Analysis, 5th Edition. McGraw-Hill, New York, 2013.Google Scholar
  42. 42.
    L. H. Lee, E. P. Chew, and P. Manikam. A general framework on the simulation-based optimization under fixed computing budget. European Journal of Operational Research, 174:1828–1841, 2006.CrossRefGoogle Scholar
  43. 43.
    L. H. Lee, E. P. Chew, S. Teng, and D. Goldsman. Optimal computing budget allocation for multi-objective simulation models. In R. G. Ingalls, M. D. Rosetti, J. S. Smith, B. A. Peters, editors, Proceedings of the 2004 Winter Simulation Conference, 586–594, 2004.Google Scholar
  44. 44.
    L. H. Lee, E. P. Chew, S. Teng, and D. Goldsman. Finding the non-dominated Pareto set for multi-objective simulation models. IIE Transactions, 42:656–674, 2010.CrossRefGoogle Scholar
  45. 45.
    L. H. Lee, N. A. Pujowidianto, L. W. Li, C. H. Chen, and C. M. Yap. Approximate simulation budget allocation for selecting the best system in the presence of stochastic constraints. IEEE Transactions on Automatic Control, 57(11):2940–2945, 2012.CrossRefGoogle Scholar
  46. 46.
    D. J. Morrice, M. W. Brantley, and C. H. Chen. An efficient ranking and selection procedure for a linear transient mean performance measure. In S. J. Mason, R. R. Hill, L. Mönch, O. Rose, T. Jefferson, J. Fowler, editors, Proceedings of the 2008 Winter Simulation Conference, 290–296, 2008.Google Scholar
  47. 47.
    D. J. Morrice, M. W. Brantley, and C. H. Chen. A transient means ranking and selection procedure with sequential sampling constraints. In M. D. Rosetti, R. R. Hill, B. Johansson, A. Dunkin, R. Ingalls, editors, Proceedings of the 2009 Winter Simulation Conference, 590–600, 2009.Google Scholar
  48. 48.
    B. L. Nelson, J. Swann, D. Goldsman, and W. M. Song. Simple procedures for selecting the best simulated system when the number of alternatives is large. Operations Research, 49(6):950–963, 2001.CrossRefGoogle Scholar
  49. 49.
    W. B. Powell and I. O. Ryzhov. Optimal Learning. Wiley, 2012.Google Scholar
  50. 50.
    Y. Rinott. On two-stage selection procedures and related probability inequalities. Communications in Statistics A7:799–811, 1978.CrossRefGoogle Scholar
  51. 51.
    R. Rubinstein. The cross-entropy method for combinatorial and continuous optimization. Methodology and Computing in Applied Probability, 1(2): 127–190, 1999.CrossRefGoogle Scholar
  52. 52.
    I. O. Ryzhov, W. B. Powell, and P. Frazier. The knowledge-gradient algorithm for a general class of online learning problems. Operations Research, 60(1):180–195, 2012.CrossRefGoogle Scholar
  53. 53.
    L. Shi and C. H. Chen. A new algorithm for stochastic discrete resource allocation optimization. Discrete Event Dynamic Systems: Theory and Applications, 10:271–294, 2000.CrossRefGoogle Scholar
  54. 54.
    J. F. Shortle, C. H. Chen, B. Crain, A. Brodsky, and D. Brod. Optimal splitting for rare-event simulation. IIE Transactions, 44(5):352–367, 2012.CrossRefGoogle Scholar
  55. 55.
    R. Szechtman and E. Yücesan. A new perspective on feasibility determination. In S. J. Mason, R. R. Hill, L. Mönch, O. Rose, T. Jefferson, J. Fowler, editors, Proceedings of the 2008 Winter Simulation Conference, 273–280, 2008.Google Scholar
  56. 56.
    E. Tekin and I. Sabuncuoglu. Simulation optimization: A comprehensive review on theory and applications. IIE Transactions, 36:1067–1081, 2004.CrossRefGoogle Scholar
  57. 57.
    L. Trailovic and L. Y. Pao. Computing budget allocation for efficient ranking and selection of variances with application to target tracking algorithms. IEEE Transactions on Automatic Control, 49:58–67, 2004.CrossRefGoogle Scholar
  58. 58.
    R. Waeber, P. Frazier, and S. G. Henderson. Performance measures for ranking and selection procedures. In B. Johansson, S. Jain, J. Montoya-Torres, J. Hugan, E. Yücesan, editors, Proceedings of the 2010 Winter Simulation Conference, 1235–1245, 2010.Google Scholar
  59. 59.
    W. P. Wong, W. Jaruphongsa, and L. H. Lee. Budget allocation for effective data collection in predicting an accurate DEA efficiency score. IEEE Transactions on Automatic Control, 56(6):1235–1245, 2011.CrossRefGoogle Scholar
  60. 60.
    S. Yan, E. Zhou, and C. H. Chen. Efficient selection of a set of good enough designs with complexity preference. IEEE Transactions on Automation Science and Engineering, 9(3):596–606, 2012.CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2015

Authors and Affiliations

  • Chun-Hung Chen
    • 1
  • Stephen E. Chick
    • 2
  • Loo Hay Lee
    • 3
  • Nugroho A. Pujowidianto
    • 4
  1. 1.George Mason UniversityFairfaxUSA
  2. 2.INSEADFontainebleauFrance
  3. 3.National University of SingaporeSingaporeSingapore
  4. 4.Hewlett-Packard SingaporeSingaporeSingapore

Personalised recommendations