Skip to main content
SpringerLink
Log in

Data envelopment analysis for algorithm efficiency assessment in metamodel-based simulation optimization

  • ORIGINAL ARTICLE
  • Published:
The International Journal of Advanced Manufacturing Technology Aims and scope Submit manuscript

Abstract

In the last years, the use of metamodel-based simulation optimization techniques to solve industrial problems stood out as a promising research field, mainly due to the advance of machine learning techniques. The number of metamodeling studies has grown considerably in recent years, but many academics and practitioners still have doubts about which metamodels to choose for their projects. In this way, some studies have compared the effectiveness of metamodeling algorithms. However, they have just analyzed the performance of one or more metrics separately; i.e., they did not analyze the overall efficiency of these metamodels. Basing the metamodels’ choice only on one or more metrics empirically might generate biases, causing distortions in decision-making. Therefore, we propose using the multi-criteria data envelopment analysis (MCDEA) model to systematically compare some of the main machine learning algorithms (support vector machine, artificial neural network, gradient-boosted trees, random forest, and Gaussian process). To evaluate the proposed approach, we developed discrete-event simulation models of three real case studies to obtain their input and output data. Moreover, we used machine learning algorithms to train and optimize the metamodels and, finally, new-MCDEA was adopted to compare the metamodels’ efficiency considering the associated error, fitting, training and prediction times, and response, among other metrics. Different from traditional comparison approaches, where different algorithms could be chosen depending on the decision-maker bias, the proposed work allowed a good balance between all metrics, and for all cases, the metamodels based on gradient-boosted trees were considered the most efficient.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Source: Pontes et al. [62]

Fig. 2
Fig. 3

Source: Amaral et al. [27]

Similar content being viewed by others

Availability of data and material

Not applicable.

Code availability

Not applicable.

References

  1. dos Santos CH, Gabriel GT, Amaral JVS, do, et al (2021) Decision-making in a fast fashion company in the Industry 4.0 era: a digital twin proposal to support operational planning. Int J Adv Manuf Technol 116:1653–1666. https://doi.org/10.1007/s00170-021-07543-z

    Article  Google Scholar 

  2. Sousa Junior WT, Montevechi JAB, de Miranda R, C, et al (2020) Shop floor simulation optimization using machine learning to improve parallel metaheuristics. Expert Syst Appl. https://doi.org/10.1016/j.eswa.2020.113272

    Article  Google Scholar 

  3. Law AM Simulation modeling and analysis, 5th ed. McGraw-Hil, New York

  4. Xu J, Huang E, Hsieh L et al (2016) Simulation optimization in the era of Industrial 4.0 and the Industrial Internet. J Simul 10:310–320. https://doi.org/10.1057/s41273-016-0037-6

    Article  Google Scholar 

  5. Yin H, Fang H, Wang Q, Wen G (2016) Design optimization of a MASH TL-3 concrete barrier using RBF-based metamodels and nonlinear finite element simulations. Eng Struct 114:122–134. https://doi.org/10.1016/j.engstruct.2016.02.009

    Article  Google Scholar 

  6. Oliveira MLM, Montevechi JAB, Pinho AF, Miranda RC (2017) Using hybrid simulation to represent the human factor in production systems. Int J Simul Model 16:263–274. https://doi.org/10.2507/IJSIMM16(2)7.378

    Article  Google Scholar 

  7. Vilela FF, Amaral JVS, do, Leal G dos S, et al (2021) Economic feasibility analysis for co-generation of power by diesel generator at peak time: Monte Carlo simulation approach in a hospital. Acta Sci Technol 43:2–9. https://doi.org/10.4025/actascitechnol.v43i3.50965

    Article  Google Scholar 

  8. Miranda RDC, Montevechi JAB, Da SAF, Marins FAS (2014) A new approach to reducing search space and increasing efficiency in simulation optimization problems via the fuzzy-DEA-BCC. Math Probl Eng. https://doi.org/10.1155/2014/450367

    Article  MathSciNet  MATH  Google Scholar 

  9. Siebers PO, MacAl CM, Garnett J et al (2010) Discrete-event simulation is dead, long live agent-based simulation! J Simul 4:204–210. https://doi.org/10.1057/jos.2010.14

    Article  Google Scholar 

  10. Gabriel GT, Campos AT, Leal F, Montevechi JAB (2020) Good practices and deficiencies in conceptual modelling: a systematic literature review. J Simul 00:1–17. https://doi.org/10.1080/17477778.2020.1764875

    Article  Google Scholar 

  11. Pereira TF, Montevechi JAB, de Miranda R, C, Friend JD, (2015) Integrating soft systems methodology to aid simulation conceptual modeling. Int Trans Oper Res 22:265–285. https://doi.org/10.1111/itor.12133

    Article  MathSciNet  MATH  Google Scholar 

  12. Barton RR (2009) Simulation optimization using metamodel. In: Winter Simulation Conference. pp 230–238

  13. Shahi MRM, Mehdipour EF, Amiri M (2016) Optimization using simulation and response surface methodology with an application on subway train scheduling. Int Trans Oper Res 23:797–811. https://doi.org/10.1111/itor.12150

    Article  MathSciNet  MATH  Google Scholar 

  14. Jalali H, Van NI (2015) Simulation optimization in inventory replenishment: a classification. IIE Trans 47:1217–1235. https://doi.org/10.1080/0740817X.2015.1019162

    Article  Google Scholar 

  15. Fu MC (2002) Feature article: Optimization for simulation: theory vs. practice. INFORMS J Comput 14:192–215. https://doi.org/10.1287/ijoc.14.3.192.113

    Article  MathSciNet  MATH  Google Scholar 

  16. de Miranda R, C, Montevechi JAB, da Silva AF, Marins FAS, (2017) Increasing the efficiency in integer simulation optimization: reducing the search space through data envelopment analysis and orthogonal arrays. Eur J Oper Res 262:673–681. https://doi.org/10.1016/j.ejor.2017.04.016

    Article  MATH  Google Scholar 

  17. Amaral JVS, do, Montevechi AB, Miranda R de C, de Sousa Junior WT, (2022) Metamodel-based simulation optimization: a systematic literature review. Simul Model Pract Theory. https://doi.org/10.1016/j.simpat.2021.102403

    Article  Google Scholar 

  18. Parnianifard A, Azfanizam AS, Ariffin MKA, Ismail MIS (2020) Comparative study of metamodeling and sampling design for expensive and semi-expensive simulation models under uncertainty. SIMULATION 96:89–110. https://doi.org/10.1177/0037549719846988

    Article  Google Scholar 

  19. Jalali H, Van NI (2015) Simulation optimization in inventory replenishment: a classification. IIE Transactions (Institute of Industrial Engineers) 47:1217–1235. https://doi.org/10.1080/0740817X.2015.1019162

    Article  Google Scholar 

  20. Pedrielli G, Wang S, Ng SH et al (2020) An extended two-stage sequential optimization approach: properties and performance. Eur J Oper Res 287:929–945. https://doi.org/10.1016/j.ejor.2020.04.045

    Article  MathSciNet  MATH  Google Scholar 

  21. dos Santos CH, Montevechi JAB, de Queiroz JA et al (2021) Decision support in productive processes through DES and ABS in the digital twin era: a systematic literature review. Int J Prod Res. https://doi.org/10.1080/00207543.2021.1898691

    Article  Google Scholar 

  22. Yousefi M, Yousefi M, Fogliatto FS (2020) Simulation-based optimization methods applied in hospital emergency departments: a systematic review. SIMULATION 96:791–806. https://doi.org/10.1177/0037549720944483

    Article  Google Scholar 

  23. Sousa Junior WT, Montevechi JABABB, de Miranda R, C, et al (2019) Economic lot-size using machine learning, parallelism, metaheuristic and simulation. Int J Simul Model 18:205–216. https://doi.org/10.2507/IJSIMM18(2)461

    Article  Google Scholar 

  24. Hüllen G, Zhai J, Kim SH et al (2020) Managing uncertainty in data-driven simulation-based optimization. Comput Chem Eng 136:106519. https://doi.org/10.1016/j.compchemeng.2019.106519

    Article  Google Scholar 

  25. Li YF, Ng SH, Xie M, Goh TN (2010) A systematic comparison of metamodeling techniques for simulation optimization in decision support Systems. Appl Soft Comput 10:1257–1273. https://doi.org/10.1016/j.asoc.2009.11.034

    Article  Google Scholar 

  26. Lal A, Datta B (2020) Performance Evaluation of Homogeneous and Heterogeneous Ensemble Models for Groundwater Salinity Predictions: a Regional-Scale Comparison Study. Water, Air, & Soil Pollution 231:320. https://doi.org/10.1007/s11270-020-04693-w

  27. Amaral JVS, do, de Carvalho Miranda R, Montevechi JAB et al (2022) Metamodeling-based simulation optimization in manufacturing problems: a comparative study. Int J Adv Manuf Technol. https://doi.org/10.1007/s00170-022-09072-9

  28. De La Fuente R, Smith R (2017) Metamodeling a system dynamics model: a contemporany comparison of methods. Proc - Winter Simul Conf. Las Vegas, NY, pp 1689–1699

    Google Scholar 

  29. Luo J, Ji Y, Lu W (2019) Comparison of surrogate models based on different sampling methods for groundwater remediation. J Water Resour Plan Manag 145:04019015. https://doi.org/10.1061/(asce)wr.1943-5452.0001062

    Article  Google Scholar 

  30. Charnes A, Cooper WW, Rhodes E (1978) Measuring the efficiency of decision making units. Eur J Oper Res 2:429–444. https://doi.org/10.1016/0377-2217(78)90138-8

    Article  MathSciNet  MATH  Google Scholar 

  31. Banker RD, Charnes A, Cooper WW et al (1989) An introduction to data envelopment analysis with some of its models and their uses. Research in Governmental and Non-Profit Accounting 5:125–163

    Google Scholar 

  32. Hatami-Marbini A, Toloo M (2017) An extended multiple criteria data envelopment analysis model. Expert Syst Appl 73:201–219. https://doi.org/10.1016/j.eswa.2016.12.030

    Article  Google Scholar 

  33. da Silva AF, Marins FAS, Dias EX, Ushizima CA (2021) Improving manufacturing cycle efficiency through new multiple criteria data envelopment analysis models: an application in green and lean manufacturing processes. Prod Plan and Control 32:104–120. https://doi.org/10.1080/09537287.2020.1713413

    Article  Google Scholar 

  34. Zhu N, Zhu C, Emrouznejad A (2021) A combined machine learning algorithms and DEA method for measuring and predicting the efficiency of Chinese manufacturing listed companies. Int J Manag Sci 6:435–448. https://doi.org/10.1016/j.jmse.2020.10.001

    Article  Google Scholar 

  35. Kleijnen JPC (2017) Regression and Kriging metamodels with their experimental designs in simulation: a review. Eur J Oper Res 256:1–16. https://doi.org/10.1016/j.ejor.2016.06.041

    Article  MathSciNet  MATH  Google Scholar 

  36. De La Fuente R, Erazo I, Smith RL (2019) Enabling intelligent processes in simulation utilizing the TensorFlow deep learning resources. Proc - Win Simul Conf 2018-Decem:1108–1119. https://doi.org/10.1109/WSC.2018.8632539

  37. Dey A (2016) Machine learning algorithms: a review. International Journal of Computer Science and Information Technologies 7:1174–1179

    Google Scholar 

  38. Géron A (2017) Hands-on machine learning with Scikit-Learn and TensorFlow. O’Reilly Media

  39. Hastie T, Tibshirani R, Friedman J (2008) The elements of statistical learning

  40. Xia Y, Liu C, Li YY, Liu N (2017) A boosted decision tree approach using Bayesian hyper-parameter optimization for credit scoring. Expert Syst Appl 78:225–241. https://doi.org/10.1016/j.eswa.2017.02.017

    Article  Google Scholar 

  41. Østergård T, Jensen RL, Maagaard SE (2018) A comparison of six metamodeling techniques applied to building performance simulations. Appl Energy 211:89–103. https://doi.org/10.1016/j.apenergy.2017.10.102

    Article  Google Scholar 

  42. Bergmeir C, Benítez JM (2012) On the use of cross-validation for time series predictor evaluation. Inf Sci 191:192–213. https://doi.org/10.1016/j.ins.2011.12.028

    Article  Google Scholar 

  43. Roy DK, Datta B (2018) A surrogate based multi-objective management model to control saltwater intrusion in multi-layered coastal aquifer systems. Civ Eng Environ Syst. https://doi.org/10.1080/10286608.2018.1431777

    Article  Google Scholar 

  44. Hu B, Palta M, Shao J (2006) Properties of R2 statistics for logistic regression. Stat Med 25:1383–1395. https://doi.org/10.1002/sim.2300

    Article  MathSciNet  Google Scholar 

  45. Kleijnen JPC, Kleijnen JPC (2017) Simulation-optimization via Kriging and bootstrapping: a survey simulation-optimization via Kriging and bootstrapping: a survey. J Simul. https://doi.org/10.1057/jos.2014.4

    Article  MATH  Google Scholar 

  46. Sain SR, Vapnik VN (1996) The nature of statistical learning theory. Technometrics 38:409. https://doi.org/10.2307/1271324

    Article  Google Scholar 

  47. Yadav B, Ch S, Mathur S, Adamowski J (2016) Estimation of in-situ bioremediation system cost using a hybrid extreme learning machine (ELM)-particle swarm optimization approach. J Hydrol 543:373–385. https://doi.org/10.1016/j.jhydrol.2016.10.013

    Article  Google Scholar 

  48. Hou Z, Dai Z, Lao W et al (2019) Application of mixed-integer nonlinear optimization programming based on ensemble surrogate model for dense nonaqueous phase liquid source identification in groundwater. Environ Eng Sci 36:699–709. https://doi.org/10.1089/ees.2018.0366

    Article  Google Scholar 

  49. Jeng JT (2006) Hybrid approach of selecting hyperparameters of support vector machine for regression. IEEE Trans Syst Man Cybern B Cybern 36:699–709. https://doi.org/10.1109/TSMCB.2005.861067

    Article  Google Scholar 

  50. Vapnik VN (1999) The nature of statistical learning theory New York

  51. Oliveira ALI, Braga PL, Lima RMF, Cornélio ML (2010) GA-based method for feature selection and parameters optimization for machine learning regression applied to software effort estimation. Inf Softw Technol 52:1155–1166. https://doi.org/10.1016/j.infsof.2010.05.009

    Article  Google Scholar 

  52. Cherkassky V, Ma Y (2004) Practical selection of SVM parameters and noise estimation for SVM regression. Neural Netw 17:113–126. https://doi.org/10.1016/S0893-6080(03)00169-2

    Article  MATH  Google Scholar 

  53. Villarreal-Marroquín MG, Mulyana R, Castro JM, Cabrera-Ríos M (2011) Selecting process parameter in injection molding via simulation optimization. J Polym Eng 31:387–395. https://doi.org/10.1515/POLYENG.2011.086

    Article  Google Scholar 

  54. Dang XP (2014) General frameworks for optimization of plastic injection molding process parameters. Simul Model Pract Theory 41:15–27. https://doi.org/10.1016/j.simpat.2013.11.003

    Article  Google Scholar 

  55. Dunke F, Nickel S (2020) Neural networks for the metamodeling of simulation models with online decision making. Simul Model Pract Theory 99:102016. https://doi.org/10.1016/j.simpat.2019.102016

    Article  Google Scholar 

  56. Zeinali F, Mahootchi M, Sepehri MM (2015) Resource planning in the emergency departments: a simulation-based metamodeling approach. Simul Model Pract Theory 53:123–138. https://doi.org/10.1016/j.simpat.2015.02.002

    Article  Google Scholar 

  57. Amiri F, Shirazi B, Tajdin A (2019) Multi-objective simulation optimization for uncertain resource assignment and job sequence in automated flexible job shop. Appl Soft Comput 75:190–202. https://doi.org/10.1016/j.asoc.2018.11.015

    Article  Google Scholar 

  58. Dengiz B, Alabas-Uslu C, Dengiz O (2009) Optimization of manufacturing systems using a neural network metamodel with a new training approach. J Oper Res Soc 60:1191–1197. https://doi.org/10.1057/palgrave.jors.2602620

    Article  MATH  Google Scholar 

  59. Mohammad Nezhad A, Mahlooji H (2014) An artificial neural network meta-model for constrained simulation optimization. J Oper Res Soc 65:1232–1244. https://doi.org/10.1057/jors.2013.73

    Article  Google Scholar 

  60. Jackson I, Tolujevs J, Lang S, Kegenbekov Z (2019) Metamodelling of inventory-control simulations based on a multilayer perceptron. Transp Telecommun 20:251–259. https://doi.org/10.2478/ttj-2019-0021

    Article  Google Scholar 

  61. Hassoun MH (1995) Fundamentals of artificial neural networks Cambridge

  62. Pontes FJ, Ferreira JR, Silva MB et al (2010) Artificial neural networks for machining processes surface roughness modeling. Int J Adv Manuf Technol 49:879–902. https://doi.org/10.1007/s00170-009-2456-2

    Article  Google Scholar 

  63. Braga PL, Oliveira ALI, Meira SRL (2007) Software effort estimation using machine learning techniques with robust confidence intervals. In: 19th IEEE International Conference on Tools with Artificial Intelligence(ICTAI 2007). Patras, pp 181–185

  64. Breiman L (2001) Random forests. Mach Learn 45:5–32. https://doi.org/10.1201/9780429469275-8

    Article  MATH  Google Scholar 

  65. Ranjbar A, Mahjouri N, Cherubini C (2020) Development of an efficient conjunctive meta-model-based decision-making framework for saltwater intrusion management in coastal aquifers. J Hydro-Environ Res 29:45–58. https://doi.org/10.1016/j.jher.2019.11.005

    Article  Google Scholar 

  66. Candelieri A, Perego R, Archetti F (2018) Bayesian optimization of pump operations in water distribution systems. J Global Optim 71:213–235. https://doi.org/10.1007/s10898-018-0641-2

    Article  MathSciNet  MATH  Google Scholar 

  67. Sousa Junior WT, de, Montevechi JAB, Miranda R de CCC, et al (2020) Shop floor simulation optimization using machine learning to improve parallel metaheuristics. Expert Syst Appl. https://doi.org/10.1016/j.eswa.2020.113272

    Article  Google Scholar 

  68. Steer KCB, Wirth A, Halgamuge SK (2012) Decision tree ensembles for online operation of large smart grids. Energy Convers Manage 59:9–18. https://doi.org/10.1016/j.enconman.2012.01.010

    Article  Google Scholar 

  69. Schapire RE (1990) The strength of weak learnability. Mach Learn 5:197–227. https://doi.org/10.1023/A:1022648800760

    Article  Google Scholar 

  70. Friedman JH (2002) Stochastic gradient boosting. Comput Stat Data Anal 38:367–378. https://doi.org/10.1016/S0167-9473(01)00065-2

    Article  MathSciNet  MATH  Google Scholar 

  71. Krige DG (1951) A statistical approach to some basic mine valuation problems on the Witwatersrand. J South Afr Inst Min Metall 52:119–139

    Google Scholar 

  72. Nguyen-Tuong D, Seeger M, Peters J (2009) Model learning with local Gaussian process regression. Adv Robot 23:2015–2034. https://doi.org/10.1163/016918609X12529286896877

    Article  Google Scholar 

  73. Wan HP, Ren WX (2015) Parameter selection in finite-element-model updating by global sensitivity analysis using Gaussian process metamodel. J Struct Eng 141:1–11. https://doi.org/10.1061/(ASCE)ST.1943-541X.0001108

    Article  Google Scholar 

  74. Rostami H, Khaksar Manshad A (2014) Application of evolutionary Gaussian processes regression by particle swarm optimization for prediction of dew point pressure in gas condensate reservoirs. Neural Comput Appl 24:705–713. https://doi.org/10.1007/s00521-012-1275-9

    Article  Google Scholar 

  75. Banker RD, Charnes A, Cooper WW (1984) Some models for estimating technical and scale inefficiencies in data envelopment analysis. Manage Sci 30:1078–1092. https://doi.org/10.1287/mnsc.30.9.1078

    Article  MATH  Google Scholar 

  76. Li XB, Reeves GR (1999) Multiple criteria approach to data envelopment analysis. Eur J Oper Res 115:507–517. https://doi.org/10.1016/S0377-2217(98)00130-1

    Article  MATH  Google Scholar 

  77. Ghasemi MR, Ignatius J, Emrouznejad A (2014) A bi-objective weighted model for improving the discrimination power in MCDEA. Eur J Oper Res 233:640–650. https://doi.org/10.1016/j.ejor.2013.08.041

    Article  MathSciNet  MATH  Google Scholar 

  78. dos Rubem AP, S, Mello JCCBS de, Meza LA, (2017) A goal programming approach to solve the multiple criteria DEA model. Eur J Oper Res 260:134–139. https://doi.org/10.1016/j.ejor.2016.11.049

    Article  MathSciNet  MATH  Google Scholar 

  79. da Silva AF, Marins FAS, Dias EX (2020) Improving the discrimination power with a new multi-criteria data envelopment model. Ann Oper Res 287:127–159. https://doi.org/10.1007/s10479-019-03446-1

    Article  MathSciNet  MATH  Google Scholar 

  80. Anderson TR, Hollingsworth K, Inman L (2002) The fixed weighting nature of a cross-evaluation model. J Prod Anal 17:249–255. https://doi.org/10.1023/A:1015012121760

    Article  Google Scholar 

  81. Roy DK, Datta B (2019) Adaptive management of coastal aquifers using entropy-set pair analysis–based three-dimensional sequential monitoring network design. J Hydrol Eng 24:04018072. https://doi.org/10.1061/(asce)he.1943-5584.0001765

    Article  Google Scholar 

  82. Halkos G, Petrou KN (2019) Treating undesirable outputs in DEA: a critical review. Econ Anal Policy 62:97–104. https://doi.org/10.1016/j.eap.2019.01.005

    Article  Google Scholar 

Download references

Funding

This work was funded by the Conselho Nacional de Desenvolvimento Científico e Tecnológico (CNPq), Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES), and Fundação de Amparo à Pesquisa do Estado de Minas Gerais (FAPEMIG).

Author information

Authors and Affiliations

  1. Production Engineering and Management Institute, Federal University of Itajubá (UNIFEI), Ave. BPS, Itajubá, Minas Gerais, 1303, Brazil

    João Victor Soares do Amaral, Rafael de Carvalho Miranda, José Arnaldo Barra Montevechi & Carlos Henrique dos Santos

  2. Department of Production, São Paulo State University, São Paulo, Brazil

    Aneirson Francisco da Silva

Authors
  1. João Victor Soares do Amaral
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Rafael de Carvalho Miranda
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. José Arnaldo Barra Montevechi
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Carlos Henrique dos Santos
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Aneirson Francisco da Silva
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to João Victor Soares do Amaral.

Ethics declarations

Conflict of interest

The authors declare no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

do Amaral, J.V.S., de Carvalho Miranda, R., Montevechi, J.A.B. et al. Data envelopment analysis for algorithm efficiency assessment in metamodel-based simulation optimization. Int J Adv Manuf Technol 121, 7493–7507 (2022). https://doi.org/10.1007/s00170-022-09864-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00170-022-09864-z

Keywords

Search

Navigation