Deterministic Global Optimization with Artificial Neural Networks Embedded

  • Artur M. Schweidtmann
  • Alexander MitsosEmail author


Artificial neural networks are used in various applications for data-driven black-box modeling and subsequent optimization. Herein, we present an efficient method for deterministic global optimization of optimization problems with artificial neural networks embedded. The proposed method is based on relaxations of algorithms using McCormick relaxations in a reduced space (Mitsos et al. in SIAM J Optim 20(2):573–601, 2009) employing the convex and concave envelopes of the nonlinear activation function. The optimization problem is solved using our in-house deterministic global solver. The performance of the proposed method is shown in four optimization examples: an illustrative function, a fermentation process, a compressor plant and a chemical process. The results show that computational solution time is favorable compared to a state-of-the-art global general-purpose optimization solver.


Surrogate-based optimization Multilayer perceptron McCormick relaxations Machine learning MAiNGO 

Mathematics Subject Classification

90C26 90C30 90C90 68T01 



The authors gratefully acknowledge the financial support of the Kopernikus project SynErgie by the Federal Ministry of Education and Research (BMBF) and the project supervision by the project management organization Projektträger Jülich (PtJ). We are grateful to Jaromił Najam, Dominik Bongartz and Susanne Sass for their work on MAiNGO and Benoît Chachuat for providing MC++. We thank Eduardo Schultz for providing the model of the Cumene process, Adrian Caspari and Pascal Schäfer for helpful discussions and Linus Netze and Nils Graß  for implementing case studies. Finally, we thank the associate editor and the anonymous reviewers for their valuable comments and suggestions.


  1. 1.
    Hornik, K., Stinchcombe, M., White, H.: Multilayer feedforward networks are universal approximators. Neural Netw. 2(5), 359–366 (1989). zbMATHCrossRefGoogle Scholar
  2. 2.
    Gasteiger, J., Zupan, J.: Neural networks in chemistry. Angew. Chem. Int. Edn. Engl. 32(4), 503–527 (1993). CrossRefGoogle Scholar
  3. 3.
    Azlan Hussain, M.: Review of the applications of neural networks in chemical process control—simulation and online implementation. Artif. Intell. Eng. 13(1), 55–68 (1999). CrossRefGoogle Scholar
  4. 4.
    Agatonovic-Kustrin, S., Beresford, R.: Basic concepts of artificial neural network modeling and its application in pharmaceutical research. J. Pharm. Biomed. Anal. 22(5), 717–727 (2000). CrossRefGoogle Scholar
  5. 5.
    Witek-Krowiak, A., Chojnacka, K., Podstawczyk, D., Dawiec, A., Pokomeda, K.: Application of response surface methodology and artificial neural network methods in modelling and optimization of biosorption process. Bioresour. Technol. 160, 150–160 (2014). CrossRefGoogle Scholar
  6. 6.
    Meireles, M., Almeida, P., Simoes, M.G.: A comprehensive review for industrial applicability of artificial neural networks. IEEE Trans. Ind. Electron. 50(3), 585–601 (2003). CrossRefGoogle Scholar
  7. 7.
    Del Rio-Chanona, E.A., Fiorelli, F., Zhang, D., Ahmed, N.R., Jing, K., Shah, N.: An efficient model construction strategy to simulate microalgal lutein photo-production dynamic process. Biotechnol. Bioeng. 114(11), 2518–2527 (2017). CrossRefGoogle Scholar
  8. 8.
    Cheema, J.J.S., Sankpal, N.V., Tambe, S.S., Kulkarni, B.D.: Genetic programming assisted stochastic optimization strategies for optimization of glucose to gluconic acid fermentation. Biotechnol. Progr. 18(6), 1356–1365 (2002). CrossRefGoogle Scholar
  9. 9.
    Desai, K.M., Survase, S.A., Saudagar, P.S., Lele, S.S., Singhal, R.S.: Comparison of artificial neural network and response surface methodology in fermentation media optimization: case study of fermentative production of scleroglucan. Biochem. Eng. J. 41(3), 266–273 (2008). CrossRefGoogle Scholar
  10. 10.
    Nagata, Y., Chu, K.H.: Optimization of a fermentation medium using neural networks and genetic algorithms. Biotechnol. Lett. 25(21), 1837–1842 (2003). CrossRefGoogle Scholar
  11. 11.
    Fahmi, I., Cremaschi, S.: Process synthesis of biodiesel production plant using artificial neural networks as the surrogate models. Comput. Chem. Eng. 46, 105–123 (2012). CrossRefGoogle Scholar
  12. 12.
    Nascimento, C.A.O., Giudici, R., Guardani, R.: Neural network based approach for optimization of industrial chemical processes. Comput. Chem. Eng. 24(9–10), 2303–2314 (2000). CrossRefGoogle Scholar
  13. 13.
    Nascimento, C.A.O., Giudici, R.: Neural network based approach for optimisation applied to an industrial nylon-6,6 polymerisation process. Comput. Chem. Eng. 22, 595–S600 (1998). CrossRefGoogle Scholar
  14. 14.
    Chambers, M., Mount-Campbell, C.A.: Process optimization via neural network metamodeling. Int. J. Prod. Econ. 79(2), 93–100 (2002). CrossRefGoogle Scholar
  15. 15.
    Henao, C.A., Maravelias, C.T.: Surrogate-based process synthesis. In: Pierucci, S., Ferraris, G.B. (eds.) 20th European Symposium on Computer Aided Process Engineering, Computer Aided Chemical Engineering, vol. 28, pp. 1129–1134. Elsevier, Milan, Italy (2010).
  16. 16.
    Henao, C.A., Maravelias, C.T.: Surrogate-based superstructure optimization framework. AIChE J. 57(5), 1216–1232 (2011). CrossRefGoogle Scholar
  17. 17.
    Sant Anna, H.R., Barreto, A.G., Tavares, F.W., de Souza, M.B.: Machine learning model and optimization of a PSA unit for methane–nitrogen separation. Comput. Chem. Eng. 104, 377–391 (2017). CrossRefGoogle Scholar
  18. 18.
    Smith, J.D., Neto, A.A., Cremaschi, S., Crunkleton, D.W.: CFD-based optimization of a flooded bed algae bioreactor. Ind. Eng. Chem. Res. 52(22), 7181–7188 (2013). CrossRefGoogle Scholar
  19. 19.
    Henao, C.A.: A superstructure modeling framework for process synthesis using surrogate models. Dissertation, University of Wisconsin, Madison (2012)Google Scholar
  20. 20.
    Kajero, O.T., Chen, T., Yao, Y., Chuang, Y.C., Wong, D.S.H.: Meta-modelling in chemical process system engineering. J. Taiwan Inst. Chem. Eng. 73, 135–145 (2017). CrossRefGoogle Scholar
  21. 21.
    Lewandowski, J., Lemcoff, N.O., Palosaari, S.: Use of neural networks in the simulation and optimization of pressure swing adsorption processes. Chem. Eng. Technol. 21(7), 593–597 (1998).\(<\)593::AID-CEAT593\(>\)3.0.CO;2-UGoogle Scholar
  22. 22.
    Gutiérrez-Antonio, C.: Multiobjective stochastic optimization of dividing-wall distillation columns using a surrogate model based on neural networks. Chem. Biochem. Eng. Q. 29(4), 491–504 (2016).
  23. 23.
    Chen, C.R., Ramaswamy, H.S.: Modeling and optimization of variable retort temperature thermal processing using coupled neural networks and genetic algorithms. J. Food Eng. 53(3), 209–220 (2002). CrossRefGoogle Scholar
  24. 24.
    Dornier, M., Decloux, M., Trystram, G., Lebert, A.: Interest of neural networks for the optimization of the crossflow filtration process. LWT-Food Sci. Technol. 28(3), 300–309 (1995). CrossRefGoogle Scholar
  25. 25.
    Fernandes, F.A.N.: Optimization of Fischer–Tropsch synthesis using neural networks. Chem. Eng. Technol. 29(4), 449–453 (2006). CrossRefGoogle Scholar
  26. 26.
    Grossmann, I.E., Viswanathan, J., Vecchietti, A., Raman, R., Kalvelagen, E.: GAMS/DICOPT: A discrete continuous optimization package. GAMS Corporation Inc, Cary (2002)Google Scholar
  27. 27.
    Drud, A.S.: Conopt—a large-scale GRG code. ORSA J. Comput. 6(2), 207–216 (1994). zbMATHCrossRefGoogle Scholar
  28. 28.
    Nandi, S., Ghosh, S., Tambe, S.S., Kulkarni, B.D.: Artificial neural-network-assisted stochastic process optimization strategies. AIChE J. 47(1), 126–141 (2001). CrossRefGoogle Scholar
  29. 29.
    Tawarmalani, M., Sahinidis, N.V.: A polyhedral branch-and-cut approach to global optimization. Math. Program. 103(2), 225–249 (2005). MathSciNetzbMATHCrossRefGoogle Scholar
  30. 30.
    de Weerdt, E., Chu, Q.P., Mulder, J.A.: Neural network output optimization using interval analysis. IEEE Trans. Neural Netw. 20(4), 638–653 (2009). CrossRefGoogle Scholar
  31. 31.
    Moore, R.E., Bierbaum, F.: Methods and Applications of Interval Analysis, 2 edn. SIAM Studies in Applied Mathematics. Society for Industrial and Applied Mathematics, Philadelphia (1979).
  32. 32.
    Misener, R., Floudas, C.A.: ANTIGONE: Algorithms for continuous/integer global optimization of nonlinear equations. J. Glob. Optim. 59(2), 503–526 (2014). MathSciNetzbMATHCrossRefGoogle Scholar
  33. 33.
    Maher, S.J., Fischer, T., Gally, T., Gamrath, G., Gleixner, A., Gottwald, R.L., Hendel, G., Koch, T., Lübbecke, M.E., Miltenberger, M., Müller, B., Pfetsch, M.E., Puchert, C., Rehfeldt, D., Schenker, S., Schwarz, R., Serrano, F., Shinano, Y., Weninger, D., Witt, J.T., Witzig, J.: The SCIP optimization suite (version 4.0)Google Scholar
  34. 34.
    Epperly, T.G.W., Pistikopoulos, E.N.: A reduced space branch and bound algorithm for global optimization. J. Glob. Optim. 11(3), 287–311 (1997). MathSciNetzbMATHCrossRefGoogle Scholar
  35. 35.
    Mitsos, A., Chachuat, B., Barton, P.I.: McCormick-based relaxations of algorithms. SIAM J. Optim. 20(2), 573–601 (2009). MathSciNetzbMATHCrossRefGoogle Scholar
  36. 36.
    Scott, J.K., Stuber, M.D., Barton, P.I.: Generalized McCormick relaxations. J. Glob. Optim. 51(4), 569–606 (2011). MathSciNetzbMATHCrossRefGoogle Scholar
  37. 37.
    Bongartz, D., Mitsos, A.: Deterministic global optimization of process flowsheets in a reduced space using McCormick relaxations. J. Glob. Optim. 20(9), 419 (2017). MathSciNetzbMATHCrossRefGoogle Scholar
  38. 38.
    Huster, W.R., Bongartz, D., Mitsos, A.: Deterministic global optimization of the design of a geothermal organic rankine cycle. Energy Proc. 129, 50–57 (2017). CrossRefGoogle Scholar
  39. 39.
    McCormick, G.P.: Computability of global solutions to factorable nonconvex programs: part I—convex underestimating problems. Math. Program. 10(1), 147–175 (1976). zbMATHCrossRefGoogle Scholar
  40. 40.
    Bompadre, A., Mitsos, A.: Convergence rate of McCormick relaxations. J. Glob. Optim. 52(1), 1–28 (2012). MathSciNetzbMATHCrossRefGoogle Scholar
  41. 41.
    Najman, J., Mitsos, A.: Convergence analysis of multivariate McCormick relaxations. J. Glob. Optim. 66(4), 597–628 (2016). MathSciNetzbMATHCrossRefGoogle Scholar
  42. 42.
    Tsoukalas, A., Mitsos, A.: Multivariate McCormick relaxations. J. Glob. Optim. 59(2–3), 633–662 (2014). MathSciNetzbMATHCrossRefGoogle Scholar
  43. 43.
    Najman, J., Bongartz, D., Tsoukalas, A., Mitsos, A.: Erratum to multivariate McCormick relaxations. J. Glob. Optim. 68(1), 219–225 (2017). zbMATHCrossRefGoogle Scholar
  44. 44.
    Khan, K.A., Watson, H.A.J., Barton, P.I.: Differentiable McCormick relaxations. J. Glob. Optim. 67(4), 687–729 (2017). MathSciNetzbMATHCrossRefGoogle Scholar
  45. 45.
    Khan, K.A., Wilhelm, M., Stuber, M.D., Cao, H., Watson, H.A.J., Barton, P.I.: Corrections to differentiable McCormick relaxations. J. Glob. Optim. 70(3), 705–706 (2018). zbMATHCrossRefGoogle Scholar
  46. 46.
    Bongartz, D., Mitsos, A.: Infeasible path global flowsheet optimization using McCormick relaxations. In: Espuna, A. (ed.) 27th European Symposium on Computer Aided Process Engineering, Computer Aided Chemical Engineering, vol. 40. Elsevier, San Diego (2017).
  47. 47.
    Wechsung, A., Scott, J.K., Watson, H.A.J., Barton, P.I.: Reverse propagation of McCormick relaxations. J. Glob. Optim. 63(1), 1–36 (2015). MathSciNetzbMATHCrossRefGoogle Scholar
  48. 48.
    Stuber, M.D., Scott, J.K., Barton, P.I.: Convex and concave relaxations of implicit functions. Optim. Methods Softw. 30(3), 424–460 (2015). MathSciNetzbMATHCrossRefGoogle Scholar
  49. 49.
    Bishop, C.M.: Pattern Recognition and Machine Learning. Information Science and Statistics, 8th edn. Springer, New York (2009)Google Scholar
  50. 50.
    Bertsekas, D.P., Nedic, A., Ozdaglar, A.E.: Convex Analysis and Optimization, Athena Scientific Optimization and Computation Series, vol. 1. Athena Scientific, Belmont (2003)zbMATHGoogle Scholar
  51. 51.
    Bongartz, D., Najman, J., Sass, S., Mitsos, A.: MAiNGO: McCormick based Algorithm for mixed integer Nonlinear Global Optimization. Technical report (2018)Google Scholar
  52. 52.
    Chachuat, B.: MC++ (version 2.0): A toolkit for bounding factorable functions (2014)Google Scholar
  53. 53.
    Chachuat, B., Houska, B., Paulen, R., Peri’c, N., Rajyaguru, J., Villanueva, M.E.: Set-theoretic approaches in analysis, estimation and control of nonlinear systems. IFAC-PapersOnLine 48(8), 981–995 (2015). CrossRefGoogle Scholar
  54. 54.
    Hofschuster, W., Krämer, W.: FILIB++ Interval Library (version 3.0.2) (1998)Google Scholar
  55. 55.
    International Business Machies: IBM ilog CPLEX (version 12.1) (2009)Google Scholar
  56. 56.
    Gleixner, A.M., Berthold, T., Müller, B., Weltge, S.: Three enhancements for optimization-based bound tightening. J. Glob. Optim. 67(4), 731–757 (2017). MathSciNetzbMATHCrossRefGoogle Scholar
  57. 57.
    Ryoo, H.S., Sahinidis, N.V.: Global optimization of nonconvex NLPs and MINLPs with applications in process design. Comput. Chem. Eng. 19(5), 551–566 (1995). CrossRefGoogle Scholar
  58. 58.
    Locatelli, M., Schoen, F. (eds.): Global optimization: theory, algorithms, and applications. MOS-SIAM series on optimization. Mathematical Programming Society, Philadelphia, PA (2013).
  59. 59.
    Kraft, D.: A software package for sequential quadratic programming. Deutsche Forschungs- und Versuchsanstalt für Luft- und Raumfahrt Köln: Forschungsbericht. Wiss. Berichtswesen d. DFVLR, Köln (1988)Google Scholar
  60. 60.
    Johnson, S.G.: The NLopt nonlinear-optimization package (version 2.4.2) (2016)Google Scholar
  61. 61.
    Bendtsen, C., Stauning, O.: Fadbad++ (version 2.1): a flexible C++ package for automatic differentiation (2012)Google Scholar
  62. 62.
    Najman, J., Mitsos, A.: Tighter McCormick relaxations through subgradient propagation. Optimization online. (2017)
  63. 63.
    Ghorbanian, K., Gholamrezaei, M.: An artificial neural network approach to compressor performance prediction. Appl. Energy 86(7–8), 1210–1221 (2009). CrossRefGoogle Scholar
  64. 64.
    Luyben, W.L.: Design and control of the cumene process. Ind. Eng. Chem. Res. 49(2), 719–734 (2010). CrossRefGoogle Scholar
  65. 65.
    Schultz, E.S., Trierweiler, J.O., Farenzena, M.: The importance of nominal operating point selection in self-optimizing control. Ind. Eng. Chem. Res. 55(27), 7381–7393 (2016). CrossRefGoogle Scholar
  66. 66.
    Lee, U., Burre, J., Caspari, A., Kleinekorte, J., Schweidtmann, A.M., Mitsos, A.: Techno-economic optimization of a green-field post-combustion CO\(_2\) capture process using superstructure and rate-based models. Ind. Eng. Chem. Res. 55(46), 12014–12026 (2016). CrossRefGoogle Scholar
  67. 67.
    Helmdach, D., Yaseneva, P., Heer, P.K., Schweidtmann, A.M., Lapkin, A.A.: A multiobjective optimization including results of life cycle assessment in developing biorenewables-based processes. ChemSusChem 10(18), 3632–3643 (2017). CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Aachener Verfahrenstechnik, Process Systems EngineeringRWTH Aachen UniversityAachenGermany

Personalised recommendations