Advances in Computational Mathematics

, Volume 44, Issue 6, pp 1717–1750 | Cite as

Neural network closures for nonlinear model order reduction

  • Omer SanEmail author
  • Romit Maulik


Many reduced-order models are neither robust with respect to parameter changes nor cost-effective enough for handling the nonlinear dependence of complex dynamical systems. In this study, we put forth a robust machine learning framework for projection-based reduced-order modeling of such nonlinear and nonstationary systems. As a demonstration, we focus on a nonlinear advection-diffusion system given by the viscous Burgers equation, which is a prototypical setting of more realistic fluid dynamics applications due to its quadratic nonlinearity. In our proposed methodology the effects of truncated modes are modeled using a single layer feed-forward neural network architecture. The neural network architecture is trained by utilizing both the Bayesian regularization and extreme learning machine approaches, where the latter one is found to be more computationally efficient. A significant emphasis is laid on the selection of basis functions through the use of both Fourier bases and proper orthogonal decomposition. It is shown that the proposed model yields significant improvements in accuracy over the standard Galerkin projection methodology with a negligibly small computational overhead and provide reliable predictions with respect to parameter changes.


Machine learning Neural networks Extreme learning machine Model order reduction Projection methods Burgers equation 

Mathematics Subject Classification (2010)

37N10 76M25 76F20 76D99 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.



The computing for this project was performed by using resources from the High Performance Computing Center (HPCC) at Oklahoma State University.


  1. 1.
    Akhtar, I., Borggaard, J., Burns, J.A., Imtiaz, H., Zietsman, L.: Using functional gains for effective sensor location in flow control: a reduced-order modelling approach. J. Fluid Mech. 781, 622–656 (2015)MathSciNetzbMATHGoogle Scholar
  2. 2.
    Akhtar, I., Wang, Z., Borggaard, J., Iliescu, T.: A new closure strategy for proper orthogonal decomposition reduced-order models. J. Comput. Nonlinear Dyn. 7(3), 034,503 (2012)Google Scholar
  3. 3.
    Amsallem, D., Farhat, C.: Interpolation method for adapting reduced-order models and application to aeroelasticity. AIAA J. 46(7), 1803–1813 (2008)Google Scholar
  4. 4.
    Amsallem, D., Farhat, C.: Stabilization of projection-based reduced-order models. Int. J. Numer. Methods Eng. 91(4), 358–377 (2012)MathSciNetzbMATHGoogle Scholar
  5. 5.
    Antoulas, A.C.: Approximation of Large-Scale Dynamical Systems. SIAM, Philadelphia (2005)zbMATHGoogle Scholar
  6. 6.
    Aubry, N., Holmes, P., Lumley, J.L., Stone, E.: The dynamics of coherent structures in the wall region of a turbulent boundary layer. J. Fluid Mech. 192(1), 115–173 (1988)MathSciNetzbMATHGoogle Scholar
  7. 7.
    Balajewicz, M., Dowell, E.H.: Stabilization of projection-based reduced order models of the Navier–Stokes. Nonlinear Dyn. 70(2), 1619–1632 (2012)MathSciNetGoogle Scholar
  8. 8.
    Banyay, G.A., Ahmadpoor, M., Brigham, J.C.: Proper orthogonal decomposition based reduced order modeling of the very high temperature reactor lower plenum hydrodynamics. In: ASME 2014 4th Joint US-European Fluids Engineering Division Summer Meeting Collocated with the ASME 2014 12th International Conference on Nanochannels, Microchannels, and Minichannels, pp. V01DT27A013–V01DT27A013. American Society of Mechanical Engineers (2014)Google Scholar
  9. 9.
    Benner, P., Gugercin, S., Willcox, K.: A survey of projection-based model reduction methods for parametric dynamical systems. SIAM Rev. 57, 483–531 (2015)MathSciNetzbMATHGoogle Scholar
  10. 10.
    Benosman, M., Borggaard, J., San, O., Kramer, B.: Learning-based robust stabilization for reduced-order models of 2D and 3D Boussinesq equations. Appl. Math. Model. 49, 162–181 (2017)MathSciNetGoogle Scholar
  11. 11.
    Benosman, M., Kramer, B., Boufounos, P.T., Grover, P.: Learning-based reduced order model stabilization for partial differential equations: application to the coupled Burgers’ equation. In: American Control Conference (ACC), 2016, pp. 1673–1678. IEEE (2016)Google Scholar
  12. 12.
    Bergmann, M., Bruneau, C.H., Iollo, A.: Improvement of reduced order modeling based on POD. Computational Fluid Dynamics 2008, 779–784 (2009)Google Scholar
  13. 13.
    Borggaard, J., Hay, A., Pelletier, D.: Interval-based reduced order models for unsteady fluid flow. Int. J. Numer. Anal. Model. 4(3-4), 353–367 (2007)MathSciNetzbMATHGoogle Scholar
  14. 14.
    Borggaard, J., Iliescu, T., Wang, Z.: Artificial viscosity proper orthogonal decomposition. Math. Comput. Model. 53(1), 269–279 (2011)MathSciNetzbMATHGoogle Scholar
  15. 15.
    Borggaard, J., Wang, Z., Zietsman, L.: A goal-oriented reduced-order modeling approach for nonlinear systems. Comput. Math. Appl. 71(11), 2155–2169 (2016)MathSciNetGoogle Scholar
  16. 16.
    Buffoni, M., Camarri, S., Iollo, A., Salvetti, M.V.: Low-dimensional modelling of a confined three-dimensional wake flow. J. Fluid Mech. 569, 141–150 (2006)zbMATHGoogle Scholar
  17. 17.
    Bui-Thanh, T., Willcox, K., Ghattas, O., van Bloemen Waanders, B.: Goal-oriented, model-constrained optimization for reduction of large-scale systems. J. Comput. Phys. 224(2), 880–896 (2007)MathSciNetzbMATHGoogle Scholar
  18. 18.
    Carlberg, K., Farhat, C.: A low-cost, goal-oriented ‘compact proper orthogonal decomposition’ basis for model reduction of static systems. Int. J. Numer. Methods Eng. 86(3), 381–402 (2011)MathSciNetzbMATHGoogle Scholar
  19. 19.
    Cazemier, W.: Proper Orthogonal Decomposition and Low Dimensional Models for Turbulent Flows. Groningen (1997)Google Scholar
  20. 20.
    Cazemier, W., Verstappen, R., Veldman, A.: Proper orthogonal decomposition and low-dimensional models for driven cavity flows. Phys. Fluids (1994-present) 10 (7), 1685–1699 (1998)Google Scholar
  21. 21.
    Cordier, L., Majd, E., Abou, B., Favier, J.: Calibration of POD reduced-order models using Tikhonov regularization. Int. J. Numer. Methods Fluids 63(2), 269–296 (2010)MathSciNetzbMATHGoogle Scholar
  22. 22.
    Dawson, C.W., Wilby, R.: An artificial neural network approach to rainfall-runoff modelling. Hydrol. Sci. J. 43(1), 47–66 (1998)Google Scholar
  23. 23.
    Demuth, H.B., Beale, M.H., De Jess, O., Hagan, M.T.: Neural Network Design. Martin Hagan (2014)Google Scholar
  24. 24.
    Dyke, S., Spencer, B. Jr, Sain, M., Carlson, J.: Modeling and control of magnetorheological dampers for seismic response reduction. Smart Mater. Struct. 5 (5), 565 (1996)Google Scholar
  25. 25.
    Efe, M., Debiasi, M., Yan, P., Ozbay, H., Samimy, M.: Control of subsonic cavity flows by neural networks-analytical models and experimental validation. In: 43rd AIAA Aerospace Sciences Meeting and Exhibit, p. 294 (2005)Google Scholar
  26. 26.
    Efe, M.O., Debiasi, M., Ozbay, H., Samimy, M.: Modeling of subsonic cavity flows by neural networks. In: Proceedings of the IEEE International Conference on Mechatronics, 2004. ICM’04, pp. 560–565. IEEE (2004)Google Scholar
  27. 27.
    El Majd, B.A., Cordier, L.: New regularization method for calibrated POD reduced-order models. Math. Model. Anal. 21(1), 47–62 (2016)MathSciNetGoogle Scholar
  28. 28.
    Faller, W.E., Schreck, S.J.: Unsteady fluid mechanics applications of neural networks. J. Aircr. 34(1), 48–55 (1997)Google Scholar
  29. 29.
    Fang, F., Pain, C., Navon, I., Gorman, G., Piggott, M., Allison, P., Farrell, P., Goddard, A.: A POD reduced order unstructured mesh ocean modelling method for moderate Reynolds number flows. Ocean Model. 28(1), 127–136 (2009)Google Scholar
  30. 30.
    Foresee, F.D., Hagan, M.T.: Gauss-Newton approximation to Bayesian learning. In: International Conference on Neural Networks, 1997, vol. 3, pp. 1930–1935. IEEE (1997)Google Scholar
  31. 31.
    Fortuna, L., Nunnari, G., Gallo, A.: Model Order Reduction Techniques with Applications in Electrical Engineering. Springer Science & Business Media, Berlin (2012)Google Scholar
  32. 32.
    Freund, R.W.: Reduced-order modeling techniques based on Krylov subspaces and their use in circuit simulation. In: Applied and Computational Ccontrol, Signals, and Circuits, pp. 435–498. Springer (1999)Google Scholar
  33. 33.
    Gaspard, P.: Chaos, Scattering and Statistical Mechanics, vol. 9. Cambridge University Press, Cambridge (2005)zbMATHGoogle Scholar
  34. 34.
    Gillies, E.: Low-dimensional characterization and control of non-linear wake flows. Ph.D. thesis, PhD. Dissertation, University of Glasgow, Scotland (1995)Google Scholar
  35. 35.
    Gillies, E.: Low-dimensional control of the circular cylinder wake. J. Fluid Mech. 371, 157–178 (1998)MathSciNetzbMATHGoogle Scholar
  36. 36.
    Gillies, E.: Multiple sensor control of vortex shedding. AIAA J. 39(4), 748–750 (2001)Google Scholar
  37. 37.
    Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. MIT press, Cambridge (2016)zbMATHGoogle Scholar
  38. 38.
    Hagan, M.T., Menhaj, M.B.: Training feedforward networks with the Marquardt algorithm. IEEE Trans. Neural Netw. 5(6), 989–993 (1994)Google Scholar
  39. 39.
    Haykin, S.S., Haykin, S.S., Haykin, S.S., Haykin, S.S.: Neural Networks and Learning Machines, vol. 3. Pearson, Upper Saddle River (2009)zbMATHGoogle Scholar
  40. 40.
    Hocevar, M., Sirok, B., Grabec, I.: Experimental turbulent field modeling by visualization and neural networks. J. Fluids Eng. 126, 316–322 (2004)Google Scholar
  41. 41.
    Holmes, P., Lumley, J.L., Berkooz, G.: Turbulence, Coherent Structures, Dynamical Systems and Symmetry. Cambridge University Press, Cambridge (1998)zbMATHGoogle Scholar
  42. 42.
    Hotelling, H.: Analysis of a complex of statistical variables into principal components. J. Educ. Psychol. 24(6), 417 (1933)zbMATHGoogle Scholar
  43. 43.
    Huang, G.B., Zhu, Q.Y., Siew, C.K.: Extreme learning machine: theory and applications. Neurocomputing 70(1), 489–501 (2006)Google Scholar
  44. 44.
    Imtiaz, H., Akhtar, I.: Closure modeling in reduced-order model of Burgers’ equation for control applications. J. Aerosp. Eng. 231(4), 642–656 (2017)Google Scholar
  45. 45.
    Iollo, A., Lanteri, S., Désidéri, J.A.: Stability properties of POD–Galerkin approximations for the compressible Navier–Stokes equations. Theor. Comput. Fluid Dyn. 13(6), 377–396 (2000)zbMATHGoogle Scholar
  46. 46.
    Kazantzis, N., Kravaris, C., Syrou, L.: A new model reduction method for nonlinear dynamical systems. Nonlinear Dyn. 59(1), 183–194 (2010)MathSciNetzbMATHGoogle Scholar
  47. 47.
    Khibnik, A., Narayanan, S., Jacobson, C., Lust, K.: Analysis of low dimensional dynamics of flow separation. In: Continuation Methods in Fluid Dynamics, vol. 74, pp. 167–178. Vieweg (2000)Google Scholar
  48. 48.
    Kim, T.W., Valdés, J.B.: Nonlinear model for drought forecasting based on a conjunction of wavelet transforms and neural networks. J. Hydrol. Eng. 8(6), 319–328 (2003)Google Scholar
  49. 49.
    Kunisch, K., Volkwein, S.: Control of the Burgers equation by a reduced-order approach using proper orthogonal decomposition. J. Optim. Theory Appl. 102(2), 345–371 (1999)MathSciNetzbMATHGoogle Scholar
  50. 50.
    Kunisch, K., Volkwein, S.: Galerkin proper orthogonal decomposition methods for parabolic problems. Numer. Math. 90(1), 117–148 (2001)MathSciNetzbMATHGoogle Scholar
  51. 51.
    Kunisch, K., Volkwein, S.: Optimal snapshot location for computing POD basis functions. ESAIM: Mathematical Modelling and Numerical Analysis 44(3), 509–529 (2010)MathSciNetzbMATHGoogle Scholar
  52. 52.
    Lassila, T., Manzoni, A., Quarteroni, A., Rozza, G.: Model order reduction in fluid dynamics: challenges and perspectives. In: Quarteroni, A., Rozza, G. (eds.) Reduced Order Methods for Modeling and Computational Reduction. Springer, Milano (2013)Google Scholar
  53. 53.
    Lee, C., Kim, J., Babcock, D., Goodman, R.: Application of neural networks to turbulence control for drag reduction. Phys. Fluids 9(6), 1740–1747 (1997)Google Scholar
  54. 54.
    Levenberg, K.: A method for the solution of certain non-linear problems in least squares. Q. Appl. Math. 2(2), 164–168 (1944)MathSciNetzbMATHGoogle Scholar
  55. 55.
    Ling, J., Kurzawski, A., Templeton, J.: Reynolds averaged turbulence modelling using deep neural networks with embedded invariance. J. Fluid Mech. 807, 155–166 (2016)MathSciNetzbMATHGoogle Scholar
  56. 56.
    Loève, M.: Probability Theory; Foundations, Random Sequences. D. Van Nostrand Company, New York (1955)zbMATHGoogle Scholar
  57. 57.
    Lorenz, E.N.: Empirical orthogonal functions and statistical weather prediction. Massachusetts Institute of Technology, Department of Meteorology, Cambridge (1956)Google Scholar
  58. 58.
    Lucia, D.J., Beran, P.S., Silva, W.A.: Reduced-order modeling: new approaches for computational physics. Prog. Aerosp. Sci. 40(1), 51–117 (2004)Google Scholar
  59. 59.
    Lumley, J.: The structures of inhomogeneous turbulent flow. In: Yaglom, A., Tatarski, V. (eds.) Atmospheric Turbulence and Radio Wave Propagation, pp. 160–178 (1967)Google Scholar
  60. 60.
    MacKay, D.J.: Bayesian interpolation. Neural Comput. 4(3), 415–447 (1992)zbMATHGoogle Scholar
  61. 61.
    Maleewong, M., Sirisup, S.: On-line and off-line POD assisted projective integral for non-linear problems: a case study with Burgers’ equation. International Journal of Mathematical, Computational, Physical, Electrical and Computer Engineering 5(7), 984–992 (2011)Google Scholar
  62. 62.
    Marquardt, D.W.: An algorithm for least-squares estimation of nonlinear parameters. J. Soc. Ind. Appl. Math. 11(2), 431–441 (1963)MathSciNetzbMATHGoogle Scholar
  63. 63.
    Maulik, R., San, O.: Explicit and implicit LES closures for Burgers turbulence. J. Comput. Appl. Math. 327, 12–40 (2018)MathSciNetzbMATHGoogle Scholar
  64. 64.
    Moin, P.: Fundamentals of Engineering Numerical Analysis. Cambridge University Press, Cambridge (2010)zbMATHGoogle Scholar
  65. 65.
    Moosavi, A., Stefanescu, R., Sandu, A.: Efficient construction of local parametric reduced order models using machine learning techniques. arXiv:1511.02909 (2015)
  66. 66.
    Moosavi, A., Stefanescu, R., Sandu, A.: Multivariate predictions of local reduced-order-model errors and dimensions. arXiv:1701.03720 (2017)
  67. 67.
    Narayanan, S., Khibnik, A., Jacobson, C., Kevrekedis, Y., Rico-Martinez, R., Lust, K.: Low-dimensional models for active control of flow separation. In: Proceedings of the 1999 IEEE International Conference on Control Applications, 1999, vol. 2, pp. 1151–1156. IEEE (1999)Google Scholar
  68. 68.
    Nguyen, D., Widrow, B.: Improving the learning speed of 2-layer neural networks by choosing initial values of the adaptive weights. In: 1990 IJCNN International Joint Conference on Neural Networks, 1990, pp. 21–26. IEEE (1990)Google Scholar
  69. 69.
    Noack, B., Papas, P., Monkewitz, P.: Low-dimensional Galerkin model of a laminar shear-layer. Tech. rep., Tech. Rep. 2002-01. Laboratoire De Mecanique des Fluides, Departement de Genie Mecanique, Ecole Polytechnique Fédérale de Lausanne, Switzerland (2002)Google Scholar
  70. 70.
    Noack, B.R., Afanasiev, K., Morzyński, M., Tadmor, G., Thiele, F.: A hierarchy of low-dimensional models for the transient and post-transient cylinder wake. J. Fluid Mech. 497, 335–363 (2003)MathSciNetzbMATHGoogle Scholar
  71. 71.
    Noack, B.R., Morzynski, M., Tadmor, G.: Reduced-Order Modelling for Flow Control, vol. 528. Springer, Berlin (2011)zbMATHGoogle Scholar
  72. 72.
    Pinnau, R.: Model reduction via proper orthogonal decomposition. Model Order Reduction: Theory, Research Aspects and Applications 13, 95–109 (2008)MathSciNetzbMATHGoogle Scholar
  73. 73.
    Raissi, M., Perdikaris, P., Karniadakis, G.E.: Physics informed deep learning (Part I): data-driven discovery of nonlinear partial differential equations. arXiv:1711.10561 (2017)
  74. 74.
    Raissi, M., Perdikaris, P., Karniadakis, G.E.: Physics informed deep learning (Part II): data-driven discovery of nonlinear partial differential equations. arXiv:1711.10566 (2017)
  75. 75.
    Ravindran, S.S.: A reduced-order approach for optimal control of fluids using proper orthogonal decomposition. Int. J. Numer. Methods Fluids 34(5), 425–448 (2000)MathSciNetzbMATHGoogle Scholar
  76. 76.
    Rowley, C.W.: Model reduction for fluids, using balanced proper orthogonal decomposition. Int. J. Bifurc. Chaos 15(03), 997–1013 (2005)MathSciNetzbMATHGoogle Scholar
  77. 77.
    Rowley, C.W., Dawson, S.T.: Model reduction for flow analysis and control. Annu. Rev. Fluid Mech. 49, 387–417 (2017)MathSciNetzbMATHGoogle Scholar
  78. 78.
    Roychowdhury, J.: Reduced-order modeling of time-varying systems. IEEE Trans. Circuits Syst. II, Analog Digit. Signal Process. 46(10), 1273–1288 (1999)Google Scholar
  79. 79.
    Sahan, R., Koc-Sahan, N., Albin, D., Liakopoulos, A.: Artificial neural network-based modeling and intelligent control of transitional flows. In: Proceedings of the 1997 IEEE International Conference on Control Applications, 1997, pp. 359–364. IEEE (1997)Google Scholar
  80. 80.
    San, O., Iliescu, T.: Proper orthogonal decomposition closure models for fluid flows: Burgers equation. Int. J. Numer. Anal. Model. 5, 217–237 (2014)MathSciNetzbMATHGoogle Scholar
  81. 81.
    San, O., Iliescu, T.: A stabilized proper orthogonal decomposition reduced-order model for large scale quasigeostrophic ocean circulation. Adv. Comput. Math. 41(5), 1289–1319 (2015)MathSciNetzbMATHGoogle Scholar
  82. 82.
    Schmid, P.J.: Dynamic mode decomposition of numerical and experimental data. J. Fluid Mech. 656, 5–28 (2010)MathSciNetzbMATHGoogle Scholar
  83. 83.
    Serre, D.: Matrices: Theory and Applications. Springer, New York (2002)zbMATHGoogle Scholar
  84. 84.
    Silveira, L.M., Kamon, M., White, J.: Efficient reduced-order modeling of frequency-dependent coupling inductances associated with 3-D interconnect structures. IEEE Transactions on Components, Packaging, and Manufacturing Technology: Part B 19(2), 283–288 (1996)Google Scholar
  85. 85.
    Sirisup, S., Karniadakis, G.E.: A spectral viscosity method for correcting the long-term behavior of POD models. J. Comput. Phys. 194(1), 92–116 (2004)MathSciNetzbMATHGoogle Scholar
  86. 86.
    Sirovich, L.: Turbulence and the dynamics of coherent structures. I-Coherent structures. II-symmetries and transformations. III-dynamics and scaling. Q. Appl. Math. 45, 561–571 (1987)zbMATHGoogle Scholar
  87. 87.
    Taira, K., Brunton, S.L., Dawson, S., Rowley, C.W., Colonius, T., McKeon, B.J., Schmidt, O.T., Gordeyev, S., Theofilis, V., Ukeiley, L.S.: Modal analysis of fluid flows: an overview. AIAA J. 55(12), 4013–4041 (2017)Google Scholar
  88. 88.
    Ullmann, S., Lang, J.: A POD-Galerkin reduced model with updated coefficients for Smagorinsky LES. In: V European Conference on Computational Fluid Dynamics, ECCOMAS CFD, p. 2010 (2010)Google Scholar
  89. 89.
    Wang, Z.: Reduced-order modeling of complex engineering and geophysical flows: analysis and computations. Ph.D. thesis, Virginia Tech (2012)Google Scholar
  90. 90.
    Wang, Z., Akhtar, I., Borggaard, J., Iliescu, T.: Two-level discretizations of nonlinear closure models for proper orthogonal decomposition. J. Comput. Phys. 230(1), 126–146 (2011)MathSciNetzbMATHGoogle Scholar
  91. 91.
    Wang, Z., Akhtar, I., Borggaard, J., Iliescu, T.: Proper orthogonal decomposition closure models for turbulent flows: a numerical comparison. Comput. Methods Appl. Mech. Eng. 237, 10–26 (2012)MathSciNetzbMATHGoogle Scholar
  92. 92.
    Weller, J., Lombardi, E., Bergmann, M., Iollo, A.: Numerical methods for low-order modeling of fluid flows based on POD. Int. J. Numer. Methods Fluids 63(2), 249–268 (2010)MathSciNetzbMATHGoogle Scholar
  93. 93.
    Wells, D., Wang, Z., Xie, X., Iliescu, T.: An evolve-then-filter regularized reduced order model for convection-dominated flows. Int. J. Numer. Methods Fluids 84, 598–615 (2017)MathSciNetGoogle Scholar
  94. 94.
    Widrow, B., Rumelhart, D.E., Lehr, M.A.: Neural networks: applications in industry, business and science. Commun. ACM 37(3), 93–106 (1994)Google Scholar
  95. 95.
    Xie, X., Mohebujjaman, M., Rebholz, L., Iliescu, T.: Data-driven filtered reduced order modeling of fluid flows. arXiv:1709.04362 (2017)
  96. 96.
    Xie, X., Wells, D., Wang, Z., Iliescu, T.: Approximate deconvolution reduced order modeling. Comput. Methods Appl. Mech. Eng. 313, 512–534 (2017)MathSciNetGoogle Scholar
  97. 97.
    Zhang, G., Patuwo, B.E., Hu, M.Y.: Forecasting with artificial neural networks: the state of the art. Int. J. Forecast. 14(1), 35–62 (1998)Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Department of Mechanical and Aerospace EngineeringOklahoma State UniversityStillwaterUSA

Personalised recommendations