Skip to main content
SpringerLink
Log in

Neural network closures for nonlinear model order reduction

  • Published:
Advances in Computational Mathematics Aims and scope Submit manuscript

Abstract

Many reduced-order models are neither robust with respect to parameter changes nor cost-effective enough for handling the nonlinear dependence of complex dynamical systems. In this study, we put forth a robust machine learning framework for projection-based reduced-order modeling of such nonlinear and nonstationary systems. As a demonstration, we focus on a nonlinear advection-diffusion system given by the viscous Burgers equation, which is a prototypical setting of more realistic fluid dynamics applications due to its quadratic nonlinearity. In our proposed methodology the effects of truncated modes are modeled using a single layer feed-forward neural network architecture. The neural network architecture is trained by utilizing both the Bayesian regularization and extreme learning machine approaches, where the latter one is found to be more computationally efficient. A significant emphasis is laid on the selection of basis functions through the use of both Fourier bases and proper orthogonal decomposition. It is shown that the proposed model yields significant improvements in accuracy over the standard Galerkin projection methodology with a negligibly small computational overhead and provide reliable predictions with respect to parameter changes.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Similar content being viewed by others

References

  1. Akhtar, I., Borggaard, J., Burns, J.A., Imtiaz, H., Zietsman, L.: Using functional gains for effective sensor location in flow control: a reduced-order modelling approach. J. Fluid Mech. 781, 622–656 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  2. Akhtar, I., Wang, Z., Borggaard, J., Iliescu, T.: A new closure strategy for proper orthogonal decomposition reduced-order models. J. Comput. Nonlinear Dyn. 7(3), 034,503 (2012)

    Article  Google Scholar 

  3. Amsallem, D., Farhat, C.: Interpolation method for adapting reduced-order models and application to aeroelasticity. AIAA J. 46(7), 1803–1813 (2008)

    Article  Google Scholar 

  4. Amsallem, D., Farhat, C.: Stabilization of projection-based reduced-order models. Int. J. Numer. Methods Eng. 91(4), 358–377 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  5. Antoulas, A.C.: Approximation of Large-Scale Dynamical Systems. SIAM, Philadelphia (2005)

    Book  MATH  Google Scholar 

  6. Aubry, N., Holmes, P., Lumley, J.L., Stone, E.: The dynamics of coherent structures in the wall region of a turbulent boundary layer. J. Fluid Mech. 192(1), 115–173 (1988)

    Article  MathSciNet  MATH  Google Scholar 

  7. Balajewicz, M., Dowell, E.H.: Stabilization of projection-based reduced order models of the Navier–Stokes. Nonlinear Dyn. 70(2), 1619–1632 (2012)

    Article  MathSciNet  Google Scholar 

  8. Banyay, G.A., Ahmadpoor, M., Brigham, J.C.: Proper orthogonal decomposition based reduced order modeling of the very high temperature reactor lower plenum hydrodynamics. In: ASME 2014 4th Joint US-European Fluids Engineering Division Summer Meeting Collocated with the ASME 2014 12th International Conference on Nanochannels, Microchannels, and Minichannels, pp. V01DT27A013–V01DT27A013. American Society of Mechanical Engineers (2014)

  9. Benner, P., Gugercin, S., Willcox, K.: A survey of projection-based model reduction methods for parametric dynamical systems. SIAM Rev. 57, 483–531 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  10. Benosman, M., Borggaard, J., San, O., Kramer, B.: Learning-based robust stabilization for reduced-order models of 2D and 3D Boussinesq equations. Appl. Math. Model. 49, 162–181 (2017)

    Article  MathSciNet  Google Scholar 

  11. Benosman, M., Kramer, B., Boufounos, P.T., Grover, P.: Learning-based reduced order model stabilization for partial differential equations: application to the coupled Burgers’ equation. In: American Control Conference (ACC), 2016, pp. 1673–1678. IEEE (2016)

  12. Bergmann, M., Bruneau, C.H., Iollo, A.: Improvement of reduced order modeling based on POD. Computational Fluid Dynamics 2008, 779–784 (2009)

    Google Scholar 

  13. Borggaard, J., Hay, A., Pelletier, D.: Interval-based reduced order models for unsteady fluid flow. Int. J. Numer. Anal. Model. 4(3-4), 353–367 (2007)

    MathSciNet  MATH  Google Scholar 

  14. Borggaard, J., Iliescu, T., Wang, Z.: Artificial viscosity proper orthogonal decomposition. Math. Comput. Model. 53(1), 269–279 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  15. Borggaard, J., Wang, Z., Zietsman, L.: A goal-oriented reduced-order modeling approach for nonlinear systems. Comput. Math. Appl. 71(11), 2155–2169 (2016)

    Article  MathSciNet  Google Scholar 

  16. Buffoni, M., Camarri, S., Iollo, A., Salvetti, M.V.: Low-dimensional modelling of a confined three-dimensional wake flow. J. Fluid Mech. 569, 141–150 (2006)

    Article  MATH  Google Scholar 

  17. Bui-Thanh, T., Willcox, K., Ghattas, O., van Bloemen Waanders, B.: Goal-oriented, model-constrained optimization for reduction of large-scale systems. J. Comput. Phys. 224(2), 880–896 (2007)

    Article  MathSciNet  MATH  Google Scholar 

  18. Carlberg, K., Farhat, C.: A low-cost, goal-oriented ‘compact proper orthogonal decomposition’ basis for model reduction of static systems. Int. J. Numer. Methods Eng. 86(3), 381–402 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  19. Cazemier, W.: Proper Orthogonal Decomposition and Low Dimensional Models for Turbulent Flows. Groningen (1997)

  20. Cazemier, W., Verstappen, R., Veldman, A.: Proper orthogonal decomposition and low-dimensional models for driven cavity flows. Phys. Fluids (1994-present) 10 (7), 1685–1699 (1998)

    Article  Google Scholar 

  21. Cordier, L., Majd, E., Abou, B., Favier, J.: Calibration of POD reduced-order models using Tikhonov regularization. Int. J. Numer. Methods Fluids 63(2), 269–296 (2010)

    MathSciNet  MATH  Google Scholar 

  22. Dawson, C.W., Wilby, R.: An artificial neural network approach to rainfall-runoff modelling. Hydrol. Sci. J. 43(1), 47–66 (1998)

    Article  Google Scholar 

  23. Demuth, H.B., Beale, M.H., De Jess, O., Hagan, M.T.: Neural Network Design. Martin Hagan (2014)

  24. Dyke, S., Spencer, B. Jr, Sain, M., Carlson, J.: Modeling and control of magnetorheological dampers for seismic response reduction. Smart Mater. Struct. 5 (5), 565 (1996)

    Article  Google Scholar 

  25. Efe, M., Debiasi, M., Yan, P., Ozbay, H., Samimy, M.: Control of subsonic cavity flows by neural networks-analytical models and experimental validation. In: 43rd AIAA Aerospace Sciences Meeting and Exhibit, p. 294 (2005)

  26. Efe, M.O., Debiasi, M., Ozbay, H., Samimy, M.: Modeling of subsonic cavity flows by neural networks. In: Proceedings of the IEEE International Conference on Mechatronics, 2004. ICM’04, pp. 560–565. IEEE (2004)

  27. El Majd, B.A., Cordier, L.: New regularization method for calibrated POD reduced-order models. Math. Model. Anal. 21(1), 47–62 (2016)

    Article  MathSciNet  Google Scholar 

  28. Faller, W.E., Schreck, S.J.: Unsteady fluid mechanics applications of neural networks. J. Aircr. 34(1), 48–55 (1997)

    Article  Google Scholar 

  29. Fang, F., Pain, C., Navon, I., Gorman, G., Piggott, M., Allison, P., Farrell, P., Goddard, A.: A POD reduced order unstructured mesh ocean modelling method for moderate Reynolds number flows. Ocean Model. 28(1), 127–136 (2009)

    Article  Google Scholar 

  30. Foresee, F.D., Hagan, M.T.: Gauss-Newton approximation to Bayesian learning. In: International Conference on Neural Networks, 1997, vol. 3, pp. 1930–1935. IEEE (1997)

  31. Fortuna, L., Nunnari, G., Gallo, A.: Model Order Reduction Techniques with Applications in Electrical Engineering. Springer Science & Business Media, Berlin (2012)

    Google Scholar 

  32. Freund, R.W.: Reduced-order modeling techniques based on Krylov subspaces and their use in circuit simulation. In: Applied and Computational Ccontrol, Signals, and Circuits, pp. 435–498. Springer (1999)

  33. Gaspard, P.: Chaos, Scattering and Statistical Mechanics, vol. 9. Cambridge University Press, Cambridge (2005)

    MATH  Google Scholar 

  34. Gillies, E.: Low-dimensional characterization and control of non-linear wake flows. Ph.D. thesis, PhD. Dissertation, University of Glasgow, Scotland (1995)

  35. Gillies, E.: Low-dimensional control of the circular cylinder wake. J. Fluid Mech. 371, 157–178 (1998)

    Article  MathSciNet  MATH  Google Scholar 

  36. Gillies, E.: Multiple sensor control of vortex shedding. AIAA J. 39(4), 748–750 (2001)

    Article  Google Scholar 

  37. Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. MIT press, Cambridge (2016)

    MATH  Google Scholar 

  38. Hagan, M.T., Menhaj, M.B.: Training feedforward networks with the Marquardt algorithm. IEEE Trans. Neural Netw. 5(6), 989–993 (1994)

    Article  Google Scholar 

  39. Haykin, S.S., Haykin, S.S., Haykin, S.S., Haykin, S.S.: Neural Networks and Learning Machines, vol. 3. Pearson, Upper Saddle River (2009)

    MATH  Google Scholar 

  40. Hocevar, M., Sirok, B., Grabec, I.: Experimental turbulent field modeling by visualization and neural networks. J. Fluids Eng. 126, 316–322 (2004)

    Article  Google Scholar 

  41. Holmes, P., Lumley, J.L., Berkooz, G.: Turbulence, Coherent Structures, Dynamical Systems and Symmetry. Cambridge University Press, Cambridge (1998)

    MATH  Google Scholar 

  42. Hotelling, H.: Analysis of a complex of statistical variables into principal components. J. Educ. Psychol. 24(6), 417 (1933)

    Article  MATH  Google Scholar 

  43. Huang, G.B., Zhu, Q.Y., Siew, C.K.: Extreme learning machine: theory and applications. Neurocomputing 70(1), 489–501 (2006)

    Article  Google Scholar 

  44. Imtiaz, H., Akhtar, I.: Closure modeling in reduced-order model of Burgers’ equation for control applications. J. Aerosp. Eng. 231(4), 642–656 (2017)

    Google Scholar 

  45. Iollo, A., Lanteri, S., Désidéri, J.A.: Stability properties of POD–Galerkin approximations for the compressible Navier–Stokes equations. Theor. Comput. Fluid Dyn. 13(6), 377–396 (2000)

    Article  MATH  Google Scholar 

  46. Kazantzis, N., Kravaris, C., Syrou, L.: A new model reduction method for nonlinear dynamical systems. Nonlinear Dyn. 59(1), 183–194 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  47. Khibnik, A., Narayanan, S., Jacobson, C., Lust, K.: Analysis of low dimensional dynamics of flow separation. In: Continuation Methods in Fluid Dynamics, vol. 74, pp. 167–178. Vieweg (2000)

  48. Kim, T.W., Valdés, J.B.: Nonlinear model for drought forecasting based on a conjunction of wavelet transforms and neural networks. J. Hydrol. Eng. 8(6), 319–328 (2003)

    Article  Google Scholar 

  49. Kunisch, K., Volkwein, S.: Control of the Burgers equation by a reduced-order approach using proper orthogonal decomposition. J. Optim. Theory Appl. 102(2), 345–371 (1999)

    Article  MathSciNet  MATH  Google Scholar 

  50. Kunisch, K., Volkwein, S.: Galerkin proper orthogonal decomposition methods for parabolic problems. Numer. Math. 90(1), 117–148 (2001)

    Article  MathSciNet  MATH  Google Scholar 

  51. Kunisch, K., Volkwein, S.: Optimal snapshot location for computing POD basis functions. ESAIM: Mathematical Modelling and Numerical Analysis 44(3), 509–529 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  52. Lassila, T., Manzoni, A., Quarteroni, A., Rozza, G.: Model order reduction in fluid dynamics: challenges and perspectives. In: Quarteroni, A., Rozza, G. (eds.) Reduced Order Methods for Modeling and Computational Reduction. Springer, Milano (2013)

  53. Lee, C., Kim, J., Babcock, D., Goodman, R.: Application of neural networks to turbulence control for drag reduction. Phys. Fluids 9(6), 1740–1747 (1997)

    Article  Google Scholar 

  54. Levenberg, K.: A method for the solution of certain non-linear problems in least squares. Q. Appl. Math. 2(2), 164–168 (1944)

    Article  MathSciNet  MATH  Google Scholar 

  55. Ling, J., Kurzawski, A., Templeton, J.: Reynolds averaged turbulence modelling using deep neural networks with embedded invariance. J. Fluid Mech. 807, 155–166 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  56. Loève, M.: Probability Theory; Foundations, Random Sequences. D. Van Nostrand Company, New York (1955)

    MATH  Google Scholar 

  57. Lorenz, E.N.: Empirical orthogonal functions and statistical weather prediction. Massachusetts Institute of Technology, Department of Meteorology, Cambridge (1956)

    Google Scholar 

  58. Lucia, D.J., Beran, P.S., Silva, W.A.: Reduced-order modeling: new approaches for computational physics. Prog. Aerosp. Sci. 40(1), 51–117 (2004)

    Article  Google Scholar 

  59. Lumley, J.: The structures of inhomogeneous turbulent flow. In: Yaglom, A., Tatarski, V. (eds.) Atmospheric Turbulence and Radio Wave Propagation, pp. 160–178 (1967)

  60. MacKay, D.J.: Bayesian interpolation. Neural Comput. 4(3), 415–447 (1992)

    Article  MATH  Google Scholar 

  61. Maleewong, M., Sirisup, S.: On-line and off-line POD assisted projective integral for non-linear problems: a case study with Burgers’ equation. International Journal of Mathematical, Computational, Physical, Electrical and Computer Engineering 5(7), 984–992 (2011)

    Google Scholar 

  62. Marquardt, D.W.: An algorithm for least-squares estimation of nonlinear parameters. J. Soc. Ind. Appl. Math. 11(2), 431–441 (1963)

    Article  MathSciNet  MATH  Google Scholar 

  63. Maulik, R., San, O.: Explicit and implicit LES closures for Burgers turbulence. J. Comput. Appl. Math. 327, 12–40 (2018)

    Article  MathSciNet  MATH  Google Scholar 

  64. Moin, P.: Fundamentals of Engineering Numerical Analysis. Cambridge University Press, Cambridge (2010)

    Book  MATH  Google Scholar 

  65. Moosavi, A., Stefanescu, R., Sandu, A.: Efficient construction of local parametric reduced order models using machine learning techniques. arXiv:1511.02909 (2015)

  66. Moosavi, A., Stefanescu, R., Sandu, A.: Multivariate predictions of local reduced-order-model errors and dimensions. arXiv:1701.03720 (2017)

  67. Narayanan, S., Khibnik, A., Jacobson, C., Kevrekedis, Y., Rico-Martinez, R., Lust, K.: Low-dimensional models for active control of flow separation. In: Proceedings of the 1999 IEEE International Conference on Control Applications, 1999, vol. 2, pp. 1151–1156. IEEE (1999)

  68. Nguyen, D., Widrow, B.: Improving the learning speed of 2-layer neural networks by choosing initial values of the adaptive weights. In: 1990 IJCNN International Joint Conference on Neural Networks, 1990, pp. 21–26. IEEE (1990)

  69. Noack, B., Papas, P., Monkewitz, P.: Low-dimensional Galerkin model of a laminar shear-layer. Tech. rep., Tech. Rep. 2002-01. Laboratoire De Mecanique des Fluides, Departement de Genie Mecanique, Ecole Polytechnique Fédérale de Lausanne, Switzerland (2002)

  70. Noack, B.R., Afanasiev, K., Morzyński, M., Tadmor, G., Thiele, F.: A hierarchy of low-dimensional models for the transient and post-transient cylinder wake. J. Fluid Mech. 497, 335–363 (2003)

    Article  MathSciNet  MATH  Google Scholar 

  71. Noack, B.R., Morzynski, M., Tadmor, G.: Reduced-Order Modelling for Flow Control, vol. 528. Springer, Berlin (2011)

    Book  MATH  Google Scholar 

  72. Pinnau, R.: Model reduction via proper orthogonal decomposition. Model Order Reduction: Theory, Research Aspects and Applications 13, 95–109 (2008)

    MathSciNet  MATH  Google Scholar 

  73. Raissi, M., Perdikaris, P., Karniadakis, G.E.: Physics informed deep learning (Part I): data-driven discovery of nonlinear partial differential equations. arXiv:1711.10561 (2017)

  74. Raissi, M., Perdikaris, P., Karniadakis, G.E.: Physics informed deep learning (Part II): data-driven discovery of nonlinear partial differential equations. arXiv:1711.10566 (2017)

  75. Ravindran, S.S.: A reduced-order approach for optimal control of fluids using proper orthogonal decomposition. Int. J. Numer. Methods Fluids 34(5), 425–448 (2000)

    Article  MathSciNet  MATH  Google Scholar 

  76. Rowley, C.W.: Model reduction for fluids, using balanced proper orthogonal decomposition. Int. J. Bifurc. Chaos 15(03), 997–1013 (2005)

    Article  MathSciNet  MATH  Google Scholar 

  77. Rowley, C.W., Dawson, S.T.: Model reduction for flow analysis and control. Annu. Rev. Fluid Mech. 49, 387–417 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  78. Roychowdhury, J.: Reduced-order modeling of time-varying systems. IEEE Trans. Circuits Syst. II, Analog Digit. Signal Process. 46(10), 1273–1288 (1999)

    Article  Google Scholar 

  79. Sahan, R., Koc-Sahan, N., Albin, D., Liakopoulos, A.: Artificial neural network-based modeling and intelligent control of transitional flows. In: Proceedings of the 1997 IEEE International Conference on Control Applications, 1997, pp. 359–364. IEEE (1997)

  80. San, O., Iliescu, T.: Proper orthogonal decomposition closure models for fluid flows: Burgers equation. Int. J. Numer. Anal. Model. 5, 217–237 (2014)

    MathSciNet  MATH  Google Scholar 

  81. San, O., Iliescu, T.: A stabilized proper orthogonal decomposition reduced-order model for large scale quasigeostrophic ocean circulation. Adv. Comput. Math. 41(5), 1289–1319 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  82. Schmid, P.J.: Dynamic mode decomposition of numerical and experimental data. J. Fluid Mech. 656, 5–28 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  83. Serre, D.: Matrices: Theory and Applications. Springer, New York (2002)

    MATH  Google Scholar 

  84. Silveira, L.M., Kamon, M., White, J.: Efficient reduced-order modeling of frequency-dependent coupling inductances associated with 3-D interconnect structures. IEEE Transactions on Components, Packaging, and Manufacturing Technology: Part B 19(2), 283–288 (1996)

    Article  Google Scholar 

  85. Sirisup, S., Karniadakis, G.E.: A spectral viscosity method for correcting the long-term behavior of POD models. J. Comput. Phys. 194(1), 92–116 (2004)

    Article  MathSciNet  MATH  Google Scholar 

  86. Sirovich, L.: Turbulence and the dynamics of coherent structures. I-Coherent structures. II-symmetries and transformations. III-dynamics and scaling. Q. Appl. Math. 45, 561–571 (1987)

    Article  MATH  Google Scholar 

  87. Taira, K., Brunton, S.L., Dawson, S., Rowley, C.W., Colonius, T., McKeon, B.J., Schmidt, O.T., Gordeyev, S., Theofilis, V., Ukeiley, L.S.: Modal analysis of fluid flows: an overview. AIAA J. 55(12), 4013–4041 (2017)

    Article  Google Scholar 

  88. Ullmann, S., Lang, J.: A POD-Galerkin reduced model with updated coefficients for Smagorinsky LES. In: V European Conference on Computational Fluid Dynamics, ECCOMAS CFD, p. 2010 (2010)

  89. Wang, Z.: Reduced-order modeling of complex engineering and geophysical flows: analysis and computations. Ph.D. thesis, Virginia Tech (2012)

  90. Wang, Z., Akhtar, I., Borggaard, J., Iliescu, T.: Two-level discretizations of nonlinear closure models for proper orthogonal decomposition. J. Comput. Phys. 230(1), 126–146 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  91. Wang, Z., Akhtar, I., Borggaard, J., Iliescu, T.: Proper orthogonal decomposition closure models for turbulent flows: a numerical comparison. Comput. Methods Appl. Mech. Eng. 237, 10–26 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  92. Weller, J., Lombardi, E., Bergmann, M., Iollo, A.: Numerical methods for low-order modeling of fluid flows based on POD. Int. J. Numer. Methods Fluids 63(2), 249–268 (2010)

    MathSciNet  MATH  Google Scholar 

  93. Wells, D., Wang, Z., Xie, X., Iliescu, T.: An evolve-then-filter regularized reduced order model for convection-dominated flows. Int. J. Numer. Methods Fluids 84, 598–615 (2017)

    Article  MathSciNet  Google Scholar 

  94. Widrow, B., Rumelhart, D.E., Lehr, M.A.: Neural networks: applications in industry, business and science. Commun. ACM 37(3), 93–106 (1994)

    Article  Google Scholar 

  95. Xie, X., Mohebujjaman, M., Rebholz, L., Iliescu, T.: Data-driven filtered reduced order modeling of fluid flows. arXiv:1709.04362 (2017)

  96. Xie, X., Wells, D., Wang, Z., Iliescu, T.: Approximate deconvolution reduced order modeling. Comput. Methods Appl. Mech. Eng. 313, 512–534 (2017)

    Article  MathSciNet  Google Scholar 

  97. Zhang, G., Patuwo, B.E., Hu, M.Y.: Forecasting with artificial neural networks: the state of the art. Int. J. Forecast. 14(1), 35–62 (1998)

    Article  Google Scholar 

Download references

Acknowledgments

The computing for this project was performed by using resources from the High Performance Computing Center (HPCC) at Oklahoma State University.

Author information

Authors and Affiliations

  1. Department of Mechanical and Aerospace Engineering, Oklahoma State University, Stillwater, OK, 74078, USA

    Omer San & Romit Maulik

Authors
  1. Omer San
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Romit Maulik
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Omer San.

Additional information

Communicated by: Peter Benner

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

San, O., Maulik, R. Neural network closures for nonlinear model order reduction. Adv Comput Math 44, 1717–1750 (2018). https://doi.org/10.1007/s10444-018-9590-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10444-018-9590-z

Keywords

Mathematics Subject Classification (2010)

Search

Navigation