## Abstract

We demonstrate several techniques to encourage practical uses of neural networks for fluid flow estimation. In the present paper, three perspectives which are remaining challenges for applications of machine learning to fluid dynamics are considered: 1. interpretability of machine-learned results, 2. bulking out of training data, and 3. generalizability of neural networks. For the interpretability, we first demonstrate two methods to observe the internal procedure of neural networks, i.e., visualization of hidden layers and application of gradient-weighted class activation mapping (Grad-CAM), applied to canonical fluid flow estimation problems—(1) drag coefficient estimation of a cylinder wake and (2) velocity estimation from particle images. It is exemplified that both approaches can successfully tell us evidences of the great capability of machine learning-based estimations. We then utilize some techniques to bulk out training data for super-resolution analysis and temporal prediction for cylinder wake and NOAA sea surface temperature data to demonstrate that sufficient training of neural networks with limited amount of training data can be achieved for fluid flow problems. The generalizability of machine learning model is also discussed by accounting for the perspectives of inter/extrapolation of training data, considering super-resolution of wakes behind two parallel cylinders. We find that various flow patterns generated by complex interaction between two cylinders can be reconstructed well, even for the test configurations regarding the distance factor. The present paper can be a significant step toward practical uses of neural networks for both laminar and turbulent flow problems.

### Similar content being viewed by others

### Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.## References

Kreinovich VY (1991) Arbitrary nonlinearity is sufficient to represent all functions by neural networks: a theorem. Neural Netw. 4:381–383

Hornik K (1991) Approximation capabilities of multilayer feedforward networks. Neural Netw. 4:251–257

Cybenko G (1989) Approximation by superpositions of a sigmoidal function. Math. Control Signals Syst. 2:303–314

Baral C, Fuentes O, Kreinovich V (2018) Why deep neural networks: a possible theoretical explanation. In: Ceberio M, Kreinovich V (eds) Constraint programming and decision making: theory and applications. Studies in systems, decision and control, vol 100. Springer, Cham. https://doi.org/10.1007/978-3-319-61753-4_1

Brunton SL, Noack BR, Koumoutsakos P (2020) Machine learning for fluid mechanics. Annu. Rev. Fluid Mech. 52:477–508

Duraisamy K, Iaccarino G, Xiao H (2019) Turbulence modeling in the age of data. Annu. Rev. Fluid. Mech. 51:357–377

Gamahara M, Hattori Y (2017) Searching for turbulence models by artificial neural network. Phys. Rev. Fluids 2(5):054604

Maulik R, San O (2017) A neural network approach for the blind deconvolution of turbulent flows. J. Fluid Mech. 831:151–181

Maulik R, San O, Jacob JD, Crick C (2019) Sub-grid scale model classification and blending through deep learning. J. Fluid Mech. 870:784–812

Maulik R, San O, Rasheed A, Vedula P (2019) Subgrid modelling for two-dimensional turbulence using neural networks. J. Fluid Mech. 858:122–144

Yang XIA, Zafar S, Wang J-X, Xiao H (2019) Predictive large-eddy-simulation wall modeling via physics-informed neural networks. Phys. Rev. Fluids 4(3):034602

Pawar S, San O, Rasheed A, Vedula P (2020) A priori analysis on deep learning of subgrid-scale parameterizations for kraichnan turbulence. Theor. Comput. Fluid Dyn. 34:429–455

Ling J, Kurzawski A, Templeton J (2016) Reynolds averaged turbulence modelling using deep neural networks with embedded invariance. J. Fluid Mech. 807:155–166

Milani PM, Ling J, Eaton JK (2020) Turbulent scalar flux in inclined jets in crossflow: counter gradient transport and deep learning modelling. J Fluid Mech 906:A27

Geneva N, Zabaras N (2019) Quantifying model form uncertainty in Reynolds-averaged turbulence models with bayesian deep neural networks. J. Comput. Phys. 383:125–147

Novati G, de Laroussilhe HL, Koumoutsakos P (2021) Automating turbulence modeling by multi-agent reinforcement learning. Nat. Mach. Intell. 3:87–96

Taira K, Hemati MS, Brunton SL, Sun Y, Duraisamy K, Bagheri S, Dawson S, Yeh CA (2020) Modal analysis of fluid flows: Applications and outlook. AIAA J. 58(3):998–1022

Wang Z, Xiao D, Fang F, Govindan R, Pain CC, Guo Y (2018) Model identification of reduced order fluid dynamics systems using deep learning. Int. J. Numer. Methods Fluids 86(4):255–268

Srinivasan PA, Guastoni L, Azizpour H, Schlatter P, Vinuesa R (2019) Predictions of turbulent shear flows using deep neural networks. Phys. Rev. Fluids 4:054603

Milano M, Koumoutsakos P (2002) Neural network modeling for near wall turbulent flow. J. Comput. Phys. 182:1–26

Fukami K, Hasegawa K, Nakamura T, Morimoto M, Fukagata K (2020) Model order reduction with neural networks: application to laminar and turbulent flows. SN Comput Sci 2:467

Murata T, Fukami K, Fukagata K (2020) Nonlinear mode decomposition with convolutional neural networks for fluid dynamics. J. Fluid Mech. 882:A13

Fukami K, Nakamura T, Fukagata K (2020) Convolutional neural network based hierarchical autoencoder for nonlinear mode decomposition of fluid field data. Phys. Fluids 32:095110

Fukami K, Fukagata K, Taira K (2020) Assessment of supervised machine learning for fluid flows. Theor. Comput. Fluid Dyn. 34(4):497–519

Fukami K, Nabae Y, Kawai K, Fukagata K (2019) Synthetic turbulent inflow generator using machine learning. Phys. Rev. Fluids 4:064603

Salehipour H, Peltier WR (2019) Deep learning of mixing by two ‘atoms’ of stratified turbulence. J. Fluid Mech. 861:R4

Fukami K, Fukagata K, Taira K (2019) Super-resolution reconstruction of turbulent flows with machine learning. J. Fluid Mech. 870:106–120

Fukami K, Fukagata K, Taira K (2019) Super-resolution analysis with machine learning for low-resolution flow data. In: 11th International Symposium on Turbulence and Shear Flow Phenomena (TSFP11), Southampton, UK, number 208,

Liu B, Tang J, Huang H, Lu X-Y (2020) Deep learning methods for super-resolution reconstruction of turbulent flows. Phys. Fluids 32:025105

Deng Z, He C, Liu Y, Kim KC (2019) Super-resolution reconstruction of turbulent velocity fields using a generative adversarial network-based artificial intelligence framework. Phys. Fluids 31:125111

Fukami K, Fukagata K, Taira K (2021) Machine-learning-based spatio-temporal super resolution reconstruction of turbulent flows. J. Fluid Mech., 909(A9),

Cai S, Zhou S, Xu C, Gao Q (2019) Dense motion estimation of particle images via a convolutional neural network. Exp. Fluids 60:60–73

Morimoto M, Fukami K, Fukagata K (2021) Experimental velocity data estimation for imperfect particle images using machine learning. Phys Fluids 33:087121

Brunton SL, Hemanti MS, Taira K (2020) Special issue on machine learning and data-driven methods in fluid dynamics. Theor. Comput. Fluid Dyn. 34(4):333–337

Lee C, Kim J, Babcock D, Goodman R (1997) Application of neural networks to turbulence control for drag reduction. Phys. Fluids 9(6):1740–1747

Choi H, Moin P, Kim J (1994) Active turbulence control for drag reduction in wall-bounded flows. J. Fluid Mech. 262(3):75–110

Garnier P, Viquerat J, Rabault J, Larcher A, Kuhnle A, Hachem E (2021) A review on deep reinforcement learning for fluid mechanics. Comput Fluids 225:104973

Rabault J, Kuchta M, Jensen A, Réglade U, Cerardi N (2019) Artificial neural networks trained through deep reinforcement learning discover control strategies for active flow control. J. Fluid Mech. 865:281–302

Tang H, Rabault J, Kuhnle A, Wang Y, Wang T (2020) Robust active flow control over a range of reynolds numbers using an artificial neural network trained through deep reinforcement learning. Phys. Fluids 32(5):053605

Maulik R, Fukami K, Ramachandra N, Fukagata K, Taira K (2020) Probabilistic neural networks for fluid flow surrogate modeling and data recovery. Phys. Rev. Fluids 5:104401

Jagodinski E, Zhu X, Verma S (2020) Uncovering dynamically critical regions in near-wall turbulence 3D convolutional neural networks. arXiv:2004.06187

Selvaraju R. R., Das A, Vedantam R, Cogswell M, Parikh D, Batra D (2016) Grad-CAM: Why did you say that? arXiv:1611.07450

Selvaraju R. R., Cogswell M, Das A, Vedantam R, Parikh D, Batra D (2017) Grad-CAM: Visual explanations from deep networks via gradient-based localization. In: Proc. IEEE Int. Conf. Comput. Vis., pages 618–626

Kim J, Lee C (2020) Prediction of turbulent heat transfer using convolutional neural networks. J. Fluid Mech. 882:A18

Kutz JN (2017) Deep learning in fluid dynamics. J. Fluid Mech. 814:1–4

Hasegawa K, Fukami K, Murata T, Fukagata K (2020) CNN-LSTM based reduced order modeling of two-dimensional unsteady flows around a circular cylinder at different Reynolds numbers. Fluid Dyn. Res. 52(6):065501

Hasegawa K, Fukami K, Murata T, Fukagata K (2020) Machine-learning-based reduced-order modeling for unsteady flows around bluff bodies of various shapes. Theor. Comput. Fluid Dyn. 34(4):367–388

Erichson NB, Mathelin L, Yao Z, Brunton SL, Mahoney MW, Kutz JN (2020) Shallow learning for fluid flow reconstruction with limited sensors. Proc. Royal Soc. A 476(2238):20200097

Kor H, Ghomizad M. Badri, Fukagata K (2017) A unified interpolation stencil for ghost-cell immersed boundary method for flow around complex geometries. J. Fluid Sci. Technol., 12(1):JFST0011

Franke R, Rodi W, Schonung B (1990) Numerical calculation of laminar vortex shedding flow past cylinders. J. Wind Eng. Ind. Aerodyn. 35:237–257

Robichaux J, Balachandar S, Vanka SP (1999) Three-dimensional floquet instability of the wake of square cylinder. Phys. Fluids 11:560

Caltagirone JP (1994) Sur l’interaction fluide-milieu poreux: application au calcul des efforts excerses sur un obstacle par un fluide visqueux. C. R. Acad. Sci. Paris 318:571–577

Bai H, Alam MdM (2018) Dependence of square cylinder wake on Reynolds number. Phys. Fluids 30:015102

Available on https://www.esrl.noaa.gov/psd/

Weller HG, Tabor G, Jasak H, Fureby C (1998) A tensorial approach to computational continuum mechanics using object-oriented techniques. Comput. Phys. 12(6):620–631

Rumelhart DE, Hinton GE, Williams RJ (1986) Learning representations by back-propagation errors. Nature 322:533–536

Domingos P (2012) A few useful things to know about machine learning. Communications of the ACM 55(10):78–87

Lui HFS, Wolf WR (2019) Construction of reduced-order models for fluid flows using deep feedforward neural networks. J. Fluid Mech. 872:963–994

Yu J, Hesthaven JS (2019) Flowfield reconstruction method using artificial neural network. AIAA J. 57(2):482–498

Kingma D. P., Ba J (2014) Adam: A method for stochastic optimization. arXiv:1412.6980

Nair V, Hinton G. E. (2010) Rectified linear units improve restricted boltzmann machines. Proc. Int. Conf. Mach. Learn., pages 807–814

LeCun Y, Bottou L, Bengio Y, Haffner P (1998) Gradient-based learning applied to document recognition. Proc. IEEE 86(11):2278–2324

Matsuo M, Nakamura T, Morimoto M, Fukami K, Fukagata K (2021) Supervised convolutional network for three-dimensional fluid data reconstruction from sectional flow fields with adaptive super-resolution assistance. arXiv:2103.09020

Moriya N, Fukami K, Nabae Y, Morimoto M, Nakamura T, Fukagata K (2021) Inserting machine-learned virtual wall velocity for large-eddy simulation of turbulent channel flows. arXiv:2106.09271

Nakamura T, Fukami K, Fukagata K (2021) Comparison of linear regressions and neural networks for fluid flow problems assisted with error-curve analysis. arXiv:2105.00913

Du X, Qu X, He Y, Guo D (2018) Single image super-resolution based on multi-scale competitive convolutional neural network. Sensors 18(789):1–17

Zhang Y, Sung W, Marvis D (2018) Application of convolutional neural network to predict airfoil lift coefficient. AIAA paper, 2018–1903

Miyanawala T.P., Jaiman R.K. (2018) A novel deep learning method for the predictions of current forces on bluff bodies. In: Proceedings of the ASME 2018 37th International Conference on Ocean, Offshore and Arctic Engineering OMAE2018, pages 1–10

Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations

Guastoni L, Güemes A, Ianiro A, Discetti S, Schlatter P, Azizpour H, Vinuesa R (2020) Convolutional-network models to predict wall-bounded turbulence from wall quantities. J Fluid Mech 928:A27

Shorten C, Khoshgoftaar TM (2019) A survey on image data augmentation for deep learning. J. Big Data 6(1):60

Huang J, Liu H, Wang Q, Cai W (2020) Limited-projection volumetric tomography for time-resolved turbulent combustion diagnostics via deep learning. Aerosp. Sci. Technol., 106(106123)

Mikołajczyk A, Grochowski M (2018) Data augmentation for improving deep learning in image classification problem. In: 2018 international interdisciplinary PhD workshop (IIPhDW), pages 117–122. IEEE

Maulik R, San O (2017) Resolution and energy dissipation characteristics of implicit LES and explicit filtering models for compressible turbulence. Fluids 2(2):14

Duraisamy K (2021) Perspectives on machine learning-augmented reynolds-averaged and large eddy simulation models of turbulence. Phys. Rev. Fluids 6:050504

Fukami K, Maulik R, Ramachandra N, Fukagata K, Taira K (2021) Global field reconstruction from sparse sensors with Voronoi tessellation-assisted deep learning. Nat Mach Intell. https://doi.org/10.1038/s42256-021-00402-2

Stengel K, Glaws A, Hettinger D, King RN (2020) Adversarial super-resolution of climatological wind and solar data. Proc. Natl. Acad. Sci. USA 117(29):16805–16815

Nakamura T, Fukami K, Hasegawa K, Nabae Y, Fukagata K (2021) Convolutional neural network and long short-term memory based reduced order surrogate for minimal turbulent channel flow. Phys. Fluids 33:025116

Brunton SL, Proctor JL, Kutz JN (2016) Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proc. Natl. Acad. Sci. USA 113(15):3932–3937

Fukami K, Murata T, Fukagata K (2021) Sparse identification of nonlinear dynamics with low-dimensionalized flow representations. J Fluid Mech 926:A10

Saku Y, Aizawa M, Ooi T, Ishigami G (2021) Spatio-temporal prediction of soil deformation in bucket excavation using machine learning. Adv Robot. https://doi.org/10.1080/01691864.2021.1943521

Morimoto M, Fukami K, Zhang K, Nair AG, Fukagata K (2021) Convolutional neural networks for fluid flow analysis: toward effective metamodeling and low-dimensionalization. Theor Comput Fluid Dyn 35:633–658

Raissi M, Yazdani A, Karniadakis GE (2020) Hidden fluid mechanics: Learning velocity and pressure fields from flow visualizations. Science 367(6481):1026–1030

Wu Z, Pan S, Chen F, Long G, Zhang C, Philip SY (2021) A comprehensive survey on graph neural networks. IEEE Trans Neural Netw Learn Syst 32(1):4–24

Inubushi M, Goto S (2019) Transferring reservoir computing: Formulation and application to fluid physics. In: International Conference on Artificial Neural Networks, pages 193–199. Springer

Manohar K, Brunton BW, Kutz JN, Brunton SL (2018) Data-driven sparse sensor placement for reconstruction: Demonstrating the benefits of exploiting known patterns. IEEE Control Syst. 38(3):63–86

Nakai K, Yamada K, Nagata T, Saito Y, Nonomura T (2020) Effect of objective function on data-driven sparse sensor optimization. IEEE Access 9:46731–46743

Saito Y, Nonomura T, Nankai K, Yamada K, Asai K, Tsubakino Y, Tsubakino D () Data-driven vector-measurement-sensor selection based on greedy algorithm. IEEE Sensors Letters, 2020

## Acknowledgments

This work was supported from Japan Society for the Promotion of Science (KAKENHI grant number: 18H03758, 21H05007).

## Author information

### Authors and Affiliations

### Corresponding author

## Ethics declarations

### Conflict of interest

The authors declare that they have no conflict of interest.

## Additional information

### Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

## Rights and permissions

## About this article

### Cite this article

Morimoto, M., Fukami, K., Zhang, K. *et al.* Generalization techniques of neural networks for fluid flow estimation.
*Neural Comput & Applic* **34**, 3647–3669 (2022). https://doi.org/10.1007/s00521-021-06633-z

Received:

Accepted:

Published:

Issue Date:

DOI: https://doi.org/10.1007/s00521-021-06633-z