Skip to main content
SpringerLink
Log in

Deep reinforcement learning in fluid mechanics: A promising method for both active flow control and shape optimization

  • Special Column on the International Symposium on High-Fidelity Computational Methods and Applications 2019 (Guest Editors Hui Xu, Wei Zhang)
  • Published:
Journal of Hydrodynamics Aims and scope Submit manuscript

Abstract

In recent years, artificial neural networks (ANNs) and deep learning have become increasingly popular across a wide range of scientific and technical fields, including fluid mechanics. While it will take time to fully grasp the potentialities as well as the limitations of these methods, evidence is starting to accumulate that point to their potential in helping solve problems for which no theoretically optimal solution method is known. This is particularly true in fluid mechanics, where problems involving optimal control and optimal design are involved. Indeed, such problems are famously difficult to solve effectively with traditional methods due to the combination of non linearity, non convexity, and high dimensionality they involve. By contrast, deep reinforcement learning (DRL), a method of optimization based on teaching empirical strategies to an ANN through trial and error, is well adapted to solving such problems. In this short review, we offer an insight into the current state of the art of the use of DRL within fluid mechanics, focusing on control and optimal design problems.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Krizhevsky A., Sutskever I., Hinton G. E. Imagenet classification with deep convolutional neural networks [C]. Advances in Neural Information Processing Systems, Lake Tahoe, USA, 2012, 1097–1105.

  2. He K., Zhang X., Ren S. et al. Deep residual learning for image recognition [C]. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, USA, 2016, 770–778.

  3. Rabault J., Kolaas J., Jensen A. Performing particle image velocimetry using artificial neural networks: A proof-of-concept [J]. Measurement Science and Technology, 2017, 28(12): 125301.

    Article  Google Scholar 

  4. Kober J., Bagnell J. A., Peters J. Reinforcement learning in robotics: A survey [J]. The International Journal of Robotics Research, 2013, 32(11): 1238–1274.

    Article  Google Scholar 

  5. Gu S., Holly E., Lillicrap T. et al. Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates [C]. 2017 IEEE international conference on robotics and automation (ICRA), IEEE, Singapore, 2017, 3389–3396.

    Chapter  Google Scholar 

  6. Schmidhuber J. Deep learning in neural networks: An overview [J]. Neural networks, 2015, 61: 85–117.

    Article  Google Scholar 

  7. Lillicrap T. P., Hunt J. J., Pritzel A. et al. Continuous control with deep reinforcement learning [EB/OL]. arXiv preprint, 2015, arXiv:1509.02971.

  8. Sutton R. S., Barto A. G. Reinforcement learning: An introduction [M]. Cambridge, USA: MIT press, 2018.

    MATH  Google Scholar 

  9. Rabault J., Zhang W., Xu H. Deep reinforcement learning in fluid mechanics: a promising method in both active flow control and shape optimization [C]. International Symposium on High Fidelity Computational Methods and Applications, Shanghai, China, 20129.

  10. Rabault J. Deep reinforcement learning applied to fluid mechanics: materials from the 2019 flow/interface school on machine learning and data driven methods [C]. International Symposium on High Fidelity Computational Methods and Applications, Shanghai, China, 2019.

  11. Rosenblatt F. The perceptron: A perceiving and recognizing automation [R]. New York, USA: Cornell Aeronautical Laboratory, 1957, Report 85-60-1.

    Google Scholar 

  12. Goodfellow I., Bengio Y., Courville A. 2017 The Deep Learning Book [M]. Cambridge, USA: MIT Press.

    MATH  Google Scholar 

  13. LeCun Y., Bengio Y., Hinton G. Deep learning [J]. Nature, 2015, 521(7553): 436–444.

    Article  Google Scholar 

  14. LeCun Y., Bengio Y. Convolutional networks for images, speech, and time series (The handbook of brain theory and neural networks) [M]. Cambridge, USA: MIT Press, 1998.

    Google Scholar 

  15. Hornik K., Stinchcombe M., White H. Multilayer feedforward networks are universal approximators [J]. Neural Networks, 1989, 2(5): 359–366.

    Article  Google Scholar 

  16. Glorot X., Bordes A., Bengio Y. Deep sparse rectifier neural networks [C]: Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, Rome, Italy, 2011, 315–323.

  17. Glorot X., Bengio Y. Understanding the difficulty of training deep feedforward neural networks [C]. Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, Chia Laguna Resort, Italy, 2010, 249–256.

  18. Srivastava N., Hinton G., Krizhevsky A. et al. Dropout: A simple way to prevent neural networks from overfitting [J]. The Journal of Machine Learning Research, 2014, 15(1): 1929–1958.

    MathSciNet  MATH  Google Scholar 

  19. Ioffe S., Szegedy C. Batch normalization: Accelerating deep network training by reducing internal covariate shift [EB/OL]. arXiv preprint, 2015, arXiv:1502.03167.

  20. He K., Zhang X., Ren S. et al. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification [C]. Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile, 2015, 1026–1034.

  21. Li X., Chen H., Qi X. et al. H-DenseUNet: Hybrid densely connected UNet for liver and tumor segmentation from CT volumes [J]. IEEE Transactions on Medical Imaging, 2018, 37(12): 2663–2674.

    Article  Google Scholar 

  22. Kingma D. P., Ba J. Adam: A method for stochastic optimization [EB/OL]. arXiv preprint, 2014, arXiv: 1412.6980.

  23. Duchi J., Hazan E., Singer Y. Adaptive subgradient methods for online learning and stochastic optimization [J]. Journal of Machine Learning Research, 2011, 12(7): 2121–2159.

    MathSciNet  MATH  Google Scholar 

  24. Abadi M., Barham P., Chen J. et al. Tensorflow: A system for large-scale machine learning [C]. Proceedings of the 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI’16), Savannah, USA, 2016, 265–283.

  25. Paszke A., Gross S., Massa F. et al. PyTorch: An imperative style, high-performance deep learning library [C]. Advances in Neural Information Processing Systems, Vancouver, Canada, 2019, 8024–8035.

  26. Silver D., Schrittwieser J., Simonyan K. et al. Mastering the game of go without human knowledge [J]. Nature, 2017, 550(7676): 354–359.

    Article  Google Scholar 

  27. Silver D., Hubert T., Schrittwieser J. et al. A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play [J]. Science, 2018, 362(6419): 1140–1144.

    Article  MathSciNet  Google Scholar 

  28. Knight W. Google just gave control over data center cooling to an AI [EB/OL]. 2018, https://www.technologyreview.com/s/611902/google-just-gave-control-over-data-center-cooling-to-an-ai/.

  29. Bellman R. A Markovian decision process [J]. Journal of Mathematics and Mechanics, 1957, 6(4): 679–684.

    MathSciNet  MATH  Google Scholar 

  30. Bellman R. E., Dreyfus S. E. Applied dynamic programming [M]. Princeton, USA: Princeton University Press, 2015.

    MATH  Google Scholar 

  31. Van Hasselt H., Guez A., Silver D. Deep reinforcement learning with double q-learning [C]. Thirtieth AAAI Conference on Artificial Intelligence, Phoenix, USA, 2016.

  32. Schulman J., Levine S., Abbeel P. et al. Trust region policy optimization [C]. International Conference on Machine Learning, Guangzhou, China, 2015, 1889–1897.

  33. Garnier P., Viquerat J., Rabault J. et al. A review on deep reinforcement learning for fluid mechanics [EB/OL]. arXiv preprint, 2019, arXiv:1908.04127.

  34. Schaul T., Quan J., Antonoglou I. et al. Prioritized experience replay [EB/OL]. arXiv preprint, 2015, arXiv:1511.05952.

  35. Pinto L., Andrychowicz M., Welinder P. et al. Asymmetric actor critic for image-based robot learning [EB/OL]. arXiv preprint, 2017, arXiv:1710.06542.

  36. Schulman J., Wolski F., Dhariwal P. et al. Proximal policy optimization algorithms [EB/OL]. arXiv preprint, 2017, arXiv:1707.06347,.

  37. Achiam J, Sastry S. Surprise-based intrinsic motivation for deep reinforcement learning[J]. arXiv preprint, 2017, arXiv:1703.01732.

  38. Savinov N., Raichuk A., Marinier R. et al. Episodic curiosity through reachability [EB/OL]. arXiv preprint, 2018, arXiv:1810.02274.

  39. Ha D., Schmidhuber J. World models [EB/OL]. arXiv preprint, 2018, arXiv:1803.10122.

  40. Salimans T., Chen R. Learning Montezuma’s revenge from a single demonstration [EB/OL]. arXiv preprint, 2018, arXiv:1812.03381.

  41. Tensorforce: A tensorflow library for applied reinforcement learning [EB/OL]. 2017, https://tensorforce.readthedocs.io/en/latest/.

  42. Stable baselines [EB/OL]. 2018, https://github.com/hill-a/stable-baselines.

  43. Duriez T., Brunton S. L., Noack B. R. Machine learning control-taming nonlinear dynamics and turbulence [M]. Cham, Switzerland: Springer International Publishing, 2017.

    Book  Google Scholar 

  44. Novati G., Verma S., Alexeev D. et al. Synchronised swimming of two fish [J]. Bioinspiration and Biomimetics, 2017, 12(3): 036001.

    Article  Google Scholar 

  45. Reddy G., Celani A., Sejnowski T. J. et al. Learning to soar in turbulent environments [J]. Proceedings of the National Academy of Sciences, 2016, 113(33): E4877–E4884.

    Article  Google Scholar 

  46. Bøhn E., Coates E. M., Moe S. et al. Deep reinforcement learning attitude control of fixed-wing UAVs using proximal policy optimization [C]. 2019 International Conference on Unmanned Aircraft Systems (ICUAS), IEEE, Atlanta, USA, 2019, 523–533.

  47. Hwangbo J., Sa I., Siegwart R. et al. Control of a quadrotor with reinforcement learning [J]. IEEE Robotics and Automation Letters, 2017, 2(4): 2096–2103.

    Article  Google Scholar 

  48. Biferale L., Bonaccorso F., Buzzicotti M. et al. Zermelo’s problem: Optimal point-to-point navigation in 2D turbulent flows using reinforcement learning [J]. Chaos: An Interdisciplinary Journal of Nonlinear Science, 2019, 29(10): 103138.

    Article  MathSciNet  Google Scholar 

  49. Rabault J., Kuchta M., Jensen A. et al. Artificial neural networks trained through deep reinforcement learning discover control strategies for active flow control [J]. Journal of Fluid Mechanics, 2019, 865: 281–302.

    Article  MathSciNet  Google Scholar 

  50. Ren F., Rabault J., Tang H. Active flow control of flow past a circular cylinder at moderate reynolds number using deep reinforcement learning [C]. International Symposium on High Fidelity Computational Methods and Applications, Shanghai, China, 2019.

  51. Rabault J., Kuhnle A. Accelerating deep reinforcement learning strategies of flow control through a multienvironment approach [J]. Physics of Fluids, 2019, 31(9): 094105.

    Article  Google Scholar 

  52. Clark J., Amodei D. Faulty reward functions in the wild [EB/OL]. 2016, https://openai.com/blog/faulty-reward-functions/.

  53. Baker B., Ingmar K., Markov T. et al. Emergent tool use from multi-agent interaction [EB/OL]. arXiv preprint, 2019, arXiv:1909.07528.

  54. Belus V., Rabault J., Viquerat J. et al. Exploiting locality and translational invariance to design effective deep reinforcement learning control of the 1-dimensional unstable falling liquid film [J]. AIP Advances, 2019, 9(12): 125014.

    Article  Google Scholar 

  55. Rabault J., Belus V., Viquerat J. et al. Exploiting locality and physical invariants to design effective deep reinforcement learning control of the unstable falling liquid film [C]. The 1st Graduate Forum of CSAA and the 7th International Academic Conference for Graduates, NUAA, Nanjing, China, 2019.

  56. Bucci M. A., Semeraro O., Allauzen A. et al. Control of chaotic systems by deep reinforcement learning [EB/OL]. arXiv preprint, 2019, arXiv:1906.07672.

  57. Corbetta A., Beintema G., Biferale L. et al. Reinforcement learning versus linear control of Rayleigh-Bénard convection [C]. American Physical Society, Division of Fluid Dynamics Meeting, Philadelphia, USA, 1998.

  58. Collis S. S., Joslin R. D., Seifert A. et al. Issues in active flow control: Theory, control, simulation, and experiment [J]. Progress in Aerospace Sciences, 2004, 40(4–5): 237–289.

    Article  Google Scholar 

  59. Cattafesta III L. N., Sheplak M. Actuators for active flow control [J]. Annual Review of Fluid Mechanics, 2011, 43: 247–272.

    Article  Google Scholar 

  60. Hwangbo J., Lee J., Dosovitskiy A. et al. Learning agile and dynamic motor skills for legged robots [EB/OL]. arXiv preprint, 2019, arXiv:1901.08652.

  61. Yan X., Zhu J., Kuang M. et al. Aerodynamic shape optimization using a novel optimizer based on machine learning techniques [J]. Aerospace Science and Technology, 2019, 86: 826–835.

    Article  Google Scholar 

  62. Viquerat J., Rabault J., Kuhnle A. et al. Direct shape optimization through deep reinforcement learning [EB/OL]. arXiv preprint, 2019, arXiv:1908.09885.

Download references

Acknowledgements

This work was supported by the National Numerical Wind Tunnel Project (Grant No. NNW2019ZT4-B09), the National Natural Science Foundation of China (Grant Nos. 91852106, 91841303).

Author information

Authors and Affiliations

  1. Department of Mathematics, University of Oslo, Oslo, Norway

    Jean Rabault

  2. Research Center of Fluid Structure Interactions Department of Mechanical Engineering, Hong Kong Polytechnical University, Hong Kong, China

    Feng Ren & Hui Tang

  3. Science and Technology on Water Jet Propulsion Laboratory, Marine Design and Research Institute of China, Shanghai, 200011, China

    Wei Zhang

  4. School of Aeronautic and Astronautic, Shanghai Jiao Tong University, Shanghai, 200011, China

    Hui Xu

  5. Department of Aeronautics, Imperial College London, London, UK

    Hui Xu

Authors
  1. Jean Rabault
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Feng Ren
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Wei Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Hui Tang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Hui Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hui Xu.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Rabault, J., Ren, F., Zhang, W. et al. Deep reinforcement learning in fluid mechanics: A promising method for both active flow control and shape optimization. J Hydrodyn 32, 234–246 (2020). https://doi.org/10.1007/s42241-020-0028-y

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s42241-020-0028-y

Key words

Search

Navigation