Finding efficient swimming strategies in a three-dimensional chaotic flow by reinforcement learning

  • K. Gustavsson
  • L. Biferale
  • A. Celani
  • S. Colabrese
Regular Article
Part of the following topical collections:
  1. Fluids and Structures: Multi-scale coupling and modeling


We apply a reinforcement learning algorithm to show how smart particles can learn approximately optimal strategies to navigate in complex flows. In this paper we consider microswimmers in a paradigmatic three-dimensional case given by a stationary superposition of two Arnold-Beltrami-Childress flows with chaotic advection along streamlines. In such a flow, we study the evolution of point-like particles which can decide in which direction to swim, while keeping the velocity amplitude constant. We show that it is sufficient to endow the swimmers with a very restricted set of actions (six fixed swimming directions in our case) to have enough freedom to find efficient strategies to move upward and escape local fluid traps. The key ingredient is the learning-from-experience structure of the algorithm, which assigns positive or negative rewards depending on whether the taken action is, or is not, profitable for the predetermined goal in the long-term horizon. This is another example supporting the efficiency of the reinforcement learning approach to learn how to accomplish difficult tasks in complex fluid environments.

Graphical abstract


Topical issue: Fluids and Structures: Multi-scale coupling and modeling 


  1. 1.
    T.J. Pedley, J.O. Kessler, Annu. Rev. Fluid Mech. 24, 313 (1992)ADSCrossRefGoogle Scholar
  2. 2.
    T. Fenchel, Science 296, 1068 (2002)ADSCrossRefGoogle Scholar
  3. 3.
    T. Kiørboe, G.A. Jackson, Limnol. Oceanogr. 46, 1309 (2001)ADSCrossRefGoogle Scholar
  4. 4.
    E. Lauga, T.R. Powers, Rep. Prog. Phys. 72, 096601 (2009)ADSCrossRefGoogle Scholar
  5. 5.
    S.J. Ebbens, J.R. Howse, Soft Matter 6, 726 (2010)ADSCrossRefGoogle Scholar
  6. 6.
    A. Ghosh, P. Fischer, Nano Lett. 9, 2243 (2009)ADSCrossRefGoogle Scholar
  7. 7.
    A.R. Hall, J. Carpenter, A. Shields, K. Ford, M. Millard, L.O. Mair, B. Evans, R. Superfine, J. Phys. D 44, 125001 (2011)ADSCrossRefGoogle Scholar
  8. 8.
    P. Fischer, A. Ghosh, Nanoscale 3, 557 (2011)ADSCrossRefGoogle Scholar
  9. 9.
    J. Wang, W. Gao, ACS Nano 6, 5745 (2012)CrossRefGoogle Scholar
  10. 10.
    B. Hejazialhossein, M. Gazzola, P. Koumoutsakos, SIAM J. Sci. Comput. 36, B622 (2014)CrossRefGoogle Scholar
  11. 11.
    D. Alexeev, A. de Brauer, P. Koumoutsakos, M. Gazzola, A.A. Tchieu, J. Fluid Mech. 789, 726 (2016)ADSMathSciNetCrossRefGoogle Scholar
  12. 12.
    B.J. Nelson, I.K. Kaliakatsos, J.J. Abbott, Annu. Rev. Biol. Eng. 12, 55 (2010)CrossRefGoogle Scholar
  13. 13.
    W. Gao, J. Wang, ACS Nano 8, 3170 (2014)CrossRefGoogle Scholar
  14. 14.
    L.K.E. Abdelmohsen, F. Peng, Y. Tu, D.A. Wilson, J. Mater. Chem. B 2, 2395 (2014)CrossRefGoogle Scholar
  15. 15.
    D. Patra, S. Sengupta, W. Duan, H. Zhang, R. Pavlick, A. Sen, Nanoscale 5, 1273 (2013)ADSCrossRefGoogle Scholar
  16. 16.
    G. Reddy, A. Celani, T.J. Sejnowski, M. Vergassola, Proc. Natl. Acad. Sci. U.S.A. 113, E4877 (2016)ADSCrossRefGoogle Scholar
  17. 17.
    S. Colabrese, K. Gustavsson, A. Celani, L. Biferale, Phys. Rev. Lett. 118, 158004 (2017)ADSCrossRefGoogle Scholar
  18. 18.
    M. Cencini, F. Santamaria, F. De Lillo, G. Boffetta, Phys. Fluids 26, 111901 (2014)ADSCrossRefGoogle Scholar
  19. 19.
    J.O. Kessler, W.M. Durham, R. Stocker, Science 323, 1067 (2009)ADSCrossRefGoogle Scholar
  20. 20.
    M. Barry, F. De Lillo, G. Boffetta, M. Cencini, W.M. Durham, E. Climent, R. Stocker, Nat. Commun. 4, 2148 (2013)Google Scholar
  21. 21.
    A. Puglisi, A. Sarracino, F. Cecconi, A. Vulpiani, Phys. Rev. Lett. 117, 174501 (2016)ADSCrossRefGoogle Scholar
  22. 22.
    F. Cecconi, A. Puglisi, A. Sarracino, A. Vulpiani, Eur. Phys. J. E 40, 81 (2017)CrossRefGoogle Scholar
  23. 23.
    E.M. Purcell, Am. J. Phys. 45, 1 (1977)ADSCrossRefGoogle Scholar
  24. 24.
    P.R. Jonsson, K. Gustavsson, F. Berglund, B. Mehlig, Phys. Rev. Lett. 116, 108104 (2016)ADSCrossRefGoogle Scholar
  25. 25.
    R.S. Sutton, A.G. Barto, Reinforcement Learning: An Introduction (MIT Press, Cambridge, 1998)Google Scholar

Copyright information

© EDP Sciences, SIF, Springer-Verlag GmbH Germany, part of Springer Nature 2017

Authors and Affiliations

  • K. Gustavsson
    • 1
  • L. Biferale
    • 2
  • A. Celani
    • 3
  • S. Colabrese
    • 2
  1. 1.Department of PhysicsUniversity of GothenburgGöteborgSweden
  2. 2.Department of Physics and INFNUniversity of Rome Tor VergataRomeItaly
  3. 3.Quantitative Life SciencesThe Abdus Salam International Centre for Theoretical PhysicsTriesteItaly

Personalised recommendations