Steering undulatory micro-swimmers in a fluid flow through reinforcement learning

Abstract

This work aims at finding optimal navigation policies for thin, deformable microswimmers that progress in a viscous fluid by propagating a sinusoidal undulation along their slender body. These active filaments are embedded in a prescribed, non-homogeneous flow, in which their swimming undulations have to compete with the drifts, strains, and deformations inflicted by the outer velocity field. Such an intricate situation, where swimming and navigation are tightly bonded, is addressed using various methods of reinforcement learning. Each swimmer has only access to restricted information on its configuration and has to select accordingly an action among a limited set. The optimisation problem then consists in finding the policy leading to the most efficient displacement in a given direction. It is found that usual methods do not converge and this pitfall is interpreted as a combined consequence of the non-Markovianity of the decision process, together with the highly chaotic nature of the dynamics, which is responsible for high variability in learning efficiencies. Still, we provide an alternative method to construct efficient policies, which is based on running several independent realisations of Q-learning. This allows the construction of a set of admissible policies whose properties can be studied in detail and compared to assess their efficiency and robustness.

Graphical abstract

Appendices

Appendix A: Reinforcement-learning algorithms

We give here details on the three reinforcement-learning algorithms that have been used in this work and whose results are discussed in Sect. 4, namely Q-learning, differential semi-gradient SARSA, and policy gradient/Actor-Critic.

1.1 A.1 Q-learning

Q-learning is a model-free reinforcement-learning algorithm used to estimate the optimal action-value function for a given environment [38]. It uses the discounted reward $\mathcal {R}^\textrm{disc}[\pi ]$ defined by Eq. (4) with discount rate $\gamma$. In this algorithm, an agent interacts with the environment, observes the state and reward, and takes an action according to an $\varepsilon$-greedy policy. An estimate of the action-value function Q defined in Sect. 3.2 is updated at each step using the immediate reward R and the maximum estimated value for the next state, using a learning rate parameter $\lambda$. The algorithm continues until convergence is reached or a stopping criterion is met.

The resulting procedure is summarised below:

1.2 A.2 Differential semi-gradient SARSA

This is another model-free reinforcement learning algorithm used to learn the optimal action-value function Q associated to the differential return $\mathcal {R}^\textrm{diff}[\pi ]$ defined in Eq. (5). In this procedure, the agent interacts with the environment, observes the state and reward, and takes an action according to an $\varepsilon$-greedy policy following an approximation $\hat{Q}_{\varvec{\eta }}$ of the action-value function. The approximation parameters $\varvec{\eta }$ are updated with a learning rate $\lambda _2$ using a temporal-difference (TD) error $\delta$, which is the difference between the observed reward and an estimated value $\bar{R}$ of the next state. Additionally, the algorithm updates the estimate $\bar{R}$ at a rate $\lambda _1$. Details can be found in [37].

The procedure is then:

1.3 A.3 Policy gradient/Actor-Critic

This last procedure is a model-based reinforcement learning algorithm used to learn a policy that maximises the expected differential return $\mathcal {R}^\textrm{diff}[\pi ]$. In this algorithm, the agent interacts with the environment, observes the state and reward, and takes an action according to an approximation $\hat{\pi }_{\varvec{\theta }}$ of the optimal policy. The TD error $\delta$ is this time defined as the difference between the observed reward and the estimated value $\hat{V}_{\varvec{\eta }}$ of the next state. The algorithm uses it to update the estimated average reward $\bar{R}$ at a rate $\lambda _1$, and the approximation parameters $\varvec{\eta }$ used for the value function at a rate $\lambda _2$. Additionally, the approximation parameters $\varvec{\theta }$ of the policy are updated at a rate $\lambda _3$ using both the TD error and the gradient of the log-likelihood of the policy with respect to $\varvec{\theta }$. This algorithm is known as the Actor-Critic algorithm [41] because it simultaneously learns an actor (policy $\hat{\pi }_{\varvec{\theta }}$) and a critic (value function $\hat{V}_{\varvec{\eta }}$).

The resulting procedure is summarised below:

Appendix B: The iterative Markovian approximation

We here give some details on the idea to approximate the dynamical evolution of the swimmer by an MDP. Our hope is that this approximation will capture the most relevant information of our optimisation problem, namely, the transition probabilities between the states of our environment and the distribution of the rewards obtained by our agent. The advantages of this approach are twofold. First, MDPs only require knowledge of the transitions and rewards, abstracting away all the other aspects of the dynamics. This approximation thus enables learning algorithms to run significantly faster, without the need to simulate the entire system at each step. Second, this approach would separate the issue of non-Markovianity from other potential difficulties.

Our procedure consists in constructing a sequence of policies $\pi _0$, $\pi _1$, ...$\pi _k$ that will hopefully converge to the optimal $\pi _\star$. At each step, we simulate a swimmer that follows the policy $\pi _k$. Once a statistical steady state is reached, we try out at every time step $t=n\Delta t$, all possible actions to monitor the new observation and reward at time $t_{n+1}$. We then use long time averages (over $10^6\Delta t$) to construct numerical Monte-Carlo approximations to the transition probability $p_{\textrm{T},k}(\omega '\vert \omega ,\alpha )$ of observing $\omega '$ at time $t+\Delta t$ given that $\omega$ was observed and action $\alpha$ was performed at time t, together with the corresponding distribution of rewards $p_{\textrm{R},k}(R\vert \omega ,\alpha )$. Both distributions depend of course on $\pi _k$. We then use the approximate probabilities $p_{\textrm{T},k}$ and $p_{\textrm{R},k}$ to run the $\varepsilon$-greedy Q-learning algorithm that, because of the Markovian formulation imposed now, is ensured to converge. This leads to construct the optimal policy $\pi _{k+1}$ associated to the approximate system. This procedure is reiterated changing the base policy to $\pi _{k+1}$, until it attains a fixed point. The method is summarised in Algorithm 4.

The motivation behind this procedure is that if the Markovian approximation is not too far off, then it is natural to think that the optimal policy $\pi _{k+1}$ of the approximate system should be at least an improvement on the policy $\pi _k$ if not also the optimal policy when we go back to the real system. Hence, if the optimal policy $\pi _\star$ is a fixed point of our procedure, then the sequence $\{\pi _k; k \ge 0\}$ would converge to it, thus solving our problem.

We have run this procedure, choosing for the initial policy $\pi _0$ the naive strategy of Sect. 3.3. After three iterations, the algorithm circled back to the policy we encountered on the first iteration $\pi _3 = \pi _1$. Hence, this proposed procedure does not lead to any improvement with respect to the naive policy. This is interpreted as a sign of the highly non-Markovian nature of our setting.