Taming Lagrangian chaos with multi-objective reinforcement learning

Abstract

We consider the problem of two active particles in 2D complex flows with the multi-objective goals of minimizing both the dispersion rate and the control activation cost of the pair. We approach the problem by means of multi-objective reinforcement learning (MORL), combining scalarization techniques together with a Q-learning algorithm, for Lagrangian drifters that have variable swimming velocity. We show that MORL is able to find a set of trade-off solutions forming an optimal Pareto frontier. As a benchmark, we show that a set of heuristic strategies are dominated by the MORL solutions. We consider the situation in which the agents cannot update their control variables continuously, but only after a discrete (decision) time, $\tau$. We show that there is a range of decision times, in between the Lyapunov time and the continuous updating limit, where reinforcement learning finds strategies that significantly improve over heuristics. In particular, we discuss how large decision times require enhanced knowledge of the flow, whereas for smaller $\tau$ all a priori heuristic strategies become Pareto optimal.

Graphic abstract

Appendix A: Q-learning implementation

To solve the optimization problem, we used the Q-learning algorithm [35] which is based on evaluating the action-value function, Q(s, a), that is the expected future cumulative reward given that the agents are in state s and take action a. The algorithm is expected to converge to the optimal policy by the following iterative trial-and-error protocol. At each decision time $t_j$, the agents pair measures its state $s_{t_j}$ and selects an action $a_{t_j}$ using an $\epsilon$-greedy strategy, where $a_{t_j}(s_{t_j})=\arg \max _a\{Q(s_{t_j},a)\}$ with probability $1-\epsilon$ or $a_{t_j}$ is chosen randomly with probability $\epsilon$. Then, we let the dynamical system evolve for a time $\tau$, according to (1), keeping both control directions and velocity intensity fixed. Afterward, the agents receive a reward $r_\textrm{tot}(t_{j+1})$ (11) and the Q-matrix is updated as

$$\begin{aligned} \begin{aligned} Q(s_{t_j},a_{t_j}) \leftarrow \,&Q(s_{t_j},a_{t_j}) + \alpha [r_\textrm{tot}(t_{j+1}) + \\ {}&+\max _{a} Q(s_{t_{j+1}},a)-Q(s_{t_j},a_{t_j})], \end{aligned} \end{aligned}$$

(17)

where $\alpha$ is the learning rate. Updates are repeated up to the end of the episode $t=T_{\max }$, when no reward is assigned. The learning protocol is repeated restarting with another pair with the same initial distance in another flow position until we reach a “local” optimum given by the equation $Q^*(s_{t_j},a)= r_\textrm{tot}(t_{j+1}) +\max _{a} Q^*(s_{t_{j+1}},a)$ and defined by the policy

$$\begin{aligned} a(s)=\arg \max _a\{Q^*(s,a)\}. \end{aligned}$$

In order to ease the convergence of the algorithm, the learning rate $\alpha$ is taken as a decreasing functions of the time spent in the state-action pair, while the exploration parameters decrease with the time spent in the visited state. Thus if n(s, a) is the number of decision times in which the couple (s, a) has been visited, and

$$\begin{aligned} n(s)=\sum _a n(s,a)/|{\mathcal {A}}|, \end{aligned}$$

(18)

$\epsilon$ and $\alpha$ are taken as:

$$\begin{aligned} \alpha= & {} 5/[200^{1/\gamma }+\tau n(s,a)]^{\gamma } \end{aligned}$$

(19)

$$\begin{aligned} \epsilon= & {} 5/[200^{1/\gamma }+\tau n(s)]^{\gamma } \end{aligned}$$

(20)

with $\gamma =4/5$, the numerical values of the constants have been determined after some preliminary tests. As for the initialization of the matrix Q, we have taken the same large (optimistic) value for all the state-action pairs.