Multi-objective deep reinforcement learning for crowd-aware robot navigation with dynamic human preference

Abstract

The growing development of autonomous systems is driving the application of mobile robots in crowded environments. These scenarios often require robots to satisfy multiple conflicting objectives with different relative preferences, such as work efficiency, safety, and smoothness, which inherently cause robots’ poor exploration in seeking policies optimizing several performance criteria. In this paper, we propose a multi-objective deep reinforcement learning framework for crowd-aware robot navigation problems to learn policies over multiple competing objectives whose relative importance preference is dynamic to the robot. First, a two-stream structure is introduced to separately extract the spatial and temporal features of pedestrian motion characteristics. Second, to learn navigation policies for each possible preference, a multi-objective deep reinforcement learning method is proposed to maximize a weighted-sum scalarization of different objective functions. We consider path planning and path tracking tasks, which focus on conflicting objectives of collision avoidance, target reaching, and path following. Experimental results demonstrate that our method can effectively navigate through crowds in simulated environments while satisfying different task requirements.

Appendix: Social force model

In the experiments, we use Social Force Model (SFM) to simulate the motion dynamics of pedestrians with human-robot interaction (HRI), which is implemented based on existing literature [43,44,45,46]. In the simulated SFM, the factors that influence the pedestrian’s motion include their internal motivations and the external forces exerted on them. The bold variables represent vectors. The specific motion dynamics of pedestrian i can be expressed as

$$\begin{aligned} m_i\frac{dv_i}{dt}={\textbf{f}}_i(t), \end{aligned}$$

(15)

where $m_i$ is the mass of pedestrian, $v_i$ is the current velocity, and $f_i$ is the acceleration force of pedestrian. $f_i$ is expressed as

$$\begin{aligned} {\textbf{f}}_i(t) = {\textbf{f}}_{id}(t)+\sum _{j\ne i}{\textbf{f}}_{ij}(t)+\sum _{w}{\textbf{f}}_{iw}(t)+{\textbf{f}}_{ir}(t), \end{aligned}$$

(16)

where ${\textbf{f}}_{id}$ denotes the self-driving force, ${\textbf{f}}_{ij}$, ${\textbf{f}}_{iw}$, and ${\textbf{f}}_{ir}$ are the external forces exerted on the pedestrian from other pedestrians, the walls, and the robot, respectively. The details are expressed as follows:

• 1. The self-driving force: This force depends on the pedestrian’s internal motivation, which reflects the intention of adjusting his/her direction and velocity to arrive at the destination. Suppose the desired direction is denoted as ${\textbf{e}}_i$, which points from the current position to the destination, and the desired velocity is $v_i^0$. Then, the self-driving force is given by [43]

  $$\begin{aligned} {\textbf{f}}_{id}(t)=\frac{m_i}{\tau }(v_i^0{\textbf{e}}_i-{\textbf{v}}_i), \end{aligned}$$

  (17)

  where $\tau$ denotes the relaxation time during which the discrepancy between the desired velocity and the current velocity.

• 2. The social force exerted by other pedestrians: This force stems from the repulsive force among pedestrians, which represents their desire to keep a safe distance from nearby humans and obtain more space in crowded environments. It is expressed as [43]

  $$\begin{aligned} {\textbf{f}}_{ij}(t)=F\Theta _{ij}\rm{exp}[-d_{ij}/D_0+(D_1/d_{ij})^k]{\textbf{e}}_{ij}, \end{aligned}$$

  (18)

  where F denotes the maximum repulsive force, $d_{ij}$ is the distance between pedestrian i and pedestrian j, and ${\textbf{e}}_{ij}$ is the normalized vector pointing from pedestrian i to pedestrian j. $D_0$, $D_1$, and k are related constant parameters. $\Theta _{ij}$ reflects the anisotropic character of the repulsive force due to the limited field of each pedestrian and is expressed as

  $$\begin{aligned} \Theta _{ij} = \lambda _i+(1-\lambda _i)\frac{1+\rm{cos}(\phi _{ij})}{2}, \end{aligned}$$

  (19)

  where $\phi _{ij}$ is a constant parameter and with $\phi _{ij}<1$, we can model the situation that pedestrians react much stronger to things happening before than behind. $\phi _{ij}$ is the angle between the desired direction ${\textbf{e}}_i$ and the relative angle ${\textbf{e}}_{ij}$.

• 3. The social force exerted by walls: This force reflects that pedestrians want to keep a safe distance from walls in crowded places. This repulsive force is expressed as [44]

  $$\begin{aligned} {\textbf{f}}_{iw}(t) = A_{iw}[(r_i-d_{iw})/B_{iw}]{\textbf{n}}_{iw}, \end{aligned}$$

  (20)

  where $d_{iw}$ denotes the nearest distance between the pedestrian and the wall, and ${\textbf{n}}_{iw}$ is the direction pointing from the position of the pedestrian to the nearest point of the wall. $A_{iw}$ and $B_{iw}$ denote the strength and the range of the respective interaction force. $r_i$ is the radius of the pedestrian.

• 4. The interaction force exerted by robot: This force reflects the human-robot interaction force which is modeled from the perspective of social force. It is expressed as [45]

  $$\begin{aligned} {\textbf{f}}_{ir}(t) = A_{ir}[(r_{ir}-d_{ir})/B_{ir}]{\textbf{n}}_{ir}\Theta _{ir}, \end{aligned}$$

  (21)

  where $d_{ir}$ is the distance between the pedestrian and the robot, and ${\textbf{n}}_{ir}$ is the vector pointing from the pedestrian to the robot. $A_{ir}$ and $B_{ir}$ denote the strength and the range of the respective human-robot interaction force. $r_{ir}$ is the sum of the pedestrian radius $r_i$ and the robot radius $r_r$.

The parameters of our implemented model are shown in Table 6, which are chosen based on [46]. ${\mathcal {N}}(\mu ,\sigma ^2)$ denotes the Gaussian distribution with mean $\mu$ and standard derivation $\sigma$.

Table 6 List of social force model parameters