RDDRL: a recurrent deduction deep reinforcement learning model for multimodal vision-robot navigation

Abstract

Existing deep reinforcement learning-based mobile robot navigation relies largely on single-modal visual perception to perform local-scale navigation. However, multimodal visual fusion-based global navigation is still under technical exploration. Visual navigation necessitates that agents drive safely in structured, changing, and even unpredictable environments; otherwise, inappropriate operations may result in mission failure and even irreversible damage to life and property. We propose a recurrent deduction deep learning model (RDDRL) for multimodal vision-robot navigation to address these issues. We incorporate a recurrent reasoning mechanism (RRM) into the reinforcement learning model, which allows the agent to store memory, predict the future, and aid in policy learning. Specifically, the RRM first stores current observations and states by learning a parameterized environment model and then predicts future transitions. The RRM then performs a self-assessment on the predicted behavior and perceives the consequences of the current policy, producing a more reliable decision-making process. Furthermore, to obtain global-scale behavioral decision-making, information from scene recognition, semantic segmentation, and pose estimation are fused and used as partial observations of the RDDRL. A large number of simulated experiments based on CARLA scenarios, as well as test results in real-world scenarios, show that RDDRL outperforms state-of-the-art RL methods in terms of driving stability and safety. The results show that by training the agent, the collision rate in the global decision-making of the unmanned vehicle decreases from 0.2 % in the training state to 0.0 % in the test state.

Appendices

Appendix A. Place recognition model

In the appendix, we introduce the establishment process of the environment perception model in detail, which includes the place recognition model, the scene segmentation model, and the pose estimation model. To improve the overall operational efficiency of the proposed RDDRL model, we pursue the principle of lightweight in the process of designing the environment model. Therefore, the designed place recognition model, scene segmentation model, and pose estimation model are all lightweight models. Fig. 17 depicts the place recognition model, which consists of five convolutional modules with a total of 13 convolutional layers.

Fig. 17

Place recognition model

The 1 to 5 convolutional modules use 64, 128, 256, 512, and 512 filters, respectively. To extract salient features, all pooling layers use maxpooling, with a kernel size of $2 \times 2$ and a stride size of $2 \times 2$. The detailed structure description is shown in Table 7.

Table 7 The structure of place recognition model

To speed up scene recognition, the method of feature aggregation is used to aggregate salient features in a certain region into salient regions, which is expressed as follows:

$$\begin{aligned} \left\{ \begin{array}{l} f_{\Omega }=\left[ f_{\Omega , 1}, f_{\Omega , 2}, \cdots , f_{\Omega , K}\right] ^{T} \\ f_{\Omega , i}=\sum _{p \in \Omega } X_{i} \end{array}\right. \end{aligned}$$

(A1)

where $X_{i}$ is the two-dimensional tensor that represents the activated responses of the $i^th$ convolutional layer. $f_{\Omega , i}$ is the $i^{th}$ valid spatial position that represents the salient region in an image. Then, these salient regions are weighted to obtain global scene representation:

$$\begin{aligned} L(\textbf{I})=\sum _{i=0}^{N} f_{\Omega , i} \end{aligned}$$

(A2)

For the query scene $\textbf{I}^q$and the reference scene $\textbf{I}^r$, the global scene representation obtained after convolutional encoding is $L^q(\textbf{I})$ and $L^r(\textbf{I})$, respectively. The cosine distance function is used to calculate the vector distance between the two to judge the place similarity, which is expressed as Eq. 1.

We carried out place recognition experiments to verify the effectiveness of the proposed model by comparing it with similar methods on four challenging datasets. The experimental results are shown in Fig. 18, which also was described in detail in our published papers [44].

Fig. 18

Experimental results for place recognition on four challenging datasets

Appendix B. Scene segmentation model

We design a lightweight scene segmentation model that adopts a full-bottleneck structure, as shown in Fig. 19. To speed up the operation, atrous convolutions are embedded in the bottleneck structure. The whole architecture is shown in Table 8.

We use I to represent the matrix representation of the image, and the following results are obtained after feature encoding (encoder):

$$\begin{aligned} T_m=s(I*F_{m}^{w}+b_m) \end{aligned}$$

(B3)

where s is the RRelu function, b is the bias vectors for the $k^{th}$ feature map, $F_{m}^{w}$ is the convolutional filters. The decoder will also output m intermediate features by the previous down-sample $T_m$. The convolutions from block 3 will cope with the encoding features, and output the results:

$$\begin{aligned} G_{m}=s\left( T_{m} * F_{m}^{w^{\prime }}+b_{m}^{\prime }\right) \end{aligned}$$

(B4)

Where $F_{m}^{w^{\prime }}$ is the convolutional filters used in the current stage, $b_{m}^{\prime }$ is the bias vectors in the current stage. Then, the decoder will cope with the output of the previous stage:

$$\begin{aligned} D_{m}=s\left( G_{m} * F_{m}^{w^{\prime }}+b_{m}^{\prime \prime }\right) \end{aligned}$$

(B5)

Where $F_{m}^{w^{\prime }}$ is the convolutional filters used in the decoding stage, $b_{m}^{\prime \prime }$ is the bias vectors in the decoding stage. Scene segmentation tries to produce labels for each pixel based on different categories appearing in a scene. In our work, we use the Softmax layer to as the object classifier to detect all objects, which is expressed as follows:

$$\begin{aligned} B_{m}^{*}={\text {softmax}}\left\{ D_{1}, D_{2}, \ldots , D_{m}\right\} \end{aligned}$$

(B6)

Where m is the number of categories.

Fig. 19

Scene segmentation model

Table 8 The structure of scene segmentation model

We carried out semantic segmentation experiments to verify the effectiveness of the proposed model by comparing it with similar methods. To speed up model reasoning, we construct a lightweight semantic segmentation model. The results show that The proposed model is second only to LedNet in terms of performance, but the reasoning speed of our model is much higher than that of LedNet.

Table 9 Experimental results for semantic segmentation on Cityscapes dataset

Appendix C. Pose estimation model

To estimate the robot’s pose, we design a pose estimator based on multi-sensor fusion. The pose estimator first uses a visual encoder and an imu encoder to encode images and inertial measurements, respectively and then uses LSTM toconstruct a pose estimator. The vector representation of the pose is output by fusing the data collected by the two sensors, and the fully connected layer network is then used to regress the robot’s pose. The whole structure of pose estimation is shown in Fig. 20.

Fig. 20

Pose estimation model

We first employ a CNN with five convolutional layers and LSTM to encode the image and imu features respectively, which is expressed as in Eqs. 5 and 6. We consider translation and rotation as motion on the manifolds and decompose the camera’s motion into a rotation motion and a translation motion in the vector space. Then, the rotation motion and translation motion are translated into a rotation matrix and a translation matrix by a special Euclidean group SE(3):

$$\begin{aligned} SE(3)=\left\{ T=\left[ \begin{array}{cc} R\left( \left( a_{v}, a_{i}\right) \right) &{} t\left( \left( a_{v}, a_{i}\right) \right) \\ 0^{T} &{} 1 \end{array}\right] \in \mathbb {R}^{4 \times 4}\right\} \end{aligned}$$

(B7)

Where R and T are the rotation and translation matrices of the camera in space motion, $\mathbb {R}^{3}$ denotes a set of 3D vector Spaces, $a_{v}$ and $a_{i}$ are the scene descriptor and imu descriptor, lstm is the fusion function. Every moment of the camera’s motion has a matching Euclidean group SE(3), which forms a trajectory flow by recording the time sequence. The pose vector is fed into three fully connected neural layers for pose regression. The fully connected neural layers regress translations and rotations as a quaternion and map the fused pose vector into a robot’s pose vector, which is expressed as Eq. B7. We carried out pose estimation experiments to verify the effectiveness of the proposed model by comparing it with similar methods on the KITTI dataset. The selected comparison method includes the handcrafted engineering method (VINS [45]) and the deep learning method (PoseNet [46] and DeepVO [47]). By comparing the average pose errors (APE) of several methods, we can see from Fig. 21 that the pose estimation method proposed by us obtained a smaller APE value, indicating a higher pose estimation accuracy. Figures C6 and C7 respectively show the Translation and pose angle changes on the six KITTI sequences. It can be seen that the experimental curves are generally consistent with the groundtruth, which also was described in detail in our published papers [48].

Fig. 21

Experimental results for pose estimation on the KITTI dataset

Fig. 22

Translation changes on the KITTI dataset

Fig. 23

Pose angle changes on the KITTI dataset