Visual SLAM for Mobile Robot

Abstract

During the early developmental stage of visual SLAM technology, other sensor information was not considered, and only the camera was used. According to different tracking algorithms, VSLAM can be classified into direct and feature (or indirect) methods. If based on optimization algorithms, visual SLAM can be classified into filter-based optimization (e.g., MonoSLAM) and graph-based optimization (e.g., parallel tracking and mapping (PTAM)). When the image features of the pure vision-based SLAM are lost, the accuracy and robustness of the pose estimation decrease rapidly, and the algorithm may fail. Therefore, during subsequent development, visual SLAM techniques based on the fusion of multiple sensors, such as vision, IMU, and LiDAR, have emerged.

Appendices

Further Reading

• 1. ORB-SLAM3

  A typical visual SLAM system mainly includes data processing, initialization, vision odometer, map maintenance, closed-loop detection, and other parts. Compared with the previous two generations, ORB-SLAM3¹ mainly adds inertial sensors on the basis of monocular, stereo vision and RBG-D and introduces vision-inertia and multi-map mode (Atlas) and map merging. It is a full multi-map and multi-session system able to work in pure vision or visual-inertial modes with monocular, stereo, or RGB-D sensors, using pin-hole and fisheye camera models. As shown in Fig 10.31, there have four main modules including Tracking thread, Loop and map merging thread, Local mapping thread, and Atlas.

  Tracking thread processes sensor information and computes the pose of the current frame with respect to the active map in real-time, minimizing the re-projection error of the matched map features. It also decides whether the current frame becomes a key frame. In visual-inertial mode, the body velocity and IMU biases are estimated by optimization including the inertial residuals. When tracking is lost, the tracking thread tries to re-localize the current frame in all the Atlas’ maps. If re-localized, tracking is resumed, switching the active map if needed. Otherwise, after a certain time, the active map is stored as nonactive, and a new active map is initialized.

  Loop and map merging thread detects common regions between the active map and the whole Atlas at key frame rate. If the common area belongs to the active map, it performs loop correction; if it belongs to a different map, both maps are seamlessly merged into a single one, which becomes the active map. After a loop correction, a full BA is launched in an independent thread to further refine the map without affecting real-time performance.

  Local mapping thread adds key frames and points to the active map, removes the redundant ones, and refines the map using visual or visual-inertial bundle adjustment, operating in a local window of key frames close to the current frame. Additionally, in the inertial case, the IMU parameters are initialized and refined by the mapping thread using novel MAP-estimation technique.

  Atlas is a multi-map representation composed of a set of disconnected maps. There is an active map where the tracking thread localizes the incoming frames, and is continuously optimized and grown with new key frames by the local mapping thread. The other maps in the Atlas are the nonactive maps. The system builds a unique DBoW2 database of key frames that is used for re-localization, loop closing, and map merging.

• 2. LVI-SAM

  LVI-SAM² is built atop a factor graph and is composed of two subsystems: a visual-inertial system (VIS) and a lidar-inertial system (LIS). The two subsystems are designed in a tightly coupled manner, in which the VIS leverages LIS estimation to facilitate initialization. The accuracy of the VIS is improved by extracting depth information for visual features using LiDAR measurements. In turn, the LIS utilizes VIS estimation for initial guesses to support scan-matching. Loop closures are first identified by the VIS and further refined by the LIS. LVI-SAM can also be effective when one of the two subsystems fails, which increases robustness in both texture-less and feature-less environments.

  As shown in Fig 10.32, the VIS and LIS can function independently while using information from each other to increase system accuracy and robustness. The system outputs pose estimates at the IMU rate. The VIS processes images and IMU measurements, with LiDAR measurements being optional. Visual Odometry is obtained by minimizing the joint residuals of visual and IMU measurements. The LIS extracts LiDAR features and performs LiDAR Odometry by matching the extracted features with a feature map. The feature map is maintained in a sliding-window manner for real-time performance. Lastly, by jointly optimizing the contributions of IMU preintegration constraints, visual odometry constraints, LiDAR odometry constraints, and loop closure constraints in a factor graph using iSAM2, the state estimation problem is solved, which can be formulated as a maximum a posteriori (MAP) problem. Note that the multi-sensor graph optimization employed in the LIS is intended to reduce data exchange and improve system efficiency.

Fig. 10.31

Main system components of ORB-SLAM3

Fig. 10.32

System structure of LVI-SAM

Exercises

• 1. What are the optimization components used in the ORB-SLAM2 algorithm process?

• 2. Are there any other optimization methods commonly used for back end optimization? Describe these methods.

• 3. How are key frames selected?

• 4. Learn about the DBoW2 library and try to find a few images to test if loop closures can be detected correctly. What other methods can be used for loop closure detection except for the bag-of-words model?

• 5. How many representations of maps for visual SLAM are there? What application scenarios are they each suitable for?

• 6. What other methods are available for solving for camera motion besides epipolar geometry and PnP? Describe these methods.

• 7. In the visual SLAM algorithm, the pose change is calculated according to the matching feature points of adjacent frames. Assume that a point coordinate is (x, y, z), and rotate a, b, and c degrees according to the x-axis, y-axis, and z-axis, respectively. Please derive the corresponding rotation matrix R.

• 8. There are many representations of rotation, including matrix representation, axis–angle representation, and quaternion representation. It is very important to understand the conversion relationship between different representations. Please derive Rodriguez formula, which shows how to express the rotation of axis–angle as a rotation matrix.

• 9. When using direct linear transformation to solve PnP, if there are too many points, the coefficient matrix A in equation Ax = b will be overdetermined. Please prove that when A is overdetermined, the least square solution of Ax = b is x = (A^TA)^-1A^Tb.

• 10. Briefly describe the steps of monocular dense mapping and RGB-D dense mapping. Explain their differences and connections.

• 11. Briefly describe the application scenarios of no less than three common visual SLAM methods. Compare their advantages and disadvantages.

• 12. The figure below contains the main functions of a typical mobile robot system in operation: pose estimation, environmental mapping, navigation, and obstacle avoidance. Analyze the individual ROS nodes shown in the figure below.