Robot End-Effector Mounted Camera Pose Optimization in Object Detection-Based Tasks

Abstract

Robots equipped with the vision systems at the end-effector provide a powerful combination in industrial contexts, allowing to execute a wide range of manufacturing tasks, such as inspection applications. While many works are dedicated to machine vision algorithms, the optimization of the vision system pose is not properly addressed. Optimizing the sensor pose, in fact, can increase the object detection performance, avoiding occlusions and collisions in the real working scene. Therefore, the development of an approach capable of optimizing the pose of a vision system is the main objective of this paper. A complete pipeline for such optimization is proposed, composed of the following main components: working scene reconstruction, robot-environment collisions modeling, object detection, sensor pose optimization (exploiting Bayesian Optimization, a state of the art methodology), and collision-free robot motion planning. To validate the proposed approach, experimental tests have been executed considering two object detection-based tasks. A Franka EMIKA Panda robot equipped with an Intel^© RealSense D400 at its end-effector has been employed as a robotic platform. Achieved results show the high-fidelity reconstruction of the real working environment for an offline optimization (i.e., performed simulations), as well as the capabilities of the employed Bayesian Optimization-based approach to define the sensor pose. The proposed optimization methodology has been compared with the grid point approach, showing an improved performance for camera pose optimization purposes. An additional experiment has been performed in order to show the possibility to exploit a digital twin (if available) of the working scene instead of the environment reconstruction (to reduce the computational resources and to avoid measurements noise in the 3D reconstruction). Obtained results show the feasibility of the proposed pipeline employing such a digital twin.

Appendix A: Object Detection State of the Art

Many object detection approaches are making use of specific datasets (created either by human annotation or incrementally placing one object in the scene and using foreground masking) for the detection of parts in operating environments [52,53,54]. Other approaches, instead, are making use of the CAD files of the target parts to be detected. Some of these works use synthetic datasets generated by rendering 3D CAD models of the target objects with different viewpoints, avoiding manual labeling [55]. In this case, an offline training (i.e., in simulation environments) for detection and pose estimation purposes is performed. However, many issues are still present in the proposed methods, making it difficult to transfer the trained behavior from simulation to the real task: modeling differences between the virtual training environment and the real testing scenario (i.e., the training can be not suitable for the target application), the generation of training objects’ poses that are not necessarily physically realistic (i.e., increasing the processing time without providing useful information to the algorithm), the presence of occlusions that are usually treated in a simplified manner (i.e., unrealistic scenes resulting in possible failures when moving to the real task) [56, 57]. The possibility to (partially) overcome such issues by exploiting an autonomous process for training a Convolutional Neural Network for object detection and pose estimation has been proposed in [58]. By employing a physics engine to generate synthetic but physically realistic images, the proposed approach makes use of multiple views to perform the object detection and its pose estimation. Other CAD-based approaches make use of the real data acquired from the operating scene to perform the object detection and its pose estimation [59], using both the 2D images [60] or the RGB-D data [61]. In such a context, feature-based methods [62], in which the object detection is based on the use of 3D data, are some of the most popular solutions adopted in many robotic applications [41], that can be divided in two main groups: local feature-based [63] and global feature-based [64] methods. Local features-based approaches are based on matching the descriptors of local surface characteristics, including three main stages [65]: 3D keypoints detection, local surface feature description, and surface matching. The first phase is the most important one, in which a set of points are labelled as keypoints as a function of the exploited detection method (e.g., surface sparse sampling, mesh decimation, fixed-scale, adaptive-scale, etc. [66, 67]). These points will be the ones on which object detection will be based. Once a keypoint has been detected, geometric information of the local surface around the keypoint can be extracted and encoded into a feature descriptor. According to the approaches employed to construct the features descriptors, it is possible to classify the existing methods into three broad categories [68]: signature based, histogram based, and transform based methods. Finally, the surface matching step establishes a set of feature correspondences between the operating scene and the target model, by matching the scene features against the model features. A comprehensive survey of these existing methods is proposed in [69]. Global feature-based methods, instead, follow a different pipeline for which the whole object surface is described by a single or small set of descriptors. Global point cloud descriptor is described extensively in [70]. Local features-based techniques are more robust considering cluttered environments and partial occlusions, that are frequently present in the real-world applications. Global features-based methods are instead more suitable for model retrieval and 3D shape classification, especially considering the weak geometric structures. Alhamzi et al. [71] describes an approach based on the exploitation of both local features and global features techniques, based on the PCL library [72]. Other approaches have been developed for CAD-based object detection and pose identification purposes. Template matching techniques have been proposed exploiting RGB-D data as in [73], in which a method based on quantized surface normal as depth cue is proposed. In a similar way, recently, [74] applied the concept of multimodal matching of [73] on an efficient cascade-style evaluation strategy. Even techniques based on supervised machine learning have been used for object detection and pose estimation exploiting RGB-D data. In [75] a review of classification techniques used in supervised machine learning is described, explaining how the goal of supervised learning is to build a concise model of the distribution of class labels in terms of predictor features and the different classification techniques. One of the methods to perform the pose estimation is represented by a deep learning approach to category-level 3D object pose tracking on RGB-D data with the use of key points [76]. This algorithm tracks novel object instances of known object categories such as bowls, laptops, and mugs in real time. It learns to compactly represent an object by a handful of 3D key points, based on which the inter frame motion of an object instance can be estimated through key point matching. Another algorithm for real-time 6 DOF pose estimation and tracking of rigid 3D objects uses a monocular RGB camera [77]. The key idea is to derive a region-based cost function using temporally consistent local color histograms. While such region-based cost functions are commonly optimized using first-order gradient descent techniques, in this paper a Gauss-Newton optimization scheme is proposed, which gives rise to drastically faster convergence and highly accurate and robust tracking performance. In numerous preliminary experiments performed by the authors [48, 49], it has been demonstrated that the proposed Gauss-Newton approach outperforms existing approaches in the presence of cluttered backgrounds, heterogeneous objects and partial occlusions. HybridPose [78], instead, leverages on multiple intermediate representations to express the geometric information in the input image for pose estimation. In addition to key points, this type of algorithm integrates a prediction network that outputs edge vectors between adjacent key points. As most objects possess a partial reflection symmetry, HybridPose also utilizes predicted dense pixel-wise correspondences that reflect the underlying symmetric relations between pixels. Another work demonstrated that Neural Networks coupled with a local voting-based approach can be used to perform reliable 3D object detection and pose estimation in a cluttered environment showing occlusions [79].