Localization and grasping of small load carriers with autonomous unmanned aerial vehicles

. The application of unmanned aerial vehicles (UAV) in the area of inspection, survey or urban logistics has become a rapidly developing research domain. While the feasibility of material transports with UAVs has already been shown in the scope of di(cid:11)erent projects, the payload is thereby usually transferred manually into the UAV’s load handling device. A decisive factor for the economic usability of UAVs for aerial transportation, however, is a fully automated system including the autonomous recognition and pick-up of the cargo. We therefore present a solution for the automated detection, localization and grasping of small load carriers with UAVs. The system includes a specialized load handing device, a camera-based real-time tracking solution for small load carriers and a fusion of the global and relative position measurements to achieve the in-(cid:13)ight positioning accuracy required for the autonomous cargo pick-up.


Introduction and related work
In recent years, UAVs have found their way into various industrial applications. In addition to tasks in the area of inspection or surveying, there are also various use cases in the field of material transport, as UAVs enable fast, highly flexible and direct material transport in the air. In addition to the transport of parcels in retail or the delivery of urgent medical products or devices, UAVs are also suitable for the transport of spare parts or urgently needed parts on the company premises or within a production hall. Due to their three-dimensional scope of movement UAVs are able to operate in the unused space above the production systems, extend the flow of materials by the third dimension and offer a highly flexible and fast transportation. [1] While different publications have already shown that autonomous UAVs can provide an efficient addition to existing logistic systems there are still only few solutions or publications describing fully functional systems for the automated aerial in-house transportation [2]. Existing research and pilot projects on the aerial transportation with autonomous UAVs commonly do not focus on the autonomous cargo pick-up. [3] [4] To enable autonomous aerial gripping of objects, different operating principles including suction-gripping, interlocking gripping or magnetic gripping have been developed during the recent years. Among the most common design are mechanic finger or jaw grippers as they are easy to produce, robust and can handle objects independently of their material and shape. However, they require a precise positioning of the UAV or an underactuated and flexible design to be able to grasp objects properly.
To compensate the displacement between the UAV and the target object caused by inaccuracies during manual flight Pounds et al. utilize an underactuated four finger gripper for the reliable in-flight grasping of different objects. Each finger features two elastic joints to compensate the contact forces between gripper and object, resulting from the positional fluctuation of the UAV. [5] [6] Optical tracking systems such as motion capturing systems allow a precise tracking of UAVs with an accuracy below 1 mm. The usage of such systems allows the application of fully actuated grippers in combination with autonomous drones. Qi et. al. utilize a basic single-DOF finger gripper for the successful aerial gripping of a cuboid [7].
The usage of interlocking connections for aerial grasping has been shown in [8] and [9]. Thereby pins or clamps are actuated by servo motors to penetrate the cargo objects and to attain a secure fastening of the object.
While the automated cargo pick-up is considered a key requirement for the cost-efficient in-house aerial transportation, the state of the art lacks of working solutions in this area. We thus present a system architecture for the autonomous detection and grasping of small load carriers with autonomous UAVs. The system includes an autonomous hexarotor system equipped with a camera system to detect and locate the cargo objects and an appropriate load handling device to realize an autonomous grasping.

System architecture
To ensure an economic usage of the transportation of small load carriers with autonomous UAVs both the cargo pick-up and the drop-off need to be fully automated. Therefore, we want to achieve the following transportation scenario: The UAV receives a transportation task including an approximate position of the load carrier (e.g. the position of the stockyard) and the drop point. After the approach to the transmitted pick-up position, the UAV detects and locates the load carrier and positions itself precisely above the object to allow a reliable pick-up. After the pick-up using an automated load-handling device the cargo is transported to the transmitted destination. Subsequent to the aerial transport the load carrier is either precisely set down on a landing area defined by optical markers or simply at the transmitted drop-off coordinates.
As we focus on the indoor transportation we use an ultra-wide band (UWB) tracking system for global positioning. The system allows a tracking with 10 cm accuracy and can cover areas of several ten-thousand square meters. When flying in outdoor areas this localization system can be easily replaced with tracking data provided by satellite-based radio-navigation systems without modifying the remaining system components.

Mechatronic cargo handling device
To ensure an automated pick-up and drop-off a mechatronic load handling device is developed and attached to a modified version of a commercially available hexarotor system. The load handling device is designed to be set down vertically on the load carrier. To compensate inaccuracies and fluctuations in the positioning of the UAV and to center the load carrier during the landing phase the system provides four lead-in chamfers (cf. Figure 4).
To fasten the load carrier after the landing, retaining bolts are used to create an interlocking connection with the indentations on the outer contour (1). As depicted in Figure 1, the individual bolts (2) are guided by linear bearings (3) and actuated by servo motors (4). A servo lever with a slotted hole (5) is used to convert the rotary motion into a linear one. As the servos are only actuated while the UAV is resting on the load carrier, no additional force or momentum other than the friction force needs to be overcome to move the bolts. During the flight phases, the weight force of the load carrier presses the bolts onto the bearing thus creating an efficient load restraint additional to the counter-momentum of the motor.

Object localization and tracking
Machine learning approaches are often considered as state of the art for object segmentation tasks. There are various neural networks available for object segmentation such as Mask R-CNN, DeepLab or FuseNet that allow an instance segmentation based on colour or combined colour and depth information [10][11][12] as well as advanced pose estimation pipelines for segmenting and grasping various objects [13]. Light-weight models and neural network architectures also allow a near real-time image analysis low-cost hardware such as the NVidia Jetson Nano Board or different single board computers equipped with the Intel Neural Compute Stick.
However, as machine learning approaches still require large amounts of suitable and annotated training data especially when the load carriers is filled with different objects, we favour a solution based on color and shape segmentation.
Our approach for the detection, segmentation and localization of the individual load carriers is based on their known colour (blue) and shape (rectangular with known size). The detailed algorithm to segment and locate the objects is presented in detail below.
To extract the load carrier from the image, in a first step a basic colour segmentation algorithm is used. Based on a tolerance range centred on the actual colour value of the load carrier all blue pixels are segmented and converted into a binary mask. As depicted in the left image in Figure 2 the resulting mask contains most of the load carrier's pixels but is still affected by noise and falsely identified areas.
Afterwards Gaussian smoothing is applied to the binary image to reduce the effect of the remaining noise that can especially occur at the edges of the segmented objects. The Gaussian filtering prevents the incorrect detection of the noise as the edge of an object during the subsequent edge detection using the Canny-algorithm (cf. Figure 2, centre).
By applying topological structural analysis the contours of the remaining objects can be extracted. As only rectangular shapes are required, we apply the Douglas-Peucker-algorithm to approximate the remaining contours with the minimal required number of points. As an rectangle is fully described by its four corner points all contours with more or less points are discarded. To extract the rectangles actually representing a load carrier their enclosed area is analysed. As the intrinsic camera parameters, the size of the load carrier and the current flight altitude is known, the expected size of the load carrier in the camera image can be estimated and all rectangles that do not fulfill this criterion can be discarded.
Next, the scalar product of the vectors defining the rectangle's borders is calculated to get an estimate of the angle between them. This is used to only store rectangular shaped contours with corner angles of approximately 90 degrees. Based on this information, it can be ensured that e.g. for trapezoidal shapes, which also consist of only four corner points, the related contours are no longer stored for further processing.
Finally, the contours are sorted by their enclosed area and only the contour with the biggest area is kept. This is to avoid calculating the position of smaller rectangles within the box resulting from the possible influence of shading or overexposure (cf. Figure 2, right). After the segmentation the transform between the camera and the computed centroid of the load carrier is determined using the camera's inverted intrinsic matrix.

Pose estimation and software architecture
As the optical tracking system features a significantly lower measurement noise than the UWB tracking system, the provided transform is not only used as setpoint but also used to increase the in-flight positioning accuracy of the UAV. This is necessary as the positioning accuracy that can be achieved by the data of the UWB tracking system and the UAV's intertial measurement unit is oftentimes not sufficient to pick up the load carriers. To improve the estimation of the current flight position we apply the covariance intersection algorithm presented by Niehsen [14].
Thereby the relative pose between the UAV and the load carrier is transformed into global coordinates and fused with the pose provided by the UWB tracking system. The inverse transformation is based on the UAVs global position at the time of the initial localization of the load carrier.
Using the covariance matrix K xx of the global UWB tracking data and the covariance matrix K yy of the pose provided by the optical tracking a combined covariance matrix K zz can be calculated as To find the optimal solution for K zz the value of w is computed in such a way that the determinant of K zz is minimized. Using the available closed-loop formulations a fused position estimation is calculated and forwarded to the internal kalman filter of the flight control unit (FCU). Thereby the position values provided by the UWB tracking system and the box detection are denoted as P G and P C .
The implemented process for cargo pick-up, transportation and drop-off is depicted in Figure 3. Initially the overlying task management sends an approximate position setpoint to the FCU, indicating where the cargo object is located (not shown in the chart). As soon as the UAV reaches the provided position the process for locating and grasping the load carrier becomes active.
Based on the provided RGB-data the load carrier is segmented and localized using the algorithm presented in section 2.2. The resulting transform and the known transforms between the camera and the FCU and the load handling device and the FCU are then used to calculate an updated position setpoint that is forwarded to the FCU.
Using the covariance intersection algorithm an improved position estimation is calculated whenever valid relative tracking data is available and forwarded to the internal kalman filter of the FCU to improve the in-flight positioning accuracy.
The internal land detector of the FCU provides information whether the UAV is already landed or still in-flight. Based on this feedback, the load handling device controller drives the individual servo motors to fasten or release the cargo.
After a successful pick-up (which is determined by the increment in the required take-off thrust) the UAV heads for the provided drop-off position. When the drop-off position is referenced by additional optical markers, they are tracked using the algorithms presented in [15] and the resulting relative position is used as input to the covariance intersection algorithm, otherwise the positioning during the landing phase only relies on the UWB tracking data.

Evaluation
The resulting system is depicted in Figure 4. The developed load-handling device is attached to a DJI F550 airframe with the corresponding electronic speed controllers and motors. As FCU we use the open-hardware system Pixracer running the 1.9.2 stable release of the PX4 autopilot [16] and a NVidia Jetson Nano on-board companion computer (OBC) for object detection and data fusion.
The implemented software utilizes the Robot Operating System (ROS) and the provided ROS communication layer. To allow a precise tracking of both While the FCU runs the real-time modules such as attitude and position controller the OBC runs additional software modules required for general flight processes and navigation. Core element of our previously developed software framework is the mission controller, a software module that is responsible for basic navigation tasks, path-following, system diagnosis and failure detection. Figure 5 compares the achieved in-flight accuracy based on the pose estimation of the internal kalman filter when hovering around 1 m above the load carrier. To simplify the evaluation and depiction the corresponding static setpoint is subtracted from the position values. All flights are conducted with the same closed-loop controller but it's effect on the positioning accuracy and potential improvements will not be discussed in detail. When only the UWB tracking information is used, the resulting in-flight position deviates around ±5 cm from the setpoint in x-and y-direction. However, the deviations in z-direction reach values of up to 20 cm. This can be traced back to the fact that the noise in the measurement values of the UWB tracking system in z-direction is significantly higher then the noise in x-and y-direction.
When fusing additional optical tracking information, the deviations decrease to maximal values of ±4 cm in every direction. Compared to the UWB-only dataset, especially the accuracy in z-direction improves significantly enabling a consistent and precise descending. Compared to the z-direction, the positioning accuracy in x-and y-direction increases less which can be traced back to inferior measurement inaccuracies of the UWB tracking system.
While the optical tracking of the load carrier improves the flight accuracy and improves the success rate of the cargo pick-up further improvements are required. The lead-in chamfers of the current load handling device can compensate movement of up to ±5 cm, however, the automated pick-up currently only succeeds approximately every second time (evaluated on the basis of 20 test flights). Besides the in-flight position oscillations also faulty tracking information, friction on the lead-in chamfers and oscillations caused by the ground effect are accountable for the poor success rate.

Conclusion and Outlook
Within this paper we have presented an approach to the autonomous aerial grasping of small load carriers with UAVs. Using a camera-based solution for segmenting and locating the cargo objects, a newly developed load-handling device and the covariance intersection algorithm for optimized pose estimation,

Positioning accuracy (covariance intersection)
x y z the automated pick-up of small load carriers was realized. While it was shown that the autonomous pick-up of small load carriers is possible with the present system, the reliability and rate of successful pick-ups still needs to be improved. To optimize the flight and grasping performance within future research, the influence of optimized and model-based controllers will be examined. Also improved cameras with optimized field of views and additional sensors to measure the current movement speed will be integrated and the influence of vibrations caused by the spinning rotors will be examined.
Open Access This chapter is licensed under the terms of the Creative Commons Attripermits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.
The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.