1 Introduction

Globalization and a changing consumer society are leading to ever higher demands on production and logistics. Influences such as a sharp increase in the number of product variants, shorter product life cycles and greater interlinking of different companies mean that modern production systems have to respond with ever greater flexibility. Numerous technologies exist to meet the increasing requirements. For intralogistics, e.g., Automated Guided Vehicles (AGVs) can be used, which can autonomously take over transport tasks in a production system and allow a flexible connection between individual workstations. Another emerging technology is the use of lightweight robots for human-robot collaboration. A combination of an AGV and a lightweight robot yields a Mobile Robot Manipulator.

In [1], the authors define a Mobile Robot Manipulator (MRM) as a system consisting of 3 subsystems—the mobile platform, the manipulation robot and the gripping system. As described in [1] the mobile manipulator is an autonomous and self-driving terrestrial platform, with omnidirectional wheels, a 6 degree of freedom (DOF) manipulator and a parallel gripper. The MRM has either 9 or 8 DOF, based on the type of manipulator installed. However, in our understanding, the term MRM can be defined more broadly and is not limited to specific wheels or end effectors. For clarity, our definition is included at the end of the chapter.

First publications on MRMs go back to the year 1961 when General Mills invented the so-called “Little Ranger” in the USA to solve tasks in a nuclear disaster area [2]. The authors in [3] give an overview of the advances in autonomous mobile robots development until 2019. However, the solutions presented cannot be operated in a shared workspace with humans unless complex safety measures are installed. The development of robots for human–robot collaboration through safeguarding employing force and power limitation offers the possibility to manage the mobile manipulators in a collaborative workspace with humans. Mobile manipulators can open up new potentials in a production system, such as the autonomous location-flexible use of robots or flexible assembly tasks common, for example, in the automotive industry [4].

To exploit the full potential of an MRM, simultaneous operation of manipulator and platform should be possible. At the same time, the use of MRMs in a shared workspace with a human may be necessary to provide maximum flexibility in production. The normative situation shows that the operation of mobile manipulators is not far advanced yet. For example, standards EN ISO 10218 and ISO/TS15066 exist for human–robot collaboration and standard DIN EN ISO 3691-4 for the operation of AGVs. A standard for the simultaneous operation of robots and AGVs currently exists only for the North American market in the form of ANSI RIA15.08.

Another technology that has grown significantly in recent years are unmanned aerial vehicles (UAV). The term UAV describes three different types of UAVs—Multirotor UAV, Helicopter UAV and Fixed-wing UAV. All of them come with their strengths and weaknesses regarding mobility, adaptability, availability of sensors, endurance and payload, as is stated in [5]. Multi-rotor UAVs, often Quadrocopters or Multicopters, are among the most common for industrial indoor environments. Vertical Take-Off and Landing (VTOL) advantages include high mobility, speed, and transportability.

Industrial applications of UAVs cover the fields of transport and logistics [6], workspace surveillance and security [7], inspection [8], construction [9], and agriculture [10]. The use of UAVs as variable antennas for mobile network technology is discussed [11]. While numerous outdoor uses exists, the number of indoor applications that involve UAVs is comparatively small for the current state of research in flight robotics [12]. One reason for this could be the complex localization of the UAVs. Due to signal shielding, GPS-based indoor localization of UAVs is only possible to a limited extent or even impossible, therefore alternative localization methods must be used [13]. In addition, the low carrying capacity of most UAVs may be a limiting factor in industrial material handling applications, and commercially available drones with high payloads tend to be expensive and too large [12].

The use of a MRM offers new potentials compared to the use of an AGV, in which e.g. loading and unloading operations can be flexibly handled by the manipulator. Furthermore, the manipulator can be used flexibly in assembly, for example, in order to react to fluctuating production capacities, human skills or variant diversity. The transport time of the AGV is normally time in which no value creation can be done. However, the manipulator could use the transport time to perform assembly or machining operations on (partial) products. The combination of MRM and UAV offers new application possibilities in a production system, like delivering goods to the mobile manipulator during the transit or picking up goods via a UAV from the platform when an assembled product is finished. For use in navigation or also in MRM workspace monitoring when used in human–robot collaboration, the UAV could be used as a mobile camera, for example. Another field of application would be the use of the MRM as a mobile loading station to provide optimized loading points in the production system for the UAV. Therefore, the combination of UAV and mobile robot manipulator even increases flexibility and productivity and offers new ways of intralogistics processes.

Currently, however, the combinations of MRM and UAV are not used in production systems despite their potential. A variety of technical issues have to be resolved, such as control and navigation, mechanical integration, and, most importantly, safety concepts for operation in a shared workspace with humans. It is the goal of this review to identify the technical challenges, and to find out in which areas preliminary work already exists and in which areas research is still needed.

Numerous publications address the use of either UAVs or MRMs in production systems. In this paper, we focus on the joint use of UAVs and MRMs. To this end, we will review the state of the art for both technologies below. Subsequently, we will explain the procedure of our systematic literature review (SLR) on the combined use of MRMs and UAVs. The results will then be categorized, and finally, the existing gaps and the necessary needs for action for future research in this area will be presented.

We introduce the following terms and abbreviations for clarity:

  • Unmanned Aerial Vehicle (UAV): Aerial robots, flying autonomously. UAVs can be divided into Multirotor UAVs, Helicopter UAVs, and Fixed-Wing UAVs. We focus on multirotor UAVs in our review.

  • Automated Guided Vehicle (AGV): Ground robotics, moving autonomously. For our review, we include both, AGVs following predefined tracks based, e.g., on optical or magnetic lines, and AGVs navigating autonomously by using, e.g., virtual models of the areas or other information. We also include all results found for AMRs (Autonomous Mobile Robots), which can be defined as AGVs of the latter category, under the term AGV, since often no clear distinction is made and similar challenges for collaboration arise.

  • Mobile Robot Manipulator (MRM): Combination of autonomous platforms, e.g., an AGV with a manipulator. It is irrelevant whether, e.g., wheels, omnidirectional wheels or chain drives are used for movement. The end effector used is not limited to a gripper, but can also be a tool.

  • Cooperation: The joint execution of an action or task involving both UAV and MRM.

2 State of the Art

For both MRMs and UAVs, reviews on specific topics such as navigation, or motion control exist. We attempt to categorize the most important existing reviews in the following two subsections with a focus on those that are relevant for intralogistics, or more generally, indoor applications in production systems. It will be evident that most existing reviews address either MRMs or UAVs, but not their joint use. We stress that we summarize only existing review articles, since an analysis of the vast body of original literature is beyond the present paper.

2.1 Literature Reviews on MRMs

When searching for existing reviews, only reviews describing systems that meet the definition of an MRM from Sect. 1 were considered. The so found reviews are listed in Table 1. The existing reviews on MRMs can be divided into different subject areas. Advances and Applications describe publications that have investigated applications or application areas for MRMs. The Challenges category examines the challenges in the use of MRMs and the technologies required for it. The Construction category summarizes publications that have investigated the use of MRMs in construction. For Motion Controlling, there is one publication that analyzes different techniques and methods. Reviews that deal with currently available methods and algorithms for path planning and navigation of MRMs fall under the category Navigation. For the planning and control of MRMs in intralogistics, a review is classified in the category Intralogistics.

Table 1 Literature reviews on MRMs
Full size table

2.2 Literature reviews on UAVs

The following literature searches on UAVs refer to quadcopters or VTOL multicopters used in indoor applications or whose outdoor application is at least not mentioned. Due to many searches found, it can be assumed that not all studies on this subject have been found. Nevertheless, Table 2 includes all findings on a particular field.

In the field of Aerial Manipulation, mechanical manipulation systems that are attached to the drone are reviewed. Within the area of Application-related Tasks, the use of UAVs for specific applications was investigated. The Collision Avoidance Systems category contains reviews that examine the various techniques and methods available for avoiding collisions. Communication and Control Topologies include all reviews dealing with communication and network architectures. The reviews dealing with the existing methods for controlling UAVs are classified under Control Methodologies. Only one review deals with current problems for implementing logistics processes with UAVs and is classified under Existing Problems. Articles in General Research do not survey a clear specific field the review is based on. Herein mentioned reviews investigate mixed fields of applications or issues, e.g. construction types of UAVs or future strategic directions.

Table 2 Literature reviews on UAVs
Full size table

The Localization, Mapping and Routing category presents the reviews that deal with UAV path planning to perform specific tasks or create maps as well as the localization within them. All reviews on legal issues are classified in the Regulation and Legislation category. The Remote Sensing category includes the reviews that examine the various technologies and applications for UAV data collection. All reviews that examine existing technologies for landing UAVs using cameras are classified in Vision-based Landing.

2.3 Literature reviews on the combination

UAVs and MRMs can cooperate to solve a wide variety of tasks and leverage their respective strengths. While UAVs are characterized by a high speed and flexibility in congested environments, an MRM can transport heavy loads, perform ground-based tasks with a manipulator, assemble products and support workers on the ground in various ways. The potential of their symbiotic use has been described in the areas of logistics, supporting manipulation, mapping terrain in disaster areas, construction, formation control between air and ground units, and warehouse picking in the literature summarized below.

We note a generally established terminology for cooperative UAVs and MRMs appears not to exist, which renders a literature search difficult. Cooperative UAVs and MRMs are often referred to as a multi-agent system, multi-robot system, UAV-AGV coordination systems, or go under the headline air-ground collaboration. Existing reviews of the literature on UAV and AGV collaboration can be categorized into studies on subsystem coordination [5, 74, 75], and target detection and tracking [76]. While it does not deal with UAV and MRM collaboration, [77] should be mentioned, because it addresses the simultaneous manipulation by several mobile manipulators. Similarly, [78] does not address UAV and MRM collaboration, but challenges for co-manipulation and transportation tasks from a more general perspective.

For the combination of UAVs and MRMs within the industrial production environment, no literature research has been conducted so far, which is why the research carried out here is intended to sort the already published literature and identify application gaps for economical use.

3 Procedure of the systematic literature research

In order to make the results of a literature search comprehensible and transparent, it is essential to document the procedure. We summarize the procedure applied for the present paper in this section. The documentation is based on [79, 80]. For the sake of clarity, the terms Unmanned Ground Vehicle (UGV) and AGV from the publications are combined under the term AGV. If an MRM is described in the publication, this term is also used explicitly.

Focus: The focus of this SLR is research outcomes as well as practices and applications.

Goal: The goal of the SLR is to collect and structure existing applications as well as the software, hardware, deployment concepts and algorithms available for them in order to identify existing gaps in the research landscape.

Perspective: In conducting the SLR, a neutral position was taken, and an attempt was made to depict an objective representation of the current research landscape. However, fields of action stated in Sect. 5, also reflect the experience of the authors.

Coverage: The coverage of the existing literature was as exhaustive as possible. However, some papers were selected as a proxy if they did not differ from other existing papers in terms of content, such as the publication of content by the same author in different journals or conferences.

Organization: The structure of the SLR follows the methodological format, where first, after an introduction, the state of the research is presented and, after the presentation of the method of the SLR, the analysis of the results follows. The literature found is organized thematically by grouping the various publications on different topics and thus sub-areas of the problem.

Audience: The target audience of the SLR are specialists from the areas of production, as well as mobile ground and aerial robotics. The target audience is to be classified as a professional audience, i.e., PhD students, professors, etc., and users from industry such as engineers.

Criteria: Publications were included if they met the following criteria:

  • The publication has been published between 2010 and 2021.

  • Since it was found during the search that very few publications exist on the cooperation between MRM and UAV, publications on the cooperation between AGV and UAV are included if they are related to production systems or if the content makes it irrelevant whether an AGV or MRM is present (e.g., during the landing process).

  • Cooperation between a UAV and an MRM/AGV is described at least as a side topic.

  • If it is irrelevant in the execution of an action whether the ground vehicle moves autonomously and the described implementation can be transferred exactly the same to an AGV or MRM, these papers are also integrated.

Procedure: An iterative search was performed. An initial keyword search with the keywords detailed below provided a starting set. Subsequently, forward and backward searches were carried out, i.e., relevant articles among those cited in the collected ones and relevant articles citing the collected ones were identified. In addition, a search for further publications by the relevant authors was carried out. The forward, backward and author-based searches were repeated iteratively. The sources listed in Table 3 were used for the search.

Table 3 Sources used in the SLR
Full size table

The following keywords were used for the initial search run. Keywords from the three categories were used both individually and in all combinations:

Mobile aerial robotics: UAV, unmanned aerial vehicle, drone, quadrocopter, multicopter, VTOL

Mobile ground robotics: mobile robot platform, mobile manipulator, mobile robot manipulator, UGV, AGV, automated guided vehicle, AMR, autonomous mobile robot, moving platform

Cooperation of UAV and MRM: Cooperation, collaboration, multi-agent system, intralogistics, landing on moving platform, MAGMaS (Multi Aerial Ground Manipulator System)

4 Analysis of the research results

This section presents the results of the systematic literature review. The publications found are first classified based on the application in Sect. 4.1 This is followed in Sect. 4.2 by an analysis of the implementation described in the publications. Finally, Sect. 4.3 examines the maturity of the publications.

4.1 Type of application

Table 4 groups the publications found based on their type of application. For the sake of clarity, publications on MRM are also included in the table under the term AGV. If publications describe a cooperation with MRM, this is explicitly mentioned in the following text. The remainder of Sect. 4.1 discusses the types of applications in the order they appear in Table 4.

Table 4 Types of application
Full size table

4.1.1 Co-Manipulation

The field of co-manipulation deals with publications where UAV and MRM together perform a transport, movement or action on an object. The listed publications therefore deal with special control strategies, task allocation and controller design for manipulation tasks that involve the UAV and MRM. Among other things, the focus here is on how a task can be performed with the lowest possible energy consumption by combining UAV and MRM. Furthermore, the control of UAV and MRM for simultaneous handling of an object is investigated. The papers mainly deal with special or individual cases, so that a transfer to the general use in production systems is difficult.

Such heterogeneous teams may be used for construction, localization, pickup, transport or assembly tasks as it is described in [82], where a UAV and an AGV cooperated in closing a valve in a disaster-stricken industrial environment. The system had to deal with a control system for aerial manipulation, self-localization and navigation, as well as task allocation algorithms to enable the collaboration of the two units. The system was validated in a Robot Operating System (ROS [141]) simulation testbed and visualized in the Gazebo robot simulation software [142].

The authors in [81] also describe a UAV and MRM wall building system for the Mohamed Bin Zayed International Robotics Challenge (MBZIRC) 2020. The coordination framework for a cooperative robotic team was tested within a physical simulation environment. The locations of all components and the hierarchical decomposition were known a priori. The results indicate a successful cooperative mission scheduling under various setups of multi-criteria objectives.

In [83], a decentralized task planning and coordination framework for multi-robot systems is proposed and tested. The robot system consisted of an AGV carrier unit for the UAV and a dual-arm UAV. The mapped environment and hierarchical decomposition of the system mission were obtained in the first exploration phase. For various mission setups in parcel delivery, the AGV carried the UAV as close as possible to the mission target and therefore found the most energy-saving strategy for the two units. The planning framework proved to reduce the energy consumption for tasks like parcel delivery or inspection by 80%.

The report in [84] deals with control strategies for an AGV and UAV team to move an object to a particular pose, i.e., its position and inclination. While the AGV moved the object to the goal position, the UAV was responsible for adjusting the inclination of the objects. The authors designed specific controllers and proved the system stability. Within numerical simulations the effectiveness of the proposed control strategy was shown. Another aerial ground co-manipulation system is presented in [85, 86], in which the units cooperated to co-manipulate a complex object. The control strategies for the system were validated in a physical simulation environment as well as in a real test scenario and showed high reliability. In an experiment, the UAV and MRM autonomously lifted a bar and transported it back to its initial position. A motion-capture system was used to determine the pose of the bar and the UAV. The robots external torques sensed from the joint sensors are monitored during the execution.

4.1.2 Landing the UAV on an AGV

Landing the UAV on a moving AGV, often referred to as a rendezvous, is crucial for physical cooperation between the two systems. Because the AGV is moving, the UAV has to calculate the translation from the current position to the desired landing spot and adjust its velocity dynamically. The usual AGVs do not have a lot of space for an inaccurate landing, so the landing process must be within a specific area at a particular time. In this context, the existing scientific reports primarily deal with vision-based methods, sensor fusions, and a proper controller design. Not all mentioned reports consider indoor applications due to testing the concept in a large area with landing on a rapid AGV. However, the reports presented apply to the indoor environment and have therefore been included in the literature research.

In terms of vision-based landing procedures, landing on a moving platform can be considered solved. In 2001, the first report on landing an UAV on a moving platform can be found in [143]. By this time, tracking a moving object came already down to the same challenges it faces today – image recognition, trajectory planning and controller design of the UAV. Most of the vision-based approaches follow the strategy of recognizing an ArUco marker or an April Tag (see [144, 145], respectively, for an introduction to these types of marker) in the images of the drones, estimate the relative pose of the UAV to the defined position of the objects and compute the path of the drones. Since it was published in 2001, [143] is not listed in Table 4.

For ease and clearness, the found reports regarding the landing procedure of an UAV on an AGV are divided into vision-based procedures, procedures involving a sensor fusion, and papers focusing on the controller design and trajectory planning. Beginning with the vision-based procedures in [87], the authors use a remote infrared Wii camera attached to the quadcopter UAV for detecting the infrared markers on a landing AGV. The system was capable of stable tracking up to a distance of 2.5 m and following along with the moving AGV. The authors discussed the trajectory planning for the pose calculation and their system test results for following the AGV in a circular path.

In [88], a landing and control algorithm using so-called image-based visual servoing (IBVS) for a VTOL UAV is presented. IBVS is described as a technique that requires less precise depth estimations, since most of the calculations are two-dimensional, in contrast to typical techniques representing the target in 3D. For experiments, the three control modes patrol, tracking and landing, were defined and validated. The landing position was detected with the help of an ArUco marker in the middle of the landing frame of the AGV. According to the authors, the experiments showed a successful implementation of the control strategy.

The authors in [89] deal with control algorithms for landing the UAV autonomously on a marine vehicle. The landing procedure on the ocean brings special conditions like strong winds, currents and the wave motion that cause the landing target to rock and drift. Implementing the image procession was achieved using MATLAB. The testing has been carried out indoor and outdoor by landing the UAV on a kayak with a success rate of 75%. Another vision-based approach for a UAV and AGV landing system is given in [90]. A circle-shaped marker is attached to the landing area, enabling the UAV to track the AGV autonomously. A two-level control strategy consisting of low-level self-stabilized flight control and a high-level position control was implemented and considered in more detail. Within their experiments, the authors demonstrated the successful autonomous taking-off as well as the tracking of and landing on the moving AGV. Furthermore, a discussion on the control performance is given.

In [91], a vision and LiDAR-based localization and tracking algorithm is implemented and verified on a UAV and AGV system. By detecting a marker on the landing AGV with a monocular down-facing camera, the relative pose between the UAV and the landing AGV is calculated. The LiDAR range finder was used to estimate the UAV’s height above ground and enabled a controlled landing procedure.

A vision-based landing is proposed in [92], validated by both, simulations and experiments. The authors deliver a complete solution for a rendezvous by designing a compound landing marker with small and large scale features for visual feedback during the entire landing process.

In [96, 111], the authors present another result of the MBZIRC 2017. The authors solved the given task of landing a UAV on a fast-moving vehicle in an outdoor environment. The system was evaluated in a simulation and later demonstrated at the MBZIRC, where the system proved to be one of the fastest competitors. The authors describe the system setup, the implemented control strategy, and their evaluation results in detail. The autonomous landing of a quadcopter on a high-speed AGV is also discussed in [93]. The authors validated the use of a Kalman filter for the visual position and velocity estimation. A two-controller strategy was used in which a so-called Proportional Navigation law was first used to approach the moving target, and a second PID controller was used for the landing phase. The experiments resulted in a maximum speed of the AGV of 50 km/h, or around 13,89 m/s, during the landing phase.

In [94], a deep reinforcement learning approach of vision-based autonomous landing is presented. The training was performed in a simulation environment with continuous state and action spaces. A final validation in a real-flight scenario was carried out successfully with a maximum speed of the AGV of 2 m/s during the landing phase. The publication [95] deals with a hybrid camera array-based autonomous landing procedure with a high field of view camera (fish-eye camera) and a tailored control strategy. The authors describe the development of their system with an initial simulation and parameter identification in Microsoft AirSim [146]. An ArUco marker was attached to the landing platform, and multiple tests validated the performance of the system in simulations and a real scenario. The authors provide a comprehensive analysis and discuss their results in detail.

In the approach in [97], the authors also use an April Tag marker to locate the moving landing platform. The UAV carries a gimbal camera for identification and has a GPS receiver for coarse localization of the landing platform. The landing process is controlled by a model predictive control (MPC) scheme. With respect to the center of the landing platform, flight experiments showed that the autonomous landing could be realized with an average error of 0.39 m, i.e., 39 cm. Another approach to enhance the landing process is sensor fusion. By using different sensor data and fusing them, for example, a faulty, inadequate or unavailable GPS signal can be compensated to allow a safe landing process. In [98], an approach for a tether-based UAV is shown for this purpose. For the landing process, data from an inertial navigation system (INS), magnetometer, altimeter and rope sensor are fused to determine the position and acceleration of the UAV. Using the tether-based approach, the rope sensor can determine the distance and twist angle to the landing platform. Experiments showed a successful landing on both, a static and a moving platform.

In [101], the authors also present a method for motion tracking and control of the landing process in environments where no GPS signal is available. For this purpose, they fuse the data from multiple red, green, blue-depth cameras (RGB-D) and an inertial measurement unit (IMU). The control is performed using a model predictive controller and resulted in consistent landing on a target moving with up to 5 m/s.

To save battery power of the UAV, the landing on moving ground vehicles is investigated in [93, 99, 100] using sensor fusion. In the first two cases, AprilTag markers are used for tracking.

The authors in [93, 100] take an approach in which data from a ground camera, an orientable camera with gimbal, the IMU of the gimbal on the UAV, and the GPS of a mobile phone on the landing platform are fused to realize a landing speed of around 13.89 m/s (50 km/h) for the landing platform.

The approach in [99] is based on the fusion of a stereo vision-based visual odometry with the IMU data and a particle filter-based visual servo in the UAV vision system. While the quality of the fused data was tested in experiments indoor and outdoor, the landing itself was shown in a simulation.

As mentioned above, controller design and trajectory tracking are other important fields of interest when realizing the landing of a UAV on an AGV. Following the AGV, i.e., tracking its trajectory and reaching a position to land on it, are the main tasks for the UAV addressed in the literature. In [102], the authors propose a nonlinear controller for a UAV equipped with a camera and IMU only, while the AGV is a moving platform at sea.

Nonlinear controllers, partly combined with optical flow, i.e., exploiting the projection of the target velocity, are used to stabilize the UAV position relative to the AGV and the landing procedure itself. Simulations and experiments showing a successful landing are also included. Optical flow is also used in [104] to measure the velocity using the translational optical flow, while a centroid for landmarks is used to determine the position. The authors provide simulation and experimental results and claim to guarantee convergence also with disturbances present.

The authors of [103] propose an approach to deal with time delay and their effect on closed-loop stability in the context of the rendezvous between UAV and AGV. They determine so-called delay margins for particular setups and showed the effectiveness using simulations and experiments. For landing the UAV on the AGV, a navigation control algorithm is proposed in [105] for a UAV equipped with a camera looking downward, so the position relative to the target can be determined. To realize the desired search, homing and landing phases, the authors create a virtual sphere around the AGV target position, with a virtual target moving on its surface. When approaching this target point, the UAV is guided to land at the desired spot. The adaptive tracking control scheme based on backstepping and dynamic surface control was validated in simulations.

In both [108] (for a vessel, however, transferable to ground vehicles) and [109], the authors redefine the problem of trajectory tracking into the stabilization of the relative motion between air and ground vehicle. The focus in [108] is on developing separate controllers, one for reaching a location above the target, and another one for descending onto the target. Both controllers make use of a saturated adaptive sliding mode technique, estimate uncertainties by adaptive algorithms and use linear compensators to deal with input saturation effects.

In [109], the authors use an adaptive radial basis function neural network for estimating unknown disturbances and designing the controller using a backstepping technique. Both [108, 109] showed in numerical simulations the proposed approaches fulfill the desired tasks.

The authors in [106, 107] both propose MPC schemes to guide the UAVs. In [106], the authors focused on a guidance function embedding three phases: target detection, target tracking and landing. Their approach was tested in simulations and experiments. Compared to other control techniques, the proposed MPC scheme showed the best overall performance, and the authors included their developed functions to the ISAE Supaero drones library. A system architecture combining the modelling of the aerial vehicle equipped with a camera, the localization of the AGV using a Kalman filter and, finally, an MPC scheme for guiding the UAV is presented in [107], here tested in simulations using the DJI Hardware-in-the-loop simulation environment. For a landing platform moving at a speed of up to 12 m/s, an accuracy of 0.37 m, i.e., 37 cm with respect to the center point could be achieved. While the trajectory tracking itself is realized by a nonlinear feedback controller in [110], the authors use an MPC scheme to generate the reference trajectory. Following and landing with respect to a moving AGV were tested in field tests and presented at the MBZIRC.

4.1.3 Landing the UAV on an MRM

For a better separation, papers addressing the landing process on MRMs are summarized separately here. The landing can occur both on the platform of the MRM, which is meant to carry the workload, or on a separate landing platform attached to the robots end effector flange. Landing in conjunction with the manipulator is considered even more complex because the manipulator grants an additional level of freedom for the landing procedure, and the manipulator may be actuated. The first approach for this challenge can be found in [112], where an image-based visual servoing control scheme was employed for the coordinated landing of a VTOL UAV at an actuated robot flange. The entire system dynamics of the UAV and MRM were simulated in a MATLAB/SIMULINK environment. A vision-based landing procedure was chosen to guide the UAV to the landing position at the AGV. For that purpose, the landing platform was endowed with several image feature points that were observed by the UAV camera. Concluding the report, the authors presented the performance in several case studies. However, it can be assumed that landing on a platform always needs some docking mechanism to prevent the UAV from falling from the landing platform. This is why the authors in [113] approached the docking of a quadrotor at the flange of an MRM within a ROS and Gazebo simulation environment with a focus on docking mechanisms. An eye-in-hand device detected visual information about the relative pose of the UAV and MRM at the end of the manipulator flange and a marker underneath the UAV. The end effector was programmed to follow the UAV to its nearest position to keep it in sight. The attachment of the UAV to the flange of the MRM was made possible by a special interlocking magnetic connection. The same authors validated their system in [114] with real scenario experiments. In addition to the simulation in [113], the authors extended their former approach with a control strategy, path planning algorithms for UAVs and MRMs, and simulating physical contact forces between robots for a robust design of a docking device. In experiments, the UAV executed exploration tasks, performed heterogeneous surveys in interest points and flew back to the MRM for safe storage and transport.

4.1.4 Docking Mechanism

Any complex interaction between the MRM and UAV needs a docking mechanism between the units that hold the UAV in place while the MRM manipulates the transported object. The main problems during docking or landing maneuvers between UAV and MRM are the motion tolerances that have to be compensated either by software or hardware. For example, when transporting the UAV on the MRM over rough terrain, picking up material from the UAV or storing it could cause the UAV to fall off the robot platform or lead to an inaccurate manipulation. Besides that, applications regarding connecting a payload to the UAV or docking it to a station for recharging are examined.

In [115], the authors described an autonomous docking mechanism for UAV that is capable of vertical take-off. The system consists of a static ring and actuated legs. With the help of an infrared beacon on the docking device and an infrared tracking device on the UAV, a stable docking process can be performed even under strong wind and poor sensor signals. The authors in [116] developed a spherical-shaped docking interface. The UAV had a sphere-shaped form and a retractable landing gear to be able to land on flat surfaces. In addition, a concave landing device was designed for the MRM, allowing safe landing even with inaccurate positioning and control. The presented concept was not proved within any real environment.

The authors of [117] developed a method for landing a UAV on an KUKA LRW4+ industrial robotic arm using computer vision. The robot arm guided a camera which detected a marker mounted on the UAV. A magnetic grabber was attached to the robot arm and UAV that would hold the UAV to the arm after a successful landing. The robot was implemented in ROS and able to follow the UAVs position while moving. The range for the magnetic connection though only had a fault tolerance of less than 1 cm.

4.1.5 Formation Control

Several difficulties arise in this area and are accounted for in the following literature. The recognition of the designated formation partner, e.g. based on special markers, and the combination of several control loops for managing different tasks simultaneously are only two of them. Further, the communication between the corresponding partners, and the compliance of constraints, e.g., regarding the distance between each partner, are also challenging in this research area. For a successful collaboration between several robots, it is essential to coordinate them in the desired way. This section gives an overview of solutions for the so-called Formation Control problem. While often accounted for in the control of swarms of UAVs, formation control becomes important also for cooperating air and ground vehicles.

The focus in [118] lies in developing a control setup that uses internal sensors only. The resulting controller was validated experimentally by following a ground vehicle equipped with an ArUco marker, i.e., a synthesized marker for pose recognition. Both underlying problems, reference tracking of a velocity profile, and position control to finally target the desired way points, were addressed by the combination of basic control laws. Based on an established concept, the main contribution of [120] is a distributed control using self-organizing wireless communication between UAVs and ground vehicles for formation, maintenance, and de-formation. The network in the form of a rooted tree here results from a self-organizing process. The functionality of all three formation control tasks was proven in simulations, combined with investigations on fault tolerance and scalability.

In contrast to the previous two publications, [119, 121] both deal with formation control of physically connected aerial and ground vehicles. In [119], the AGVs connected to the UAVs via tethered cables act as additional actuators to, e.g., stabilize the aerial vehicle, and thus require a strategy for a suitable formation. Control allocation techniques serve as the basis for the proposed approach. In simulations, the authors tested the ability to maintain the position of a UAV using three AGVs subject to disturbances induced by wind. In [121], the physical connection between aerial and ground vehicles is used to provide power to the UAV in the Powered-over-Tether fashion, such that greater distances in the context of infrastructure inspection can be reached. Thus, the authors propose a control framework to, on the one hand, coordinating aerial and ground vehicles to maintain the connection, and on the other hand, to plan the paths for infrastructure inspections. The leader–follower approach for coordinating both parties is combined with a path planning algorithm, and the functionality was shown in validation experiments.

4.1.6 Mapping and Localization

Besides a communication between UAV and AGV, whether directly or via a central node, the large amount of data that is generated by mapping the ground are problems arising in this context. The processing of the data and the fusion of information from processed images of the UAVs, built-in sensors of both, UAV and AGV, and the implementation of tailored control actions are another challenges. While the ground vehicle uses its vision sensors to map the ground, the flying UAV has a greater angle of view. It can guide the MRM or AGV, assist the manipulation with the vision from another perspective, or find obstacles and equipment faster than the ground vehicle by itself. Although the cooperation between UAV and AGV for mapping and localization has primarily been used in outdoor applications in the publications summarized below, the feasibility of a transfer from outdoor experiments to an indoor environment are assumed.

For cooperating systems of aerial and ground vehicles, we can divide the remaining contributions in this section into basic path planning, solutions for special applications and target detection. In [122], the UAV recorded ground images, which were sent to the AGV. On the AGV, the image was processed into a ground map, and, using a combination of genetic algorithm and rolling optimization, a path was planned, which could be adjusted according to the location on the map. The method was validated using images with obstacles to generate optimal paths. AGVs or UAVs for real implementation were not used.

The combination of visual information provided by the UAV and the sensors of the AGV on the ground for general path planning is also addressed in [127, 129, 133]. The authors in [129] propose a planning framework for path planning combining established approaches to exploit both the endurance of the AGV and the flexibility of the UAV. Here, for localizing UAV and AGV, fiducial markers and a camera were used to detect the AGV from the UAV, and the UAV was then localized using the position of the AGV. A Segway-based UGV and a Pixhawk-based UAV were used for real implementation and validation.

In [133], the UAV is located on the AGV, which follows a path predefined by the operator. In an ambiguous situation for the AGV, the aerial vehicle takes off and follows the AGV trajectory, providing information about the future path to the operator for trajectory planning. The method was applied and validated outdoors using real hardware. To overcome the restrictions from limited fuel to the flight distance of the UAV, in [126], the aerial vehicle mapping the ground is combined with an AGV that can refuel the UAV when necessary. Algorithms to coordinate the UAVs and the mobile fuel stations are presented. Validation only took place in a simulator, which was specially developed for this purpose in MATLAB.

Special applications are addressed in [124, 125, 128, 131, 132]. While a 3D map of a house damaged by an earthquake is in the focus of [124], authors in [125] use a team of unmanned vehicles for the localization of gamma radiation hotspots. Another humanitarian mission is described in [132], where the cooperation of UAV and AGV is used for demining missions. Besides these applications, teams of autonomous robots are also used for mapping and localization for economic reasons. In [128], mapping and localization with autonomous robots is used to reduce the need for human actions in intelligent manufacturing. The authors in [131] present an approach to mapping areas with a power-over-tether aerial and ground vehicle team.

A validation in reality with corresponding hardware was carried out in [124, 125, 132]. A validation in simulation was applied in [128, 131]. Another area of interest that requires mapping and localization techniques is the field of target detection. The authors in [123] propose to merge the visual sensor information of the UAV with the information about the absolute position of the ground vehicle to provide a map with absolute coordinates of obstacles. For validation, the scenario was first simulated and then implemented in reality.

In [126], measurements from bearing-only sensors were triangulated by a multi-robot team for target tracking. In bearing-only localization, the angle of arrival of a signal from a known transmitter is measured. The local measurements were filtered by each robot, which could then communicate and exchange information. The approach was tested in simulations and field experiments. The author in [130] focuses on the coordination of aerial and ground vehicle teams to systematically scan defined areas. These teams were synchronized by switching between different state-dependent and condition-triggered controllers. Validation only took place in a simulation. The authors in [134] demonstrate an approach to cooperative area monitoring and object identification. The AGV and UAV are equipped with cameras and transmit the data to a cloud, where it is preprocessed and made available to the operator. In one use case, it was shown that the UAV can detect an object from the air and then generate automatic path planning to the object for the AGV. The implementation of the approach was presented in an outdoor environment.

4.1.7 Transport and Logistics

AGVs are nowadays broadly used as intralogistics systems that transport goods through multiple levels of a shop floor. Flexibility in intralogistics can be increased even further through the use of UAVs. The main problems here are the low load-bearing capacity and battery capacity of the UAV. Furthermore, UAVs and MRMs differ greatly in their characteristics with regard to load capacity and transport time, so that property-based task planning and control is necessary for flexible intralogistics. Research questions of the listed papers are therefore how several UAV or MRM together can transport an object, how the range of UAV can be increased by MRM as transport base and how a task distribution between UAV and MRM can look like.

Known solutions for the control of cable-driven parallel manipulators were transferred to MRMs in [137] and the modelling of a combined transport with MRMs grabbing load and UAVs using cables was investigated.

Since an AGV can typically carry heavier loads than a UAV, the AGV can also serve as a mobile transport base for the UAV. A graph-based optimization approach is presented in [139], in which a truck formed the ground unit for package delivery and the UAVs delivered the packages along the route. An implementation took place in a MATLAB simulation.

The use of an AGV as a central unit can also serve, for example, to transport UAVs thus extending their range of action. It is an obvious extension to use the AGV as a recharging station for the UAV in such a setup. One approach to this is presented in [136], where UAVs were used for transportation of an AGV for inspecting civil infrastructure. This was presented in a real environment. In [138], this approach was extended so that the UAV is connected to the AGV with a cable and thus supplied with energy. The combined system can be used for inventory and inspection of warehouses, for example.

For the coordination of cooperating UAVs and AGVs, a decentralized control approach is presented in [135], which performs optimal task allocation and scheduling considering the different capabilities of the participants and a known map. In [140], an approach is proposed to solve the problem of the limited battery capacity of the UAV. For this purpose, an AGV was used as a mobile charging station. The authors developed an algorithm to enable automatic route planning of UAV and AGV under the constraint that the UAV has to fly to a defined number of locations in the shortest possible time. The authors used algorithms for the travelling salesperson problem to determine the optimal number of AGVs, the route planning of the AGVs and the charging time points of the UAV.

Table 5 Types of implementation for different applications
Full size table

4.2 Type of implementation

The publications summarized here differ significantly with respect to their focus, which ranges from proposals of new concepts, over discussions of algorithmic details, to real-life implementations of demonstrators. It therefore proves useful to categorize the publications not only by the treated topic (Sect. 4.1), but also by the focus of the implementation. After categorizing the publications by their topic in the previous section, we further group them with respect to the focus of their implementation in the present section. Note that this is followed by a discussion of the maturity level in the next section. We use the four categories of implementations, presented in Fig. 1.

Fig. 1
figure 1

Implementation stages

Full size image

Publications assigned to the concept category deal with a general discussion about the collaboration of a UAV and MRM in any form. Publications in the concept category may regard a system architecture or economic efficiency of a system. However, an actual implementation of algorithms, hardware designs or control software is not dealt with. A publication is assigned to the algorithm category, if it addresses control and scheduling algorithms. A validation in the form of a simulation may be carried out. If so, however, it is of a general mathematical nature and not related to an actual implementation or application. A publication is assigned to the category software, if its focus is on software for the implementation of a specific application. Publications in this category do not include hardware testing or implementations in real environments. A publication is listed in the hardware category, if it reports test that were conducted with real hardware, i.e., test beyond mere simulations. In addition, publications that describe pure hardware development also fall into this category. The results of the analysis are shown in Table 5.

Table 6 Maturity level for different implementations
Full size table
Fig. 2
figure 2

Number of publications per implementation category

Full size image

The number of publications for each implementation and application is shown in Fig. 2. It is evident from the figure that most publications present an implementation of hardware solutions. The largest fraction of publications in the Hardware category address the topic Landing of UAV on AGV. Publications that exclusively deal with software development amount to only about a quarter of those in the Hardware category. Furthermore, there are many publications that present concepts, mainly for Transport/Logistics and Mapping/Localization. Publications dealing with new algorithms exist only for Landing UAV on AGV, Mapping/Localization and Transport/Logistics.

4.3 Maturity level

Finally, the results of the publications will be examined for their degree of maturity. The degree of maturity indirectly describes how promising an approach is for use in practice. Three categories were established for this analysis:

  1. 1.

    Concept

  2. 2.

    Prototype

  3. 3.

    Ready for market

Distinguishing the maturity of an application between a concept or prototype is a narrow burr. The term Concept is here understood to mean the authors did not show an implementation of the presented approach nor carry out any testing nor show the application’s functionality. As long as the type of implementation is suggested as a concept by the authors themselves, it will automatically be classified with the maturity of a concept. Applications in which an exemplary implementation including the corresponding evaluation is described are assigned to the Prototype category. An application that is considered to be mature enough to be classified as Ready for Market must be commercially available or soon available. The results of this classification are given in Table 6. Note that only one publication presented a result that can be classified as ready for market.

Fig. 3
figure 3

Number of publications per maturity level

Full size image

Figure 3 shows that the number of publications that presented a result with a Prototype maturity level is more than three times higher than the number of publications with a Concept maturity level.

5 Derivation of the fields of action

Our analysis shows that numerous publications exist on the cooperation between AGVs and UAVs. However, the cooperation between MRMs and UAVs is only very rarely treated. In the following, we will show which results from the existing publications can be transferred to a cooperative use of UAVs and MRMs in production systems and in which areas there is still a need for action. To this end, the findings from the analysis of the various application fields are first presented and then research gaps are identified.

5.1 Co-Manipulation

Currently, UAVs and AGVs are mainly used in combination for the transport of goods. The advantage of the MRM over an AGV is the ability to carry out activities while driving and thus create value while driving. However, to the knowledge of the authors there are no publications on the combination of MRMs and UAVs that treat the use of robotic arms for production. One research question here is what use cases exist in production that a manipulator can take over while driving. The need for action here lies first of all in the creation of application concepts and their evaluation with regard to applicability and the achievable benefits. Another identified research gap is the development of appropriate control algorithms suitable for simultaneous control of multiple UAVs and MRMs on a single component or application context.

5.2 Landing the UAV on the AGV

For the interaction of UAV with an MRM, the UAV must be capable of vertical take-off and landing. Our review shows that various approaches for VTOL of UAVs on AGVs exist, characterized by a large number of different methods and algorithms. However, the methods are used almost exclusively outdoors. Therefore, a major research gap in this area is the transfer of the methods and algorithms to the indoor environment. Here, the different boundary conditions for localization, such as a missing GPS signal, must be taken into account in particular.

5.3 Landing the UAV on an MRM

Because an MRM differs from an AGV only by an additional manipulator, these approaches can be easily transferred. Therefore, the aspect of landing on the platform is identical to the previous category. The procedures for landing on an MRM focus explicitly on landing on the manipulator. The analyzed papers present approaches that could be used in production system. Drones that could potentially be used in production systems for transporting goods must have a sufficiently large payload, which implies a large size. Therefore, a gap in research is mainly in the development of algorithms and hardware design for larger drones.

5.4 Docking Mechanism

Docking Mechanism always depend very much on the application scenario for which they have to be used. General research questions could therefore not currently be identified in this field. However, one research gap that has not been addressed is the development of docking mechanisms for autonomous UAV charging using MRM.

5.5 Formation Control

The found approaches for formation control were all used outdoor. Here, a research gap lies in the transfer to the indoor environment, e.g. for the use of power-over-tether UAV in interaction with the flight space, which is significantly limited by the infrastructure of production facilities. Another research gap here is primarily in the development of suitable network and communication topologies, for the communication of multiple UAV and MRM from different manufacturers with each other and with external systems for planning and control in a production system. A research question here also lies in how control algorithms can be executed centrally and how real-time communication, e.g. via 5G, could look like in a production system.

5.6 Mapping and Localization

Numerous approaches to mapping and localization have been described in the publications presented. However, these are used exclusively outdoors. The biggest research gap here is the transfer of the algorithms to the indoor area. This is especially interesting for the control of MRMs, which have to navigate autonomously and adapt to changing environments. Here, a UAV could generate a map for navigation much faster or provide live instructions to the MRM. Another major research question is how UAV and MRM can be combined to enable novel safety concepts in human-robot collaboration. By monitoring the MRM’s workspace and travel path with a UAV, movement and travel speeds could be dramatically increased.

5.7 Transport and Logistics

In order to use UAVs and MRMs flexibly in production, scheduling algorithms are needed that perform comprehensive resource and task scheduling for all participating mobile units. The algorithms must, e.g., take into account the different capabilities and equipment of the participants and consider the loading states and processes in terms of time and space. Our analysis revealed only very few approaches that address this issue. The need for action in this area exists to the extent that no concrete algorithms have been developed and tested for this application.

In summary, some preliminary work exists for combined use, especially for VTOL. However, two major research gaps exist that need to be considered in the future. These can be divided into research on basic functionalities for interaction and research on suitable application scenarios. The first research gap is that almost all publications describe outdoor use. Here, research is needed on how the existing algorithms and approaches can be transferred to an indoor environment and which algorithms may need to be newly developed to perform the basic functions for interactions such as VTOL, joint control, or task planning and control.

6 Conclusion

The cooperation of UAVs and MRMs in production processes promises great potential for the flexibility, economy, and demand of future manufacturing standards. Therefore, in this paper, we have examined the current research landscape on this topic. We conducted a systematic literature search and analyzed the results based on their area of application, form of collaboration, type of implementation, and level of maturity. Based on the analysis, recommendations for action were presented, proposing areas in which further research is necessary to make the full potential of the cooperation usable in the future.

Publications on the cooperation between UAVs and AGVs mainly deal with VTOL. Furthermore, most publications address other applications than production. Nevertheless, the existing publications provide a reasonable basis for a transfer to the cooperative use of MRMs and UAVs in production. However, it has been shown that only a few publications exist that explicitly deal with the cooperation between MRMs and UAVs. Therefore, there is still a strong need to apply the existing results to production systems and the development of specific solutions for MRMs in contrast to AGVs, which enables new possibilities.