Precise Cable-Suspended Pick-and-Place with an Aerial Multi-robot System

This work introduces the G-Fly-Crane, a proof-of-concept aerial multi-robot system designed to demonstrate the advantage of using multiple aerial robots as a valuable tool for novel construction techniques, not requiring the use of heavy engines and costly infrastructures. We experimentally demonstrate its capability to perform pick-and-place and manipulation tasks in a construction scenario, with an increased payload capacity and dexterity compared to the single robot case. The system is composed of three aerial robots connected to a platform by three pairs of cables. The platform is equipped with a gripper, enabling the grasping of objects. The paper describes in detail the hardware and software architecture of our prototype and explains the implemented control methods. A shared control strategy incorporates the human operator in the control loop, thus increasing the overall system reliability when performing complex tasks. The paper also discusses the next steps required to bring this technology from indoor laboratory conditions to real-world applications.


Introduction
In construction, and more in general in the context of civil engineering, the manipulation of heavy and bulky loads is often required. In some cases, areas difficult to access make the use of ground cranes unfeasible, thus requiring the use of aerial cranes. Large helicopters, for example, are employed to transport and assemble elements for the construction of power transmission lines [27]. Decommissioning of civil structures is also a fundamental and costly task in this field. Both assembly and decommissioning tasks can be accomplished using such heavy aerial machines. However, in addition to being extremely difficult to pilot, they could represent a danger for the construction workers. The use of Unmanned Aerial Vehicles (UAVs) for civil engineering applications, and in particular of multi-rotor platforms, would reduce the above-mentioned risk for the human workers. Moreover, UAVs working autonomously or semi-autonomously could potentially speed up the completion of construction/decommissioning tasks, thus reducing the associated costs. Hence, it is not surprising that the use of aerial robots for construction has gained popularity in the civil engineering domain.
In this context, "drone compatible" bricks have recently been designed so that UAVs can rapidly construct real-scale structures, particularly masonry structures, without the use of concrete or other gluing materials [9].
In the field of robotics, researchers have already investigated the problems of transporting objects and assembling small-scale structures. These problems have been firstly addressed by considering a single standard underactuated aerial vehicle equipped with a rigidly-attached gripper (mechanical or magnetic), allowing pick-and-place operations [3,14,22]. However, having the load fixed on the aerial robot significantly reduces its agility. Moreover, the coupling of the translational and rotational dynamics makes pick-andplace operations more challenging. A possible solution is to transport the load by means of a cable attached to the robot, which makes it possible to keep the rotational dynamics of the vehicle decoupled from that of translation [6,24,26]. Although the simplicity of such a system makes it suitable for transportation purposes, this solution is not optimal for construction uses. The limited payload of a single aerial robot and the inability to control the load attitude make the single-robot single-cable solution unsuitable for pick-andplace and complex manipulation tasks required in construction applications.
To the best of our knowledge, in this work, we introduce for the first time a real cable-suspended aerial multi-robot manipulator able to perform pick-and-place and manipulation operations in a construction/decommissioning scenario.
The system developed for this proof of concept, called the G-Fly-Crane (Grasping Fly-Crane), is an extension of the Fly-Crane [15]. Similarly to the Fly-Crane, the G-Fly-Crane consists of three quadrotors attached by pairs of cables to a rigid platform. However, the G-Fly-Crane is now equipped with a magnetic gripper (see Fig. 1) and incorporates a software architecture to deal with pick-and-place operations involving a shared control approach. The latter represents the main contribution of this work .
This new system is meant to solve the problems previously mentioned: it has an increased payload thanks to the use of multiple quadrotors, and can precisely control all the 6 degrees of freedom (d.o.f.) of the cable-suspended platform, and thus of the grasped object. We shall demonstrate through experiments, with real pick-and-place and manipulation operations, that these properties make the G-Fly-Crane particularly suited for construction and decommissioning applications. This proof of concept represents a preliminary, but fundamental, step in scaling up the system for real-world outdoor applications, with the inclusion of on-board sensors, more sophisticated grippers and larger aerial robots with increased payload capabilities.
Although the G-Fly-Crane could control its motion fully autonomously, the complexity of assembly operations in real construction sites, in addition to current safety standards and regulations, requires the possibility for a human operator to supervise and possibly intervene in the control of the system. For this purpose, we designed a shared Fig. 1 G-Fly-Crane and its main variables. Several types of grippers can be attached to the platform. In this work, we integrated a simple magnetic gripper control strategy, which allows a single user to control both the position and the attitude of the platform, without having to manage the simultaneous control of each robot. Such an assignment would be impossible for a single operator and would require a challenging coordination of multiple operators. Given the operator commands for the platform, the proposed control approach optimally resolves the system redundancy and computes the proper commands for each robot. The proposed control method relies on the kinematic controller presented in [19,20].
Other multi-robot systems and control strategies have been proposed for the cooperative transportation and manipulation of cable suspended loads. Detailed surveys have been presented in [2,18]. Different strategies based on inverse dynamics, differential flatness, and formation control, have been proposed to control a team of aerial vehicles connected by cables to a point mass load for transportation purposes [1,5,11,16]. Aiming at more complex manipulation tasks, a two-robot system was considered proposing vision-based [8] and communication-less strategies [25,28] for the control of a cable-suspended bar. To control the full pose of a cable-suspended rigid load (requiring the ability to attain six-dimensional wrenches) at least three robots connected to three non-collinear points on the load by one cable each are needed [12,23]. However, to enhance the precision of the system by avoiding internal motions, a statically rigid [4] design is preferable. This can be obtained using six (or more) robots connected to six (or more) non-collinear points on the load by one cable each [17]. The Fly-Crane [15,20], and so the G-Fly-Crane (as well as similar designs [21]), use a simpler and more efficient statically rigid design involving only three robots, which is the minimum number to fully control the platform pose. This makes it preferable over the other statically-rigid solutions. Nevertheless, the current literature mainly focuses on demonstrating full-pose control capabilities without real interaction tasks. In this work, we go further by experimentally demonstrating the effectiveness of the G-Fly-Crane together with a shared control approach for pick-and-place and manipulation tasks in a construction scenario.
The paper is organized as follows. The G-Fly-Crane hardware and the scenario designed for the experiments are described in Section 2. Section 3 presents the software architecture implemented, and Section 4 describes the proposed control strategy. Experimental results of pick-and-place operations with the G-Fly-Crane are shown in Section 5. Finally, Section 6 concludes the paper with final discussions and comments. Figure 2 shows the scenario designed to demonstrate pickand-place and manipulation operations with the G-Fly-Crane. This scenario takes place inside an indoor arena equipped with a motion capture system and emulates the construction of a wall. Initially, the wall is incomplete. The two missing bricks, (red bricks in Fig. 2), necessary to complete the wall, are located in a pile on the ground. To increase the task difficulty, the wall was designed simulating an unevenness of the ground, presenting both flat and inclined surfaces. While one brick has to be placed on a flat spot, the other has to be positioned tilted, thus requiring 6D (position and orientation) manipulation capabilities. The bricks weight 0.5 [kg]. The G-Fly-Crane starts from a random location on the ground.

Experimental Setup
The G-Fly-Crane is made of three quadrotors (robots) and a platform. Figure 3 shows a graphical scheme of the hardware mounted on each quadrotor. Each robot is equipped with a Mikrokopter FlightCtrl V2.1 and four Six nylon cables are used to attach the robots to the platform, which is made of carbon fiber bars for a total weight of 0.350 [kg]. The platform was designed to dock different types of grippers. For the present work, the G-Fly-Crane is equipped with a gripper consisting of a matrix of four electromagnetic devices. This set-up, chosen after performing several tests, allows compensating undesirable torques while manipulating objects, and 0.1 [kg] to the total weight of the platform. The gripper control is implemented on an Arduino Board mounted on one of the robots and connected by USB to its onboard PC. A metal plate glued on top of the bricks allows their manipulation using this magnetic gripper. Table 1 compares the weight and loading capability of a single quadrotor with respect to (w.r.t.) the one of the G-Fly-Crane system. The comparison is made fair by considering for the G-Fly-Crane system the extra weight introduced by the platform, and that the quadrotors always fly slightly tilted (on average χ = 10 [deg]). This is needed to keep the cables taut and to ensure the balance of forces needed for the 6D manipulation (see [20]). This causes a loss of efficiency of about 1 − cos( ) ≃ 1.5% . Following these considerations, the G-Fly-Crane can manipulate loads that are almost 2.5  times heavier compared to what a single quadrotor unit, equipped with an identical gripper, can bear. Furthermore, the G-Fly-Crane has also the capability of manipulating the load in both position and orientation, and of keeping the rotating propellers farther from the environment during the pick-and-place operations. This configuration increases safety, reduces ground effects and creates less downwash on the environment thanks to the larger distance between the propellers and the ground.

Software Architecture
The software architecture for the G-Fly-Crane is implemented using GenoM3, an open-source (BSD-like license) tool developed at LAAS-CNRS to design realtime software architectures. GenoM3 allows the design and production of modules that encapsulate algorithms with inter-process communication, datapool capabilities with external access, and automatic telemetry generation. This framework has been tested in a variety of research robotic prototypes, and used in several European Space Agency (ESA) projects [10]. Figure 4 shows the architecture implemented. It is composed mostly by GenoM modules and divided into three layers, namely: (i) UAV layer, which includes the modules related with the flight controllers, state estimation and motor speed controller for each robot; (ii) shared control layer, which provides the connection between robots, platform and the user, implementing the controllers needed to operate the whole G-Fly-Crane system; and (iii) user layer, the interface with the human operator, which provides the visualization of the system status and the access to safety parameters, controller gains, logs, etc.
The UAV layer is composed of standard GenoM3 modules 1 running on the onboard PC of each quadrotor. On the other hand, the user layer is composed of a graphical interface and the joystick module running on the ground station PC. Both layers communicate thanks to the shared control layer, which runs also on the ground station PC. This layer implements the Fly-Crane Shared Control module (FSC) developed for this application. The FSC is based on the control strategy presented in the next section.

Control
The G-Fly-Crane is designed to perform complex tasks related to structure construction. Because of the complexity of the involved problems, such as object localization and classification, obstacle avoidance and so on, real application scenarios and the corresponding regulations require the presence of humans to supervise or operate the robotic systems. However, when a system has a large number of degrees of freedom, such as the G-Fly-Crane, it is impossible to simultaneously command all of them for a single operator. Thus, we have developed the Fly-Crane Shared Control module (FSC) to assist the operator. Such a control strategy aims at allowing the operator to focus solely on the end-effector and the proper execution of the task, while the remaining degrees of freedom and the control of each robotic agent are automatically managed.
The control strategy, represented in Fig. 5, implements two flight modes: -CRANE: the modality allows the operator to control the position and orientation of the platform/gripper. It is used during pick-and-place and manipulation operations; -MANUAL: this mode consists of two sub-modes. The first sub-mode maintains the formation shape and lets the human perform a rigid translation of the entire team; the second sub-mode allows the operator to control the position of a single robot while the other two remain stationary. This mode is used during take-off and landing operations and in case of emergency.
Depending on the active flight mode, the FSC module receives and interprets the commands from the human and computes

Modeling
To present the derivation of the FSC, let us briefly introduce the kinematic model of the G-Fly-Crane (more details are available in [20]). The main variables are schematically represented in Fig. 1. We define two frames: The platform is attached by pairs of cables of the same length to three robots whose position w.r.t. F W is given by the vectors p Ri ∈ ℝ 3 with i = 1,2,3, respectively. The configuration of each pair of cables w.r.t. F P is represented by the angle i ∈ ℝ between the plane {x P ,y P } and the plane shaped by the cables attached to robot i.
The system configuration is then entirely described by the variable q = [p ⊤ We also collect the position of all robots in the vector p R = [p ⊤ R1 p ⊤ R2 p ⊤ R3 ] ⊤ which can be written as function of q [20], p R = f(q). Differentiating such a kinematic relation, we obtain the differential kinematic model: is a square invertible matrix in C , except for some singular configurations investigated in [21].

CRANE Flight Mode
In this mode, the human can directly specify the endeffector pose, i.e., the position and orientation of the platform, p P and η P , respectively. To do so, we rely on the inverse kinematics controller presented in [20]. Given a desired system trajectory in the configuration space , the reference velocities of the aerial vehicles are computed as >0 is a positive definite matrix and e q = q d − q . In [20], we showed how to optimally compute k q to guarantee the system stability and the minimum H ∞ gain under parametric uncertainties. We then integrate v R ⋆ to obtain the position reference p R ⋆ . Both are sent to the geometric position controller [13] running on each aerial vehicle, which implements a PID action based on the position error. We remark that in the considered context, the integral term is sufficient to compensate external disturbances and changes of the platform mass during pick-and-place operations.
The desired system trajectory in the configuration space q d (t) is defined in a shared way between the human operator and an online local planner. The human specifies the platform position and attitude, while the local planner resolves the redundancy of the system. Thus, we map the joystick commands available to the operator into desired translational and rotational velocities of the platform, i.e., v d P and ̇ d P , respectively (see Section 4.4). Such quantities are then integrated (starting from the initial configuration) to obtain the desired position and rotation of the platform, i.e., p d P and d P , respectively. The final quantities to accomplish the desired configuration trajectory, namely α d , are computed through an online local planner [19] as the result of a optimization problem: where J 1 (q),J 2 (q) are two configuration-dependent cost functions and 1 , 2 ∈ ℝ >0 are the corresponding weights.
In particular, where f Ri ∈ ℝ ≥0 is the thrust intensity delivered by each robot, which depend on the configuration and the known parameters of the system. Solving Eq. 3, the local planner computes the cables angles that balance the effort among all the robots (J 1 (q)) while maximizing the manipulability index 4 (J 2 (q)). The time derivative of α d can be computed numerically or simply assumed zero in the domain of interest. Notice that the distribution of efforts among robots, or in other words the value of J 1 (q), also depends on the position of CoM of the platform. In this first proof of concept, we designed the platform and the gripping mechanism such that the position of the platform CoM can be assumed constant during the pick-and-place operations.

MANUAL Flight Mode
This flight mode allows the operator to command the position of each aerial vehicle. As for the CRANE flight mode, the operator uses the joystick to send the desired velocities, which are then integrated to obtain the desired position. In this specific flight mode, the operator can choose between two sub-modes: -MANUAL-1: all robots are moved as a rigid body such that , -MANUAL-2: only one robot is displaced in an independent way while the others maintain their position. In particular v Ri is the velocity command provided by the user.
The MANUAL-1 modality is mostly used for take-off and landing operations while MANUAL-2 in case of emergency. For example, if one the robots face a technical issue, the operator can relocate the others until they reach a safe location, assuming the load and/or the platform can be detached.

Human Interface
The operator interacts with the robotic system through a standard joystick (see Fig. 4). In this section, we describe how the human commands are mapped into directives for the system according to the flight mode. To keep the human interface as similar as possible to the one used by multi-rotor pilots, we employed a similar joystick set up by providing two analogue sticks with two axes each, (a 1 , a 2 , a 3 , a 4 ) with a i ∈ [− 1,1], and several buttons.

CRANE flight mode:
In this mode, commands available to the human are mapped into desired linear and rotational velocities of the platform. Mimicking a standard multi-rotor control interface, two axes of one stick are mapped into yaw rate and linear velocity along z W : where k , k z ∈ ℝ are scaling gains.
Through axes a 1 and a 2 , the human operator can control the yaw angle and the altitude of the platform, respectively. To control the planar position and attitude along the remaining x P and y P axes, we use the same joystick axes, a 3 and a 4 , according to the status of a button, called B ∈ (0,1). The mapping between the axes a 3 ,a 4 and the desired system commands depending on the value of B are described in Table 2.
In practice, with the joystick axes a 3 and a 4 , the operator controls the platform attitude when the button B is pressed, and the platform position otherwise. Separating the control of the horizontal position and the attitude, as well as making the human interface similar to the one used for standard multi-rotors, make the G-Fly-Crane operations simpler and more intuitive.

MANUAL flight mode:
In this mode, the mapping is similar to the previous mode, with the difference that the axis a 1 and the B button are ignored. In particular: Additional joystick buttons are used to choose the mode MANUAL-1 or MANUAL-2 and select the robot to be moved.

Results
To demonstrate and validate the pick-and-place and manipulation capabilities of the G-Fly-Crane, we present experiments performed in the indoor scenario shown in Section 2. A video showing the experiments is provided as supplementary material. In these experiments, the G-Fly-Crane is operated by a single person via a Logitech F310 gamepad, supported by the presented shared control strategy.
In the first experiment, the task is to complete a wall by placing two missing bricks. Figure 6 shows the different phases of the experiment: 1. Take-off: the user commands the vertical position of the robots in MANUAL-1 mode to lift the platform; 2. Pick first brick: the user commands the position of the platform in CRANE mode (B = 0) placing the platform on top of the brick and triggering the magnetic gripper; 3. Place first brick: using the same mode, the user commands the G-Fly-Crane to lift the brick, bring it to the target position, and place it on the uncomplete wall; 4. Pick and place second brick: operations 3 and 4 are repeated to pick and place the second brick. However, the second brick has to be placed on the tilted wall. Therefore, when the platform is above the target position and before placing the brick, the operator commands the orientation of the platform (B = 1) until the brick is parallel to the wall slope; 5. Landing: finally, the G-Fly-Crane is driven to its initial position for landing.
Considering the more challenging pick-and-place operation of the second brick, Fig. 7 shows the position tracking of the G-Fly-Crane during the phases 5, 6, 7, and 8 starting . Notice that during the pick operation (phase 5), the position error along z W increases. This is because the user commands a desired platform position such that the magnetic gripper slightly goes inside the brick. Thanks to the natural compliance of the cables, this does not constitute a problem, and the brick can be gently attached once perfectly in contact with the gripping system. During the lift of the brick (phase 6), again, the position error along z W increases. This is due to the additional mass of the brick that is compensated by the integral action of the robots' position controller only after a few seconds (around t = 225[s]) when the error goes back to zero.
The tracking of the desired trajectory for the attitude and α angles, respectively, are shown in Fig. 8. The noise in these measurements is due to the internal forces acting through the cables and other disturbances generated by communication delays, wind turbulence caused by the rotors of each aerial vehicle, etc.
Let us define the position, attitude and cable angles errors as e p P = p d P − p P , e P = d P − P and e = d − , respectively. In Table 3 we report the mean and variance of the norm of the previous errors. Figure 8 also shows the extra thrust for each robot, f Ri , defined as the total thrust minus the amount of thrust needed by each robot to hover the platform without the load. Thanks to the local optimization of the cables angles, α, the thrust values of each robot remain always close to each other indicating a good balance of the effort among the quadrotors. This keeps the energy consumption similar for each of the robots.
At t = 270[s], during phase 8, the brick is located over the wall surface (see video provided as supplementary material). Interestingly, as shown in Fig. 8c, the extra thrust for each quadrotor has a sudden decrease because the weight of the brick is temporarily compensated by the contact with the wall.
Additionally, we performed a second experiment further showing the manipulation capabilities of the G-Fly-Crane (see video provided as supplementary material). This test emulates the decommissioning of a structure where bricks must be removed from the inclined wall. As in the previous experiment, the G-Fly-Crane takes off and the operator places it on top of the target brick. Before descending to perform the pick, the pilot provides the commands to adjust the platform orientation according to the inclination of the brick. The brick can be then removed from the wall safely and accurately, thanks to the dexterity provided by the G-Fly-Crane.

Discussion and Conclusions
For the first time, we have experimentally demonstrated the physical interaction capabilities of an aerial multi-robot system for complex pick-and-place and manipulation operations in the context of construction scenarios. We have proposed a shared control strategy that allows a single person to operate such a complex robotic system. As a result, the human has complete control over the manipulation operations rather than having to separately provide commands to each quadrotor. The obtained results successfully  represent the first proof of concept for the potential use of the G-Fly-Crane in construction. However, we recognize that the presented experimental validation was conducted in an indoor laboratory setting and that there are several challenges ahead in bringing this technology to real-world applications. One of these challenges is the localization of the system (the robots and the platform). In general, a motion capture system (such as the one used in this work) is unsuitable for large outdoor workspaces, and the accuracy provided by a global positioning system is insufficient to perform precise manipulation tasks. A possible solution is the combination of a global positioning system with cameras and other onboard sensors, using VIO algorithms and sensor fusion techniques. Another key consideration is robustness, which is especially significant when dealing with diverse atmospheric conditions or when the system has to interact with the environment. Therefore, in addition to a more robust mechatronic design, robust control approaches will be necessary to enhance the proposed shared control strategy. Moreover, the optimization of the redundancy, i.e., the cables angles, is only local in our current implementation. A global planning method considering internal and external constraints (e.g., obstacles) in addition to an optimal balance of the payload among the robots would be relevant tools to increase robustness and to ease the system's operations. Furthermore, in the future, when new regulations will allow for manipulation activities to be performed with such complex aerial robotic systems, fully autonomous execution of the tasks could be envisioned. For this first proof of concept, we deliberately simplified the grasping problem by considering a magnetic gripper that can easily grasp and release bricks in our demonstration scenario. Nevertheless, the application of the G-Fly-Crane technology to real-world construction tasks will require the integration of more sophisticated automated grasping methods and grippers suited to the shape and material of the manipulated parts. The mentioned improvements are considered as future works.