Performing low-altitude photogrammetric surveys, a comparative analysis of user-grade unmanned aircraft systems

This paper shows results of comparing performances of four unmanned aircraft systems (UAS) in terms of photogrammetric survey’s quality. This study aims to investigate what is the more suitable UAS for specific applications considering the required scale factor, such as for architectural, environmental, and restoration purposes. A series of photogrammetric surveys were conducted in a hilly area of about 5 ha using Phantom 4 Adv, Mavic 2 Pro, Mavic Air 2, and Mavic Mini 2. These unmanned aircrafts are commercial user–grade systems used mainly by private professionals. Several photogrammetric reconstructions were performed by varying essential parameters, such as flight altitude and cameras of remotely piloted aircraft systems (RPAS), applying structure-from-motion (SfM) algorithms to the images taken from the UAS. The surveys’ quality was analyzed by comparing the ground targets’ coordinates extrapolated from the point clouds to those measured on the field with indirect georeferencing through GNSS technology. Fifty targets were installed and arranged following a reasonably regular mesh. The boundary conditions were maintained the same for each flight mission, flight trajectories, and the ground control point distribution on the ground. For each survey made by each of the four UAS, altimetric and planimetric residuals were reported and compared. Average residuals from Phantom 4 Adv, about 15 mm, almost disappear compared to the other UASs; the discrepancy is one order of magnitude. With a regular grid geometry of ground targets, the Mavic Mini 2 led to an error average of about 5 cm. Remembering that the Mavic Mini 2 is an ultralight drone (does not require a pilot's license), it could significantly reduce cost compared to the other systems.


Introduction
Structure-from-motion (SfM) photogrammetry is often used as a topographic modelling technique. It combines the utility of digital photogrammetry and ease of use derived from multi-view computer vision methods. Thanks to the increasing availability of imagery, particularly from unmanned aerial vehicles, SfM photogrammetry represents a powerful tool (James et al. 2019).
Due to the technical improvements and miniaturization of avionics and quality advancements of digital cameras, UASs have been increasingly used as remote sensing platforms (Parisi et al. 2019;Rau et al. 2016;Sarwar et al. 2016;Turner et al. 2013).
At the same time, SfM photogrammetric processing has played an increasing role in delivering digital elevation models (DEMs) from UAS-based imagery (James & Robson 2014). Several commercial software, such as Agisoft Metashape, Meshroom, and 3DZefir, offer automated photogrammetric reconstruction routines. Investigating photogrammetric error and the uncertainties associated with SfM photogrammetric results are crucial tasks.
Mapping with unmanned aerial vehicles (RPASs) typically involves the deployment of ground control points (GCPs) to georeference the images for generating topographic models (Hugenholtz et al. 2016). Even if recent UAS are equipped with direct georeferencing systems (Gabrlik 2015;Pfeifer et al. 2012;Sanz-Ablanedo et al. 2018), due to the poor performances of the low-cost inertial measurement units hosted by the tested vehicles, we performed indirect georeferencing (Ekaso et al. 2020;Eling et al. 2015;Stöcker et al. 2017).
Depending on the type of representation that a performed topographic survey has to deliver, a specific type of instrument can be adopted for the survey. For architectural drawing, for instance, 1:50 or 1:100 graphical outputs have been often used (Bonora et al. 2021;Sun & Zhang 2018). For other applications, such as vast landscape, landslides, or riverbeds surveying, smaller graphical scales have been used (Bolkas et al. 2018;Gracchi et al. 2021;Michez et al. 2016).
For other applications, such as vast landscape, landslides, or riverbeds surveying, smaller than 1:1000 graphical scales have been used (Bolkas, 2019;Gracchi et al. 2021;Michez et al. 2016;Mucchi et al. 2018). This study considers scale factors smaller than 1:100 only; to achieve a 1:50 scale factor, a planimetric error of less than 1 cm must be guaranteed and it is generally out of the range of drones. Considering results from the resulting accuracy on a cartographic representation, some considerations can also be made. It represents the uncertainty associated with the graphically represented information; historically, ± 0.2 mm is the minimum distinguishable value from the human eye without a lens. In general, the graphic error depends on the scale of the map, as shown in Table 1.
Nowadays, in which CAD software or digital maps allow for almost infinite enlargements, the graphical error is still the parameter that governs measurement accuracy based on the client's requests. For example, to return the survey on a scale of 1:1000, where the graphic error is ± 20 cm, it will not be necessary to go up to an accuracy of less than 5 cm, as this would only involve a waste of energy and unnecessary costs. In photogrammetric topographic surveys from UAS, some authors worked on scale such as from 1:3000 to 1:100 (Barba et al., 2019;Lane et al., 2000). Obtaining a product on a scale greater than 1: 100 is not possible with RTK mode; for this reason, the considerations will be carried out starting from the scale factor 100. The altimetric error can be traditionally considered double in topography compared to the planimetric one. The required threshold value on the Z coordinate for three-dimensional can be regarded as equal to twice those imposed on planimetric axes.
Recalling that the ground sampling distance represents the size of the pixel on the field and is a function of the focal length of the camera, flight altitude, and size of the sensor's pixel, it is a parameter that sets a lower limit to the precision achievable on the points on the ground. The GSD value of the 80 m height above ground level (AGL) flight of the Phantom 4 Adv is 2.1 cm.
Tuning the choice of an appropriate surveying technique, considering the expected result in terms of graphical output, could help optimize the campaign costs and find a good balance between available resources and expected outcomes.
Integrating GNSS control network and photogrammetric technique to design, implement, and perform a rigorous topographic survey methodology has been depicted (Forlani et al. 2019;Gabrlik et al. 2018).
The quality of a 3D model mainly depends on the survey's quality and the photogrammetric reconstruction process. The survey's quality, in terms of accuracy, is dependent on various parameters: method, performances of UAS avionics, quality of cameras, the accuracy of GNSS observations (Lee & Choi 2016), camera calibration (Fraser 2013) (Remondino & Fraser 2006), and georeferencing method (Forlani et al. 2018).
This This research has been carried out to investigate outcomes of a series of photogrammetric surveys performed through four DJI UAS different models, Phantom 4 Adv, Mavic 2 Pro, Mavic Air 2, and Mavic Mini 2. Predominant national and international regulations are increasingly favoring small drones in urban areas (Alamouri et al. 2021;Marshall 2021;Rango & Laliberte 2010). For this reason and considering a wide variety of urban applications for restoration purposes, we focused the tests on small weight drones. The aircraft is part of commercial user-grade systems primarily used by private professionals. On the one hand, thanks to their offthe-shelf configurations, they can help in rapidly planning and performing low-altitude surveys.
On the other hand, due to their extraordinary easy-to-use vocation, they are often deployed, paying little attention to photogrammetric best practices. Following these considerations, the tests have been designed to reproduce common critical issues such as poor planning of camera network geometry (Dai et al. 2014;Nocerino et al. 2013), camera autocalibration, and different flighting AGLs. The tested UASs that present different configurations achieve different overall mission performances and survey quality.

UAS
Four off-the-shelf consumer-grade UAS, namely, Phantom 4 Adv, Mavic 2 Pro, Mavic Air 2, and Mavic Mini 2, have been used. In Table 2, the main specs have been reported.

GNSS receiver
The used GNSS receiver has been the TRIMBLE R8s system with a 2-m-high pole and bipod support to guarantee a steady equilibrium during acquisitions. The observations were made in real-time kinematic (RTK) mode with area correction from NETGEO permanent network (NRTK). A number of satellites higher than 12 were verified for each positioning, which was carried out with 3 acquisitions of 10 epochs each. The measured values were transformed using the Verto software [45], developed by the Istituto Geografico Militare (IGM) with GK2 grid and georeferenced in the EPSG 3003 reference system (Gauss Boaga fused west). The three measured values were averaged, and this value was considered as a reference on which to perform both the checks and the photogrammetric frames. For altimetric measurements 1.5 cm error and 0.8 for planimetric measurements were considered.

SfM software
The SfM technique has been implemented through automated photogrammetric reconstruction routines. Concerning the photogrammetric reconstruction, the Agisoft Metashape's professional version (1.6.6) has been used. The software works through a standardized processing pipeline: structure from motion automatic processing to image block orientation ( Fig. 1), generating a 3D point cloud of the acquired scene, causing a triangular mesh from the point cloud, creating raster products such as digital elevation model (DEM) and orthophotos [46]. As first step, the images have been imported without camera specifications and have been filtered following a quality threshold. By applying EXIF georeferencing information, the software then estimated interior and exterior parameters. GCPs and CkPs were measured trough GNSS receiver and manually selected on the project images as a second step, 51 targets were selected. The GCPs were then selected as a constraint during the bundle block adjustment (BBA) procedure to put the photogrammetric reconstruction within a local coordinate system. CkPs were selected as check points. Once the bundle adjustment processes had been performed, exterior and interior camera parameters were adjusted accurately. A comparison between GCPs and CkPs model coordinates and the coordinates observed by the GNSS survey has been performed to assess georeferencing process accuracy. The accuracy has been expressed in pixels and meters. Root mean  square error has been calculated for the GCPs and CkPs to better depict the error distribution in the overall study area.

Surveying campaign
The performances of the various drones were investigated, flying over an inclined terrain. The surveying campaign was performed within 3 days. During the first 2 days, target arrangement and GNSS survey were performed. The photogrammetric flights were carried out during the third day to maintain a reasonable stability of boundary conditions as wind, temperature, humidity, and cloud coverage. The flights were carried out over a portion of land, including an olive grove, a vineyard, and some buildings (Fig. 2). Fiftyone targets 0.3 × 0.3 m sized (Fig. 2) were positioned on the ground based on a relatively regular grid and fixed on the ground using stable anchoring supports.
Furthermore, a topographic nail has been solidly secured in each target's center, allowing for an accurate GNSS survey. The targets' coordinates were measured with GNSS observation using a 2-m stick. The observations made through local area correction with a local station have been performed stationing on each point for three acquisitions of 10 epochs each. The average value of the three observations has been considered for each GCP. An instrumental 15-mm altimetric error and a 7-mm planimetric error were considered. The coordinates have been transformed using a local grid and framed in the EPSG 3003 reference system (Gauss-Boaga West fuse). The targets (Fig. 3) have been used as ground control points (GCPs) and check points (CkPs) to improve and verify the quality of the photogrammetric reconstruction.

Performing surveying
The surveys have been performed using the four UAS models described in the previous section. A regular speed and a comparable overall flighting dynamic have been adopted to guarantee a more stable flight. In particular, the surveying operations have been performed using automatic flight mode for Phantom 4 Adv and Mavic 2 Pro. For Mavic Air 2 and Mavic Mini 2, the manual mode has been used as the mission planning software was not available. The flying AGL has been maintained constant both in manual and automatic missions. However, a flighting chart has been used during flighting operations to maintain the same root followed by the automatic flights and the same speed. In this way, the overlapping images have been held close to the one obtained through the automatic flight mode. The study area has been divided into South-West and North-East (Fig. 4) sections to reduce the error due to the slope inclination. The complete area coverage has been performed, planning two missions for each UAS, one for each area. UAS performances for different ground sampling distances (GSD) have been investigated in terms of photogrammetric efficiency, performing flights at four different AGLs for each mission. For Phantom 4 Adv and Mavic 2 Pro, flights were performed at 30, 45, 60, and 80 m AGLs (Table 3). Due to logistics reasons, for Mavic Air 2 and Mavic Mini 2, the flights have been carried out for 30 and 60 m only. Table 2 reports for each UAS and for various flying heights the ground sampling distance on the ground. Values were calculated following Eq. (1). (1) where X_GSD is the GSD, H is the flying height, f is the focal length, and x_img is the sensor pixel size. A 60% side overlap and an 80% end overlap were adopted.

Results
The following results have been obtained performing flights at pre-established altitudes (30, 45, 60, 80 m) with a nearly regular GCPs grid on the ground for each of the UAS.

Phantom 4 and Mavic 2 Pro
For the Phantom 4 Adv and Mavic 2 Pro, the whole study area has been considered; for the Mavic Air 2 and Mavic Mini 2, the NE area only has been considered. The shorter distance between two consecutive GCPs is about 40 m. For  In Table 4, GCPs' and CKPs' residuals are calculated on the photogrammetric reconstruction made by Phantom 4 Adv's pictures. The worst case is represented for the 80-m altitude. The higher residual value is lower than 0.025 m. Figure 5 reports residuals for X, Y, Z axes and 30-, 45-, 60-, and 80-m altitude for Phantom 4 Adv.
In comparison with Phantom 4 Adv, the Mavic 2 Pro led to worse results. The total deviation varies from 55.6 cm at 30 m to 18.9 cm at 60 m. Best results have been obtained at 60 and 80 m AGL. Also, the average deviations on the ground control point and checkpoint can be considered homogeneous in this situation.
Attention was placed on targets 306 and 406, from which it can be observed again how the vertical component of the error is prevalent (Table 5). Figure 6 reports residuals for X, Y, Z axes and 30, 45, 60, and 80 m AGL for Mavic 2 Pro on target 306. The values reported in the chart for each flight AGL represent errors on CKPs.

Mavic Air 2 and Mavic Mini 2
In this case, the surveys have been carried out within the NW area only (Fig. 7). Two targets, 308 and 408, belonging to the central part of the survey area were randomly chosen to compare different flights and different UASs. Table 6 and Fig. 8, respectively, resume residuals in X, Y, and Z axes measured during a photogrammetric survey made by Mavic air 2. Table 7 and Fig. 9, respectively, resume residuals in X, Y, and Z axes measured during a photogrammetric survey made by Mavic air 2.
The Mavic Mini 2, unlike the Mavic Air 2, despite the relatively small size and weight (< 250 g), has interesting results. The deviations calculated from the photogrammetric reconstruction show good potential, especially in the case of flying at 60 m, where the errors are even lower than the Mavic 2 Pro.
Hereafter, a comparison of residuals for 60 m AGL flights of the four UAS (Table 8) is shown.    All things considered, average residuals from Phantom 4 Adv, about 15 mm, almost disappear compared to the other UASs; the discrepancy is one order of magnitude. We can even assert that Mavic Air 2, limited to the proposed set up and to the border conditions on which tests have been performed, could be difficult to use for topographic survey purposes. The average error is around 10 mm. The Mavic 2 Pro and the Mavic Mini 2 show similar planimetric residuals. The Mavic 2 Pro is better for elevation error; however, the Mavic Mini 2 demonstrated good performances. This last represents the most surprising result of this UASs comparison. With a regular grid geometry of ground targets, the Mavic Mini 2 led to an error average of about 5 cm. Remembering that the Mavic Mini 2 is an ultralight drone (does not require a pilot's license), it could significantly reduce cost compared to all the others.

Discussions
Phantom 4 Adv brought excellent results for the four analyzed flight AGLs. The errors reported for the three axes are around 2 cm. With a minimal variance, we can say that values are similar for all ground targets; the point clouds have been close settled around to the GCP allowing for the same CKP accuracy.
Two targets belonging to the central part of the survey area were chosen to make a more immediate comparison amongst different flights made by different UASs: the 306 GCP and 406 CKP.
The prevailing error is the planimetric one; on target 406 (CKP), the predominant deviation is in the vertical direction z. This statement is valid on targets 306 and 406 and a general level on all GCPs and CKPs. Furthermore, it is possible to see how the 80 m has led to slightly worse results than the other flight AGLs, which can be considered similar in terms of obtained results.
A targets' single raw of the ground target grid was chosen to carry out a general comparison on the targets, formed by a GCP (107,308,506,703) and a CKP (207,408,605) alternately. As previously highlighted, the geometry of the ground points' grid ensures that there are no significant differences between GCP and CKP. Figures 10 and 11 shows the planimetric deviations on the targets; in Fig. 12, the altimetric deviations.
The next Tables 9, 10, 11, 12, 13 report statistics of the survey in terms of median and standard deviation (STD) for planimetric and altimetric error on targets GCP (107,308,506,703) and a CKP (207,408,605). Statistics are calculated for GCPs, CKPs, and the total amount of targets. Table 13 shows a STD value for GCPs s and CKPs substantially equal on planimetric error. A slight difference between GCPs and CKPs is otherwise reported for altimetric error. The total altimetric error is 1 cm higher than the planimetric one. Figures 12 and 13 show medians for planimetric and altimetric error on targets GCP (107, 308, 506, 703) and a CKP (207,408,605). The median value for the altimetric error of Mavic Air 2 is twice with respect to the others. Figures 14 and 15 show standard deviation for planimetric and altimetric error on targets GCP (107, 308, 506, 703) and a CKP (207,408,605). Even in this case, Mavic Air 2 reached worse results. Tables 14 and 15 show the flight AGL suitable for a photogrammetric survey respecting the graphical error limits imposed by the required representation scale and GSD. The values indicate for each drone the flight AGL at which it is possible to fly to ensure the success of a survey at the defined representation scale in terms of planimetric error.

Conclusions
By varying essential parameters such as flight AGL and cameras (RPAS models), several photogrammetric reconstructions were performed applying structure-from-motion (SfM) algorithms using the images taken from the UAS. The surveys' quality was analyzed by comparing the ground targets' coordinates extrapolated from the point clouds to those measured on the field with indirect georeferencing through GNSS technology.
Looking at the results, the difference between GCPs and CKPs, in terms of error, is moderated. If, usually, the error associated to CKPs should represent the more severe quality control parameter, in this case for some UAS the GCP error is higher than the one from CKPs.
The Phantom 4 Adv confirmed the expectations, one of the most used drones for photogrammetry. All four flight AGLs used guarantee accuracy limits to the 1:200 scale. Flight AGLs up to 45 m can generate 1:100 products.
The Mavic 2 Pro cannot assure an acceptable average error for scale factors 100 and 200; however, it is suitable from 1:500 upwards.
The Mavic Air 2 is difficult to be used for 1: 100 and 1: 200 scales. It is within 1: 500 at an AGL of 30 m. It is also worth noticing that the sensor is a 48-MP pixel 2 × 2 binning. With 2 × 2 binning, four adjacent pixels are binned into one larger pixel and readout.
The Mavic Mini 2 has exceeded expectations; at the height of 60 m, it could be used for a 1:200 scale. The flight at 60 m resulted better than at 30 m: this could be due to the non-optimal network camera geometry. A low signal-noise ratio, which is probably due to the sensor size (1/2.3" for 12 MP), could even play a role.