Abstract
In this study, we investigated the accuracy of surface models and orthophoto mosaics generated from images acquired using different data acquisition methods at different processing levels in two urban study areas with different characteristics. Experimental investigations employed single- and double-grid flight directions with nadir and tilted (60°) camera angles, alongside the Perimeter 3D method. Three processing levels (low, medium, and high) were applied using SfM software, resulting in 42 models. Ground truth data from RTK GNSS points and aerial LiDAR surveys were used to assess horizontal and vertical accuracies. For the horizontal accuracy test, neither the oblique camera angle nor the double grid resulted in an improvement in accuracy. In contrast, when examining the vertical accuracy, it was concluded that for several processing levels, the tilted camera angle yielded better results, and in these cases, the double grid also improved accuracy. Feature importance analysis revealed that, among the four variables, the data acquisition method was the most important factor affecting accuracy in two out of three cases.
Similar content being viewed by others
Avoid common mistakes on your manuscript.
Introduction
Currently, UASs play an increasingly pivotal role in the realm of spatial data collection. Unlike conventional air and satellite-based remote sensing methods, the UAS offers significant advantages in facilitating data acquisition and adapting to the specific requirements of different survey studies with a variety of spatial and temporal resolutions. This versatility is further underscored by the utilization of a broad spectrum of sensor types, all of which are achieved at a comparatively low cost (Yao et al. 2019; Iglhaut et al. 2019; Nex et al. 2022). The SfM method is a key solution for processing aerial imagery obtained by UASs. Notably, the parameters are integral to the processing workflow, including camera calibration, bundle adjustment, and orientation parameter adjustments, all of which are implemented in a user-friendly manner (Uysal et al. 2015; Iglhaut et al. 2019; Deliry and Avdan 2021).
The increasing adoption of UAS-based SfM techniques for mapping and modeling is evident from its growing utilization across various domains, including agriculture, environmental studies (Akay et al. 2022; Dai et al. 2022), and industrial applications (Haas et al. 2016; Rauhala et al. 2017), as well as urban environments (Ahmed et al. 2021). In contrast to previous methods dominated by aerial and terrestrial laser scanning for high-resolution topographic data production, UAS SfM demonstrates superior efficiency through its capability for flexible, target-oriented point cloud generation at resolutions approaching sub-decimeter scales, thereby facilitating more intricate analyses (Manfreda and Eyal 2023).
The efficacy of UAS SfM compared with laser scanning has been substantiated in various contexts. For instance, a comparison of the accuracy of the UAS-based SfM and the LiDAR digital surface model (DSM) in an urban area for flood estimation yielded inconsequential disparities (Escobar Villanueva et al. 2019). Studies also investigated the feasibility of employing the UAS-based SfM for pre-disaster condition assessment, determining that its vertical error tolerance (sub-decimeter) qualifies it for comparative analyses before and after disasters (Kucharczyk and Hugenholtz 2019). Furthermore, the method’s morphometric accuracy was evaluated alongside a terrestrial laser scanner (TLS), particularly for documenting historic districts (Parrinello and Picchio 2019). Additionally, UAS-based aerial mapping has proven invaluable in generating foundational data for estimating the roof area of agricultural structures, facilitating the efficient planning of solar panel installations (Koc et al. 2020). In built-up areas, Liu et al. (2022a) investigated the effects of UAS flight altitude and image overlap degrees, while Ahmed et al. (2022) explored the influence of varying camera angles and flight directions on the accuracy of UAS-derived outputs, namely, DSM and orthophoto mosaics.
The accuracy of the method must be rigorously assessed to ensure the efficiency and reliability of UAS SfM-based data. This entails quantification of the horizontal errors inherent in the produced orthophoto mosaics and vertical errors within the DSMs. The exigency for such assessments is grounded in the profound impact of these parameters on the overall uncertainty associated with the generated models (Inzerillo et al. 2018; Belcore et al. 2021). Various factors pertaining to UAS-based SfMs can influence the reliability of generated models. These include properties associated with direct or indirect external orientation parameters, such as the scale of GNSS measurement errors, the number and spatial distribution of ground control points (GCPs), camera characteristics (e.g., tilt of the camera principal axis, focal length, lens distortions), diverse weather conditions (e.g., lighting and atmospheric properties), and the quality of the resultant images (e.g., sharpness, resolution, blur). In addition, the processing software environment significantly contributes to the quality of the derived datasets (Barkóczi and Szabó 2017; Sanz-Ablanedo et al. 2018; Lehoczky and Siki 2020; Casella et al. 2020).
The significance of external orientation has been prominently underscored by numerous studies dedicated to revealing its details and effects. These investigations have collectively determined that an optimal spatial distribution of GCPs is achieved through their homogeneous and nonlinear dispersion within the study area (Manfreda et al. 2019; Villanueva and Blanco 2019; Gomes Pessoa et al. 2021; Ulvi 2021). The establishment of universally accepted principles dictating the minimum and optimal quantity of GCPs remains elusive, as studies employ varied benchmarking systems. For instance, Sanz-Ablanedo et al. (2018) investigated the influence of the number of GCPs per image on accuracy, revealing that beyond 3–3.5 GCPs per 100 images, no statistically significant improvement in accuracy was observed. Conversely, contrasting findings have been reported, indicating that there is no statistically significant enhancement in accuracy when the number of GCPs surpasses 10 per square kilometer (Liu et al. 2022a, b). In addition to the size of the area under consideration, the topography of the site markedly influences the required number of GCPs. Consequently, a site characterized by heterogeneous topography, with an average slope of 19–30%, may necessitate a higher number of GCPs (Agüera-Vega et al. 2017), in contrast to a homogeneous flat area (Reshetyuk and Mårtensson 2016).
An increasing number of investigations have emphasized the benefits associated with utilizing the oblique data acquisition method for evaluating UAS-based aerial data (Li et al. 2018; Wu et al. 2018; Vetrivel et al. 2018). The primary justification for this preference stems from the unique capability of oblique data acquisition to capture detailed information not only from the top but also from the sides of objects (Svennevig et al. 2015; Jaud et al. 2019; Cao et al. 2023), contrasting with nadir imaging. Its application is favored in scenarios where vertical or near-vertical surfaces dominate the study area. Research dedicated to quarry wall surveys has demonstrated that integrating nadir and oblique imagery improves the accuracy of DSMs while simultaneously addressing data gaps caused by shadowing effects (Rossi et al. 2017; Zapico et al. 2021). Similar results have been noted in investigations carried out in naturally high-relief regions (Nesbit and Hugenholtz 2019; Bi et al. 2021; Nesbit et al. 2022). The enhanced accuracy and heightened informational content evident in models created by integrating nadir and oblique images support the utility of this approach in computing the leaf area index (LAI) (Che et al. 2020; Lin et al. 2021). Furthermore, its progressive deployment in urban settings is attributed to its ability to provide adequate data on building frontages, which can be harnessed for precise generation of 3D building models (Verykokou and Ioannidis 2016; Wu et al. 2018; Pepe and Costantino 2020).
In addition to the aforementioned factors, subsequent investigations into the evaluation of UAS-SfM methods have focused on additional categories of variables that influence accuracy. These include parameters such as flight altitude and image overlap (Jeong et al. 2020; Anders et al. 2020; Liu et al. 2022b). However, the novelty of our research lies in (1) its comprehensive comparison of three distinct factors influencing accuracy, (2) the comparison of two different built-in study areas with different characteristics, and (3) the utilization of three reference groups, one horizontal and two vertical. Our study is unique in its simultaneous examination of these variables, which have not been studied together in this manner previously. This holistic approach allows for a more nuanced understanding of the interplay between various factors affecting accuracy in UAS-based surveys, particularly in urban environments. Furthermore, our investigation highlights the impact of processing software settings and duration on model quality and processing time, aspects often overlooked in accuracy studies. By addressing these gaps, our research not only advances the current understanding of UAS-based survey methodologies but also provides valuable insights for optimizing aerial mapping campaigns in urban settings. Considering our results, we aimed to underscore the universal limitations in applying the investigated survey conditions to enhance the general efficacy of urban aerial mapping campaigns with UASs.
Materials and Methods
Study Area
This research was conducted in two distinct study areas within Debrecen, Hungary, as illustrated in Fig. 1. The first area, designated as “Csapókert,” is situated in the north-eastern part of the city. It is characterized by a regular street network, expansive gardens, lower building density, and predominantly one- or two-floored structures. The second study area, “Úrrétje,” is in the north-western part of the city. This area features a denser and more irregular road network with smaller gardens and increased building density. It includes multistory blocks of flats, as well as terraced and detached houses. In contrast to Csapókert, Úrrétje exhibits a homogeneous flat terrain. Despite Debrecen’s overall flat landscape, there were disparities in the topography of the two study areas. Csapókert comprises two sand mounds ranging in height from 1.5 to 2 m, contributing to its topographical variation. On the other hand, Úrrétje represents a consistently flat terrain. The selection of these two areas was intentional and based on prior research findings (Kucharczyk and Hugenholtz 2019), which suggested that varying characteristics, such as building height, built-in area density, and spatial patterns, may influence the accuracy of both orthophoto mosaics and DSMs.
UAS Data Acquisition
Image acquisition was executed using a DJI Mavic Pro quadcopter, equipped with an integrated 12-megapixel 1/2.3″ sensor featuring a focal length of 26 mm. The camera is fixed to the drone through a 3-axis gimbal, enabling the adjustment of the camera tilt along the X-axis from − 90° (nadir) to + 30°. Aerial surveys were conducted in the two designated study areas at almost simultaneous times, with merely a 5-day difference between the two assessments. To ensure comparability, both study areas were surveyed during the same time of day (early afternoon), and strict consistency was maintained in the flight parameters, encompassing various aspects. Throughout the entirety of the data collection process, a consistent flight altitude of 90 m above ground level was maintained, with a frontlap and sidelap of imagery set at 70%. Image acquisitions were performed using three different flight modes (Fig. 2):
-
- Single grid: the UAS flies over the area to be mapped in a series of parallel paths in a defined direction.
-
- Double grid: at the end of the initial flight pass, the UAS flies along a series of passes but perpendicular to the previous one.
-
- Perimeter 3D: the UAS flew a single-grid plan and a square bounding the area to be mapped, the former with a camera angle of 90° (nadir) and the latter with a camera angle positioned at the spatial center of the area (Fig. 2).
For data collection, two single-grid flights were implemented, one conducted in an east–west (E-W) flight direction and the other in a north–south (N-S) flight direction, in addition to a double-grid flight pattern. All three flight modes were executed with a 90° nadir and a 60° oblique camera angle. The number of photos captured for each data acquisition method is listed in Table 1. To establish accurate external orientation parameters, the aerial data collection process was complemented with GCP surveys. Prior to aerial data collection, the GCPs were established by positioning the signs along the roadway. The coordinates of these GCPs were surveyed using a Stonex S9i RTK GNSS system, which demonstrated an average positioning error of approximately ± 2 cm. In total, 23 GCPs were distributed for the Úrrétje area, while 25 GCPs were designated for the Csapókert area (Fig. 3).
Reference Data
To evaluate the accuracy of the orthophoto mosaics and DSMs produced via SfM processing, two distinct types of reference data were utilized. In both study areas, an RTK GNSS device (Stonex S9i) was used to survey the GCPs and collect reference coordinates, as previously described. Given the nature of the areas, which are characterized by a high number of enclosed gardens, reference points were surveyed exclusively in public areas, typically on roadways. These points were employed to evaluate both the horizontal and vertical accuracies of the resulting orthophoto mosaics and DSMs. A uniform sampling approach was adopted for the analysis, involving the measurement of 100 points in each of the two study areas (Fig. 4). Additionally, to assess the vertical data accuracy, a LiDAR point cloud, acquired from a survey conducted using a RIEGL VQ-780II sensor, was utilized. Airborne laser scanning (ALS) data collection occurred 2 months after the UAS-based survey, with a flight altitude of 1078 m above ground level and an average point density of 16 points per square meter. DSMs were generated for both study areas at a spatial resolution of 0.25 m per pixel using LiDAR point cloud data. For this type of reference, 100 points were randomly generated for each area (Fig. 4). These points were then used to extract the height values from the rooftops of the buildings.
Data Processing
UAS-based aerial imagery was processed using the Agisoft Metashape Professional (v1.5.1) software environment.
In our SfM processing workflow using Metashape, four distinct stages are included, which encompass the following procedure:
-
I)
During the Align Photos stage, the software identifies tie points to establish connections between different images, thereby creating a block for subsequent block adjustments. The outcome of this process is a sparse point cloud with default software settings. Following this, georeferencing of the models was conducted using GCPs.
-
II)
A dense point cloud was created using the built-in dense point cloud option. This involved employing low, medium, and high processing modes for various types of acquisitions, with aggressive depth filtering applied in all cases. The processing modes correspond to different image resolutions, with the software calculating 1/8 of the pixels for low,1/4 for medium, and 1/2 for high.
-
III)
The generation of DSMs followed, utilizing the dense point cloud data with interpolation enabled, and the default resolution specified by the software according to the ground sampling distance characteristics of the lens and image acquisition.
-
IV)
Finally, orthophotos were generated using the previously derived DSMs. The mosaic blending mode was employed, and the resolution was determined using software.
SfM processing was performed on a dedicated Workstation PC with an Intel Xeon E5-2670 CPU (12/24 cores/thread), 64 GB DDR5 RAM memory, and a Nvidia Quadro K420 GPU (2 GB VRAM) configuration.
Accuracy Assessment
The horizontal accuracies of the orthophoto mosaics were determined by calculating the absolute distances between the reference and model coordinates along the X- and Y-axes using trigonometry. These distances were treated as absolute (i.e., positive) values and did not incorporate directional information. For the DSMs, the disparities between the reference data (elevation data obtained from GPS measurements for roads and extracted elevation values from LiDAR DSM for roofs) and elevation values derived from the generated models were assessed.
Initially, the obtained difference values were analyzed using standard descriptive statistical methods. The RMSE was subsequently determined for both vertical and horizontal tests by evaluating the differences between the reference and model data.
Modeling with Residuals
The residuals, which denote the disparities between the SfM models and the reference date, were subjected to evaluation using random forest regression. In this analysis, the dependent variable was the residual values, while the independent variables included the resolution (level), flight pattern (image acquisition), flight direction (direction), and the study area. The reference data were divided into training and testing datasets at a 70:30 ratio, and the models were trained using the training set. The modeled values for the testing data were subsequently calculated, and the minimum, median, and maximum values were selected for further analysis by employing explainable machine learning techniques.
The models were evaluated using explainable AI metrics. The rank of the feature importance was determined by performing 50 permutations, and the contribution of the independent variables was evaluated using breakdown plots. These plots provide a visual representation of the variables’ contributions to a specific value of the dependent variable, indicating both positive and negative contributions. The calculations were carried out using R 4.2.2 (R Core Team) with DALEX and the forester packages.
The whole data collection, processing, and analysis process is shown in Fig. 5.
Results
Horizontal Accuracy
Comparing the error ranges between the two study areas using the same method and processing level, it was observed that in 19 modeling cases, the Úrrétje area exhibited smaller differences. However, for the Csapókert area, only the MPL (0.23 m) and LPL (0.37 m) processing levels of the P3D method demonstrated a better performance (Fig. 6). In terms of the RMSE values for distances, it was noted that only the P3D LPL mode yielded better results (0.5 cm) for the Csapókert area compared to the Úrrétje area. Importantly, for the same processing mode and level between the two study areas, the discrepancy in the RMSE values was less than 2 cm.
The results demonstrated consistent patterns when comparing data collection methods in both study areas. Specifically, at the same processing level, the NG1 method consistently yielded the smallest error range, whereas the P3D method consistently yielded the largest error range. Correspondingly, for RMSE values, the NG1 method consistently provided the most accurate results, whereas the P3D method resulted in the least accurate outcomes (Table 2). However, when specifically considering RMSE values for oblique images, it was observed that the double-grid flight employed for both study areas outperformed the single-grid flight at all three processing levels, yielding better overall results.
Comparing the three processing levels examined when looking at the range values for both study areas, there were pairs of results: for example, in the NG1 processing Úrrétje area and in the OG2 method’s Csapókert. In the investigation of RMSE values, there were no cases in which lower processing levels resulted in better accuracy of the Úrrétje results. In addition, for Csapókert, only the OG3 method had a 0.2 cm higher MPL than the HPL.
According to the feature importance, the image acquisition method (0.0303) had the greatest impact, whereas the processing level (0.0298) had the least impact on the error rates. Except for the acquisition mode, the other three features were observed with minimal differences (Fig. 7). Breakdown analysis revealed that the Csapókert area and the oblique acquisition mode produced the best results with the smallest error.
The processing level (medium) did not significantly affect the error, whereas the double-grid method contributed to the error reduction. In terms of the average error value, the error was reduced by factors such as the Úrrétje area, camera angle, and N-S flight direction. Conversely, a low processing level increases error. Regarding the maximum error, the Úrrétje location contributed to error reduction, whereas the P3D method, low processing level, and the associated flight direction with P3D increased the error.
Vertical Accuracy—Road Surface
When comparing the two study areas, variations in error ranges were observed, depending on the data collection method and processing level. Specifically, the Úrrétje area showed lower error values in 12 instances, while the oblique and P3D methods yielded superior results for the Csapókert area. Regarding RMSE, the Úrrétje area demonstrated better outcomes in 18 cases, with only three exceptions. These exceptions occurred in OG1 LPL, where the difference between the two areas was 3.5 cm, and in OG2 for OG3 LPL, with a difference of less than 0.4 cm. Compared to the error ranges of different data collection methods at the same level, the NG1 method consistently demonstrated the highest accuracy. However, in both cases, the error range exceeded 1 m. This was notably observed with the NG3 LPL method (1.04 m) for the Csapókert area and the P3D LPL method (1.15 m) for the Úrrétje area (Fig. 8). In both areas, the utilization of the double-grid approach for the oblique method showed a significant improvement in the range values compared to the single-grid approach.
Regarding RMSE, our results did not identify a universally superior method. Instead, they indicated that the OG3 method provided the lowest values of HPL in the Csapókert area, while the NG1 method was the best for MPL and LPL. In the Úrrétje area, NG3 achieved the best results for HPL and MPL, whereas NG1 obtained the best results for LPL (Table 3).
When comparing different processing levels, it was observed that in the Csapókert area, MPL yielded lower values than HPL for the NG1 and NG2 methods. In the Úrrétje area, only the NG2 method demonstrated lower values in the MPL than in the HPL. In terms of RMSE, only the NG1 and NG2 methods in the Csapókert area failed to exhibit lower errors for the HPL than for the MPL. Notably, we achieved 2.4 times lower errors in the error range (MPL-HPL) using the P3D method in the Csapókert area and 2.9 times lower errors using the OG3 method in the Úrrétje area. However, similar improvements were not observed for the RMSE values.
According to the feature importance test results, the processing level exerted the most substantial influence on the alteration of vertical accuracy values for roadways, registering a value of 0.1307. In contrast, the study area had the least impact, with a value of 0.1148. There was a more noticeable disparity in the significance of the different factors compared to the previous reference type. Further examination through the breakdown analysis unveiled intriguing insights.
When examining the minimum value (negative error) for the Csapókert area, the P3D mode, LPL, and E-W directional flight increased the magnitude of the error (Fig. 9). In contrast, for the median error in the Úrrétje area, the tilted camera, HPL, and double grid slightly increased the difference from zero. At the maximum value, except for the study area (Csapókert), the other three factors—oblique camera angle, MPL, and E-W directional flight—increased the error.
Vertical Accuracy—Roof
Assessment of the accuracy of the roof surfaces revealed substantial errors and range values. Upon comparison of the two study areas, it was evident that the Úrrétje area consistently demonstrated superior performance over the Csapókert area across all processing methods and levels, with disparities frequently exceeding several tens of centimeters. However, regarding RMSE values, the Csapókert area exhibited lower errors in six instances, specifically for the NG1 HPL, MPL, LPL, NG2 HPL, MPL, and Perimeter 3D LPL methods. The differences in RMSE between the two areas are notably smaller than the range of error values. When comparing the various data collection methods, it was found that the P3D method consistently yielded the smallest range of errors at the same processing levels in both study areas. Additionally, when focusing on the HPL and MPL, the OG3 method demonstrated considerable improvement in results compared to the other methods, with differences of only a few centimeters lower than those of P3D. Regarding the RMSEs in the Csapókert area, the OG1 method exhibited the lowest values for HPL (21.6 cm) and MPL (22.7 cm). The NG1 method produced the lowest values for LPL (23.8 cm). In the Úrrétje area, the NG3 method consistently performed the best across all three processing levels, with RMSEs of 15.3 cm for HPL, 15.6 cm for MPL, and 16.4 cm for LPL. Conversely, the P3D method yielded the poorest RMSE results for both the areas.
Concerning various processing levels, the considered reference type exhibited a noteworthy number of instances where the anticipated hierarchy (HPL > MPL > LPL) was not consistently observed. Regarding the range of errors, in the Csapókert area, MPL produced superior results for the NG1 and NG2 methods, while the LPL level yielded the best outcomes for OG1 (refer to Fig. 10). Conversely, in the Úrrétje area, the expected relationship was only observed for OG1. Upon analyzing the models generated from the three nadir flights, it was noted that the LPL resulted in the narrowest range of errors, while the MPL proved most effective for the OG2, OG3, and P3D methods.
Examining the RMSEs in the Csapókert area, it was found that the HPL yielded the lowest error values in three cases (NG2, OG1, and OG2), with no discernible difference between the HPL and MPLs in one case (NG3) (Table 4). MPL resulted in the lowest errors for NG1 and OG3, whereas LPL produced the lowest errors for the P3D method. In the Úrrétje area, the best results were obtained with the HPL for NG2, NG3, OG1, and OG2 methods, while the LPL yielded the lowest RMSE values in the remaining three cases.
The analysis of feature importance revealed valuable insights regarding the vertical accuracy test conducted on the rooftops (Fig. 11). Image acquisition exerted the most substantial influence on accuracy, with a value of 0.2247. In contrast, the processing level had the smallest impact, with a value of 0.1543, which was notably lower than that of the other three factors.
Further investigation through breakdown analysis indicated intriguing findings. In the case of the minimum value, it was found that the Csapókert area, P3D mode, LPL, and P3D flight direction increased from zero. Considering the median value, the Csapókert area and LPL decreased the error; however, the tilted camera had no discernible effect, while the E-W flight direction resulted in an increase. For the maximum, it was observed that the Úrrétje area, nadir camera angle, MPL, and E-W flight direction increased the error.
Data Collection and Processing Time
Considerable disparities in flight duration were evident in the analysis of the data collected. Single-grid flights demonstrated the shortest flight durations, as they followed the same trajectory irrespective of the camera angles, yielding negligible differences. In contrast, P3D flights were characterized by the longest duration. The duration of double-grid flights was less than twice that of single-grid flights, as detailed in Table 5. By comparing the two surveyed areas, it was observed that the average flight time in the Úrrétje area was 17% longer. Notably, within this area, the flight time associated with P3D exceeded the recommended overall flight time of 21 min by the drone manufacturer (Internet 1).
Upon analysis of processing times, our findings revealed an approximately threefold increase in computation time across distinct levels, regardless of the data collection methods employed. In the comparison between the two surveyed regions, the processing time for the Úrrétje data was observed to be 34% longer on average (Fig. 12). Specifically, at identical processing levels, NG3 processing required the greatest amount of time, whereas OG1 processing required the least time. Remarkably, data acquired through the double-grid method took approximately three times longer to process compared to each single grid for all three processing levels in both areas.
Discussion
Experimental investigations were conducted in two distinct urban study areas with different built-up characteristics. This approach facilitated the comparative analysis and validation of general statements regarding the accuracy of UAS-based mapping outputs in urban environments. Overall, our study observed better accuracies and modeling reliability in the case of the Úrrétje area compared with Csapókert. This discrepancy can be attributed to the spatial distribution of the GCPs. In the Csapókert area, the GCP distribution followed the surface of four parallel roads, resulting in a relatively regular pattern that was ultimately deemed suboptimal. Consistent with prior studies by Hugenholtz et al. (2016) and Ahmed et al. (2022), our investigation confirmed that the horizontal accuracy rates exceeded the vertical accuracy rates.
In the context of the horizontal accuracy test results, it was evident that the accuracy values improved with higher processing levels, consistent with our anticipated outcomes. The chosen data acquisition methods played a pivotal role in the surveys, with NG1 yielding the most favorable results across all three tested factors (RMSE, error range, and median), whereas P3D demonstrated comparatively inferior outcomes (Table 2).
These findings notably differ from those reported in studies, where the tilted camera and the double-grid method were reported to achieve the highest accuracy. In our study, the double-grid method only demonstrated accuracy improvement for the tilted camera angle, in contrast to the results of Ahmed et al. (2022) and Strząbała et al. (2022), where the double-grid method enhanced the accuracy of nadir acquisition.
The disparities in the results could be attributed to variations in data collection methods. Specifically, our study employed a flight altitude that was 30 m higher, and the camera tilt angle was 10° more pronounced during oblique shooting than in the study by Ahmed et al. (2022). Similarly, Jaud et al. (2018) found that a tilted camera provided better horizontal accuracy in Agisoft PhotoScan (the older version of Agisoft Metashape software). The variation in outcomes between our study and theirs could also originate from differences in data acquisition methods. Their oblique survey employed a 40-m lower flight altitude and a 20° higher camera tilt angle compared to ours. A recent study by Mueller et al. (2023) introduced innovative UAS flight design scenarios, yet it is important to highlight that their conceptual framework was specifically tailored for application within urban environments.
The results of the vertical accuracy test revealed distinct impacts of flight direction and camera angle factors on the RMSE values across the two study areas and the three processing levels. Notably, in the Csapókert area, the tilted double grid (OG3) method produced superior results for ground points at the HPL, consistent with the findings reported in studies by Jaud et al. (2018) and Ahmed et al. (2022). In contrast, in the Úrrétje area, the NG3 method achieved the highest accuracy, mirroring the third-best vertical accuracy result obtained in the study by Ahmed et al. (2022).
For vertical accuracy on road surfaces in the Úrrétje area, our results corroborated a similar trend observed by Dai et al. (2023) study, indicating that a tilted camera does not necessarily lead to increased accuracy when a substantial number (> 5–8) of GCPs are employed. The optimal method for rooftop points was OG1 in the Csapókert area and NG3 in the Úrrétje area. In conclusion, in line with the findings of Nesbit and Hugenholtz (2019), our results emphasize that utilizing both a double-grid flight direction and tilted camera angle contributes to improved vertical accuracy (Tables 3 and 4) compared to employing a single grid and nadir configurations for these two reference point types. Previous studies have shown that in situations involving oblique camera angles, optimal accuracy levels are attained when the automatic flight plan is supplemented by manual flight operations for additional image acquisition (Kovanič et al. 2023).
Jakovljevic et al. (2019) observed that point clouds generated through UAS-SfM exhibited larger vertical errors, resulting in an overestimation of elevations compared to point clouds derived from LiDAR surveys. According to the authors, these errors limited the suitability of SfM-generated point clouds for applications such as accurately modeling flood hazard areas. However, in our study, the vertical errors of the ground surface points were, on average, 17 cm smaller than those in Kameyama and Sugiura (2020). Kameyama and Sugiura (2020) examined different data acquisition parameters, including flight altitude and image overlap to assess the height and volume of forest trees. They discovered that the methods of data collection had minimal impacts on error magnitudes, which, in their study, proved to be excessively large for the intended measurements. In terms of non-ground surface vertical accuracy, our models produced RMSEs within the sub-meter range, contrasting with the RMSEs exceeding a meter defined by Kameyama and Sugiura (2020).
It is essential to note that our study primarily focused on the quantitative analysis of the unprocessed results generated by the SfM process. In contrast, prior research has emphasized the potential for substantial enhancements in the vertical accuracy of UAS-derived DEMs through the incorporation of weighted averaging and additive median filtering algorithms (Ajibola et al. 2019). In our investigation of accuracy factors, we observed that the recording method held the greatest significance for two reference types (horizontal and vertical roofs), whereas the processing level emerged as the most influential factor for the vertical roof reference. Unlike Mora-Felix et al. (2020), we found that flight direction played a minor role in our study. The discrepancies in the results between the two studies may be attributed to variations in the specific factors investigated.
The software processing time for the images produced by various methods was aligned with the specifications provided by the software developer (Internet 2). According to these specifications, projects with larger image datasets and higher processing levels generally require longer processing times. However, it is noteworthy that the accuracy of the models produced from these processes does not consistently surpass that of models produced with lower processing levels. In contexts where the timely delivery of results is of significant importance, as in applications such as agriculture and monitoring, the optimization of SfM processing time becomes paramount. Based on our findings, we infer that employing MPL can offer a satisfactory level of accuracy for situations where expeditious results are of primary concern.
Conclusions
The primary objective of this study was to evaluate and compare the impact of various UAS flight orientations and camera angles on the accuracy of SfM models, while considering different processing levels. This investigation was conducted across two urban study areas, with distinct characteristics enabling a thorough examination of their combined effects.
In the present study, we generated 42 distinct models including both digital surface models (DSMs) and orthophoto mosaics, by varying image acquisition and processing parameters. Our evaluation of horizontal accuracy, specifically regarding ground elevation, through orthophoto mosaics, consistently showed that the nadir camera angle combined with the single-grid flight mode provided the highest accuracy across all processing levels. Notably, we found that image acquisition emerged as the primary factor influencing accuracy values, with minimal disparities among the outcomes of the four tested factors.
On the other hand, our vertical accuracy assessment on road surfaces did not identify a single optimal method for all the processing levels. We observed instances where the double-grid and oblique camera angles produced superior results. The processing level had a more profound impact on accuracy. Similarly, our examination of roof surface measurements revealed that different data acquisition methods across varied processing levels yielded optimal results, with image acquisition exerting the greatest impact on accuracy.
When analyzing the processing levels for both vertical references, we noted that, in several cases, the medium processing level (MPL) outperformed the high processing level (HPL) in terms of accuracy. This suggests that MPL could serve as a viable alternative, particularly when rapid results are required. Despite the potential benefits of oblique camera angles and double grids in certain scenarios, it is essential to consider the significant increase in processing time, associated with these methods, which may be a crucial factor in practical applications.
Our research findings offer a comprehensive understanding of the potential accuracy variations associated with different flight patterns and image acquisition modes. Collectively, these results contribute substantially to the broader applicability of UAS-based aerial mapping in urban settings.
According to our experiences, urban mapping is constrained by several factors. One of the most significant challenges is the regulatory framework that may impede lawful surveys. Additionally, UAV flight time restrictions can pose obstacles when mapping large areas. To further investigate this subject, our team intends to explore the feasibility of employing various external orientation techniques in urban settings and their effect on model precision.
Data Availability
The participants of this study did not give written consent for their data to be shared publicly, so due to the sensitive nature of the research supporting data is not available.
Change history
20 May 2024
A Correction to this paper has been published: https://doi.org/10.1007/s41651-024-00182-4
Abbreviations
- DSM:
-
Digital surface model
- GCP:
-
Ground control point
- HPL:
-
High processing level
- MPL:
-
Medium processing level
- LPL:
-
Low processing level
- NG1:
-
East-west nadir
- NG2:
-
North-south nadir
- NG3:
-
Nadir double grid
- OG1:
-
East-west oblique
- OG1:
-
East-west oblique
- OG2:
-
North-south oblique
- OG3:
-
Oblique double grid
- P3D:
-
Perimeter 3D
- RMSE:
-
Root mean square error
- SfM:
-
Structure from motion
- UAS:
-
Unmanned aerial system
References
Agisoft Metashape User Manual. https://www.agisoft.com/pdf/metashape-pro_1_5_en.pdf. Accessed 16 May 2023b
Agüera-Vega F, Carvajal-Ramírez F, Martínez-Carricondo P (2017) Accuracy of digital surface models and orthophotos derived from unmanned aerial vehicle photogrammetry. J Surv Eng 143:04016025. https://doi.org/10.1061/(ASCE)SU.1943-5428.0000206
Ahmed R, Mahmud KH, Tuya JH (2021) A GIS-based mathematical approach for generating 3D terrain model from high-resolution UAV imageries. J Geovis Spat Anal 5:24. https://doi.org/10.1007/s41651-021-00094-7
Ahmed S, El-Shazly A, Abed F, Ahmed W (2022) The influence of flight direction and camera orientation on the quality products of UAV-based SfM-photogrammetry. Appl Sci 12:10492. https://doi.org/10.3390/app122010492
Ajibola II, Mansor S, Pradhan B, MohdShafri HZ (2019) Fusion of UAV-based DEMs for vertical component accuracy improvement. Measurement 147:106795. https://doi.org/10.1016/j.measurement.2019.07.023
Akay SS, Özcan O, Balık Şanlı F (2022) Quantification and visualization of flood-induced morphological changes in meander structures by UAV-based monitoring. Eng Sci Technol, an Int J 27:101016. https://doi.org/10.1016/j.jestch.2021.05.020
Anders N, Smith M, Suomalainen J, Cammeraat E, Valente J, Keesstra S (2020) Impact of flight altitude and cover orientation on digital surface model (DSM) accuracy for flood damage assessment in Murcia (Spain) using a fixed-wing UAV. Earth Sci Inform 13:391–404. https://doi.org/10.1007/s12145-019-00427-7
Barkóczi N, Szabó G (2017) Accuracy assessment of digital surface models based on a small format action camera in a North-East Hungarian sample area. Geographica Pannonica 21:224–234. https://doi.org/10.5937/gp21-16076
Belcore E, Pittarello M, Lingua AM, Lonati M (2021) Mapping riparian habitats of Natura 2000 network (91E0*, 3240) at individual tree level using UAV multi-temporal and multi-spectral data. Remote sensing 13:1756. https://doi.org/10.3390/rs13091756
Bi R, Gan S, Yuan X, Li R, Gao S, Luo W, Hu L (2021) Studies on three-dimensional (3D) accuracy optimization and repeatability of UAV in complex pit-rim landforms as assisted by oblique imaging and RTK positioning. Sensors 21:8109. https://doi.org/10.3390/s21238109
Cao D, Zhang B, Zhang X, Yin L, Man X (2023) Optimization methods on dynamic monitoring of mineral reserves for open pit mine based on UAV oblique photogrammetry. Measurement 207:112364. https://doi.org/10.1016/j.measurement.2022.112364
Casella V, Chiabrando F, Franzini M, Manzino AM (2020) Accuracy assessment of a UAV block by different software packages, processing schemes and validation strategies. ISPRS Int J Geo Inf 9:164. https://doi.org/10.3390/ijgi9030164
Che Y, Wang Q, Xie Z, Zhou L, Li S, Hui F, Wang X, Li B, Ma Y (2020) Estimation of maize plant height and leaf area index dynamics using an unmanned aerial vehicle with oblique and nadir photography. Ann Bot 126:765–773. https://doi.org/10.1093/aob/mcaa097
Dai W, Qian W, Liu A, Wang C, Yang X, Hu G, Tang G (2022) Monitoring and modeling sediment transport in space in small loess catchments using UAV-SfM photogrammetry. CATENA 214:106244. https://doi.org/10.1016/j.catena.2022.106244
Dai W, Zheng G, Antoniazza G, Zhao F, Chen K, Lu W, Lane SN (2023) Improving UAV-SfM photogrammetry for modelling high-relief terrain: image collection strategies and ground control quantity. Earth Surf Processes Landf 48:5665. https://doi.org/10.1002/esp.5665
Deliry SI, Avdan U (2021) Accuracy of unmanned aerial systems photogrammetry and structure from motion in surveying and mapping: a review. J Indian Soc Remote Sens 49:1997–2017. https://doi.org/10.1007/s12524-021-01366-x
DJI Mavic Pro Specs. https://www.dji.com/hu/mavic. Accessed 16 May 2023a
Escobar Villanueva JR, Iglesias Martínez L, Pérez Montiel JI (2019) DEM generation from fixed-wing UAV imaging and LiDAR-derived ground control points for flood estimations. Sensors 19:3205. https://doi.org/10.3390/s19143205
Gomes Pessoa G, Caceres Carrilho A, Takahashi Miyoshi G, Amorim A, Galo M (2021) Assessment of UAV-based digital surface model and the effects of quantity and distribution of ground control points. Int J Remote Sens 42:65–83. https://doi.org/10.1080/01431161.2020.1800122
Haas F, Hilger L, Neugirg F, Umstädter K, Breitung C, Fischer P, Hilger P, Heckmann T, Dusik J, Kaiser A, Schmidt J, Della Seta M, Rosenkranz R, Becht M (2016) Quantification and analysis of geomorphic processes on a recultivated iron ore mine on the Italian island of Elba using long-term ground-based lidar and photogrammetric SfM data by a UAV. Nat Hazard 16:1269–1288. https://doi.org/10.5194/nhess-16-1269-2016
Hugenholtz C, Brown O, Walker J, Barchyn T, Nesbit P, Kucharczyk M, Myshak S (2016) Spatial accuracy of UAV-derived orthoimagery and topography: comparing photogrammetric models processed with direct geo-referencing and ground control points. Geomatica 70:21–30. https://doi.org/10.5623/cig2016-102
Iglhaut J, Cabo C, Puliti S, Piermattei L, O’Connor J, Rosette J (2019) Structure from motion photogrammetry in forestry: a review. Curr Forestry Rep 5:155–168. https://doi.org/10.1007/s40725-019-00094-3
Inzerillo L, Di Mino G, Roberts R (2018) Image-based 3D reconstruction using traditional and UAV datasets for analysis of road pavement distress. Autom Constr 96:457–469. https://doi.org/10.1016/j.autcon.2018.10.010
Jakovljevic G, Govedarica M, Alvarez-Taboada F, Pajic V (2019) Accuracy assessment of deep learning based classification of LiDAR and UAV points clouds for DTM Creation and flood risk mapping. Geosciences 9:323. https://doi.org/10.3390/geosciences9070323
Jaud M, Passot S, Allemand P, Le Dantec N, Grandjean P, Delacourt C (2018) Suggestions to Limit geometric distortions in the reconstruction of linear coastal landforms by SfM photogrammetry with PhotoScan® and MicMac® for UAV surveys with restricted GCPs pattern. Drones 3:2. https://doi.org/10.3390/drones3010002
Jaud M, Letortu P, Théry C, Grandjean P, Costa S, Maquaire O, Davidson R, Le Dantec N (2019) UAV survey of a coastal cliff face – selection of the best imaging angle. Measurement 139:10–20. https://doi.org/10.1016/j.measurement.2019.02.024
Jeong GY, Nguyen TN, Tran DK, Hoang TBH (2020) Applying unmanned aerial vehicle photogrammetry for measuring dimension of structural elements in traditional timber building. Measurement 153:107386. https://doi.org/10.1016/j.measurement.2019.107386
Kameyama S, Sugiura K (2020) Estimating tree height and volume using unmanned aerial vehicle photography and SfM technology, with verification of result accuracy. Drones 4:19. https://doi.org/10.3390/drones4020019
Koc AB, Anderson PT, Chastain JP, Post C (2020) Estimating rooftop areas of poultry houses using UAV and satellite images. Drones 4:76. https://doi.org/10.3390/drones4040076
Kovanič Ľ, Štroner M, Blistan P, Urban R, Boczek R (2023) Combined ground-based and UAS SfM-MVS approach for determination of geometric parameters of the large-scale industrial facility – case study. Measurement 216:112994. https://doi.org/10.1016/j.measurement.2023.112994
Kucharczyk M, Hugenholtz CH (2019) Pre-disaster mapping with drones: an urban case study in Victoria, British Columbia, Canada. Nat Hazards Earth Syst Sci 19:2039–2051. https://doi.org/10.5194/nhess-19-2039-2019
Lehoczky M, Siki Z (2020) Fotogrammetriai feldolgozószoftverek. Geodkart 72:23–27. https://doi.org/10.30921/GK.72.2020.2.4
Li J, Yang B, Chen C, Huang R, Dong Z, Xiao W (2018) Automatic registration of panoramic image sequence and mobile laser scanning data using semantic features. ISPRS J Photogramm Remote Sens 136:41–57. https://doi.org/10.1016/j.isprsjprs.2017.12.005
Lin L, Yu K, Yao X, Deng Y, Hao Z, Chen Y, Wu N, Liu J (2021) UAV based estimation of forest leaf area index (LAI) through oblique photogrammetry. Remote Sensing 13:803. https://doi.org/10.3390/rs13040803
Liu X, Lian X, Yang W, Wang F, Han Y, Zhang Y (2022a) Accuracy assessment of a UAV direct georeferencing method and impact of the configuration of ground control points. Drones 6:30. https://doi.org/10.3390/drones6020030
Liu Y, Han K, Rasdorf W (2022b) Assessment and prediction of impact of flight configuration factors on UAS-based photogrammetric survey accuracy. Remote Sensing 14:4119. https://doi.org/10.3390/rs14164119
Manfreda S, Eyal BD (eds) (2023) Unmanned aerial systems for monitoring soil, vegetation, and river systems. Elsevier, Philadelphia
Manfreda S, Dvorak P, Mullerova J, Herban S, Vuono P, Arranz Justel J, Perks M (2019) Assessing the accuracy of digital surface models derived from optical imagery acquired with unmanned aerial systems. Drones 3:15. https://doi.org/10.3390/drones3010015
Mora-Felix ZD, Sanhouse-Garcia AJ, Bustos-Terrones YA, Loaiza JG, Monjardin-Armenta SA, Rangel-Peraza JG (2020) Effect of photogrammetric RPAS flight parameters on plani-altimetric accuracy of DTM. Open Geosciences 12:1017–1035. https://doi.org/10.1515/geo-2020-0189
Mueller MM, Dietenberger S, Nestler M, Hese S, Ziemer J, Bachmann F, Leiber J, Dubois C, Thiel C (2023) Novel UAV flight designs for accuracy optimization of structure from motion data products. Remote Sensing 15:4308. https://doi.org/10.3390/rs15174308
Nesbit P, Hugenholtz C (2019) Enhancing UAV–SfM 3D model accuracy in high-relief landscapes by incorporating oblique images. Remote Sensing 11:239. https://doi.org/10.3390/rs11030239
Nesbit PR, Hubbard SM, Hugenholtz CH (2022) Direct georeferencing UAV-SfM in high-relief topography: accuracy assessment and alternative ground control strategies along steep inaccessible rock slopes. Remote Sensing 14:490. https://doi.org/10.3390/rs14030490
Nex F, Armenakis C, Cramer M, Cucci DA, Gerke M, Honkavaara E, Kukko A, Persello C, Skaloud J (2022) UAV in the advent of the twenties: where we stand and what is next. ISPRS J Photogramm Remote Sens 184:215–242. https://doi.org/10.1016/j.isprsjprs.2021.12.006
Parrinello S, Picchio F (2019) Integration and comparison of close-range SfM methodologies for the analysis and the development of the historical city center of Bethlehem. Int Arch Photogramm Remote Sens Spatial Inf Sci XLII-2/W9:589–595. https://doi.org/10.5194/isprs-archives-XLII-2-W9-589-2019
Pepe M, Costantino D (2020) UAV photogrammetry and 3D modelling of complex architecture for maintenance purposes: the case study of the masonry bridge on the Sele River, Italy. Period Polytech Civil Eng. https://doi.org/10.3311/PPci.16398
Rauhala A, Tuomela A, Davids C, Rossi P (2017) UAV Remote sensing surveillance of a mine tailings impoundment in sub-Arctic conditions. Remote Sensing 9:1318. https://doi.org/10.3390/rs9121318
Reshetyuk Y, Mårtensson S-G (2016) Generation of highly accurate digital elevation models with unmanned aerial vehicles. Photogram Rec 31:143–165. https://doi.org/10.1111/phor.12143
Rossi P, Mancini F, Dubbini M, Mazzone F, Capra A (2017) Combining nadir and oblique UAV imagery to reconstruct quarry topography: methodology and feasibility analysis. Eur J Remote Sens 50:211–221. https://doi.org/10.1080/22797254.2017.1313097
Sanz-Ablanedo E, Chandler J, Rodríguez-Pérez J, Ordóñez C (2018) Accuracy of unmanned aerial vehicle (UAV) and SfM photogrammetry survey as a function of the number and location of ground control points Used. Remote Sensing 10:1606. https://doi.org/10.3390/rs10101606
Strząbała K, Ćwiąkała P, Gruszczyński W, Puniach E, Matwij W (2022) Determining changes in building tilts based on UAV photogrammetry. Measurement 202:111772. https://doi.org/10.1016/j.measurement.2022.111772
Svennevig K, Guarnieri P, Stemmerik L (2015) From oblique photogrammetry to a 3D model – structural modeling of Kilen, eastern North Greenland. Comput Geosci 83:120–126. https://doi.org/10.1016/j.cageo.2015.07.008
Ulvi A (2021) The effect of the distribution and numbers of ground control points on the precision of producing orthophoto maps with an unmanned aerial vehicle. J Asian Archit Building Eng 20:806–817. https://doi.org/10.1080/13467581.2021.1973479
Uysal M, Toprak AS, Polat N (2015) DEM generation with UAV photogrammetry and accuracy analysis in Sahitler hill. Measurement 73:539–543. https://doi.org/10.1016/j.measurement.2015.06.010
Verykokou S, Ioannidis C (2016) Automatic rough georeferencing of multiview oblique and vertical aerial image datasets of urban scenes. Photogram Rec 31:281–303. https://doi.org/10.1111/phor.12156
Vetrivel A, Gerke M, Kerle N, Nex F, Vosselman G (2018) Disaster damage detection through synergistic use of deep learning and 3D point cloud features derived from very high resolution oblique aerial images, and multiple-kernel-learning. ISPRS J Photogramm Remote Sens 140:45–59. https://doi.org/10.1016/j.isprsjprs.2017.03.001
Villanueva JKS, Blanco AC (2019) Optimization of ground control point (GCP) configuration for unmanned aerial vehicle (UAV) survey using structure from motion (SfM). Int Arch Photogramm Remote Sens Spatial Inf Sci XLII-4/W12:167–174. https://doi.org/10.5194/isprs-archives-XLII-4-W12-167-2019
Wu B, Xie L, Hu H, Zhu Q, Yau E (2018) Integration of aerial oblique imagery and terrestrial imagery for optimized 3D modeling in urban areas. ISPRS J Photogramm Remote Sens 139:119–132. https://doi.org/10.1016/j.isprsjprs.2018.03.004
Yao H, Qin R, Chen X (2019) Unmanned aerial vehicle for remote sensing applications—a review. Remote Sensing 11:1443. https://doi.org/10.3390/rs11121443
Zapico I, Laronne JB, Sánchez Castillo L, Martín Duque JF (2021) Improvement of workflow for topographic surveys in long highwalls of open pit mines with an unmanned aerial vehicle and structure from motion. Remote Sensing 13:3353. https://doi.org/10.3390/rs13173353
Funding
Open access funding provided by University of Debrecen. The research was funded by the K138079 project of the NKFI.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Ethical Approval
All ethical responsibilities for authors have been read and adhered to. All authors approve manuscript with no ethical misconduct.
Informed Consent
The research does not involve any human participant or animal. It is strictly GIS and geospatial analysis. There is therefore no consent with respect to participants in any analysis.
Conflict of Interest
The authors declare no competing interests.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Nagy, L.A., Szabó, S., Burai, P. et al. Improving Urban Mapping Accuracy: Investigating the Role of Data Acquisition Methods and SfM Processing Modes in UAS-Based Survey Through Explainable AI Metrics. J geovis spat anal 8, 15 (2024). https://doi.org/10.1007/s41651-024-00179-z
Accepted:
Published:
DOI: https://doi.org/10.1007/s41651-024-00179-z