Introduction

Anthropogenic influences on the earth's surface, for example, due to surface or underground mining of raw materials, bring the risk that implications in the form of subsidence, displacements or slope movements can be expected in the affected areas. Therefore, it is important to detect changes in time and to make possible predictions about the object behavior to obtain indications of possible hazards or damages. Accordingly, safety can be ensured and necessary measures can be initiated. For this reason, monitoring systems [20] are becoming increasingly important and can also be found in larger research projects.

The project “Integrated Mining Impact Monitoring—i2MON” with a duration of four years (July 2018–July 2022) consisted of a group of recognized European institutions. The aim was to develop an integrated monitoring service to detect and evaluate ground and slope movements resulting from coal mining. The service includes both classic point-based sensor technology (e.g., low-cost global navigation satellite system) and area-based monitoring instruments. Terrestrial laser and radar technologies, as well as space- and airborne remote sensing, among others, are used. The sensor data is combined in a web-based monitoring platform with geological-geomorphological models and forecast models of the acquired area. A collective analysis with all multidimensional information is used to determine movement models. This will allow the mining industry to have an efficient, web-based surface condition assessment and decision-making tool for damage and risk mitigation that will contribute to the understanding of the physical movement process [9].

In this project, we had the task of testing areal object acquisition using terrestrial laser scanners, processing point clouds, visualizing them in near real-time [18], and detecting epochal changes [36, 37]. The results of various works from our wide task portfolio, including for example Schröder and Klonowski [36], are summarized in this article using a sand slide simulation in the laboratory. In particular, we deal with terrestrial laser scanning (TLS) acquisition techniques, but also image-based approaches with Structure-from-Motion (SfM) methods, exclude vegetation areas from the data, and apply evaluation algorithms to detect geometric changes of the observed object.

The multiple acquisition of a survey area at different times (epochs) offers the possibility of detecting geometric changes within the measurement accuracy of the instrumentation used. Continuous areal object detection using TLS has become a standard technology in many geodetic focused issues and has several advantages over single point measurements, for example, compared to total station or global navigation satellite system (GNSS) measurements, especially for monitoring deformation [5, 43]. Non-contact, non-destructive, fully automated scanning with variably adjustable point density and high spatial resolution in a relatively short acquisition time [26] can capture areas too sensitive for foot traffic. However, due to the predefined point grid of the laser scanner, there is no unique identical point assignment with the point cloud comparison in the case of several scans of different states. The exact position of the individual points on the object cannot be reproduced. As a result, differences in discrete points cannot be determined by a significance test. In this case, methods must be used to compare point clouds with each other and to derive deformations from them [16]. In addition to TLS, ALS [airborne laser scanning—using a small lightweight laser scanner mounted on an unmanned aircraft system (UAS)], as well as the image-based methods with SfM (handheld or mounted on a carrier platform such as UAS), offer great potential for areal object detection. The recurrent flights and the image data obtained from them can contribute greatly to monitoring tasks and evaluations of deformation measurements. Another advantage of using UAS is the time-efficient, image-based acquisition of large-scale surfaces or hard-to-reach areas with limited visibility of the object. The generation of point clouds in the evaluation process using SfM is sometimes very computationally intensive, depending on the number of images. In addition, specular (reflections) or monotonous areas (monochrome surfaces without texture and structure) can lead to problems in the calculation. Using TLS, the point cloud is available immediately after acquisition. The measurement takes place stationary on the ground and the instrument must be positioned outside the range of the moving object. For permanent monitoring, it is suitable to install a fixed TLS point at a sufficient distance in order to capture the object with the highest possible sampling interval (depending on the movement frequency) and to derive changes from it. Especially for early warning systems, this methodology proves to be efficient and deformation analyses can evaluate the captured data in real-time. Depending on the application, it must be decided which method is more promising and effective.

Point clouds acquired with TLS or SfM-based UAS represent the object with an extensive set of points. In the last decade, several methods have been proposed to analyze TLS point clouds to detect changes or measure deformations [26]. The choice of an appropriate deformation model for point cloud comparisons are differentiated into five methods [28, 42]: point-based models, point cloud-based models [12], surface-based models [1, 13, 34], geometry-based models [8, 10], and parameter-based models [15, 22, 35]. Point-based models represent the simplest form of modeling. With repetitive measurements using identical sampling settings and point address the resulting coordinates or distances can be compared directly. Point cloud-based models examine the relationships of two point clouds. One example is the Iterative Closest Point algorithm (ICP) presented by Besl and McKay (1992), which was designed for fine registration without the aid of control points and can be used for successive optimization of the transformation parameters. For surface-based models, at least one of the two point clouds to be compared must be available as a surface model and is often represented by a triangular meshing. In geometry-based models, the approximation of point clouds with a geometric model takes place and is compared with a reference shape or the other epoch. Parameter-based models can be understood as a further development of geometry-based modeling and use the estimated parameters to analyze deformations. For a comprehensive overview of the individual models we recommend the two references from [28] and [42]. In summary, point-based models are difficult to realize due to the identical point address and the parametric as well as geometric models require a prior information, called model knowledge, about the captured object [16]. In this article, we compare different approaches to deformation analysis, orienting ourselves mainly to the point cloud-based models, and qualitatively evaluating the suitability of the data bases with respect to a landslide under laboratory conditions. In addition to TLS point clouds, we also investigate the suitability of SfM-based point clouds with the point cloud-based models. We use the CloudCompare software to validate the point cloud comparisons using standard methods [17] such as cloud-to-cloud (C2C) [40] and multiscale model-to-model cloud (M3C2) [19]. Various distances between point clouds are evaluated. As an extended approach to point cloud comparisons, we introduce a subcategory called feature-based model. Point clouds are also compared and point correspondences are found but based on similar features of neighborhood relations. In the feature-based model, we apply the Fast Point Feature Histogram (FPFH) algorithm, which is also used in the registration of single scans [32, 33] and investigate the suitability on deformation analysis. The point-based models using CloudCompare are already used in many publications (e.g., [27, 39]). In this article, the results are used as a basis for comparison with respect to the feature-based model.

Since the data basis of the SfM-based point clouds are images of the captured area, 2D information and thus different evaluation strategies are also available. The idea is to be able to include the acquired UAS images for a statement about the object movements as well and to compare the results with the 3D point clouds. For this reason, we adopt an image-based approach, using computer version to detect deformations in a different way. The optical flow method can be used to calculate displacement vectors of object movements in two sequential images. For several years, the method has been increasingly used in the geosciences to identify landslide movements [6, 7, 14]. Chanut et al. [6] captures the area using images, which in turn are processed to 3D point clouds and then converted to 2D slope maps. With the help of optical flow, 3D displacements are computed based on these maps. Hermle et al. [14] uses single images on which time series analyses with optical flow are applied. Both publications show potentials in application with optical flow and have obtained satisfactory results. Based on previous research, we also apply optical flow to single images to detect deformations. In addition, we investigate a new approach in which the individual images are computed into a composite orthomosaic to always obtain the same area.

We present the different deformation analyses using a small-scale object deformation in the laboratory. To demonstrate the landslide realistically, vegetation areas are considered by integrating plant leaves into the sand slide simulation. Especially when capturing different states, moving objects or objects that change over time (for example plant growth) can be disturbing or even lead to falsified results. In the case of earth volume calculations or point cloud comparisons, this can lead to erroneous volume calculations or corresponding points in the data material [29], whereby the calculated differences do not reflect the real displacements. Likewise, for other measurement objects, such as buildings, dams, or rock faces [38], certain areas can be covered with plant growth and have different characteristics depending on the season. For this reason, all point clouds are processed after acquisition using the CANUPO algorithm [5] by segmenting the datasets into different classes based on a training dataset to filter out vegetation. The analysis of the three-dimensional dataset without the vegetation part follows. To qualify all three models and to compare them with each other, in addition to controlling them, the experiment provides individual target movements, which are also observed.

We summarize the main objectives of the article as follows:

  • Investigation of the suitability of TLS point cloud-based models for deformation analysis on SfM-based point clouds.

  • Comparison of popular point cloud-based methods (C2C, M3C2) with a feature-based model in relation to deformation analyses of dynamic surfaces.

  • Comparison between 2D-based (optical flow) and 3D-based (C2C, M3C2, FPFH) approaches for deformation detection.

Experimental setup and measurement process

To assess the movements of an object surface, we develop a landslide simulation in the laboratory. For the object acquisition, a terrestrial laser scanner, a total station, and two digital cameras are used. We describe this in the second section of this chapter. The experiment takes place on a wood panel (approximately 0.8 m × 1.0 m) with a three-sided frame used to simulate a lifelike earth surface with a sloped area. A mixture commonly known as kinetic sand is prepared, which consists of four parts fine sand, two parts cornstarch, and one part water. Kinetic sand has the advantage over basic sand that it is less permeable to air and therefore keeps its shape. The viscosity changes with the relative force applied to it and resembles a “non-Newtonian” fluid. To make the movement of the sand also visually perceptible, the kinetic sand is colored. Three parts are colored in blue, green and red. The fourth part is left in its natural color. We position the four layers of sand one behind the other in the upper area of the wood panel, which lies flat on the floor. Wood stakes are laid flat in the middle of the plate as barriers so the sand cannot slide down uniformly across the entire wood panel. This is to counteract the pure displacement of the sand (rigid body motion) in the vertical direction and to generate dynamic motions in an attenuated form. The frame of the wood panel is assumed to be stable compared to the sand. Therefore, black and white targets (frame markers) are attached as control points for the transformation of the TLS data. Outside the frame, additional control points are installed to ensure a uniform reference system for the measuring instruments used (total station and TLS). These points will not be discussed further. Plant leaves are fixed individually in the right edge area and the lower right corner of the object in order to consider the vegetation aspect in the experiment. Twelve markers (object points, represented as black crosses) are placed on the colored sand, from which the point-wise target movements are derived. Figure 1 shows the test object without movements (epoch 0) on the left and on the right are the time-shifted changes of state in epoch 1 and epoch 2.

Fig. 1
figure 1

Test object; epoch 0 (left), epoch 1 and epoch 2 (right)

Full size image

Epoch 0 is the condition at the beginning of the experiment. The sand is set in motion by changing the slope, slightly lifting the top of the wooden panel. After each change of state, the panel is moved to the starting position to perform the same measurement process epoch by epoch.

The Leica P20 terrestrial laser scanner is used to scan the entire surface of the measurement object from a short distance of approximately 5 m, resulting in a 3D point cloud. Before the measurement, resolution and quality levels are preset, which determines the density and accuracy of the point cloud. In the resulting point clouds, movements within a centimeter accuracy should be detected, and therefore we choose the resolution of the TLS acquisition to 3.1 mm at 10 m. According to the manufacturer, the 3D position accuracy at 50 m is 3 mm.

The camera images are important for two different reasons. The Nikon D800 digital camera with a resolution of 36.8 MP captures a series of images, which are processed via the SfM method to a 3D point cloud and an orthomosaic in post-processing. The second digital camera is a Sony Alpha 7R with a resolution of 36.4 MP. It is mounted on a stable tripod and is set so that one captured image per epoch includes the whole object. The camera position remains unchanged. These single images as well as the orthomosaics rendered from the serial imagery are subsequently required for the optical flow method. The coordinates of the frame points and the object points are determined with submillimeter accuracy using the Leica TS60 total station. The test setup, which includes the measurement equipment used and the resulting measurement data is visualized in Fig. 2. In total three different states (epoch 0 through epoch 2) are simulated and recorded by measurement.

Fig. 2
figure 2

Test setup with measuring instruments (TLS, total station, and cameras) and resulting measurement data

Full size image

Workflow and data preparation

To get an overview of the next steps, the evaluation process is shown as a schematic workflow in Fig. 3 and described in more detail below. Starting from the epochal recorded measurement data of the instrumentation used (step 1; Fig. 2), data preparation and processing (step 2) is carried out in the intermediate steps in order to subsequently link the methods of deformation detection (step 3). The TLS data is available as 3D point clouds directly after acquisition. From camera 1, the 2D image set is processed to point clouds through the SfM method using the Agisoft Metashape software. For the 3D datasets from TLS and SfM, both must be in the same coordinate reference system. For this purpose, the frame points are applied. In addition, the point clouds are cropped to the relevant areas and manually cleaned of erroneous points (outliers). We treated the vegetation areas in the point clouds in the CloudCompare software using the CANUPO algorithm and eliminated the datasets from the vegetation areas. The next steps are performed on datasets that have been filtered for vegetation. In addition to the 3D data, we have single images from camera 2 and an orthomosaic (2D) from the image set from camera 1, which we also generated in the Agisoft Metashape software.

Fig. 3
figure 3

Schematic evaluation process

Full size image

After we have processed the data, the different methods of deformation detection follow at the end. Here, the 3D data, point clouds from TLS and SfM, are evaluated via point cloud and feature-based models in the software CloudCompare and Matlab. We use Python to implement the image-based approach using optical flow. To check the results of the deformation detection, the measurements of the total station are used by calculating the known movements of the object points. A detailed discussion of the evaluation methods is given in chapter four.

The CANUPO algorithm is assigned more importance in the intermediate step so that a more detailed description takes place in the following. The explanation of the processing of the point clouds as well as the generation of the orthomosaic from the multi-images is excluded in this article. The CANUPO algorithm [4] can classify and segment data (e.g., 3D point clouds or 2.5D aerial LiDAR data) and is provided as a plug-in in the CloudCompare software. To separate the 3D point clouds by classes (two or more), a classifier is needed, which is learned from a training dataset. In the beginning, categories (vegetation and sand) are defined and the corresponding representative subareas in the point cloud (samples) are stored as classes. As natural surfaces are heterogeneous and have characteristic properties, it is rarely possible to determine them in a single dimension. Spherical diameters are used to determine various, local geometric relations over diverse scales and to parameterize class signatures that provide information about the distinction between classes. The more scales are selected, the more differentiated the calculation can be. For further details, please refer to the above-mentioned authors on the CANUPO algorithm.

The success of the classifier is assessed. Balanced accuracy (BA) is used to evaluate the results. The different number of points of each class is taken into account and ranges between zero and one. A high BA value implies a good recognition rate of the classifier. Additionally, the Fisher distribution is given to assess the distinctness of the classes. Simply, the larger the value, the more successful the class separation. On the available dataset, the BA value is 0.9997 and the Fisher distribution is 11.7466. The values represent satisfactory results.

The calculated classifier is saved as a parameter file and can be applied to any point cloud. A threshold value for the confidence of the point assignment can be set (here 90%).

As a result, all points below this threshold are not used for segmentation. The areas in the point cloud where the classes are adjacent are particularly affected. Figure 4 shows the TLS point cloud of epoch 1 without segmentation (left) and after applying the CANUPO algorithm (right). Both the plant leaves and the sand areas are successfully assigned to the vegetation class (red) and the sand class (blue), respectively, in the TLS point cloud. Gray represents areas below the set threshold.

Fig. 4
figure 4

Epoch 1 of the TLS point cloud without segmentation (left) and result of the CANUPO classifier with vegetation areas (red), sand areas (blue), and points below the threshold (gray; right)

Full size image

At the same time, the sand class also includes the wood panel. This can be explained by the surface texture. The surface of the sand and of the wood panel both have little structure and similar geometry. The wood stakes and the frame do not belong to either class, yet the wood stakes are assigned to the sand class and the frame to the vegetation class. In Fig. 5, the result of the SfM point cloud for epoch 1 is visualized and shows a similar result.

Fig. 5
figure 5

Epoch 1 of the SfM point cloud without segmentation (left) and result of the CANUPO classifier with vegetation areas (red), sand areas (blue), and points below the threshold (gray; right)

Full size image

The SfM point cloud is very sparse in the area of the wood panel. This is due to the surface texture of the object. Especially in shiny or redundant areas with little structure or texture, the algorithm has difficulty processing and cannot generate a uniformly dense point cloud. In these areas, only a few feature points are determined via SfM so that no points or only a few points appear. On a slope, natural components (rocks, soil, vegetation) with significant points occur. Thus, such areas with large gaps are not expected over a large area. In this test, the unprocessed areas on the wood panel can be neglected, as the focus lies on the sand area. The lighting conditions during recording also affect the image data. Consequently, care must be taken to select the appropriate acquisition configuration of the camera images as well as a sufficient number of images for the overlaps from various perspectives. For the following deformation analysis, all point clouds from both TLS and SfM are reduced by the segmented vegetation areas.

Methods for deformation detection

To compute and visualize deformations, we will discuss point cloud-based and feature-based models in more detail in this chapter. In point cloud-based models, two point clouds are compared whereby the object is used for the observation in an area-wide fashion. In contrast, in feature-based models individual points from a point cloud are considered based on local descriptors and assigned to the comparison point cloud using correlation calculations. Additionally, the optical flow is used as a comparison as an image-based approach.

Point-based target movements serve as a control of the measurements. Through the total station measurement of the object points, the movements of the individual points between the two epochs can be calculated and are available as known quantities in the form of 3D space vectors. The point displacements are visualized in Fig. 6.

Fig. 6
figure 6

Difference vectors of the measured object points by total station between epoch 1 and epoch 2. Overlay of the images and drawn vectors (left), 3D distances in the table and sorted by color into five categories (right)

Full size image

The length of the difference vectors corresponds to the values of the 3D distance vectors of the individual point movements. Likewise, the colors provide information about their magnitude. Point 21 is no longer visible in epoch 2 because of sand movement. We therefore could not determine a coordinate and exclude it for the following analyses. In general, the largest changes (> 8.7 cm) are in the middle area of the landslide simulation. The wood stake lying on the left side in horizontal alignment slows down the sand and causes a slight compression. Accordingly, the smallest movements take place in this region. Somewhat longer vectors between 6.5 cm and 8.7 cm occur on the right side, given that no resistance was installed there.

Point cloud-based models

The CloudCompare software offers different methods to compare point clouds. In the context of this article, we apply C2Cor M3C2. Generally, corresponding point pairs of the point clouds are matched based on distance calculations. The point clouds to be compared must be in the same coordinate system. In the following chapter, we will briefly discuss the common methods. The comparison of the results follows in Fig. 7 for both the TLS and SfM data.

Fig. 7
figure 7

Comparison of point cloud comparisons from SfM (left) and TLS (right) of epoch 1 and epoch 2 with models C2C (a) and M3C2 (b) in CloudCompare

Full size image

Using C2C, the spatially nearest neighbor is determined from the reference point cloud via “nearest neighbor search (NNS)” [40] in the comparison point cloud and the distance is calculated. This means that the point movements are always given absolutely in relation to the reference point cloud [16]. Accordingly, no distinction can be made between material uplift and downlift.

In contrast to the C2C method, the M3C2 algorithm can distinguish between material uplift and downlift, but the particularity of the algorithm is the use of an accuracy approach. The M3C2 algorithm performs an accuracy estimation in the form of a confidence interval for the determined distance between the point clouds while using the individual points as a basis. The algorithm combines several points into one core point in the individual point cloud. The scatter around this core point is used as an indicator of the accuracy of the measurement. The distance between the core points and their number are specified manually. By forming the kernel points, random deviations are minimized and the computational speed is increased [19]. Afterwards, a normal vector is calculated for each kernel point. A plane is estimated from the points that lie within a certain radius around the core point. In parallel, a standard deviation is calculated from this estimate, which provides information about the surface condition of the environment [16]. Subsequently, the core point is projected along the normal vector into both point clouds. The distance between two points—the point cloud difference—is calculated and visualized.

Figure 7a shows the result of the C2C model. The right color scale indicates the magnitude of the change, which ranges from 0.0 cm to 2.5 cm. The green dots identify the areas that are not subject to differences and are located mainly in the horizontal, middle area of the sand surface (SfM, TLS) and on the wood panel from TLS. In these areas, the weakness of the model becomes particularly clear in earth mass movements. Since the sand movement runs vertically from top to bottom, erosion is to be expected in the upper area and an application in the lower area, which are represented by yellow and red coloring. Only epochal differences without sign changes become apparent. Since the middle area also slides down but does not cause any major material changes, these areas are highlighted in green. The M3C2 result in Fig. 7b shows the thinned point cloud due to the use of kernel points M3C2 enables the detection of material buildup and removal. The right color scale shows both positive and negative values, with the middle, green area describing no changes. The blue dots in the upper area indicate the material removal. The application of material between the wood stakes is visible due to the yellow and red coloring. An increase in the material can also be seen above the wood stakes, which can be explained by the slipped material. Since a landslide happened, it can be assumed that the sand slides from the top to the bottom. Thus, the sand in the green area is effectively the erosion from the blue area. Additionally, M3C2 offers the possibility to indicate significant changes and standard deviations of the point clouds considering the given registration accuracy (1.0 mm).

The standard deviations are displayed in the range between 0.0 cm and 0.4 cm (Fig. 8). It is noticeable that the SfM dataset has significantly smaller standard deviations (0.1 cm or better) than the TLS point clouds. Only at the edges and at the transitions to the wood are larger deviations detected. The epochal states are similar between TLS and SfM.

Fig. 8
figure 8

Standard deviations of M3C2 point cloud comparison of epoch 1 and epoch 2 for SfM (left) and TLS (right)

Full size image

These standard deviations are equivalent to the internal accuracy and give a measure of the surface condition of the environment of the core points [16].

For both models, the indication of the areal changes is done by coloring the points. However, no pointwise spatial displacement vector, as in Fig. 6, can be localized and indicated, because the differences are considered as Euclidean distances between the point clouds. Consequently, there is no information on which direction a point has moved. The surface change of the sand can only be interpreted visually using previous data. No significant differences in point cloud results can be found between TLS and SfM in terms of measurement accuracy. Both techniques are suitable for the models and are approximately in the same difference range.

Feature-based model

To discuss another approach for point cloud comparison, the following section presents a model based on 3D feature descriptors. In contrast to point cloud-based models, the detection of single point movements by vector differences within the object is enabled. The algorithm, Fast Point Feature Histogram (FPFH) descriptor, is originally used for single scan registration [32, 33, 41]. In this process, point clouds are automatically transformed to each other by finding corresponding point pairs between two epochs based on calculations of similar geometries within neighborhood relationships [21, 41]. FPFH is the successor of the Point Feature Histogram (PFH) descriptor [30, 31] and is expected to benefit the computation time due to the reduction of the algorithm complexity from \(O(n\cdot {k}^{2})\) to \(O(n\cdot k)\) [32], where \(n\) is the set of points of the respective point cloud and \(k\) is the number of neighborhoods.

Both descriptors determine the geometric properties of the neighborhood points when parameterizing the local spatial differences between the query point and its neighborhood points. However, the FPFH descriptor does not consider the relationships of all neighborhood points of the query point (\({p}_{q}\)), rather only the points (\({p}_{i}\)) that lie within the predefined spherical radius of the direct neighborhood point (\({p}_{q}\)) (Fig. 9).

Fig. 9
figure 9

Graphical representation of the FPFH descriptor based on a query point (\({p}_{q}\)) with spherical radiuses drawn in 2D (left) and mathematical relationship of each point relationship (right) (modified after [32] and [33])

Full size image

Any query point which has fewer neighbors than the minimum number is eliminated for further calculations. For each connection of a pair of points, where \({P}_{S}\) is the starting point and \(C\) is the target point, three angles: alpha, phi, and theta (α, θ, Φ), are calculated as features (see Fig. 9). For a detailed explanation, the reader is referred to Rusu et al. [31], (2009), and (2010). The equations of the features are given below and visually represented in Fig. 9:

$$\alpha = v\bullet {n}_{C}$$
(1)
$$\theta = arctan(w\bullet {n}_{C}, u\bullet {n}_{C})$$
(2)
$$\Phi = u\bullet \frac{({C-P}_{S})}{d}$$
(3)

Likewise, the mathematical composition of the vectors u, v, and w in Fig. 9 becomes clear and defines the spatial axes for further calculations. The point whose angle between the normal vector and the distance vector of the two points is the smallest is defined as the starting point (\({P}_{S}\)). The normal vector of the target point is characterized by \({n}_{C}\) and the Euclidean distance between the points is calculated by \(d= {\Vert C-{P}_{S}\Vert }^{2}\).

To reduce the size of the descriptor, the calculated features are sorted into a histogram (Simplified Point Feature Histogram, SPFH). The histogram is evenly divided into \(m\) interval “bins”. The step size of the individual bin is composed of the difference between the minimum and maximum value of the respective feature for the entire point cloud and is divided by the number of bins. After the value ranges have been defined, the occurrence of the features in the bins is allocated and counted. The columns of the histogram thus characterize the absolute frequency of the feature. Each bin is normalized by the number of neighborhood points to describe the proportion of absolute frequency in the total number [30].

Formula (4) indicates the mathematical composition of each point. In the first step, the histogram (\(SPFH\)) of the respective query point \((p)\) is calculated with the direct neighborhood points. In the second step, the histogram is constructed over the points (\({p}_{i}\)) of the neighborhood points (\({p}_{k}\)) and adjusted by weighting over the distance (\({w}_{k}\)). Finally, the values are summed and divided by the number of neighborhood points (\(k\)). Both proportions are additively combined to \(FPFH(p)\) [32]:

$$FPFH\left(p\right)=SPFH\left(p\right)+\frac{1}{k}\sum_{i=1}^{k}\frac{1}{{w}_{k}}*SPFH({p}_{k})$$
(4)

The histograms of the individual point clouds are compared with each other via a correlation calculation. All pairs of points which are above a corresponding threshold are found as a possible combination. By concatenating all features, the final histogram contains \(m\bullet 3\) bins. The calculations are implemented in Matlab. As input data, two point clouds including normal vectors are required, which are available in the same coordinate system. The calculation of the normal vectors is done with the help of the plane estimation function in CloudCompare.

For each point per point cloud, neighborhood points are collected within a sphere radius. The search radius (\(r\)) for the neighborhood points as well as the minimum number of points that lie within the radius (\(iMin\)) must be defined in advance. The points that satisfy the conditions are stored in a matrix. The rest of the points are not considered for further procedure. For each stored point, the three features: α, θ, and Φ are determined, divided into the number of bins and compared with each other matrix of the point cloud by a correlation calculation. We set the condition that the point movements are aligned vertically downwards and opposing vectors are excluded. With the TLS point clouds available in high resolution, the point densities are reduced from about 24,000 to 6000 and have a point spacing of about 5 mm. This process mainly speeds up the computation time. The result of the TLS dataset, with a search radius (\(r\)) of 0.012 m, a minimum number of neighborhoods (\(iMin\)) of 16, and five bins (\(nBin\)), as shown in Fig. 10. The threshold value of the correlation is 99.0%.

Fig. 10
figure 10

Plot of point movements between epoch 1 (blue point cloud) and epoch 2 (red point cloud) of the TLS dataset across the calculated features at a correlation of 99.0% (black arrows) compared to the target movements (blue arrows)

Full size image

Most of the points detected are in the middle sand area. The pairs of points located in the lower area have similar sizes to the nearby number 10 point of the target movement. Compared to point 7, a significantly longer vector and two point movements oriented to the right are identified, which are considered to have low probability because of the known timing information. Particularly at the sides, the wood stakes barriers cause deformations in the sand. In the center, the dynamic movements are less pronounced so that the algorithm can determine individual vectors based on similar geometric properties in the TLS dataset. However, the algorithm is unstable with different settings, given that the results vary greatly. If the minimum number of neighborhood points is too low as a function of the radius, many connections are calculated which does not contribute to a satisfactory result and makes a clear statement difficult. A balance must be found between the two setting options. In this study, we cannot make a recommendation for the number of bins because the result varies with different data. The calculation time is enormous, especially with large datasets or with the choice of parameters, so attention must also be paid to the point density.

No useful result is obtained for the SfM point cloud. The point assignments are made arbitrarily with various setting options on the entire object surface and do not allow any clear patterns to be recognized. Based on this, the roughness, which is known as a geometric property of a point cloud, is considered for TLS and SfM at a radius of 0.01 m. It is noticeable that the TLS point cloud in the interval between 0.0 mm and 5.0 mm is significantly rougher than the SfM point cloud (Fig. 11) and corresponds to the results from Fig. 8. In the non-rough SfM point cloud therefore, many geometric similarities are likely detected, which can occur in the entire object. Characteristic properties, which should only exist in corresponding points, cannot be determined in the feature-based model.

Fig. 11
figure 11

Roughness of point clouds SfM (left) and TLS (right) in epoch 1 at a radius of 0.01 m

Full size image

Optical flow

In this chapter, deformation detection via 3D point clouds is contrasted with an image-based approach. The different states (epoch 1 and epoch 2) are each captured with an image and evaluated using the optical flow algorithm. The CANUPO algorithm applied to point clouds cannot be used on images, so vegetation remains present at this point.

Using optical flow, the motion of a camera (observer) or object within two consecutive images of a sequence can be represented over a 2D vector field. A velocity vector is assigned to each image pixel (Horn and Schunck 1981). In this experiment, the changes, which are characterized by the 2D displacement vectors, occur in object space. The vectors show the movements of the points from the first to the second image. The wood panel must be in the same position so that only the relevant changes in the sand areas are detected. Alternatively, the section can be adjusted by using an identical point in both images and 2D rotation to counteract any unwanted displacement of the wood panel that may occur.

A well-known algorithm is the Gunnar Farnebäck algorithm [11], which looks at each pixel of the image and uses it to calculate the density of the optical flow. However, the direction of the optical flow cannot be determined. The algorithm of Lucas and Kanade [25], in contrast, works with feature points in consecutive images and considers the direction of displacement. For the determination of feature points, different methods can be used, for example, to identify significant corners and edges in images. The SIFT (Scale Invariant Feature Transform) algorithm by Lowe [23] describes a method that detects rotation and scale invariant feature points in images. Due to the use of different scaling levels and gradients, the application is more robust.

With the help of a Python script, the algorithm can be applied to the image sequence. A special image region of interest (ROI) is defined manually via user input as well as the number of calculated feature points. The found feature points are stored for the first image and are used for the optical flow calculation. The output vector consists of 2D points and contains the new positions of the initially determined feature points in the second image.

Figure 12 shows the result of the images (epoch 1 and epoch 2) with difference vectors and given ROI (blue rectangle). The number of feature points is limited to 500, so that while relevant areas are not lost, the clarity is also not impaired. To visualize the point shift, a line is drawn between both pixel coordinates of the feature points and the displaced point is marked (Fig. 12, top right). The effect is classified into five categories and visualized in different colors. From this, it can be seen that the magnitude of the color scale corresponds to the difference vectors of the target movements (Fig. 6). To calculate the distance, the pixel size (0.2 mm) is determined using a comparison value of a TLS point cloud.

Fig. 12
figure 12

ROI for feature calculation for the first image (left) and result of optical flow from epoch 1 to epoch 2 (right)

Full size image

The largest movements (red) are in the middle range and mostly coincide with the known dimensions of the point movements (6, 7, and 10). The smallest movements (blue, turquoise, and green) on the left side are also analogous to the arrow lengths and colors (see Fig. 6). This can be explained by the change in the red sand area (point 1), which is larger compared to the green area below (point 5). Furthermore, the displacement in point 9 of the blue area is the smallest due to the compression of the sand at the wood stake. On the right side, slightly smaller point displacements (cyan and green) are determined, which lie in the yellow range for the target movements. Both points 4 and 22 are close to the lower limit of the yellow range, differing by a few millimeters. Additionally, the found feature points are not in the same position as the object points, so only a general rough statement is possible. In summary the optical flow result is sufficiently accurate compared to the target movements.

Particularly in the case of extensive object areas, such as a landslide, it is not possible to record the entire area from one camera position so that the scene is captured with adequate resolution over several multi-images. The generation of a georeferenced orthomosaic from the multi-images of the respective epochs represents the following step, which serves as an additional data basis for the optical flow. The application is only guaranteed if the orientation, position as well as image section of both orthomosaics are the same. The camera positions and orientations do not exist in the same way in repetitive image acquisitions. For this reason, orthomosaics can be transformed to an identical section with the same dimension via control points or natural striking points in the object. In this example, the black and white targets on the wood frame are used for this purpose. Figure 13 shows the result of the optical flow based on the orthomosaic. An ROI is also defined for this by determining the feature points in both datasets and visualizing the differences in color. Compared to Fig. 12, slightly different feature points are detected. The magnitude of the point movements largely agrees with the previous result. Consequently, it can be stated that the application of optical flow also ensures satisfactory results on orthomosaics with sufficient quality.

Fig. 13
figure 13

Orthomosaic of epoch 1 with the ROI (left) and result of the optical flow from epoch 1 to epoch 2 (right)

Full size image

To avoid losing the relevant peripheral areas when generating the orthomosaic, it is important to ensure that area in particular is captured more extensively. Due to the existing vegetation in the image data, changes are also visible in the areas that will be neglected for the time being. At this point, it is worth mentioning that image-based algorithms can be used for the classification and segmentation of image pixels over object classes [24] to reduce vegetation areas, similar to the way the CANUPO algorithm works. However, this was not implemented here.

Discussion

In this article, different methods for the detection of deformations based on a sand slide simulation under laboratory settings are presented. In comparison, landslides in real-world environments encompass much larger areas, which are additionally influenced by various environmental conditions, e.g., accessibility or light conditions. Data collection using TLS or UAS should be possible in most cases. However, the attachment of reference points is not always given or difficult to realize. For reference points to be used to align two epochs, it must be ensured that the points have not changed. If this is not possible, the point clouds of two epochs must be related to each other via geometry or via natural points. In general, for all deformation analyses based on point clouds, the quality of the point cloud is important and depends on the acquisition method (TLS or UAS), sensor resolution and its acquisition configurations as well as on the environmental conditions. The application of the CANUPO algorithm for vegetation removal is also possible with larger point clouds. We assume that the CANUPO gives better results on real data, since the surface provided in this study was very limited and there was a small amount of training data as well as variation for training the classifier. The point cloud-based models C2C and M3C2 are also suitable for in-situ slopes. The feature-based evaluation with the FPFH descriptor is not recommended for a point cloud that covers the total landslide area because of the high computation time depending on the parameter settings. Here, prior knowledge from the field of geomorphology would be beneficial to predict coarse movement patterns and thus divide the area into corresponding sectors with similar dynamics and characteristics. Alternatively, based on the point cloud-based evaluation, relevant areas could be identified and examined on a smaller grid for accurate evaluation. This is applicable to both the feature-based method and optical flow. Moreover, the transfer of additional conditions can increase the expressiveness of the FPFH algorithm. For example, if the effect of the possible movement can be estimated by prior knowledge, the spatial scales are limited. Furthermore, this descriptor is used specifically for local movements. Especially for large areas and depending on the magnitude of the change, it has to be investigated whether global descriptors, for example Viewpoint Feature Histogram VFH [33], provide more satisfactory results. The optical flow can also be applied to larger areas. By using orthomosaics instead of single images, areas of arbitrary size can be monitored. With the feature-based methods and optical flow, the functionality is only given if the properties of the neighborhoods in the point clouds as well as in the images remain similar in both epochs. Otherwise, the detected feature points cannot be assigned correctly. The application should detect both small and large movements. In the sand slide simulation, we used object points that were determined point by point in each epoch. This allowed us to calculate the reference movements and verify the results of the deformation analyses. Since this possibility is not available in reality, other ways of checking the results should be considered. One option would be a manual verification whether the results of the methods fit together.

Conclusions and future work

In this article, we present different approaches for deformation detection based on a landslide simulation in the laboratory, considering the implementation on an approximate real environment. To prepare the 3D point clouds for the analyses, areas of vegetation are removed, which is done by segmentation and filtering based on CANUPO algorithm. The point cloud-based models C2C and M3C2 detect areal-based changes between two point clouds. M3C2 also provide information about the direction of the changes. Both the TLS point clouds and the SfM point clouds are suitable as a data basis for the point cloud-based methods and present similar results. However, a point cloud-based model cannot determine discrete point movements. For this reason, we apply a feature-based descriptor (FPFH). Using the FPFH descriptor, individual point movements can only be detected in the TLS point cloud. The SfM-based point cloud did not provide useful results due to the low roughness. The algorithm is unsuitable for strongly deformed, dynamic areas because the properties of the neighborhood relations differ greatly, making finding point correspondences almost impossible. The evaluation is unstable because the parameters are difficult to set and depend strongly on the acquisition conditions and the object surface. Another way to detect single point displacements is to use the optical flow based on two sequential images. During a UAS flight there are multiple images of the object. To generate a single image of the scene without distortion, an orthomosaic can be created. We show in this experiment that with the same section and dimension, the optical flow can also be used on orthomosaics, so that the application of the method is extended and aerial images can provide information about the deformations. Since optical flow requires only two image files, it is much less computationally intensive than the method that uses 3D point clouds. The optical flow results represent the point-based target displacements more satisfactorily than the FPFH method. Furthermore, the optical flow results allow more detailed conclusions about surface motion than the feature-based algorithm.

In summary, point cloud based methods can be applied to obtain rough information about areal deformations independent of the dimension of the object. For discrete point-based displacements, optical flow using UAS-image data is appropriate. Possibly, both methods can also be considered as a combination or for checking the results of the other method. In the future, methods for deformation detection should be verified on a real object. In addition, it can be helpful to use the expert knowledge about the area and the interdisciplinary cooperation of e.g., geoscientists to estimate impacts and movements of the surface in advance on the one hand and to choose better evaluation strategies based on predictions on the other hand.