1 Introduction

Automated detection of humans in aerial images is one of key elements in wilderness search and rescue (WiSAR) activities. According to Goodrich et al. (2009), there are two search and rescue (SAR) roles, namely information acquisition and information analysis. The use of unmanned aerial vehicles (UAVs) to collect aerial images along with their near-real-time processing to identify lost persons fits the two roles.

The growing popularity of UAVs and the increasing number of UAV operations broaden their use in SAR applications (e.g. Van Tilburg 2017; Silvagni et al. 2017). They include scientific experiments (e.g. Doherty and Rudol 2007; Goodrich et al. 2008; Miller et al. 2008; Molina et al. 2012) as well as comprehensive systems or communities such as for instance: ALCEDO (http://www.alcedo.ethz.ch, access date: 24/09/2017), SHERPA (Marconi et al. 2013), SWARM, also known as SARDrones (http://sardrones.org, access date: 24/09/2017) and SARUAV (Niedzielski et al. 2017; Jurecka and Niedzielski 2017).

To quickly find a person who is lost in the wilderness, human-assisted searches (e.g. Goodrich et al. 2009; Molina et al. 2012) or automated algorithmic searches (e.g. Rudol and Doherty 2008; Flynn and Cameron 2013) are carried out. In the current state of the UAV-based SAR technologies, there is a general consensus that the automated procedures may support human-assisted searches, but cannot fully substitute an expert who operates a sensor (Molina et al. 2012; Niedzielski et al. 2018).

Cloudiness belongs to weather conditions influencing persons’ needs for SAR team involvement, and they are often reported as the second group of environmental factors, after darkness, that contributes to the needs for SAR assistance (Gretchen 2004). Thus, SAR operations are likely to be carried out in cloudy weather. When visibility is poor it may constrain making appropriate decisions on where to go, in specific cases leading a person to be lost. In operational SAR missions, clouds lead to considerable limitations, in particular when airborne or satellite sensors are employed (O’Donnell 1999).

Clouds, fog or smoke may impact performance of different cameras, and therefore the presence of such barriers is treated as a disadvantage in remote sensing (Oakley and Satherley 1998; Woodell et al. 2015; Gultepe et al. 2009; Djuricic and Jutzi 2013). In the context of UAV-assisted SAR work, bad weather not only influences image clarity but also limits the effectiveness of search through constraining flights and other field activities (Sumimoto et al. 2000; Niedzielski et al. 2018). However, the negative impact of deteriorated image clarity on the usability of such imagery can be reduced by data processing techniques (Steinvall et al. 1999; Woodell et al. 2015; Yuan et al. 2010). In this study, we approach the problem from the opposite perspective and verify the hypothesis that clouds form a natural filter which decreases false hit rates in the process of human detection. To verify the hypothesis, we apply the nested k-means algorithm to detect persons in aerial photographs taken by a UAV during the field experiment in the Izerskie Mountains (southwestern Poland).

2 Experimental Setup

The eBee UAV was controlled from a dedicated mobile UAV laboratory (Niedzielski et al. 2017, Fig. 5 therein). The stable altitude of 123 m above take-off location (ATO) was kept over the entire experiment conducted on 20/10/2016. Aerial images were taken using the visible light Canon S110 RGB camera. The ground resolution of imagery was approximately equal to 4.3 cm/px. The camera operated in the automatic mode set by the manufacturer. The wavelengths with maximum spectral responses are 660 (R), 520 (G), and 450 (B). Five persons were located in the field. Three individuals occupied fixed locations over the entire day and the other two persons were UAV operators who kept changing their sites. Figure 1 presents the study area of Polana Izerska and person locations (P1, P2, P3—persons at fixed locations who were subjects of detection; P45—UAV operators who were siting or standing very close to each other and their location varied over the day). Polana Izerska is a mountain meadow of size 250 \(\times \) 170 m, with elevations ranging from 951 to 976 m a.s.l., located in the Izerskie Mountains in southwestern Poland.

Fig. 1
figure 1

Orthophotomap of study area along with locations of persons who simulated to be lost (codes: P1, P2, P3) and tightly settled two UAV operators (joint code: P45)

Full size image
Fig. 2
figure 2

Approximate locations of cameras mounted onboard the UAV along with locations of the selected images (IMG_) and settled persons (P1, P2, P3) in the morning (a), noon (b), afternoon (c) and evening (d)

Full size image
Fig. 3
figure 3

Statistical measures of person detection performance: a graphical presentation

Full size image

Polana Izerska is specifically located, approximately 3 km eastward from Stóg Izerski which is the first orographic barrier for westerly winds that prevail in western Sudetes. As a consequence, low-altitude clouds and fog occur frequently in Polana Izerska. According to Błaś et al. (2002), the area between Stóg Izerski and Polana Izerska is very foggy, with the annual number of days with fog significantly exceeding 200. The UAV missions targeted at person detection are conduced at low altitudes above ground level to ensure high spatial resolution of imagery, with the recommended altitudes between 60 and 100 m (Goodrich et al. 2008). The weather on 20/10/2016 was typical for the study area. The variable cloud cover over the entire day enabled to collect both cloud-blurred and sharp imagery to allow the comparison.

For each period of day, a single UAV flight was performed to acquire a set of aerial images covering the entire study area. From each set of photographs—corresponding to morning, noon, afternoon or evening—we selected four images, the centres of which were located in the closest vicinity of persons P1, P2 and P3 (Fig. 2). Individual aerial images in the central projection, not orthophotos, were subject to further processing.

Table 1 Selected meteorological characteristics of weather on 20/10/2016 in Polana Izerska (969.8 m a.s.l.), based on the weather station of Świeradów Forest Inspectorate located within the study area, both at the moments of UAV image acquisition over Polana Izerska and at the closest round hours (see Table 3 for details).
Full size table
Table 2 Selected meteorological characteristics of weather on 20/10/2016 in Liberec, based on the World Meteorological Organization (WMO) synoptic station no. 11603 (398 m a.s.l.) located in the vicinity of the study area, at the closest times to the moments of UAV image acquisition over Polana Izerska (see Tables 1, 3 for details).
Full size table

Over the entire exercise, air temperatures in Polana Izerska varied between 2.7 and 4.7 \(^{\circ }\)C, while sea level pressure was stable and equal to 1014.7–1016.2 hPa (Table 1). The aerial images acquired in the morning and noon were highly impacted by clouds, while photographs taken in the afternoon and evening were clear due to better visibility. This enables the comparison of the performance of the nested k-means method for clean and cloud-blurred input images.

The analysis of a few meteorological characteristics of weather on 20/10/2016 in Liberec, i.e. the synoptic station no. 11603 (398 m a.s.l.) recognized by the World Meteorological Organization (WMO) and located close (approximately 25 km) to the study area, was carried out (Table 2). Stable weather conditions over the region, in which Polana Izerska and Liberec are located, were inferred from the comparison of Tables 1 and 2. Hence, the extrapolation of cloudiness characteristics from Liberec to Polana Izerska was done. The absolute heights of base of lowest clouds were of 698–997 m a.s.l. which, when extrapolated to Polana Izerska, confirms the low-level cloudiness in the study area. Stratocumulus (Sc) prevailed in the morning and noon, with highly overcast sky reaching the maximum of 8 oktas in the noon. In the afternoon and evening Cumulus (Cu) co-occurred with Sc, reaching the minimum cloudiness of 3 oktas in the evening.

3 Methods

The following step-by-step procedure may be employed to investigate the role of clouds in the effectiveness of automated human detection in the UAV-acquired aerial imagery.

  • A flight plan over a test area should be prepared so that a UAV operates at altitudes which are suitable for person detection in aerial images (Goodrich et al. 2008; Niedzielski et al. 2018) and flies above persons who pretend being lost in the wilderness.

  • A specific day should be chosen so that changes in vegetation do not occur, but the amount of cloud cover varies over this day—ideally with and without cloud impact on visibility of terrain from the UAV.

  • The UAV missions ought to be carried out in different cloud-related visibility conditions and JPG aerial photographs of terrain in which these persons are present should be taken.

    • (Cloud-blurred imagery) The images with deteriorated clarity should be processed by the nested k-means algorithm to highlight potential locations of persons (nested k-means + true clouds).

    • (Clear imagery) The clear images should be processed twofold:

      • by the nested k-means algorithm to highlight potential locations of persons (nested k-means + no clouds),

      • clear imagery should also be artificially blurred and processed by the nested k-means algorithm to highlight potential locations of persons (nested k-means + artificial clouds).

The nested k-means method assumes the segmentation of an aerial image into squares (moving windows), built with superimposing stripes to avoid splitting person’s signal into two segments. The nesting is based on performing three consecutive tests in each segment. The first test uses the k-means method on a three-dimensional space of colours (RGB) to identify the most unique class, highly coherent in terms of colour composition. The second test uses the k-means method on a two-dimensional space of locations to check if the previously selected class is spatially coherent. The third test is applied to check if the size of the class fits the typical areas of standing or lying person. The details about the nested k-means method can be found in the paper by Niedzielski et al. (2017, Fig. 1 therein).

If the first test produces a flag 0, the overall output in a given segment is also equal to 0 (no person found). If the first test ends up with a flag 1 (person is likely to be present in a segment), the second test is run. If the second test produces a flag 0, the overall output in a segment is 0 (no person found). However, if the second test produces a flag 1 (person is likely to be present in a segment), the third test is run. If its application ends up with a flag 0, the overall procedure in a studied segment gives 0 (no person found). In contrast, if the third test produces a flag 1, the overall output in a given segment is equal to 1 (person found). The segment which gets 1 in the third test is hereinafter known as “find segment” or “highlighted segment” (Niedzielski et al. 2017, Fig. 3 therein).

An aerial photograph processed using the nested k-means algorithm produces a new image which is a composition of the original photo and transparent squared segments. The intensity of segment greyness corresponds to a probability of finding a person (zero probability does not imply image colouring). There are three statistics associated with the analysis of a single aerial photograph (Fig. 3): (1) detection rate (the percentage of persons who were found, i.e. 100% \(\times \) the number of correctly highlighted segments divided by a true number of persons within the entire image), (2) false hit rate (the percentage of highlighted segments which did not detect a person, i.e. 100% \(\times \) the number of incorrect or simply overestimated highlighted segments divided by the total number of highlighted segments) and (3) percentage of segments suggested to be searched, abbreviated as POSSTBS (100% \(\times \) the number of highlighted segments divided by all the possible segments within the entire image). The POSSTBS statistics measures the fraction of terrain area which is advised to be searched by ground searchers.

4 Results

4.1 Detection Based on Cloud-Blurred Images

Figures 4 and 5 show the selected images, the locations of which are marked in Fig. 2, with superimposed highlighted segments. Figure 4 (left column) presents the performance of the nested k-means method for images targeted at the person P1, with no other human individuals present in the scene. Figure 4 (right column) offers the same presentation for the person P2 who also was the only one person in the scene. Figure 5 is targeted at the person P3, however, the other persons P1 or P2 and, in addition, the two UAV operators were also recorded in the images. The UAV operators were standing or sitting very close to each other, therefore they were treated as a single person P45. The low-level Sc clouds blurred meaningful parts of background images in first two rows of Figs. 4 and 5 (morning, noon), while Cu and Sc clouds did not deteriorate the clearness of background images in third and fourth rows of Figs. 4 and 5 (afternoon, evening).

Fig. 4
figure 4

The UAV-acquired images with superimposed highlighted segments: left column—observations of the person P1 in four periods of day, right column—observations of the person P2 in four periods of day. Letters A–F mark characteristic points in geographical space to enable orientation

Full size image
Fig. 5
figure 5

The UAV-acquired images with superimposed highlighted segments: left column—observations of the persons P3, P1 and P45 in four periods of day, right column—observations of the person P3, P2, P45 in four periods of day. Letters G–J mark characteristic points in geographical space to enable orientation

Full size image
Fig. 6
figure 6

Example of artificial blurring of image no. 6654 performed by superimposing a white layer with dissimilar transparencies (top to bottom): 0, 25, 50, 75, 80, 90%

Full size image

For each aerial photograph under study, Table 3 juxtaposes information on image acquisition features (actual cloudiness which may constrain the view, code of targeted person, image number, period of day, detailed data collection times in CEST and UTC, number and codes of true persons in image) as well as on the performance of the nested k-means method (number of find segments, detection rate, false fit rate, POSSTBS). The records in Table 3 are grouped into four sets which correspond to observations of the similar scene containing the same person or persons (with one exception) in four periods of day covering cloudy and non-cloudy conditions. It should be noted that in all cloudy situations different levels of transparency through clouds were recorded, however, in the worst visibility cases the terrain view from the UAV was highly blurred but still visible.

The person P1 (images: 6167, 6317, 6467, 6666) was detected with rates of 100% for cloudy conditions, however, false hit rates were low (0%) when cloud cover was continuous and high (85.7%) when patchy clouds occurred. The latter statistics show overestimation of find segments, probably due to the fact that cloud patches occupied minor parts of the image (the case is therefore similar to the non-cloudy situation). When clouds did not form a barrier between the terrain and the camera, the nested k-means method returned either perfect fit (100% detection rate, 0% false hit rate, 0.5% POSSTBS) or incorrect result (0% detection rate, 100% false fit rate). Overall, the analysis of four images targeted at the person P1 did not convey a clear message as to whether cloud cover improved the performance of person detection using the nested k-means method.

Table 3 Performance of the nested k-means method in detecting persons in UAV-taken aerial images acquired through continuous clouds (low visibility over the entire image) and patchy clouds (low visibility in parts of image) versus the skills of the same algorithm run using clean input photographs
Full size table
Table 4 Performance of the nested k-means method in detecting persons in artificially blurred UAV-taken aerial images
Full size table

In contrast, the person P2 (images: 6177, 6343, 6476, 6651) was correctly detected with 100% detection rates in the four photographs, with and without cloud impact. However, the false hit rates were of 0% for cloudy conditions (patchy and continuous cover) and increased to 66.7% when images became clean after the clouds departed. That impacted the POSSTBS values which grew from 0.5 to 1.5% when switching from cloudy to non-cloudy conditions.

Even more pronounced evidence for the positive role of transparent clouds in improving the performance of human detection using the nested k-means approach is provided by the analysis of images targeted at the person P3 and, additionally, the person P1 and tightly settled two UAV operators P45 (images: 6180, 6329, 6479, 6654). The case covers two analyses with input images blurred by continuous clouds and the other two images without any cloud impacts. The number of highlighted segments was low for cloudy scenes, with high detection rate of 100% and false hit rates ranging between 0 and 40%. The values of statistics deteriorated when clean images were analysed. Indeed, detection rates varied between 100 and 66.7%, false hit rates were as high as 57.1 and 80%. The POSSTBS values confirmed the overrepresentation of highlighted segments for non-cloudy images (3.4 and 4.9%) in comparison to the cloudy scenes (1.5 and 2.5%).

The other scene targeted at the person P3, also including the signals of P2 and P45 (images: 6178, 6328, 6493, 6653) does not enable the comparison between cloudy and non-cloudy cases. Patchy clouds were only present in the image no. 6178, but it did not captured P45 which constrains the thorough comparison. However, a lesson can be learnt from the analysis of non-cloudy situations. Although high detection rates of 100% were revealed, the image processing concurrently led to considerable false hit rates ranging from 50 to 66.7%.

The entire analysis shows that the increased false hits rates seem to characterize the analyses performed using clean images (8 of 9 non-cloudy cases reveal false hit rates \(\ge 50\)%). In contrast, low false hit rates tend to correspond to the analyses carried out using cloud-blurred images (5 of 7 cloudy cases reveal false hit rates \(\le 40\)%). Therefore, the presence of transparent cloud cover is likely to reduce false hits and overrepresentation of highlighted segments.

4.2 Detection Based on Artificially Blurred Images

To confirm the above-mentioned findings obtained on a basis of cloud-blurred images, a numerical experiment was carried out. From the analyses presented in Figs. 4 and 5, clear images—those for which the nested k-means algorithm produced high false hit rates and potential locations were overrepresented (POSSRBS \(\ge \) 2.9)—were selected (noon image: 6328; afternoon images: 6479, 6493; evening images: 6654, 6653). They were pre-processed by superimposing a new uniform white layer with different levels of transparency applied (25, 50, 75, 80 and 90%). As a result, the clarity of original image was artificially deteriorated to simulate the effect of cloud blurriness (Fig. 6). Such artificially blurred images became subject to further processing using the nested k-means approach. Table 4 juxtaposes performance statistics of the nested k-means algorithm for these five images and different levels of transparency. It is apparent from Table 4 that there exist certain levels of blurriness for which false hit rates are minimized. Indeed, these minimum values were attained for transparencies ranging from 25 to 75%.

4.3 Interpretation

It is known that the presence of atmospheric aerosols leads to the reduction of image contrast and colour. Aerosol particles cause light to scatter when travelling toward the camera and, additionally, weaken the light signal reflected from the ground (Oakley and Satherley 1998; Gultepe et al. 2009). Our analysis based on input images blurred by true clouds and the scrutiny based on the artificially blurred input photos revealed a decrease in false hit rates of human detection performed with the nested k-means method. Clouds and their artificial equivalents cause a significant reduction in contrast and colour of imagery, decreasing a number of details. Such details may be incorrectly classified as persons by the nested k-means method, contributing to false hits where no person is actually present. The clouds and their artificial equivalents work as filters which remove noise from imagery. As a result, the most evident signals remain visible. Hence, the nested k-means method seems to perform better because false hit rates are decreased due to reduced clarity of images. However, as shown above in the experiment with artificially blurred images, the effect of decreasing a number of false hits occurs to a certain level of blurriness above which (when transparency becomes very low) a number of false hits increases again. This can be explained by too intensive reduction of clarity and a consequent removal of too many details, including persons. In the context of our exercise, such a noise reduction is particularly meaningful because persons who took part in the field experiment wore blue and red jackets, and the signal of their clothes remained strong.

5 Conclusions

This paper shows that the presence of transparent clouds in UAV-taken oblique aerial images may improve the performance of the automated human detection using the nested k-means method. Although no impact of cloudiness on person detection rates were reported, clouds were found to reduce false hit rates and, as a consequence, to reduce overrepresentation of sites suggested to be searched. Sensitivity analysis, in which the clarity of UAV-taken aerial images was artificially deteriorated by superimposing white layers with different transparencies to imitate the effect naturally caused by clouds, confirmed that the reduced clarity may improve the performance of the method. The decrease in the number of false hits was found for intermediate transparencies (25–75%), and the number of false hits tended to grow when clarity of images was highly reduced (superimposed white layers with 80–90% transparency).

The results may have a considerable practical potential—in a cloudy weather UAVs may be manoeuvered to fly at low altitudes so that cloud cover does not constrain the view but gives a transparent blurred picture of terrain. Such a flight planning for SAR purposes may, according to our preliminary study published in this paper, improve the performance of the automated human detection algorithms. Alternatively, artificial filtering to simulate the effect of transparent clouds may be considered. Usually the presence of clouds is considered to constrain the usefulness of different remote sensors (e.g. O’Donnell 1999; Woodell et al. 2015), but our results show that under specific circumstances the effect of clouds can be considered as a virtue. Although the findings are promising, further studies are required to investigate similar relationships when different person detection algorithms are employed. They may include either the investigation based on a bigger set of cloud-influenced images or the comparison of clean images with photographs artificially blurred by image filters. At present, the results may therefore be method-specific and may inherit certain limitations of the algorithms themselves. In the context of the nested k-means approach, discrepancies between snow-covered and snow-free terrain concerned overrepresentation, with high values of the POSSTBS statistics recorded in winter (Niedzielski et al. 2017). Therefore, it is likely that transparent clouds may reduce differences between the performance of the method in various seasons of year.