Introduction

Cloud is a visible mass of condensed water vapor in the atmosphere typically floating high above the general level of the ground. Clouds are important as they bring rain, but on the other hand, clouds become an obstacle when earth surface study is the objective of satellite images. Clouds on the input image are treated as noise that is why detection and removal of cloud from satellite image are an important preprocessing phase on most of the applications in remote sensing. Clouds are important in the regulation of the Earth climate, terrestrial atmospheric dynamics, thermodynamic chemistry, and radiative transfer [1]. Satellite images are one of the most powerful and important tools used by the scientist for the study of earth and space science.

In recent years, researchers have used satellite image data available from different sources such as Spot series of satellites from Europe, Sentinel-1/2 satellites from France, FY2G—India Ocean InfraRed Satellite, GaoFen-1 (Gaofen-1 is a high-resolution Chinese Earth observation satellite), MODIS data, GOSAT—The Greenhouse Gases Observing Satellite, Landsat-8 which is an American Earth observation satellite, GOES—Geostationary Satellites, Meteosat-Series of satellites are geostationary meteorological satellites, NOAA—Advanced Very High-Resolution Radiometer, QuickBird Satellites, Ikonos satellites, INSAT-3D/3DR, Kalpana-1 satellites from Indian Space Research Organization. Satellites observe the behavior of the cloud tops far from above the earth.

Various forms of cloud

Cloud detection techniques are employed by researchers using physical parameters of the cloud, such as (a) shape attributes; (b) fusion of multi-scale convolutional features of cloud net; (c) color transformation; (d) density of cloud; (e) cloud shadow; (f) clear-sky background difference; (g) feature extraction of the image.

Some researchers explored cloud detection using cloud optical properties, such as (a) spectral content, viz, near infrared (NIR), visible–infrared (VIR), thermal infrared (T-IR), and fusion of different bands; (b) spectral and spatial content; (c) cloud reflectance variation, brightness temperature; (d) cloud displacement index infrared (IR) based on different view angles; (e) multi-spectral and polarization characteristics.

The literature survey reviewed various forms of cloud detection such as cloud/no cloud, thin cloud/thick cloud, snow/ice detection, and cloud/cloud shadow.

Cloud/no cloud

Many researchers focused on the detection of cloud on the satellite image, i.e., whether cloud present or not in the image. Many cloud detection techniques are employed.

Deep learning using a multi-scale cloud net (MSCN) for cloud segmentation is proposed in Refs. [2, 3]. MSCN produces more accurate results, because it used a convolutional layers and skip connection in MSCN help to fuse the feature hierarchy to combine shallow appearance information and deep semantic information. They [2] have used global cloud cover validation collection for training. Authors [4] used the potential of deep learning methods for cloud detection and introduced the concept of patches with a convolution neural network, which allows the classifier to learn features such as spectral content and shape attributes of the clouds. Authors [5] proposed deep learning for cloud detection using multilevel image features of satellite imagery. They [5] performed two major operations; (i) obtain cloud probability map and (ii) get refined cloud masks through image filter. The proposed methods are very effective methods for cloud detection. Authors of [6] studied the deep learning method for cloud detection using the SPOT 6 image database. The concept of classification to build a decision tree for cloud detection on the snow cover area is used in [7]. It deals with the repeating scene from the same area by replaced each pixel with its actual surface types as a pixel, vegetation, or water. An automatic cloud detection algorithm based on the spatial texture analysis and neural network is used in Ref. [8]. It applied the multi-spectral synthesis method and cloud detection index on the spectrum analysis for each pixel cloud detection.

Authors of [9] experimented through a convolutional neural network (CNN) and deep forest. They have used a segmented super pixel level of remote-sensing image database. The other method proposed by them is Hierarchical Fusion CNN (HFCNN) is more focused on low-level features. Bayesian cloud detection scheme for cloud detection [10] used two concepts: one is an application of empirical cloudy lookup tables and the other is the use of visible channel simulation from the radiative transfer model (RTTOV) v11.3 in daytime cloud detection. Authors of [11] proposed a cloud detection method based on dual-color space using cloud features. They analyzed a large number of cloudy high-resolution remote-sensing images of the urban area. Authors [11] built cloud regions maps based on hue—saturation—intensity model and blue, green, red model, respectively. Authors of [12] indicated that only typical data are sufficient for the SVM training data set. They further proposed that the use of discriminant analysis (DA), can determine cloudy and clear sky. In an approach to adjust the classifier, they [12] have added typical data of certain conditions to the training data set that leads to incorrect results.

Authors [13] proposed the cloud detection algorithm, named “clear-sky background differencing (CSBD)”. It uses the original image and the corresponding clear-sky images. A clear-sky background image with various solar elevation angles data set is developed [13]. The same solar position and similar brightness distribution images are compared for analysis. CSBD algorithm still has some limitations. The single CSBD algorithm may misclassify these cloud pixels as clear pixels. Authors of [14] performed segmentation of satellite images into super pixels, extraction of spectral, texture, frequency features, and super pixel information extraction from feature histogram operations in sequence. An optimal threshold and support vector machine (SVM) is applied for cloud mask extraction. This method is time-consuming as both the algorithms are iterative optimization algorithms. Authors of [15] used spectral image rationing technique for generating ratio image based on the color transformation of the input image. Further fuzzy C-means clusters the ratio image and detects the clouds automatically. Ratio value of Croma and Lumais is used for cloud detection. This method is efficient in detecting thick clouds and thin clouds. Authors of [16] proposed a cloud detection method from satellite data. The proposed method allows detecting clouds with high spatial resolution and good accuracy and calculating the quantitative parameters of clouds in the automatic mode. The great advantage of the method is the possibility of analyzing the cloud cover over the most European part of Russia regardless of the season and time of day. Cloud Displacement Index (CDI) for cloud detection is proposed by [17] which make use of infrared bands with different view angles. It helps to separate clouds from bright objects. They [17] also proposed the integration of this new functionality into the Fmask algorithm which is initially developed for the Landsat suite of satellites. Authors of [18] proposed region-growing cloud detection procedure based on multi-temporal satellite images. The author first extracted seeds, which correspond to the cloud, based on a high variation of reflectance between two images. The method [18] is dependent on only four parameters and easy to implement.

Authors of [19] assessed the cloudiness prediction performance of weather research and forecasting (WRF) model using INSAT cloud imagery data over 3 months’ period during monsoon 2009. Authors [19] have used the brightness threshold test (BT) as a basis of the algorithm. It is seen that the probability of detection (POD) of cloud is 84% and the false alarm rate (FAR) is around 18%. Authors of [20] constituted a texture feature vector by extracting gray feature vector and frequency feature vector from remote-sensing images. They compared actual feature values and thresholds by the training process and judged the cloud region. The method efficiently separates the cloud and ground objects.

Authors of [21] proposed a threshold-based technique for cloud detection using (NOAA-AVHRR) satellite time-series data. They used two tests based on reflectance and temperature of the images for cloud detection. They used a combination of both, reflectance and temperature of the images for cloud shadow detection. Gradient-based pixel segmentation method is used for cloud detection in Ref. [22] which considered the local threshold for each pixel and the gradient values of the whole image. Authors of [23] proposed a cloud detection algorithm polarized image characteristic. They used characteristics such as multi-spectrum and polarization. The dynamic threshold is obtained by the different atmosphere and surface models at different time and areas [23]. Ensemble threshold and random forest-based methods for cloud detection given in [24] which performed two-step process: (i) binarization using ten thresholds of the first infrared band and visible channel of the image; (ii) binarized images of the two channels are combined to form an ensemble threshold image. Paper [25] focuses on different color models which makes utilization of multi-spectral thresholds with pixel-to-pixel handling. The model identifies cloud and clear cloud in the view of the threshold value, but finds trouble to separate snow/Ice and cloud because, in the visible region, the reflectance of snow/Ice and cloud is the same.

Authors of [26] used partial least squared (PLS) regression which analyzed different color spaces and components and proposed a supervised segmentation framework for ground-based sky/cloud images. The method is a learning-based approach which they do not require any manually defined thresholds, conditions, or parameters at any stage of the algorithm. Authors of [27] proposed a cloud detection method for multi-spectral remote-sensing images from Landsat 8. They applied the Simple Linear Iterative Cluster (SLIC) method, and then, two-step super pixel classification strategy is used to predict each pixel as cloud or non-cloud. They further used a double branch principal component analysis (PCA) Network (PCANet) and support vector machine classifier to extract high-level information of the cloud. Author of [28] improved previous Himawari-8 cloud and haze mask (HCHM) algorithm for cloud and aerosol loading detection for China. This algorithm [28] separates each pixel to clear, cloudy, and aerosol loading. They used infrared bands’ dust storms’ brightness temperature sampling results, the tests, and their thresholds to identify dust from cloud and clear-sky areas. Authors of [29] proposed a method to choose the initial clustering center for the k-means algorithm and then optimal threshold detection using the Otsu method is used on MODIS data to determine cloud/no cloud pixel.

Thin cloud/thick cloud

In this section, the literature survey of cloud detection of thick or thin clouds is discussed. Authors of [30] proposed multiple convolutional neural networks designed for high-resolution remote-sensing imagery. They applied the adaptive simple linear iterative clustering (A-SCLI) algorithm to the segmentation of the satellite images. Convolutional neural networks’ (MCNNs) architecture separates multi-scale features from each super pixel. Based on results, cloud is identified into different forms of cloud. Authors of [31] developed a model based on deep Convolutional Neural Networks (CNNs). It consists of four convolutional layers and two fully connected layers. The image is clustered into sub-region through a simple linear iterative cluster (SLIC) method. The proposed model [31] can detect well thin cloud and thick cloud.

Authors of [32] designed a simple linear iterative clustering algorithm. It is capable to classify thin cloud, thick cloud, and non-cloudy categories. They used a natural scene statistics model to separate cloud and surface buildings and used Gabor feature which is computed within each pixel. A support vector machine is used to distinguish cloud from snow regions. Authors of [33] proposed a fuzzy logic and neural network-based method for Infrared Imager (MSG-SEVIRI) images. It used the threshold mechanism and ancillary data such as the Numerical Weather Prediction (NWP) for model development.

Snow/ice detection

In this section, the literature survey of the differentiation of clouds from snow/ice is discussed. Authors of [34] studied cloud mask coupling of statistical and physical methods’ (MACSP) algorithm which uses a physical, statistical, and temporal algorithm approaches to improve cloud identification. The MACSP may be a useful algorithm, for users that cannot benefit from ancillary data. They identified as cloudy 90.6%. The cloud detection method on polarization satellite images [23] has used the multi-spectrum and polarization characteristics and the concept of a dynamic threshold for underlying surfaces in different time and areas. This proposed method can discriminate clouds from snow/ice. Authors of [35] presented a new test for the detection of snow which uses an image gradient to detect regions of snow. Among the many bands provided in Landsat 8 images, band 2 demonstrates the most promising characteristics for the detection of snow in this method. A set of 13 images with varying amounts of snow and cloud were used to assess this test’s ability to develop a snow mask and improve the accuracy of function of Mask (Fmask) cloud detection algorithms [35]. The gradient test is an effective way of leveraging this difference to improve the overall accuracy of automated algorithms for cloud and snow detection.

Authors of [36] developed a cloud detection algorithm which utilizes the multiple-channel feature of the Special Sensor Microwave Imager/Sounder (SSMIS) and classifies cloud-affected data into three categories based on hydrometeor types (i.e., cloud liquid particles, snow crystals, and ice crystals). SSMIS is a microwave radiometer with 24 channels covering a wide range of microwave frequencies. Two kinds of ensemble learning methods, boosting and random forest (RF) for fusion of visible–infrared (VIR) and thermal classifiers, are described in Ref. [37]. They explored four kinds of boosting, adaboost. M1, adaboost SVM, logitboost, and totalboostv and RF algorithm for cloud, snow/ice, and thin cirrus pixels detection. Authors of [38] presented a novel convolutional neural network (CNN) to classify cloud and snow. CNN learns cloud and snow multi-scale semantic features from multi-spectral imagery. To generate super pixels from high-resolution multi-spectral images, a simple linear iterative clustering algorithm is used here.

Cloud/cloud shadow

In this section, the literature survey of cloud detection based on cloud shadow is discussed. Authors of [3] proposed a method named multi-scale convolutional feature fusion (MSCFF). It performs pixel-wisecloud/cloud shadow segmentation of remote-sensing images. The results show that the proposed method [3] has advantages over traditional rule-based cloud detection methods. Cloud/shadow-detection algorithm based on spectral indices (CSD-SI) [39] proposed the cloud index (CI) and cloud shadow index (CSI) to indicate the potential clouds and cloud shadows based on their physical reflective characteristics. Authors of [40] proposed a multilevel feature fused segmentation network (MFFSNet) for automatic training. They proposed a fully convolutional network for cloud and cloud shadow feature learning. Method is extended for the pyramid pooling module. This module is used to extract contextual relation between cloud and shadow.

Cloud detection approaches

Researchers applied various methods for cloud detection. Based on the literature survey, cloud detection approach can be broadly categorized into the following two categories:

  1. 1.

    Classical algorithm-based approach.

  2. 2.

    Machine learning approach.

Here, classical algorithms are considered as those methods which have known specific steps to be followed for a specific input image. The output, i.e., cloud detection, will vary based on the input image and the algorithms used for it.

Machine learning is an area of artificial intelligence (AI) that enables systems to learn automatically based on the existed data.

Classical algorithm-based approach

Various researchers attempted a threshold-based approach for different types of cloud detection.

Author [41] performed a comparison with the use of ground-based cloud-base height measurements and short laser pulses sent by ceilometer for fog detection. They used general expression connecting time delay and backscattering height. The researcher [42] proposed a new spectral–spatial classification strategy that enhanced the performance of an orbiting cloud screen of hyperspectral images. The author integrated a threshold exponential spectral angle map (TESAM), adaptive Markov random field (aMRF), and dynamic stochastic resonance (DSR).

The presented survey shows that threshold-based algorithms find a series of proper thresholds of apparent reflectance or brightness temperatures via certain channels for different sensors and achieve cloud detection. Certain threshold varies as per season, as per sun elevation which causes variation in threshold for all area and time. The threshold varies on deserts, fog/haze, and ice/snow cover. The climates and surfaces by time and space are changing on a global scale, so varying threshold with surface type, temperature, atmospheric humidity, and viewing geometry will influence cloud detection results of satellite images [43]. Many cloud detection methods are threshold-based, but the threshold has poor universality. Classical algorithm-based methods for cloud detection are given in Table 1.

Table 1 Classical algorithm approach methods for cloud detection
Full size table

Machine learning approach

The machine learning techniques train forecast models by fusing model output with input samples to provide additional decision support. The detail about methods and models is given in Table 2. Various researchers [2,3,4,5,6] used artificial neural net with many variations such as support vector machine, fusing multi-scale convolution features, deep learning, decision tree, Bayesian classification, random forest-based methods, object-based convolution neural network, etc. Machine learning methods are very flexible and less complex since they simulate decisions on training data, but are not consistent, because model training depends on input data.

Table 2 Machine learning approach methods for cloud detection
Full size table

Other than above the two approaches, recent (2019) literature survey shows new methods. Author [44] describes a cloud detection algorithm based on Level 1 data of Polarization and Directionality of Earth’s Reflectance (POLDER). This study uses the multi-angular polarized data to achieve cloud detection. The simulation of multi-angular normalized polarized reflectance is done and the difference between the simulation and POLDER observation of polarized reflectance is used as the third criterion to distinguish cloudless pixels from cloud ones. Author [45] presented a conditional random field (CRF) model for cloud detection on ground-based sky. They showed that very high cloud detection accuracy can be achieved by combining a discriminative classifier and a higher order clique potential in a CRF framework. They demonstrated that the idea of combining a discriminative classifier with a higher order clique potential in a CRF framework is increasing efficiency in the DRF model.

Cloud detection from the visual band of a satellite image is developed by [46]. They consider the differences between the cloud and ground including various gray levels. They considered a reference satellite image (Ikonos) and introduced a reference satellite image by comparing the variance corresponding to the reference. This method detects multiple cloud regions and determines whether or not the cloud exists in an image is described.

Researchers [47] proposed a cloud detection method for satellite images with high resolution using ground objects multi-features, such as color, texture, and shape. The multi-scale decomposition of the domain transform is then used to extract the textures.

This method has a high overall accuracy rate, but if the non-cloud area is highly reflective, as the cloud area, it is easily misidentified.

Author [48] summarized simple and effective cloud and shadow-detection algorithms for both Landsat and Worldview images. They propose a very simple shadow detector based on an inverted map to convert an image into a greyscale image. They inverted map is threshold to generate the shadow mask. However, this method can generate false results if improper threshold is selected.

The literature survey shows that more work is required to achieve the desired accuracy and correctness for cloud detection methods.

In Table 3, strength and limitation of the selected cloud detection method are mentioned, so it is suggested to use the hybrid model which is a combination of machine learning techniques and classical algorithms to reach desired accuracy.

Table 3 Brief about strength and limitations of cloud detection methods
Full size table

Conclusion

Recent research from 2004 to 2018 on techniques of cloud detection from satellite imagery data is discussed in this paper. Detection of various forms of cloud such as cloud/no cloud, thin cloud/thick cloud, snow/ice detection, and cloud/cloud shadow are reviewed. Various threshold-based classical algorithms and machine learning techniques are reviewed along with the accuracy achieved by different researchers.

Based on this literature review, the following recommendations can be drawn: (a) Accuracy of cloud detection can be improved by adding different areas and time situations of consideration using different atmosphere models. (b) The potential clouds, cloud shadows based on their physical reflective characteristics, and consideration of physical parameters will increase the accuracy of cloud detection. (c) Hybrid method which combines atmospheric parameters and methods like artificial neural networks are needed to overcome constraints and drawbacks of existing algorithms.