1 Introduction

The World Meteorological Organization (WMO) defines fog as a suspension of extremely minute, typically microscopic water or ice droplets that reduces horizontal visibility on the earth's surface to less than one kilometre [1, 2]. But as per our national practices, fog is reported wherever the visibility is 1000 m or less and the relative humidity is 90% or more [3]. Transportation [4, 5] and aviation services at airports [6,7,8,9,10] are significantly impacted by low visibility conditions brought on by the onset of fog. Fog limits visibility to about 1000 m, making it challenging for pilots to locate the airport [11]. There has been a continuous evolution in Instrument Landing System (ILS) technology, but fog and poor visibility still pose a significant challenge to the airport's ability to function normally. When the fog rolls in, airports may have to close or reduce their operations, which can have serious financial repercussions [12,13,14]. Flight delays, cancellations, and diversions are common when dense fog (surface visibility ≤ 200 m) occurs, adding to the misery of passengers and increasing the cost to airlines. In December 2017, 21 flights (the most in a month's time) were diverted because of poor visibility at the JPNI Airport in Patna.

In northern and northeastern India, especially in the Indo-Gangetic Plain (IGP), December, January, and February are foggy months [15, 16]. Recent fog research in India found a worrying spike in fog and land pollution, as well as fog in December, January, and February [17, 18]. The Numerical Weather Prediction (NWP) model, which is used by scientists throughout the world to predict the weather, is also used to predict visibility [19,20,21]. Many studies have tested high-resolution NWP models to predict fog [22]. Predicting fog is challenging due to the small scale and limited knowledge of the atmospheric process, which is so complicated that the NWP model does not capture many aspects of fog formation [21, 23]. Because low-visibility incidents are local, numerical weather prediction is a difficult procedure. Fog prediction is susceptible to small-scale alterations in meteorological factors (wind or atmospheric stability), which many existing models don't represent.

In this study, we focus on machine learning (ML) approaches for low visibility prediction due to the NWP model's limitations. We address statistical and machine learning approaches for low-visibility prediction and provide promising results. In the 1980s, academics proposed using a linear regression technique to make fog predictions, although limited success [24]. ML-based algorithms have been effectively applied to several fog prediction problems in the recent decade to address the constraints of conventional (e.g., linear) techniques. A multi-layer perceptron (MLP) approach [25] was tested at Canberra International Airport in Australia, utilizing meteorologically measured parameters. The Australian Bureau of Meteorology's data was utilized to train and test a neural network model. A fuzzy logic-based fog prediction system was proposed and analyzed at Perth airport [26]. The fog prediction model averaged two different fog forecasting systems using majority voting [27] and suggested a decision tree induction ML technique for fog event prediction in Dubai, improving on numerical model approaches. The performance of MLPs with backpropagation [28] training is examined in a Brazilian fog event prediction challenge. [29] applies a Bayesian network to Melbourne airport fog prediction. In this scenario, an 8-h prediction time horizon is employed to train the network with 34 years of data. This fog prediction system at Melbourne Airport has better results than the previous one. Different ML regression algorithms have been explored at Valladolid Airport in Spain [30]. Machine learning (ML) and ANN-based approaches have successfully predicted low visibility and fog-related extreme events. Examples include support vector regression or Extreme Learning Machine (ELM) [31, 32], the Bayesian Decision Method [29, 33, 34], neural networks [25, 35,36,37,38,39], regression algorithms [40], ordinal regression techniques [7]. Studies also examine the meteorological causes of fog [18, 41, 42]. Some studies use the Decision Tree algorithm from the NWP model's output weather data to predict fog [43]. Others use a technique based on synoptic weather observation data [44]. These studies focus on solitary events. Data-driven AI has the potential to revolutionize weather forecasting or location-specific fog prediction. ML prediction approaches are objective and independent of forecasters' subjectivity. Receiving weather predictions faster also facilitates aeroplane operation, which requires precision and efficiency. India's location-specific weather forecasting hasn't been studied extensively. Customized services require reliable location-specific visibility predictions. This forecast may reduce weather-impacted costs. Predicting whether the formation or dissipation of fog will create low or reduced visibility is, thus, a critical issue that has unintended repercussions in many fields of weather, transportation, and aviation.

To that end, this study aims to.

  • Characterize fog events locally specific to the JPNI Airport located in the Indo Gangetic Plain (IGP) region.

  • We present a discussion on the effectiveness of various ML approaches in predicting fog (surface visibility ≤ 1000 m) and dense fog (surface visibility ≤ 200 m) and propose a method for making precise predictions of fog and dense fog.

Therefore, fog and dense fog predictions with the utmost accuracy are the objectives of this research paper, along with the local characterization of fog events. The rest of the paper is organized as follows: the next section describes the fog events database specific to the JPNI Airport, Patna, India, along with the studied areas. The most important methodology of the research paper is discussed in Sect. 3. Section 4 presents the comprehensive experimental results and comparisons of different ML approaches considered in fog prediction. Discussion of the proposed ML approaches is placed in Sect. 5. Section 6 closes the paper with conclusions and remarks on the research carried out.

2 Measurement site and data

We consider real-hourly synoptic weather observations for Jay Prakash Narayan International Airport (JPNI), Patna (25.5947° N, 85.0908° E) (shown in Fig. 1), and its surrounding areas, which are prone to frequent low-visibility events caused by local features as well as fog patterns associated with western disturbances (WD). Patna Airport has an elevation of only 52 m on the Ganga basin's alluvial plains. Its location on the Ganges's southern bank makes it a notable landmark (because of prevailing winds). From November through February, the Western Disturbance (WD) often causes radiation fog and advection fog to appear in this region [18, 45]. Extreme fog in this region frequently causes airport closures or significant delays, having far-reaching economic and social consequences. Thus, we focused on one of the Indo-Gangetic Plain (IGP) region’s airports that experience fog very often (as indicated in Fig. 1).

Fig. 1
figure 1

The geographical location of Jay Prakash Narayan International Airport Patna (a) India (b) Capital Cities of Patna and its Airport

Full size image

In order to anticipate the low visibility (fog or dense fog) in the following hour, we use the data we have at the moment (xn) and the previous hours (xn−1 to xn−7)to make a prediction about the target value in the hour (yn+1) for the lead time of one hour and subsequently (yn+2) for a two-hour lead time, and so on. We trained machine learning models (algorithms) to predict fog (surface visibility ≤ 1000 m) and dense fog (surface visibility ≤ 200 m) using synoptic hourly meteorological parameters that represent the availability of moisture and its distribution at the surface and in the lower boundary layer, including dry bulb temperature, dew point temperature, wind speed, wind direction, relative humidity, and cloud amount. The selected input variables and the current value of the predictive variables (fog or dense fog) do not have a direct or linear relationship, indicating that non-linear prediction models are more appropriate for this dataset in order to produce reliable predictions. However, it is crucial to remember that the linear correlation between variables is merely a suggestive parameter and should be considered with caution due to the non-linear interactions between meteorological variables that cause the fog events.

In this study, meteorological data from December, January, and February of 2014–2015 to 2020–2021 (21 months) were used to train the models, and meteorological data from 2021 to 2022 (3 months) was used to test the best-performing machine learning (ML) approaches (level 0 and level 1). In the train and test datasets, the fog/no fog ratio is 140/493 and 43/47, respectively. Similarly, the dense fog/no dense fog ratio in the train and test datasets is 51/582 and 11/79.

3 Methodology

The fundamental principles underlying all methods can be reduced to the concept of a dataset (x,y)N i = 1, where x = (× 1,…xd) refers to the input of the descriptive variables, and y is the equivalent response variable label. The purpose of these estimating procedures is to reconstruct the unknown functional dependence \(x\to ^{f} y\) by estimate \(\widehat{f\left( x \right)}\). As a result, the loss function Ψ(y,f) value is minimized.

$$\widehat{f\left( x \right)} = {\text{y}}$$
(1)
$$\widehat{f\left( x \right)} = arg\begin{array}{*{20}c} {min} \\ {f\left( x \right)} \\ \end{array} \Psi \left( {{\text{y}},{\text{f}}\left( {\text{x}} \right)} \right)$$
(2)

Reducing the function of the expected loss over the change in the response variable as a result of the Ey(Ψ[y, f (x)]), where x is the observed descriptive data, the equivalent expression is shown in equation no.3.

$$\widehat{f\left( x \right)} = {\text{arg}}\begin{array}{*{20}c} {min} \\ {f\left( x \right)} \\ \end{array} {\text{Ex}}\left[ {{\text{Ey}}\Psi } \right[{\text{y}},{\text{f}}\left( {\text{x}} \right)]\left] {\text{x}} \right]$$
(3)

Response y can be selected from a skewed distribution. There are a variety of loss functions that can affect this. The role of the binomial loss can be easily taken into account in classification problems if the variance of the response is dual (y = 0, 1). In Fig. 2, we present a block schematic of the entire process by which our proposed machine-learning prediction model operates the mechanism of the detailed methodology described in the algorithm 1 shown below. We implemented the explained algorithms in Python through h2o and Anaconda.

Fig. 2
figure 2

Process block diagram of the proposed machine learning prediction model

Full size image
figure a

More specifically, the following are the research paper's novelty and significant contributions:

  • We offer a thorough benchmark for the problems in the prediction of low-visibility events caused by fog (with intensity), taking into account a wide range of comparisons of relevant ML approaches.

  • This is one of the first attempts to use statistics to describe the characterization of fog (intensity scale) at JPNI Airport in Patna, India, which is located in one of the foggiest Indo-Gangetic regions (IGP).

  • Assessment and optimum tuning of a wide range of relevant ML techniques (i.e., optimally six) at level 0, and subsequently stacked ensembling of these wide ranges of machine learning models at level 1, provide robust ML approaches with optimum performance and high accuracy in different time scales (a lead time of 6 h to 1 h).

  • This research paper comprehensively compares the usability of a wide range of machine learning classification algorithms like GBM, DRF, GLM, XRT, XGBoost, Feed Forward Neural Network (at level 0), and Stacked Ensemble (at level 1). And finally, it proposes that the ML technique has the best trade-off among three meta-algorithm approaches: bagging (which decreases variance), boosting (which decreases bias), and stacking (which improves predictive forces). While finding the best prediction algorithms among them, using ensemble algorithms (level 1) is another fun way to combine the information that level 0 techniques provide.

  • Based on the results of this study, we found that the proposed ML techniques had a high statistical skill score and performance metric and that the output was stable.

3.1 Supervised ML methods

Many proposed ensemble ML methods have the goal of training a large number of relevant base ML models and then combining their predictions to boost the accuracy of the resulting model or to make the model more robust or generalizable. Ensemble approaches, which have a learning paradigm that may be summed up as "better together," consistently outperform the predictions of other, more sophisticated machine learning methods, such as incredibly complex ANNs. Participants in a learning experience are the ensemble's building blocks. Although there are a wide variety of ensemble algorithms to choose from, most of them may be grouped into the categories of bagging, boosting, and stacking methods.

3.1.1 Random forest

When it comes to solving classification problems, Random Forest (RF) [46, 47] is one of the most well-known bagging-like algorithms. In this method, the decision trees are used as the learners, and then subsets are created using the bootstrap aggregating methodology, similar to the bagging method but with the added feature of dynamic tree topology. From a theoretical standpoint, this runs against the bagging paradigm because the trees in the ensemble (the forest) may differ in length, topology, or input factors, dramatically increasing the variability of the learners.

3.1.2 Bagging

Bootstrap aggregation, or bagging, is a more straightforward ensemble technique for training many learners and delivering a consistent output. Learners are bagged together because their topology, number of input–output variables, and parameters are all the same. The most common learners used to create the bagging ensemble method are decision trees with the same number of branches, the same training parameters, and the same input–output variables. Figure 3 depicts the decision procedure by majority vote (in classification) [48] used to generate the result of an ensemble model.

Fig. 3
figure 3

Presentation of bagging technique for classification problems

Full size image

3.1.3 Boosting

Boosting algorithms are a special kind of ensemble algorithm with their own method of teaching new information. They are effective classifiers [49]. The ensemble consists of a set of learners, each of whom has its own set of learning parameters or input–output variables.

3.1.4 Stacking

To create an ensemble, a stacked ensemble "stacks" the predictions of numerous basic classifiers using a meta-learner. Stacking, also known as "super learning" [50] or "stacked regression” [47], refers to a family of algorithms that require training a "meta learner" at a higher level to determine the best possible combination of the learners at the lower levels. In order to boost a single predictor's robustness and generalizability, this ensemble approach combines the predictions of multiple base estimators. This study proposes a stacked ensemble model consisting of XGBoost, GLM, GBM, and XRT. With this method, the final prediction of the ensemble model is made by training a meta-model at a higher level (level 1) on the unintentional predictions of base models at a lower level (level 0).

3.2 ANN-based models

In this section, we also briefly describe the ANN-based approaches that were investigated in this study.

3.2.1 ANN based learning

To classify and predict complex data sets, we use information-processing algorithms called neural networks. Starting with a series of features (input variables) with predetermined labels, feedforward neural networks are trained to make predictions (classification). The internal layers perform their function by taking a directional (training-based) and non-linear approach to combining input and weights. Although there are a number of established algorithms for training a system, the back-propagation techniques used here are among the most effective. It uses a backpropagation technique to maximize the weight of each perceptron unit across an input layer, a hidden layer, and an output layer. Backpropagation is used to train a multi-layer feed-forward neural network to reduce the stochastic gradient. Improvements in prediction accuracy can be achieved by the use of more sophisticated techniques, including adaptive learning rate, rate annealing, momentum training, drop-out, standard L1 or L2 regularization, checkpointing, and grid search [51]. It can also automatically learn data features [52].

3.3 Statistical skill score

The statistical analysis of the skill scores was carried out using a contingency table (shown in Table 1) with the logic "if fog (Surface visibility ≤ 1000 m)occurs then 1 otherwise 0 (Surface visibility greater than 1000 m) ". The same applies to instances of dense fog (Surface visibility ≤ 200 m). This evaluation takes into account the statistical aspects of the skill scores: Accuracy, Selectivity and F1 Score.

$$Accuracy\frac{{\left( {TP + TN} \right)}}{{\left( {TP + FP + FN + TP} \right)}}$$
(4)
$$Specificity\,or\,Selectivity = \frac{{\left( {TN} \right)}}{{\left( {TN + FP} \right)}}$$
(5)
$$F_{1} = \frac{2}{{\frac{1}{recall}{*}\frac{1}{precision}}}$$
(6)
Table 1 Model contingency table for computation of forecast quality
Full size table

3.4 Performance classification metrics

The outcomes of machine-learning experiments on a testing data set are presented in order to assess the feasibility of using such algorithms to estimate low visibility. N samples from the testing data set compared with the predicted value with the actual values and proposed algorithms compared using verification results listed in Table 2.

Table 2 List of verification scores used in the study
Full size table

4 Results

4.1 Statistical characterization of fog at JPNI Airport Patna

In this subsection, we offer the statistical description of the low visibility/fog at the JPNI Airport, Patna. Out of 1210 days during the study period 2014–2022 (November to March), 604 days have surface visibility below 1000 m, indicating fog, and 166 days have surface visibility below 200 m, indicating dense fog. This information was used to propose comprehensive and generalized best-trade state-of-the-art ML techniques. Fig. 4 displays the average number of low visibility and fog hours at JPNI Airport Patna from 2014 to 2022, while Fig. 5 displays the average number of days with low visibility and fog (with the intensity scale).

Fig. 4
figure 4

Statistical analysis of the number of foggy days for November to March for the JPNI Airport Patna

Full size image
Fig. 5
figure 5

Statistical analysis of the number of foggy days for November to March for the JPNI Airport Patna (Foggy days represented as at least once in a day visibility ≤ 1000 m)

Full size image
Fig. 6
figure 6

Comparison of ML approaches performance for the prediction of fog (surface visibility ≤ 1000 m)for the lead time of 02 h (The blue color indicates the AUC and Ginni Index in the shown fig shows the AUC (Prediction of fog), shows AUC (prediction of dense fog),Similarly show shows ginni index for the prediction of fog, ginni index for the prediction of dense fog. Similarly green represents the MPCE and AUCPR for the prediction of fog as well as dense fog

Full size image

In this analysis, low visibility owing to fog formation is most often in January, December, February and November, and least common in March. Fog and low visibility last for the longest time in January and for the shortest time in March. When compared to December to February, January has more dense fog days and dense fog hours (Surface visibility ≤ 200 m). In addition, February has a similar number of foggy days (Surface Visibility ≤ 1000 m) but fewer days with the other intensity scale values. Standard deviation (given in Figs. 4, 5) indicates that advection fog (fog in the rear sector of WD) occurs in January and sometimes in December, whereas radiation fog (more standard deviation) occurs in November and February (less standard deviation). Standard deviation for fog at JPNI airport Patna are largest between November and February, while the opposite is true in December and January. Figure 5 clearly demonstrates how the frequency of foggy days varies by month, with November and December having the largest fluctuation and December and January having the lowest. Table 3 displays the average number of low visibility/fog hours (surface visibility ≤ 1000 m) and dense fog (surface visibility ≤ 200 m) for the study period.

Table 3 Mean no. of hours of Fog (Surface Visibility ≤ 1000 m) and Dense Fog (Surface Visibility ≤ 200 m) events (In hours) for the JPNI Airport, Patna
Full size table

4.2 Prediction of fog (surface visibility ≤ 1000 m) by using proposed ML (machine learning) approaches

We investigate the classification outcomes of the prediction of fog (surface visibility ≤ 1000 m) and give a comprehensive evaluation of how well different ML approaches performed on these datasets. Which method proved most effective for these datasets with a lead time of six to one hour? Here, seven machine-learning approaches are used to predict fog (surface visibility ≤ 1000 m). A trial-and-error process is used to fine-tune the six base models (GBM, XGBoost, GLM, XRT, DRF, and Feed Forward Neural Network) at level 0 for optimal performance. The suggested stacked ensemble of level-0 models outperforms the aforementioned six ML approaches at the level-1 evaluation stage (GBM, XGBoost, GLM, XRT, DRF, and Feed Forward Neural Network). Figure 6 presents the comparison of all seven ML approaches for the prediction of fog (surface visibility ≤ 1000 m) and dense fog (surface visibility ≤ 200 m) for the 02-h lead time. Where, SE outperforms the other base model approaches, including feed-forward neural networks. Similarly, it compares performance metrics for the lead times from 6 to 1 h and finds that the proposed SE approaches get better as the lead time decreases (presented in Fig. 7). Figure 8 presents the statistical skill score of the proposed stacked ensemble approaches for a lead time of 6 to 1 h for the fog (surface visibility ≤ 1000 m) and dense fog (surface visibility ≤ 200 m). Empirical evidence shows that the AUC gets better as the time stamp gets smaller, with the best values being between 0.8721 and 0.9231 for the prediction fog.

Fig. 7
figure 7

The performance metric of the proposed machine learning algorithms for the prediction of fog (surface visibility ≤ 1000 m) and dense fog ((surface visibility ≤ 200 m) for the lead time of 6 to 1 h

Full size image
Fig. 8
figure 8

The Statistical skill score of the proposed machine learning algorithms for the prediction of Fog (Surface Visibility ≤ 1000 m) and dense fog ((Surface Visibility ≤ 200 m) for the lead time of 6 to 1 h

Full size image

Also, the statistical skill score like hit rate is more than 87%, selectivity > 80%, and a high F1 Score, which signifies the usability of the proposed models in fog prediction specific to the Patna airport.

4.3 Prediction of dense Fog (surface visibility ≤ 200 m) by using ML (machine learning) approaches

Figure 6 provides a thorough examination of various ML (Machine Learning) strategies for predicting dense fog (surface visibility ≤ 200 m) on the test dataset for the lead time of 02 h. It suggested stacked ensemble approaches at level 1 outperform the six base models at level 0 just as the algorithms outlined before in Sect. 4.2. The results have also improved significantly as compared to Sect. 4.2. The stacked ensemble approaches at level 1 have also surpassed level 0 (GBM, XGBoost, GLM, XRT, DRF, and Feed Forward Neural Network) ML approaches in terms of metric performance (shown in Fig. 7) and statistical skill score (shown in Fig. 8) when predicting dense fog (surface visibility ≤ 200 m). Compared to the performance in dense fog (i.e. surface visibility ≤ 200 m), all of the metrics and statistical skill scores show considerable improvements for the lead time of prediction from 6 to 1 h over the test dataset. In the range of a six-hour to a one-hour lead time, the accuracy varies between 91.2 and 96.7%. This results is important for the efficient management of one of the IGP (Indo Gangetic Plain) region’s busiest airports and the timely scheduling of aircraft. This means that the most cost-effective operations could take place on an exceptionally busy day for the airlines (i.e., foggy days). Empirical evidence shows that the AUC gets better as the time stamp gets smaller, with the best values being between 0.9162 and 0.9856 for the prediction dense fog. Also, the statistical skill score like hit rate is more than 91.8%, selectivity > 80%, and a high F1 Score (0.9161 (for 6 h lead time) to 0.9856 (for 6 h lead time), which signifies the usability of the proposed models in fog prediction specific to the Patna airport.

5 Discussion

Investigating the potential of ML (Machine Learning) approaches for regional or local low visibility (fog/dense fog) prediction in the context of Now cast (with a lead time of 06 h to 01 h) is the focus of this research. To put it another way, the suggested Ensemble ML approaches at level 1 perform significantly better than the corresponding base models at level 0. Predictions of fog (surface visibility ≤ 1000 m) receive an F1 score between 0.90 and 0.92 ( for a lead time of 06 h to 01 h). Also, F1 scores range from 0.95 (06 h lead time) to 0.98 (lead time of 01 h) for the prediction of dense fog (surface visibility ≤ 1000 m). This result confirms the findings of [21, 53, 54] which found that tree-based algorithms perform exceptionally well [43, 44]. Moreover, when compared to other forecast difficulties linked with meteorological factors like wind speed, temperature, and even precipitation, the prediction of low-visibility/fog is often more challenging. This is attributable to the extreme local characteristics of fog events. Despite this challenge, any advancement towards accurate techniques to improve the explicit variance of low visibility event prediction is crucial for the characterization and accurate prediction of these events and the respective application of such predictive models in fog modeling. Based on the results, this experimentation with competing ML algorithms led to important conclusions.

  • It is clearly shown that as the time stamp decreases the AUC also increases and its best values vary from 0.8721 to 0.9231 for the prediction of fog (visibility ≤ 1000 m) and varies between 0.9162 to 0.9856 for the prediction of dense fog (visibility ≤ 200 m) for the lead time of 06 h to 01 h.

  • Accordingly, our proposed approaches can be viewed as instruments that harness the power of several relevant ML approaches to develop accurate prediction of local-scale characteristics like fog/low visibility (with intensity scale).

6 Conclusion

Even with the advancement of numerical weather prediction models and the guidance of fog and low visibility prediction models, fog/low visibility prediction remains challenging. Accurately predicting fog requires a deep understanding of the complicated and chaotic atmospheric processes occurring at the boundary layer over a very short time and domain scale. The complexity involved in fog prediction is also linked to the inherent limitations in understanding micro-scale factors that contribute to the initiation, intensification, uplift, and dispersion of fog in the area of interest. As a result, accurate fog prediction is necessary for the smooth operation of aviation services. This article presents an alternative to traditional low visibility and fog forecasting techniques based on historical data, allowing for the accurate and precise prediction of fog/low visibility (with an intensity scale) at specific locations. This study aims to fill the gaps in the regional outlook and explore a variety of options. Therefore, the current research compares various ML algorithms for supervised classification, such as GBM, DRF, GLM, XRT, XGBoost, and ANN based learning at level 0, that model data in various ways, and a stacked ensembling of level 0 algorithms at level 1 for the best prediction of fog (surface visibility ≤ 1000 m) and dense fog (surface visibility ≤ 200 m). The proposed algorithm (model) optimally balances the benefits of three ML meta-algorithms. In order to increase the effectiveness of the model's metrics and the statistical skill score, bagging, boosting, and stacking are employed. Increasing predictability through stacking and decreasing bias through boosting. It's important to identify the most effective prediction algorithms when doing so. A nowcasting system tailored to JPNI Airport Patna has been developed using the stacked ensembling concept we've been discussing. This method's main benefit is that it generates output that may be used by the final consumers—air traffic controllers, airline managers, and so on—to determine exactly what it is they need.