Keywords

The Need for Short-Range Forecasts

Utilities and Independent Transmission Operators (ITOs) depend on accurate forecasts for the next hours to several weeks. They often need to plan operation and maintenance outages weeks to months ahead. Weather events may make it difficult to maintain power lines, wind turbines and other infrastructure. If they can plan around events such as lightning storms and high wind events, the safety of the workers and efficiency of the maintenance can be greatly improved.

Weather also impacts day ahead planning of how to commit units. Although weather impacts all types of power production, it actually also drives the renewable units. It is important to be able to forecast the wind, solar and hydro power available the next day, or often, over the next several days too. The marginal cost to run these renewable resources is quite low and it is economically advantageous to allocate as much power from those units as possible. But overallocation of those units when the wind, irradiance or water power is not available could lead to using much more expensive reserve units in real time. Thus it is critical to produce high quality wind and solar power forecasts.

For timescales less than a day, it is important to know the most recent update to the forecast in order to balance the load in close to real time. Sharp up or down ramps in renewable energy must be balanced with the reserve units if the grid operators were not expecting them.

Overview of Scales

To forecast across these scales from minutes to a few weeks, meteorologists must combine forecast methods that are most appropriate to each scale. Figure 7.1 illustrates the big picture of the types of systems that may be useful for such forecasts. Figure 7.1 focuses on the short time range where the stakeholders require optimal accuracy. That figure demonstrates that observation-based nowcasting, which refers to the first few hours of the forecast, provides a much more accurate forecast in the short range, but its skill drops off rapidly. Numerical weather prediction (NWP) becomes more important at about three hours and provides value out to about two weeks. Because NWP run at high resolution over a sizable domain requires on the order of hours to run on supercomputers and often requires spin-up time, it is not available for real-time use in the shortest ranges. Modern methods of forecasting renewable energy output employing postprocessing methods to blend disparate models or ensembles are shown to greatly improve the forecast skill (Giebel and Kariniotakis 2007; Monteiro et al. 2009; Mahoney et al. 2012; Ahlstrom et al. 2013; Orwig et al. 2014; Tuohy et al. 2015). Figure 7.1 indicates that this blending can provide value beyond that of the input models.

Fig. 7.1
figure 1

Blending of NWP models with observation-based nowcasting enables optimization of the short-range forecast

Full size image

Nowcasting

The shortest time frames of the first several hours benefit from forecasting based on data sensors that determine the current situation of the variable of interest. For instance, having in situ measurements of wind speed or solar radiation in the field allow us to train statistical learning or artificial intelligence models to recognize current conditions and likely changes. Over the first 15–45 minutes, it is often difficult to beat a persistence forecast. If we know the wind now, the best forecast in the first few minutes is that there is no change. For solar radiation, we often use a “smart persistence” where we expect the cloud cover to stay the same, but recognize that the solar angle will change. For solar energy, sky imaging traces cloud conditions in real time, which allows us to use motion vectors derived from image processing succeeding images, or if there is a co-located wind profiler, one can project the cloud motion for the next few steps (Kleissl 2010; Chu et al. 2013; Nguyen and Kleissl 2014; Peng et al. 2015). Cloud imager-based forecasts are effective for about 15–30 minutes.

Statistical learning methods have been shown to be effective from about 30 minutes to around three hours. Such methods may consist of using techniques such as artificial neural networks (ANN—Mellit 2008; Wang et al. 2012), autoregressive models (Hassanzadeh et al. 2010; Yang et al. 2012), Markov process models (Morf 2014) or support vector machines (Sharma et al. 2011; Bouzerdoum et al. 2013) to recognize patterns in the changes of wind speed or solar radiation. Blended techniques, such as that of Pedro and Coimbra (2012), that use a genetic algorithm to optimize an ANN have also been effective. In addition, new methods blend weather observations with irradiance observations and use clustering techniques to identify regimes and then train ANN for the individual regimes (Kazor and Hering 2015), which was shown to improve upon non-regime-dependent ANN and upon smart persistence (McCandless et al. 2016a). Recent work has added satellite data to this type of forecasting (McCandless et al. 2016b). Some methods also predict the variability of the resource (McCandless et al. 2015).

In the time range from about an hour out to six hours, cloud motion vectors derived from satellite data are often used to forecast solar irradiance (Miller et al. 2013). When a satellite observes the cloud cover and the forecast system then advects it using motion vectors derived from successive images or uses observed or modeled data to advect those clouds, the derived liquid water path can be used to provide good estimates of irradiance attenuation. Satellite-based methods depend on the satellite data being received and processed before being used in the motion vector models, which take on the order of 30 minutes after observation. They also do not account for cloud development or dissipation, so the forecasts of individual clouds are only accurate for a limited period of time, beginning to degrade after the first couple hours.

Wind energy forecasting at these scales of the first few hours can be improved by other methods of remote sensing. Mahoney et al. (2012) describe the Variational Doppler Radar Analysis System (VDRAS) that assimilates radar data into a cloud-resolving model to better predict winds. Because that model does not include the full physics, it can be updated at frequencies as high as every 15 minutes. That work showed that in case studies, the winds could be well predicted for the first two hours and could identify weather ramps as they approach. This method, however, relies on having radars sited in close proximity to the wind farms.

Numerical Weather Prediction

Beyond about three to six hours, the workhorse of forecasting is NWP. NWP consists of the integration of the nonlinear partial differential equations governing atmospheric flow and includes appropriate models for the physics of clouds, radiation, turbulence, land surface conditions and more (Warner 2011). As computer power has advanced, so has our capability to provide higher resolution simulations in closer to real time. The national centers now run very short-range simulations at about three kilometers horizontal resolution over limited regions (such as over the USA) as often as hourly. Global models have necessarily coarser resolution and run less frequently. As of mid-2017, the US Global Forecast System is run every six hours at 13 kilometers resolution with hourly output for the first five days and at 70 kilometers out to 16 days. The European Center for Medium Forecast is run twice a day at nine kilometers resolution for 16 days with output at three hours temporal resolution. The national centers are continually updating their model resolutions, lengths of simulations and frequency of the runs as computer power is upgraded. In addition, they are including ensemble runs, which provide probabilistic information as well as improving upon the deterministic forecast. As mentioned in the introduction and elaborated later, the sensitivity to initial conditions is what partly limits predictability, which necessitates running ensembles of models.

It is important to assimilate observations of weather data from the global networks to provide the best possible initial condition to the runs. To improve forecasts at specific points, such as at a wind farm, it is advantageous to also assimilate specialized data (such as wind speed measurements) at that farm. Mahoney et al. (2012) and Wilczak et al. (2015) provide evidence that assimilating local wind farm data can improve the NWP forecasts. In a case study, Cheng et al. (2017) show that real-time four-dimensional data assimilation can reduce the mean absolute error in the forecast by 30–40% in the first three hours. Kosovic et al. (2017) indicate that such local data assimilation can significantly improve forecasts over the national models for the first 15 hours. Versions of the models are now being built to specifically improve upon predicting variables of particular importance to energy, such as wind (Wilczak et al. 2015), solar (Jimenez et al. 2016) and hydro (Gochis et al. 2014), although the use of national hydrological models is still in its infancy.

Blending the Forecasts and Predicting Power

Modern forecasting includes postprocessing the NWP output and forecast blending to improve upon the results. At a basic level, a multivariate statistical regression model known as model output statistics (MOS) is applied to remove biases (Glahn and Lowry 1972). More complex methods (such as ANN, autoregressive models and others) are also used to provide nonlinear corrections to models (Myers et al. 2011; Myers and Linden 2011; Giebel and Kariniotakis 2007; Pelland et al. 2013). Ensemble MOS (Wilks and Hamill 2007) not only corrects the individual models but also optimizes weights for blending forecasts from multiple models. It is typical to blend multiple models together using some statistical learning technique to produce a better forecast than any single model could produce consistently, often with a 10–15% improvement over the best model forecast (Mahoney et al. 2012; Myers et al. 2011). Figure 7.2 displays an example of wind speed forecasts using National Center for Atmospheric Research (NCAR’s) Dynamic Integrated foreCast (DICast®) system to blend multiple models, with the blended forecast showing a lower average root mean square error than any input model.

Fig. 7.2
figure 2

Mean root mean square error for wind speed forecasts at METAR sites over the contiguous USA from multiple models and the DICast forecast (red) for the month from 5 October to 5 November, 2015

Full size image

Probabilistic Forecasts and the Analog Ensemble

Utilities and ITOs are also requesting probabilistic information to plan reserves, alleviating transmission bottlenecks, and better planning for renewable operations. To this end, the meteorology sector has been providing probabilistic forecasts. All major national forecasting centers now run ensemble forecasts. The rate of spread of the members of the ensemble quantifies the uncertainty. Various postprocessing methods are used to remove bias, sharpen the probability density function and calibrate the spread, or reliability, of the forecast.

Another interesting postprocessing method is the analog ensemble (AnEn) technique. In this case, however, rather than running multiple models, the historical output from a single, often high resolution, model is used to generate the ensemble. A search of the historical forecast records is made to identify the forecasts that are most similar to the current forecast. We then compare the forecasts to the corresponding observations. Those observations then become the analog ensemble. This method is effective at both improving on the deterministic forecast and using the multiple analogs to form an ensemble that can be used to quantify the uncertainty of the forecast (Delle Monache et al. 2013; Haupt and Delle Monache 2014; Alessandrini et al. 2015).

Accuracy of forecasts has been improving steadily with some areas now seeing single digit errors in terms of percentage of capacity at a wind farm (Orwig et al. 2014; Haupt and Mahoney 2015). These improvements have stemmed from including observations in the immediate vicinity of the resource, both in the nowcasting and assimilated into the NWP models, as well as better methods of blending multiple models for the appropriate timescales. Solar power predictions have not been a focus for very long, but rapid improvement is also happening here (Lorenz et al. 2009; 2014; Tuohy et al. 2015; Haupt et al. 2017).